[ https://issues.apache.org/jira/browse/SPARK-29238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ARUN KINDRA updated SPARK-29238: -------------------------------- Description: Hi I'm trying to add a newColumn to an empty DF but I don't see the new column is getting added. Dataset<Row> newDF = sparkSession.emptyDataFrame(); Dataset<Row> newDf_DateConverted = newDF.withColumn("year", lit("2019")); newDf_DateConverted.show(); *Output:* +-------------------+ |year| +-------------------+ +-------------------+ Basically, I am reading one HBase table and if there is no data into the table i get empty JavaPairRDD. I convert that JavaPairRDD to JavaRDD<Row> and then using schema I convert into a DF. Later I need to insert that DF values into the Hive external partitioned table. But when there is no data in HBase i am not seeing the partition getting created. So, I tried the above 2 lines to code where I have empty DF and I try to add partitioned column into it, it doesn't work. was: Hi I'm trying to add a newColumn to an empty DF but I don't see the new column is getting added. Dataset<Row> newDF = sparkSession.emptyDataFrame(); Dataset<Row> newDf_DateConverted = newDF.withColumn("year", expr("2019")); newDf_DateConverted.show(); *Output:* +-------------------+ |year| +-------------------+ +-------------------+ Basically, I am reading one HBase table and if there is no data into the table i get empty JavaPairRDD. I convert that JavaPairRDD to JavaRDD<Row> and then using schema I convert into a DF. Later I need to insert that DF values into the Hive external partitioned table. But when there is no data in HBase i am not seeing the partition getting created. So, I tried the above 2 lines to code where I have empty DF and I try to add partitioned column into it, it doesn't work. > Add newColumn using withColumn to an empty Dataframe > ---------------------------------------------------- > > Key: SPARK-29238 > URL: https://issues.apache.org/jira/browse/SPARK-29238 > Project: Spark > Issue Type: Question > Components: SQL > Affects Versions: 2.3.2 > Reporter: ARUN KINDRA > Priority: Major > > Hi > I'm trying to add a newColumn to an empty DF but I don't see the new column > is getting added. > Dataset<Row> newDF = sparkSession.emptyDataFrame(); > Dataset<Row> newDf_DateConverted = newDF.withColumn("year", lit("2019")); > newDf_DateConverted.show(); > *Output:* > +-------------------+ > |year| > +-------------------+ > +-------------------+ > > Basically, I am reading one HBase table and if there is no data into the > table i get empty JavaPairRDD. I convert that JavaPairRDD to JavaRDD<Row> and > then using schema I convert into a DF. Later I need to insert that DF values > into the Hive external partitioned table. But when there is no data in HBase > i am not seeing the partition getting created. > So, I tried the above 2 lines to code where I have empty DF and I try to add > partitioned column into it, it doesn't work. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org