[ https://issues.apache.org/jira/browse/SPARK-29238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939379#comment-16939379 ]
Hyukjin Kwon commented on SPARK-29238: -------------------------------------- Since there's no record, there would be anything will be added as a value. There's 0 records so no record that holds the literal. > Add newColumn using withColumn to an empty Dataframe > ---------------------------------------------------- > > Key: SPARK-29238 > URL: https://issues.apache.org/jira/browse/SPARK-29238 > Project: Spark > Issue Type: Question > Components: SQL > Affects Versions: 2.3.2 > Reporter: ARUN KINDRA > Priority: Major > > Hi > I'm trying to add a newColumn to an empty DF but I don't see the new column > is getting added. > Dataset<Row> newDF = sparkSession.emptyDataFrame(); > Dataset<Row> newDf_DateConverted = newDF.withColumn("year", lit("2019")); > newDf_DateConverted.show(); > *Output:* > +-------------------+ > |year| > +-------------------+ > +-------------------+ > > Basically, I am reading a HBase table and if there is no data into the table > i get empty JavaPairRDD. I convert that JavaPairRDD to JavaRDD<Row> and then > using schema I convert into a DF. Later I need to insert that DF values into > the Hive external partitioned table. But when there is no data in HBase i am > not seeing the partition getting created in HDFS. > So, I tried the above 2 lines to code where I have empty DF and I try to add > partitioned column into it, that also doesn't work. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org