[GitHub] [incubator-hudi] AnkitMishra3220 opened a new issue #1591: Skip Null Value on Upsert

[GitBox]

AnkitMishra3220 opened a new issue #1591:
URL: https://github.com/apache/incubator-hudi/issues/1591


   **_Tips before filing an issue_**
   
   - Have you gone through our 
[FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)? Yes
   
   - Join the mailing list to engage in conversations and get faster support at 
dev-subscr...@hudi.apache.org.
   
   - If you have triaged this as a bug, then file an 
[issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
   
   **Describe the problem you faced**
   
   My use-case is to design data lakes from like events which I am getting from 
Kafka. It also requires
   functionality '**Skip Null Value on Upsert**' because we have data for 
fields specific to events.   
   
   Can we skip null values on Upsert ? I have not found any switch for this 
functionality. 
    
   A clear and concise description of the problem.
    Want to achieve **Skip null value on upsert** functionality. 
   
   **To Reproduce**
   
   Steps to reproduce the behavior:
   
   1.  Insert Data to Hudi Table hudi_trips_snapshot
   
   
df.write.format("hudi").options(getQuickstartWriteConfigs).option(PRECOMBINE_FIELD_OPT_KEY,
 "ts").option(RECORDKEY_FIELD_OPT_KEY, 
"uuid").option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").option(TABLE_NAME, 
tableName). mode(Overwrite).save(basePath)
   
   val tripsSnapshotDF = spark.read.format("hudi").load(basePath + "/*/*/*/*")
      tripsSnapshotDF.createOrReplaceTempView("hudi_trips_snapshot")
   
   <img width="1234" alt="Screenshot 2020-05-04 at 8 22 55 PM" 
src="https://user-images.githubusercontent.com/40383808/80984339-59e39580-8e4b-11ea-8f29-640bb1a6dd09.png";>
   
   
   2. Create a Dataframe which contain null values for one column and updated 
values for another column.
   
   <img width="1164" alt="Screenshot 2020-05-04 at 8 22 43 PM" 
src="https://user-images.githubusercontent.com/40383808/80984400-6ff15600-8e4b-11ea-9058-96a58c597a76.png";>
   
   
   3. Update the Hudi Table with new dataframe. 
   
   
updateDf.write.format("org.apache.hudi").options(getQuickstartWriteConfigs).option(PRECOMBINE_FIELD_OPT_KEY,
 "ts").option(RECORDKEY_FIELD_OPT_KEY, 
"uuid").option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").option(TABLE_NAME, 
tableName).mode(Append).save(basePath)
   
   4. Check updated table. It is providing updated values but not skipping null 
values on Upsert. 
   
   <img width="1184" alt="Screenshot 2020-05-04 at 8 22 06 PM" 
src="https://user-images.githubusercontent.com/40383808/80984454-813a6280-8e4b-11ea-8b4d-aefdcefecaa8.png";>
   
   
   **Expected behavior**
    There should be some switch/property present for functionality '**Skip Null 
Value on Upsert**'
   
   **Environment Description**
   
   * Hudi version : 0.5.2-incubating
   
   * Spark version : 2.4.4
   
   * Hive version :  3.1.2
   
   * Storage (HDFS/S3/GCS..) : S3 
   
   * Running on Docker? (yes/no) : No
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org