Balaji Varadarajan created HUDI-1153:
----------------------------------------

             Summary: Spark DataSource and Streaming Write must fail when 
operation type is misconfigured
                 Key: HUDI-1153
                 URL: https://issues.apache.org/jira/browse/HUDI-1153
             Project: Apache Hudi
          Issue Type: Improvement
          Components: Spark Integration
            Reporter: Balaji Varadarajan
             Fix For: 0.6.1


Context: [https://github.com/apache/hudi/issues/1902#issuecomment-669698259]

 

If you look at DataSourceUtils.java, 
[https://github.com/apache/hudi/blob/master/hudi-spark/src/main/java/org/apache/hudi/DataSourceUtils.java#L257]
 

we are string comparison to determine operation type which is a bad idea and a 
typo could result in "upsert" being used silently. 

 

Just like 
[https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java#L187]
 being used for DeltaStreamer, we need similar enums defined in 
DataSourceOptions.scala for OPERATION_OPT_KEY but care must be taken to ensure 
we do not cause backwards compatibility issue by changing the property value. 
In other words, we need to retain the lower case values ("bulk_insert", 
"insert" and "upsert") but make it an enum. 

 

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to