Balaji Varadarajan created HUDI-1153:
----------------------------------------
Summary: Spark DataSource and Streaming Write must fail when
operation type is misconfigured
Key: HUDI-1153
URL: https://issues.apache.org/jira/browse/HUDI-1153
Project: Apache Hudi
Issue Type: Improvement
Components: Spark Integration
Reporter: Balaji Varadarajan
Fix For: 0.6.1
Context: [https://github.com/apache/hudi/issues/1902#issuecomment-669698259]
If you look at DataSourceUtils.java,
[https://github.com/apache/hudi/blob/master/hudi-spark/src/main/java/org/apache/hudi/DataSourceUtils.java#L257]
we are string comparison to determine operation type which is a bad idea and a
typo could result in "upsert" being used silently.
Just like
[https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java#L187]
being used for DeltaStreamer, we need similar enums defined in
DataSourceOptions.scala for OPERATION_OPT_KEY but care must be taken to ensure
we do not cause backwards compatibility issue by changing the property value.
In other words, we need to retain the lower case values ("bulk_insert",
"insert" and "upsert") but make it an enum.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)