sivabalan narayanan created HUDI-3889:
-----------------------------------------
Summary: Do not validate table config if save mode is set to
Overwrite
Key: HUDI-3889
URL: https://issues.apache.org/jira/browse/HUDI-3889
Project: Apache Hudi
Issue Type: Task
Components: spark
Reporter: sivabalan narayanan
with spark datasource write, if Overwrite is set as save mode, we should not do
table config validation
{code:java}
scala> df.write.format("hudi").
| option(PRECOMBINE_FIELD_OPT_KEY, "tpep_dropoff_datetime").
| option(RECORDKEY_FIELD_OPT_KEY, "tpep_pickup_datetime").
| option(PARTITIONPATH_FIELD_OPT_KEY, "date_col").
| option(TABLE_NAME, "hudi_tbl1").
| option("hoodie.embed.timeline.server","false").
| mode(Overwrite).
| save(basePath)
warning: one deprecation; for details, enable `:setting -deprecation' or
`:replay -deprecation'
org.apache.hudi.exception.HoodieException: Config conflict(key current value
existing value):
RecordKey: tpep_pickup_datetime id
PreCombineKey: tpep_dropoff_datetime created_at
at
org.apache.hudi.HoodieWriterUtils$.validateTableConfig(HoodieWriterUtils.scala:161)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:87)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:161)
at
org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
