[
https://issues.apache.org/jira/browse/HUDI-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-3889:
--------------------------------------
Fix Version/s: 0.12.0
> Do not validate table config if save mode is set to Overwrite
> -------------------------------------------------------------
>
> Key: HUDI-3889
> URL: https://issues.apache.org/jira/browse/HUDI-3889
> Project: Apache Hudi
> Issue Type: Task
> Components: spark
> Reporter: sivabalan narayanan
> Priority: Critical
> Fix For: 0.12.0
>
>
> with spark datasource write, if Overwrite is set as save mode, we should not
> do table config validation
>
> {code:java}
> scala> df.write.format("hudi").
> | option(PRECOMBINE_FIELD_OPT_KEY, "tpep_dropoff_datetime").
> | option(RECORDKEY_FIELD_OPT_KEY, "tpep_pickup_datetime").
> | option(PARTITIONPATH_FIELD_OPT_KEY, "date_col").
> | option(TABLE_NAME, "hudi_tbl1").
> | option("hoodie.embed.timeline.server","false").
> | mode(Overwrite).
> | save(basePath)
> warning: one deprecation; for details, enable `:setting -deprecation' or
> `:replay -deprecation'
> org.apache.hudi.exception.HoodieException: Config conflict(key current
> value existing value):
> RecordKey: tpep_pickup_datetime id
> PreCombineKey: tpep_dropoff_datetime created_at
> at
> org.apache.hudi.HoodieWriterUtils$.validateTableConfig(HoodieWriterUtils.scala:161)
> at
> org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:87)
> at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:161)
> at
> org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
