xushiyan commented on code in PR #7856:
URL: https://github.com/apache/hudi/pull/7856#discussion_r1096769218
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -480,6 +486,7 @@ object HoodieSparkSqlWriter {
|Table's schema ${latestTableSchema.toString(true)}
|""".stripMargin)
throw new SchemaCompatibilityException("Incoming batch schema is
not compatible with the table's one")
+ }
Review Comment:
/nit fix indentation
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -415,6 +415,7 @@ object HoodieSparkSqlWriter {
val allowAutoEvolutionColumnDrop =
opts.getOrDefault(HoodieWriteConfig.SCHEMA_ALLOW_AUTO_EVOLUTION_COLUMN_DROP.key,
HoodieWriteConfig.SCHEMA_ALLOW_AUTO_EVOLUTION_COLUMN_DROP.defaultValue).toBoolean
+ val mergeIntoWrites =
opts.getOrDefault(DataSourceWriteOptions.MERGE_INTO_WRITES,
DataSourceWriteOptions.DEFAULT_MERGE_INTO_WRITES).toBoolean
Review Comment:
we need to infer this when user is using merge into. it's gonna be pretty
bad exp if someone uses merge into and still need to set the config = true.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -463,6 +463,9 @@ object DataSourceWriteOptions {
val RECONCILE_SCHEMA: ConfigProperty[Boolean] =
HoodieCommonConfig.RECONCILE_SCHEMA
+ val MERGE_INTO_WRITES = "hoodie.merge.into.writes"
+ val DEFAULT_MERGE_INTO_WRITES = "false"
Review Comment:
why not make it a ConfigProperty
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -415,6 +415,7 @@ object HoodieSparkSqlWriter {
val allowAutoEvolutionColumnDrop =
opts.getOrDefault(HoodieWriteConfig.SCHEMA_ALLOW_AUTO_EVOLUTION_COLUMN_DROP.key,
HoodieWriteConfig.SCHEMA_ALLOW_AUTO_EVOLUTION_COLUMN_DROP.defaultValue).toBoolean
+ val mergeIntoWrites =
opts.getOrDefault(DataSourceWriteOptions.MERGE_INTO_WRITES,
DataSourceWriteOptions.DEFAULT_MERGE_INTO_WRITES).toBoolean
Review Comment:
look into
`org.apache.spark.sql.hudi.command.MergeIntoHoodieTableCommand#buildMergeIntoConfig`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
