hudi-agent commented on code in PR #18650:
URL: https://github.com/apache/hudi/pull/18650#discussion_r3461614528
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala:
##########
@@ -159,11 +156,18 @@ class DefaultSource extends RelationProvider
mode: SaveMode,
optParams: Map[String, String],
df: DataFrame): BaseRelation = {
+ // Pull `hoodie.*` and `spark.hoodie.*` from SparkConf, normalize
`spark.hoodie.*` to
+ // canonical `hoodie.*`, and merge with explicit options (explicit options
win). This
+ // mirrors what the read createRelation already does, so configs like
+ // `--conf spark.hoodie.datasource.hive_sync.use_spark_catalog=true` are
honored on
+ // writes too. `HoodieSparkSqlWriter` and downstream callers see only
canonical keys.
+ val effectiveOpts =
+ DataSourceOptionsHelper.collectHoodieAndSparkHoodieConfs(sqlContext,
optParams)
Review Comment:
🤖 The filter inside `collectHoodieAndSparkHoodieConfs` matches both
`hoodie.*` and `spark.hoodie.*`, so this now forwards plain `hoodie.*`
SparkConf entries to the write path as well — not just the `spark.hoodie.*`
ones the PR title/description focus on. That's a broader behavior change: a
user who set, say, `hoodie.metadata.enable` or a read-oriented `hoodie.*` in
SparkConf for reads will now see it applied to writes too. Is that intended
parity, or should the write path only pick up the `spark.hoodie.*` prefix?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
