rangareddy commented on issue #17328: URL: https://github.com/apache/hudi/issues/17328#issuecomment-5351232106
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-8665). **Findings: needs a `spark-sql` reproduction on current `master` before it can be dispositioned.** The report is specific and actionable as written - `SET hoodie.metadata.enable`, `hoodie.metadata.index.column.stats.enable` and `hoodie.enable.data.skipping` only take effect when issued before `CREATE TABLE`, and are ignored inside the `OPTIONS` clause. What it needs is confirmation against a current build, because the config-resolution path has changed substantially since it was filed. I did not want to guess at this one from code alone: whether a `SET` loses to a table property, an `OPTIONS` entry, or an inferred default depends on the precedence order that `HoodieCatalog` / `HoodieWriterUtils` apply at create time, and that is not decidable by reading without running it. One pointer that may be the same root cause, since it is independently confirmed: #17398 (HUDI-9153) documents that `HoodieConfig.setDefaults` iterates `Class#getDeclaredFields()` and writes each resolved default into `props` as it goes, so a `ConfigProperty` infer function calling `contains(...)` cannot tell "the user set this" from "we just defaulted it". That is exactly the failure shape described here. Worth checking whether these two are the same bug before doing independent work on them. Keeping this open; a repro on a recent version (ideally without the 1TB dataset) would unblock it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
