gengliangwang opened a new pull request, #59119: URL: https://github.com/apache/spark/pull/59119
### What changes were proposed in this pull request? Add `TableCatalog.changelogStateOptionKeys()`, defaulting to `tableStateOptionKeys()`, and pass only these declared options to the existing catalog-level `loadChangelog` API. Keep all options on each relation for scan planning. Reuse changelog metadata within an analysis or refresh pass by `(catalog, identifier, ChangelogContext, changelog state options)`. Preserve CDC context and post-processing when refreshing cached data or re-resolving DataFrame temporary views, and reject metadata changes that invalidate an analyzed CDC rewrite. ### Why are the changes needed? [SPARK-59841](https://issues.apache.org/jira/browse/SPARK-59841): `loadTable` uses declared state options, while `loadChangelog` receives all options and does not share the same metadata-reuse semantics. Reads that differ only in scan options should reuse CDC metadata; different ranges or state options must remain separate. Retaining direct catalog-level CDC loading also lets a connector read changes from versions 1 through 10 when the latest version is 100 without Spark first loading the current table snapshot. ### Does this PR introduce _any_ user-facing change? Yes. `loadChangelog` now receives only options declared by `changelogStateOptionKeys()`. Existing CDC connectors must declare any options required during metadata loading and obtain Spark-parsed range and processing parameters from `ChangelogContext`. Scan-only options remain available to `Changelog.newScanBuilder`. The new method defaults to the table state keys. An override supplies the complete CDC state-key set, including any table keys the connector still needs. The new API is marked `@since 4.3.0`. ### How was this patch tested? Added coverage for inherited and independent CDC keys, case handling, load deduplication, range isolation, retained scan options, bounded historical reads, temporary views, cache refresh, and incompatible CDC metadata changes. Passed Java Checkstyle for the changed Java source and Scala style checks for all 14 changed/new Scala files using the repository configurations and cached build-tool dependencies. Whitespace, added-source ASCII, and error JSON checks also passed. Attempted the focused suites: ```text build/sbt -batch \ 'catalyst/testOnly *CatalogV2UtilSuite *TableLookupCacheSuite *ChangelogTableSuite' \ 'sql/testOnly *ChangelogResolutionSuite *ChangelogEndToEndSuite *ResolveChangelogTable*Suite *DataSourceV2OptionSuite' ``` The build stopped before compilation: the configured Maven mirror returned `forbidden` for `at.yawk.lz4:lz4-java:1.12.0`. The exact JAR is not cached locally, and an attempt to access Maven Central failed DNS resolution. The functional suites have not run. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
