SanJSp opened a new pull request, #59596: URL: https://github.com/apache/spark/pull/59596
### What changes were proposed in this pull request? Add a default `Changelog.metadataColumns()` method and expose those columns through the internal `ChangelogTable` wrapper using `SupportsMetadataColumns`. Keep metadata resolution lazy and exclude metadata from `SELECT *`. Mark the CDC post-processing subtree and reject explicit user access to changelog metadata outside that subtree with `UNSUPPORTED_FEATURE.CHANGELOG_METADATA_COLUMN`. ### Why are the changes needed? Addresses [SPARK-60341](https://issues.apache.org/jira/browse/SPARK-60341). Connectors may provide row identity and row version as scan-generated metadata rather than data columns. Spark needs to resolve those references for carryover removal, update detection, and net change computation without adding them to the user-visible changelog schema. Direct user access is not supported by this change. ### Does this PR introduce _any_ user-facing change? Yes. CDC connectors can expose metadata needed for Spark's internal post-processing. Existing connectors retain the default empty metadata list. Changelog metadata remains absent from `SELECT *`, and explicit access through SQL or the Dataset API is rejected. ### How was this patch tested? Local verification passed on current Spark master using Java 17: - `ChangelogTableSuite`: 20 tests passed. - `ChangelogResolutionSuite`: 50 tests passed, including internal metadata resolution, `SELECT *`, SQL/Dataset metadata rejection, subqueries, and combined carryover removal and net changes. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
