AnishMahto opened a new pull request, #58483: URL: https://github.com/apache/spark/pull/58483
This is a stacked PR. See incremental diff here: https://github.com/AnishMahto/spark/compare/SPARK-59164-nested-schema-diff...SPARK-59165-scd2-version-map-schema ### What changes were proposed in this pull request? In order to support ignore-null in SCD2, we need to persist additional metadata per row that tells us which column values were actually authored by the upsert event that created the row, and which columns were unauthored by the same upsert event and instead need to inherit a value from a previous row. This ticket is concerned with simply defining the version map schema/concept, and adding it to the `_cdc_metadata` per row. The version map will not be used yet, but will start materializing in both old and new AutoCDC tables. ### Why are the changes needed? To support the ignore-null feature for AutoCDC SCD2 (as per the [approved SPIP](https://docs.google.com/document/d/10gtGoUOJvY_XoXpJuJ8LGJWPLMkdpgzseiMcKnjDloo/edit?tab=t.0#heading=h.9g6a5f8v6xig)), we need to introduce a new map data structure to persist column authorship. This is a per-row data structure and holds operational metadata, so it naturally fits in `_cdc_metadata`. ### Does this PR introduce _any_ user-facing change? Yes. The (nullable) version map field is added to the `_cdc_metadata` column. That means existing AutoCDC target tables that do not contain this subfield will undergo schema evolution on their next pipeline run after this change. Nested-field schema evolution is supported in SDP as of the previous PR in this stack, and the evolution case is captured via unit tests using the in-memory DSv2 catalog. ### How was this patch tested? Unit tests. ### Was this patch authored or co-authored using generative AI tooling? Yes, using Claude Opus 4.6. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
