AnishMahto opened a new pull request, #59311: URL: https://github.com/apache/spark/pull/59311
This is a stacked PR. See incremental diff here: https://github.com/AnishMahto/spark/compare/scd1-per-leaf-metadata-schema...scd1-version-map-construction See previous PR in stack here: https://github.com/apache/spark/pull/58973 ### What changes were proposed in this pull request? Concretely define the contract for the contents of the version map; the format the keys are written in, which leaves get an entry, and what the stored sequence represents when null or not null. Then implement the logic to construct a version map column expression as per the contract, given an upsert-representing row. Delete-representing rows need no equivalent operation, as they always hold null version maps ### Why are the changes needed? AutoCDC SCD1 ignore-null support. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Unit tested in `Scd1VersionMapSuite`. ### Was this patch authored or co-authored using generative AI tooling? Co-authored with Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
