AnishMahto opened a new pull request, #59309: URL: https://github.com/apache/spark/pull/59309
### What changes were proposed in this pull request? Until AutoCDC ignore-null is fully built E2E for both SCD1 and SCD2, we should explicitly gate its usage. In ignore-null's current partial implementation state, if it runs then we may produce partial results that get persisted against existing AutoCDC target/aux tables. Both the current (partially complete) implementation and API support for ignore-null is merged into Spark master, and is planned to go out as of Spark 4.4. Once the implementation is complete, this gating can be removed. **Gating is intentionally done as a `GraphValidation`, to avoid reverting ignore-null Python/SQL API and tests, which are currently unreleased and only in Spark master.** ### Why are the changes needed? Gate incomplete AutoCDC ignore-null implementation. ### Does this PR introduce _any_ user-facing change? No, currently the user facing ignore-null APIs are unreleased. ### How was this patch tested? Unit tests. ### Was this patch authored or co-authored using generative AI tooling? Co-authored with Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
