AnishMahto opened a new pull request, #56683:
URL: https://github.com/apache/spark/pull/56683
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR:
https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g.,
'[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a
faster review.
7. If you want to add a new configuration, please read the guideline first
for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the
guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
`requireDestinationSupportsRowLevelOps` is a check executed during AutoCDC
flow execution today that asserts the configured catalog implements the
`SupportsRowLevelOperations` DSv2 interface.
This check is unnecessarily strict; the intention was to validate that the
configured catalog supports MERGE (which requires row level operations) but
indifferent to DSv2 vs DSv1 API.
The check however by construction also asserts the catalog is implementing
row level operations via DSv2 specifically. There are real catalogs connectors
(ex. Delta) that do indeed support MERGE but not via DSv2 contract, and instead
through custom analyzer rules.
The proposal is to drop this unnecessarily strict check, and instead rely on
execution-time failure/validation for MERGE support.
### Why are the changes needed?
Enables support for catalogs/connectors that implement MERGE, but do not
implement via DSv2.
### Does this PR introduce _any_ user-facing change?
Yes, ungates additional catalog/connectors for use with AutoCDC. For
incompatible catalogs, an exception is still thrown during flow execution, but
now specifically when attempting to execute MERGE and with a different error
message.
### How was this patch tested?
Updated test in `AutoCdcScd1SinglePipelineSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Yes, co-authored with Claude Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
