Anish Mahto created SPARK-57623:
-----------------------------------
Summary: Drop AutoCDC `requireDestinationSupportsRowLevelOps` Check
Key: SPARK-57623
URL: https://issues.apache.org/jira/browse/SPARK-57623
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.3.0
Reporter: Anish Mahto
`requireDestinationSupportsRowLevelOps` is a check executed during AutoCDC flow
execution today that asserts the configured catalog implements the
`SupportsRowLevelOperations` DSv2 interface.
This check is unnecessarily strict; the intention was to validate that the
configured catalog supports MERGE (which requires row level operations) but
indifferent to DSv2 vs DSv1 API.
The check however by construction also asserts the catalog is implementing row
level operations via DSv2 specifically. There are real catalogs connectors (ex.
Delta) that do indeed support MERGE but not via DSv2 contract, and instead
through custom analyzer rules.
The proposal is to drop this unnecessarily strict check, and instead rely on
execution-time failure/validation for MERGE support.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
