Anish Mahto created SPARK-60368:
-----------------------------------
Summary: SCD1 Ignore-null support; Dispatch leaf-level
reconciliation and target merge
Key: SPARK-60368
URL: https://issues.apache.org/jira/browse/SPARK-60368
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.4.0
Reporter: Anish Mahto
Connects the previously introduced SCD1 ignore-null (leaf-level) components, so
that a microbatch is reconciled and merged end to end while maintaining each
target row's version map.
Concretely, `Scd1BatchProcessor` now uses
`Scd1LeafLevelReconciliation.reconcileMicrobatch` and
`Scd1LeafLevelTargetMerge` whenever an ignore-null selection is set, including
one that selects no columns. Without a selection, it keeps the row-level path,
so existing flows behave exactly as before.
Clients still can't use ignore-null, because
`GraphValidations.validateNoAutoCdcIgnoreNullSelection` rejects it. This change
adds end-to-end tests that bypass graph validation by constructing
`Scd1BatchProcessor` directly. They cover ignore-null selection changes between
microbatches and retries after a partial merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
