[
https://issues.apache.org/jira/browse/SPARK-60368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60368:
-----------------------------------
Labels: pull-request-available (was: )
> SCD1 Ignore-null support; Dispatch leaf-level reconciliation and target merge
> -----------------------------------------------------------------------------
>
> Key: SPARK-60368
> URL: https://issues.apache.org/jira/browse/SPARK-60368
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 4.4.0
> Reporter: Anish Mahto
> Priority: Major
> Labels: pull-request-available
>
> Connects the previously introduced SCD1 ignore-null (leaf-level) components,
> so that a microbatch is reconciled and merged end to end while maintaining
> each target row's version map.
> Concretely, `Scd1BatchProcessor` now uses
> `Scd1LeafLevelReconciliation.reconcileMicrobatch` and
> `Scd1LeafLevelTargetMerge` whenever an ignore-null selection is set,
> including one that selects no columns. Without a selection, it keeps the
> row-level path, so existing flows behave exactly as before.
> Clients still can't use ignore-null, because
> `GraphValidations.validateNoAutoCdcIgnoreNullSelection` rejects it. This
> change adds end-to-end tests that bypass graph validation by constructing
> `Scd1BatchProcessor` directly. They cover ignore-null selection changes
> between microbatches and retries after a partial merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
