[
https://issues.apache.org/jira/browse/SPARK-59343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Mahto updated SPARK-59343:
--------------------------------
Affects Version/s: 4.4.0
(was: 4.3.0)
> SCD2 warn or prevent users from changing ignore-null selection between
> microbatch retries
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-59343
> URL: https://issues.apache.org/jira/browse/SPARK-59343
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 4.4.0
> Reporter: Anish Mahto
> Priority: Major
>
> When an AutoCDC flow partially fails, specifically between merging to the
> auxiliary table and merging to the target table, some of the partially merged
> rows will be pulled again for reconciliation on the next run of the AutoCDC
> flow (which re-runs the same microbatch).
> When that happens, there can be rows with duplicate rows pulled in for
> reconciliation, with sequence numbers. When the rows are truly duplicates,
> the behavior is effectively stable; either the already persisted row wins or
> the incoming row in the re-processed microbatch wins, but both resolve to the
> same values.
> Ignore-null changes how a row in the microbatch is interpreted, and can be
> changed between flow runs. If the ignore-null selection is changed between
> when the microbatch was partially processed and when it is retried, then rows
> can be pulled in for reconciliation with the same sequence as rows in the
> microbatch, but they are not true duplicates - they were ingested with
> different ignore-null selections, and persist a different version map.
> Changing the ignore-null selection between retries for the same microbatch is
> a very narrow edge case, but we should decide between:
> - Documenting that changing ignore-null selection between retries can lead to
> undefined behavior
> - On microbatch retries, implement some kind of check to make sure the
> ignore-null selection is not changing. This will likely require persisting
> additional information
> - On AutoCDC execution failures, re-wrap the exception with a generic AutoCDC
> execution exception, which documents the risk of changing ignore-null
> selection before the current microbatch is succesfully replayed
> - Make retries deterministic when ignore-null selection is changing between
> runs. Ex. stamp existing version maps/rows with a timestamp, and let incoming
> rows always win
> This only affects SCD2, where the auxiliary table contains both tombstones
> and no-op upserts. In SCD1 the auxiliary table only contains tombstones,
> which always have a null version map.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
