[
https://issues.apache.org/jira/browse/SPARK-59343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Mahto updated SPARK-59343:
--------------------------------
Description:
When an AutoCDC flow partially fails, specifically between merging to the
auxiliary table and merging to the target table, some of the partially merged
rows will be pulled again for reconciliation on the next run of the AutoCDC
flow (which re-runs the same microbatch).
When that happens, there can be rows with duplicate rows pulled in for
reconciliation, with sequence numbers. When the rows are truly duplicates, the
behavior is effectively stable; either the already persisted row wins or the
incoming row in the re-processed microbatch wins, but both resolve to the same
values.
Ignore-null changes how a row in the microbatch is interpreted, and can be
changed between flow runs. If the ignore-null selection is changed between when
the microbatch was partially processed and when it is retried, then rows can be
pulled in for reconciliation with the same sequence as rows in the microbatch,
but they are not true duplicates - they were ingested with different
ignore-null selections, and persist a different version map.
Changing the ignore-null selection between retries for the same microbatch is a
very narrow edge case, but we should decide between:
- Documenting that changing ignore-null selection between retries can lead to
undefined behavior
- On microbatch retries, implement some kind of check to make sure the
ignore-null selection is not changing. This will likely require persisting
additional information
- On AutoCDC execution failures, re-wrap the exception with a generic AutoCDC
execution exception, which documents the risk of changing ignore-null selection
before the current microbatch is succesfully replayed
- Make retries deterministic when ignore-null selection is changing between
runs. Ex. stamp existing version maps/rows with a timestamp, and let incoming
rows always win
This only affects SCD2, where the auxiliary table contains both tombstones and
no-op upserts. In SCD1 the auxiliary table only contains tombstones, which
always have a null version map.
was:
When an AutoCDC flow partially fails, specifically between merging to the
auxiliary table and merging to the target table, some of the partially merged
rows will be pulled again for reconciliation on the next run of the AutoCDC
flow (which re-runs the same microbatch).
When that happens, there can be rows with duplicate rows pulled in for
reconciliation, with sequence numbers. When the rows are truly duplicates, the
behavior is effectively stable; either the already persisted row wins or the
incoming row in the re-processed microbatch wins, but both resolve to the same
values.
Ignore-null changes how a row in the microbatch is interpreted, and can be
changed between flow runs. If the ignore-null selection is changed between when
the microbatch was partially processed and when it is retried, then rows can be
pulled in for reconciliation with the same sequence as rows in the microbatch,
but they are not true duplicates - they were ingested with different
ignore-null selections, and persist a different version map.
Changing the ignore-null selection between retries for the same microbatch is a
very narrow edge case, but we should decide between:
- Documenting that changing ignore-null selection between retries can lead to
undefined behavior
- On microbatch retries, implement some kind of check to make sure the
ignore-null selection is not changing. This will likely require persisting
additional information
- On AutoCDC execution failures, re-wrap the exception with a generic AutoCDC
execution exception, which documents the risk of changing ignore-null selection
before the current microbatch is succesfully replayed
- Make retries deterministic when ignore-null selection is changing between
runs. Ex. stamp existing version maps/rows with a timestamp, and let incoming
rows always win
> SCD2 warn or prevent users from changing ignore-null selection between
> microbatch retries
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-59343
> URL: https://issues.apache.org/jira/browse/SPARK-59343
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 4.3.0
> Reporter: Anish Mahto
> Priority: Major
>
> When an AutoCDC flow partially fails, specifically between merging to the
> auxiliary table and merging to the target table, some of the partially merged
> rows will be pulled again for reconciliation on the next run of the AutoCDC
> flow (which re-runs the same microbatch).
> When that happens, there can be rows with duplicate rows pulled in for
> reconciliation, with sequence numbers. When the rows are truly duplicates,
> the behavior is effectively stable; either the already persisted row wins or
> the incoming row in the re-processed microbatch wins, but both resolve to the
> same values.
> Ignore-null changes how a row in the microbatch is interpreted, and can be
> changed between flow runs. If the ignore-null selection is changed between
> when the microbatch was partially processed and when it is retried, then rows
> can be pulled in for reconciliation with the same sequence as rows in the
> microbatch, but they are not true duplicates - they were ingested with
> different ignore-null selections, and persist a different version map.
> Changing the ignore-null selection between retries for the same microbatch is
> a very narrow edge case, but we should decide between:
> - Documenting that changing ignore-null selection between retries can lead to
> undefined behavior
> - On microbatch retries, implement some kind of check to make sure the
> ignore-null selection is not changing. This will likely require persisting
> additional information
> - On AutoCDC execution failures, re-wrap the exception with a generic AutoCDC
> execution exception, which documents the risk of changing ignore-null
> selection before the current microbatch is succesfully replayed
> - Make retries deterministic when ignore-null selection is changing between
> runs. Ex. stamp existing version maps/rows with a timestamp, and let incoming
> rows always win
> This only affects SCD2, where the auxiliary table contains both tombstones
> and no-op upserts. In SCD1 the auxiliary table only contains tombstones,
> which always have a null version map.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
