Anish Mahto created SPARK-56927:
-----------------------------------
Summary: Implement SCD1 Batch Processor; Merge Microbatch onto
Target
Key: SPARK-56927
URL: https://issues.apache.org/jira/browse/SPARK-56927
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.2.0, 4.3.0
Reporter: Anish Mahto
{*}Preamble{*}:
The SCD type 1 flow is a foreachBatch streaming query on an input
change-data-feed, and is responsible for reconciling the incoming change data
onto some target table that follows SCD1 replication semantics.
SCD1 flows also maintain an "auxiliary" table to keep track of early-arriving
out-of-order received events state. Each microbatch will need to reconcile
against this auxiliary table as well, and update the auxiliary table's state
appropriately for future microbatches.
{*}Merge Microbatch onto Target{*}:
The final part of an SCD1 reconciliation - merge the reconciled dataframe
(deduplicated, target columns and operational metadata projected, and tombstone
applied) onto the target table.
The target table's contents represent the SCD1 transformation of the change
data feed, for the specified target column selection in the `ChangeArgs`.
Merging onto the target under SCD1 semantics means deleting existing rows that
match against the reconciled microbatch's delete events, and inserting or
updating rows as appropriate for the reconciled microbatch's upsert events.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
