[
https://issues.apache.org/jira/browse/SPARK-59347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho reassigned SPARK-59347:
---------------------------------
Assignee: Anish Mahto
> SCD2 Ignore-null support; union microbatch with target table during
> preprocessing
> ---------------------------------------------------------------------------------
>
> Key: SPARK-59347
> URL: https://issues.apache.org/jira/browse/SPARK-59347
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 4.4.0
> Reporter: Anish Mahto
> Assignee: Anish Mahto
> Priority: Major
> Labels: pull-request-available
>
> Part of the contract for the SCD2 version map is the keys in the version map
> should track exactly with the data column names.
> Complexity arises when case insensitivity is used, and the incoming
> microbatch has a different case-spelling for certain columns than the
> persisted target/aux tables.
> When `Scd2ForeachBatchHandler` does a [union with the incoming microbatch
> rows and the existing target/aux table
> rows|https://github.com/apache/spark/blob/master/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2ForeachBatchHandler.scala#L104-L106],
> due to ordering of the union operators, the resulting DF will respect the
> case-spelling on the incoming microbatch.
> But when this dataframe is eventually merged back into the target/aux tables,
> the existing case-spelling on those tables will win. This is no longer
> acceptable since the version map will be constructed by referencing column
> names in the microbatch dataframe. If those column names change, the version
> map's keys become inconsistent.
> The proposal is to union the target schema onto the microbatch dataframe
> during microbatch preprocessing, and before we eventually construct the
> version map. This appraoch has the added benefit that the version map will
> now see rows reductively schema evolved (i.e dropped) from the microbatch.
> This change is a no-op behaviorally, and should have no observable side
> effect to users.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
