Anish Mahto created SPARK-59358:
-----------------------------------
Summary: SCD2 Ignore-null suport; Project missing columns to
microbatch during preprocessMicrobatch
Key: SPARK-59358
URL: https://issues.apache.org/jira/browse/SPARK-59358
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.4.0
Reporter: Anish Mahto
During SCD2 microbatch reconciliation, it's possible an incoming microbatch
does not contain a column that the target table already contains - this can
happen if the upstream change data feed undergoes reductive schema evolution
and drops a column.
When this happens, in `Scd2ForeachBatchHandler` we union the microbatch with
the target/aux tables using `allowMissingColumns=true`. The microbatch then
desirably gains null entries for columns it does not contain but the target
table does contain (and vice versa).
With ignore-null, we need to do the microbatch projection specifically before
the version map is constructed, so that those projected null columns can obey
the active ignore-null selection during version map construction.
In isolation, this is a no-op change; we project null columns onto the
microbatch the same as before, we just do it earlier during microbatch
reconciliation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
