AnishMahto commented on code in PR #58483:
URL: https://github.com/apache/spark/pull/58483#discussion_r3964455895
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala:
##########
@@ -194,6 +194,9 @@ case class Scd2BatchProcessor(
colName = AutoCdcReservedNames.cdcMetadataColName,
col = Scd2BatchProcessor.constructCdcMetadataCol(
recordStartAt = changeArgs.sequencing,
+ // TODO (SPARK-59183): actually populate version map according to
ignore-null selection and
+ // actual authorship in microbatch.
+ versionMap = F.lit(null),
Review Comment:
That’s a good question. First, the comment does not mean version-map
construction will happen at this exact call site. This call will continue
initializing the field to null. A future PR will replace it for incoming
upserts after applying output selection and aligning the microbatch with the
target schema.
In this PR, ignore-null API is not yet exposed to users, so every row must
have a null map. A null map means ignore-null was off and all nulls are
authoritative. These rows will not be retroactively assigned maps.
With ignore-null enabled, there are two cases I'm aware of where an event
does not have a value at all for a column:
1. A column did not exist when an existing event was ingested and was added
later. Its map has no entry for that column. After schema evolution supplies a
null value, “non-null map + null data value + absent entry” identifies the
column as unauthored. This is case 3 of the `Scd2VersionMap` contract.
2. The target contains a column that is absent from a later microbatch
schema, representing reductive schema evolution. The microbatch is padded to
the target schema before its map is constructed, and the synthesized null
desirably receives an entry according to the active ignore-null selection.
The former case is retroactive schema evolution where the event truly has no
authorship over a later added column, and should indeed inherit its value
instead. The latter is a case where the event explicitly chose not to author a
known/existing column, and so that column should respect the active ignore-null
selection. Both cases are uniquely identifiable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
