AnishMahto commented on code in PR #58483:
URL: https://github.com/apache/spark/pull/58483#discussion_r3921561460
##########
sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/graph/AutoCdcScd2SchemaEvolutionSuite.scala:
##########
@@ -64,7 +71,78 @@ class AutoCdcScd2SchemaEvolutionSuite
import testImplicits._
/** The SCD2 target's `_cdc_metadata` struct value for a given
recordStartAt. */
- private def scd2Meta(recordStartAt: Long): Row = Row(recordStartAt)
+ private def scd2Meta(recordStartAt: Long): Row = Row(recordStartAt, null)
+
+ test("legacy SCD2 CDC metadata schema evolves to include the version map") {
Review Comment:
The major focus for this PR should be - what should we do with existing
AutoCDC tables created before version map is added to the CDC metadata column?
What I rely on in this PR, as is illustrated by this test, is SDP + DSv2's
ability to handle schema evolution for nested columns. That means existing
AutoCDC tables will gain a version map sub-field in their CDC metadata column,
set to null, on their next run after these changes lands.
Now what is technically not guaranteed, is whether each popular connector
actually supports and respects the table alters and `TableChanges` we emit, via
DSv2. If they don't, then those tables will be not be able to be refreshed
going forward, on the first Spark version that includes these changes. But from
SDP's perspective, we are fully compliant with and using DSv2 API.
Of the connectors that do support the MERGE operation (which is a
prerequisite for AutoCDC anyway), I'm not aware of any that don't support
nested column evolution via DSv2 - ex. both Delta and Iceberg do support nested
evolution. But technically Parquet tables in HMS do not support nested schema
evolution AFAIK. Now Parquet does not support MERGE either, so an AutoCDC table
could never have been materialized using it.
**Alternative option**
If we're not willing to rely on the assumption that all AutoCDC compatible
table formats and catalogs support nested schema evolution via DSv2, then
another option is too just support two variants of the CDC metadata column
schema going forward. Existing tables will use the schema definition that does
not include the version map, and new tables will be created with the schema
definition that includes the version map.
This does however mean:
- We need to support both schema variants forever going forward
- Existing AutoCDC tables are not allowed to convert to ignore-null. That
also means they will never be eligible for AutoCDC multiflow, which relies on
ignore-null
- We'll need to add extra logic to check the existing table's schema, and
determine which schema variant its using
- The `Scd2BatchProcessor` code will have to be written in a way that
supports both types of schemas
Which I am generally not a fan of, and would prefer to avoid if justifiable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
