anew commented on code in PR #57584:
URL: https://github.com/apache/spark/pull/57584#discussion_r3668116892
##########
sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/graph/AutoCdcOutOfOrderConvergenceSuite.scala:
##########
@@ -140,20 +146,33 @@ class AutoCdcScd1OutOfOrderConvergenceSuite
deleteCondition = Some(functions.col(isDeleteColumn) === true),
columnSelection = Some(ColumnSelection.ExcludeColumns(
Seq(UnqualifiedColumnName(isDeleteColumn))
- ))
+ )),
+ scdType = scdType
))
}
}
- private def createTargetTable(targetTable: String): Unit = {
+ /**
+ * DDL fragment for the SCD-type-specific reserved columns a target table
carries after the
+ * user-selected data columns: just the CDC metadata column for SCD1, plus
the __START_AT /
+ * __END_AT interval bounds for SCD2. The sequencing type is BIGINT here.
+ */
+ private def reservedColumnsDdl(scdType: ScdType): String = scdType match {
Review Comment:
I made this symmetric by
- renaming `cdcMetadataDdl` to `scd1MetadataDdl`
- adding a new `scd2MetadataDdl` to the Mixin
- using these constants everywhere.
But digging into this, I realized that there are not many paces where the
new `scd2MetadataDdl` are used. The reason is a gap in test coverage: while we
have quite a few end to end tests for SCD1, we have very few for SCD2. I will
close that test gap in a follow-up PR with Jira:
https://issues.apache.org/jira/browse/SPARK-58409
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
