Andreas Neumann created SPARK-58409:
---------------------------------------
Summary: Add SCD2 end-to-end test suites to match SCD1 AutoCDC
coverage
Key: SPARK-58409
URL: https://issues.apache.org/jira/browse/SPARK-58409
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 5.0.0
Reporter: Andreas Neumann
SCD2 AutoCDC end-to-end (graph-execution) test coverage is much thinner than
SCD1's. SCD1 has eight E2E suites under sql/pipelines/src/test/scala/.../graph,
but only two have SCD2 counterparts today.
Have SCD2 equivalents:
- AutoCdcScd1SinglePipelineSuite -> AutoCdcScd2SinglePipelineSuite
- AutoCdcScd1AuxiliaryTableSpecSuite -> AutoCdcScd2AuxiliaryTableSpecSuite
Also, the randomized order-invariance suite AutoCdcOutOfOrderConvergenceSuite
(formerly AutoCdcScd1OutOfOrderConvergenceSuite) now runs for both SCD1 and
SCD2 (SPARK-58321), and AutoCdcConfigDriftSuite covers SCD2 sequencing-type and
track-history drift (SPARK-58391).
Missing SCD2 counterparts (SCD1 suite -> gap):
- AutoCdcScd1FullRefreshSuite -> no SCD2 full-refresh suite. Full refresh drops
and recreates both the target and the (richer) SCD2 auxiliary table; worth
dedicated coverage rather than assuming the SCD1 test generalizes.
- AutoCdcScd1MultiPipelineSuite -> no SCD2 multi-pipeline suite (multiple
pipelines / flows writing to the same SCD2 target).
- AutoCdcScd1SchemaEvolutionSuite -> no SCD2 schema-evolution suite (evolving
user columns on an SCD2 target alongside the framework columns,
nested/struct/array evolution, etc.).
- AutoCdcScd1TargetTableDurabilitySuite -> no SCD2 target-table durability
suite.
- AutoCdcScd1AuxiliaryTableDurabilitySuite -> no SCD2 auxiliary-table
durability suite.
- AutoCdcScd1KeyDriftSuite -> no SCD2 key-drift suite. (Key-drift validation is
SCD-type-agnostic, exercised end-to-end only for SCD1; the SCD2 path should
have equivalent coverage.)
Proposed work: add the missing SCD2 E2E suites, reusing the shared
AutoCdcGraphExecutionTestMixin helpers (scd2MetadataDdl,
singleAutoCdcFlowPipeline/autoCdcFlow with scdType = Type2). Discovered while
extending the out-of-order convergence test to SCD2 (SPARK-58321) and auditing
AutoCDC test coverage.
Note: this became actionable only once SCD2 flows are runnable end to end
(SPARK-58321), which these suites depend on.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
