Anish Mahto created SPARK-58572:
-----------------------------------
Summary: Add Cross-SCD Fuzz Testing
Key: SPARK-58572
URL: https://issues.apache.org/jira/browse/SPARK-58572
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.4.0
Reporter: Anish Mahto
Given the same CDF (but can be shuffled in arbitrary order), every key's output
row in SCD1 should be exactly equal (in data columns, not necessarily CDC
metadata) to the last row per key in SCD2.
Using this principle, and the existing AutoCDC fuzz test harnesses, we should
setup a cross-SCD fuzz test so that one implementation can keep the other in
check for correctness, and detect regressions made to one SCD's implementation
but not the other.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
