Hyukjin Kwon created SPARK-58987:
------------------------------------
Summary: [spark-connect-rust] Faster Connect parity CI (ours-only
+ skip manifest) and 100% Rust coverage
Key: SPARK-58987
URL: https://issues.apache.org/jira/browse/SPARK-58987
Project: Spark
Issue Type: Improvement
Components: Project Infra
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The Connect parity gate runs all 84 official connect test files twice each
(reference pyspark AND our client via transport injection) against one live
server, ~1h20m-2h. Now that the client is at full parity, the reference run
only serves to compute a per-file environmental-failure baseline.
h2. Changes
* Parity gate: run ONLY our client against the official connect suite,
deselecting a checked-in per-test manifest of known environmental failures
(tests the reference client also fails in this single-node pure-Connect CI env,
each annotated with the reason). Drops the reference run, roughly halving
CI/build time. The manifest is regenerated from a reference run when the Spark
version bumps.
* Add in-repo Rust tests (plan-building golden + e2e_integration) covering the
functionality behind the env-skipped official tests, so skipping them loses no
real coverage.
* Enforce 100% Rust test coverage across all crates (spark-connect-core,
spark-connect, and the pyspark-rs PyO3 layer measured via the Python suite
driving an instrumented extension) via a cargo-llvm-cov CI gate; fill the gaps
with unit/integration tests.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
