[
https://issues.apache.org/jira/browse/SPARK-61359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61359:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Read the crate directory when the proto build script runs, and pin the hash
> seed and the UDTF file timeout of the parity scripts
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61359
> URL: https://issues.apache.org/jira/browse/SPARK-61359
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Three small build and test-infrastructure fixes from the s16 review:
> - crates/spark-connect-proto/build.rs takes the directory of its .proto files
> from env!("CARGO_MANIFEST_DIR"), which is baked in when the script is
> compiled. A build directory shared by several checkouts (worktrees) can run a
> script compiled in another checkout; once that checkout is deleted, the build
> fails with "protoc failed: Could not make proto path relative: <deleted
> checkout>/crates/spark-connect-proto/proto/spark/connect/base.proto". The
> script now reads CARGO_MANIFEST_DIR when it runs, as Cargo sets it for every
> build-script run.
> - scripts/gen_parity_skiplist.py, which runs the reference client to
> regenerate the manifest of environmental failures, does not pin
> PYTHONHASHSEED, while scripts/run_official_tests.py pins it to 0 (as Apache's
> bin/pyspark does). With a random seed, about 7% of the runs of the official
> test_crossvalidator_with_random_forest_classifier fail, so a regenerated
> manifest could pick up that flake. It now pins it too.
> - scripts/run_official_tests.py runs test_parity_udtf.py (387 tests, most
> with a Python worker on the server) under the default 360 s per-file cap. It
> takes 145-204 s on CI runners but 466 s on a loaded machine, so it gets the
> 1200 s cap of test_parity_arrow_python_udf.py.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]