Hyukjin Kwon created SPARK-61359:
------------------------------------

             Summary: Read the crate directory when the proto build script 
runs, and pin the hash seed and the UDTF file timeout of the parity scripts
                 Key: SPARK-61359
                 URL: https://issues.apache.org/jira/browse/SPARK-61359
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


Three small build and test-infrastructure fixes from the s16 review:
- crates/spark-connect-proto/build.rs takes the directory of its .proto files 
from env!("CARGO_MANIFEST_DIR"), which is baked in when the script is compiled. 
A build directory shared by several checkouts (worktrees) can run a script 
compiled in another checkout; once that checkout is deleted, the build fails 
with "protoc failed: Could not make proto path relative: <deleted 
checkout>/crates/spark-connect-proto/proto/spark/connect/base.proto". The 
script now reads CARGO_MANIFEST_DIR when it runs, as Cargo sets it for every 
build-script run.
- scripts/gen_parity_skiplist.py, which runs the reference client to regenerate 
the manifest of environmental failures, does not pin PYTHONHASHSEED, while 
scripts/run_official_tests.py pins it to 0 (as Apache's bin/pyspark does). With 
a random seed, about 7% of the runs of the official 
test_crossvalidator_with_random_forest_classifier fail, so a regenerated 
manifest could pick up that flake. It now pins it too.
- scripts/run_official_tests.py runs test_parity_udtf.py (387 tests, most with 
a Python worker on the server) under the default 360 s per-file cap. It takes 
145-204 s on CI runners but 466 s on a loaded machine, so it gets the 1200 s 
cap of test_parity_arrow_python_udf.py.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to