[ 
https://issues.apache.org/jira/browse/SPARK-61359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61359:
------------------------------------

    Assignee: Hyukjin Kwon

> Read the crate directory when the proto build script runs, and pin the hash 
> seed and the UDTF file timeout of the parity scripts
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61359
>                 URL: https://issues.apache.org/jira/browse/SPARK-61359
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> Three small build and test-infrastructure fixes from the s16 review:
> - crates/spark-connect-proto/build.rs takes the directory of its .proto files 
> from env!("CARGO_MANIFEST_DIR"), which is baked in when the script is 
> compiled. A build directory shared by several checkouts (worktrees) can run a 
> script compiled in another checkout; once that checkout is deleted, the build 
> fails with "protoc failed: Could not make proto path relative: <deleted 
> checkout>/crates/spark-connect-proto/proto/spark/connect/base.proto". The 
> script now reads CARGO_MANIFEST_DIR when it runs, as Cargo sets it for every 
> build-script run.
> - scripts/gen_parity_skiplist.py, which runs the reference client to 
> regenerate the manifest of environmental failures, does not pin 
> PYTHONHASHSEED, while scripts/run_official_tests.py pins it to 0 (as Apache's 
> bin/pyspark does). With a random seed, about 7% of the runs of the official 
> test_crossvalidator_with_random_forest_classifier fail, so a regenerated 
> manifest could pick up that flake. It now pins it too.
> - scripts/run_official_tests.py runs test_parity_udtf.py (387 tests, most 
> with a Python worker on the server) under the default 360 s per-file cap. It 
> takes 145-204 s on CI runners but 466 s on a loaded machine, so it gets the 
> 1200 s cap of test_parity_arrow_python_udf.py.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to