[ 
https://issues.apache.org/jira/browse/SPARK-61385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61385.
----------------------------------
    Resolution: Fixed

Issue resolved by pull request 473
[https://github.com/apache/spark-connect-rust/pull/473]

> Run the tests of the execution hook without a server on a client that does 
> not retry, so that they do not wait 10 minutes for the plan compression lookup
> ---------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61385
>                 URL: https://issues.apache.org/jira/browse/SPARK-61385
>             Project: Spark
>          Issue Type: Test
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> Since SPARK-61263 (plan compression), the first plan of a session reads the 
> plan compression confs of the server with a Config call before the execution 
> hook runs, as PySpark's _get_plan_compression_threshold_and_algorithm does. 
> The two offline tests of crates/spark-connect/tests/execute_plan_hook.rs use 
> a session of the dead port sc://127.0.0.1:1 with the default retry policy, so 
> that Config call is retried for its whole backoff before the hook runs: the 
> test binary takes 639 s, where it took 0.00 s before.
> The CI jobs run it several times: "Rust build and test (stable)" twice, and 
> "Rust checks", "WASM UDF feature" and "Measure and publish coverage badges" 
> once each, 10.7 minutes each time. "Rust behavioral e2e" runs it one test at 
> a time, 1278 s. The "Rust build and test (stable)" job went from 7.1 minutes 
> at aad559aa to 34.7 minutes at 2c768214, the first commit after SPARK-61263 
> with a full run.
> Fix: run those two tests in the crate on a session of the dead port whose 
> client does not retry (RetryPolicy::no_retries, as the tests of the streaming 
> module do), and keep the test of a server in the integration test file.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to