[
https://issues.apache.org/jira/browse/SPARK-61385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61385:
------------------------------------
Assignee: Hyukjin Kwon
> Run the tests of the execution hook without a server on a client that does
> not retry, so that they do not wait 10 minutes for the plan compression lookup
> ---------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61385
> URL: https://issues.apache.org/jira/browse/SPARK-61385
> Project: Spark
> Issue Type: Test
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Since SPARK-61263 (plan compression), the first plan of a session reads the
> plan compression confs of the server with a Config call before the execution
> hook runs, as PySpark's _get_plan_compression_threshold_and_algorithm does.
> The two offline tests of crates/spark-connect/tests/execute_plan_hook.rs use
> a session of the dead port sc://127.0.0.1:1 with the default retry policy, so
> that Config call is retried for its whole backoff before the hook runs: the
> test binary takes 639 s, where it took 0.00 s before.
> The CI jobs run it several times: "Rust build and test (stable)" twice, and
> "Rust checks", "WASM UDF feature" and "Measure and publish coverage badges"
> once each, 10.7 minutes each time. "Rust behavioral e2e" runs it one test at
> a time, 1278 s. The "Rust build and test (stable)" job went from 7.1 minutes
> at aad559aa to 34.7 minutes at 2c768214, the first commit after SPARK-61263
> with a full run.
> Fix: run those two tests in the crate on a session of the dead port whose
> client does not retry (RetryPolicy::no_retries, as the tests of the streaming
> module do), and keep the test of a server in the integration test file.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]