Qianlan Chen created SPARK-59938:
------------------------------------
Summary: Improve cancellatio for server-side Maven artifact
resolution
Key: SPARK-59938
URL: https://issues.apache.org/jira/browse/SPARK-59938
Project: Spark
Issue Type: Improvement
Components: Connect, SQL
Affects Versions: 4.4.0
Reporter: Qianlan Chen
This is follow-up work from SPARK-59553.
Current behavior
Server-side Maven artifact resolution checks whether the gRPC context is
cancelled or the thread is interrupted:
* before starting resolution;
* while polling for the global Ivy lock;
* immediately after acquiring the lock; and
* after resolution completes.
Cancellation is not checked while the synchronous ivy.resolve call is running.
During active resolution, we currently rely on the network-operation timeouts:
* spark.jars.ivyConnectTimeout: 30 seconds by default
* spark.jars.ivyReadTimeout: 5 minutes by default
Both timeouts are capped by the remaining RPC deadline. However, these are
per-network-operation timeouts rather than a total resolution timeout. A
cancelled request may therefore retain the global Ivy lock until resolution
completes or a network operation times out.
Follow-up work
1. Improve cancellation behavior while Ivy resolution is actively running.
2. Add coverage for resolution timeouts when custom Ivy settings are used.
3. Add an atomicity test where a valid uploaded artifact is followed by a
failing Maven artifact, verifying that no artifacts are registered.
4. Add an end-to-end Spark Connect handler test that exercises Maven
resolution without using the fake injected artifact.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
