qianlan717 commented on code in PR #59196:
URL: https://github.com/apache/spark/pull/59196#discussion_r4255298664
##########
common/utils/src/main/scala/org/apache/spark/util/MavenUtils.scala:
##########
@@ -62,32 +65,126 @@ private[spark] object MavenUtils extends Logging {
}
}
- private def acquireIvyLock(isCancelled: () => Boolean): Unit = {
- def checkCancelled(): Unit = {
- if (isCancelled() || Thread.currentThread().isInterrupted) {
- throw new CancellationException("Maven dependency resolution was
cancelled")
+ private class IvyResolutionCancellation {
+ private val requested = new AtomicBoolean(false)
+ private val httpHandler = new AtomicReference[HttpClientHandler]()
+
+ def isCancelled: Boolean = requested.get()
+
+ def installUrlHandler(): () => Unit = {
+ val previousHandler = URLHandlerRegistry.getDefault
+ val handler = new HttpClientHandler
Review Comment:
Replacing Ivy’s HTTP handler solely for cancellation introduces unnecessary
authentication risk. I propose removing that replacement and preserving Ivy’s
existing transport and authentication behavior, and change to use a best-effort
cancellation
**What already has:** runtime resolution has `spark.jars.ivyConnectTimeout`
and `spark.jars.ivyReadTimeout`, cancellation checks before/after resolution,
and cancellable waiting for the Ivy lock. However, once ivy.resolve starts,
those Spark checks cannot observe cancellation until it returns. The timeouts
bound connection attempts and individual blocking reads—not the whole
resolution.
**What we propose to add:** run resolution in a worker and monitor request
cancellation. On cancellation, set a persistent cancellation flag and directly
interrupt the worker. Ivy’s existing artifact-copy loop can then notice
interruption between reads; Spark’s phase checkpoints prevent subsequent
resolve/retrieve work. We would **wait for worker cleanup** before returning
cancellation, without replacing Ivy’s transport or authentication behavior. Ivy
already provides the copy-loop interruption check.
**Why this is best-effort:** interruption does not necessarily unblock an
ongoing read, and some Ivy metadata and cleanup loops do not check
interruption. The existing read timeout helps with stalled reads, but does not
guarantee that the entire resolution terminates within one timeout. This
improves cancellation responsiveness where Ivy cooperates, without promising
immediate termination or reimplementing its downloader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
