dongjoon-hyun commented on PR #58574: URL: https://github.com/apache/spark/pull/58574#issuecomment-5581431234
Thank you for the detailed reply, @sarutak. I agree with the direction: keep the selection/resolution split, keep resolution late where `HadoopDelegationTokenManager` also starts, and make selection safe in every context where resolution will not follow. Your Kerberos comparison checks out; `LocalSchedulerBackend` does override `createTokenManager()`, so (b) is real parity. **Local mode.** (a) for this PR and (b) as a follow-up sounds right to me. (a) removes the regression with minimal scope, and (b) brings STS calls and a renewal thread into local mode, which deserves its own PR and tests. **Constructor window.** Documentation is fine for `spark.jars` / `spark.files` on `s3a://`. Two small asks on top of that: - Since `checkRemoteJarFile` only logs and drops the jar, please have the selection phase log one line at INFO/WARN noting that driver-side access to the wired scheme will fail until credentials are acquired at scheduler start. That makes the eventual `ClassNotFoundException` on executors much easier to diagnose. - Please call out the `SparkSubmit` `FileSystem` cache case as its own item in the docs rather than folding it into the same caveat. It affects the default K8s cluster-mode path (`--jars s3a://BUCKET/...` with output on the same bucket) and results in a mixed-identity job rather than a failure, which is harder to notice. The rest of the plan (I/O-free selection without `init()`, WARN plus idempotent fallback in `start()`, single owner for the loader, `stop()` ordering, scaladoc and description updates, `SparkContext`-level tests, moving the call earlier) covers everything I raised. One nit for the next push: the commit message still says `SPARK-57703` while the PR title is `SPARK-59296`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
