SteNicholas opened a new pull request, #2482: URL: https://github.com/apache/celeborn/pull/2482
### What changes were proposed in this pull request? `SparkShuffleManager` print warning log for `spark.executor.userClassPathFirst=true` with ShuffleManager defined in user jar via `--jar` or `spark.jars`. ### Why are the changes needed? When `spark.executor.userClassPathFirst` is enabled with ShuffleManager defined in user jar, the `ClassLoader` of `handle` is `ChildFirstURLClassLoader`, which is different from `CelebornShuffleHandle` of which the `ClassLoader` is `AppClassLoader` in `SparkShuffleManager#getWriter/getReader`. The local test log is as follows: ``` ./bin/spark-sql --master yarn --deploy-mode client \ --conf spark.celeborn.master.endpoints=localhost:9099 \ --conf spark.executor.userClassPathFirst=true \ --conf spark.jars=/tmp/celeborn-client-spark-3-shaded_2.12-0.5.0-SNAPSHOT.jar \ --conf spark.shuffle.manager=org.apache.spark.shuffle.celeborn.SparkShuffleManager \ --conf spark.shuffle.service.enabled=false ./bin/spark-sql --master yarn --deploy-mode client --jars /tmp/celeborn-client-spark-3-shaded_2.12-0.5.0-SNAPSHOT.jar \ --conf spark.celeborn.master.endpoints=localhost:9099 \ --conf spark.executor.userClassPathFirst=true \ --conf spark.shuffle.manager=org.apache.spark.shuffle.celeborn.SparkShuffleManager \ --conf spark.shuffle.service.enabled=false ``` ``` 24/04/28 18:03:31 [Executor task launch worker for task 0.0 in stage 5.0 (TID 8)] WARN SparkShuffleManager: [getWriter] handle classloader: org.apache.spark.util.ChildFirstURLClassLoader, CelebornShuffleHandle classloader: sun.misc.Launcher$AppClassLoader ``` It causes that `SparkShuffleManager` fallback to vanilla Spark `SortShuffleManager` for `spark.executor.userClassPathFirst=true` with `ShuffleManager` defined in user jar before https://github.com/apache/spark/pull/43627. Therefore, `SparkShuffleManager` should print warning log to remind for spark.executor.userClassPathFirst=true with `ShuffleManager` defined in user jar. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
