mridulm commented on code in PR #2482:
URL: https://github.com/apache/celeborn/pull/2482#discussion_r1591934586
##########
client-spark/spark-2/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java:
##########
@@ -254,6 +257,20 @@ private int executorCores(SparkConf conf) {
}
}
+ private void checkUserClassPathFirst() {
+ if ((Boolean) conf.get(package$.MODULE$.EXECUTOR_USER_CLASS_PATH_FIRST()))
{
Review Comment:
Emit log message only when classloaders are different ? (same for the other
classes as well)
```suggestion
if ((Boolean)
conf.get(package$.MODULE$.EXECUTOR_USER_CLASS_PATH_FIRST()) &&
!Objects.equals(getClass().getClassLoader(),
conf.getClass().getClassLoader())
) {
```
##########
README.md:
##########
@@ -301,6 +301,10 @@ spark.sql.adaptive.skewJoin.enabled true
spark.shuffle.sort.io.plugin.class
org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO
# Required Spark version >= 3.4.0, highly recommended to disable
spark.dynamicAllocation.shuffleTracking.enabled false
+
+# Support ShuffleManager defined in user jar
Review Comment:
super nit:
```suggestion
# Support ShuffleManager when defined in user jars
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
