xkrogen commented on a change in pull request #35223:
URL: https://github.com/apache/spark/pull/35223#discussion_r789813717
##########
File path: docs/running-on-yarn.md
##########
@@ -923,7 +927,11 @@ above. For example, you may have configuration like:
```properties
yarn.nodemanager.aux-services = spark_shuffle_x,spark_shuffle_y
yarn.nodemanager.aux-services.spark_shuffle_x.classpath =
/path/to/spark-x-yarn-shuffle.jar,/path/to/spark-x-config
+ # workaround for YARN-11053
+ yarn.nodemanager.aux-services.spark_shuffle_x.system-classes =
classes.other.than.spark.shuffle.service.classes
yarn.nodemanager.aux-services.spark_shuffle_y.classpath =
/path/to/spark-y-yarn-shuffle.jar,/path/to/spark-y-config
+ # workaround for YARN-11053
+ yarn.nodemanager.aux-services.spark_shuffle_y.system-classes =
classes.other.than.spark.shuffle.service.classes
Review comment:
Rather than setting it to some arbitrary class name, it would be better
to set it to an empty string. In this way, the `ApplicationClassLoader` will be
configured to use the default system classes
([ref](https://github.com/apache/hadoop/blob/03cfc852791c14fad39db4e5b14104a276c08e59/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ApplicationClassLoader.java#L94-L97)).
This is probably more desirable behavior than completely overriding the system
classes config, and is more similar to the changes from YARN-11053.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
