srowen commented on a change in pull request #35223: URL: https://github.com/apache/spark/pull/35223#discussion_r790287713
########## File path: docs/running-on-yarn.md ########## @@ -913,7 +913,11 @@ with a mixed workload of applications running multiple Spark versions, since a g the shuffle service is not always compatible with other versions of Spark. YARN versions since 2.9.0 support the ability to run shuffle services within an isolated classloader (see [YARN-4577](https://issues.apache.org/jira/browse/YARN-4577)), meaning multiple Spark versions -can coexist within a single NodeManager. The +can coexist within a single NodeManager. There is an issue Review comment: Let's clean up the sentence a little: "Note that versions of YARN prior to 3.3.2/3.4.0 have an issue[...] which requires setting `yarn.nodemanager.aux-services.<service-name>.system-classes` as a workaround. See issue for details." ########## File path: docs/running-on-yarn.md ########## @@ -923,7 +927,11 @@ above. For example, you may have configuration like: ```properties yarn.nodemanager.aux-services = spark_shuffle_x,spark_shuffle_y yarn.nodemanager.aux-services.spark_shuffle_x.classpath = /path/to/spark-x-yarn-shuffle.jar,/path/to/spark-x-config + # workaround for YARN-11053 + yarn.nodemanager.aux-services.spark_shuffle_x.system-classes = classes.other.than.spark.shuffle.service.classes yarn.nodemanager.aux-services.spark_shuffle_y.classpath = /path/to/spark-y-yarn-shuffle.jar,/path/to/spark-y-config + # workaround for YARN-11053 + yarn.nodemanager.aux-services.spark_shuffle_y.system-classes = classes.other.than.spark.shuffle.service.classes Review comment: Let's not add this as it is a workaround for older YARN. It does not need to be a part of Spark docs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
