This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 74ebef2 [SPARK-37925][DOC] Update document to mention the workaround
for YARN-11053
74ebef2 is described below
commit 74ebef243c18e7a8f32bf90ea75ab6afed9e3132
Author: Cheng Pan <[email protected]>
AuthorDate: Sat Feb 5 09:47:15 2022 -0600
[SPARK-37925][DOC] Update document to mention the workaround for YARN-11053
### What changes were proposed in this pull request?
Update document "Running multiple versions of the Spark Shuffle Service" to
mention the workaround for YARN-11053
### Why are the changes needed?
User may stuck when they following the current document to deploy
multi-versions Spark Shuffle Service on YARN because of
[YARN-11053](https://issues.apache.org/jira/browse/YARN-11053)
### Does this PR introduce _any_ user-facing change?
User document changes.
### How was this patch tested?
Closes #35223 from pan3793/SPARK-37925.
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
---
docs/running-on-yarn.md | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md
index c55ce86..63c0376 100644
--- a/docs/running-on-yarn.md
+++ b/docs/running-on-yarn.md
@@ -916,9 +916,12 @@ support the ability to run shuffle services within an
isolated classloader
can coexist within a single NodeManager. The
`yarn.nodemanager.aux-services.<service-name>.classpath` and, starting from
YARN 2.10.2/3.1.1/3.2.0,
`yarn.nodemanager.aux-services.<service-name>.remote-classpath` options can be
used to configure
-this. In addition to setting up separate classpaths, it's necessary to ensure
the two versions
-advertise to different ports. This can be achieved using the
`spark-shuffle-site.xml` file described
-above. For example, you may have configuration like:
+this. Note that YARN 3.3.0/3.3.1 have an issue which requires setting
+`yarn.nodemanager.aux-services.<service-name>.system-classes` as a workaround.
See
+[YARN-11053](https://issues.apache.org/jira/browse/YARN-11053) for details. In
addition to setting
+up separate classpaths, it's necessary to ensure the two versions advertise to
different ports.
+This can be achieved using the `spark-shuffle-site.xml` file described above.
For example, you may
+have configuration like:
```properties
yarn.nodemanager.aux-services = spark_shuffle_x,spark_shuffle_y
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
