xkrogen commented on a change in pull request #31936:
URL: https://github.com/apache/spark/pull/31936#discussion_r600685042
##########
File path: docs/running-on-yarn.md
##########
@@ -761,8 +761,27 @@ The following extra configuration options are available
when the shuffle service
NodeManagers where the Spark Shuffle Service is not running.
</td>
</tr>
+<tr>
+ <td><code>spark.yarn.shuffle.service.metrics.namespace</code></td>
+ <td><code>sparkShuffleService</code></td>
+ <td>
+ The namespace to use when emitting shuffle service metrics into Hadoop
metrics2 system of the
+ NodeManager.
Review comment:
> it looks like the name referenced by the node manager works with the
Hadoop 2.9+ custom class loader, but I assume with Hadoop 2.7 it requires the
spark_shuffle name ? hence the spark.shuffle.service.name won't work unless you
have recompiled the code and manually changed it.
No, this is not correct. YARN ignores the hard-coded name on _all_ versions
of YARN. Take a look at `AuxServices` on the 2.7.0 branch:
https://github.com/apache/hadoop/blob/f95b390df2ca7d599f0ad82cf6e8d980469e7abb/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java#L129-L136
`spark.shuffle.service.name` works fine on Hadoop 2.7, it is only the
isolated classloader that won't work on older versions.
> I'm worrying about the situation some users try to use `Apache Spark
distribution (with Hadoop 2.7)` at YARN 2.9+ cluster. Does it work?
I don't quite understand the concern here. Does my explanation above address
your question? We haven't changed any of the interfaces used to interact with
YARN, there should be no binary compatibility issues or anything of that sort.
I can test whichever combination of `Spark Version + Hadoop Version
Distribution` running on top of `Hadoop Version YARN` you like, but I am
failing to see where the concern is / what you'd like me to look for.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
