squito commented on a change in pull request #26000: [SPARK-29330][CORE][YARN]
Allow users to chose the name of Spark Shuffle service
URL: https://github.com/apache/spark/pull/26000#discussion_r354952158
##########
File path:
common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
##########
@@ -136,7 +136,11 @@
private DB db;
public YarnShuffleService() {
- super("spark_shuffle");
+ this("spark_shuffle");
+ }
+
+ protected YarnShuffleService(String serviceName) {
Review comment:
I think there are a few things getting muddled together here -- one is how
you'd support running two shuffle services, and the other is how a client could
choose which shuffle service it talks to.
The client can already set the port for the shuffle server with
`spark.shuffle.service.port`, it just can't set the name used in the
`ExecutorRunnable`.
The other thing to add about how the names of the shuffle servers matter in
yarn is that the name goes into `yarn-site.xml` as described in the
"Configuring the External Shuffle Service" in
[`running-on-yarn.md`](https://github.com/apache/spark/blame/c1a5f94973213b1cad15388f3ef8a488424c34a7/docs/running-on-yarn.md#L660).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
