nonsleepr commented on a change in pull request #26000:
[SPARK-29330][CORE][YARN] Allow users to chose the name of Spark Shuffle service
URL: https://github.com/apache/spark/pull/26000#discussion_r334009122
##########
File path:
common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
##########
@@ -136,7 +136,11 @@
private DB db;
public YarnShuffleService() {
- super("spark_shuffle");
+ this("spark_shuffle");
+ }
+
+ protected YarnShuffleService(String serviceName) {
Review comment:
The name specified here is actually useful only in tests. YARN's service
instantiation logic wouldn't even pass the name of the service used in the
config to instantiated service. I guess that's the main reason the names and
ports are hardcoded or bound to non-namespaced configuration keys.
The way HDP overcomes that is by providing different classpaths with
different implementations for different versions of the service
(`spark_shuffle` for Spark 1.6.x and `spark2_shuffle` for Spark 2+). The only
way I see it's possible to pass different parameters to the same implementation
of the service is by providing different configs on the classpath.
I will add a comment here stating that the name is actually only used for
the tests, but otherwise would always be hardcoded to `spark_shuffle`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
