[
https://issues.apache.org/jira/browse/YARN-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17493624#comment-17493624
]
tonydoen edited comment on YARN-4577 at 2/17/22, 3:28 AM:
----------------------------------------------------------
Thanks [~xgong] , [xkro...@apache.org |mailto:xkro...@apache.org]for your
contribution!
we have tried to run multiple versions of the Spark Shuffle Service according
[https://github.com/apache/spark/blob/master/docs/running-on-yarn.md]
but, it wont work. any suggestion, we would appreciate it.
yarn-site.xml configuration
...
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle,spark_shuffle_1_6,spark_shuffle_3_2</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle_1_6.class</name>
<value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle_1_6.classpath</name>
<value>/home/work/app/yarn/c4prc-preview/nodemanager/spark-yarn-shuffle-lib/spark-1-6-yarn-shuffle-lib,/home/work/app/yarn/c4prc-preview/nodemanager/spark-yarn-shuffle-lib/spark-1-6-yarn-shuffle-config</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle_3_2.class</name>
<value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle_3_2.classpath</name>
<value>/home/work/app/yarn/c4prc-preview/nodemanager/spark-yarn-shuffle-lib/spark-3-2-yarn-shuffle-lib,/home/work/app/yarn/c4prc-preview/nodemanager/spark-yarn-shuffle-lib/spark-3-2-yarn-shuffle-config</value>
</property>
...
/home/work/app/yarn/c4prc-preview/nodemanager/spark-yarn-shuffle-lib/spark-3-2-yarn-shuffle-lib
is spark-yarn-shuffle lib location
/home/work/app/yarn/c4prc-preview/nodemanager/spark-yarn-shuffle-lib/spark-3-2-yarn-shuffle-config
is spark-yarn-shuffle config, just one file named [spark-shuffle-site.xml] is
in the above path.
spark-shuffle-site.xml content is
<configuration>
<property>
<name>spark.shuffle.service.port</name>
<value>20811</value>
</property>
<property>
<name>spark.yarn.shuffle.service.metrics.namespace</name>
<value>sparkShuffleService1</value>
</property>
</configuration>
was (Author: JIRAUSER285351):
Thanks [~xgong] , [xkro...@apache.org |mailto:xkro...@apache.org]for your
contribution!
we have tried to run multiple versions of the Spark Shuffle Service according
[https://github.com/apache/spark/blob/master/docs/running-on-yarn.md]
but, it wont work. any suggestion, we would appreciate it.
yarn-site.xml configuration
...
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle,spark_shuffle_1_6,spark_shuffle_3_2</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle_1_6.class</name>
<value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle_1_6.classpath</name>
<value>/home/work/app/yarn/c4prc-preview/nodemanager/spark-yarn-shuffle-lib/spark-1-6-yarn-shuffle-lib,/home/work/app/yarn/c4prc-preview/nodemanager/spark-yarn-shuffle-lib/spark-1-6-yarn-shuffle-config</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle_3_2.class</name>
<value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle_3_2.classpath</name>
<value>/home/work/app/yarn/c4prc-preview/nodemanager/spark-yarn-shuffle-lib/spark-3-2-yarn-shuffle-lib,/home/work/app/yarn/c4prc-preview/nodemanager/spark-yarn-shuffle-lib/spark-3-2-yarn-shuffle-config</value>
</property>
...
/home/work/app/yarn/c4prc-preview/nodemanager/spark-yarn-shuffle-lib/spark-3-2-yarn-shuffle-lib
is spark-yarn-shuffle lib location
/home/work/app/yarn/c4prc-preview/nodemanager/spark-yarn-shuffle-lib/spark-3-2-yarn-shuffle-config
is spark-yarn-shuffle config, just one file named [spark-shuffle-site.xml] is
in the above path.
spark-shuffle-site.xml content is
<configuration>
<property>
<name>spark.shuffle.service.port</name>
<value>20811</value>
</property>
<property>
<name>spark.yarn.shuffle.service.metrics.namespace</name>
<value>sparkShuffleService1</value>
</property>
</configuration>
> Enable aux services to have their own custom classpath/jar file
> ---------------------------------------------------------------
>
> Key: YARN-4577
> URL: https://issues.apache.org/jira/browse/YARN-4577
> Project: Hadoop YARN
> Issue Type: Improvement
> Affects Versions: 2.8.0
> Reporter: Xuan Gong
> Assignee: Xuan Gong
> Priority: Major
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-4577.1.patch, YARN-4577.2.patch,
> YARN-4577.20160119.1.patch, YARN-4577.20160204.patch,
> YARN-4577.20160428.patch, YARN-4577.20160509.patch, YARN-4577.20160510.patch,
> YARN-4577.20160511.1.patch, YARN-4577.20160511.patch, YARN-4577.3.patch,
> YARN-4577.3.rebase.patch, YARN-4577.4.patch, YARN-4577.5.patch,
> YARN-4577.poc.patch
>
>
> Right now, users have to add their jars to the NM classpath directly, thus
> put them on the system classloader. But if multiple versions of the plugin
> are present on the classpath, there is no control over which version actually
> gets loaded. Or if there are any conflicts between the dependencies
> introduced by the auxiliary service and the NM itself, they can break the NM,
> the auxiliary service, or both.
> The solution could be: to instantiate aux services using a classloader that
> is different from the system classloader.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
