[
https://issues.apache.org/jira/browse/SPARK-6436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14371691#comment-14371691
]
Marcelo Vanzin commented on SPARK-6436:
---------------------------------------
It shouldn't really depend on the Hadoop version. netty comes in compile scope
from network/common, so it should be included in the final assembly for
network/yarn.
Perhaps take a look at "mvn -f network/yarn/pom.xml dependency:tree -Dverbose"
to see if something looks wrong?
> io/netty missing from external shuffle service jars for yarn
> ------------------------------------------------------------
>
> Key: SPARK-6436
> URL: https://issues.apache.org/jira/browse/SPARK-6436
> Project: Spark
> Issue Type: Bug
> Components: Shuffle, YARN
> Affects Versions: 1.3.0
> Reporter: Thomas Graves
>
> I was trying to use the external shuffle service on yarn but it appears that
> io/netty isn't included in the network jars. I loaded up network-common,
> network-yarn, and network-shuffle. If there is some other jar supposed to be
> included please let me know.
> 2015-03-20 14:25:07,142 [main] FATAL
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting
> NodeManager
> java.lang.NoClassDefFoundError: io/netty/channel/EventLoopGroup
> at
> org.apache.spark.network.shuffle.ExternalShuffleBlockManager.<init>(ExternalShuffleBlockManager.java:64)
> at
> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.<init>(ExternalShuffleBlockHandler.java:53)
> at
> org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:105)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:143)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
