[
https://issues.apache.org/jira/browse/SPARK-6436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14371750#comment-14371750
]
Thomas Graves commented on SPARK-6436:
--------------------------------------
netty io is listed in the dependency tree. When I build though I see this:
INFO] --- maven-shade-plugin:2.2:shade (default) @ spark-network-common_2.10 ---
[INFO] Excluding io.netty:netty-all:jar:4.0.23.Final from the shaded jar.
[INFO] Including com.google.guava:guava:jar:14.0.1 in the shaded jar.
[INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
I assume perhaps the top level pom.xml where the shade has specific
includes/excludes is preventing it. I'll investigate more
> io/netty missing from external shuffle service jars for yarn
> ------------------------------------------------------------
>
> Key: SPARK-6436
> URL: https://issues.apache.org/jira/browse/SPARK-6436
> Project: Spark
> Issue Type: Bug
> Components: Shuffle, YARN
> Affects Versions: 1.3.0
> Reporter: Thomas Graves
>
> I was trying to use the external shuffle service on yarn but it appears that
> io/netty isn't included in the network jars. I loaded up network-common,
> network-yarn, and network-shuffle. If there is some other jar supposed to be
> included please let me know.
> 2015-03-20 14:25:07,142 [main] FATAL
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting
> NodeManager
> java.lang.NoClassDefFoundError: io/netty/channel/EventLoopGroup
> at
> org.apache.spark.network.shuffle.ExternalShuffleBlockManager.<init>(ExternalShuffleBlockManager.java:64)
> at
> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.<init>(ExternalShuffleBlockHandler.java:53)
> at
> org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:105)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:143)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
