[
https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097026#comment-15097026
]
Steve Loughran commented on SPARK-12807:
----------------------------------------
Stack
{code}
So connecting it gives error:
2016-01-11 16:56:51,222 INFO containermanager.AuxServices
(AuxServices.java:addService(72)) - Adding auxiliary service
spark_shuffle, "spark_shuffle"
2016-01-11 16:56:51,439 FATAL nodemanager.NodeManager
(NodeManager.java:initAndStartNodeManager(465)) - Error starting
NodeManager
java.lang.NoSuchMethodError:
com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z
at
com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:457)
at
com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:379)
at
org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.<clinit>(ExternalShuffleBlockResolver.java:57)
at
org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.<init>(ExternalShuffleBlockHandler.java:56)
at
org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:128)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:143)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:237)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:253)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:462)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:509)
{code}
> Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
> ------------------------------------------------------------------------
>
> Key: SPARK-12807
> URL: https://issues.apache.org/jira/browse/SPARK-12807
> Project: Spark
> Issue Type: Bug
> Components: Shuffle, YARN
> Affects Versions: 1.6.0
> Environment: A Hadoop cluster with Jackson 2.2.3, spark running with
> dynamic allocation enabled
> Reporter: Steve Loughran
>
> When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster,
> you get to see a stack trace in the NM logs, indicating a jackson 2.x version
> mismatch.
> (reported on the spark dev list)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
