Hi guys:
I am using spark 2.1.1 to test on CDH 5.7.1, when i run on yarn with
following command, error 'NoSuchMethodError:
org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;'
appears sometimes:
command:
*su cloudera-scm -s "/bin/sh" -c "/opt/spark2/bin/spark-shell --master
yarn --deploy-mode client --files
/opt/spark2/conf/log4j_all.properties#log4j.properties --driver-memory 8g
--num-executors 2 --executor-memory 8g --executor-cores 5
--driver-library-path :/opt/cloudera/parcels/CDH/lib/hadoop/lib/native
--driver-class-path /opt/spark2/libs/mysql-connector-java-5.1.36.jar --jars
/opt/spark2/libs/mysql-connector-java-5.1.36.jar " *
error messages:
2017-07-17 17:15:25,255 - WARN -
org.apache.spark.network.server.TransportChannelHandler.exceptionCaught(TransportChannelHandler.java:78)
- rpc-client-1-1 -Exception in connection from /ip:60099
java.lang.NoSuchMethodError:
org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;
at
org.apache.spark.rpc.netty.NettyRpcHandler.channelActive(NettyRpcEnv.scala:614)
at
org.apache.spark.network.server.TransportRequestHandler.channelActive(TransportRequestHandler.java:87)
at
org.apache.spark.network.server.TransportChannelHandler.channelActive(TransportChannelHandler.java:88)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
at
org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
at
org.spark_project.io.netty.handler.timeout.IdleStateHandler.channelActive(IdleStateHandler.java:251)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
at
org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
at
org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
at
org.spark_project.io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1282)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
at
org.spark_project.io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:887)
at
org.spark_project.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:262)
at
org.spark_project.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:292)
at
org.spark_project.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:640)
at
org.spark_project.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575)
at
org.spark_project.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489)
at
org.spark_project.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451)
at
org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
at
org.spark_project.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:745)
2017-07-17 17:15:25,256 - DEBUG -
org.spark_project.io.netty.util.internal.logging.Slf4JLogger.debug(Slf4JLogger.java:81)
- rpc-client-1-1 -An exception java.lang.NoSuchMethodError:
org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;
at
org.apache.spark.rpc.netty.NettyRpcHandler.exceptionCaught(NettyRpcEnv.scala:595)
at
org.apache.spark.network.server.TransportRequestHandler.exceptionCaught(TransportRequestHandler.java:82)
at
org.apache.spark.network.server.TransportChannelHandler.exceptionCaught(TransportChannelHandler.java:80)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:291)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:854)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:221)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
at
org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
at
org.spark_project.io.netty.handler.timeout.IdleStateHandler.channelActive(IdleStateHandler.java:251)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
at
org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
my conf:
spark.serializer
org.apache.spark.serializer.KryoSerializer
spark.rdd.compress true
spark.default.parallelism 10
spark.rpc.netty.dispatcher.numThreads 3
spark.shuffle.manager tungsten-sort
spark.memory.useLegacyMode false
spark.memory.fraction 0.5
spark.memory.storageFraction 0.3
spark.memory.offHeap.enabled false
spark.memory.offHeap.size 0
spark.yarn.executor.memoryOverhead 1024
spark.yarn.am.memoryOverhead 1024
spark.yarn.maxAppAttempts 6
spark.shuffle.file.buffer 32k
spark.locality.wait 0s
spark.scheduler.minRegisteredResourcesRatio 1.0
spark.scheduler.maxRegisteredResourcesWaitingTime 60s
Does anyone encounters this problems? Please help me, Thanks.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-2-1-1-Error-java-lang-NoSuchMethodError-org-apache-spark-network-client-TransportClient-getCha-tp28868.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
