?????? Spark 2.1.1 Error:java.lang.NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;

2017-07-17 Thread ???? 
Thanks for your reply. Can you describe it in more detail? Which dependency 
mismatch?
It works well sometimes, but somtimes fails becaus of error 'NoSuchMethodError'.


Thanks.




--  --
??: "vaquar khan";<vaquar.k...@gmail.com>;
: 2017??7??17??(??) 9:38
??: ""<441586...@qq.com>; 
: "User"<user@spark.apache.org>; 
: Re: Spark 2.1.1 Error:java.lang.NoSuchMethodError: 
org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;



Following error we are getting because of dependency mismatch.

Regards, 
vaquar khan 


On Jul 17, 2017 3:50 AM, "zzcclp" <441586...@qq.com> wrote:
Hi guys:
   I am using spark 2.1.1 to test on CDH 5.7.1, when i run on yarn with
 following command, error 'NoSuchMethodError:
 
org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;'
 appears sometimes:
 
   command:
   *su cloudera-scm -s "/bin/sh" -c "/opt/spark2/bin/spark-shell --master
 yarn --deploy-mode client --files
 /opt/spark2/conf/log4j_all.properties#log4j.properties --driver-memory 8g
 --num-executors 2 --executor-memory 8g --executor-cores 5
 --driver-library-path :/opt/cloudera/parcels/CDH/lib/hadoop/lib/native
 --driver-class-path /opt/spark2/libs/mysql-connector-java-5.1.36.jar --jars
 /opt/spark2/libs/mysql-connector-java-5.1.36.jar " *
 
   error messages:
   2017-07-17 17:15:25,255 - WARN -
 
org.apache.spark.network.server.TransportChannelHandler.exceptionCaught(TransportChannelHandler.java:78)
 - rpc-client-1-1 -Exception in connection from /ip:60099
 java.lang.NoSuchMethodError:
 
org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;
 at
 org.apache.spark.rpc.netty.NettyRpcHandler.channelActive(NettyRpcEnv.scala:614)
 at
 
org.apache.spark.network.server.TransportRequestHandler.channelActive(TransportRequestHandler.java:87)
 at
 
org.apache.spark.network.server.TransportChannelHandler.channelActive(TransportChannelHandler.java:88)
 at
 
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
 at
 
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
 at
 
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
 at
 
org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
 at
 
org.spark_project.io.netty.handler.timeout.IdleStateHandler.channelActive(IdleStateHandler.java:251)
 at
 
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
 at
 
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
 at
 
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
 at
 
org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
 at
 
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
 at
 
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
 at
 
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
 at
 
org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
 at
 
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
 at
 
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
 at
 
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
 at
 
org.spark_project.io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1282)
 at
 
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
 at
 
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
 at
 
org.spark_project.io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:887)
 at
 
org.spark_project.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:262)
 at
 
org.s

Re: Spark 2.1.1 Error:java.lang.NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;

2017-07-17 Thread vaquar khan 
Following error we are getting because of dependency mismatch.

Regards,
vaquar khan

On Jul 17, 2017 3:50 AM, "zzcclp" <441586...@qq.com> wrote:

Hi guys:
  I am using spark 2.1.1 to test on CDH 5.7.1, when i run on yarn with
following command, error 'NoSuchMethodError:
org.apache.spark.network.client.TransportClient.
getChannel()Lio/netty/channel/Channel;'
appears sometimes:

  command:
  *su cloudera-scm -s "/bin/sh" -c "/opt/spark2/bin/spark-shell --master
yarn --deploy-mode client --files
/opt/spark2/conf/log4j_all.properties#log4j.properties --driver-memory 8g
--num-executors 2 --executor-memory 8g --executor-cores 5
--driver-library-path :/opt/cloudera/parcels/CDH/lib/hadoop/lib/native
--driver-class-path /opt/spark2/libs/mysql-connector-java-5.1.36.jar --jars
/opt/spark2/libs/mysql-connector-java-5.1.36.jar " *

  error messages:
  2017-07-17 17:15:25,255 - WARN -
org.apache.spark.network.server.TransportChannelHandler.exceptionCaught(
TransportChannelHandler.java:78)
- rpc-client-1-1 -Exception in connection from /ip:60099
java.lang.NoSuchMethodError:
org.apache.spark.network.client.TransportClient.
getChannel()Lio/netty/channel/Channel;
at
org.apache.spark.rpc.netty.NettyRpcHandler.channelActive(
NettyRpcEnv.scala:614)
at
org.apache.spark.network.server.TransportRequestHandler.channelActive(
TransportRequestHandler.java:87)
at
org.apache.spark.network.server.TransportChannelHandler.channelActive(
TransportChannelHandler.java:88)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.
invokeChannelActive(AbstractChannelHandlerContext.java:219)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.
invokeChannelActive(AbstractChannelHandlerContext.java:205)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.
fireChannelActive(AbstractChannelHandlerContext.java:198)
at
org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.
channelActive(ChannelInboundHandlerAdapter.java:64)
at
org.spark_project.io.netty.handler.timeout.IdleStateHandler.channelActive(
IdleStateHandler.java:251)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.
invokeChannelActive(AbstractChannelHandlerContext.java:219)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.
invokeChannelActive(AbstractChannelHandlerContext.java:205)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.
fireChannelActive(AbstractChannelHandlerContext.java:198)
at
org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.
channelActive(ChannelInboundHandlerAdapter.java:64)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.
invokeChannelActive(AbstractChannelHandlerContext.java:219)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.
invokeChannelActive(AbstractChannelHandlerContext.java:205)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.
fireChannelActive(AbstractChannelHandlerContext.java:198)
at
org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.
channelActive(ChannelInboundHandlerAdapter.java:64)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.
invokeChannelActive(AbstractChannelHandlerContext.java:219)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.
invokeChannelActive(AbstractChannelHandlerContext.java:205)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.
fireChannelActive(AbstractChannelHandlerContext.java:198)
at
org.spark_project.io.netty.channel.DefaultChannelPipeline$
HeadContext.channelActive(DefaultChannelPipeline.java:1282)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.
invokeChannelActive(AbstractChannelHandlerContext.java:219)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.
invokeChannelActive(AbstractChannelHandlerContext.java:205)
at
org.spark_project.io.netty.channel.DefaultChannelPipeline.fireChannelActive(
DefaultChannelPipeline.java:887)
at
org.spark_project.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.
fulfillConnectPromise(AbstractNioChannel.java:262)
at
org.spark_project.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.
finishConnect(AbstractNioChannel.java:292)
at
org.spark_project.io.netty.channel.nio.NioEventLoop.processSelectedKey(
NioEventLoop.java:640)
at
org.spark_project.io.netty.channel.nio.NioEventLoop.
processSelectedKeysOptimized(NioEventLoop.java:575)
at
org.spark_project.io.netty.channel.nio.NioEventLoop.processSelectedKeys(
NioEventLoop.java:489)
at
org.spark_project.io.netty.channel.nio.NioEventLoop.run(
NioEventLoop.java:451)
at
org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor$2.
run(SingleThreadEventExecutor.java:140)
at

Spark 2.1.1 Error:java.lang.NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;

2017-07-17 Thread zzcclp 
Hi guys:
  I am using spark 2.1.1 to test on CDH 5.7.1, when i run on yarn with
following command, error 'NoSuchMethodError:
org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;'
appears sometimes:

  command:
  *su cloudera-scm -s "/bin/sh" -c "/opt/spark2/bin/spark-shell --master
yarn --deploy-mode client --files
/opt/spark2/conf/log4j_all.properties#log4j.properties --driver-memory 8g
--num-executors 2 --executor-memory 8g --executor-cores 5
--driver-library-path :/opt/cloudera/parcels/CDH/lib/hadoop/lib/native
--driver-class-path /opt/spark2/libs/mysql-connector-java-5.1.36.jar --jars
/opt/spark2/libs/mysql-connector-java-5.1.36.jar " *

  error messages:
  2017-07-17 17:15:25,255 - WARN -
org.apache.spark.network.server.TransportChannelHandler.exceptionCaught(TransportChannelHandler.java:78)
- rpc-client-1-1 -Exception in connection from /ip:60099
java.lang.NoSuchMethodError:
org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;
at
org.apache.spark.rpc.netty.NettyRpcHandler.channelActive(NettyRpcEnv.scala:614)
at
org.apache.spark.network.server.TransportRequestHandler.channelActive(TransportRequestHandler.java:87)
at
org.apache.spark.network.server.TransportChannelHandler.channelActive(TransportChannelHandler.java:88)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
at
org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
at
org.spark_project.io.netty.handler.timeout.IdleStateHandler.channelActive(IdleStateHandler.java:251)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
at
org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
at
org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
at
org.spark_project.io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1282)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
at
org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
at
org.spark_project.io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:887)
at
org.spark_project.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:262)
at
org.spark_project.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:292)
at
org.spark_project.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:640)
at
org.spark_project.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575)
at
org.spark_project.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489)
at
org.spark_project.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451)
at
org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
at
org.spark_project.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:745)
2017-07-17 17:15:25,256 - DEBUG -