SparkStreaming + Flume: org.jboss.netty.channel.ChannelException: Failed to bind to: master60/10.0.10.60:31001

[luohui20001]

Hi there      I have a flume cluster sending messages to SparkStreaming. I got 
an exception like below:16/08/25 23:00:54 ERROR ReceiverTracker: Deregistered 
receiver for stream 0: Error starting receiver 0 - 
org.jboss.netty.channel.ChannelException: Failed to bind to: 
master60/10.0.10.60:31001
        at 
org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
        at org.apache.avro.ipc.NettyServer.<init>(NettyServer.java:106)
        at org.apache.avro.ipc.NettyServer.<init>(NettyServer.java:119)
        at org.apache.avro.ipc.NettyServer.<init>(NettyServer.java:74)
        at org.apache.avro.ipc.NettyServer.<init>(NettyServer.java:68)
        at 
org.apache.spark.streaming.flume.FlumeReceiver.initServer(FlumeInputDStream.scala:162)
        at 
org.apache.spark.streaming.flume.FlumeReceiver.onStart(FlumeInputDStream.scala:169)
        at 
org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:148)
        at 
org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:130)
        at 
org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:575)
        at 
org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:565)
        at 
org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1992)
        at 
org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1992)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
        at org.apache.spark.scheduler.Task.run(Task.scala:89)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.BindException: Cannot assign requested address
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at 
org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
        at 
org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372)
        at 
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296)
        at 
org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
        ... 3 more


I checked the port like below:[hadoop@master60 shellscripts]$ netstat -an|grep 
31001
tcp        0      0 :::31001                    :::*                        
LISTEN
tcp        0      0 ::ffff:10.0.10.60:31001     ::ffff:10.0.30.199:34773    
ESTABLISHED

It seems that the workers on other nodes(except the one on master60) could not 
connect the port 31001 on master60 However I tried[hadoop@slave61 ~]$ telnet 
master60 31001
Trying 10.0.10.60...
Connected to master60.
Escape character is '^]'.
So I didn't quit get why the executors can not build connections?BTW, I am 
using Spark1.6.1, flume 1.6.0
Any idea will be appreciated.



--------------------------------

 

Thanks&amp;Best regards!
San.Luo