[ 
https://issues.apache.org/jira/browse/SPARK-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582039#comment-15582039
 ] 

Vitaly Gerasimov commented on SPARK-17954:
------------------------------------------

There is a way to do it? I can't find how to bind executor to port with 
specific host listening.

Try make it clearer. I want executor open port with listening any port like 
that (in Spark 1.6):
{code}
~$ netstat -ntlp
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
PID/Program name
tcp6       0      0 :::22939                :::*                    LISTEN      
22277/java
{code}

Now it is:
{code}
~$ netstat -ntlp
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
PID/Program name
tcp6       0      0 127.0.0.1:48105         :::*                    LISTEN      
1318/java
{code}

So in Spark 1.6 I can do this:
{code}
~$ telnet worker1.test 22939
Trying x.x.x.x...
Connected to worker1.test.
Escape character is '^]'.
{code}

But in Spark 2.0 I cannot:
{code}
~# telnet worker1.test 48105
Trying x.x.x.x...
telnet: Unable to connect to remote host: Connection refused
{code}

I didn't find a way to explicitly bind by Spark config.

> FetchFailedException executor cannot connect to another worker executor
> -----------------------------------------------------------------------
>
>                 Key: SPARK-17954
>                 URL: https://issues.apache.org/jira/browse/SPARK-17954
>             Project: Spark
>          Issue Type: Bug
>    Affects Versions: 2.0.0, 2.0.1
>            Reporter: Vitaly Gerasimov
>
> I have standalone mode spark cluster wich have three nodes:
> master.test
> worker1.test
> worker2.test
> I am trying to run the next code in spark shell:
> {code}
> val json = spark.read.json("hdfs://master.test/json/a.js.gz", 
> "hdfs://master.test/json/b.js.gz")
> json.createOrReplaceTempView("messages")
> spark.sql("select count(*) from messages").show()
> {code}
> and I am getting the following exception:
> {code}
> org.apache.spark.shuffle.FetchFailedException: Failed to connect to 
> worker1.test/x.x.x.x:51029
>       at 
> org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:357)
>       at 
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:332)
>       at 
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:54)
>       at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
>       at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
>       at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
>       at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
>       at 
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
>       at 
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>       at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
>       at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.agg_doAggregateWithoutKey$(Unknown
>  Source)
>       at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
>  Source)
>       at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>       at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:370)
>       at 
> org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:246)
>       at 
> org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:240)
>       at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:784)
>       at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:784)
>       at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>       at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>       at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>       at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
>       at org.apache.spark.scheduler.Task.run(Task.scala:85)
>       at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>       at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Failed to connect to 
> worker1.test/x.x.x.x:51029
>       at 
> org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228)
>       at 
> org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179)
>       at 
> org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:96)
>       at 
> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)
>       at 
> org.apache.spark.network.shuffle.RetryingBlockFetcher.access$200(RetryingBlockFetcher.java:43)
>       at 
> org.apache.spark.network.shuffle.RetryingBlockFetcher$1.run(RetryingBlockFetcher.java:170)
>       at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>       ... 3 more
> Caused by: java.net.ConnectException: Connection refused: 
> worker1.test/x.x.x.x:51029
>       at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>       at 
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
>       at 
> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
>       at 
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>       ... 1 more
> {code}
> This exception happens on both workers. If I run spark cluster without one of 
> worker, it works fine. File size is ~ 128 MB. Both workers have 2 cores and 2 
> GB memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to