Guoqiang Li created SPARK-2701:
----------------------------------
Summary: ConnectionManager throws out of "Could not find
reference for received ack message xxx" exception
Key: SPARK-2701
URL: https://issues.apache.org/jira/browse/SPARK-2701
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.0.0, 1.0.1
Reporter: Guoqiang Li
{noformat}
14/07/26 22:05:15 INFO network.ConnectionManager: Key not valid ?
sun.nio.ch.SelectionKeyImpl@13c7f699
14/07/26 22:05:15 INFO network.ConnectionManager: Removing SendingConnection to
ConnectionManagerId(tuan222,49423)
14/07/26 22:05:15 INFO spark.CacheManager: Partition rdd_51_418 not found,
computing it
14/07/26 22:05:15 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator:
maxBytesInFlight: 50331648, targetRequestSize: 10066329
14/07/26 22:05:15 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator:
Getting 1024 non-empty blocks out of 1024 blocks
14/07/26 22:05:15 INFO network.ConnectionManager: key already cancelled ?
sun.nio.ch.SelectionKeyImpl@13c7f699
java.nio.channels.CancelledKeyException
at
org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:363)
at
org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:116)
14/07/26 22:05:15 INFO network.ConnectionManager: Notifying
org.apache.spark.network.ConnectionManager$MessageStatus@67c56c61
14/07/26 22:05:15 ERROR storage.BlockFetcherIterator$BasicBlockFetcherIterator:
Could not get block(s) from ConnectionManagerId(tuan222,49423)
14/07/26 22:05:15 INFO network.SendingConnection: Initiating connection to
[tuan222/172.16.1.222:49423]
14/07/26 22:05:15 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator:
Started 79 remote fetches in 69 ms
14/07/26 22:05:15 INFO network.ConnectionManager: Removing ReceivingConnection
to ConnectionManagerId(tuan222,49423)
14/07/26 22:05:15 ERROR executor.ExecutorUncaughtExceptionHandler: Uncaught
exception in thread Thread[pool-1-thread-2,5,main]
java.lang.Error: java.lang.Exception: Could not find reference for received ack
message 222491
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1151)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.Exception: Could not find reference for received ack
message 222491
at
org.apache.spark.network.ConnectionManager.org$apache$spark$network$ConnectionManager$$handleMessage(ConnectionManager.scala:647)
at
org.apache.spark.network.ConnectionManager$$anon$9.run(ConnectionManager.scala:504)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
... 2 more
14/07/26 22:05:15 WARN network.SendingConnection: Error finishing connection to
tuan222/172.16.1.222:49423
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at
org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:318)
at
org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:203)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
14/07/26 22:05:15 INFO network.ConnectionManager: Removing SendingConnection to
ConnectionManagerId(tuan222,49423)
14/07/26 22:05:15 INFO network.ConnectionManager: Notifying
org.apache.spark.network.ConnectionManager$MessageStatus@1324c9dc
14/07/26 22:05:15 ERROR storage.BlockFetcherIterator$BasicBlockFetcherIterator:
Could not get block(s) from ConnectionManagerId(tuan222,49423)
14/07/26 22:05:15 INFO network.ConnectionManager: Handling connection error on
connection to ConnectionManagerId(tuan222,49423)
14/07/26 22:05:15 INFO network.ConnectionManager: Removing SendingConnection to
ConnectionManagerId(tuan222,49423)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
