Hi, I have a stand alone Spark Cluster, where worker and master reside on the same machine. I submit a job to the cluster, the job is executed for a while and suddenly I get this exception with no additional trace.
ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@2490dce9 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) Any idea where should I look for the cause? best, /shahab This following is the part of printout from "driver application" logs: 14/10/27 15:21:15 INFO BlockManagerInfo: Removed broadcast_1_piece0 on ip-10-89-32-179.eu-west-1.compute.internal:40479 in memory (size: 3.4 KB, free: 1565.6 MB) 14/10/27 15:21:15 INFO ContextCleaner: Cleaned broadcast 1 14/10/27 15:21:15 INFO ShuffleBlockManager: Could not find files for shuffle 1 for deleting 14/10/27 15:21:15 INFO ContextCleaner: Cleaned shuffle 1 14/10/27 15:21:15 INFO ShuffleBlockManager: Could not find files for shuffle 0 for deleting 14/10/27 15:21:15 INFO ContextCleaner: Cleaned shuffle 0 14/10/27 15:21:15 INFO BlockManagerInfo: Removed taskresult_9 on ip-10-zz.xx-yy:40479 in memory (size: 24.1 MB, free: 1589.8 MB) 14/10/27 15:21:16 INFO DAGScheduler: Stage 7 (collect at TimeBenchmarking_SimpleModel.scala:55) finished in 3.209 s 14/10/27 15:21:16 INFO TaskSetManager: Finished task 0.0 in stage 7.0 (TID 9) in 2640 ms onip-10-zz.xx-yy (1/1) 14/10/27 15:21:16 INFO SparkContext: Job finished: collect at TimeBenchmarking_SimpleModel.scala:55, took 102.661420511 s 14/10/27 15:21:16 INFO TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool 14/10/27 15:21:16 INFO SparkUI: Stopped Spark web UI at http://ip-10-zz.xx-yy:4040 14/10/27 15:21:16 INFO DAGScheduler: Stopping DAGScheduler 14/10/27 15:21:16 INFO SparkDeploySchedulerBackend: Shutting down all executors 14/10/27 15:21:16 INFO SparkDeploySchedulerBackend: Asking each executor to shut down 14/10/27 15:21:16 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-10-zz.xx-yy, 40479) 14/10/27 15:21:16 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-10-zz.xx-yy,40479) 14/10/27 15:21:16 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-10-zz.xx-yy,40479) 14/10/27 15:21:16 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@2490dce9 14/10/27 15:21:16 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@2490dce9 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 14/10/27 15:21:17 INFO MapOutputTrackerMasterActor: MapOutputTrackerActor stopped! 14/10/27 15:21:17 INFO ConnectionManager: Selector thread was interrupted!
