Hi Sparklers,
I was wondering if some else has also encountered this... (Actually I am not
even sure if this is an issue)...
I have a spark job that reads data from Hbase does a bunch of transformation
sparkContext.newAPIHadoopRDD -> flatMapToPair -> groupByKey -> mapValues
After this I do a take(10) on the result to print it out in the log file.
I always get the results which I am 100% sure are correct. However, every once
in a while I get the following in the log file even if the results are correct -
14/09/22 20:16:22 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
14/09/22 20:16:22 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down
remote daemon.
14/09/22 20:16:22 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon
shut down; proceeding with flushing remote transports.
14/09/22 20:16:22 INFO Remoting: Remoting shut down
14/09/22 20:16:22 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut
down.
14/09/22 20:16:24 INFO ConnectionManager: Key not valid ?
sun.nio.ch.SelectionKeyImpl@a4503d8
14/09/22 20:16:24 INFO ConnectionManager: Removing SendingConnection to
ConnectionManagerId(tr-pan-xxxx-04,55008)
14/09/22 20:16:24 INFO ConnectionManager: Removing ReceivingConnection to
ConnectionManagerId(tr-pan-xxxx-04,55008)
14/09/22 20:16:24 ERROR ConnectionManager: Corresponding
SendingConnectionManagerId not found
14/09/22 20:16:24 INFO ConnectionManager: key already cancelled ?
sun.nio.ch.SelectionKeyImpl@a4503d8
java.nio.channels.CancelledKeyException
at
org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:363)
at
org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:116)
The spark dashboard also does not show any error of executors failing.
Would it be possible for someone to throw some light into what this actually
means? and whether we should be concerned about it?
I am running a Cloudera CDH 5.1.2 cluster with I believe spark v1.0.0
The spark job is submitted to yarn.
-abe
