Hi All,
I am facing this issue in my production environment.
My worker dies by throwing this exception.
But i see the space is available on all the partitions on my disk
I did NOT see any abrupt increase in DIsk IO, which might have choked the
executor to write on to the stderr file.
But still my worker dies, this is not happening on all my workers, it's one
machine that is performing this way.
Could you please help me debug if it is happening because i am doing
something wrong, or some issue from hardware/OS perspective, that i can
debug and fix.
15/07/11 18:05:45 ERROR Worker: RECEIVED SIGNAL 1: SIGHUP
15/07/11 18:05:45 INFO ExecutorRunner: Killing process!
15/07/11 18:05:45 ERROR FileAppender: Error writing stream to file
/opt/spark-1.4.0-bin-hadoop2.6/work/app-20150710162005-0001/16517/stderr
java.io.IOException: Stream closed
at
java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at
org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70)
at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
at
org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772)
at
org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
15/07/11 18:05:46 INFO Utils: Shutdown hook called
15/07/11 18:05:46 INFO Utils: Deleting directory
/tmp/spark-f269acd9-3ab0-4b3c-843c-bcf2e8c2669f
15/07/11 18:05:46 INFO Worker: Executor app-20150710162005-0001/16517
finished with state EXITED message Command exited with code 129 exitStatus
129
