Hi Spark devs,
I'm seeing a stacktrace where the classloader that reads from the REPL is
hung, and blocking all progress on that executor. Below is that hung
thread's stacktrace, and also the stacktrace of another hung thread.
I thought maybe there was an issue with the REPL's JVM on the other side,
but didn't see anything useful in that stacktrace either.
Any ideas what I should be looking for?
Thanks!
Andrew
"Executor task launch worker-0" daemon prio=10 tid=0x00007f780c208000
nid=0x6ae9 runnable [0x00007f78c2eeb000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00007f7e13ea9560> (a java.io.BufferedInputStream)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
- locked <0x00007f7e13e9eeb0> (a
sun.net.www.protocol.http.HttpURLConnection)
at java.net.URL.openStream(URL.java:1037)
at
org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:86)
at
org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:63)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
- locked <0x00007f7fc9018980> (a
org.apache.spark.repl.ExecutorClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.avro.util.ClassUtils.forName(ClassUtils.java:102)
at org.apache.avro.util.ClassUtils.forName(ClassUtils.java:82)
at
org.apache.avro.specific.SpecificData.getClass(SpecificData.java:132)
at
org.apache.avro.specific.SpecificDatumReader.setSchema(SpecificDatumReader.java:69)
at
org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:126)
at
org.apache.avro.file.DataFileReader.<init>(DataFileReader.java:97)
at
org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:59)
at
org.apache.avro.mapred.AvroRecordReader.<init>(AvroRecordReader.java:41)
at
org.apache.avro.mapred.AvroInputFormat.getRecordReader(AvroInputFormat.java:71)
at
org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:193)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:184)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:93)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
And the other threads are stuck on the Class.forName0() method too:
"Executor task launch worker-4" daemon prio=10 tid=0x00007f780c20f000
nid=0x6aed waiting for monitor entry [0x00007f78c2ae8000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.avro.util.ClassUtils.forName(ClassUtils.java:102)
at org.apache.avro.util.ClassUtils.forName(ClassUtils.java:79)
at
org.apache.avro.specific.SpecificData.getClass(SpecificData.java:132)
at
org.apache.avro.specific.SpecificDatumReader.setSchema(SpecificDatumReader.java:69)
at
org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:126)
at
org.apache.avro.file.DataFileReader.<init>(DataFileReader.java:97)
at
org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:59)
at
org.apache.avro.mapred.AvroRecordReader.<init>(AvroRecordReader.java:41)
at
org.apache.avro.mapred.AvroInputFormat.getRecordReader(AvroInputFormat.java:71)
at
org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:193)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:184)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:93)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
asdf
