Could you try to comment out some lines in `extract_sift_features_opencv` to find which line cause the crash?
If the bytes came from sequenceFile() is broken, it's easy to crash a C library in Python (OpenCV). On Thu, May 28, 2015 at 8:33 AM, Sam Stoelinga <sammiest...@gmail.com> wrote: > Hi sparkers, > > I am working on a PySpark application which uses the OpenCV library. It runs > fine when running the code locally but when I try to run it on Spark on the > same Machine it crashes the worker. > > The code can be found here: > https://gist.github.com/samos123/885f9fe87c8fa5abf78f > > This is the error message taken from STDERR of the worker log: > https://gist.github.com/samos123/3300191684aee7fc8013 > > Would like pointers or tips on how to debug further? Would be nice to know > the reason why the worker crashed. > > Thanks, > Sam Stoelinga > > > org.apache.spark.SparkException: Python worker exited unexpectedly (crashed) > at org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:172) > at > org.apache.spark.api.python.PythonRDD$$anon$1.<init>(PythonRDD.scala:176) > at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:94) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) > at org.apache.spark.scheduler.Task.run(Task.scala:64) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.EOFException > at java.io.DataInputStream.readInt(DataInputStream.java:392) > at org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:108) > > > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
