Hi,
We are encountering java OptionalDataException in one of our spark jobs.
All the tasks of a stage passes (or atleast we do not see any error), but stage
fails with above exception while getting task result.
And and this exception gets printed on driver.
Any pointers in this regard would be helpful.
Here is the stack-trace:
java.io.OptionalDataException
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1555)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at
scala.collection.mutable.HashMap$$anonfun$readObject$1.apply(HashMap.scala:174)
at
scala.collection.mutable.HashMap$$anonfun$readObject$1.apply(HashMap.scala:174)
at scala.collection.mutable.HashTable$class.init(HashTable.scala:109)
at scala.collection.mutable.HashMap.init(HashMap.scala:40)
at scala.collection.mutable.HashMap.readObject(HashMap.scala:174)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2136)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at
org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1$$anonfun$apply$mcV$sp$2.apply(TaskResult.scala:67)
at
org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1$$anonfun$apply$mcV$sp$2.apply(TaskResult.scala:66)
at scala.collection.immutable.Range.foreach(Range.scala:160)
at
org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1.apply$mcV$sp(TaskResult.scala:66)
at
org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1.apply(TaskResult.scala:55)
at
org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1.apply(TaskResult.scala:55)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1326)
at
org.apache.spark.scheduler.DirectTaskResult.readExternal(TaskResult.scala:55)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:2076)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2025)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:108)
at
org.apache.spark.scheduler.TaskResultGetter$$anon$3$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:64)
at
org.apache.spark.scheduler.TaskResultGetter$$anon$3$$anonfun$run$1.apply(TaskResultGetter.scala:63)
at
org.apache.spark.scheduler.TaskResultGetter$$anon$3$$anonfun$run$1.apply(TaskResultGetter.scala:63)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
at
org.apache.spark.scheduler.TaskResultGetter$$anon$3.run(TaskResultGetter.scala:62)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Thanks
[Image removed by sender.]<http://adobe.com/>
Ravi Aggarwal
408.536.6719 (tel)
E11-355
Computer Scientist
669.214.1491 (cell)
San Jose, CA, 95110, US
Adobe. Make It an Experience.
[email protected]
Adobe.com<http://www.adobe.com/>
Any additional, business necessary information, such as legal requirements for
your region can go here – Arial, size 8, italicized. Otherwise, delete this
text.
