Re: OptionalDataException in spark

[Ravi Aggarwal]

Hi,

We are encountering java OptionalDataException in one of our spark jobs.
All the tasks of a stage passes (or atleast we do not see any error), but stage 
fails with above exception while getting task result.
And and this exception gets printed on driver.

Any pointers in this regard would be helpful.

Here is the stack-trace:

java.io.OptionalDataException
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1555)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at 
scala.collection.mutable.HashMap$$anonfun$readObject$1.apply(HashMap.scala:174)
at 
scala.collection.mutable.HashMap$$anonfun$readObject$1.apply(HashMap.scala:174)
at scala.collection.mutable.HashTable$class.init(HashTable.scala:109)
at scala.collection.mutable.HashMap.init(HashMap.scala:40)
at scala.collection.mutable.HashMap.readObject(HashMap.scala:174)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2136)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at 
org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1$$anonfun$apply$mcV$sp$2.apply(TaskResult.scala:67)
at 
org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1$$anonfun$apply$mcV$sp$2.apply(TaskResult.scala:66)
at scala.collection.immutable.Range.foreach(Range.scala:160)
at 
org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1.apply$mcV$sp(TaskResult.scala:66)
at 
org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1.apply(TaskResult.scala:55)
at 
org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1.apply(TaskResult.scala:55)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1326)
at 
org.apache.spark.scheduler.DirectTaskResult.readExternal(TaskResult.scala:55)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:2076)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2025)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at 
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
at 
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:108)
at 
org.apache.spark.scheduler.TaskResultGetter$$anon$3$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:64)
at 
org.apache.spark.scheduler.TaskResultGetter$$anon$3$$anonfun$run$1.apply(TaskResultGetter.scala:63)
at 
org.apache.spark.scheduler.TaskResultGetter$$anon$3$$anonfun$run$1.apply(TaskResultGetter.scala:63)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
at 
org.apache.spark.scheduler.TaskResultGetter$$anon$3.run(TaskResultGetter.scala:62)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Thanks

[Image removed by sender.]
Ravi Aggarwal
408.536.6719 (tel)
E11-355
Computer Scientist
669.214.1491 (cell)
San Jose, CA, 95110, US
Adobe. Make It an Experience.
raagg...@adobe.com
Adobe.com



Any additional, business necessary information, such as legal requirements for 
your region can go here – Arial, size 8, italicized.  Otherwise, delete this 
text.

Unsubscribe

[Amit Jain]

Unsubscribe

Hive External Table Partiton Data Type.

[SB M]

Hi All,
   Am trying to create a dynamic partition with external table on hive
metastore using spark sql.

when am trying to create a partition column data type as bigint, partition
is not working even i tried with repair table. data is not shown when i ran
sample query select * from table.


but if i tried to create a dynamic partition with string as data type for
partition its working fine. partition are working as expected.


is there something am doing it wrong ??


Regards,
Sree