I have recently moved to Kryo for serialization to get better performance.
Have written some of the serializers for my custom DS.
What could below exception be about: (I dont see any of my code line in the
stack trace)

java.lang.ArrayIndexOutOfBoundsException: -2
        at java.util.ArrayList.get(Unknown Source)
        at
com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42)
        at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:773)
        at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:727)
        at
org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:106)
        at
org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:101)
        at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
        at scala.collection.Iterator$$anon$21.hasNext(Iterator.scala:440)
        at
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:26)
        at scala.collection.Iterator$class.foreach(Iterator.scala:772)
        at
org.apache.spark.util.CompletionIterator.foreach(CompletionIterator.scala:23)
        at
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
        at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:102)
        at
scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:250)
        at
org.apache.spark.util.CompletionIterator.toBuffer(CompletionIterator.scala:23)
        at
scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:237)
        at
org.apache.spark.util.CompletionIterator.toArray(CompletionIterator.scala:23)
        at
org.apache.spark.rdd.OrderedRDDFunctions$$anonfun$sortByKey$1.apply(OrderedRDDFunctions.scala:44)
        at
org.apache.spark.rdd.OrderedRDDFunctions$$anonfun$sortByKey$1.apply(OrderedRDDFunctions.scala:43)
        at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:36)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:237)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:226)
        at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:29)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:237)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:226)
        at org.apache.spark.scheduler.ResultTask.run(ResultTask.scala:99)
        at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:158)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
        at java.lang.Thread.run(Unknown Source)

Any ideas? or Suggestions would help.

Thanks,
Archit.

Reply via email to