Re: Spark program throws NIO Buffer over flow error (TDigest - Ted Dunning lib)

[๏̯͡๏]

Same exception with different values of compression (10,100)

  var digest: TDigest = TDigest.createAvlTreeDigest(100)

On Wed, Jul 8, 2015 at 6:50 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> wrote:

> Any suggestions ?
> Code:
>
>     val dimQuantiles = genericRecordsAndKeys
>
>       .map {
>
>         case (keyToOutput, rec) =>
>
>           var digest: TDigest = TDigest.createAvlTreeDigest(10000)
>
>           val fpPaidGMB = rec.get("fpPaidGMB").asInstanceOf[Double]
>
>           digest.add(fpPaidGMB)
>
>           var bbuf: ByteBuffer = ByteBuffer.allocate(digest.byteSize());
>
>           digest.asBytes(bbuf);
>
>           (keyToOutput.toString, bbuf.array())
>
>       }
>
>
> Trace:
> aborted due to stage failure: Task 4487 in stage 1.0 failed 4 times, most
> recent failure: Lost task 4487.3 in stage 1.0 (TID 281,
> phxaishdc9dn0111.phx.ebay.com): java.nio.BufferOverflowException
> at java.nio.Buffer.nextPutIndex(Buffer.java:519)
> at java.nio.HeapByteBuffer.putDouble(HeapByteBuffer.java:519)
> at com.tdunning.math.stats.AVLTreeDigest.asBytes(AVLTreeDigest.java:336)
> at
> com.ebay.ep.poc.spark.reporting.process.service.OutlierService$$anonfun$4.apply(OutlierService.scala:96)
> at
> com.ebay.ep.poc.spark.reporting.process.service.OutlierService$$anonfun$4.apply(OutlierService.scala:89)
> at
> org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:197)
> at
> org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:196)
> at
> org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:138)
> at
> org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32)
> at
> org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:202)
> at
> org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:56)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:70)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
>
>
> --
> Deepak
>
>


-- 
Deepak