Hi Patrick,
I still get the exception on lastest
master 05be7047744c88e64e7e6bd973f9bcfacd00da5f. A bit more info on the
subject. I'm using KryoSerialzation with a custom serialization function,
and the exception come from a rdd
operation
combineByKey(createDict,combineKey,mergeDict,partitioner,true,"org.apache.spark.serializer.KryoSerializer").
All previous operation seems ok. The only difference is that this operation
can generate some a large dict object around 1 gb size. I hope this can
give you some clue what might go wrong. I'm still having trouble figure
out the cause.
Thanks,
Jiacheng Guo
On Wed, Jan 22, 2014 at 1:36 PM, Patrick Wendell <[email protected]> wrote:
> This code has been modified since you reported this so you may want to
> try the current master.
>
> - Patrick
>
> On Mon, Jan 20, 2014 at 4:22 AM, guojc <[email protected]> wrote:
> > Hi,
> > I'm tring out lastest master branch of spark for the exciting external
> > hashmap feature. I have a code that is running correctly at spark 0.8.1
> and
> > I only make a change for its easily to be spilled to disk. However, I
> > encounter a few task failure of
> > java.util.NoSuchElementException (java.util.NoSuchElementException)
> >
> org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.next(ExternalAppendOnlyMap.scala:277)org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.next(ExternalAppendOnlyMap.scala:212)org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:29)
> > And the job seems to fail to recover.
> > Can anyone give some suggestion on how to investigate the issue?
> > Thanks,Jiacheng Guo
>
