Hi!

After a quick look over the code, this seems like a bug. One cornercase of
the overflow handling code does not check for the "running out of memory"
condition.

I would like to wait if Robert Waury has some ideas about that, he is the
one most familiar with the code.

I would guess, though, that you should  be able to work around that by
either setting the solution set as "unmanaged", or by slightly changing the
memory configuration. It seems a rare cornercase, only if the memory runs
out in a very special situation. You may be able to avoid running into it
when using slightly more or less memory.

Greetings,
Stephan


On Fri, Apr 17, 2015 at 3:59 PM, Stefan Bunk <[email protected]>
wrote:

> Hi Squirrels,
>
> I have some trouble with a delta-iteration transitive closure program [1].
> When I run the program, I get the following error:
>
> java.io.EOFException
> at
> org.apache.flink.runtime.operators.hash.InMemoryPartition$WriteView.nextSegment(InMemoryPartition.java:333)
> at
> org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.advance(AbstractPagedOutputView.java:140)
> at
> org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.writeByte(AbstractPagedOutputView.java:223)
> at
> org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.writeLong(AbstractPagedOutputView.java:291)
> at
> org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.writeDouble(AbstractPagedOutputView.java:307)
> at
> org.apache.flink.api.common.typeutils.base.DoubleSerializer.serialize(DoubleSerializer.java:62)
> at
> org.apache.flink.api.common.typeutils.base.DoubleSerializer.serialize(DoubleSerializer.java:26)
> at
> org.apache.flink.api.scala.typeutils.CaseClassSerializer.serialize(CaseClassSerializer.scala:89)
> at
> org.apache.flink.api.scala.typeutils.CaseClassSerializer.serialize(CaseClassSerializer.scala:29)
> at
> org.apache.flink.runtime.operators.hash.InMemoryPartition.appendRecord(InMemoryPartition.java:219)
> at
> org.apache.flink.runtime.operators.hash.CompactingHashTable.insertOrReplaceRecord(CompactingHashTable.java:536)
> at
> org.apache.flink.runtime.operators.hash.CompactingHashTable.buildTableWithUniqueKey(CompactingHashTable.java:347)
> at
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.readInitialSolutionSet(IterationHeadPactTask.java:209)
> at
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:270)
> at
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:362)
> at
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:217)
> at java.lang.Thread.run(Thread.java:745)
>
> Both input files have been generated and written to HDFS by Flink jobs. I
> already ran the Flink program that generated them several times: the error
> persists.
> You can find the logs at [2] and [3].
>
> I am using the 0.9.0-milestone-1 release.
>
> Best,
> Stefan
>
>
> [1] Flink job: https://gist.github.com/knub/0bfec859a563009c1d57
> [2] Job manager logs: https://gist.github.com/knub/01e3a4b0edb8cde66ff4
> [3] One task manager's logs:
> https://gist.github.com/knub/8f2f953da95c8d7adefc
>

Reply via email to