Hi,
I also get the EOFException on 0.9-SNAPSHOT when using a modified
SingleSourceShortestPaths on big graphs.
After I set the SolutionSet to "unmanaged" the job finishes. But this
only works until a certain point, over which I cannot increase the input
size further without getting an OutOfMemoryError exception.
Best,
Mihail
On 19.04.2015 13:52, Stefan Bunk wrote:
Hi,
I tested three configurations:
taskmanager.heap.mb: 6144, taskmanager.memory.fraction: 0.5
taskmanager.heap.mb: 5544, taskmanager.memory.fraction: 0.6
taskmanager.heap.mb: 5144, taskmanager.memory.fraction: 0.7
The error occurs in all three configurations.
In the last configuration, I can even find another exception in the
logs of one of the taskmanagers:
19.Apr. 13:39:29 INFO Task -
IterationHead(WorksetIteration (Resolved-Redirects)) (10/10) switched
to FAILED : java.lang.IndexOutOfBoundsException: Index: 161, Size: 161
at java.util.ArrayList.rangeCheck(ArrayList.java:635)
at java.util.ArrayList.get(ArrayList.java:411)
at
org.apache.flink.runtime.operators.hash.InMemoryPartition$WriteView.resetTo(InMemoryPartition.java:352)
at
org.apache.flink.runtime.operators.hash.InMemoryPartition$WriteView.access$100(InMemoryPartition.java:301)
at
org.apache.flink.runtime.operators.hash.InMemoryPartition.appendRecord(InMemoryPartition.java:226)
at
org.apache.flink.runtime.operators.hash.CompactingHashTable.insertOrReplaceRecord(CompactingHashTable.java:536)
at
org.apache.flink.runtime.operators.hash.CompactingHashTable.buildTableWithUniqueKey(CompactingHashTable.java:347)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.readInitialSolutionSet(IterationHeadPactTask.java:209)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:270)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:362)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:217)
at java.lang.Thread.run(Thread.java:745)
Greetings
Stefan
On 17 April 2015 at 16:05, Stephan Ewen <[email protected]
<mailto:[email protected]>> wrote:
Hi!
After a quick look over the code, this seems like a bug. One
cornercase of the overflow handling code does not check for the
"running out of memory" condition.
I would like to wait if Robert Waury has some ideas about that, he
is the one most familiar with the code.
I would guess, though, that you should be able to work around
that by either setting the solution set as "unmanaged", or by
slightly changing the memory configuration. It seems a rare
cornercase, only if the memory runs out in a very special
situation. You may be able to avoid running into it when using
slightly more or less memory.
Greetings,
Stephan
On Fri, Apr 17, 2015 at 3:59 PM, Stefan Bunk
<[email protected] <mailto:[email protected]>>
wrote:
Hi Squirrels,
I have some trouble with a delta-iteration transitive closure
program [1].
When I run the program, I get the following error:
java.io.EOFException
at
org.apache.flink.runtime.operators.hash.InMemoryPartition$WriteView.nextSegment(InMemoryPartition.java:333)
at
org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.advance(AbstractPagedOutputView.java:140)
at
org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.writeByte(AbstractPagedOutputView.java:223)
at
org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.writeLong(AbstractPagedOutputView.java:291)
at
org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.writeDouble(AbstractPagedOutputView.java:307)
at
org.apache.flink.api.common.typeutils.base.DoubleSerializer.serialize(DoubleSerializer.java:62)
at
org.apache.flink.api.common.typeutils.base.DoubleSerializer.serialize(DoubleSerializer.java:26)
at
org.apache.flink.api.scala.typeutils.CaseClassSerializer.serialize(CaseClassSerializer.scala:89)
at
org.apache.flink.api.scala.typeutils.CaseClassSerializer.serialize(CaseClassSerializer.scala:29)
at
org.apache.flink.runtime.operators.hash.InMemoryPartition.appendRecord(InMemoryPartition.java:219)
at
org.apache.flink.runtime.operators.hash.CompactingHashTable.insertOrReplaceRecord(CompactingHashTable.java:536)
at
org.apache.flink.runtime.operators.hash.CompactingHashTable.buildTableWithUniqueKey(CompactingHashTable.java:347)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.readInitialSolutionSet(IterationHeadPactTask.java:209)
at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:270)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:362)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:217)
at java.lang.Thread.run(Thread.java:745)
Both input files have been generated and written to HDFS by
Flink jobs. I already ran the Flink program that generated
them several times: the error persists.
You can find the logs at [2] and [3].
I am using the 0.9.0-milestone-1 release.
Best,
Stefan
[1] Flink job: https://gist.github.com/knub/0bfec859a563009c1d57
[2] Job manager logs:
https://gist.github.com/knub/01e3a4b0edb8cde66ff4
[3] One task manager's logs:
https://gist.github.com/knub/8f2f953da95c8d7adefc