What do you exactly mean by dataset size? Is that the size of the input data or the size of the data that is being held in-memory by the iterations?
The exception is not caused by the input data size. I think you are working on the graph closeness algorithms. In this case, the exception is probably caused because your sketches in the nodes are too large (or you have too many nodes). On Tue, Jun 24, 2014 at 5:03 PM, Stephan Ewen <[email protected]> wrote: > The delta iterations are currently still memory sensitive (the last > component in flink that is). > > We are working on providing better memory usage for them, but the only > short term solution is to increase the memory size of the JVMs that run the > code, or to use more machines. > > Stephan > > > > On Tue, Jun 24, 2014 at 4:59 PM, Janani Chakkaradhari < > [email protected]> wrote: > > > Hi, > > > > I am facing the following exception while using Delta iteration. The > > dataset size is 27GB. Any suggestions to overcome this issue? > > > > 15:14:30,431 ERROR > > eu.stratosphere.pact.runtime.task.RegularPactTask - Error in > > task code: SolutionSet Delta (142/284) > > java.lang.RuntimeException: Memory ran out. Compaction failed. > > numPartitions: 32 minPartition: 23 maxPartition: 40 number of overflow > > segments: 15 bucketSize: 242 Message: Index: 59, Size: 24 > > at > > > > > eu.stratosphere.pact.runtime.hash.CompactingHashTable.insertOrReplaceRecord(CompactingHashTable.java:526) > > at > > > > > eu.stratosphere.pact.runtime.iterative.io.SolutionSetUpdateOutputCollector.collect(SolutionSetUpdateOutputCollector.java:55) > > at > > eu.stratosphere.pact.runtime.task.NoOpDriver.run(NoOpDriver.java:64) > > at > > > > > eu.stratosphere.pact.runtime.task.RegularPactTask.run(RegularPactTask.java:505) > > at > > > > > eu.stratosphere.pact.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:132) > > at > > > > > eu.stratosphere.pact.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:100) > > at > > > > > eu.stratosphere.pact.runtime.task.RegularPactTask.invoke(RegularPactTask.java:370) > > at > > > > > eu.stratosphere.nephele.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:284) > > at java.lang.Thread.run(Thread.java:853) > > > > Thanks, > > Janani > > >
