Hi Marc, The CompactingHashtable is not spillable (the only operator in batch actually that isn't), so you can only either reduce your data size or increase your memory. When you are using unmanaged memory, make sure the allocation of managed memory is reduced to a minimum.
On Mon, May 29, 2017 at 9:29 AM, Kaepke, Marc <marc.kae...@haw-hamburg.de> wrote: > Hi, > > I tried to run an vertex-centric and a gather scatter iteration in Gelly > and receive an exception like > > Exception in thread "main" > org.apache.flink.runtime.client.JobExecutionException: > Job execution failed. > at org.apache.flink.runtime.jobmanager.JobManager$$ > anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$ > mcV$sp(JobManager.scala:900) > at org.apache.flink.runtime.jobmanager.JobManager$$ > anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:843) > at org.apache.flink.runtime.jobmanager.JobManager$$ > anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:843) > at scala.concurrent.impl.Future$PromiseCompletingRunnable. > liftedTree1$1(Future.scala:24) > at scala.concurrent.impl.Future$PromiseCompletingRunnable.run( > Future.scala:24) > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) > at akka.dispatch.ForkJoinExecutorConfigurator$ > AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) > at scala.concurrent.forkjoin.ForkJoinTask.doExec( > ForkJoinTask.java:260) > at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue. > runTask(ForkJoinPool.java:1339) > at scala.concurrent.forkjoin.ForkJoinPool.runWorker( > ForkJoinPool.java:1979) > at scala.concurrent.forkjoin.ForkJoinWorkerThread.run( > ForkJoinWorkerThread.java:107) > Caused by: java.lang.RuntimeException: Memory ran out. Compaction failed. > numPartitions: 32 minPartition: 37 maxPartition: 96 number of overflow > segments: 0 bucketSize: 476 Overall memory: 85688320 Partition memory: > 67043328 Message: null > at org.apache.flink.runtime.operators.hash.CompactingHashTable. > insertRecordIntoPartition(CompactingHashTable.java:456) > at org.apache.flink.runtime.operators.hash.CompactingHashTable. > insertOrReplaceRecord(CompactingHashTable.java:405) > at org.apache.flink.runtime.operators.hash.CompactingHashTable. > buildTableWithUniqueKey(CompactingHashTable.java:316) > at org.apache.flink.runtime.iterative.task.IterationHeadTask. > readInitialSolutionSet(IterationHeadTask.java:210) > at org.apache.flink.runtime.iterative.task.IterationHeadTask.run( > IterationHeadTask.java:271) > at org.apache.flink.runtime.operators.BatchTask.invoke( > BatchTask.java:355) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655) > at java.lang.Thread.run(Thread.java:745) > > > parameter.setSolutionSetUnmanagedMemory(true); doesn’t work for me. > > > Please help. > > > Best, > Marc
