Memory segment error when migrating functional code from Flink 0.9 to 0.8
Hello everyone, I am implementing a graph algorithm as part of a course and I will also add it to the Flink- Gelly examples. My problem is that I started developing it in the Gelly repository, which runs on flink 0.9. It works like a charm there, but in order to test in on a cluster to see its real capabilities, I need to move it to the course repository, which runs on flink 0.8. Initially, I thought this migration should occur without incidents since flink 0.8 is more stable. Instead, I got the following exception: java.lang.IllegalArgumentException: *Too few memory segments provided. Hash Table needs at least 33 memory segments.* at org.apache.flink.runtime.operators.hash.CompactingHashTable.init(CompactingHashTable.java:238) at org.apache.flink.runtime.operators.hash.CompactingHashTable.init(CompactingHashTable.java:227) at org.apache.flink.runtime.iterative.task.IterationHeadPactTask.initCompactingHashTable(IterationHeadPactTask.java:177) at org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:279) at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360) at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257) at java.lang.Thread.run(Thread.java:745) This is the code for Gelly, where all tests pass: https://github.com/andralungu/flink-graph/tree/minSpanningTree Unfortunately, the code for the course is private, so you cannot actually see it... maybe @aalexandrov can do something about the privacy settings of this repo. https://github.com/andralungu/IMPRO-3.WS14/tree/dmst_algorithm Thank you! Andra
Re: Memory segment error when migrating functional code from Flink 0.9 to 0.8
Hi Andra, have you tried increasing the number of network buffers in your cluster? You can control by the configuration value: taskmanager.network.numberOfBuffers: #numberBuffers Greets, Till On Mon, Feb 9, 2015 at 9:56 AM, Andra Lungu lungu.an...@gmail.com wrote: Hello everyone, I am implementing a graph algorithm as part of a course and I will also add it to the Flink- Gelly examples. My problem is that I started developing it in the Gelly repository, which runs on flink 0.9. It works like a charm there, but in order to test in on a cluster to see its real capabilities, I need to move it to the course repository, which runs on flink 0.8. Initially, I thought this migration should occur without incidents since flink 0.8 is more stable. Instead, I got the following exception: java.lang.IllegalArgumentException: *Too few memory segments provided. Hash Table needs at least 33 memory segments.* at org.apache.flink.runtime.operators.hash.CompactingHashTable.init(CompactingHashTable.java:238) at org.apache.flink.runtime.operators.hash.CompactingHashTable.init(CompactingHashTable.java:227) at org.apache.flink.runtime.iterative.task.IterationHeadPactTask.initCompactingHashTable(IterationHeadPactTask.java:177) at org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:279) at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360) at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257) at java.lang.Thread.run(Thread.java:745) This is the code for Gelly, where all tests pass: https://github.com/andralungu/flink-graph/tree/minSpanningTree Unfortunately, the code for the course is private, so you cannot actually see it... maybe @aalexandrov can do something about the privacy settings of this repo. https://github.com/andralungu/IMPRO-3.WS14/tree/dmst_algorithm Thank you! Andra
Re: Memory segment error when migrating functional code from Flink 0.9 to 0.8
This is actually a problem of the number of memory segments available to the hash table for the solution set. For complex pipelines, memory currently gets too fragmented. There are two workarounds, until we do the dynamic memory management, or break it into shorter pipelines: Break the job up into subparts, or move the solution set to the user code part of the heap. There is a flag to attach to the Delta Iterations, see DeltaIteration#setSolutionSetUnManaged() Greetinigs, Stephan On Mon, Feb 9, 2015 at 10:32 AM, Till Rohrmann trohrm...@apache.org wrote: Hi Andra, have you tried increasing the number of network buffers in your cluster? You can control by the configuration value: taskmanager.network.numberOfBuffers: #numberBuffers Greets, Till On Mon, Feb 9, 2015 at 9:56 AM, Andra Lungu lungu.an...@gmail.com wrote: Hello everyone, I am implementing a graph algorithm as part of a course and I will also add it to the Flink- Gelly examples. My problem is that I started developing it in the Gelly repository, which runs on flink 0.9. It works like a charm there, but in order to test in on a cluster to see its real capabilities, I need to move it to the course repository, which runs on flink 0.8. Initially, I thought this migration should occur without incidents since flink 0.8 is more stable. Instead, I got the following exception: java.lang.IllegalArgumentException: *Too few memory segments provided. Hash Table needs at least 33 memory segments.* at org.apache.flink.runtime.operators.hash.CompactingHashTable.init(CompactingHashTable.java:238) at org.apache.flink.runtime.operators.hash.CompactingHashTable.init(CompactingHashTable.java:227) at org.apache.flink.runtime.iterative.task.IterationHeadPactTask.initCompactingHashTable(IterationHeadPactTask.java:177) at org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:279) at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360) at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257) at java.lang.Thread.run(Thread.java:745) This is the code for Gelly, where all tests pass: https://github.com/andralungu/flink-graph/tree/minSpanningTree Unfortunately, the code for the course is private, so you cannot actually see it... maybe @aalexandrov can do something about the privacy settings of this repo. https://github.com/andralungu/IMPRO-3.WS14/tree/dmst_algorithm Thank you! Andra
