On 13 May 2016 at 14:28, Arkay <robkee...@gmail.com> wrote: > Hi Vasia, > > It seems to work OK up to about 50MB of input, and dies after that point. > If i disable just this connected components step the rest of my program is > happy with the full 1.5GB test dataset. It seems to be specifically > limited > to GraphAlgorithms in my case. >
So your program has other steps before/after the connected components algorithm? Could it be that you have some expensive operation that competes for memory with the hash table? > > Do you know what the units are when it is saying Partition memory: 8388608? > If it is bytes then it sounds like its using around 256MB per hash table of > 32 partitions (which is then multiplied by number of task slots i guess). > Yes, that's bytes. > Can this number be configured do you know? Perhaps the windows version of > the JVM is defaulting it to a lower value than on Linux? > By default, the hash table uses Fink's managed memory. That's 3.0GB in your case (0.7 of the total memory by default). You can change this fraction by setting the "taskmanager.memory.fraction" in the configuration. See [1] for other managed memory options. Hope this helps! -Vasia. [1]: https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#managed-memory > > Thanks, > Rob > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Memory-ran-out-error-when-running-connected-components-tp6888p6899.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. >
