Hi Gabor, I don't think there is a way to tune the memory settings for specific operators.
For that you would need to change the memory allocation in the optimizers, which is possible but not a lightweight change either. If you want to get something working, you could add a method to the API to manually specify a memory fraction. The information could be passed through the API to the optimizer which takes the explicitly specified fraction into account when assigning memory budgets (see Optimizer.java [1] and PlanFinalizer.java [2]) for how that works. Cheers, Fabian [1] https://github.com/apache/flink/blob/master/flink-optimizer/src/main/java/org/apache/flink/optimizer/Optimizer.java [2] https://github.com/apache/flink/blob/master/flink-optimizer/src/main/java/org/apache/flink/optimizer/traversals/PlanFinalizer.java 2016-11-16 17:29 GMT+01:00 Gábor Hermann <[email protected]>: > Hi all, > > Is there any way to increase the sort buffer size other than increasing > the overall TaskManager memory? > The following error comes up running a job with huge matrix block objects > on a cluster: > > Error obtaining the sorted input: Thread 'SortMerger Reading Thread' > terminated due to an exception: The record exceeds the maximum size of a > sort buffer (current maximum: 100499456 bytes). > > Every TM has at least 40 GB of memory while the maximum sort buffer size > is at 100 MB. What is the reason for this limit? Sorry if I'm missing > something, but I have not found any related discussion or documentation yet. > > Cheers, > Gabor > >
