Github user ramkrish86 commented on the issue:
https://github.com/apache/flink/pull/2495
@StephanEwen
Thank you. I agree this is a super important piece of FLINK where the
memory is involved and that is one reason why I wanted to know its managment
and usage and wanted to work on this. I have already split this into 2 tasks.
One that deals with sorters and another is with the Iterators.
In an iterative task I found that memory is being allocated with the
iterators and then with the sorters.
This PR is only for sorters.
the other one is for the iterators
https://github.com/apache/flink/pull/2510/commits/e17462881cc2f0137cc8412eb578fb4c42baeb15
@ggevay is sheperding it and helping me out with the idea of making
ResettableDrivers which I was not fully sure if it can be done.
I had added some docs to the JIRA itself about splitting up the tasks and
what is being addressed here and how it can be addressed.
`SorterMemoryAllocators` are nothing but just holders of the memory
segments that needs to be passed to the sorters. They hold the memory so that
the read buffers, write buffers and large memory buffers can pull in memory and
put back the memory to them at the end of each iteration. I initially
understood that changing this should have minimal impact on other areas which
are not impacted without which it is going to be difficult to review.
As said above it was just to show the intention of the changes and I am
very much aware that these change needs time to get a thorough review and any
design needs to be focussed on future enhancements as I had mentioned in the
doc. A PR will always help to understand better because the changes are in code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
