GitHub user ramkrish86 opened a pull request:
https://github.com/apache/flink/pull/2495
FLINK-3322 - Make sorters to reuse the memory pages allocated for iterative
tasks
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
This is part1 for FLINK-3322 where only the Sorters are made to reuse the
memory pages. As @ggevay pointed out we have to handle the iterators also
where the memory pages are allocated. I have a seperate PR for that because
that involves touching lot of places. But am open to feedback here. It is fine
with me to combine both also but it was making the changes much bigger.
I would like to get the feed back here on this apporach.
Here a SorterMemoryAllocator is now passed to the UnilateralSortMergers.
That will allocate the required memory pages and it will allocate the required
read, write and large buffers. As per the existing logic the buffers will be
released. But if the task is an iterative task we wait for the tasks to be
released until a close or termination call happens for the iterative task.
In case of pages that were grabbed in between for keysort or record sort
those will be put back to the respective pages so that we have the required
number of pages through out the life cycle of the iterative task.
As said this is only part 1. We need to address the iterators also. But
that according to me touches more places. I have done the changes for that but
it is not in a shape to be pushed as a PR but am open to feed back here. Thanks
all.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ramkrish86/flink FLINK-3322_part1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2495.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2495
----
commit 705ee5294bc5263971c2924a55c9230d72806527
Author: Ramkrishna <[email protected]>
Date: 2016-09-13T06:33:59Z
FLINK-3322 - Make sorters to reuse the memory pages allocated for
iterative tasks
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
