[
https://issues.apache.org/jira/browse/IGNITE-7526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519424#comment-16519424
]
ASF GitHub Bot commented on IGNITE-7526:
----------------------------------------
GitHub user tledkov-gridgain opened a pull request:
https://github.com/apache/ignite/pull/4238
IGNITE-7526 SQL: Introduce memory region for reducer merge results with
disk offload
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gridgain/apache-ignite ignite-7526
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/ignite/pull/4238.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4238
----
commit 47232c571d1efcad9017286ba10d6ec362eae100
Author: devozerov <vozerov@...>
Date: 2018-06-13T13:01:12Z
It works!
commit 199b93202848a2e87bc4ba82f584ed51cca67b43
Author: tledkov-gridgain <tledkov@...>
Date: 2018-06-19T15:03:28Z
Merge branch '_master' into ignite-7526
commit 85dbca4d0f68202769aeae163f59bb2cd2698920
Author: tledkov-gridgain <tledkov@...>
Date: 2018-06-20T11:59:50Z
IGNITE-7526: minor
commit c02ba93abc953fe7c39b2d0b0d4a093c351504f5
Author: tledkov-gridgain <tledkov@...>
Date: 2018-06-21T14:07:44Z
IGNITE-7526: save the progress
commit 2d66389cc1a70cff0765cf3967651fe4994c280e
Author: tledkov-gridgain <tledkov@...>
Date: 2018-06-21T14:10:41Z
Merge branch '_master' into ignite-7526
----
> SQL: Introduce memory region for reducer merge results with disk offload
> ------------------------------------------------------------------------
>
> Key: IGNITE-7526
> URL: https://issues.apache.org/jira/browse/IGNITE-7526
> Project: Ignite
> Issue Type: Task
> Components: sql
> Reporter: Vladimir Ozerov
> Assignee: Taras Ledkov
> Priority: Major
>
> Currently all results received from map nodes are stored inside reducer's
> heap memory. What is worse, in case of complex queries, such as having sorts
> or groupings, need to collect all results from mappers first before final
> processing could be applied. In case of big results set (or intermediate
> results) this could easily lead to OOME on reducer.
> To mitigate this we should introduce special memory area where intermediate
> results could be stored. All final processing should be stored in the same
> area as well. This area should be of limited size and should be able to
> offload results to disk in case of overflow.
> We could start with our B+Tree and free list and store results in some K-V
> form.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
