[
https://issues.apache.org/jira/browse/IGNITE-7526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510929#comment-16510929
]
Vladimir Ozerov commented on IGNITE-7526:
-----------------------------------------
Looks like we can try to piggyback on H2 external storage as follows:
1) Make our H2 instance persistable and allow to set max memory rows; this way
H2 will store intermediate results on disk
2) Make our merge table use the same infrastructure to get rid of OOME on a
client
3) Create a patch for H2 to offload GROUP BY results to disk
> SQL: Introduce memory region for reducer merge results with disk offload
> ------------------------------------------------------------------------
>
> Key: IGNITE-7526
> URL: https://issues.apache.org/jira/browse/IGNITE-7526
> Project: Ignite
> Issue Type: Task
> Components: sql
> Reporter: Vladimir Ozerov
> Assignee: Taras Ledkov
> Priority: Major
>
> Currently all results received from map nodes are stored inside reducer's
> heap memory. What is worse, in case of complex queries, such as having sorts
> or groupings, need to collect all results from mappers first before final
> processing could be applied. In case of big results set (or intermediate
> results) this could easily lead to OOME on reducer.
> To mitigate this we should introduce special memory area where intermediate
> results could be stored. All final processing should be stored in the same
> area as well. This area should be of limited size and should be able to
> offload results to disk in case of overflow.
> We could start with our B+Tree and free list and store results in some K-V
> form.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
