Vladimir Ozerov created IGNITE-7526:
---------------------------------------
Summary: SQL: Introduce memory region for reducer merge results
with disk offload
Key: IGNITE-7526
URL: https://issues.apache.org/jira/browse/IGNITE-7526
Project: Ignite
Issue Type: Task
Components: sql
Reporter: Vladimir Ozerov
Currently all results received from map nodes are stored inside reducer's heap
memory. What is worse, in case of complex queries, such as having sorts or
groupings, need to collect all results from mappers first before final
processing could be applied. In case of big results set (or intermediate
results) this could easily lead to OOME on reducer.
To mitigate this we should introduce special memory area where intermediate
results could be stored. All final processing should be stored in the same area
as well. This area should be of limited size and should be able to offload
results to disk in case of overflow.
We could start with our B+Tree and free list and store results in some K-V
form.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
