Vladimir Ozerov commented on IGNITE-7526:

Created a prototype with persistent database - was able to execute SQL query 

> SQL: Introduce memory region for reducer merge results with disk offload
> ------------------------------------------------------------------------
>                 Key: IGNITE-7526
>                 URL: https://issues.apache.org/jira/browse/IGNITE-7526
>             Project: Ignite
>          Issue Type: Task
>          Components: sql
>            Reporter: Vladimir Ozerov
>            Assignee: Taras Ledkov
>            Priority: Major
> Currently all results received from map nodes are stored inside reducer's 
> heap memory. What is worse, in case of complex queries, such as having sorts 
> or groupings, need to collect all results from mappers first before final 
> processing could be applied. In case of big results set (or intermediate 
> results) this could easily lead to OOME on reducer. 
> To mitigate this we should introduce special memory area where intermediate 
> results could be stored. All final processing should be stored in the same 
> area as well. This area should be of limited size and should be able to 
> offload results to disk in case of overflow.
> We could start with our B+Tree and free list and store results in some K-V 
> form. 

This message was sent by Atlassian JIRA

Reply via email to