Vladimir Ozerov created IGNITE-6019:

             Summary: SQL: client node should not hold the whole data set 
in-memory when possible
                 Key: IGNITE-6019
                 URL: https://issues.apache.org/jira/browse/IGNITE-6019
             Project: Ignite
          Issue Type: Bug
          Components: sql
    Affects Versions: 2.1
            Reporter: Vladimir Ozerov
            Assignee: Alexander Paschenko
             Fix For: 2.2

Our SQL engine requests request data from server nodes in pieces called "page". 
This allows us to control memory consumption on client side. However, currently 
our client code is designed in a way that all pages are requested from all 
servers before a single cursor row is returned to the user. It defeats the 
whole idea of "cursor" and "page", and could easily crash client node with 

We need to fix that and request further pages in a kind of sliding window, 
keeping no more than "N" pages in memory simultaneously. Note that sometimes it 
is not possible, e.g. in case of {{DISTINCT}} or non-collocated {{GROUP BY}}. 
In this case we would have to build the whole result set first anyway. So let's 
focus on a scenario when the whole result set is not needed.

As currently everything is requested synchronously page-by-page, in the first 
version it would be enough to distribute synchronous page requests between 
cursor reads, without any prefetch. 

Implementation details:
1) Optimization should be applied only to {{skipMergeTbl=true}} cases, when 
complete result set of map queries is not needed.
2) Starting point is {{GridReduceQueryExecutor#query}}, see 
{{skipMergeTbl=true}} branch - this is where we get all pages eagerly.
3) Get no more than one page from the server at a time. We request the page, 
iterate over it, then request another page.

This message was sent by Atlassian JIRA

Reply via email to