Ivan, I was involved in a couple of such use cases personally, so, that's not my imagination ;) Even more, as far as I remember, the primary reason why we improved our affinityRuns ensuring no partition is purged from a node until a task is completed is because many users were running local SQL from compute tasks and needed a guarantee that SQL will always return a correct result set.
- Denis On Fri, Nov 1, 2019 at 10:01 AM Ivan Pavlukhin <vololo...@gmail.com> wrote: > Denis, > > Would be nice to see real use-cases of affinity call + local SQL > combination. Generally, new engine will be able to infer collocation > resulting in the same collocated execution automatically. > > пт, 1 нояб. 2019 г. в 19:11, Denis Magda <dma...@apache.org>: > > > > Hi Igor, > > > > Local queries feature is broadly used together with affinity-based > compute > > tasks: > > > https://apacheignite.readme.io/docs/collocate-compute-and-data#section-affinity-call-and-run-methods > > > > The use case is as follows. The user knows that all required data needed > > for computation is collocated, and SQL is used as an advanced API for > data > > retrieval from the computation code. The affinity task ensures that > > partitions won't be discarded from the node(s) if the topology changes > > during the task execution and, thus, it's safe to run SQL locally > skipping > > distributed phases. > > > > The combination of affinity compute tasks with local SQL is a real and > > valuable use case, and this is what we need to support with Calcite. Do > you > > see any challenges? > > > > - > > Denis > > > > > > On Fri, Nov 1, 2019 at 8:46 AM Roman Kondakov <kondako...@mail.ru.invalid > > > > wrote: > > > > > Hi Igor! > > > > > > IMO we need to maintain the backward compatibility between old and new > > > query engines as much as possible. And therefore we shouldn't change > the > > > behavior of local queries. > > > > > > So, for local queries Calcite's planner shouldn't consider the > > > distribution trait at all. > > > > > > > > > -- > > > Kind Regards > > > Roman Kondakov > > > > > > On 01.11.2019 17:07, Seliverstov Igor wrote: > > > > Hi Igniters, > > > > > > > > Working on new generation of Ignite SQL I faced a question: «Do we > need > > > local queries at all and, if so, what semantic they should have?». > > > > > > > > Current planing flow consists of next steps: > > > > > > > > 1) Parsing SQL to AST > > > > 2) Validating AST (against Schema) > > > > 3) Optimizing (Building execution graph) > > > > 4) Splitting (into query fragments which executes on target nodes) > > > > 5) Mapping (query fragments to nodes/partitions) > > > > > > > > At last step we check that all Fragment sources (a table or result) > have > > > the same distribution (in other words all sources have to be > co-located) > > > > > > > > Planner and Splitter guarantee that all caches in a Fragment are > > > co-located, an Exchange is produced otherwise. But if we force local > > > execution we cannot produce Exchanges, that means we may face two > > > non-co-located caches inside a single query fragment (result of local > query > > > planning is a single query fragment). So, we cannot pass the check. > > > > > > > > Should we throw an exception or omit the check for local query > planning > > > or prohibit local queries at all? > > > > > > > > Your thoughts? > > > > > > > > Regards, > > > > Igor > > > > > > > -- > Best regards, > Ivan Pavlukhin >
