[
https://issues.apache.org/jira/browse/IGNITE-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17002806#comment-17002806
]
Igor Seliverstov commented on IGNITE-12449:
-------------------------------------------
[~ustas], this is a brand new query execution flow (designed for Calcite based
query engine) that uses a reactive approach and "push" semantics.
This is needed to execute a query, with possible cross dependencies, by limited
number of threads without blocking operations when all dependency conflicts are
solving automatically by operations reordering (if an operation needs some data
that unavailable at the moment, it stops executing until a data source signals
there is some new data available).
At the other hand "push" approach reduces request-reply roundtrips, that
reduces query execution latency.
Back pressure is a part of the protocol (if a consumer is not ready to process
a data producer stops data sending until a consumer signal) it allows us to use
small operation buffers that reduces possible OOM during execution - a reducer
doesn't collect whole dataset except several cases, like data sorting, that
effectively may be solved by an index introducing.
> Calcite integration. Execution flow.
> ------------------------------------
>
> Key: IGNITE-12449
> URL: https://issues.apache.org/jira/browse/IGNITE-12449
> Project: Ignite
> Issue Type: Task
> Reporter: Igor Seliverstov
> Assignee: Igor Seliverstov
> Priority: Major
>
> We need to introduce query execution environment.
> Execution should:
> * use streaming approach
> * have suspend/resume ability
> * work in event loop threads
> Rough protocol description:
> The flow is defined as a tree of operations, like filter, project etc.
> Each node provides a sink to consume data from its children and has a target
> sink to push data to upper node.
> Upper node may signal that it's ready to consume data. After a node received
> a signal it starts to push data into a target sink until the sink says there
> is no place for new data, after that the node stops pushing data until a new
> signal.
> Some of nodes (like inbox node, describing remote input) may signal that
> there is some new data, it forces a root node to signal its children top to
> bottom.
> When a signal arrived an inbox node, the inbox starts to push the new data.
> When a node realizes the data is over, it sends "end" signal to a target
> sink, after that an upper node wont signal the node to continue data pushing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
