[jira] [Created] (IGNITE-13710) Calcite integration. Fix or to union rule logic
Igor Seliverstov created IGNITE-13710: - Summary: Calcite integration. Fix or to union rule logic Key: IGNITE-13710 URL: https://issues.apache.org/jira/browse/IGNITE-13710 Project: Ignite Issue Type: Bug Components: sql Reporter: Igor Seliverstov -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (IGNITE-13541) Calcite integration. Support of Date/Time types
[ https://issues.apache.org/jira/browse/IGNITE-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov reassigned IGNITE-13541: - Assignee: Igor Seliverstov > Calcite integration. Support of Date/Time types > --- > > Key: IGNITE-13541 > URL: https://issues.apache.org/jira/browse/IGNITE-13541 > Project: Ignite > Issue Type: New Feature > Components: sql >Reporter: Yury Gerzhedovich >Assignee: Igor Seliverstov >Priority: Critical > Original Estimate: 40h > Remaining Estimate: 40h > > Seems currently the new Calcite engine doesn't support date/time types. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13579) Calcite integration. Wrong distribution assertion in case of nested request execution.
[
https://issues.apache.org/jira/browse/IGNITE-13579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215381#comment-17215381
]
Igor Seliverstov commented on IGNITE-13579:
---
[~zstan], looks OK, merged to the feature branch
> Calcite integration. Wrong distribution assertion in case of nested request
> execution.
> ---
>
> Key: IGNITE-13579
> URL: https://issues.apache.org/jira/browse/IGNITE-13579
> Project: Ignite
> Issue Type: Bug
> Components: sql
>Affects Versions: 2.8.1
>Reporter: Stanilovsky Evgeny
>Assignee: Stanilovsky Evgeny
>Priority: Major
> Labels: calcite
> Time Spent: 10m
> Remaining Estimate: 0h
>
> request :
> {noformat}
> SELECT NAME FROM products WHERE CAT_ID IN (SELECT CAT_ID FROM products WHERE
> CAT_ID > 1) and ID > 2;
> {noformat}
> brings assertion like:
> {noformat}
> java.lang.AssertionError: All produced nodes must have equal traits.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (IGNITE-13542) Calcite integration. Type system
[ https://issues.apache.org/jira/browse/IGNITE-13542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov reassigned IGNITE-13542: - Assignee: Igor Seliverstov > Calcite integration. Type system > > > Key: IGNITE-13542 > URL: https://issues.apache.org/jira/browse/IGNITE-13542 > Project: Ignite > Issue Type: New Feature > Components: sql >Reporter: Yury Gerzhedovich >Assignee: Igor Seliverstov >Priority: Critical > Original Estimate: 80h > Remaining Estimate: 80h > > System of types for the new Calcite engine . Just review, seems it may be > used as-is + write many tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13521) Calcite integration. New implementation of IgniteLogical[Table|Index]Scan.
[ https://issues.apache.org/jira/browse/IGNITE-13521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212436#comment-17212436 ] Igor Seliverstov commented on IGNITE-13521: --- [~zstan], merged to the feature branch > Calcite integration. New implementation of IgniteLogical[Table|Index]Scan. > -- > > Key: IGNITE-13521 > URL: https://issues.apache.org/jira/browse/IGNITE-13521 > Project: Ignite > Issue Type: Improvement > Components: sql >Affects Versions: 2.8.1 >Reporter: Stanilovsky Evgeny >Assignee: Stanilovsky Evgeny >Priority: Major > Labels: calcite > > After [1] (ProjectionNode) was implemented, we need additional logical scans > with Convention#NONE convention for optimizations requests with predicates. > Details: > 1. Implements IgniteLogicalTableScan and IgniteLogicalIndexScan with > inheritance from ProjectableFilterableTableScan > 2. Move and change ValuesConverterRule form HEURISTIC_OPTIMIZATION into > OPTIMIZATION > 3. ExposeIndexRule refactoring. > 4. New converter IgniteLogical[Table|Index]Scan -> Ignite[Table|Index]Scan > [1] https://issues.apache.org/jira/browse/IGNITE-13476 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-13547) Calcite integration. CREATE TABLE support
[ https://issues.apache.org/jira/browse/IGNITE-13547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-13547: -- Remaining Estimate: 168h Original Estimate: 168h > Calcite integration. CREATE TABLE support > - > > Key: IGNITE-13547 > URL: https://issues.apache.org/jira/browse/IGNITE-13547 > Project: Ignite > Issue Type: New Feature > Components: sql >Reporter: Yury Gerzhedovich >Priority: Critical > Original Estimate: 168h > Remaining Estimate: 168h > > We need to support DDL commands. The task about the support of CREATE TABLE. > Potentially with syntaxis as we already have in the H2 engine. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-13548) Calcite integration. DROP TABLE support
[ https://issues.apache.org/jira/browse/IGNITE-13548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-13548: -- Remaining Estimate: 168h Original Estimate: 168h > Calcite integration. DROP TABLE support > --- > > Key: IGNITE-13548 > URL: https://issues.apache.org/jira/browse/IGNITE-13548 > Project: Ignite > Issue Type: New Feature > Components: sql >Reporter: Yury Gerzhedovich >Priority: Critical > Original Estimate: 168h > Remaining Estimate: 168h > > We need to support DDL commands. The task about the support of DROP TABLE. > Potentially with syntaxis as we already have in the H2 engine. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-13544) Calcite integration. Merge joins
[ https://issues.apache.org/jira/browse/IGNITE-13544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-13544: -- Remaining Estimate: 336h Original Estimate: 336h > Calcite integration. Merge joins > > > Key: IGNITE-13544 > URL: https://issues.apache.org/jira/browse/IGNITE-13544 > Project: Ignite > Issue Type: New Feature > Components: sql >Reporter: Yury Gerzhedovich >Priority: Critical > Original Estimate: 336h > Remaining Estimate: 336h > > To have a performance boost need to implement merge joins. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-13545) Calcite integration. Index spoon
[ https://issues.apache.org/jira/browse/IGNITE-13545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-13545: -- Remaining Estimate: 336h Original Estimate: 336h > Calcite integration. Index spoon > > > Key: IGNITE-13545 > URL: https://issues.apache.org/jira/browse/IGNITE-13545 > Project: Ignite > Issue Type: New Feature > Components: sql >Reporter: Yury Gerzhedovich >Priority: Major > Original Estimate: 336h > Remaining Estimate: 336h > > To have a performance boost need to implement index spoons. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-13546) Calcite integration. Hash index spoon
[ https://issues.apache.org/jira/browse/IGNITE-13546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-13546: -- Remaining Estimate: 336h Original Estimate: 336h > Calcite integration. Hash index spoon > - > > Key: IGNITE-13546 > URL: https://issues.apache.org/jira/browse/IGNITE-13546 > Project: Ignite > Issue Type: New Feature > Components: sql >Reporter: Yury Gerzhedovich >Priority: Critical > Original Estimate: 336h > Remaining Estimate: 336h > > To have a performance boost need to implement hash index spoons. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-13543) Calcite integration. Support of merge aggregates
[ https://issues.apache.org/jira/browse/IGNITE-13543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-13543: -- Remaining Estimate: 336h Original Estimate: 336h > Calcite integration. Support of merge aggregates > > > Key: IGNITE-13543 > URL: https://issues.apache.org/jira/browse/IGNITE-13543 > Project: Ignite > Issue Type: New Feature > Components: sql >Reporter: Yury Gerzhedovich >Priority: Critical > Original Estimate: 336h > Remaining Estimate: 336h > > To have a performance boost need to implement merge aggregates. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13541) Calcite integration. Support of Date/Time types
[ https://issues.apache.org/jira/browse/IGNITE-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210204#comment-17210204 ] Igor Seliverstov commented on IGNITE-13541: --- Seems we need only small client side converter to get full dates support, but in addition we need to investigate possibility to pass user context (like a timezone) to get proper date/time processing on server nodes, that may have different time zone settings > Calcite integration. Support of Date/Time types > --- > > Key: IGNITE-13541 > URL: https://issues.apache.org/jira/browse/IGNITE-13541 > Project: Ignite > Issue Type: New Feature > Components: sql >Reporter: Yury Gerzhedovich >Priority: Critical > Original Estimate: 168h > Remaining Estimate: 168h > > Seems currently the new Calcite engine doesn't support date/time types. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-13542) Calcite integration. Type system
[ https://issues.apache.org/jira/browse/IGNITE-13542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-13542: -- Remaining Estimate: 336h Original Estimate: 336h > Calcite integration. Type system > > > Key: IGNITE-13542 > URL: https://issues.apache.org/jira/browse/IGNITE-13542 > Project: Ignite > Issue Type: New Feature > Components: sql >Reporter: Yury Gerzhedovich >Priority: Critical > Original Estimate: 336h > Remaining Estimate: 336h > > System of types for the new Calcite engine . Just review, seems it may be > used as-is + write many tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-13541) Calcite integration. Support of Date/Time types
[ https://issues.apache.org/jira/browse/IGNITE-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-13541: -- Remaining Estimate: 168h Original Estimate: 168h > Calcite integration. Support of Date/Time types > --- > > Key: IGNITE-13541 > URL: https://issues.apache.org/jira/browse/IGNITE-13541 > Project: Ignite > Issue Type: New Feature > Components: sql >Reporter: Yury Gerzhedovich >Priority: Critical > Original Estimate: 168h > Remaining Estimate: 168h > > Seems currently the new Calcite engine doesn't support date/time types. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-13515) Performance drop when there are many thin clients per server
[
https://issues.apache.org/jira/browse/IGNITE-13515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Seliverstov updated IGNITE-13515:
--
Docs Text: New configuration property added:
ClientConnectorConfiguration#selectorCount
Ignite Flags: Docs Required,Release Notes Required
> Performance drop when there are many thin clients per server
>
>
> Key: IGNITE-13515
> URL: https://issues.apache.org/jira/browse/IGNITE-13515
> Project: Ignite
> Issue Type: Task
> Components: clients
>Reporter: Igor Seliverstov
>Assignee: Igor Seliverstov
>Priority: Major
> Fix For: 2.10
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> There is a performance drop when running many thin clients per node:
> {noformat}
> 4 nodes - 1 thin client java - ~5500 pps (put per second)
> 4 nodes - 2 thin clients java - ~5000 pps
> 4 nodes - 4 thin clients java - ~4000 pps
> 4 nodes - 10 thin clients java - ~1500 pps
> 4 nodes - ~100 thin clients java - ~100 pps
> {noformat}
> May be related to sync request processing in connection thread pool on server
> side. Need to investigate and fix if possible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-13515) Performance drop when there are many thin clients per server
[
https://issues.apache.org/jira/browse/IGNITE-13515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17206407#comment-17206407
]
Igor Seliverstov commented on IGNITE-13515:
---
[~ptupitsyn], please look at
> Performance drop when there are many thin clients per server
>
>
> Key: IGNITE-13515
> URL: https://issues.apache.org/jira/browse/IGNITE-13515
> Project: Ignite
> Issue Type: Task
> Components: clients
>Reporter: Igor Seliverstov
>Assignee: Igor Seliverstov
>Priority: Major
> Fix For: 2.10
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There is a performance drop when running many thin clients per node:
> {noformat}
> 4 nodes - 1 thin client java - ~5500 pps (put per second)
> 4 nodes - 2 thin clients java - ~5000 pps
> 4 nodes - 4 thin clients java - ~4000 pps
> 4 nodes - 10 thin clients java - ~1500 pps
> 4 nodes - ~100 thin clients java - ~100 pps
> {noformat}
> May be related to sync request processing in connection thread pool on server
> side. Need to investigate and fix if possible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (IGNITE-13515) Performance drop when there are many thin clients per server
[
https://issues.apache.org/jira/browse/IGNITE-13515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Seliverstov updated IGNITE-13515:
--
Fix Version/s: 2.9
> Performance drop when there are many thin clients per server
>
>
> Key: IGNITE-13515
> URL: https://issues.apache.org/jira/browse/IGNITE-13515
> Project: Ignite
> Issue Type: Task
> Components: clients
>Reporter: Igor Seliverstov
>Assignee: Igor Seliverstov
>Priority: Major
> Fix For: 2.9
>
>
> There is a performance drop when running many thin clients per node:
> {noformat}
> 4 nodes - 1 thin client java - ~5500 pps (put per second)
> 4 nodes - 2 thin clients java - ~5000 pps
> 4 nodes - 4 thin clients java - ~4000 pps
> 4 nodes - 10 thin clients java - ~1500 pps
> 4 nodes - ~100 thin clients java - ~100 pps
> {noformat}
> May be related to sync request processing in connection thread pool on server
> side. Need to investigate and fix if possible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-13515) Performance drop when there are many thin clients per server
Igor Seliverstov created IGNITE-13515:
-
Summary: Performance drop when there are many thin clients per
server
Key: IGNITE-13515
URL: https://issues.apache.org/jira/browse/IGNITE-13515
Project: Ignite
Issue Type: Task
Components: clients
Reporter: Igor Seliverstov
Assignee: Igor Seliverstov
There is a performance drop when running many thin clients per node:
{noformat}
4 nodes - 1 thin client java - ~5500 pps (put per second)
4 nodes - 2 thin clients java - ~5000 pps
4 nodes - 4 thin clients java - ~4000 pps
4 nodes - 10 thin clients java - ~1500 pps
4 nodes - ~100 thin clients java - ~100 pps
{noformat}
May be related to sync request processing in connection thread pool on server
side. Need to investigate and fix if possible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (IGNITE-12633) Calcite integration. Query cancel purpose.
[ https://issues.apache.org/jira/browse/IGNITE-12633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov resolved IGNITE-12633. --- Resolution: Implemented Done in scope of IGNITE-13198 > Calcite integration. Query cancel purpose. > -- > > Key: IGNITE-12633 > URL: https://issues.apache.org/jira/browse/IGNITE-12633 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Igor Seliverstov >Assignee: Ravil Galeyev >Priority: Minor > > Currently on a query participant node left whole query is cancelled, but the > end user doesn't know the purpose. We should pass causing cancel exception to > end user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13463) Calcite improvements. Rework RootNode rows exchange.
[ https://issues.apache.org/jira/browse/IGNITE-13463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199966#comment-17199966 ] Igor Seliverstov commented on IGNITE-13463: --- [~zstan], LGTM, merged to the feature branch. > Calcite improvements. Rework RootNode rows exchange. > > > Key: IGNITE-13463 > URL: https://issues.apache.org/jira/browse/IGNITE-13463 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8.1 > Environment: In current implementation calcite.exec.rel.RootNode > generates huge number of context switches, looks like we have a field for > optimizations here. Need additional investigations and benchmarks. >Reporter: Stanilovsky Evgeny >Assignee: Stanilovsky Evgeny >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13461) Calcite integration. Wrong behavior on thick client side planning.
Igor Seliverstov created IGNITE-13461: - Summary: Calcite integration. Wrong behavior on thick client side planning. Key: IGNITE-13461 URL: https://issues.apache.org/jira/browse/IGNITE-13461 Project: Ignite Issue Type: Bug Components: sql Reporter: Igor Seliverstov Assignee: Igor Seliverstov Query planner tries to use index physical structures at a client side, also it tries to access affinity manager, that isn't started on the client. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (IGNITE-13198) Calcite integration. Rework error / cancel logic at execution
[ https://issues.apache.org/jira/browse/IGNITE-13198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184684#comment-17184684 ] Igor Seliverstov edited comment on IGNITE-13198 at 8/25/20, 7:01 PM: - Reworked in a next way: Because a node having intermediate fragment may leave, there will be nobody who closes upstream fragment execution. Now a root fragment node is responsible for closing(cancelling) all query fragments. It seems overcomplicated to pass an error through all query fragments, so, an error is sent right to a root fragment node (after the error reaches an Outbox node inside a local execution fragment). Node left events are processed by Inbox nodes, it allows simpler failover handling and an ability to successfully finish a query in case it got all needed data from a failed node. On node left an error will be passed to a user and query will be automatically closed. Root node left event is processed by Outbox node, this way a query will be automatically closed. was (Author: gvvinblade): Reworked in a next way: Because a node, having intermediate fragment, may leave there will be nobody who closes upstream fragment execution. Now a root fragment node is responsible for closing(cancelling) all query fragments. It seems overcomplicated to pass an error through all query fragments, so, an error is sent right to a root fragment node (after the error reaches an Outbox node inside a local execution fragment). Node left events are processed by Inbox nodes, it allows simpler failover handling and an ability to successfully finish a query in case it got all needed data from a failed node. On node left an error will be passed to a user and query will be automatically closed. Root node left event is processed by Outbox node, this way a query will be automatically closed. > Calcite integration. Rework error / cancel logic at execution > - > > Key: IGNITE-13198 > URL: https://issues.apache.org/jira/browse/IGNITE-13198 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Taras Ledkov >Assignee: Igor Seliverstov >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Change error & cancel logic on query execution. > *Error* > - propagated from source to downstream nodes. > - Always propagated to the RootNode; > - RootNode receive an error and cancel all execution tree. > *Cancel* > - propagated from downstream to source; > - any node may cancel its execution sub-tree; > - (technical details) to prevent race on Inbox creation Outbox resend Cancel > message to Inbox. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-13198) Calcite integration. Rework error / cancel logic at execution
[ https://issues.apache.org/jira/browse/IGNITE-13198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-13198: -- Ignite Flags: (was: Docs Required,Release Notes Required) > Calcite integration. Rework error / cancel logic at execution > - > > Key: IGNITE-13198 > URL: https://issues.apache.org/jira/browse/IGNITE-13198 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Taras Ledkov >Assignee: Igor Seliverstov >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Change error & cancel logic on query execution. > *Error* > - propagated from source to downstream nodes. > - Always propagated to the RootNode; > - RootNode receive an error and cancel all execution tree. > *Cancel* > - propagated from downstream to source; > - any node may cancel its execution sub-tree; > - (technical details) to prevent race on Inbox creation Outbox resend Cancel > message to Inbox. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13198) Calcite integration. Rework error / cancel logic at execution
[ https://issues.apache.org/jira/browse/IGNITE-13198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184684#comment-17184684 ] Igor Seliverstov commented on IGNITE-13198: --- Reworked in a next way: Because a node, having intermediate fragment, may leave there will be nobody who closes upstream fragment execution. Now a root fragment node is responsible for closing(cancelling) all query fragments. It seems overcomplicated to pass an error through all query fragments, so, an error is sent right to a root fragment node (after the error reaches an Outbox node inside a local execution fragment). Node left events are processed by Inbox nodes, it allows simpler failover handling and an ability to successfully finish a query in case it got all needed data from a failed node. On node left an error will be passed to a user and query will be automatically closed. Root node left event is processed by Outbox node, this way a query will be automatically closed. > Calcite integration. Rework error / cancel logic at execution > - > > Key: IGNITE-13198 > URL: https://issues.apache.org/jira/browse/IGNITE-13198 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Taras Ledkov >Assignee: Igor Seliverstov >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Change error & cancel logic on query execution. > *Error* > - propagated from source to downstream nodes. > - Always propagated to the RootNode; > - RootNode receive an error and cancel all execution tree. > *Cancel* > - propagated from downstream to source; > - any node may cancel its execution sub-tree; > - (technical details) to prevent race on Inbox creation Outbox resend Cancel > message to Inbox. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (IGNITE-12350) MVCC activated and causing memory leak (OOM) despite no mvccEnabled caches
[ https://issues.apache.org/jira/browse/IGNITE-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17178396#comment-17178396 ] Igor Seliverstov edited comment on IGNITE-12350 at 8/15/20, 10:30 PM: -- [~v.pyatkov], LGTM, merged to master was (Author: gvvinblade): [~v.pyatkov]LGTM, merged to master > MVCC activated and causing memory leak (OOM) despite no mvccEnabled caches > -- > > Key: IGNITE-12350 > URL: https://issues.apache.org/jira/browse/IGNITE-12350 > Project: Ignite > Issue Type: Bug > Components: mvcc >Affects Versions: 2.7.5 >Reporter: Ming Vuong >Assignee: Vladislav Pyatkov >Priority: Major > Fix For: 2.9 > > Attachments: MATheap.jpg, proxy-memory-leak.png, proxy-oom-2.png > > Time Spent: 20m > Remaining Estimate: 0h > > We have a critical memory leak where mvccCoordinator is still selected and > mvcc is still 'activated' despite having no mvccEnabled caches, this is > causing constant OOM crashes and issues on both server and client nodes. > All client CacheAtomicityModes are of type ATOMIC, and for server config > persistence is false, TCPDiscoverySpi used and PeerClassLoadingEnabled=true. > There are two server JVM instances on two separate nodes. One of the servers > consistently get an OOM error, as well as all clients, due to the same > recoveryBallotBoxes HashMap constantly increasing in size and unable to be > Garbage Collected. > Attached is Eclipse Memory Analyzer Tool screenshot on the heap dump close to > one of the server JVM OOM crashes. The same heap analysis result is apparent > for all clients as well. > !MATheap.jpg! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12350) MVCC activated and causing memory leak (OOM) despite no mvccEnabled caches
[ https://issues.apache.org/jira/browse/IGNITE-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-12350: -- Component/s: mvcc > MVCC activated and causing memory leak (OOM) despite no mvccEnabled caches > -- > > Key: IGNITE-12350 > URL: https://issues.apache.org/jira/browse/IGNITE-12350 > Project: Ignite > Issue Type: Bug > Components: mvcc >Affects Versions: 2.7.5 >Reporter: Ming Vuong >Assignee: Vladislav Pyatkov >Priority: Major > Fix For: 2.9 > > Attachments: MATheap.jpg, proxy-memory-leak.png, proxy-oom-2.png > > Time Spent: 20m > Remaining Estimate: 0h > > We have a critical memory leak where mvccCoordinator is still selected and > mvcc is still 'activated' despite having no mvccEnabled caches, this is > causing constant OOM crashes and issues on both server and client nodes. > All client CacheAtomicityModes are of type ATOMIC, and for server config > persistence is false, TCPDiscoverySpi used and PeerClassLoadingEnabled=true. > There are two server JVM instances on two separate nodes. One of the servers > consistently get an OOM error, as well as all clients, due to the same > recoveryBallotBoxes HashMap constantly increasing in size and unable to be > Garbage Collected. > Attached is Eclipse Memory Analyzer Tool screenshot on the heap dump close to > one of the server JVM OOM crashes. The same heap analysis result is apparent > for all clients as well. > !MATheap.jpg! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12350) MVCC activated and causing memory leak (OOM) despite no mvccEnabled caches
[ https://issues.apache.org/jira/browse/IGNITE-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-12350: -- Fix Version/s: 2.9 > MVCC activated and causing memory leak (OOM) despite no mvccEnabled caches > -- > > Key: IGNITE-12350 > URL: https://issues.apache.org/jira/browse/IGNITE-12350 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7.5 >Reporter: Ming Vuong >Assignee: Vladislav Pyatkov >Priority: Major > Fix For: 2.9 > > Attachments: MATheap.jpg, proxy-memory-leak.png, proxy-oom-2.png > > Time Spent: 20m > Remaining Estimate: 0h > > We have a critical memory leak where mvccCoordinator is still selected and > mvcc is still 'activated' despite having no mvccEnabled caches, this is > causing constant OOM crashes and issues on both server and client nodes. > All client CacheAtomicityModes are of type ATOMIC, and for server config > persistence is false, TCPDiscoverySpi used and PeerClassLoadingEnabled=true. > There are two server JVM instances on two separate nodes. One of the servers > consistently get an OOM error, as well as all clients, due to the same > recoveryBallotBoxes HashMap constantly increasing in size and unable to be > Garbage Collected. > Attached is Eclipse Memory Analyzer Tool screenshot on the heap dump close to > one of the server JVM OOM crashes. The same heap analysis result is apparent > for all clients as well. > !MATheap.jpg! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-12350) MVCC activated and causing memory leak (OOM) despite no mvccEnabled caches
[ https://issues.apache.org/jira/browse/IGNITE-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17178396#comment-17178396 ] Igor Seliverstov commented on IGNITE-12350: --- [~v.pyatkov]LGTM, merged to master > MVCC activated and causing memory leak (OOM) despite no mvccEnabled caches > -- > > Key: IGNITE-12350 > URL: https://issues.apache.org/jira/browse/IGNITE-12350 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7.5 >Reporter: Ming Vuong >Assignee: Vladislav Pyatkov >Priority: Major > Attachments: MATheap.jpg, proxy-memory-leak.png, proxy-oom-2.png > > Time Spent: 20m > Remaining Estimate: 0h > > We have a critical memory leak where mvccCoordinator is still selected and > mvcc is still 'activated' despite having no mvccEnabled caches, this is > causing constant OOM crashes and issues on both server and client nodes. > All client CacheAtomicityModes are of type ATOMIC, and for server config > persistence is false, TCPDiscoverySpi used and PeerClassLoadingEnabled=true. > There are two server JVM instances on two separate nodes. One of the servers > consistently get an OOM error, as well as all clients, due to the same > recoveryBallotBoxes HashMap constantly increasing in size and unable to be > Garbage Collected. > Attached is Eclipse Memory Analyzer Tool screenshot on the heap dump close to > one of the server JVM OOM crashes. The same heap analysis result is apparent > for all clients as well. > !MATheap.jpg! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-5038) BinaryMarshaller might need to use context class loader for deserialization
[ https://issues.apache.org/jira/browse/IGNITE-5038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177674#comment-17177674 ] Igor Seliverstov commented on IGNITE-5038: -- [~ivan.glukos]I went across the PR, generally I'm OK with changes. > BinaryMarshaller might need to use context class loader for deserialization > --- > > Key: IGNITE-5038 > URL: https://issues.apache.org/jira/browse/IGNITE-5038 > Project: Ignite > Issue Type: Improvement > Components: binary >Affects Versions: 2.0 >Reporter: Dmitry Karachentsev >Assignee: Ivan Rakov >Priority: Major > Labels: features > Attachments: results-compound-20170802.zip, > results-compound-20170808.zip > > Time Spent: 1h 40m > Remaining Estimate: 0h > > There is a special use case discussed on the dev list: > http://apache-ignite-developers.2346864.n4.nabble.com/Re-BinaryObjectImpl-deserializeValue-with-specific-ClassLoader-td17126.html#a17224 > According to the use case, BinaryMarshaller might need to try to deserialize > an object using a context class loader if it failed to do so with a custom > classloader (`IgniteConfiguration.getClassLoader()`) or the system one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-11942) IGFS and Hadoop Accelerator Discontinuation
[ https://issues.apache.org/jira/browse/IGNITE-11942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164607#comment-17164607 ] Igor Seliverstov commented on IGNITE-11942: --- [~akalashnikov], Java part looks OK, quickly looked at js part, seems it's OK too, if [~kuaw26] and [~vsisko] don't mind, proceed with merging. > IGFS and Hadoop Accelerator Discontinuation > --- > > Key: IGNITE-11942 > URL: https://issues.apache.org/jira/browse/IGNITE-11942 > Project: Ignite > Issue Type: Task >Reporter: Denis A. Magda >Assignee: Anton Kalashnikov >Priority: Blocker > Fix For: 2.9 > > Time Spent: 20m > Remaining Estimate: 0h > > The community has voted for the following decision: > * IGFS and In-Memory Hadoop Accelerator components are to be discontinued and > no longer supported by the community > * The existing source code of IGFS and In-Memory Hadoop Accelerator is to be > removed from Ignite master. Before that, a special branch like > "ignite-igfs-and-hadoop-accelerator" to be forked off the master in order to > preserve the sources in Git history for those who might need it. > The voting thread: > http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42405.html > Once the changes are made for Ignite 2.8, please contact Denis Magda to > update a public documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (IGNITE-4003) Slow or faulty client can stall the whole cluster.
[
https://issues.apache.org/jira/browse/IGNITE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Seliverstov reassigned IGNITE-4003:
Assignee: Igor Seliverstov
> Slow or faulty client can stall the whole cluster.
> --
>
> Key: IGNITE-4003
> URL: https://issues.apache.org/jira/browse/IGNITE-4003
> Project: Ignite
> Issue Type: Bug
> Components: cache, general
>Affects Versions: 1.7
>Reporter: Vladimir Ozerov
>Assignee: Igor Seliverstov
>Priority: Critical
>
> Steps to reproduce:
> 1) Start two server nodes and some data to cache.
> 2) Start a client from Docker subnet, which is not visible from the outside.
> Client will join the cluster.
> 3) Try to put something to cache or start another node to force rabalance.
> Cluster is stuck at this moment. Root cause - servers are constantly trying
> to establish outgoing connection to the client, but fail as Docker subnet is
> not visible from the outside. It may stop virtually all cluster operations.
> Typical thread dump:
> {code}
> org.apache.ignite.IgniteCheckedException: Failed to send message (node may
> have left the grid or TCP connection cannot be established due to firewall
> issues) [node=TcpDiscoveryNode [id=a15d74c2-1ec2-4349-9640-aeacd70d8714,
> addrs=[127.0.0.1, 172.17.0.6], sockAddrs=[/127.0.0.1:0, /127.0.0.1:0,
> /172.17.0.6:0], discPort=0, order=7241, intOrder=3707,
> lastExchangeTime=1474096941045, loc=false, ver=1.5.23#20160526-sha1:259146da,
> isClient=true], topic=T4 [topic=TOPIC_CACHE,
> id1=949732fd-1360-3a58-8d9e-0ff6ea6182cc,
> id2=a15d74c2-1ec2-4349-9640-aeacd70d8714, id3=2], msg=GridContinuousMessage
> [type=MSG_EVT_NOTIFICATION, routineId=7e13c48e-6933-48b2-9f15-8d92007930db,
> data=null, futId=null], policy=2]
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1129)
> [ignite-core-1.5.23.jar:1.5.23]
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1347)
> [ignite-core-1.5.23.jar:1.5.23]
> at
> org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1227)
> ~[ignite-core-1.5.23.jar:1.5.23]
> at
> org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1198)
> ~[ignite-core-1.5.23.jar:1.5.23]
> at
> org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1180)
> ~[ignite-core-1.5.23.jar:1.5.23]
> at
> org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:841)
> ~[ignite-core-1.5.23.jar:1.5.23]
> at
> org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:800)
> ~[ignite-core-1.5.23.jar:1.5.23]
> at
> org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:787)
> [ignite-core-1.5.23.jar:1.5.23]
> at
> org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$700(CacheContinuousQueryHandler.java:91)
> [ignite-core-1.5.23.jar:1.5.23]
> at
> org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$1.onEntryUpdated(CacheContinuousQueryHandler.java:412)
> [ignite-core-1.5.23.jar:1.5.23]
> at
> org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:343)
> [ignite-core-1.5.23.jar:1.5.23]
> at
> org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:250)
> [ignite-core-1.5.23.jar:1.5.23]
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.initialValue(GridCacheMapEntry.java:3476)
> [ignite-core-1.5.23.jar:1.5.23]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtForceKeysFuture$MiniFuture.onResult(GridDhtForceKeysFuture.java:548)
> [ignite-core-1.5.23.jar:1.5.23]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtForceKeysFuture.onResult(GridDhtForceKeysFuture.java:207)
> [ignite-core-1.5.23.jar:1.5.23]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader.processForceKeyResponse(GridDhtPreloader.java:636)
> [ignite-core-1.5.23.jar:1.5.23]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader.access$1000(GridDhtPreloader.java:81)
> [ignite-core-1.5.23.jar:1.5.23]
> at
>
[jira] [Created] (IGNITE-13247) Calcite integration. Move QueryCursorEx.isQuery() flag to a proper place.
Igor Seliverstov created IGNITE-13247: - Summary: Calcite integration. Move QueryCursorEx.isQuery() flag to a proper place. Key: IGNITE-13247 URL: https://issues.apache.org/jira/browse/IGNITE-13247 Project: Ignite Issue Type: Improvement Reporter: Igor Seliverstov Since QueryCursorEx is used in continuous queries, we should move SQL query related flag to another place. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-12620) Calcite integration. Index Nested Loop Join/Hash Join
[ https://issues.apache.org/jira/browse/IGNITE-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148795#comment-17148795 ] Igor Seliverstov commented on IGNITE-12620: --- [~agoncharuk],[~amashenkov],[~korlov],[~tledkov-gridgain], guys, please look at > Calcite integration. Index Nested Loop Join/Hash Join > - > > Key: IGNITE-12620 > URL: https://issues.apache.org/jira/browse/IGNITE-12620 > Project: Ignite > Issue Type: New Feature >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > We may implement the feature the next way: > # For each row from left node consume whole dataset from right node > # Pass join condition as an argument of request() method of the right node > # In case a data source at right is an index scan > ## If there is no cursor opened - create a new cursor using bounds from > request > ## If there is an existing cursor - just check the cursor was opened using > the same condition as passed one. > ## After the right node signals EOD consume a next row from left and repeat > from p 2. > # In case a data source at right is a table scan with huge amount of rows > ## If there is no cursor opened - create a new cursor ignoring passed > condition > ## If there is an existing cursor - just check the cursor was opened using > the same condition as passed one. > ## After the right node signals EOD consume a next row from left and repeat > from p 2. > # In case a data source at right is remote / is a table scan with small > number of rows/ is a filter node with good enough selectivity > ## Create a hash index for a data source > ## If there is no cursor opened - create a new cursor using bounds from > request > ## If there is an existing cursor - just check the cursor was opened using > the same condition as passed one. > ## After the right node signals EOD consume a next row from left and repeat > from p 2. >  Consider implementation specifics at optimization time, choose the cheapest > variant -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-13117) Calcite integration. Update Calcite version to 1.23.0
[ https://issues.apache.org/jira/browse/IGNITE-13117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-13117: -- Description: There are two important features in the new Calcite release that we need to solve current planner issues: top-down traits propagation and bottom-up traits derivation (was: There are two important features in the new Calcite release what we need to solve current planner issues: top-down traits propagation and bottom-up traits derivation) > Calcite integration. Update Calcite version to 1.23.0 > - > > Key: IGNITE-13117 > URL: https://issues.apache.org/jira/browse/IGNITE-13117 > Project: Ignite > Issue Type: Improvement >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > There are two important features in the new Calcite release that we need to > solve current planner issues: top-down traits propagation and bottom-up > traits derivation -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13117) Calcite integration. Update Calcite version to 1.23.0
Igor Seliverstov created IGNITE-13117: - Summary: Calcite integration. Update Calcite version to 1.23.0 Key: IGNITE-13117 URL: https://issues.apache.org/jira/browse/IGNITE-13117 Project: Ignite Issue Type: Improvement Reporter: Igor Seliverstov Assignee: Igor Seliverstov There are two important features in the new Calcite release what we need to solve current planner issues: top-down traits propagation and bottom-up traits derivation -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13051) Optimized affinity switch on baseline node left is broken for client topology and MVCC
[ https://issues.apache.org/jira/browse/IGNITE-13051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119368#comment-17119368 ] Igor Seliverstov commented on IGNITE-13051: --- [~timonin.maksim], Looks good to me, at least I see no issues. > Optimized affinity switch on baseline node left is broken for client topology > and MVCC > -- > > Key: IGNITE-13051 > URL: https://issues.apache.org/jira/browse/IGNITE-13051 > Project: Ignite > Issue Type: Bug >Reporter: Alexey Scherbakov >Assignee: Maksim Timonin >Priority: Critical > Fix For: 2.9 > > Time Spent: 10m > Remaining Estimate: 0h > > If a node contains only client cache topology with MVCC enabled, PME will > hang after changes introduced in IGNITE-12617. > Reproduced by > CachePartitionLossWithRestartsTest.testPartitionLossDetectionOnClientTopology[0 > false false -1] and enabled MVCC. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (IGNITE-12620) Calcite integration. Index Nested Loop Join/Hash Join
[ https://issues.apache.org/jira/browse/IGNITE-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov reassigned IGNITE-12620: - Assignee: Igor Seliverstov > Calcite integration. Index Nested Loop Join/Hash Join > - > > Key: IGNITE-12620 > URL: https://issues.apache.org/jira/browse/IGNITE-12620 > Project: Ignite > Issue Type: New Feature >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > > We may implement the feature the next way: > # For each row from left node consume whole dataset from right node > # Pass join condition as an argument of request() method of the right node > # In case a data source at right is an index scan > ## If there is no cursor opened - create a new cursor using bounds from > request > ## If there is an existing cursor - just check the cursor was opened using > the same condition as passed one. > ## After the right node signals EOD consume a next row from left and repeat > from p 2. > # In case a data source at right is a table scan with huge amount of rows > ## If there is no cursor opened - create a new cursor ignoring passed > condition > ## If there is an existing cursor - just check the cursor was opened using > the same condition as passed one. > ## After the right node signals EOD consume a next row from left and repeat > from p 2. > # In case a data source at right is remote / is a table scan with small > number of rows/ is a filter node with good enough selectivity > ## Create a hash index for a data source > ## If there is no cursor opened - create a new cursor using bounds from > request > ## If there is an existing cursor - just check the cursor was opened using > the same condition as passed one. > ## After the right node signals EOD consume a next row from left and repeat > from p 2. >  Consider implementation specifics at optimization time, choose the cheapest > variant -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (IGNITE-12900) Calcite integration. Use RowHandler to access fields.
[ https://issues.apache.org/jira/browse/IGNITE-12900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov resolved IGNITE-12900. --- Release Note: Done in scope of IGNITE-12715 Resolution: Done > Calcite integration. Use RowHandler to access fields. > - > > Key: IGNITE-12900 > URL: https://issues.apache.org/jira/browse/IGNITE-12900 > Project: Ignite > Issue Type: Task >Reporter: Igor Seliverstov >Priority: Major > > Currently only Object[] can be used as a row because most of execution nodes > require Object[] as a row type. Let's use generic row types with appropriate > RowHandler in execution nodes and compiled expressions to get more > flexibility. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12991) Calcite integration. Pass cancel flag to VolcanoPlanner
Igor Seliverstov created IGNITE-12991:
-
Summary: Calcite integration. Pass cancel flag to VolcanoPlanner
Key: IGNITE-12991
URL: https://issues.apache.org/jira/browse/IGNITE-12991
Project: Ignite
Issue Type: Improvement
Reporter: Igor Seliverstov
see {{AbstractRelOptPlanner.java:91}}, here {{CancelFlag}}Â is used to cancel
planning loop. We need to pass it into a newly created context and bind its
state with {{PlanningContext#queryCancel}}Â to break possible infinite planning
loop. See also {{PlanningContext#unwrap}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (IGNITE-12592) Calcite integration. Query load balancing.
[
https://issues.apache.org/jira/browse/IGNITE-12592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Seliverstov resolved IGNITE-12592.
---
Resolution: Fixed
Already done in scope of other issues.
> Calcite integration. Query load balancing.
> --
>
> Key: IGNITE-12592
> URL: https://issues.apache.org/jira/browse/IGNITE-12592
> Project: Ignite
> Issue Type: Task
>Reporter: Igor Seliverstov
>Priority: Major
>
> Currently all query tasks execute in a query pool. The pool contains a
> limited number of threads, so, in case of long running scan operation other
> query tasks will starve.
> We need to limit time a thread spends inside {{ScanNode#requestInternal
> method}}.
> See appropriate TODO
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12972) Calcite integration. Serialization refactoring
Igor Seliverstov created IGNITE-12972: - Summary: Calcite integration. Serialization refactoring Key: IGNITE-12972 URL: https://issues.apache.org/jira/browse/IGNITE-12972 Project: Ignite Issue Type: Improvement Reporter: Igor Seliverstov Assignee: Igor Seliverstov Currently we need quite a lot of classes to serialize, send and deserialize a prepared plan (in scope of node-to-node communications). It's better to do that by analogy with Calcite's RelJsonReader/RelJsonWriter. This way we may omit necessity to maintain lots of classes preserving functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12909) Calcite integration. Explain command support
Igor Seliverstov created IGNITE-12909: - Summary: Calcite integration. Explain command support Key: IGNITE-12909 URL: https://issues.apache.org/jira/browse/IGNITE-12909 Project: Ignite Issue Type: Task Reporter: Igor Seliverstov Support EXPLAIN command -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12900) Calcite integration. Use RowHandler to access fields.
Igor Seliverstov created IGNITE-12900: - Summary: Calcite integration. Use RowHandler to access fields. Key: IGNITE-12900 URL: https://issues.apache.org/jira/browse/IGNITE-12900 Project: Ignite Issue Type: Task Reporter: Igor Seliverstov Currently only Object[] can be used as a row because most of execution nodes require Object[] as a row type. Let's use generic row types with appropriate RowHandler in execution nodes and compiled expressions to get more flexibility. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12899) Calcite integration. Distribution multitrait
[ https://issues.apache.org/jira/browse/IGNITE-12899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-12899: -- Description: Currently Ignite nodes have single distribution value which isn't true in several cases like: # a partitioned table with a key alias should have 2 distribution traits: by _key and by key alias (id for example) - without distribution by _key column almost impossible to implement IGNITE-12692 # After inner / cross join the result should have several hash distributions (by left and by right keys) - it may be important on join transpose optimization # On project a source distribution key may appear more than once, so the result should contain a distribution for each distribution key position. was: Currently Ignite nodes have single distribution value which isn't true in several cases like: # a partitioned table with a key alias should have 2 distribution traits: by _key and by key alias (id for example) - without distribution by _key column almost impossible to implement IGNITE-12692 # After inner / cross join the result should have several hash distributions (by left and by right keys) - it may be important on join transpose optimization > Calcite integration. Distribution multitrait > > > Key: IGNITE-12899 > URL: https://issues.apache.org/jira/browse/IGNITE-12899 > Project: Ignite > Issue Type: Task >Reporter: Igor Seliverstov >Priority: Major > > Currently Ignite nodes have single distribution value which isn't true in > several cases like: > # a partitioned table with a key alias should have 2 distribution traits: by > _key and by key alias (id for example) - without distribution by _key column > almost impossible to implement IGNITE-12692 > # After inner / cross join the result should have several hash distributions > (by left and by right keys) - it may be important on join transpose > optimization > # On project a source distribution key may appear more than once, so the > result should contain a distribution for each distribution key position. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (IGNITE-12566) Calcite integration. Expressions evaluation.
[ https://issues.apache.org/jira/browse/IGNITE-12566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083931#comment-17083931 ] Igor Seliverstov edited comment on IGNITE-12566 at 4/15/20, 9:00 AM: - List of needed operations implementations: AND OR ABS ACOS AND ANY_VALUE ASCII ASIN ATAN ATAN2 BIT_AND BIT_OR BIT_XOR CARDINALITY CASE CAST CBRT CEIL CHARACTER_LENGTH CHAR_LENGTH CLASSIFIER COALESCE COLLECT CONCAT COS COT COUNT CURRENT_CATALOG CURRENT_DATE CURRENT_PATH CURRENT_ROLE CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURRENT_VALUE DATETIME_PLUS DEFAULT DEGREES DENSE_RANK DIVIDE DIVIDE_INTEGER ELEMENT EQUALS EXP EXTRACT FIRST_VALUE FLOOR FUSION GREATER_THAN GREATER_THAN_OR_EQUAL GROUPING GROUPING_ID INITCAP IS_A_SET IS_EMPTY IS_FALSE IS_JSON_ARRAY IS_JSON_OBJECT IS_JSON_SCALAR IS_JSON_VALUE IS_NOT_A_SET IS_NOT_EMPTY IS_NOT_FALSE IS_NOT_JSON_ARRAY IS_NOT_JSON_OBJECT IS_NOT_JSON_SCALAR IS_NOT_JSON_VALUE IS_NOT_NULL IS_NOT_TRUE IS_NULL IS_TRUE ITEM JSON_ARRAY JSON_ARRAYAGG JSON_EXISTS JSON_OBJECT JSON_OBJECTAGG JSON_QUERY JSON_VALUE_ANY JSON_VALUE_EXPRESSION LAG LAST LAST_DAY LAST_VALUE LEAD LESS_THAN LESS_THAN_OR_EQUAL LIKE LISTAGG LN LOCALTIME LOCALTIMESTAMP LOG10 LOWER MAP_VALUE_CONSTRUCTOR MAX MEMBER_OF MIN MINUS MINUS_DATE MOD MULTIPLY MULTISET_EXCEPT MULTISET_EXCEPT_DISTINCT MULTISET_INTERSECT MULTISET_INTERSECT_DISTINCT MULTISET_UNION MULTISET_UNION_DISTINCT NEXT_VALUE NOT NOT_EQUALS NOT_LIKE NOT_SIMILAR_TO NOT_SUBMULTISET_OF NTH_VALUE NTILE OR OVERLAY PI PLUS POSITION POWER PREV RADIANS RAND RAND_INTEGER RANK REGR_COUNT REINTERPRET REPLACE ROUND ROW ROW_NUMBER SESSION_USER SIGN SIMILAR_TO SIN SINGLE_VALUE SLICE STRUCT_ACCESS SUBMULTISET_OF SUBSTRING SUM SUM0 SYSTEM_USER TAN TRIM TRUNCATE TUMBLE_TVF UNARY_MINUS UNARY_PLUS UPPER USER Also we have to take into consideration types and types inference on different types are passed as function parameters (for example for math operations: DOUBLE + INTEGER = DOUBLE; INTEGER + BIG_INT = BIG_INT etc) Calcite uses code generation to generate functions for each passed types combinations at runtime, it's possible to act the same way, but having classes cache to bind logical operation to already generated implementation (not to compile code each time) was (Author: gvvinblade): List of needed operations implementations: AND OR ABS ACOS AND ANY_VALUE ASCII ASIN ATAN ATAN2 BIT_AND BIT_OR BIT_XOR CARDINALITY CASE CAST CBRT CEIL CHARACTER_LENGTH CHAR_LENGTH CLASSIFIER COALESCE COLLECT CONCAT COS COT COUNT CURRENT_CATALOG CURRENT_DATE CURRENT_PATH CURRENT_ROLE CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURRENT_VALUE DATETIME_PLUS DEFAULT DEGREES DENSE_RANK DIVIDE DIVIDE_INTEGER ELEMENT EQUALS EXP EXTRACT FIRST_VALUE FLOOR FUSION GREATER_THAN GREATER_THAN_OR_EQUAL GROUPING GROUPING_ID INITCAP IS_A_SET IS_EMPTY IS_FALSE IS_JSON_ARRAY IS_JSON_OBJECT IS_JSON_SCALAR IS_JSON_VALUE IS_NOT_A_SET IS_NOT_EMPTY IS_NOT_FALSE IS_NOT_JSON_ARRAY IS_NOT_JSON_OBJECT IS_NOT_JSON_SCALAR IS_NOT_JSON_VALUE IS_NOT_NULL IS_NOT_TRUE IS_NULL IS_TRUE ITEM JSON_ARRAY JSON_ARRAYAGG JSON_EXISTS JSON_OBJECT JSON_OBJECTAGG JSON_QUERY JSON_VALUE_ANY JSON_VALUE_EXPRESSION LAG LAST LAST_DAY LAST_VALUE LEAD LESS_THAN LESS_THAN_OR_EQUAL LIKE LISTAGG LN LOCALTIME LOCALTIMESTAMP LOG10 LOWER MAP_VALUE_CONSTRUCTOR MAX MEMBER_OF MIN MINUS MINUS_DATE MOD MULTIPLY MULTISET_EXCEPT MULTISET_EXCEPT_DISTINCT MULTISET_INTERSECT MULTISET_INTERSECT_DISTINCT MULTISET_UNION MULTISET_UNION_DISTINCT NEXT_VALUE NOT NOT_EQUALS NOT_LIKE NOT_SIMILAR_TO NOT_SUBMULTISET_OF NTH_VALUE NTILE OR OVERLAY PI PLUS POSITION POWER PREV RADIANS RAND RAND_INTEGER RANK REGR_COUNT REINTERPRET REPLACE ROUND ROW ROW_NUMBER SESSION_USER SIGN SIMILAR_TO SIN SINGLE_VALUE SLICE STRUCT_ACCESS SUBMULTISET_OF SUBSTRING SUM SUM0 SYSTEM_USER TAN TRIM TRUNCATE TUMBLE_TVF UNARY_MINUS UNARY_PLUS UPPER USER Also we have to take into consideration types and types inference on different types are passed as function parameters (for example for math operations: DOUBLE + INTEGER = DOUBLE; INTEGER + BIG_INT = BIG_INT etc) > Calcite integration. Expressions evaluation. > > > Key: IGNITE-12566 > URL: https://issues.apache.org/jira/browse/IGNITE-12566 > Project: Ignite > Issue Type: Task >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > > Currently we use a part of Calcite "Bindables" to evaluate expressions at the > execution time. Using it we > * lose a control on how expressions are evaluated > * cannot implement several important optimizations, like "keepBinary" > * can use only Object[] as a row, which causes a lot of array
[jira] [Commented] (IGNITE-12566) Calcite integration. Expressions evaluation.
[ https://issues.apache.org/jira/browse/IGNITE-12566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083931#comment-17083931 ] Igor Seliverstov commented on IGNITE-12566: --- List of needed operations implementations: AND OR ABS ACOS AND ANY_VALUE ASCII ASIN ATAN ATAN2 BIT_AND BIT_OR BIT_XOR CARDINALITY CASE CAST CBRT CEIL CHARACTER_LENGTH CHAR_LENGTH CLASSIFIER COALESCE COLLECT CONCAT COS COT COUNT CURRENT_CATALOG CURRENT_DATE CURRENT_PATH CURRENT_ROLE CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURRENT_VALUE DATETIME_PLUS DEFAULT DEGREES DENSE_RANK DIVIDE DIVIDE_INTEGER ELEMENT EQUALS EXP EXTRACT FIRST_VALUE FLOOR FUSION GREATER_THAN GREATER_THAN_OR_EQUAL GROUPING GROUPING_ID INITCAP IS_A_SET IS_EMPTY IS_FALSE IS_JSON_ARRAY IS_JSON_OBJECT IS_JSON_SCALAR IS_JSON_VALUE IS_NOT_A_SET IS_NOT_EMPTY IS_NOT_FALSE IS_NOT_JSON_ARRAY IS_NOT_JSON_OBJECT IS_NOT_JSON_SCALAR IS_NOT_JSON_VALUE IS_NOT_NULL IS_NOT_TRUE IS_NULL IS_TRUE ITEM JSON_ARRAY JSON_ARRAYAGG JSON_EXISTS JSON_OBJECT JSON_OBJECTAGG JSON_QUERY JSON_VALUE_ANY JSON_VALUE_EXPRESSION LAG LAST LAST_DAY LAST_VALUE LEAD LESS_THAN LESS_THAN_OR_EQUAL LIKE LISTAGG LN LOCALTIME LOCALTIMESTAMP LOG10 LOWER MAP_VALUE_CONSTRUCTOR MAX MEMBER_OF MIN MINUS MINUS_DATE MOD MULTIPLY MULTISET_EXCEPT MULTISET_EXCEPT_DISTINCT MULTISET_INTERSECT MULTISET_INTERSECT_DISTINCT MULTISET_UNION MULTISET_UNION_DISTINCT NEXT_VALUE NOT NOT_EQUALS NOT_LIKE NOT_SIMILAR_TO NOT_SUBMULTISET_OF NTH_VALUE NTILE OR OVERLAY PI PLUS POSITION POWER PREV RADIANS RAND RAND_INTEGER RANK REGR_COUNT REINTERPRET REPLACE ROUND ROW ROW_NUMBER SESSION_USER SIGN SIMILAR_TO SIN SINGLE_VALUE SLICE STRUCT_ACCESS SUBMULTISET_OF SUBSTRING SUM SUM0 SYSTEM_USER TAN TRIM TRUNCATE TUMBLE_TVF UNARY_MINUS UNARY_PLUS UPPER USER Also we have to take into consideration types and types inference on different types are passed as function parameters (for example for math operations: DOUBLE + INTEGER = DOUBLE; INTEGER + BIG_INT = BIG_INT etc) > Calcite integration. Expressions evaluation. > > > Key: IGNITE-12566 > URL: https://issues.apache.org/jira/browse/IGNITE-12566 > Project: Ignite > Issue Type: Task >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > > Currently we use a part of Calcite "Bindables" to evaluate expressions at the > execution time. Using it we > * lose a control on how expressions are evaluated > * cannot implement several important optimizations, like "keepBinary" > * can use only Object[] as a row, which causes a lot of array copy > operations and unnecessary allocations at the execution time > * suffer from delays, which go from expressions compilation process when > it's more efficient to start from interpretation and compile an expression on > repeatable execution only > We need to implement expressions interpreter and our own expression compiler > taking Ignite specifics in consideration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12899) Calcite integration. Distribution multitrait
Igor Seliverstov created IGNITE-12899: - Summary: Calcite integration. Distribution multitrait Key: IGNITE-12899 URL: https://issues.apache.org/jira/browse/IGNITE-12899 Project: Ignite Issue Type: Task Reporter: Igor Seliverstov Currently Ignite nodes have single distribution value which isn't true in several cases like: # a partitioned table with a key alias should have 2 distribution traits: by _key and by key alias (id for example) - without distribution by _key column almost impossible to implement IGNITE-12692 # After inner / cross join the result should have several hash distributions (by left and by right keys) - it may be important on join transpose optimization -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (IGNITE-12864) Calcite integration. UNION support.
[ https://issues.apache.org/jira/browse/IGNITE-12864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov reassigned IGNITE-12864: - Assignee: Igor Seliverstov > Calcite integration. UNION support. > --- > > Key: IGNITE-12864 > URL: https://issues.apache.org/jira/browse/IGNITE-12864 > Project: Ignite > Issue Type: New Feature >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (IGNITE-12871) Calcite integration. Broadcast to hash conversion without exchange.
[ https://issues.apache.org/jira/browse/IGNITE-12871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov reassigned IGNITE-12871: - Assignee: Igor Seliverstov > Calcite integration. Broadcast to hash conversion without exchange. > --- > > Key: IGNITE-12871 > URL: https://issues.apache.org/jira/browse/IGNITE-12871 > Project: Ignite > Issue Type: Improvement >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > > Use a partition filter instead of network communications -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12871) Calcite integration. Broadcast to hash conversion without exchange.
Igor Seliverstov created IGNITE-12871: - Summary: Calcite integration. Broadcast to hash conversion without exchange. Key: IGNITE-12871 URL: https://issues.apache.org/jira/browse/IGNITE-12871 Project: Ignite Issue Type: Improvement Reporter: Igor Seliverstov Use a partition filter instead of network communications -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12868) Calcite integration. LEFT, RIGHT join support.
Igor Seliverstov created IGNITE-12868: - Summary: Calcite integration. LEFT, RIGHT join support. Key: IGNITE-12868 URL: https://issues.apache.org/jira/browse/IGNITE-12868 Project: Ignite Issue Type: New Feature Reporter: Igor Seliverstov -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12864) Calcite integration. UNION support.
Igor Seliverstov created IGNITE-12864: - Summary: Calcite integration. UNION support. Key: IGNITE-12864 URL: https://issues.apache.org/jira/browse/IGNITE-12864 Project: Ignite Issue Type: New Feature Reporter: Igor Seliverstov -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (IGNITE-12820) Calcite integration. Do not use AbstarctConverter while planning.
[ https://issues.apache.org/jira/browse/IGNITE-12820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov reassigned IGNITE-12820: - Assignee: Igor Seliverstov > Calcite integration. Do not use AbstarctConverter while planning. > - > > Key: IGNITE-12820 > URL: https://issues.apache.org/jira/browse/IGNITE-12820 > Project: Ignite > Issue Type: Improvement >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > > We need to change traits explicitly without AbstractConverter's and > appropriate rules. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12820) Calcite integration. Do not use AbstarctConverter while planning.
Igor Seliverstov created IGNITE-12820: - Summary: Calcite integration. Do not use AbstarctConverter while planning. Key: IGNITE-12820 URL: https://issues.apache.org/jira/browse/IGNITE-12820 Project: Ignite Issue Type: Improvement Reporter: Igor Seliverstov We need to change traits explicitly without AbstractConverter's and appropriate rules. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12819) Calcite integration. Cost system.
Igor Seliverstov created IGNITE-12819: - Summary: Calcite integration. Cost system. Key: IGNITE-12819 URL: https://issues.apache.org/jira/browse/IGNITE-12819 Project: Ignite Issue Type: Task Reporter: Igor Seliverstov Current implementation doesn't suit our needs for particular reasons. We need to introduce our own one taking into account such parameters as IO operations, memory usage, CPU usage, disk usage etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12792) Calcite integration. Update Calcite version to 1.22.0
Igor Seliverstov created IGNITE-12792: - Summary: Calcite integration. Update Calcite version to 1.22.0 Key: IGNITE-12792 URL: https://issues.apache.org/jira/browse/IGNITE-12792 Project: Ignite Issue Type: Improvement Reporter: Igor Seliverstov Assignee: Igor Seliverstov -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12776) Calcite integration. An exception occurred when a replicated cache is used in a query.
Igor Seliverstov created IGNITE-12776: - Summary: Calcite integration. An exception occurred when a replicated cache is used in a query. Key: IGNITE-12776 URL: https://issues.apache.org/jira/browse/IGNITE-12776 Project: Ignite Issue Type: Bug Reporter: Igor Seliverstov Assignee: Igor Seliverstov -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12747) Calcite integration. Correlated queries support.
Igor Seliverstov created IGNITE-12747: - Summary: Calcite integration. Correlated queries support. Key: IGNITE-12747 URL: https://issues.apache.org/jira/browse/IGNITE-12747 Project: Ignite Issue Type: New Feature Reporter: Igor Seliverstov -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12708) Calcite integration. Expressions factory base implementation.
Igor Seliverstov created IGNITE-12708: - Summary: Calcite integration. Expressions factory base implementation. Key: IGNITE-12708 URL: https://issues.apache.org/jira/browse/IGNITE-12708 Project: Ignite Issue Type: New Feature Components: sql Reporter: Igor Seliverstov Assignee: Igor Seliverstov We need to implement basic environment for expressions evaluation. Expressions should exist in two types - compiled and interpreted. Expressions should be compiled in dedicated thread pull and eventually replace interpreted ones. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (IGNITE-12566) Calcite integration. Expressions evaluation.
[ https://issues.apache.org/jira/browse/IGNITE-12566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov reassigned IGNITE-12566: - Assignee: Igor Seliverstov > Calcite integration. Expressions evaluation. > > > Key: IGNITE-12566 > URL: https://issues.apache.org/jira/browse/IGNITE-12566 > Project: Ignite > Issue Type: Task >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > > Currently we use a part of Calcite "Bindables" to evaluate expressions at the > execution time. Using it we > * lose a control on how expressions are evaluated > * cannot implement several important optimizations, like "keepBinary" > * can use only Object[] as a row, which causes a lot of array copy > operations and unnecessary allocations at the execution time > * suffer from delays, which go from expressions compilation process when > it's more efficient to start from interpretation and compile an expression on > repeatable execution only > We need to implement expressions interpreter and our own expression compiler > taking Ignite specifics in consideration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12700) Calcite integration. Aggregates support.
Igor Seliverstov created IGNITE-12700: - Summary: Calcite integration. Aggregates support. Key: IGNITE-12700 URL: https://issues.apache.org/jira/browse/IGNITE-12700 Project: Ignite Issue Type: New Feature Components: sql Reporter: Igor Seliverstov Assignee: Igor Seliverstov -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12695) Calcite integration. DML support. MERGE support
Igor Seliverstov created IGNITE-12695: - Summary: Calcite integration. DML support. MERGE support Key: IGNITE-12695 URL: https://issues.apache.org/jira/browse/IGNITE-12695 Project: Ignite Issue Type: New Feature Reporter: Igor Seliverstov -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-12692) Calcite integration. DML. Skip reducer optimization.
[ https://issues.apache.org/jira/browse/IGNITE-12692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039027#comment-17039027 ] Igor Seliverstov commented on IGNITE-12692: --- UPD seems it's more efficient and natural to request from a modify node underlying table distribution and add a single distributed SUM aggregation node before it - this way a planner will try to collocate modification and source select. Adding a single distributed table modify to search scope as an alternative will cover a case when collecting phase required. This way we'll get desired behavior automatically. > Calcite integration. DML. Skip reducer optimization. > > > Key: IGNITE-12692 > URL: https://issues.apache.org/jira/browse/IGNITE-12692 > Project: Ignite > Issue Type: Improvement >Reporter: Igor Seliverstov >Priority: Major > > Having plan tree we easily can check whether a final modification may be > executed on data nodes directly or not. We should implement such kind of > optimization. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12692) Calcite integration. DML. Skip reducer optimization.
Igor Seliverstov created IGNITE-12692: - Summary: Calcite integration. DML. Skip reducer optimization. Key: IGNITE-12692 URL: https://issues.apache.org/jira/browse/IGNITE-12692 Project: Ignite Issue Type: Improvement Reporter: Igor Seliverstov Having plan tree we easily can check whether a final modification may be executed on data nodes directly or not. We should implement such kind of optimization. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12639) Calcite integration. DML support.
Igor Seliverstov created IGNITE-12639: - Summary: Calcite integration. DML support. Key: IGNITE-12639 URL: https://issues.apache.org/jira/browse/IGNITE-12639 Project: Ignite Issue Type: New Feature Reporter: Igor Seliverstov Assignee: Igor Seliverstov -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12633) Calcite integration. Query cancel purpose.
[ https://issues.apache.org/jira/browse/IGNITE-12633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-12633: -- Ignite Flags: (was: Docs Required,Release Notes Required) > Calcite integration. Query cancel purpose. > -- > > Key: IGNITE-12633 > URL: https://issues.apache.org/jira/browse/IGNITE-12633 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Igor Seliverstov >Priority: Major > > Currently on a query participant node left whole query is cancelled, but the > end user doesn't know the purpose. We should pass causing cancel exception to > end user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12633) Calcite integration. Query cancel purpose.
Igor Seliverstov created IGNITE-12633: - Summary: Calcite integration. Query cancel purpose. Key: IGNITE-12633 URL: https://issues.apache.org/jira/browse/IGNITE-12633 Project: Ignite Issue Type: Improvement Components: sql Reporter: Igor Seliverstov Currently on a query participant node left whole query is cancelled, but the end user doesn't know the purpose. We should pass causing cancel exception to end user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12620) Calcite integration. Index Nested Loop Join/Hash Join
Igor Seliverstov created IGNITE-12620: - Summary: Calcite integration. Index Nested Loop Join/Hash Join Key: IGNITE-12620 URL: https://issues.apache.org/jira/browse/IGNITE-12620 Project: Ignite Issue Type: New Feature Reporter: Igor Seliverstov We may implement the feature the next way: # For each row from left node consume whole dataset from right node # Pass join condition as an argument of request() method of the right node # In case a data source at right is an index scan ## If there is no cursor opened - create a new cursor using bounds from request ## If there is an existing cursor - just check the cursor was opened using the same condition as passed one. ## After the right node signals EOD consume a next row from left and repeat from p 2. # In case a data source at right is a table scan with huge amount of rows ## If there is no cursor opened - create a new cursor ignoring passed condition ## If there is an existing cursor - just check the cursor was opened using the same condition as passed one. ## After the right node signals EOD consume a next row from left and repeat from p 2. # In case a data source at right is remote / is a table scan with small number of rows/ is a filter node with good enough selectivity ## Create a hash index for a data source ## If there is no cursor opened - create a new cursor using bounds from request ## If there is an existing cursor - just check the cursor was opened using the same condition as passed one. ## After the right node signals EOD consume a next row from left and repeat from p 2.  Consider implementation specifics at optimization time, choose the cheapest variant -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12602) Calcite integration. JDBC Thin driver support.
Igor Seliverstov created IGNITE-12602: - Summary: Calcite integration. JDBC Thin driver support. Key: IGNITE-12602 URL: https://issues.apache.org/jira/browse/IGNITE-12602 Project: Ignite Issue Type: Task Reporter: Igor Seliverstov Provide a way to use experimental engine via JDBC thin driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (IGNITE-12602) Calcite integration. JDBC Thin driver support.
[ https://issues.apache.org/jira/browse/IGNITE-12602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov reassigned IGNITE-12602: - Assignee: Igor Seliverstov > Calcite integration. JDBC Thin driver support. > -- > > Key: IGNITE-12602 > URL: https://issues.apache.org/jira/browse/IGNITE-12602 > Project: Ignite > Issue Type: Task >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > > Provide a way to use experimental engine via JDBC thin driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (IGNITE-12449) Calcite integration. Execution flow.
[ https://issues.apache.org/jira/browse/IGNITE-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov resolved IGNITE-12449. --- Resolution: Done Done in scope of IGNITE-12448 > Calcite integration. Execution flow. > > > Key: IGNITE-12449 > URL: https://issues.apache.org/jira/browse/IGNITE-12449 > Project: Ignite > Issue Type: Task >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > > We need to introduce query execution environment. > Execution should: > * use streaming approach > * have suspend/resume ability > * work in event loop threads > Rough protocol description: > The flow is defined as a tree of operations, like filter, project etc. > Each node provides a sink to consume data from its children and has a target > sink to push data to upper node. > Upper node may signal that it's ready to consume data. After a node received > a signal it starts to push data into a target sink until the sink says there > is no place for new data, after that the node stops pushing data until a new > signal. > Some of nodes (like inbox node, describing remote input) may signal that > there is some new data, it forces a root node to signal its children top to > bottom. > When a signal arrived an inbox node, the inbox starts to push the new data. > When a node realizes the data is over, it sends "end" signal to a target > sink, after that an upper node wont signal the node to continue data pushing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12592) Calcite integration. Query load balancing.
Igor Seliverstov created IGNITE-12592:
-
Summary: Calcite integration. Query load balancing.
Key: IGNITE-12592
URL: https://issues.apache.org/jira/browse/IGNITE-12592
Project: Ignite
Issue Type: Task
Reporter: Igor Seliverstov
Currently all query tasks execute in a query pool. The pool contains a limited
number of threads, so, in case of long running scan operation other query tasks
will starve.
We need to limit time a thread spends inside {{ScanNode#requestInternal
method}}.
See appropriate TODO
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12586) Calcite integration. NodesMapping simplification.
Igor Seliverstov created IGNITE-12586:
-
Summary: Calcite integration. NodesMapping simplification.
Key: IGNITE-12586
URL: https://issues.apache.org/jira/browse/IGNITE-12586
Project: Ignite
Issue Type: Improvement
Reporter: Igor Seliverstov
Seems turning {{List assignments}} to {{Map
assignments}} may significantly reduce occupied memory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12585) Calcite integration. Splitter improvements.
Igor Seliverstov created IGNITE-12585: - Summary: Calcite integration. Splitter improvements. Key: IGNITE-12585 URL: https://issues.apache.org/jira/browse/IGNITE-12585 Project: Ignite Issue Type: Improvement Reporter: Igor Seliverstov Currently in case a head fragment of a query plan have broadcast distribution it causes OptimisticPlanningException and additional fragment split each time it's mapped on topology on a client node. Seems it's quite usual case, so, we should cover it properly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12566) Calcite integration. Expressions evaluation.
Igor Seliverstov created IGNITE-12566: - Summary: Calcite integration. Expressions evaluation. Key: IGNITE-12566 URL: https://issues.apache.org/jira/browse/IGNITE-12566 Project: Ignite Issue Type: Task Reporter: Igor Seliverstov Currently we use a part of Calcite "Bindables" to evaluate expressions at the execution time. Using it we * lose a control on how expressions are evaluated * cannot implement several important optimizations, like "keepBinary" * can use only Object[] as a row, which causes a lot of array copy operations and unnecessary allocations at the execution time * suffer from delays, which go from expressions compilation process when it's more efficient to start from interpretation and compile an expression on repeatable execution only We need to implement expressions interpreter and our own expression compiler taking Ignite specifics in consideration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12564) Calcite integration. Exceptions handling.
Igor Seliverstov created IGNITE-12564: - Summary: Calcite integration. Exceptions handling. Key: IGNITE-12564 URL: https://issues.apache.org/jira/browse/IGNITE-12564 Project: Ignite Issue Type: Task Reporter: Igor Seliverstov There are several exceptions types that may be thrown at * parsing and validation time * planning time * mapping time * execution time All of them should be handled properly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12563) Calcite integration. Reset and remap queries on topology change.
Igor Seliverstov created IGNITE-12563: - Summary: Calcite integration. Reset and remap queries on topology change. Key: IGNITE-12563 URL: https://issues.apache.org/jira/browse/IGNITE-12563 Project: Ignite Issue Type: Task Components: sql Reporter: Igor Seliverstov A query should: * be silently remapped in case an actual topology version at query plan materialization time differs from a topology version at the time locations was calculated * be cancelled with appropriate exception in case one of responsible nodes left a cluster after execution is started. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12448) Calcite integration. Communication protocol.
[ https://issues.apache.org/jira/browse/IGNITE-12448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-12448: -- Reviewer: Roman Kondakov > Calcite integration. Communication protocol. > > > Key: IGNITE-12448 > URL: https://issues.apache.org/jira/browse/IGNITE-12448 > Project: Ignite > Issue Type: Task > Components: sql >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > We need to introduce a communication protocol between query fragments in > scope of query execution. > * Protocol should have Push semantics with back pressure ability > * Protocol have to provide ordering guaranties - the data batches processed > in same order they sent to remote node. > * Protocol have to provide delivery guaranties. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-12449) Calcite integration. Execution flow.
[ https://issues.apache.org/jira/browse/IGNITE-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17002813#comment-17002813 ] Igor Seliverstov commented on IGNITE-12449: --- Current limitations: only limited operation types are implemented: nested loop join, filter, project, exchange, table scan (sort, index scan are under development) what is out of scope: query parsing, planning, validation, etc. So, let's say you have several query fragments (executing on different nodes), each of them is an operations tree, each of them executes in any order. that we need to prove: * dependency conflicts are solving as expected, so, regardless of initial start order all, the query parts eventually executes in proper order. * There is no deadlocks * There is no OOM, operation buffers are used as expected, except known cases when an operation requires full dataset (nested loop joins, sort operations, grouping operations, etc) > Calcite integration. Execution flow. > > > Key: IGNITE-12449 > URL: https://issues.apache.org/jira/browse/IGNITE-12449 > Project: Ignite > Issue Type: Task >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > > We need to introduce query execution environment. > Execution should: > * use streaming approach > * have suspend/resume ability > * work in event loop threads > Rough protocol description: > The flow is defined as a tree of operations, like filter, project etc. > Each node provides a sink to consume data from its children and has a target > sink to push data to upper node. > Upper node may signal that it's ready to consume data. After a node received > a signal it starts to push data into a target sink until the sink says there > is no place for new data, after that the node stops pushing data until a new > signal. > Some of nodes (like inbox node, describing remote input) may signal that > there is some new data, it forces a root node to signal its children top to > bottom. > When a signal arrived an inbox node, the inbox starts to push the new data. > When a node realizes the data is over, it sends "end" signal to a target > sink, after that an upper node wont signal the node to continue data pushing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-12449) Calcite integration. Execution flow.
[ https://issues.apache.org/jira/browse/IGNITE-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17002806#comment-17002806 ] Igor Seliverstov commented on IGNITE-12449: --- [~ustas], this is a brand new query execution flow (designed for Calcite based query engine) that uses a reactive approach and "push" semantics. This is needed to execute a query, with possible cross dependencies, by limited number of threads without blocking operations when all dependency conflicts are solving automatically by operations reordering (if an operation needs some data that unavailable at the moment, it stops executing until a data source signals there is some new data available). At the other hand "push" approach reduces request-reply roundtrips, that reduces query execution latency. Back pressure is a part of the protocol (if a consumer is not ready to process a data producer stops data sending until a consumer signal) it allows us to use small operation buffers that reduces possible OOM during execution - a reducer doesn't collect whole dataset except several cases, like data sorting, that effectively may be solved by an index introducing. > Calcite integration. Execution flow. > > > Key: IGNITE-12449 > URL: https://issues.apache.org/jira/browse/IGNITE-12449 > Project: Ignite > Issue Type: Task >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > > We need to introduce query execution environment. > Execution should: > * use streaming approach > * have suspend/resume ability > * work in event loop threads > Rough protocol description: > The flow is defined as a tree of operations, like filter, project etc. > Each node provides a sink to consume data from its children and has a target > sink to push data to upper node. > Upper node may signal that it's ready to consume data. After a node received > a signal it starts to push data into a target sink until the sink says there > is no place for new data, after that the node stops pushing data until a new > signal. > Some of nodes (like inbox node, describing remote input) may signal that > there is some new data, it forces a root node to signal its children top to > bottom. > When a signal arrived an inbox node, the inbox starts to push the new data. > When a node realizes the data is over, it sends "end" signal to a target > sink, after that an upper node wont signal the node to continue data pushing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-12449) Calcite integration. Execution flow.
[ https://issues.apache.org/jira/browse/IGNITE-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17002796#comment-17002796 ] Igor Seliverstov commented on IGNITE-12449: --- [~ustas], please look at > Calcite integration. Execution flow. > > > Key: IGNITE-12449 > URL: https://issues.apache.org/jira/browse/IGNITE-12449 > Project: Ignite > Issue Type: Task >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > > We need to introduce query execution environment. > Execution should: > * use streaming approach > * have suspend/resume ability > * work in event loop threads > Rough protocol description: > The flow is defined as a tree of operations, like filter, project etc. > Each node provides a sink to consume data from its children and has a target > sink to push data to upper node. > Upper node may signal that it's ready to consume data. After a node received > a signal it starts to push data into a target sink until the sink says there > is no place for new data, after that the node stops pushing data until a new > signal. > Some of nodes (like inbox node, describing remote input) may signal that > there is some new data, it forces a root node to signal its children top to > bottom. > When a signal arrived an inbox node, the inbox starts to push the new data. > When a node realizes the data is over, it sends "end" signal to a target > sink, after that an upper node wont signal the node to continue data pushing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12449) Calcite integration. Execution flow.
[ https://issues.apache.org/jira/browse/IGNITE-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-12449: -- Description: We need to introduce query execution environment. Execution should: * use streaming approach * have suspend/resume ability * work in event loop threads Rough protocol description: The flow is defined as a tree of operations, like filter, project etc. Each node provides a sink to consume data from its children and has a target sink to push data to upper node. Upper node may signal that it's ready to consume data. After a node received a signal it starts to push data into a target sink until the sink says there is no place for new data, after that the node stops pushing data until a new signal. Some of nodes (like inbox node, describing remote input) may signal that there is some new data, it forces a root node to signal its children top to bottom. When a signal arrived an inbox node, the inbox starts to push the new data. When a node realizes the data is over, it sends "end" signal to a target sink, after that an upper node wont signal the node to continue data pushing. was: We need to introduce query execution environment. Execution should: * use streaming approach * have suspend/resume ability * work in event loop threads Rough protocol description: The flow defined as a tree of operations, like filter, project etc > Calcite integration. Execution flow. > > > Key: IGNITE-12449 > URL: https://issues.apache.org/jira/browse/IGNITE-12449 > Project: Ignite > Issue Type: Task >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > > We need to introduce query execution environment. > Execution should: > * use streaming approach > * have suspend/resume ability > * work in event loop threads > Rough protocol description: > The flow is defined as a tree of operations, like filter, project etc. > Each node provides a sink to consume data from its children and has a target > sink to push data to upper node. > Upper node may signal that it's ready to consume data. After a node received > a signal it starts to push data into a target sink until the sink says there > is no place for new data, after that the node stops pushing data until a new > signal. > Some of nodes (like inbox node, describing remote input) may signal that > there is some new data, it forces a root node to signal its children top to > bottom. > When a signal arrived an inbox node, the inbox starts to push the new data. > When a node realizes the data is over, it sends "end" signal to a target > sink, after that an upper node wont signal the node to continue data pushing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12449) Calcite integration. Execution flow.
[ https://issues.apache.org/jira/browse/IGNITE-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-12449: -- Description: We need to introduce query execution environment. Execution should: * use streaming approach * have suspend/resume ability * work in event loop threads Rough protocol description: The flow defined as a tree of operations, like filter, project etc was: We need to introduce query execution environment. Execution should: * use streaming approach * have suspend/resume ability * work in event loop threads > Calcite integration. Execution flow. > > > Key: IGNITE-12449 > URL: https://issues.apache.org/jira/browse/IGNITE-12449 > Project: Ignite > Issue Type: Task >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > > We need to introduce query execution environment. > Execution should: > * use streaming approach > * have suspend/resume ability > * work in event loop threads > Rough protocol description: > The flow defined as a tree of operations, like filter, project etc -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (IGNITE-12449) Calcite integration. Execution flow.
[ https://issues.apache.org/jira/browse/IGNITE-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov reassigned IGNITE-12449: - Assignee: Igor Seliverstov > Calcite integration. Execution flow. > > > Key: IGNITE-12449 > URL: https://issues.apache.org/jira/browse/IGNITE-12449 > Project: Ignite > Issue Type: Task >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > > We need to introduce query execution environment. > Execution should: > * use streaming approach > * have suspend/resume ability > * work in event loop threads -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (IGNITE-12448) Calcite integration. Communication protocol.
[ https://issues.apache.org/jira/browse/IGNITE-12448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov reassigned IGNITE-12448: - Assignee: Igor Seliverstov > Calcite integration. Communication protocol. > > > Key: IGNITE-12448 > URL: https://issues.apache.org/jira/browse/IGNITE-12448 > Project: Ignite > Issue Type: Task > Components: sql >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > > We need to introduce a communication protocol between query fragments in > scope of query execution. > * Protocol should have Push semantics with back pressure ability > * Protocol have to provide ordering guaranties - the data batches processed > in same order they sent to remote node. > * Protocol have to provide delivery guaranties. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12455) Calcite integration. Partition pruning.
Igor Seliverstov created IGNITE-12455: - Summary: Calcite integration. Partition pruning. Key: IGNITE-12455 URL: https://issues.apache.org/jira/browse/IGNITE-12455 Project: Ignite Issue Type: Improvement Reporter: Igor Seliverstov In scope of fragment info calculation we're able to prune partitions on the basis of filter condition, query parameters, affinity and tables distribution. This optimization may dramatically reduce query execution time/needed resources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12449) Calcite integration. Execution flow.
Igor Seliverstov created IGNITE-12449: - Summary: Calcite integration. Execution flow. Key: IGNITE-12449 URL: https://issues.apache.org/jira/browse/IGNITE-12449 Project: Ignite Issue Type: Task Reporter: Igor Seliverstov We need to introduce query execution environment. Execution should: * use streaming approach * have suspend/resume ability * work in event loop threads -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12448) Calcite integration. Communication protocol.
Igor Seliverstov created IGNITE-12448: - Summary: Calcite integration. Communication protocol. Key: IGNITE-12448 URL: https://issues.apache.org/jira/browse/IGNITE-12448 Project: Ignite Issue Type: Task Components: sql Reporter: Igor Seliverstov We need to introduce a communication protocol between query fragments in scope of query execution. * Protocol should have Push semantics with back pressure ability * Protocol have to provide ordering guaranties - the data batches processed in same order they sent to remote node. * Protocol have to provide delivery guaranties. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12446) Calcite integration. Fix javadocs and code style.
Igor Seliverstov created IGNITE-12446: - Summary: Calcite integration. Fix javadocs and code style. Key: IGNITE-12446 URL: https://issues.apache.org/jira/browse/IGNITE-12446 Project: Ignite Issue Type: Task Components: sql Reporter: Igor Seliverstov Assignee: Igor Seliverstov Fix javadocs and code style -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (IGNITE-12248) Apache Calcite based query execution engine
[ https://issues.apache.org/jira/browse/IGNITE-12248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov reassigned IGNITE-12248: - Assignee: (was: Igor Seliverstov) > Apache Calcite based query execution engine > --- > > Key: IGNITE-12248 > URL: https://issues.apache.org/jira/browse/IGNITE-12248 > Project: Ignite > Issue Type: Task > Components: sql >Reporter: Igor Seliverstov >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > Currently used H2 based query execution engine has a number of critical > limitations Which do not allow to execute an arbitrary query. > The ticket aims to show potential of a new, Calcite based, execution engine > which may act not worse than current one on co-located queries, provide a > boost for queries, using distributed joins, and provide an ability to execute > arbitrary queries, requiring more than one map-reduce step in execution flow. > [IEP > link|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=130028084] > [Dev list > thread|http://apache-ignite-developers.2346864.n4.nabble.com/New-SQL-execution-engine-td43724.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (IGNITE-12251) Calcite exchange rules
[ https://issues.apache.org/jira/browse/IGNITE-12251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov resolved IGNITE-12251. --- Resolution: Fixed Done as a part of initial prototype. > Calcite exchange rules > -- > > Key: IGNITE-12251 > URL: https://issues.apache.org/jira/browse/IGNITE-12251 > Project: Ignite > Issue Type: Task > Components: sql >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > > Need to implement IgniteDistributionTraitDef and utility methods (to > calculate distribution trait for each node type based on its position, type > and children nodes) > Next distribution types needed: random, hash, single, broadcast > hash distribution trait should have affinity information -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12251) Calcite exchange rules
[ https://issues.apache.org/jira/browse/IGNITE-12251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-12251: -- Ignite Flags: (was: Docs Required,Release Notes Required) > Calcite exchange rules > -- > > Key: IGNITE-12251 > URL: https://issues.apache.org/jira/browse/IGNITE-12251 > Project: Ignite > Issue Type: Task > Components: sql >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > > Need to implement IgniteDistributionTraitDef and utility methods (to > calculate distribution trait for each node type based on its position, type > and children nodes) > Next distribution types needed: random, hash, single, broadcast > hash distribution trait should have affinity information -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (IGNITE-12295) Faster index eviction
[ https://issues.apache.org/jira/browse/IGNITE-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957806#comment-16957806 ] Igor Seliverstov edited comment on IGNITE-12295 at 10/23/19 12:02 PM: -- [~skalashnikov], in general the approach is OK, at least I don't see any issues. Some comments: -- a deleting entries cursor looks unclear and redundant. -- need to clean up the code or document it, why we need so many Runnables in EvictionManager? Seems you should refactor it and make simpler. For example let indexing returns a single purge task which is registering in Eviction manager. -- other issues highlighted by [~northdragon] was (Author: gvvinblade): [~skalashnikov], in general the approach is OK, at least I don't see any issues. Some comments: -- a deleting entries cursor looks unclear and redundantly. -- need to clean up the code or document it, why we need so many Runnables in EvictionManager? Seems you should refactor it and make simpler. For example let indexing returns a single purge task which is registering in Eviction manager. -- other issues highlighted by [~northdragon] > Faster index eviction > - > > Key: IGNITE-12295 > URL: https://issues.apache.org/jira/browse/IGNITE-12295 > Project: Ignite > Issue Type: Sub-task >Reporter: Sergey Kalashnikov >Assignee: Sergey Kalashnikov >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > For the file-based rebalancing approach, it seems feasible to avoid iterating > the old partition data in order to clear the indexes. > One can independently clear the shared index structures of all the rows > referencing entries from moving partitions by deducing partition id from the > links in the leaf pages. > The proposed algorithm is simple and takes the set of integer partition ids > as an input: > 1. Iterate over leaf pages of the index and remove items attributed to any of > indicated partitions, unless it is the only or the rightmost item on a page. > 2. If the rightmost item (or the only item) on a page happens to belong to > any of the indicated partitions, employ a regular remove algorithm > (descending from the root) so that inner pages get correctly updated. > Restart iteration from the leaf page where the removed item would be inserted > (descend from the root to find it). > The use of such algorithm can be justified (as having performance advantage) > when the number of keys that'd be removed is bigger than the number of leaf > pages in the index. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-12295) Faster index eviction
[ https://issues.apache.org/jira/browse/IGNITE-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957806#comment-16957806 ] Igor Seliverstov commented on IGNITE-12295: --- [~skalashnikov], in general the approach is OK, at least I don't see any issues. Some comments: -- a deleting entries cursor looks unclear and redundantly. -- need to clean up the code or document it, why we need so many Runnables in EvictionManager? Seems you should refactor it and make simpler. For example let indexing returns a single purge task which is registering in Eviction manager. -- other issues highlighted by [~northdragon] > Faster index eviction > - > > Key: IGNITE-12295 > URL: https://issues.apache.org/jira/browse/IGNITE-12295 > Project: Ignite > Issue Type: Sub-task >Reporter: Sergey Kalashnikov >Assignee: Sergey Kalashnikov >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > For the file-based rebalancing approach, it seems feasible to avoid iterating > the old partition data in order to clear the indexes. > One can independently clear the shared index structures of all the rows > referencing entries from moving partitions by deducing partition id from the > links in the leaf pages. > The proposed algorithm is simple and takes the set of integer partition ids > as an input: > 1. Iterate over leaf pages of the index and remove items attributed to any of > indicated partitions, unless it is the only or the rightmost item on a page. > 2. If the rightmost item (or the only item) on a page happens to belong to > any of the indicated partitions, employ a regular remove algorithm > (descending from the root) so that inner pages get correctly updated. > Restart iteration from the leaf page where the removed item would be inserted > (descend from the root to find it). > The use of such algorithm can be justified (as having performance advantage) > when the number of keys that'd be removed is bigger than the number of leaf > pages in the index. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12251) Calcite exchange rules
[ https://issues.apache.org/jira/browse/IGNITE-12251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-12251: -- Description: Need to implement IgniteDistributionTraitDef and utility methods (to calculate distribution trait for each node type based on its position, type and children nodes) Next distribution types needed: random, hash, single, broadcast hash distribution trait should have affinity information was: Need to implement IgniteDistributionTraitDef and utility methods (to calculate distribution trait for each node type based on its position, type and children nodes) Next distribution types needed: random, hash, single, broadcast > Calcite exchange rules > -- > > Key: IGNITE-12251 > URL: https://issues.apache.org/jira/browse/IGNITE-12251 > Project: Ignite > Issue Type: Task > Components: sql >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > > Need to implement IgniteDistributionTraitDef and utility methods (to > calculate distribution trait for each node type based on its position, type > and children nodes) > Next distribution types needed: random, hash, single, broadcast > hash distribution trait should have affinity information -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12251) Calcite exchange rules
[ https://issues.apache.org/jira/browse/IGNITE-12251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov updated IGNITE-12251: -- Description: Need to implement IgniteDistributionTraitDef and utility methods (to calculate distribution trait for each node type based on its position, type and children nodes) Next distribution types needed: random, hash, single, broadcast was:Need to implement IgniteDistributionTraitDef and utility methods (to calculate distribution trait for each node type based on its position, type and children nodes) > Calcite exchange rules > -- > > Key: IGNITE-12251 > URL: https://issues.apache.org/jira/browse/IGNITE-12251 > Project: Ignite > Issue Type: Task > Components: sql >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > > Need to implement IgniteDistributionTraitDef and utility methods (to > calculate distribution trait for each node type based on its position, type > and children nodes) > Next distribution types needed: random, hash, single, broadcast -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12251) Calcite exchange rules
Igor Seliverstov created IGNITE-12251: - Summary: Calcite exchange rules Key: IGNITE-12251 URL: https://issues.apache.org/jira/browse/IGNITE-12251 Project: Ignite Issue Type: Task Components: sql Reporter: Igor Seliverstov Assignee: Igor Seliverstov Need to implement IgniteDistributionTraitDef and utility methods (to calculate distribution trait for each node type based on its position, type and children nodes) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12248) Apache Calcite based query execution engine
Igor Seliverstov created IGNITE-12248: - Summary: Apache Calcite based query execution engine Key: IGNITE-12248 URL: https://issues.apache.org/jira/browse/IGNITE-12248 Project: Ignite Issue Type: Task Components: sql Reporter: Igor Seliverstov Assignee: Igor Seliverstov Currently used H2 based query execution engine has a number of critical limitations Which do not allow to execute an arbitrary query. The ticket aims to show potential of a new, Calcite based, execution engine which may act not worse than current one on co-located queries, provide a boost for queries, using distributed joins, and provide an ability to execute arbitrary queries, requiring more than one map-reduce step in execution flow. [IEP link|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=130028084] [Dev list thread|http://apache-ignite-developers.2346864.n4.nabble.com/New-SQL-execution-engine-td43724.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-5714) Implementation of suspend/resume for pessimistic transactions
[ https://issues.apache.org/jira/browse/IGNITE-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904961#comment-16904961 ] Igor Seliverstov commented on IGNITE-5714: -- [~alex_pl], now it looks much better, I have no objections to merge the PR. > Implementation of suspend/resume for pessimistic transactions > - > > Key: IGNITE-5714 > URL: https://issues.apache.org/jira/browse/IGNITE-5714 > Project: Ignite > Issue Type: Sub-task > Components: general >Reporter: Alexey Kuznetsov >Assignee: Aleksey Plekhanov >Priority: Major > Labels: iep-34 > Time Spent: 4.5h > Remaining Estimate: 0h > > Support transaction suspend()\resume() operations for pessimistic > transactions. Resume can be called in another thread. >_+But there is a problem+_: Imagine, we started pessimistic transaction in > thread T1 and then perform put operation, which leads to sending > GridDistributedLockRequest to another node. Lock request contains thread id > of the transaction. Then we call suspend, resume in another thread and we > also must send messages to other nodes to change thread id. > It seems complicated task.It’s better to get rid of sending thread id to the > nodes. > We can use transaction xid on other nodes instead of thread id. Xid is sent > to nodes in GridDistributedLockRequest#nearXidVer >_+Proposed solution+_ : On remote nodes instead of thread id of near > transaction GridDistributedLockRequest#threadId use its xid > GridDistributedLockRequest#nearXidVer. > Remove usages of near transaction's thread id on remote nodes. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (IGNITE-6809) Use MPSC queue in striped pool
[ https://issues.apache.org/jira/browse/IGNITE-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Seliverstov resolved IGNITE-6809. -- Resolution: Won't Do The ticket is closed since several MPSC queue implementations in striped pools don't bring expected performance improvements. Unlike JMH microbenchmarks show 2 times better throughput the overall picture is even worse because MPSC queues bring improvements only on massive tasks stream when the stripe acts almost without parking, that's really rare case. On sparse workload (usual case) the MPSC queue works almost the same way as jdk's concurrent linked queue but with additional parkings (there is no spin as it did in current stripes implementations). Seems the change is premature and it makes sense to put efforts into improvement of other system parts. > Use MPSC queue in striped pool > -- > > Key: IGNITE-6809 > URL: https://issues.apache.org/jira/browse/IGNITE-6809 > Project: Ignite > Issue Type: Improvement > Components: general >Affects Versions: 2.3 >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > Labels: performance > Attachments: [IGNITE-6809]_Use_MPSC_queue_in_striped_pool_.patch > > > Use MPSC queue in striped pool > Let's start from [MP-SC concurrent linked queue implementation based on > Dmitry > Vyukov's|http://www.1024cores.net/home/lock-free-algorithms/queues/non-intrusive-mpsc-node-based-queue] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
