[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894459#comment-15894459
 ] 

ASF GitHub Bot commented on IGNITE-4106:


Github user AMashenkov closed the pull request at:

https://github.com/apache/ignite/pull/1476


> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
> Fix For: 1.9
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[Andrew Mashenkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880386#comment-15880386
 ] 

Andrew Mashenkov commented on IGNITE-4106:
--

Segmented spatial index support added as well.

> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
> Fix For: 1.9
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878059#comment-15878059
 ] 

ASF GitHub Bot commented on IGNITE-4106:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1556


> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
> Fix For: 1.9
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875007#comment-15875007
 ] 

ASF GitHub Bot commented on IGNITE-4106:


GitHub user AMashenkov opened a pull request:

https://github.com/apache/ignite/pull/1556

IGNITE-4106: SQL: parallelize sql queries over cache local partitions

Merged with IGNITE-1.9.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4106-1.9

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1556.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1556


commit 608032b900a51a7fba99e383c0fe0d8a37558cbb
Author: AMRepo 
Date:   2017-02-20T18:24:29Z

Implemented.

commit b7c5ae313fa7c4cc6765e2f3ce8eb30d46c93f10
Author: AMRepo 
Date:   2017-02-20T19:53:33Z

Merge remote-tracking branch 'apache/ignite-1.9' into ignite-4106-1.9

# Conflicts:
#   
modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/IgniteH2Indexing.java
#   
modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/opt/GridH2IndexBase.java
#   
modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/twostep/GridMapQueryExecutor.java
#   
modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/twostep/GridReduceQueryExecutor.java
#   
modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/twostep/msg/GridH2QueryRequest.java
#   
modules/indexing/src/test/java/org/apache/ignite/internal/processors/query/IgniteSqlSplitterSelfTest.java




> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
> Fix For: 1.9
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[Andrew Mashenkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874980#comment-15874980
 ] 

Andrew Mashenkov commented on IGNITE-4106:
--

Minor API fix: qryParallelismLevel can be set on per cache base.
Allow REPLICATED cache to be used with segmented PARTITIONED in one query.
Tests added.

Sync-ed with master.

> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
> Fix For: 1.9
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874918#comment-15874918
 ] 

ASF GitHub Bot commented on IGNITE-4106:


Github user AMashenkov closed the pull request at:

https://github.com/apache/ignite/pull/1278


> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
> Fix For: 1.9
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[Andrew Mashenkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856711#comment-15856711
 ] 

Andrew Mashenkov commented on IGNITE-4106:
--

[~sergi.vladykin],
I've done. Would you please review a PR?

> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
> Fix For: 1.9
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[Andrew Mashenkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854808#comment-15854808
 ] 

Andrew Mashenkov commented on IGNITE-4106:
--

Found an issue. Local queries work incorrect with segmented index due to 
TwoStepQuery does not support local properly.
Need a few time to fix it.

> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
> Fix For: 1.9
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[Andrew Mashenkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848539#comment-15848539
 ] 

Andrew Mashenkov commented on IGNITE-4106:
--

Local queries is fixed.
Waiting for TC tests.

> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
> Fix For: 1.9
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843190#comment-15843190
 ] 

ASF GitHub Bot commented on IGNITE-4106:


GitHub user AMashenkov opened a pull request:

https://github.com/apache/ignite/pull/1476

IGNITE-4106: SQL: parallelize sql queries over cache local partitions



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4106

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1476.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1476


commit 5d0417c8cb0f95bc5506a41028b830b831f19863
Author: AMRepo 
Date:   2017-01-19T20:35:06Z

Implemented

commit e638a91eaba80c1f71c0b8a74a59cd5e32122ac2
Author: Andrey V. Mashenkov 
Date:   2017-01-20T10:54:21Z

Minors: Styles & Javadoc

commit 14066d16d429a236ff7bde3e4b5b2653343709c7
Author: Andrey V. Mashenkov 
Date:   2017-01-27T17:02:02Z

Local queries over stripped indices fixed.




> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
> Fix For: 1.9
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843172#comment-15843172
 ] 

ASF GitHub Bot commented on IGNITE-4106:


Github user AMashenkov closed the pull request at:

https://github.com/apache/ignite/pull/1444


> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
> Fix For: 1.9
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830113#comment-15830113
 ] 

ASF GitHub Bot commented on IGNITE-4106:


GitHub user AMashenkov opened a pull request:

https://github.com/apache/ignite/pull/1444

IGNITE-4106: SQL: parallelize sql queries over cache local partitions

Implemented

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4106

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1444.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1444


commit 6fffa4afb6699fdb54fc5e3ca02797547c4d09ea
Author: Andrey V. Mashenkov 
Date:   2017-01-19T15:24:03Z

Implemented




> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
> Fix For: 1.9
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830104#comment-15830104
 ] 

ASF GitHub Bot commented on IGNITE-4106:


Github user AMashenkov closed the pull request at:

https://github.com/apache/ignite/pull/1386


> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
> Fix For: 1.9
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[Andrew Mashenkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15829852#comment-15829852
 ] 

Andrew Mashenkov commented on IGNITE-4106:
--

[~sergi.vladykin] thanks for review, I'll fix PR shortly.

> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
> Fix For: 1.9
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778467#comment-15778467
 ] 

ASF GitHub Bot commented on IGNITE-4106:


GitHub user AMashenkov opened a pull request:

https://github.com/apache/ignite/pull/1386

IGNITE-4106: SQL: parallelize sql queries over cache local partitions

Tree index splitted into segmentes in GridH2StripedTreeIndex
Tests added

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4106

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1386.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1386


commit d013500bd897505ec6cbfb2f25d54f1f938f2f83
Author: AMRepo 
Date:   2016-12-26T14:54:01Z

Tree index splitted into segmentes in GridH2StripedTreeIndex

Tests added

commit 3202889b6005eaad77026e4682e7b6e8fb336ba5
Author: AMRepo 
Date:   2016-12-26T15:06:11Z

Tree index splitted into segmentes in GridH2StripedTreeIndex
Tests added




> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.6, 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance, sql
> Fix For: 2.0
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778462#comment-15778462
 ] 

ASF GitHub Bot commented on IGNITE-4106:


Github user AMashenkov closed the pull request at:

https://github.com/apache/ignite/pull/1385


> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.6, 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance, sql
> Fix For: 2.0
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778443#comment-15778443
 ] 

ASF GitHub Bot commented on IGNITE-4106:


Github user AMashenkov closed the pull request at:

https://github.com/apache/ignite/pull/1198


> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.6, 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance, sql
> Fix For: 2.0
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[Andrew Mashenkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722743#comment-15722743
 ] 

Andrew Mashenkov commented on IGNITE-4106:
--

Segmented tree index implementation moved to separate class. 
ReduceQueryExecutor now sends only one query request per node. 
MapQueryExecutor bothered with starting query threads if needed.
Corresponding configuration options added.



> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.6, 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance, sql
> Fix For: 2.0
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[Andrew Mashenkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15693238#comment-15693238
 ] 

Andrew Mashenkov commented on IGNITE-4106:
--

I've done with splitting indices. There are some broken tests left, unresolved 
issue with distributed joins and issue with prepared statement cache.

Prepared statement cache is now broken due to H2 knows nothing about splitted 
indices and how they work. Reused statement can return same result on different 
segments, so I've just disabled this feature.

Serj, would you please review PR #1278 for if I go the right way.



> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.6, 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
> Fix For: 2.0
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15693220#comment-15693220
 ] 

ASF GitHub Bot commented on IGNITE-4106:


GitHub user AMashenkov opened a pull request:

https://github.com/apache/ignite/pull/1278

IGNITE-4106: SQL: parallelize sql queries over cache local partitions

Indexes splitted

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4106-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1278.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1278


commit cede430c27f1bc73a3fd61a3569c164d16869816
Author: Andrey V. Mashenkov 
Date:   2016-11-18T12:19:33Z

Merged

commit e8329f93003c77d1070cf88be3cbaf19e7845656
Author: Andrey V. Mashenkov 
Date:   2016-11-18T12:47:05Z

index splitted

commit 4e88be252ce35482ac4b885e8e97b7fa16d3e53b
Author: Andrey V. Mashenkov 
Date:   2016-11-24T12:46:15Z

Merge remote-tracking branch 'apache/master' into ignite-4106-2

# Conflicts:
#   
modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/twostep/GridMergeIndex.java




> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.6, 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
> Fix For: 2.0
>
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[Andrew Mashenkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15667581#comment-15667581
 ] 

Andrew Mashenkov commented on IGNITE-4106:
--

We need to split GridH2TreeIndex into segments, to each thread can work with 
its own segment. 

GridH2Tree.doFind method is feasible candidate. Partition filter that was saved 
in GridH2QueryContext is accessed in doFind method. We can add additional 
context property to transfer some "segment index" instead of using partition 
filter. This will reduce message size and simplify code, due to we shouldn't be 
bothered with distributing partitions among the threads in 
GridReduceQueryExecutor. 


> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.6, 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[Andrew Mashenkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15650767#comment-15650767
 ] 

Andrew Mashenkov commented on IGNITE-4106:
--

It seems that 
Every query thread read full node index and then filter data within its own 
partition subset. Thus, query splitted to several threads will induce full node 
index to be read once per partition subset.
In current (1 thread per node) approach, all query threads  work with its own 
smaller index.


> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.6, 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[Andrew Mashenkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629132#comment-15629132
 ] 

Andrew Mashenkov commented on IGNITE-4106:
--

JFR profiling results is attached.

> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.6, 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
> Attachments: 1node-4thread.jfr, 4node-1thread.jfr
>
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[Andrew Mashenkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629128#comment-15629128
 ] 

Andrew Mashenkov commented on IGNITE-4106:
--

Approach to split sql query by sending Map queries messages with subset of 
primary partitions is seems to be more simple.
Although it mechanics similar to multi-node, I would expect equals trend of 
throughput as for threads count as nodes count. 
I've no success to find a reason yet, it seems there are contentions somewhere 
and I don't have enough expertise yet to find it alone.

> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.6, 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622307#comment-15622307
 ] 

ASF GitHub Bot commented on IGNITE-4106:


GitHub user AMashenkov opened a pull request:

https://github.com/apache/ignite/pull/1198

IGNITE-4106: SQL: parallelize sql queries over cache local partitions



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4106

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1198.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1198


commit 87a1928a4f90b4f8a221041cfff9d22e3dd801cc
Author: vozerov-gridgain 
Date:   2016-08-26T12:22:15Z

IGNITE-3776: Removed code duplication in GridNearAtomicAbstractUpdateFuture.

commit 92f18bf353cc8c3821c6500ce9f1cd397a7cf17c
Author: Andrey V. Mashenkov 
Date:   2016-08-26T12:31:30Z

IGNITE-3745: ODBC: Implemented date/time/timestamp escape sequence parsing. 
This closes #991.

commit b5757642e135908d9baa027a605035dd0d4acfc9
Author: tledkov-gridgain 
Date:   2016-08-26T12:47:02Z

IGNITE-3670 IGFS: Improved symlink handling for delete operation and added 
more tests. This closes #975.

commit 289170346e40a89243b80d3743c1b66543a117ef
Author: Igor Sapego 
Date:   2016-08-29T12:00:08Z

IGNITE-3773: ODBC: Added tests for UUID data type. This closes #993.

commit 0465874d9dddcf962a82a2ef38589121201f0b75
Author: agura 
Date:   2016-08-24T18:13:29Z

ignite-2968 Deadlock detection for optimistic tx and near caches

commit fea3eeaac839381ec3662275a82f4d12af6f12dd
Author: agura 
Date:   2016-08-29T15:38:35Z

Merged ignite-1.6.7 to ignite-1.7.2

commit 407071e466d1a4fcec86571d4791ace8bb206f53
Author: Eduard Shangareev 
Date:   2016-08-30T00:32:31Z

IGNITE-2546 - Transformers for SCAN queries. Fixes #949.

commit f9ff97c91374dcd9cd8ad08d46d1d2de44193060
Author: Andrey V. Mashenkov 
Date:   2016-08-30T06:31:20Z

ignite-2560 Support resource injection for entry processor, optimizations 
for resource injection.

commit 3244a5c9dabf6d4fcaa32a661cc0adc6f8ea30de
Author: Andrey V. Mashenkov 
Date:   2016-08-30T08:49:11Z

IGNITE-3742: ODBC: Added support for OUTER JOIN escape sequence. This 
closes #1000.

commit fbbcaf4322548f61d2f63bf5d4e8f6d5284e73d3
Author: Andrey V. Mashenkov 
Date:   2016-08-30T10:22:29Z

IGNITE-3798: ODBC: Added literals support. This closes #1005.

commit 1ef150eba52eb63c2bfc3fafa0d036cf26be1c5b
Author: Alexey Kuznetsov 
Date:   2016-08-30T11:18:20Z

Revert wrong merge.

commit 4a259da296946740e180d25559e5cacba60a97ad
Author: EdShangGG 
Date:   2016-08-30T11:39:13Z

Merge remote-tracking branch 'ignite-gg/ignite-1.6.7' into ignite-1.7.2

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/GridCacheQueryManager.java
#   
modules/indexing/src/test/java/org/apache/ignite/testsuites/IgniteCacheQuerySelfTestSuite.java
#   
modules/spring/src/test/java/org/apache/ignite/testsuites/IgniteSpringTestSuite.java

commit 31dbc5d65f8ea51010d2129e7c6e9a27acbf8528
Author: isapego 
Date:   2016-08-30T14:28:34Z

Fix for C++ tests.

commit dabd86c62e39eb983ef3d198c8b8ee96d2623c84
Author: Andrey Novikov 
Date:   2016-08-31T09:00:19Z

IGNITE-3811 Fixed query for merge with complex key on SQL Server.

commit 70e69cb7aa08c268b07920838add4a40e28fe25d
Author: isapego 
Date:   2016-08-31T13:47:11Z

IGNITE-3390: Added DSN configuration window.

commit bdbc5a37c1924ba4427221ee6cdf506c7f3ba248
Author: isapego 
Date:   2016-08-31T14:13:34Z

Merge branch 'ignite-1.6.7' into ignite-1.7.2

commit 12f532986677c30a716f73aeaa7d3587dd701f55
Author: sboikov 
Date:   2016-09-01T14:05:15Z

Merge remote-tracking branch 'remotes/community/ignite-1.6.6' into 
ignite-1.6.7

commit ae7765329fd6f7d50d13183d13626f39c5682334
Author: dkarachentsev 
Date:   2016-09-02T15:01:12Z

IGNITE-2208 - Queries with object arguments doesn't work wth 
BinaryMarshaller.

commit 7a84ab6a9163ca31fbcfcc6d7ff27e06bf9babef
Author: vozerov-gridgain 
Date:   2016-09-02T15:05:16Z

IGNITE-3827: Removed double marshalling of keys in 
DataStreamerImpl.addData(Map) method.

commit 7bb961fd0a2e78334d33b6bc279c4edc323c246a
Author: vozerov-gridgain 

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[Andrew Mashenkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15616263#comment-15616263
 ] 

Andrew Mashenkov commented on IGNITE-4106:
--

I've implementes 2 prototypes. In both I try to speed up SQL query Map phase 
with multi-threading approach.
Compared scenarios: 1 node with splitting into 4 threads vs 4 nodes without 
splitting.

1) The first one. MapQuery message processing splits into several threads. Each 
thread runs same query over certain cache local partitions. When all threads 
fiished - results merged and return to Reducer. This approach shows significant 
speedup, but throughput is 10-15% slower than if we just add more nodes to 
grid. Code is far from ideal, i believe we can fix this 10-15% slowdown.

2)  The second. I try to split queries with sending more Map queries messages 
from query initiator node. But subset of primary partitions for target node 
were specified in these messages . So, remote nodes process these messages in 
parallel. This approach give worse results, throughput is 50% slower than if we 
just add more nodes to grid.

> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.6, 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split local SQL queries over cache partitions, so we 
> would be able to parallelize SQL query on every single node and utilize CPU 
> more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4106) SQL: parallelize sql queries over cache local partitions

[Andrew Mashenkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602127#comment-15602127
 ] 

Andrew Mashenkov commented on IGNITE-4106:
--

I've noticed, that jdbc connection cached in ThreadLocal cache, one connection 
per schema (ignite cache). It seems that public\system or may be other pool's 
threads, that used to execute sql queries, hold connection objects.

If we will split queries in thread per partition manner, we can get 
uncontrolled growth of opened connections. So, imho, 
[IGNITE-4105|https://issues.apache.org/jira/browse/IGNITE-4105] should be done 
at first.



> SQL: parallelize sql queries over cache local partitions
> 
>
> Key: IGNITE-4106
> URL: https://issues.apache.org/jira/browse/IGNITE-4106
> Project: Ignite
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 1.6, 1.7
>Reporter: Andrew Mashenkov
>Assignee: Andrew Mashenkov
>  Labels: performance
>
> If we run SQL query on cache partitioned over several cluster nodes, it will 
> be split into several queries running in parallel. But really we will have 
> one thread per query on each node.
> So, for now, to improve SQL query performance we need to run more Ignite 
> instances or split caches manually.
> It seems to be better to split SQL queries over cache partitions, so we would 
> be able to parallelize SQL query on every single node and utilize CPU more 
> efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)