[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17191792#comment-17191792 ] Tanmay Ambre commented on IGNITE-13270: --- [~zstan] - we dont have distributed joins. the query I provided uses the affinity key to join - hence i think the joins will be collocated. also regarding passing integers to query statements. we dont do that. Our input is a text string. Hope this helps. I was not able to understand your comment completely. Hence unable to answer in detail. > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Priority: Major > Attachments: IDX13270.java, IgniteSqlDistributedJoinSelfTest.java > > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[
https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183055#comment-17183055
]
Taras Ledkov commented on IGNITE-13270:
---
[~zstan]
About reproducer with invalid results:
The root cause is using the index on VARCHAR column {{idx_B_COL2}} and lookup
the integer value at this index.
We have a problem with lookup value with type conversion at the index.
The reproducer may be shorter:
{code}
sql("CREATE TABLE TEST (ID INTEGER PRIMARY KEY, VAL VARCHAR(100), VAL1
int)");
sql("CREATE INDEX IDX_VAL ON TEST (VAL)");
// Fail
res = sql("SELECT * FROM TEST WHERE VAL=?", i).getAll();
// OK
res = sql("SELECT * FROM TEST WHERE VAL=?",
Integer.toString(i)).getAll();
{code}
> Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
> --
>
> Key: IGNITE-13270
> URL: https://issues.apache.org/jira/browse/IGNITE-13270
> Project: Ignite
> Issue Type: Bug
> Components: sql
>Affects Versions: 2.8, 2.8.1
>Reporter: Tanmay Ambre
>Priority: Major
> Attachments: IDX13270.java, IgniteSqlDistributedJoinSelfTest.java
>
>
> After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing
> massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on
> each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes
> are above 80%. On 2.7.5 the same query was performing very nicely.
> The query is a join with 3 tables having same affinity key. however, we dont
> know the affinity key when we are querying and there this results in a merge
> scan.
> Query:
> select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C
> where
> A.A_ID = B.A_ID AND
> B.B_ID = C.B_ID AND
> A.AFFINITYKEY = B.AFFINITYKEY AND
> B.AFFINITYKEY = C.AFFINITYKEY AND
> B.B_ID = ? AND
> C.C_ID = ?
>
> Performance:
> 2.7.5 ~ 2 to 5 ms
> 2.8.0 ~ 6 to 12 seconds
> 2.8.1 ~ within 3 seconds
>
> Volume (we have a EDA based system where messages come in burst. One burst
> has approx. 150K events - each event correspond to one query).
>
> throughput of our java based processor (that fires this query):
> 2.7.5 ~ 750 transactions per second
> 2.8.0 ~ 3 transactions per second
> 2.8.1 ~ 6 transactions per second
>
> CPU burn
> 2.7.5 ~ 10 to 15% on each node
> 2.8.0 ~ 100% on 2 nodes other nodes are idling
> 2.8.1 ~ 80 to 90% on each node
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17181986#comment-17181986 ] Stanilovsky Evgeny commented on IGNITE-13270: - [~tambre] check this ticket [1] and comments near, in this case we have the same story - not collocated joins without distributed joins flag [2] works unproperly, i suppose we need to add additional logging in such cases. Additionally reproducer attached highlighting the bug with differ table population, i.r. if you have *VARCHAR(N)* in table definition and *INSERT INTO VALUES ... N* where *N* used without quotation - here would be a problem. [1] https://issues.apache.org/jira/browse/IGNITE-13019 [2] https://apacheignite-sql.readme.io/docs/distributed-joins > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Priority: Major > Attachments: IDX13270.java, IgniteSqlDistributedJoinSelfTest.java > > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17181852#comment-17181852 ] Stanilovsky Evgeny commented on IGNITE-13270: - still have problems in master, org.apache.ignite.internal.processors.query.IgniteSqlDistributedJoinSelfTest#testJoinWithEqualAffinityKeys check reproducer attached, uncomment one line with index and test will be ok. > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Assignee: Stanilovsky Evgeny >Priority: Major > Attachments: IDX13270.java, IgniteSqlDistributedJoinSelfTest.java > > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17178561#comment-17178561 ] Tanmay Ambre commented on IGNITE-13270: --- [~zstan] - yes 2.8.0 is good. There was this one query which spiked the CPU. We have broken it down into 2 queries and the performance is acceptable. when we moved to 2.8.0 - 2.8.1 was not available. However, in the coming months - we would upgrade. Yes - I do agree for having the test case for avoiding potential issues in the future. What I dont understand is even with 2.7.5 we were getting accurate results with lower CPU and with 2.8.0 we are getting the same results (in our case we are looking for just one record). I think what you found out was when the result set holds multiple records - 2.7.5 had issues. > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Assignee: Stanilovsky Evgeny >Priority: Major > Attachments: IDX13270.java > > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17176084#comment-17176084 ] Stanilovsky Evgeny commented on IGNITE-13270: - ticket still in Open state because i want to form appropriate test and merge it for further possible problems prevention in this part. Is 2.8.0 acceptable for you usage for now ? Btw 2.8.1 already released is better to use it. > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Assignee: Stanilovsky Evgeny >Priority: Major > Attachments: IDX13270.java > > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17176078#comment-17176078 ] Tanmay Ambre commented on IGNITE-13270: --- hi - we have decided to split the query and optimize it and we will be upgrading to 2.8.0. Thanks a lot for your help. I think if this is an expected behavior - you can close this issue. > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Assignee: Stanilovsky Evgeny >Priority: Major > Attachments: IDX13270.java > > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165115#comment-17165115 ] Tanmay Ambre commented on IGNITE-13270: --- Thanks - I will try to reproduce it. Thanks again > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Assignee: Stanilovsky Evgeny >Priority: Major > Attachments: IDX13270.java > > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163442#comment-17163442 ] Stanilovsky Evgeny commented on IGNITE-13270: - [~tambre] check for attached reproducer plz, i found that ignite 2.7.5 has bugs, thus you can obtain incorrect resultset here (attached test prove it) . If you want to repeat my experiments just download appropriate sources and put this class in neighbor of BasicIndexTest.java for example. So my conclusion - your cpu burn is due to increasing result sets and further processing > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Assignee: Stanilovsky Evgeny >Priority: Major > Attachments: IDX13270.java > > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162725#comment-17162725 ] Tanmay Ambre commented on IGNITE-13270: --- query that we fire is: select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C where A.A_ID = B.A_ID AND B.B_ID = C.B_ID AND A.AFFINITYKEY = B.AFFINITYKEY AND B.AFFINITYKEY = C.AFFINITYKEY AND B.COL2 = ? AND C.COL6 = ? > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Priority: Major > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162724#comment-17162724 ] Tanmay Ambre commented on IGNITE-13270: --- CREATE TABLE IF NOT EXISTS A ( A_ID bigint, affinityKey varchar(100), COL1 varchar(30), COL2 bigint, COL3 varchar(40), COL4 varchar(40), COL5 varchar (100), COL6 varchar (100), COL7 int, COL8 int COL9 timestamp, COl10 varchar(30), COL11 bigint, COL12 varchar(50), COL13 timestamp, primary key(A_ID, affinityKey) ) WITH "template=inputCacheTemplate, CACHE_NAME=ACache, AFFINITY_KEY=affinityKey, CACHE_GROUP=INPUT_GROUP KEY_TYPE=com.foo.AKey, VALUE_TYPE=com.foo.A"; CREATE TABLE IF NOT EXISTS B ( B_ID bigint, affinityKey varchar(100), A_ID bigint, COL1 varchar(30), COL2 varchar(100), COL3 bigint, COL4 varchar(50), COL5 varchar(30), COL6 varchar(20), COL7 timestamp, COL8 timestamp, COL9 timestamp, COL10 timestamp, primary key(B_ID, affinityKey) ) WITH "template=inputCacheTemplate, CACHE_NAME=BCache, AFFINITY_KEY=affinityKey, CACHE_GROUP=INPUT_GROUP KEY_TYPE=com.foo.BKey, VALUE_TYPE=com.foo.B"; CREATE TABLE IF NOT EXISTS C( C_ID bigint, B_ID bigint, affinityKey varchar(100), COL1 varchar(20), COL2 bigint, COL3 varchar(50), COL4 varchar(20), COL5 varchar(20), COL6 int, COL7 timestamp, primary key(C_ID, affinityKey) ) WITH "template=inputCacheTemplate, CACHE_NAME=CCache, AFFINITY_KEY=affinityKey, CACHE_GROUP=INPUT_GROUP KEY_TYPE=com.foo.CKey, VALUE_TYPE=com.foo.C"; CREATE INDEX IF NOT EXISTS idx_A_A_ID ON A(A_ID); CREATE INDEX IF NOT EXISTS idx_B_B_ID ON B(B_ID); CREATE INDEX IF NOT EXISTS idx_B_A_ID ON B(A_ID); CREATE INDEX IF NOT EXISTS idx_B_COL2 ON B(COL2) INLINE SIZE 128; CREATE INDEX IF NOT EXISTS idx_C_C_ID on C(C_ID); CREATE INDEX IF NOT EXISTS idx_C_B_ID on C(B_ID); CREATE INDEX IF NOT EXISTS idx_C_COL6 on C(COL6); inputCacheTemplate is PARTITIONED, backups=1, atomicityMode=ATOMIC,partitionLossPolicy=READ_WRITE_SAFE, writeSyncrhonizationMode=PRIMARY_SYNC, statisticsEnabled=true, affinity=RendezvousAffinityFunction (partitions=256), dataRegionName=dr.input dr.input is specified as: persistenceEnabled=false initialSize=64G maxSize=200G metricsEnabled=true > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Priority: Major > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162070#comment-17162070 ] Stanilovsky Evgeny commented on IGNITE-13270: - ok, i try to reproduce it if you give me table and index creation queries for A B and C tables plz. > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Priority: Major > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162018#comment-17162018 ] Tanmay Ambre commented on IGNITE-13270: --- No our caches are transient. We don't use persistence To give you a background of the tests - # we do a complete data wipe of data in ignite (as if its a new install) # activate the cluster # Recreate our caches and indexes # Load data # and then test We followed the same process for testing all 3 versions of ignite. Problem is the same query with same plan works nicely on 2.7.5 but not on 2.8.0 and 2.8.1 - This is the concern. The CPU burn is order of magnitude higher in 2.8.0 and 2.8.1 > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Priority: Major > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162019#comment-17162019 ] Tanmay Ambre commented on IGNITE-13270: --- h2 version is 1.4.197 > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Priority: Major > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17161895#comment-17161895 ] Stanilovsky Evgeny commented on IGNITE-13270: - ok, without deep investigation i can suggest only : 1. check that your sql plans are correct now : explain select ... <- you need to find correct index usage here 2. if p 1. is incorrect or cpu burn still in progress try to rebuild current indexes, stop node by node, remove or move index.bin file from all or suspicious caches and start node - index will rebuild automatically. And one point - we are talking about cluster with persistent caches isn`t it ? > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Priority: Major > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17161784#comment-17161784 ] Tanmay Ambre commented on IGNITE-13270: --- Trace 5: org.h2.value.Value.cache line: 410 org.h2.value.ValueString.get line: 153 org.h2.value.ValueString.get line: 134 org.apache.ignite.internal.processors.query.h2.H2Utils.wrap line: 582 org.apache.ignite.internal.processors.query.h2.opt.H2CacheRow.wrap line: 169 org.apache.ignite.internal.processors.query.h2.opt.H2CacheRow.getValue0 line: 104 org.apache.ignite.internal.processors.query.h2.opt.H2CacheRow.getValue line: 86 org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare line: 392 org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare line: 63 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare line: 5200 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.findLowerBound line: 5317 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0 line: 5588 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer line: 5376 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.nextPage line: 5428 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.next line: 5661 org.apache.ignite.internal.processors.query.h2.H2Cursor.next line: 66 org.h2.index.IndexCursor.next line: 316 org.h2.table.TableFilter.next line: 502 org.h2.command.dml.Select$LazyResultQueryFlat.fetchNextRow line: 1452 org.h2.result.LazyResult.hasNext line: 79 org.h2.result.LazyResult.next line: 59 org.h2.command.dml.Select.queryFlat line: 527 org.h2.command.dml.Select.queryWithoutCache line: 633 org.h2.command.dml.Query.queryWithoutCacheLazyCheck line: 114 org.h2.command.dml.Query.query line: 352 org.h2.command.dml.Query.query line: 333 org.h2.command.CommandContainer.query line: 114 org.h2.command.Command.executeQuery line: 202 org.h2.jdbc.JdbcPreparedStatement.executeQuery line: 114 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery line: 824 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer line: 912 org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0 line: 417 org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest line: 242 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.onMessage line: 2138 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.lambda$start$17 line: 2095 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$$Lambda$482/222427158.onMessage line: not available org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage line: 3379 org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener line: 1843 org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0 line: 1468 org.apache.ignite.internal.managers.communication.GridIoManager.access$5200 line: 229 org.apache.ignite.internal.managers.communication.GridIoManager$9.run line: 1365 java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149 java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624 java.lang.Thread.run line: 748 > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Priority: Major > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst >
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17161783#comment-17161783 ] Tanmay Ambre commented on IGNITE-13270: --- Trace 4: org.apache.ignite.internal.processors.cache.persistence.DataStructure.read line: 341 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.read line: 5876 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.askNeighbor line: 2569 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1300 line: 94 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0 line: 342 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run line: 5709 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run line: 278 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run line: 5695 org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage line: 169 org.apache.ignite.internal.processors.cache.persistence.DataStructure.read line: 364 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.read line: 5896 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown line: 1365 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown line: 1374 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown line: 1374 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown line: 1374 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown line: 1374 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown line: 1374 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown line: 1374 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown line: 1374 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doFind line: 1332 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findOne line: 1300 org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.find line: 360 org.h2.index.BaseIndex.find line: 130 org.h2.index.IndexCursor.find line: 176 org.h2.table.TableFilter.next line: 471 org.h2.table.TableFilter.next line: 541 org.h2.command.dml.Select$LazyResultQueryFlat.fetchNextRow line: 1452 org.h2.result.LazyResult.hasNext line: 79 org.h2.result.LazyResult.next line: 59 org.h2.command.dml.Select.queryFlat line: 527 org.h2.command.dml.Select.queryWithoutCache line: 633 org.h2.command.dml.Query.queryWithoutCacheLazyCheck line: 114 org.h2.command.dml.Query.query line: 352 org.h2.command.dml.Query.query line: 333 org.h2.command.CommandContainer.query line: 114 org.h2.command.Command.executeQuery line: 202 org.h2.jdbc.JdbcPreparedStatement.executeQuery line: 114 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery line: 824 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer line: 912 org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0 line: 417 org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest line: 242 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.onMessage line: 2138 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.lambda$start$17 line: 2095 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$$Lambda$482/222427158.onMessage line: not available org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage line: 3379 org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener line: 1843 org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0 line: 1468 org.apache.ignite.internal.managers.communication.GridIoManager.access$5200 line: 229 org.apache.ignite.internal.managers.communication.GridIoManager$9.run line: 1365 java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149 java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624 java.lang.Thread.run line: 748 > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Priority: Major > > After upgrading to ignite 2.8.0 fr
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17161781#comment-17161781 ] Tanmay Ambre commented on IGNITE-13270: --- Trace 2: org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.acquirePage line: 481 org.apache.ignite.internal.processors.cache.persistence.DataStructure.acquirePage line: 158 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.acquirePage line: 5858 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.nextPage line: 5417 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.next line: 5661 org.apache.ignite.internal.processors.query.h2.H2Cursor.next line: 66 org.h2.index.IndexCursor.next line: 316 org.h2.table.TableFilter.next line: 502 org.h2.command.dml.Select$LazyResultQueryFlat.fetchNextRow line: 1452 org.h2.result.LazyResult.hasNext line: 79 org.h2.result.LazyResult.next line: 59 org.h2.command.dml.Select.queryFlat line: 527 org.h2.command.dml.Select.queryWithoutCache line: 633 org.h2.command.dml.Query.queryWithoutCacheLazyCheck line: 114 org.h2.command.dml.Query.query line: 352 org.h2.command.dml.Query.query line: 333 org.h2.command.CommandContainer.query line: 114 org.h2.command.Command.executeQuery line: 202 org.h2.jdbc.JdbcPreparedStatement.executeQuery line: 114 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery line: 824 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer line: 912 org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0 line: 417 org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest line: 242 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.onMessage line: 2138 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.lambda$start$17 line: 2095 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$$Lambda$482/222427158.onMessage line: not available org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage line: 3379 org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener line: 1843 org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0 line: 1468 org.apache.ignite.internal.managers.communication.GridIoManager.access$5200 line: 229 org.apache.ignite.internal.managers.communication.GridIoManager$9.run line: 1365 java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149 java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624 java.lang.Thread.run line: 748 > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Priority: Major > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17161782#comment-17161782 ] Tanmay Ambre commented on IGNITE-13270: --- Trace 3: org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage line: 165 org.apache.ignite.internal.processors.cache.persistence.DataStructure.read line: 364 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.read line: 5896 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown line: 1365 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown line: 1374 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown line: 1374 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown line: 1374 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown line: 1374 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown line: 1374 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown line: 1374 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown line: 1374 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doFind line: 1332 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findOne line: 1300 org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.find line: 360 org.h2.index.BaseIndex.find line: 130 org.h2.index.IndexCursor.find line: 176 org.h2.table.TableFilter.next line: 471 org.h2.table.TableFilter.next line: 541 org.h2.command.dml.Select$LazyResultQueryFlat.fetchNextRow line: 1452 org.h2.result.LazyResult.hasNext line: 79 org.h2.result.LazyResult.next line: 59 org.h2.command.dml.Select.queryFlat line: 527 org.h2.command.dml.Select.queryWithoutCache line: 633 org.h2.command.dml.Query.queryWithoutCacheLazyCheck line: 114 org.h2.command.dml.Query.query line: 352 org.h2.command.dml.Query.query line: 333 org.h2.command.CommandContainer.query line: 114 org.h2.command.Command.executeQuery line: 202 org.h2.jdbc.JdbcPreparedStatement.executeQuery line: 114 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery line: 824 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer line: 912 org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0 line: 417 org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest line: 242 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.onMessage line: 2138 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.lambda$start$17 line: 2095 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$$Lambda$482/222427158.onMessage line: not available org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage line: 3379 org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener line: 1843 org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0 line: 1468 org.apache.ignite.internal.managers.communication.GridIoManager.access$5200 line: 229 org.apache.ignite.internal.managers.communication.GridIoManager$9.run line: 1365 java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149 java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624 java.lang.Thread.run line: 748 > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Priority: Major > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system wher
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17161780#comment-17161780 ] Tanmay Ambre commented on IGNITE-13270: --- Have taken multiple snapshots 1 -- org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.read line: 5896 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown line: 1365 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown line: 1374 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doFind line: 1332 org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findOne line: 1300 org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.find line: 360 org.h2.index.BaseIndex.find line: 130 org.h2.index.IndexCursor.find line: 176 org.h2.table.TableFilter.next line: 471 org.h2.table.TableFilter.next line: 541 org.h2.command.dml.Select$LazyResultQueryFlat.fetchNextRow line: 1452 org.h2.result.LazyResult.hasNext line: 79 org.h2.result.LazyResult.next line: 59 org.h2.command.dml.Select.queryFlat line: 527 org.h2.command.dml.Select.queryWithoutCache line: 633 org.h2.command.dml.Query.queryWithoutCacheLazyCheck line: 114 org.h2.command.dml.Query.query line: 352 org.h2.command.dml.Query.query line: 333 org.h2.command.CommandContainer.query line: 114 org.h2.command.Command.executeQuery line: 202 org.h2.jdbc.JdbcPreparedStatement.executeQuery line: 114 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery line: 824 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer line: 912 org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0 line: 417 org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest line: 242 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.onMessage line: 2138 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.lambda$start$17 line: 2095 org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$$Lambda$482/222427158.onMessage line: not available org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage line: 3379 org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener line: 1843 org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0 line: 1468 org.apache.ignite.internal.managers.communication.GridIoManager.access$5200 line: 229 org.apache.ignite.internal.managers.communication.GridIoManager$9.run line: 1365 java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149 java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624 java.lang.Thread.run line: 748 > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Priority: Major > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17161756#comment-17161756 ] Stanilovsky Evgeny commented on IGNITE-13270: - you can erase sensitive info, looks like it would be better than only description) > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Priority: Major > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17161482#comment-17161482 ] Tanmay Ambre commented on IGNITE-13270: --- I will try. But due to confidentiality agreement - I am unable to ship the trace as-is. but will provide some additional info. Thanks. > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Priority: Major > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17161338#comment-17161338 ] Stanilovsky Evgeny commented on IGNITE-13270: - [~tambre] can you attach these dumps, please? > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Priority: Major > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17161192#comment-17161192 ] Tanmay Ambre commented on IGNITE-13270: --- What we are seeing in threadumps is that the CPU is consumed by "#query" threads. Also they cause blocking for each other. Most of the stacktraces for these threads were inside h2. > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Priority: Major > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
[ https://issues.apache.org/jira/browse/IGNITE-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17161185#comment-17161185 ] Tanmay Ambre commented on IGNITE-13270: --- Due to confidentiality of the application and data - can't share the exact query and data. However, Table A has approx. 0.75 million records Table B has approx. 1.13 million records Table C has approx. 1.17 million records > Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 > -- > > Key: IGNITE-13270 > URL: https://issues.apache.org/jira/browse/IGNITE-13270 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.8, 2.8.1 >Reporter: Tanmay Ambre >Priority: Major > > After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing > massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on > each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes > are above 80%. On 2.7.5 the same query was performing very nicely. > The query is a join with 3 tables having same affinity key. however, we dont > know the affinity key when we are querying and there this results in a merge > scan. > Query: > select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C > where > A.A_ID = B.A_ID AND > B.B_ID = C.B_ID AND > A.AFFINITYKEY = B.AFFINITYKEY AND > B.AFFINITYKEY = C.AFFINITYKEY AND > B.B_ID = ? AND > C.C_ID = ? > > Performance: > 2.7.5 ~ 2 to 5 ms > 2.8.0 ~ 6 to 12 seconds > 2.8.1 ~ within 3 seconds > > Volume (we have a EDA based system where messages come in burst. One burst > has approx. 150K events - each event correspond to one query). > > throughput of our java based processor (that fires this query): > 2.7.5 ~ 750 transactions per second > 2.8.0 ~ 3 transactions per second > 2.8.1 ~ 6 transactions per second > > CPU burn > 2.7.5 ~ 10 to 15% on each node > 2.8.0 ~ 100% on 2 nodes other nodes are idling > 2.8.1 ~ 80 to 90% on each node > -- This message was sent by Atlassian Jira (v8.3.4#803005)
