[jira] [Commented] (PHOENIX-4460) High GC / RS shutdown when we use select query with "IN" clause using 4.10 phoenix client on 4.13 phoenix server
[
https://issues.apache.org/jira/browse/PHOENIX-4460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418673#comment-16418673
]
Rajeshbabu Chintaguntla commented on PHOENIX-4460:
--
This isn't applicable for 5.x branch.
> High GC / RS shutdown when we use select query with "IN" clause using 4.10
> phoenix client on 4.13 phoenix server
>
>
> Key: PHOENIX-4460
> URL: https://issues.apache.org/jira/browse/PHOENIX-4460
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Blocker
> Fix For: 4.14.0, 4.13.2
>
> Attachments: PHOENIX-4460-v2.patch, PHOENIX-4460.patch
>
>
> We were able to reproduce the High GC / RS shutdown / phoenix KeyRange query
> high object count issue on cluster today.
> Main observation is that this is reproducible when firing lots of query
> select from xyz where abc in (?, ?, ...) of this type with 4.10 phoenix
> client hitting 4.13 phoenix on HBase server side
> (4.10 client/4.10 server works fine, 4.13 client with 4.13 server works fine)
> We wrote a loader client (attached) with the below table/query , upserted
> ~100 million rows and fired the query in parallel using 4-5 loader clients
> with 16 threads each
> {code}
> TABLE: = "CREATE TABLE " + TABLE_NAME_TEMPLATE
> + " (\n" + " TestKey varchar(255) PRIMARY KEY, TestVal1 varchar(200),
> TestVal2 varchar(200), " + "TestValue varchar(1))";
> QUERY: = "SELECT * FROM " + TABLE_NAME_TEMPLATE + " WHERE TestKey IN (?, ?,
> ?, ?, ?, ?, ?, ?, ?, ?)"
> {code}
> After running this client immediately within a min or two we see the
> phoenix.query.KeyRange object count immediately going up to several lakhs and
> keeps on increasing continuously. This count doesn't seem to come down even
> after shutting down the clients
> {code}
> -bash-4.1$ ~/current/bigdata-util/tools/Linux/jdk/jdk1.8.0_102_x64/bin/jmap
> -histo:live 90725 | grep KeyRange
> 47:2748526596448 org.apache.phoenix.query.KeyRange
> 1851: 2 48 org.apache.phoenix.query.KeyRange$Bound
> 2434: 1 24 [Lorg.apache.phoenix.query.KeyRange$Bound;
> 3411: 1 16 org.apache.phoenix.query.KeyRange$1
> 3412: 1 16 org.apache.phoenix.query.KeyRange$2
> {code}
> After some time we also started seeing High GC issues and RegionServers
> crashing
> Experiment Summary:
> - 4.13 client/4.13 Server --- Issue not reproducible (we do see KeyRange
> count increasing upto few 100's)
> - 4.10 client/4.10 Server --- Issue not reproducible (we do see KeyRange
> count increasing upto few 100's)
> - 4.10 client/4.13 Server --- Issue reproducible as described above
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4460) High GC / RS shutdown when we use select query with "IN" clause using 4.10 phoenix client on 4.13 phoenix server
[
https://issues.apache.org/jira/browse/PHOENIX-4460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296329#comment-16296329
]
Hudson commented on PHOENIX-4460:
-
SUCCESS: Integrated in Jenkins build Phoenix-master #1897 (See
[https://builds.apache.org/job/Phoenix-master/1897/])
PHOENIX-4460 High GC / RS shutdown when we use select query with IN (tdsilva:
rev 9d8be0e9214ba3680a81c399c5da316c1b91c99b)
* (edit)
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/BaseScannerRegionObserver.java
> High GC / RS shutdown when we use select query with "IN" clause using 4.10
> phoenix client on 4.13 phoenix server
>
>
> Key: PHOENIX-4460
> URL: https://issues.apache.org/jira/browse/PHOENIX-4460
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Blocker
> Fix For: 4.14.0, 4.13.2
>
> Attachments: PHOENIX-4460-v2.patch, PHOENIX-4460.patch
>
>
> We were able to reproduce the High GC / RS shutdown / phoenix KeyRange query
> high object count issue on cluster today.
> Main observation is that this is reproducible when firing lots of query
> select from xyz where abc in (?, ?, ...) of this type with 4.10 phoenix
> client hitting 4.13 phoenix on HBase server side
> (4.10 client/4.10 server works fine, 4.13 client with 4.13 server works fine)
> We wrote a loader client (attached) with the below table/query , upserted
> ~100 million rows and fired the query in parallel using 4-5 loader clients
> with 16 threads each
> {code}
> TABLE: = "CREATE TABLE " + TABLE_NAME_TEMPLATE
> + " (\n" + " TestKey varchar(255) PRIMARY KEY, TestVal1 varchar(200),
> TestVal2 varchar(200), " + "TestValue varchar(1))";
> QUERY: = "SELECT * FROM " + TABLE_NAME_TEMPLATE + " WHERE TestKey IN (?, ?,
> ?, ?, ?, ?, ?, ?, ?, ?)"
> {code}
> After running this client immediately within a min or two we see the
> phoenix.query.KeyRange object count immediately going up to several lakhs and
> keeps on increasing continuously. This count doesn't seem to come down even
> after shutting down the clients
> {code}
> -bash-4.1$ ~/current/bigdata-util/tools/Linux/jdk/jdk1.8.0_102_x64/bin/jmap
> -histo:live 90725 | grep KeyRange
> 47:2748526596448 org.apache.phoenix.query.KeyRange
> 1851: 2 48 org.apache.phoenix.query.KeyRange$Bound
> 2434: 1 24 [Lorg.apache.phoenix.query.KeyRange$Bound;
> 3411: 1 16 org.apache.phoenix.query.KeyRange$1
> 3412: 1 16 org.apache.phoenix.query.KeyRange$2
> {code}
> After some time we also started seeing High GC issues and RegionServers
> crashing
> Experiment Summary:
> - 4.13 client/4.13 Server --- Issue not reproducible (we do see KeyRange
> count increasing upto few 100's)
> - 4.10 client/4.10 Server --- Issue not reproducible (we do see KeyRange
> count increasing upto few 100's)
> - 4.10 client/4.13 Server --- Issue reproducible as described above
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-4460) High GC / RS shutdown when we use select query with "IN" clause using 4.10 phoenix client on 4.13 phoenix server
[
https://issues.apache.org/jira/browse/PHOENIX-4460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16294010#comment-16294010
]
James Taylor commented on PHOENIX-4460:
---
[~pboado] - you’ll want the fix for this before cutting an RC. We’ll likely do
a 4.13.2 with this fix.
> High GC / RS shutdown when we use select query with "IN" clause using 4.10
> phoenix client on 4.13 phoenix server
>
>
> Key: PHOENIX-4460
> URL: https://issues.apache.org/jira/browse/PHOENIX-4460
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Blocker
> Fix For: 4.14.0, 4.13.2
>
> Attachments: PHOENIX-4460-v2.patch, PHOENIX-4460.patch
>
>
> We were able to reproduce the High GC / RS shutdown / phoenix KeyRange query
> high object count issue on cluster today.
> Main observation is that this is reproducible when firing lots of query
> select from xyz where abc in (?, ?, ...) of this type with 4.10 phoenix
> client hitting 4.13 phoenix on HBase server side
> (4.10 client/4.10 server works fine, 4.13 client with 4.13 server works fine)
> We wrote a loader client (attached) with the below table/query , upserted
> ~100 million rows and fired the query in parallel using 4-5 loader clients
> with 16 threads each
> {code}
> TABLE: = "CREATE TABLE " + TABLE_NAME_TEMPLATE
> + " (\n" + " TestKey varchar(255) PRIMARY KEY, TestVal1 varchar(200),
> TestVal2 varchar(200), " + "TestValue varchar(1))";
> QUERY: = "SELECT * FROM " + TABLE_NAME_TEMPLATE + " WHERE TestKey IN (?, ?,
> ?, ?, ?, ?, ?, ?, ?, ?)"
> {code}
> After running this client immediately within a min or two we see the
> phoenix.query.KeyRange object count immediately going up to several lakhs and
> keeps on increasing continuously. This count doesn't seem to come down even
> after shutting down the clients
> {code}
> -bash-4.1$ ~/current/bigdata-util/tools/Linux/jdk/jdk1.8.0_102_x64/bin/jmap
> -histo:live 90725 | grep KeyRange
> 47:2748526596448 org.apache.phoenix.query.KeyRange
> 1851: 2 48 org.apache.phoenix.query.KeyRange$Bound
> 2434: 1 24 [Lorg.apache.phoenix.query.KeyRange$Bound;
> 3411: 1 16 org.apache.phoenix.query.KeyRange$1
> 3412: 1 16 org.apache.phoenix.query.KeyRange$2
> {code}
> After some time we also started seeing High GC issues and RegionServers
> crashing
> Experiment Summary:
> - 4.13 client/4.13 Server --- Issue not reproducible (we do see KeyRange
> count increasing upto few 100's)
> - 4.10 client/4.10 Server --- Issue not reproducible (we do see KeyRange
> count increasing upto few 100's)
> - 4.10 client/4.13 Server --- Issue reproducible as described above
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-4460) High GC / RS shutdown when we use select query with "IN" clause using 4.10 phoenix client on 4.13 phoenix server
[
https://issues.apache.org/jira/browse/PHOENIX-4460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291919#comment-16291919
]
Hadoop QA commented on PHOENIX-4460:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12902178/PHOENIX-4460-v2.patch
against master branch at commit 5cb02da74c15b0ae7c0fb4c880d60a2d1b6d18aa.
ATTACHMENT ID: 12902178
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.rpc.PhoenixServerRpcIT
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1666//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1666//artifact/patchprocess/patchReleaseAuditWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1666//console
This message is automatically generated.
> High GC / RS shutdown when we use select query with "IN" clause using 4.10
> phoenix client on 4.13 phoenix server
>
>
> Key: PHOENIX-4460
> URL: https://issues.apache.org/jira/browse/PHOENIX-4460
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Blocker
> Fix For: 4.14.0, 4.13.2
>
> Attachments: PHOENIX-4460-v2.patch, PHOENIX-4460.patch
>
>
> We were able to reproduce the High GC / RS shutdown / phoenix KeyRange query
> high object count issue on cluster today.
> Main observation is that this is reproducible when firing lots of query
> select from xyz where abc in (?, ?, ...) of this type with 4.10 phoenix
> client hitting 4.13 phoenix on HBase server side
> (4.10 client/4.10 server works fine, 4.13 client with 4.13 server works fine)
> We wrote a loader client (attached) with the below table/query , upserted
> ~100 million rows and fired the query in parallel using 4-5 loader clients
> with 16 threads each
> {code}
> TABLE: = "CREATE TABLE " + TABLE_NAME_TEMPLATE
> + " (\n" + " TestKey varchar(255) PRIMARY KEY, TestVal1 varchar(200),
> TestVal2 varchar(200), " + "TestValue varchar(1))";
> QUERY: = "SELECT * FROM " + TABLE_NAME_TEMPLATE + " WHERE TestKey IN (?, ?,
> ?, ?, ?, ?, ?, ?, ?, ?)"
> {code}
> After running this client immediately within a min or two we see the
> phoenix.query.KeyRange object count immediately going up to several lakhs and
> keeps on increasing continuously. This count doesn't seem to come down even
> after shutting down the clients
> {code}
> -bash-4.1$ ~/current/bigdata-util/tools/Linux/jdk/jdk1.8.0_102_x64/bin/jmap
> -histo:live 90725 | grep KeyRange
> 47:2748526596448 org.apache.phoenix.query.KeyRange
> 1851: 2 48 org.apache.phoenix.query.KeyRange$Bound
> 2434: 1 24 [Lorg.apache.phoenix.query.KeyRange$Bound;
> 3411: 1 16 org.apache.phoenix.query.KeyRange$1
> 3412: 1 16 org.apache.phoenix.query.KeyRange$2
> {code}
> After some time we also started seeing High GC issues and RegionServers
> crashing
> Experiment Summary:
> - 4.13 client/4.13 Server --- Issue not reproducible (we do see KeyRange
> count increasing upto few 100's)
> - 4.10 client/4.10 Server --- Issue not reproducible (we do see KeyRange
> count increasing upto few 100's)
> - 4.10 client/4.13 Server --- Issue reproducible as described above
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-4460) High GC / RS shutdown when we use select query with "IN" clause using 4.10 phoenix client on 4.13 phoenix server
[
https://issues.apache.org/jira/browse/PHOENIX-4460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291830#comment-16291830
]
Lars Hofhansl commented on PHOENIX-4460:
Yep. +1
> High GC / RS shutdown when we use select query with "IN" clause using 4.10
> phoenix client on 4.13 phoenix server
>
>
> Key: PHOENIX-4460
> URL: https://issues.apache.org/jira/browse/PHOENIX-4460
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Blocker
> Fix For: 4.14.0, 4.13.2
>
> Attachments: PHOENIX-4460-v2.patch, PHOENIX-4460.patch
>
>
> We were able to reproduce the High GC / RS shutdown / phoenix KeyRange query
> high object count issue on cluster today.
> Main observation is that this is reproducible when firing lots of query
> select from xyz where abc in (?, ?, ...) of this type with 4.10 phoenix
> client hitting 4.13 phoenix on HBase server side
> (4.10 client/4.10 server works fine, 4.13 client with 4.13 server works fine)
> We wrote a loader client (attached) with the below table/query , upserted
> ~100 million rows and fired the query in parallel using 4-5 loader clients
> with 16 threads each
> {code}
> TABLE: = "CREATE TABLE " + TABLE_NAME_TEMPLATE
> + " (\n" + " TestKey varchar(255) PRIMARY KEY, TestVal1 varchar(200),
> TestVal2 varchar(200), " + "TestValue varchar(1))";
> QUERY: = "SELECT * FROM " + TABLE_NAME_TEMPLATE + " WHERE TestKey IN (?, ?,
> ?, ?, ?, ?, ?, ?, ?, ?)"
> {code}
> After running this client immediately within a min or two we see the
> phoenix.query.KeyRange object count immediately going up to several lakhs and
> keeps on increasing continuously. This count doesn't seem to come down even
> after shutting down the clients
> {code}
> -bash-4.1$ ~/current/bigdata-util/tools/Linux/jdk/jdk1.8.0_102_x64/bin/jmap
> -histo:live 90725 | grep KeyRange
> 47:2748526596448 org.apache.phoenix.query.KeyRange
> 1851: 2 48 org.apache.phoenix.query.KeyRange$Bound
> 2434: 1 24 [Lorg.apache.phoenix.query.KeyRange$Bound;
> 3411: 1 16 org.apache.phoenix.query.KeyRange$1
> 3412: 1 16 org.apache.phoenix.query.KeyRange$2
> {code}
> After some time we also started seeing High GC issues and RegionServers
> crashing
> Experiment Summary:
> - 4.13 client/4.13 Server --- Issue not reproducible (we do see KeyRange
> count increasing upto few 100's)
> - 4.10 client/4.10 Server --- Issue not reproducible (we do see KeyRange
> count increasing upto few 100's)
> - 4.10 client/4.13 Server --- Issue reproducible as described above
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-4460) High GC / RS shutdown when we use select query with "IN" clause using 4.10 phoenix client on 4.13 phoenix server
[
https://issues.apache.org/jira/browse/PHOENIX-4460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291819#comment-16291819
]
Geoffrey Jacoby commented on PHOENIX-4460:
--
+1
> High GC / RS shutdown when we use select query with "IN" clause using 4.10
> phoenix client on 4.13 phoenix server
>
>
> Key: PHOENIX-4460
> URL: https://issues.apache.org/jira/browse/PHOENIX-4460
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Blocker
> Fix For: 4.14.0, 4.13.2
>
> Attachments: PHOENIX-4460-v2.patch, PHOENIX-4460.patch
>
>
> We were able to reproduce the High GC / RS shutdown / phoenix KeyRange query
> high object count issue on cluster today.
> Main observation is that this is reproducible when firing lots of query
> select from xyz where abc in (?, ?, ...) of this type with 4.10 phoenix
> client hitting 4.13 phoenix on HBase server side
> (4.10 client/4.10 server works fine, 4.13 client with 4.13 server works fine)
> We wrote a loader client (attached) with the below table/query , upserted
> ~100 million rows and fired the query in parallel using 4-5 loader clients
> with 16 threads each
> {code}
> TABLE: = "CREATE TABLE " + TABLE_NAME_TEMPLATE
> + " (\n" + " TestKey varchar(255) PRIMARY KEY, TestVal1 varchar(200),
> TestVal2 varchar(200), " + "TestValue varchar(1))";
> QUERY: = "SELECT * FROM " + TABLE_NAME_TEMPLATE + " WHERE TestKey IN (?, ?,
> ?, ?, ?, ?, ?, ?, ?, ?)"
> {code}
> After running this client immediately within a min or two we see the
> phoenix.query.KeyRange object count immediately going up to several lakhs and
> keeps on increasing continuously. This count doesn't seem to come down even
> after shutting down the clients
> {code}
> -bash-4.1$ ~/current/bigdata-util/tools/Linux/jdk/jdk1.8.0_102_x64/bin/jmap
> -histo:live 90725 | grep KeyRange
> 47:2748526596448 org.apache.phoenix.query.KeyRange
> 1851: 2 48 org.apache.phoenix.query.KeyRange$Bound
> 2434: 1 24 [Lorg.apache.phoenix.query.KeyRange$Bound;
> 3411: 1 16 org.apache.phoenix.query.KeyRange$1
> 3412: 1 16 org.apache.phoenix.query.KeyRange$2
> {code}
> After some time we also started seeing High GC issues and RegionServers
> crashing
> Experiment Summary:
> - 4.13 client/4.13 Server --- Issue not reproducible (we do see KeyRange
> count increasing upto few 100's)
> - 4.10 client/4.10 Server --- Issue not reproducible (we do see KeyRange
> count increasing upto few 100's)
> - 4.10 client/4.13 Server --- Issue reproducible as described above
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-4460) High GC / RS shutdown when we use select query with "IN" clause using 4.10 phoenix client on 4.13 phoenix server
[
https://issues.apache.org/jira/browse/PHOENIX-4460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291490#comment-16291490
]
Hadoop QA commented on PHOENIX-4460:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12902112/PHOENIX-4460.patch
against master branch at commit 5cb02da74c15b0ae7c0fb4c880d60a2d1b6d18aa.
ATTACHMENT ID: 12902112
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.AlterSessionIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.AggregateIT
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1665//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1665//artifact/patchprocess/patchReleaseAuditWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1665//console
This message is automatically generated.
> High GC / RS shutdown when we use select query with "IN" clause using 4.10
> phoenix client on 4.13 phoenix server
>
>
> Key: PHOENIX-4460
> URL: https://issues.apache.org/jira/browse/PHOENIX-4460
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Blocker
> Fix For: 4.14.0, 4.13.2
>
> Attachments: PHOENIX-4460.patch
>
>
> We were able to reproduce the High GC / RS shutdown / phoenix KeyRange query
> high object count issue on cluster today.
> Main observation is that this is reproducible when firing lots of query
> select from xyz where abc in (?, ?, ...) of this type with 4.10 phoenix
> client hitting 4.13 phoenix on HBase server side
> (4.10 client/4.10 server works fine, 4.13 client with 4.13 server works fine)
> We wrote a loader client (attached) with the below table/query , upserted
> ~100 million rows and fired the query in parallel using 4-5 loader clients
> with 16 threads each
> {code}
> TABLE: = "CREATE TABLE " + TABLE_NAME_TEMPLATE
> + " (\n" + " TestKey varchar(255) PRIMARY KEY, TestVal1 varchar(200),
> TestVal2 varchar(200), " + "TestValue varchar(1))";
> QUERY: = "SELECT * FROM " + TABLE_NAME_TEMPLATE + " WHERE TestKey IN (?, ?,
> ?, ?, ?, ?, ?, ?, ?, ?)"
> {code}
> After running this client immediately within a min or two we see the
> phoenix.query.KeyRange object count immediately going up to several lakhs and
> keeps on increasing continuously. This count doesn't seem to come down even
> after shutting down the clients
> {code}
> -bash-4.1$ ~/current/bigdata-util/tools/Linux/jdk/jdk1.8.0_102_x64/bin/jmap
> -histo:live 90725 | grep KeyRange
> 47:2748526596448 org.apache.phoenix.query.KeyRange
> 1851: 2 48 org.apache.phoenix.query.KeyRange$Bound
> 2434: 1 24 [Lorg.apache.phoenix.query.KeyRange$Bound;
> 3411: 1 16 org.apache.phoenix.query.KeyRange$1
> 3412: 1 16 org.apache.phoenix.query.KeyRange$2
> {code}
> After some time we also started seeing High GC issues and RegionServers
> crashing
> Experiment Summary:
> - 4.13 client/4.13 Server --- Issue not reproducible (we do see KeyRange
> count increasing upto few 100's)
> - 4.10 client/4.10 Server --- Issue not reproducible (we do see KeyRange
> count increasing upto few 100's)
> - 4.10 client/4.13 Server --- Issue reproducible as described above
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-4460) High GC / RS shutdown when we use select query with "IN" clause using 4.10 phoenix client on 4.13 phoenix server
[
https://issues.apache.org/jira/browse/PHOENIX-4460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291345#comment-16291345
]
Lars Hofhansl commented on PHOENIX-4460:
Can we capture different clients in a unittest? This might be a worthy thing to
add.
(If that needs a whole new plumbing then it's overkill)
> High GC / RS shutdown when we use select query with "IN" clause using 4.10
> phoenix client on 4.13 phoenix server
>
>
> Key: PHOENIX-4460
> URL: https://issues.apache.org/jira/browse/PHOENIX-4460
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Blocker
> Fix For: 4.14.0, 4.13.2
>
> Attachments: PHOENIX-4460.patch
>
>
> We were able to reproduce the High GC / RS shutdown / phoenix KeyRange query
> high object count issue on cluster today.
> Main observation is that this is reproducible when firing lots of query
> select from xyz where abc in (?, ?, ...) of this type with 4.10 phoenix
> client hitting 4.13 phoenix on HBase server side
> (4.10 client/4.10 server works fine, 4.13 client with 4.13 server works fine)
> We wrote a loader client (attached) with the below table/query , upserted
> ~100 million rows and fired the query in parallel using 4-5 loader clients
> with 16 threads each
> {code}
> TABLE: = "CREATE TABLE " + TABLE_NAME_TEMPLATE
> + " (\n" + " TestKey varchar(255) PRIMARY KEY, TestVal1 varchar(200),
> TestVal2 varchar(200), " + "TestValue varchar(1))";
> QUERY: = "SELECT * FROM " + TABLE_NAME_TEMPLATE + " WHERE TestKey IN (?, ?,
> ?, ?, ?, ?, ?, ?, ?, ?)"
> {code}
> After running this client immediately within a min or two we see the
> phoenix.query.KeyRange object count immediately going up to several lakhs and
> keeps on increasing continuously. This count doesn't seem to come down even
> after shutting down the clients
> {code}
> -bash-4.1$ ~/current/bigdata-util/tools/Linux/jdk/jdk1.8.0_102_x64/bin/jmap
> -histo:live 90725 | grep KeyRange
> 47:2748526596448 org.apache.phoenix.query.KeyRange
> 1851: 2 48 org.apache.phoenix.query.KeyRange$Bound
> 2434: 1 24 [Lorg.apache.phoenix.query.KeyRange$Bound;
> 3411: 1 16 org.apache.phoenix.query.KeyRange$1
> 3412: 1 16 org.apache.phoenix.query.KeyRange$2
> {code}
> After some time we also started seeing High GC issues and RegionServers
> crashing
> Experiment Summary:
> - 4.13 client/4.13 Server --- Issue not reproducible (we do see KeyRange
> count increasing upto few 100's)
> - 4.10 client/4.10 Server --- Issue not reproducible (we do see KeyRange
> count increasing upto few 100's)
> - 4.10 client/4.13 Server --- Issue reproducible as described above
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-4460) High GC / RS shutdown when we use select query with "IN" clause using 4.10 phoenix client on 4.13 phoenix server
[ https://issues.apache.org/jira/browse/PHOENIX-4460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291313#comment-16291313 ] James Taylor commented on PHOENIX-4460: --- Something must be holding onto the Listpreventing it from being GCed. This simply clears that when the filtering is done such that it should be GCed (and GCed earlier). Still need to figure out the *why* (only happens with old client/new server) and the *who* (is something holding on to the filter instance?), but focusing mainly on a quick solution right now. Let's see if it helps - looks like the test to repo is pretty straightforward. > High GC / RS shutdown when we use select query with "IN" clause using 4.10 > phoenix client on 4.13 phoenix server > > > Key: PHOENIX-4460 > URL: https://issues.apache.org/jira/browse/PHOENIX-4460 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Blocker > Fix For: 4.14.0, 4.13.2 > > Attachments: PHOENIX-4460.patch > > > We were able to reproduce the High GC / RS shutdown / phoenix KeyRange query > high object count issue on cluster today. > Main observation is that this is reproducible when firing lots of query > select from xyz where abc in (?, ?, ...) of this type with 4.10 phoenix > client hitting 4.13 phoenix on HBase server side > (4.10 client/4.10 server works fine, 4.13 client with 4.13 server works fine) > We wrote a loader client (attached) with the below table/query , upserted > ~100 million rows and fired the query in parallel using 4-5 loader clients > with 16 threads each > {code} > TABLE: = "CREATE TABLE " + TABLE_NAME_TEMPLATE > + " (\n" + " TestKey varchar(255) PRIMARY KEY, TestVal1 varchar(200), > TestVal2 varchar(200), " + "TestValue varchar(1))"; > QUERY: = "SELECT * FROM " + TABLE_NAME_TEMPLATE + " WHERE TestKey IN (?, ?, > ?, ?, ?, ?, ?, ?, ?, ?)" > {code} > After running this client immediately within a min or two we see the > phoenix.query.KeyRange object count immediately going up to several lakhs and > keeps on increasing continuously. This count doesn't seem to come down even > after shutting down the clients > {code} > -bash-4.1$ ~/current/bigdata-util/tools/Linux/jdk/jdk1.8.0_102_x64/bin/jmap > -histo:live 90725 | grep KeyRange > 47:2748526596448 org.apache.phoenix.query.KeyRange > 1851: 2 48 org.apache.phoenix.query.KeyRange$Bound > 2434: 1 24 [Lorg.apache.phoenix.query.KeyRange$Bound; > 3411: 1 16 org.apache.phoenix.query.KeyRange$1 > 3412: 1 16 org.apache.phoenix.query.KeyRange$2 > {code} > After some time we also started seeing High GC issues and RegionServers > crashing > Experiment Summary: > - 4.13 client/4.13 Server --- Issue not reproducible (we do see KeyRange > count increasing upto few 100's) > - 4.10 client/4.10 Server --- Issue not reproducible (we do see KeyRange > count increasing upto few 100's) > - 4.10 client/4.13 Server --- Issue reproducible as described above -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-4460) High GC / RS shutdown when we use select query with "IN" clause using 4.10 phoenix client on 4.13 phoenix server
[
https://issues.apache.org/jira/browse/PHOENIX-4460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291300#comment-16291300
]
Lars Hofhansl commented on PHOENIX-4460:
That's it? Can you explain? :)
> High GC / RS shutdown when we use select query with "IN" clause using 4.10
> phoenix client on 4.13 phoenix server
>
>
> Key: PHOENIX-4460
> URL: https://issues.apache.org/jira/browse/PHOENIX-4460
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Blocker
> Fix For: 4.14.0, 4.13.2
>
> Attachments: PHOENIX-4460.patch
>
>
> We were able to reproduce the High GC / RS shutdown / phoenix KeyRange query
> high object count issue on cluster today.
> Main observation is that this is reproducible when firing lots of query
> select from xyz where abc in (?, ?, ...) of this type with 4.10 phoenix
> client hitting 4.13 phoenix on HBase server side
> (4.10 client/4.10 server works fine, 4.13 client with 4.13 server works fine)
> We wrote a loader client (attached) with the below table/query , upserted
> ~100 million rows and fired the query in parallel using 4-5 loader clients
> with 16 threads each
> {code}
> TABLE: = "CREATE TABLE " + TABLE_NAME_TEMPLATE
> + " (\n" + " TestKey varchar(255) PRIMARY KEY, TestVal1 varchar(200),
> TestVal2 varchar(200), " + "TestValue varchar(1))";
> QUERY: = "SELECT * FROM " + TABLE_NAME_TEMPLATE + " WHERE TestKey IN (?, ?,
> ?, ?, ?, ?, ?, ?, ?, ?)"
> {code}
> After running this client immediately within a min or two we see the
> phoenix.query.KeyRange object count immediately going up to several lakhs and
> keeps on increasing continuously. This count doesn't seem to come down even
> after shutting down the clients
> {code}
> -bash-4.1$ ~/current/bigdata-util/tools/Linux/jdk/jdk1.8.0_102_x64/bin/jmap
> -histo:live 90725 | grep KeyRange
> 47:2748526596448 org.apache.phoenix.query.KeyRange
> 1851: 2 48 org.apache.phoenix.query.KeyRange$Bound
> 2434: 1 24 [Lorg.apache.phoenix.query.KeyRange$Bound;
> 3411: 1 16 org.apache.phoenix.query.KeyRange$1
> 3412: 1 16 org.apache.phoenix.query.KeyRange$2
> {code}
> After some time we also started seeing High GC issues and RegionServers
> crashing
> Experiment Summary:
> - 4.13 client/4.13 Server --- Issue not reproducible (we do see KeyRange
> count increasing upto few 100's)
> - 4.10 client/4.10 Server --- Issue not reproducible (we do see KeyRange
> count increasing upto few 100's)
> - 4.10 client/4.13 Server --- Issue reproducible as described above
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
