[jira] [Commented] (HBASE-25277) postScannerFilterRow impacts Scan performance a lot in HBase 2.x
[
https://issues.apache.org/jira/browse/HBASE-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265634#comment-17265634
]
Hudson commented on HBASE-25277:
Results for branch branch-2.4
[build #32 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/32/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/32/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/32/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/32/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/32/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> postScannerFilterRow impacts Scan performance a lot in HBase 2.x
>
>
> Key: HBASE-25277
> URL: https://issues.apache.org/jira/browse/HBASE-25277
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, scan
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Labels: perfomance, scanning
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.3.4, 2.5.0, 2.4.1
>
> Attachments: test_report.png
>
>
> In our test we observed Scan performance is degraded by more than 60% in
> HBase-2.x as compared to 1.3.x. As per the flamegraph report, RS spent 31%
> of the time in postScannerFilterRow, however the coprocessors
> (AccessController, VisibilityController & ConstraintProcessor) does nothing
> in that hook.
> HBASE-14489 added the logic to avoid the call to postScannerFilterRow when
> not needed which is not working as expected in HBase 2.x. AccessController,
> VisibilityController & ConstraintProcessor override the postScannerFilterRow
> with dummy (same as RegionObserver) implementation, so
> RegionCoprocessorHost.hasCustomPostScannerFilterRow will be TRUE and call the
> hook for all configured CPs while processing each row . Suppose we have
> configured 5 region CPs and there are 1 M rows in table, then there will be 1
> * 5 M dummy call to postScannerFilterRow during whole table scan.
> We need to remove postScannerFilterRow hook from these CPs as these are not
> doing anything.
> Another problem is in RegionCoprocessorHost.hasCustomPostScannerFilterRow
> init logic, currently it is always TRUE even though we remove
> postScannerFilterRow hook from AccessController, VisibilityController &
> ConstraintProcessor, because we are finding postScannerFilterRow until it
> is found (we look in configured CP's super class also) or clazz is NULL.
> https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L301
> Supper class of Object (super class of AccessController) will be NULL, so
> RegionCoprocessorHost.hasCustomPostScannerFilterRow will be set as TRUE
> https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L279
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25277) postScannerFilterRow impacts Scan performance a lot in HBase 2.x
[
https://issues.apache.org/jira/browse/HBASE-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17253726#comment-17253726
]
Hudson commented on HBASE-25277:
Results for branch branch-2.2
[build #140 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/140/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/140//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/140//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/140//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/140//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> postScannerFilterRow impacts Scan performance a lot in HBase 2.x
>
>
> Key: HBASE-25277
> URL: https://issues.apache.org/jira/browse/HBASE-25277
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, scan
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Labels: perfomance, scanning
> Fix For: 3.0.0-alpha-1, 2.4.0, 2.2.7, 2.3.4, 2.5.0
>
> Attachments: test_report.png
>
>
> In our test we observed Scan performance is degraded by more than 60% in
> HBase-2.x as compared to 1.3.x. As per the flamegraph report, RS spent 31%
> of the time in postScannerFilterRow, however the coprocessors
> (AccessController, VisibilityController & ConstraintProcessor) does nothing
> in that hook.
> HBASE-14489 added the logic to avoid the call to postScannerFilterRow when
> not needed which is not working as expected in HBase 2.x. AccessController,
> VisibilityController & ConstraintProcessor override the postScannerFilterRow
> with dummy (same as RegionObserver) implementation, so
> RegionCoprocessorHost.hasCustomPostScannerFilterRow will be TRUE and call the
> hook for all configured CPs while processing each row . Suppose we have
> configured 5 region CPs and there are 1 M rows in table, then there will be 1
> * 5 M dummy call to postScannerFilterRow during whole table scan.
> We need to remove postScannerFilterRow hook from these CPs as these are not
> doing anything.
> Another problem is in RegionCoprocessorHost.hasCustomPostScannerFilterRow
> init logic, currently it is always TRUE even though we remove
> postScannerFilterRow hook from AccessController, VisibilityController &
> ConstraintProcessor, because we are finding postScannerFilterRow until it
> is found (we look in configured CP's super class also) or clazz is NULL.
> https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L301
> Supper class of Object (super class of AccessController) will be NULL, so
> RegionCoprocessorHost.hasCustomPostScannerFilterRow will be set as TRUE
> https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L279
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25277) postScannerFilterRow impacts Scan performance a lot in HBase 2.x
[ https://issues.apache.org/jira/browse/HBASE-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17253265#comment-17253265 ] Pankaj Kumar commented on HBASE-25277: -- Thanks [~zghao]. > postScannerFilterRow impacts Scan performance a lot in HBase 2.x > > > Key: HBASE-25277 > URL: https://issues.apache.org/jira/browse/HBASE-25277 > Project: HBase > Issue Type: Bug > Components: Coprocessors, scan >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Critical > Labels: perfomance, scanning > Fix For: 3.0.0-alpha-1, 2.4.0, 2.2.7, 2.3.4, 2.5.0 > > Attachments: test_report.png > > > In our test we observed Scan performance is degraded by more than 60% in > HBase-2.x as compared to 1.3.x. As per the flamegraph report, RS spent 31% > of the time in postScannerFilterRow, however the coprocessors > (AccessController, VisibilityController & ConstraintProcessor) does nothing > in that hook. > HBASE-14489 added the logic to avoid the call to postScannerFilterRow when > not needed which is not working as expected in HBase 2.x. AccessController, > VisibilityController & ConstraintProcessor override the postScannerFilterRow > with dummy (same as RegionObserver) implementation, so > RegionCoprocessorHost.hasCustomPostScannerFilterRow will be TRUE and call the > hook for all configured CPs while processing each row . Suppose we have > configured 5 region CPs and there are 1 M rows in table, then there will be 1 > * 5 M dummy call to postScannerFilterRow during whole table scan. > We need to remove postScannerFilterRow hook from these CPs as these are not > doing anything. > Another problem is in RegionCoprocessorHost.hasCustomPostScannerFilterRow > init logic, currently it is always TRUE even though we remove > postScannerFilterRow hook from AccessController, VisibilityController & > ConstraintProcessor, because we are finding postScannerFilterRow until it > is found (we look in configured CP's super class also) or clazz is NULL. > https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L301 > Supper class of Object (super class of AccessController) will be NULL, so > RegionCoprocessorHost.hasCustomPostScannerFilterRow will be set as TRUE > https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25277) postScannerFilterRow impacts Scan performance a lot in HBase 2.x
[ https://issues.apache.org/jira/browse/HBASE-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17253189#comment-17253189 ] Guanghao Zhang commented on HBASE-25277: Merged to branch-2.2 too. And set 2.2.7 as one of the resolved version. > postScannerFilterRow impacts Scan performance a lot in HBase 2.x > > > Key: HBASE-25277 > URL: https://issues.apache.org/jira/browse/HBASE-25277 > Project: HBase > Issue Type: Bug > Components: Coprocessors, scan >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Critical > Labels: perfomance, scanning > Fix For: 3.0.0-alpha-1, 2.4.0, 2.2.7, 2.3.4, 2.5.0 > > Attachments: test_report.png > > > In our test we observed Scan performance is degraded by more than 60% in > HBase-2.x as compared to 1.3.x. As per the flamegraph report, RS spent 31% > of the time in postScannerFilterRow, however the coprocessors > (AccessController, VisibilityController & ConstraintProcessor) does nothing > in that hook. > HBASE-14489 added the logic to avoid the call to postScannerFilterRow when > not needed which is not working as expected in HBase 2.x. AccessController, > VisibilityController & ConstraintProcessor override the postScannerFilterRow > with dummy (same as RegionObserver) implementation, so > RegionCoprocessorHost.hasCustomPostScannerFilterRow will be TRUE and call the > hook for all configured CPs while processing each row . Suppose we have > configured 5 region CPs and there are 1 M rows in table, then there will be 1 > * 5 M dummy call to postScannerFilterRow during whole table scan. > We need to remove postScannerFilterRow hook from these CPs as these are not > doing anything. > Another problem is in RegionCoprocessorHost.hasCustomPostScannerFilterRow > init logic, currently it is always TRUE even though we remove > postScannerFilterRow hook from AccessController, VisibilityController & > ConstraintProcessor, because we are finding postScannerFilterRow until it > is found (we look in configured CP's super class also) or clazz is NULL. > https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L301 > Supper class of Object (super class of AccessController) will be NULL, so > RegionCoprocessorHost.hasCustomPostScannerFilterRow will be set as TRUE > https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25277) postScannerFilterRow impacts Scan performance a lot in HBase 2.x
[
https://issues.apache.org/jira/browse/HBASE-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248274#comment-17248274
]
Hudson commented on HBASE-25277:
Results for branch branch-2.4
[build #7 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/7/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/7/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/7/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/7/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/7/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> postScannerFilterRow impacts Scan performance a lot in HBase 2.x
>
>
> Key: HBASE-25277
> URL: https://issues.apache.org/jira/browse/HBASE-25277
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, scan
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Labels: perfomance, scanning
> Fix For: 3.0.0-alpha-1, 2.5.0
>
> Attachments: test_report.png
>
>
> In our test we observed Scan performance is degraded by more than 60% in
> HBase-2.x as compared to 1.3.x. As per the flamegraph report, RS spent 31%
> of the time in postScannerFilterRow, however the coprocessors
> (AccessController, VisibilityController & ConstraintProcessor) does nothing
> in that hook.
> HBASE-14489 added the logic to avoid the call to postScannerFilterRow when
> not needed which is not working as expected in HBase 2.x. AccessController,
> VisibilityController & ConstraintProcessor override the postScannerFilterRow
> with dummy (same as RegionObserver) implementation, so
> RegionCoprocessorHost.hasCustomPostScannerFilterRow will be TRUE and call the
> hook for all configured CPs while processing each row . Suppose we have
> configured 5 region CPs and there are 1 M rows in table, then there will be 1
> * 5 M dummy call to postScannerFilterRow during whole table scan.
> We need to remove postScannerFilterRow hook from these CPs as these are not
> doing anything.
> Another problem is in RegionCoprocessorHost.hasCustomPostScannerFilterRow
> init logic, currently it is always TRUE even though we remove
> postScannerFilterRow hook from AccessController, VisibilityController &
> ConstraintProcessor, because we are finding postScannerFilterRow until it
> is found (we look in configured CP's super class also) or clazz is NULL.
> https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L301
> Supper class of Object (super class of AccessController) will be NULL, so
> RegionCoprocessorHost.hasCustomPostScannerFilterRow will be set as TRUE
> https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L279
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25277) postScannerFilterRow impacts Scan performance a lot in HBase 2.x
[
https://issues.apache.org/jira/browse/HBASE-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248044#comment-17248044
]
Hudson commented on HBASE-25277:
Results for branch branch-2.3
[build #123 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/123/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/123/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/123/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/123/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/123/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> postScannerFilterRow impacts Scan performance a lot in HBase 2.x
>
>
> Key: HBASE-25277
> URL: https://issues.apache.org/jira/browse/HBASE-25277
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, scan
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Labels: perfomance, scanning
> Fix For: 3.0.0-alpha-1, 2.5.0
>
> Attachments: test_report.png
>
>
> In our test we observed Scan performance is degraded by more than 60% in
> HBase-2.x as compared to 1.3.x. As per the flamegraph report, RS spent 31%
> of the time in postScannerFilterRow, however the coprocessors
> (AccessController, VisibilityController & ConstraintProcessor) does nothing
> in that hook.
> HBASE-14489 added the logic to avoid the call to postScannerFilterRow when
> not needed which is not working as expected in HBase 2.x. AccessController,
> VisibilityController & ConstraintProcessor override the postScannerFilterRow
> with dummy (same as RegionObserver) implementation, so
> RegionCoprocessorHost.hasCustomPostScannerFilterRow will be TRUE and call the
> hook for all configured CPs while processing each row . Suppose we have
> configured 5 region CPs and there are 1 M rows in table, then there will be 1
> * 5 M dummy call to postScannerFilterRow during whole table scan.
> We need to remove postScannerFilterRow hook from these CPs as these are not
> doing anything.
> Another problem is in RegionCoprocessorHost.hasCustomPostScannerFilterRow
> init logic, currently it is always TRUE even though we remove
> postScannerFilterRow hook from AccessController, VisibilityController &
> ConstraintProcessor, because we are finding postScannerFilterRow until it
> is found (we look in configured CP's super class also) or clazz is NULL.
> https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L301
> Supper class of Object (super class of AccessController) will be NULL, so
> RegionCoprocessorHost.hasCustomPostScannerFilterRow will be set as TRUE
> https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L279
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25277) postScannerFilterRow impacts Scan performance a lot in HBase 2.x
[ https://issues.apache.org/jira/browse/HBASE-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246745#comment-17246745 ] Andrew Kyle Purtell commented on HBASE-25277: - +1 Please merge to all branch-2.x. This is a perf regression / bug fix that does not change API semantics if a CP implements those hooks. > postScannerFilterRow impacts Scan performance a lot in HBase 2.x > > > Key: HBASE-25277 > URL: https://issues.apache.org/jira/browse/HBASE-25277 > Project: HBase > Issue Type: Bug > Components: Coprocessors, scan >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Critical > Labels: perfomance, scanning > Fix For: 3.0.0-alpha-1, 2.5.0 > > Attachments: test_report.png > > > In our test we observed Scan performance is degraded by more than 60% in > HBase-2.x as compared to 1.3.x. As per the flamegraph report, RS spent 31% > of the time in postScannerFilterRow, however the coprocessors > (AccessController, VisibilityController & ConstraintProcessor) does nothing > in that hook. > HBASE-14489 added the logic to avoid the call to postScannerFilterRow when > not needed which is not working as expected in HBase 2.x. AccessController, > VisibilityController & ConstraintProcessor override the postScannerFilterRow > with dummy (same as RegionObserver) implementation, so > RegionCoprocessorHost.hasCustomPostScannerFilterRow will be TRUE and call the > hook for all configured CPs while processing each row . Suppose we have > configured 5 region CPs and there are 1 M rows in table, then there will be 1 > * 5 M dummy call to postScannerFilterRow during whole table scan. > We need to remove postScannerFilterRow hook from these CPs as these are not > doing anything. > Another problem is in RegionCoprocessorHost.hasCustomPostScannerFilterRow > init logic, currently it is always TRUE even though we remove > postScannerFilterRow hook from AccessController, VisibilityController & > ConstraintProcessor, because we are finding postScannerFilterRow until it > is found (we look in configured CP's super class also) or clazz is NULL. > https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L301 > Supper class of Object (super class of AccessController) will be NULL, so > RegionCoprocessorHost.hasCustomPostScannerFilterRow will be set as TRUE > https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25277) postScannerFilterRow impacts Scan performance a lot in HBase 2.x
[
https://issues.apache.org/jira/browse/HBASE-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246730#comment-17246730
]
Hudson commented on HBASE-25277:
Results for branch branch-2
[build #125 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/125/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/125/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/125/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/125/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/125/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> postScannerFilterRow impacts Scan performance a lot in HBase 2.x
>
>
> Key: HBASE-25277
> URL: https://issues.apache.org/jira/browse/HBASE-25277
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, scan
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Labels: perfomance, scanning
> Fix For: 3.0.0-alpha-1, 2.5.0
>
> Attachments: test_report.png
>
>
> In our test we observed Scan performance is degraded by more than 60% in
> HBase-2.x as compared to 1.3.x. As per the flamegraph report, RS spent 31%
> of the time in postScannerFilterRow, however the coprocessors
> (AccessController, VisibilityController & ConstraintProcessor) does nothing
> in that hook.
> HBASE-14489 added the logic to avoid the call to postScannerFilterRow when
> not needed which is not working as expected in HBase 2.x. AccessController,
> VisibilityController & ConstraintProcessor override the postScannerFilterRow
> with dummy (same as RegionObserver) implementation, so
> RegionCoprocessorHost.hasCustomPostScannerFilterRow will be TRUE and call the
> hook for all configured CPs while processing each row . Suppose we have
> configured 5 region CPs and there are 1 M rows in table, then there will be 1
> * 5 M dummy call to postScannerFilterRow during whole table scan.
> We need to remove postScannerFilterRow hook from these CPs as these are not
> doing anything.
> Another problem is in RegionCoprocessorHost.hasCustomPostScannerFilterRow
> init logic, currently it is always TRUE even though we remove
> postScannerFilterRow hook from AccessController, VisibilityController &
> ConstraintProcessor, because we are finding postScannerFilterRow until it
> is found (we look in configured CP's super class also) or clazz is NULL.
> https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L301
> Supper class of Object (super class of AccessController) will be NULL, so
> RegionCoprocessorHost.hasCustomPostScannerFilterRow will be set as TRUE
> https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L279
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25277) postScannerFilterRow impacts Scan performance a lot in HBase 2.x
[ https://issues.apache.org/jira/browse/HBASE-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246426#comment-17246426 ] Pankaj Kumar commented on HBASE-25277: -- Ping [~zghao], [~ndimiduk] & [~andrew.purt...@gmail.com] ... Shall we merge this in 2.2, 2.3 & 2.4 branches? > postScannerFilterRow impacts Scan performance a lot in HBase 2.x > > > Key: HBASE-25277 > URL: https://issues.apache.org/jira/browse/HBASE-25277 > Project: HBase > Issue Type: Bug > Components: Coprocessors, scan >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Critical > Labels: perfomance, scanning > Fix For: 3.0.0-alpha-1, 2.5.0 > > Attachments: test_report.png > > > In our test we observed Scan performance is degraded by more than 60% in > HBase-2.x as compared to 1.3.x. As per the flamegraph report, RS spent 31% > of the time in postScannerFilterRow, however the coprocessors > (AccessController, VisibilityController & ConstraintProcessor) does nothing > in that hook. > HBASE-14489 added the logic to avoid the call to postScannerFilterRow when > not needed which is not working as expected in HBase 2.x. AccessController, > VisibilityController & ConstraintProcessor override the postScannerFilterRow > with dummy (same as RegionObserver) implementation, so > RegionCoprocessorHost.hasCustomPostScannerFilterRow will be TRUE and call the > hook for all configured CPs while processing each row . Suppose we have > configured 5 region CPs and there are 1 M rows in table, then there will be 1 > * 5 M dummy call to postScannerFilterRow during whole table scan. > We need to remove postScannerFilterRow hook from these CPs as these are not > doing anything. > Another problem is in RegionCoprocessorHost.hasCustomPostScannerFilterRow > init logic, currently it is always TRUE even though we remove > postScannerFilterRow hook from AccessController, VisibilityController & > ConstraintProcessor, because we are finding postScannerFilterRow until it > is found (we look in configured CP's super class also) or clazz is NULL. > https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L301 > Supper class of Object (super class of AccessController) will be NULL, so > RegionCoprocessorHost.hasCustomPostScannerFilterRow will be set as TRUE > https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25277) postScannerFilterRow impacts Scan performance a lot in HBase 2.x
[ https://issues.apache.org/jira/browse/HBASE-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246289#comment-17246289 ] Pankaj Kumar commented on HBASE-25277: -- Thanks for the review [~ram_krish], [~anoop.hbase] & [~zhangduo]. Pushed the changes to master & branch-2. > postScannerFilterRow impacts Scan performance a lot in HBase 2.x > > > Key: HBASE-25277 > URL: https://issues.apache.org/jira/browse/HBASE-25277 > Project: HBase > Issue Type: Bug > Components: Coprocessors, scan >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Critical > Labels: perfomance, scanning > Fix For: 3.0.0-alpha-1, 2.5.0 > > Attachments: test_report.png > > > In our test we observed Scan performance is degraded by more than 60% in > HBase-2.x as compared to 1.3.x. As per the flamegraph report, RS spent 31% > of the time in postScannerFilterRow, however the coprocessors > (AccessController, VisibilityController & ConstraintProcessor) does nothing > in that hook. > HBASE-14489 added the logic to avoid the call to postScannerFilterRow when > not needed which is not working as expected in HBase 2.x. AccessController, > VisibilityController & ConstraintProcessor override the postScannerFilterRow > with dummy (same as RegionObserver) implementation, so > RegionCoprocessorHost.hasCustomPostScannerFilterRow will be TRUE and call the > hook for all configured CPs while processing each row . Suppose we have > configured 5 region CPs and there are 1 M rows in table, then there will be 1 > * 5 M dummy call to postScannerFilterRow during whole table scan. > We need to remove postScannerFilterRow hook from these CPs as these are not > doing anything. > Another problem is in RegionCoprocessorHost.hasCustomPostScannerFilterRow > init logic, currently it is always TRUE even though we remove > postScannerFilterRow hook from AccessController, VisibilityController & > ConstraintProcessor, because we are finding postScannerFilterRow until it > is found (we look in configured CP's super class also) or clazz is NULL. > https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L301 > Supper class of Object (super class of AccessController) will be NULL, so > RegionCoprocessorHost.hasCustomPostScannerFilterRow will be set as TRUE > https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25277) postScannerFilterRow impacts Scan performance a lot in HBase 2.x
[
https://issues.apache.org/jira/browse/HBASE-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17245878#comment-17245878
]
Hudson commented on HBASE-25277:
Results for branch master
[build #152 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/152/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/152/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/152/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/152/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> postScannerFilterRow impacts Scan performance a lot in HBase 2.x
>
>
> Key: HBASE-25277
> URL: https://issues.apache.org/jira/browse/HBASE-25277
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, scan
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Labels: perfomance, scanning
> Fix For: 3.0.0-alpha-1
>
> Attachments: test_report.png
>
>
> In our test we observed Scan performance is degraded by more than 60% in
> HBase-2.x as compared to 1.3.x. As per the flamegraph report, RS spent 31%
> of the time in postScannerFilterRow, however the coprocessors
> (AccessController, VisibilityController & ConstraintProcessor) does nothing
> in that hook.
> HBASE-14489 added the logic to avoid the call to postScannerFilterRow when
> not needed which is not working as expected in HBase 2.x. AccessController,
> VisibilityController & ConstraintProcessor override the postScannerFilterRow
> with dummy (same as RegionObserver) implementation, so
> RegionCoprocessorHost.hasCustomPostScannerFilterRow will be TRUE and call the
> hook for all configured CPs while processing each row . Suppose we have
> configured 5 region CPs and there are 1 M rows in table, then there will be 1
> * 5 M dummy call to postScannerFilterRow during whole table scan.
> We need to remove postScannerFilterRow hook from these CPs as these are not
> doing anything.
> Another problem is in RegionCoprocessorHost.hasCustomPostScannerFilterRow
> init logic, currently it is always TRUE even though we remove
> postScannerFilterRow hook from AccessController, VisibilityController &
> ConstraintProcessor, because we are finding postScannerFilterRow until it
> is found (we look in configured CP's super class also) or clazz is NULL.
> https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L301
> Supper class of Object (super class of AccessController) will be NULL, so
> RegionCoprocessorHost.hasCustomPostScannerFilterRow will be set as TRUE
> https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L279
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25277) postScannerFilterRow impacts Scan performance a lot in HBase 2.x
[ https://issues.apache.org/jira/browse/HBASE-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238003#comment-17238003 ] Pankaj Kumar commented on HBASE-25277: -- Env Detail: 1 HM, 5 RS Test Input: 1 Table with 500 region No of records: 10135632 (10 M) Data Size: 11.98 GB Prerequisite: Scan whole table and make sure whole data is in L1 Test Scenario: Full table scan with SCVF *Test Report:* *!test_report.png!* > postScannerFilterRow impacts Scan performance a lot in HBase 2.x > > > Key: HBASE-25277 > URL: https://issues.apache.org/jira/browse/HBASE-25277 > Project: HBase > Issue Type: Bug > Components: Coprocessors, scan >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Critical > Fix For: 3.0.0-alpha-1 > > Attachments: test_report.png > > > In our test we observed Scan performance is degraded by more than 60% in > HBase-2.x as compared to 1.3.x. As per the flamegraph report, RS spent 31% > of the time in postScannerFilterRow, however the coprocessors > (AccessController, VisibilityController & ConstraintProcessor) does nothing > in that hook. > HBASE-14489 added the logic to avoid the call to postScannerFilterRow when > not needed which is not working as expected in HBase 2.x. AccessController, > VisibilityController & ConstraintProcessor override the postScannerFilterRow > with dummy (same as RegionObserver) implementation, so > RegionCoprocessorHost.hasCustomPostScannerFilterRow will be TRUE and call the > hook for all configured CPs while processing each row . Suppose we have > configured 5 region CPs and there are 1 M rows in table, then there will be 1 > * 5 M dummy call to postScannerFilterRow during whole table scan. > We need to remove postScannerFilterRow hook from these CPs as these are not > doing anything. > Another problem is in RegionCoprocessorHost.hasCustomPostScannerFilterRow > init logic, currently it is always TRUE even though we remove > postScannerFilterRow hook from AccessController, VisibilityController & > ConstraintProcessor, because we are finding postScannerFilterRow until it > is found (we look in configured CP's super class also) or clazz is NULL. > https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L301 > Supper class of Object (super class of AccessController) will be NULL, so > RegionCoprocessorHost.hasCustomPostScannerFilterRow will be set as TRUE > https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25277) postScannerFilterRow impacts Scan performance a lot in HBase 2.x
[ https://issues.apache.org/jira/browse/HBASE-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238002#comment-17238002 ] Pankaj Kumar commented on HBASE-25277: -- Env Detail: 1 HM, 5 RS Test Input: 1 Table with 500 region No of records: 10135632 (10 M) Data Size: 11.98 GB Prerequisite: Scan whole table and make sure whole data is in L1 Test Scenario: Full table scan with SCVF *Test Report:* !Screenshot from 2020-11-24 15-23-29.png! > postScannerFilterRow impacts Scan performance a lot in HBase 2.x > > > Key: HBASE-25277 > URL: https://issues.apache.org/jira/browse/HBASE-25277 > Project: HBase > Issue Type: Bug > Components: Coprocessors, scan >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > In our test we observed Scan performance is degraded by more than 60% in > HBase-2.x as compared to 1.3.x. As per the flamegraph report, RS spent 31% > of the time in postScannerFilterRow, however the coprocessors > (AccessController, VisibilityController & ConstraintProcessor) does nothing > in that hook. > HBASE-14489 added the logic to avoid the call to postScannerFilterRow when > not needed which is not working as expected in HBase 2.x. AccessController, > VisibilityController & ConstraintProcessor override the postScannerFilterRow > with dummy (same as RegionObserver) implementation, so > RegionCoprocessorHost.hasCustomPostScannerFilterRow will be TRUE and call the > hook for all configured CPs while processing each row . Suppose we have > configured 5 region CPs and there are 1 M rows in table, then there will be 1 > * 5 M dummy call to postScannerFilterRow during whole table scan. > We need to remove postScannerFilterRow hook from these CPs as these are not > doing anything. > Another problem is in RegionCoprocessorHost.hasCustomPostScannerFilterRow > init logic, currently it is always TRUE even though we remove > postScannerFilterRow hook from AccessController, VisibilityController & > ConstraintProcessor, because we are finding postScannerFilterRow until it > is found (we look in configured CP's super class also) or clazz is NULL. > https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L301 > Supper class of Object (super class of AccessController) will be NULL, so > RegionCoprocessorHost.hasCustomPostScannerFilterRow will be set as TRUE > https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
