[GitHub] [hbase] bharathv commented on a change in pull request #1821: HBASE-24480: Deflake TestRSGroupsBasics#testClearDeadServers
bharathv commented on a change in pull request #1821:
URL: https://github.com/apache/hbase/pull/1821#discussion_r433054824
##
File path:
hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
##
@@ -385,7 +386,7 @@ public RSGroupInfoManager getRSGroupInfoManager() throws
IOException {
}
@Override
- public void removeServers(Set servers) throws IOException {
+ public void removeServers(@Nullable Set servers) throws IOException
{
Review comment:
Fair point, it does look awkward when very few places in the codebase
use that. Removed it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bharathv merged pull request #1820: HBASE-24479: Deflake TestCompaction#testStopStartCompaction
bharathv merged pull request #1820: URL: https://github.com/apache/hbase/pull/1820 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1801: [HBASE-24441]CacheConfig details logged at Store open is not really u…
Apache-HBase commented on pull request #1801: URL: https://github.com/apache/hbase/pull/1801#issuecomment-636623642 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 40s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 18s | master passed | | +1 :green_heart: | checkstyle | 1m 17s | master passed | | +1 :green_heart: | spotbugs | 2m 13s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 55s | the patch passed | | +1 :green_heart: | checkstyle | 1m 16s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 42s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 18s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 38m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1801/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1801 | | JIRA Issue | HBASE-24441 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 641558d33d7e 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4f49a96258 | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1801/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1801: [HBASE-24441]CacheConfig details logged at Store open is not really u…
Apache-HBase commented on pull request #1801: URL: https://github.com/apache/hbase/pull/1801#issuecomment-636622847 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 40s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 5s | master passed | | +1 :green_heart: | compile | 1m 1s | master passed | | +1 :green_heart: | shadedjars | 6m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 56s | the patch passed | | +1 :green_heart: | compile | 1m 1s | the patch passed | | +1 :green_heart: | javac | 1m 1s | the patch passed | | +1 :green_heart: | shadedjars | 6m 10s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 9m 9s | hbase-server in the patch failed. | | | | 35m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1801/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1801 | | JIRA Issue | HBASE-24441 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ee26448416bf 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4f49a96258 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1801/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1801/4/testReport/ | | Max. process+thread count | 640 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1801/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24482) [hbase-operator-tools] build of hbck2 fails with HBase branch-2.3, due to missing edu.umd.cs.findbugs.annotations dependency
[
https://issues.apache.org/jira/browse/HBASE-24482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120761#comment-17120761
]
Matthew Foley commented on HBASE-24482:
---
Interesting, grepping the branch-2.3 of HBase, I see the poms do _not_ mention
edu.umd.cs.findbugs.annotations, and _do_ mention javax.annotation. BUT there
are still scads of uses of edu.umd.cs.findbugs.annotations in java files. The
tag 2.1.6RC1 has the same behavior. So I infer that the
edu.umd.cs.findbugs.annotations package was being pulled in by an indirect
dependency, and either an assembly or a shadowing somewhere along the line
changed since 2.1.6.
> [hbase-operator-tools] build of hbck2 fails with HBase branch-2.3, due to
> missing edu.umd.cs.findbugs.annotations dependency
>
>
> Key: HBASE-24482
> URL: https://issues.apache.org/jira/browse/HBASE-24482
> Project: HBase
> Issue Type: Bug
> Components: hbase-operator-tools
>Affects Versions: hbase-operator-tools-1.1.0
>Reporter: Matthew Foley
>Priority: Major
> Attachments: HBASE-24482_compile_error.log
>
>
> If we do a local build of the current HBase branch-2.3 as version
> 2.3.0-SNAPSHOT, and 'mvn install' it in the local maven repository,
> then attempt to build hbase-operator-tools master branch, with
> -Dhbase.version=2.3.0-SNAPSHOT,
> the HBCK2 class file HBCKMetaTableAccessor.java (line 25) fails to compile
> with error: "package edu.umd.cs.findbugs.annotations does not exist",
> followed by multiple instances of missing "symbol: class Nullable". (A
> longer log extract is attached.)
> It would appear the default HBase version 2.1.6 made this dependency package
> available in some manner, but the newer HBase does not.
> If the @Nullable annotation is still needed, it may be sufficient to simply
> add it to the hbase-operator-tools pom.xml file, so maven will load it.
> However, it is actually deprecated in the current documentation for
> edu.umd.cs.findbugs.annotations. Various commentary online suggest that we
> should be using the similar jsr305.jar (*{{javax.annotation)}}* instead; see
> eg
> *
> [https://stackoverflow.com/questions/4963300/which-notnull-java-annotation-should-i-use]
> Does anyone know if that was HBase's approach?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24482) [hbase-operator-tools] build of hbck2 fails with HBase branch-2.3, due to missing edu.umd.cs.findbugs.annotations dependency
[
https://issues.apache.org/jira/browse/HBASE-24482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthew Foley updated HBASE-24482:
--
Description:
If we do a local build of the current HBase branch-2.3 as version
2.3.0-SNAPSHOT, and 'mvn install' it in the local maven repository,
then attempt to build hbase-operator-tools master branch, with
-Dhbase.version=2.3.0-SNAPSHOT,
the HBCK2 class file HBCKMetaTableAccessor.java (line 25) fails to compile with
error: "package edu.umd.cs.findbugs.annotations does not exist", followed by
multiple instances of missing "symbol: class Nullable". (A longer log extract
is attached.)
It would appear the default HBase version 2.1.6 made this dependency package
available in some manner, but the newer HBase does not.
If the @Nullable annotation is still needed, it may be sufficient to simply add
it to the hbase-operator-tools pom.xml file, so maven will load it. However,
it is actually deprecated in the current documentation for
edu.umd.cs.findbugs.annotations. Various commentary online suggest that we
should be using the similar jsr305.jar (*{{javax.annotation)}}* instead; see eg
*
[https://stackoverflow.com/questions/4963300/which-notnull-java-annotation-should-i-use]
Does anyone know if that was HBase's approach?
was:
If we do a local build of the current HBase branch-2.3 as version
2.3.0-SNAPSHOT, and 'mvn install' it in the local maven repository,
then attempt to build hbase-operator-tools master branch, with
-Dhbase.version=2.3.0-SNAPSHOT,
the HBCK2 class file HBCKMetaTableAccessor.java (line 25) fails to compile with
error: "package edu.umd.cs.findbugs.annotations does not exist", followed by
multiple instances of missing "symbol: class Nullable". (A longer log extract
is attached.)
It would appear the default HBase version 2.1.6 made this dependency package
available in some manner, but the newer HBase does not.
If the @Nullable annotation is still needed, it may be sufficient to simply add
it to the hbase-operator-tools pom.xml file, so maven will load it. However,
it is actually deprecated in the current documentation for
edu.umd.cs.findbugs.annotations. Various commentary online suggest that we
should be using the similar jsr305.jar (*{{javax.annotation)}}* instead; see eg
*
[https://stackoverflow.com/questions/4963300/which-notnull-java-annotation-should-i-use]
> [hbase-operator-tools] build of hbck2 fails with HBase branch-2.3, due to
> missing edu.umd.cs.findbugs.annotations dependency
>
>
> Key: HBASE-24482
> URL: https://issues.apache.org/jira/browse/HBASE-24482
> Project: HBase
> Issue Type: Bug
> Components: hbase-operator-tools
>Affects Versions: hbase-operator-tools-1.1.0
>Reporter: Matthew Foley
>Priority: Major
> Attachments: HBASE-24482_compile_error.log
>
>
> If we do a local build of the current HBase branch-2.3 as version
> 2.3.0-SNAPSHOT, and 'mvn install' it in the local maven repository,
> then attempt to build hbase-operator-tools master branch, with
> -Dhbase.version=2.3.0-SNAPSHOT,
> the HBCK2 class file HBCKMetaTableAccessor.java (line 25) fails to compile
> with error: "package edu.umd.cs.findbugs.annotations does not exist",
> followed by multiple instances of missing "symbol: class Nullable". (A
> longer log extract is attached.)
> It would appear the default HBase version 2.1.6 made this dependency package
> available in some manner, but the newer HBase does not.
> If the @Nullable annotation is still needed, it may be sufficient to simply
> add it to the hbase-operator-tools pom.xml file, so maven will load it.
> However, it is actually deprecated in the current documentation for
> edu.umd.cs.findbugs.annotations. Various commentary online suggest that we
> should be using the similar jsr305.jar (*{{javax.annotation)}}* instead; see
> eg
> *
> [https://stackoverflow.com/questions/4963300/which-notnull-java-annotation-should-i-use]
> Does anyone know if that was HBase's approach?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24482) [hbase-operator-tools] build of hbck2 fails with HBase branch-2.3, due to missing edu.umd.cs.findbugs.annotations dependency
[
https://issues.apache.org/jira/browse/HBASE-24482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthew Foley updated HBASE-24482:
--
Description:
If we do a local build of the current HBase branch-2.3 as version
2.3.0-SNAPSHOT, and 'mvn install' it in the local maven repository,
then attempt to build hbase-operator-tools master branch, with
-Dhbase.version=2.3.0-SNAPSHOT,
the HBCK2 class file HBCKMetaTableAccessor.java (line 25) fails to compile with
error: "package edu.umd.cs.findbugs.annotations does not exist", followed by
multiple instances of missing "symbol: class Nullable". (A longer log extract
is attached.)
It would appear the default HBase version 2.1.6 made this dependency package
available in some manner, but the newer HBase does not.
If the @Nullable annotation is still needed, it may be sufficient to simply add
it to the hbase-operator-tools pom.xml file, so maven will load it. However,
it is actually deprecated in the current documentation for
edu.umd.cs.findbugs.annotations. Various commentary online suggest that we
should be using the similar jsr305.jar (*{{javax.annotation)}}* instead; see eg
*
[https://stackoverflow.com/questions/4963300/which-notnull-java-annotation-should-i-use]
was:
If we do a local build of the current HBase branch-2.3 as version
2.3.0-SNAPSHOT, and 'mvn install' it in the local maven repository,
then attempt to build hbase-operator-tools master branch, with
-Dhbase.version=2.3.0-SNAPSHOT,
the HBCK2 class file HBCKMetaTableAccessor.java (line 25) fails to compile with
error: "package edu.umd.cs.findbugs.annotations does not exist", followed by
multiple instances of missing "symbol: class Nullable". (A longer log extract
is attached.)
It would appear the default HBase version 2.1.6 made this dependency package
available in some manner, but the newer HBase does not.
If the @Nullable annotation is still needed, it may be sufficient to simply add
it to the hbase-operator-tools pom.xml file, so maven will load it. However,
various commentary online suggest that we should be using the similar
jsr305.jar (*{{javax.annotation)}}* instead; see eg
*
[https://stackoverflow.com/questions/4963300/which-notnull-java-annotation-should-i-use]
> [hbase-operator-tools] build of hbck2 fails with HBase branch-2.3, due to
> missing edu.umd.cs.findbugs.annotations dependency
>
>
> Key: HBASE-24482
> URL: https://issues.apache.org/jira/browse/HBASE-24482
> Project: HBase
> Issue Type: Bug
> Components: hbase-operator-tools
>Affects Versions: hbase-operator-tools-1.1.0
>Reporter: Matthew Foley
>Priority: Major
> Attachments: HBASE-24482_compile_error.log
>
>
> If we do a local build of the current HBase branch-2.3 as version
> 2.3.0-SNAPSHOT, and 'mvn install' it in the local maven repository,
> then attempt to build hbase-operator-tools master branch, with
> -Dhbase.version=2.3.0-SNAPSHOT,
> the HBCK2 class file HBCKMetaTableAccessor.java (line 25) fails to compile
> with error: "package edu.umd.cs.findbugs.annotations does not exist",
> followed by multiple instances of missing "symbol: class Nullable". (A
> longer log extract is attached.)
> It would appear the default HBase version 2.1.6 made this dependency package
> available in some manner, but the newer HBase does not.
> If the @Nullable annotation is still needed, it may be sufficient to simply
> add it to the hbase-operator-tools pom.xml file, so maven will load it.
> However, it is actually deprecated in the current documentation for
> edu.umd.cs.findbugs.annotations. Various commentary online suggest that we
> should be using the similar jsr305.jar (*{{javax.annotation)}}* instead; see
> eg
> *
> [https://stackoverflow.com/questions/4963300/which-notnull-java-annotation-should-i-use]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24482) [hbase-operator-tools] build of hbck2 fails with HBase branch-2.3, due to missing edu.umd.cs.findbugs.annotations dependency
[
https://issues.apache.org/jira/browse/HBASE-24482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthew Foley updated HBASE-24482:
--
Attachment: HBASE-24482_compile_error.log
> [hbase-operator-tools] build of hbck2 fails with HBase branch-2.3, due to
> missing edu.umd.cs.findbugs.annotations dependency
>
>
> Key: HBASE-24482
> URL: https://issues.apache.org/jira/browse/HBASE-24482
> Project: HBase
> Issue Type: Bug
> Components: hbase-operator-tools
>Affects Versions: hbase-operator-tools-1.1.0
>Reporter: Matthew Foley
>Priority: Major
> Attachments: HBASE-24482_compile_error.log
>
>
> If we do a local build of the current HBase branch-2.3 as version
> 2.3.0-SNAPSHOT, and 'mvn install' it in the local maven repository,
> then attempt to build hbase-operator-tools master branch, with
> -Dhbase.version=2.3.0-SNAPSHOT,
> the HBCK2 class file HBCKMetaTableAccessor.java (line 25) fails to compile
> with error: "package edu.umd.cs.findbugs.annotations does not exist",
> followed by multiple instances of missing "symbol: class Nullable". (A
> longer log extract is attached.)
> It would appear the default HBase version 2.1.6 made this dependency package
> available in some manner, but the newer HBase does not.
> If the @Nullable annotation is still needed, it may be sufficient to simply
> add it to the hbase-operator-tools pom.xml file, so maven will load it.
> However, various commentary online suggest that we should be using the
> similar jsr305.jar (*{{javax.annotation)}}* instead; see eg
> *
> [https://stackoverflow.com/questions/4963300/which-notnull-java-annotation-should-i-use]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24482) [hbase-operator-tools] build of hbck2 fails with HBase branch-2.3, due to missing edu.umd.cs.findbugs.annotations dependency
Matthew Foley created HBASE-24482:
-
Summary: [hbase-operator-tools] build of hbck2 fails with HBase
branch-2.3, due to missing edu.umd.cs.findbugs.annotations dependency
Key: HBASE-24482
URL: https://issues.apache.org/jira/browse/HBASE-24482
Project: HBase
Issue Type: Bug
Components: hbase-operator-tools
Affects Versions: hbase-operator-tools-1.1.0
Reporter: Matthew Foley
If we do a local build of the current HBase branch-2.3 as version
2.3.0-SNAPSHOT, and 'mvn install' it in the local maven repository,
then attempt to build hbase-operator-tools master branch, with
-Dhbase.version=2.3.0-SNAPSHOT,
the HBCK2 class file HBCKMetaTableAccessor.java (line 25) fails to compile with
error: "package edu.umd.cs.findbugs.annotations does not exist", followed by
multiple instances of missing "symbol: class Nullable". (A longer log extract
is attached.)
It would appear the default HBase version 2.1.6 made this dependency package
available in some manner, but the newer HBase does not.
If the @Nullable annotation is still needed, it may be sufficient to simply add
it to the hbase-operator-tools pom.xml file, so maven will load it. However,
various commentary online suggest that we should be using the similar
jsr305.jar (*{{javax.annotation)}}* instead; see eg
*
[https://stackoverflow.com/questions/4963300/which-notnull-java-annotation-should-i-use]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] songxincun commented on pull request #1801: [HBASE-24441]CacheConfig details logged at Store open is not really u…
songxincun commented on pull request #1801: URL: https://github.com/apache/hbase/pull/1801#issuecomment-636613454 @anoopsjohn Thank you for your suggestion and review, thanks a lot : ) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24481) HBase Rest: Request for region detail of a table which doesn't exits is success(200 success code) instead of 403
Ajeet Rai created HBASE-24481: - Summary: HBase Rest: Request for region detail of a table which doesn't exits is success(200 success code) instead of 403 Key: HBASE-24481 URL: https://issues.apache.org/jira/browse/HBASE-24481 Project: HBase Issue Type: Bug Affects Versions: 2.2.3 Reporter: Ajeet Rai Request for region detail of a table which doesn't exits is success(200 success code) however response doesn't have any region data. However response for a request of schema detail for a table which doesn't exits, is 404(Not found) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] anoopsjohn commented on a change in pull request #1801: [HBASE-24441]CacheConfig details logged at Store open is not really u…
anoopsjohn commented on a change in pull request #1801:
URL: https://github.com/apache/hbase/pull/1801#discussion_r433034825
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -379,6 +379,9 @@ private MemStore getMemstore() {
protected void createCacheConf(final ColumnFamilyDescriptor family) {
this.cacheConf = new CacheConfig(conf, family, region.getBlockCache(),
region.getRegionServicesForStores().getByteBuffAllocator());
+LOG.info("Created cacheConfig: " + this.getCacheConfig()
Review comment:
Ya that should be fine IMO
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-24472) Enhanced version of KeyPrefixRegionSplitPolicy
[ https://issues.apache.org/jira/browse/HBASE-24472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120748#comment-17120748 ] Anil Sadineni commented on HBASE-24472: --- Thanks [~anoop.hbase] for response. I will work on patch and upload it here for review. > Enhanced version of KeyPrefixRegionSplitPolicy > -- > > Key: HBASE-24472 > URL: https://issues.apache.org/jira/browse/HBASE-24472 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Anil Sadineni >Priority: Major > > With KeyPrefixRegionSplitPolicy and DelimitedKeyPrefixRegionSplitPolicy, > splitting regions policy is limited to either fixed length or based on > delimiter. With Yarn Application Timeline Server V2, as discussed in > YARN-10077, it will be nice to have enhanced version of > DelimitedKeyPrefixRegionSplitPolicy that will give more flexibility to users > to define the ordinance of delimiter to consider. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24472) Enhanced version of KeyPrefixRegionSplitPolicy
[ https://issues.apache.org/jira/browse/HBASE-24472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120743#comment-17120743 ] Anoop Sam John commented on HBASE-24472: It make sense. You might not need new Split policy. To existing policy DelimitedKeyPrefixRegionSplitPolicy can add n optional param regarding the ordinal of the delimiter (Default is 1 anyways) > Enhanced version of KeyPrefixRegionSplitPolicy > -- > > Key: HBASE-24472 > URL: https://issues.apache.org/jira/browse/HBASE-24472 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Anil Sadineni >Priority: Major > > With KeyPrefixRegionSplitPolicy and DelimitedKeyPrefixRegionSplitPolicy, > splitting regions policy is limited to either fixed length or based on > delimiter. With Yarn Application Timeline Server V2, as discussed in > YARN-10077, it will be nice to have enhanced version of > DelimitedKeyPrefixRegionSplitPolicy that will give more flexibility to users > to define the ordinance of delimiter to consider. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[ https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120737#comment-17120737 ] Anoop Sam John commented on HBASE-24440: When 2 such mutate reqs comes and applied at same TS, need to keep both of these as 2 versions. That is the need right? Right now it will be treated as duplicate entry and the one with higher seqNo will come out of the scan. > Prevent temporal misordering on timescales smaller than one clock tick > -- > > Key: HBASE-24440 > URL: https://issues.apache.org/jira/browse/HBASE-24440 > Project: HBase > Issue Type: Brainstorming >Reporter: Andrew Kyle Purtell >Priority: Major > > When mutations are sent to the servers without a timestamp explicitly > assigned by the client the server will substitute the current wall clock > time. There are edge cases where it is at least theoretically possible for > more than one mutation to be committed to a given row within the same clock > tick. When this happens we have to track and preserve the ordering of these > mutations in some other way besides the timestamp component of the key. Let > me bypass most discussion here by noting that whether we do this or not, we > do not pass such ordering information in the cross cluster replication > protocol. We also have interesting edge cases regarding key type precedence > when mutations arrive "simultaneously": we sort deletes ahead of puts. This, > especially in the presence of replication, can lead to visible anomalies for > clients able to interact with both source and sink. > There is a simple solution that removes the possibility that these edge cases > can occur: > We can detect, when we are about to commit a mutation to a row, if we have > already committed a mutation to this same row in the current clock tick. > Occurrences of this condition will be rare. We are already tracking current > time. We have to know this in order to assign the timestamp. Where this > becomes interesting is how we might track the last commit time per row. > Making the detection of this case efficient for the normal code path is the > bulk of the challenge. One option is to keep track of the last locked time > for row locks. (Todo: How would we track and garbage collect this efficiently > and correctly. Not the ideal option.) We might also do this tracking somehow > via the memstore. (At least in this case the lifetime and distribution of in > memory row state, including the proposed timestamps, would align.) Assuming > we can efficiently know if we are about to commit twice to the same row > within a single clock tick, we would simply sleep/yield the current thread > until the clock ticks over, and then proceed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1821: HBASE-24480: Deflake TestRSGroupsBasics#testClearDeadServers
Apache-HBase commented on pull request #1821: URL: https://github.com/apache/hbase/pull/1821#issuecomment-636588211 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-1 Compile Tests _ | | +1 :green_heart: | mvninstall | 9m 45s | branch-1 passed | | +1 :green_heart: | compile | 0m 18s | branch-1 passed with JDK v1.8.0_252 | | +1 :green_heart: | compile | 0m 23s | branch-1 passed with JDK v1.7.0_262 | | +1 :green_heart: | checkstyle | 0m 26s | branch-1 passed | | +1 :green_heart: | shadedjars | 3m 0s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 32s | branch-1 passed with JDK v1.8.0_252 | | +1 :green_heart: | javadoc | 0m 21s | branch-1 passed with JDK v1.7.0_262 | | +0 :ok: | spotbugs | 1m 9s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 7s | branch-1 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 52s | the patch passed | | +1 :green_heart: | compile | 0m 18s | the patch passed with JDK v1.8.0_252 | | +1 :green_heart: | javac | 0m 18s | the patch passed | | +1 :green_heart: | compile | 0m 23s | the patch passed with JDK v1.7.0_262 | | +1 :green_heart: | javac | 0m 23s | the patch passed | | -1 :x: | checkstyle | 0m 16s | hbase-rsgroup: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 2m 47s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 4m 32s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 0m 16s | the patch passed with JDK v1.8.0_252 | | +1 :green_heart: | javadoc | 0m 20s | the patch passed with JDK v1.7.0_262 | | +1 :green_heart: | findbugs | 0m 57s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 11m 51s | hbase-rsgroup in the patch passed. | | +1 :green_heart: | asflicense | 0m 22s | The patch does not generate ASF License warnings. | | | | 43m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1821/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1821 | | JIRA Issue | HBASE-24480 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux f3d73612efcf 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1821/out/precommit/personality/provided.sh | | git revision | branch-1 / 3c13884 | | Default Java | 1.7.0_262 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_252 /usr/lib/jvm/zulu-7-amd64:1.7.0_262 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1821/1/artifact/out/diff-checkstyle-hbase-rsgroup.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1821/1/testReport/ | | Max. process+thread count | 1607 (vs. ulimit of 1) | | modules | C: hbase-rsgroup U: hbase-rsgroup | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1821/1/console | | versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Reidddddd commented on a change in pull request #1821: HBASE-24480: Deflake TestRSGroupsBasics#testClearDeadServers
Reidd commented on a change in pull request #1821:
URL: https://github.com/apache/hbase/pull/1821#discussion_r433021673
##
File path:
hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
##
@@ -385,7 +386,7 @@ public RSGroupInfoManager getRSGroupInfoManager() throws
IOException {
}
@Override
- public void removeServers(Set servers) throws IOException {
+ public void removeServers(@Nullable Set servers) throws IOException
{
Review comment:
I don't mean annotation `Nullable` is not good, but HBase doesn't use it
around the codebases. So here makes me feel not unified, what about comment the
nullable at the method @ parma description.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-21905) TestFIFOCompactionPolicy is flaky
[ https://issues.apache.org/jira/browse/HBASE-21905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120707#comment-17120707 ] Bharath Vissapragada commented on HBASE-21905: -- Repros pretty frequently for me actually, attached the failure log from the latest run on branch-1 HEAD. > TestFIFOCompactionPolicy is flaky > - > > Key: HBASE-21905 > URL: https://issues.apache.org/jira/browse/HBASE-21905 > Project: HBase > Issue Type: Test > Components: test >Affects Versions: 1.5.0 >Reporter: Andrew Kyle Purtell >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Labels: branch-1 > Fix For: 1.7.0 > > Attachments: > org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy-output.txt, > testFIFOCompactionPolicyExpiredEmptyHFiles-failure-log.txt > > > java.lang.IllegalArgumentException , overlaps with > For example: > [ERROR] > testFIFOCompactionPolicyExpiredEmptyHFiles(org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy) > Time elapsed: 3.321 s <<< ERROR! > java.io.IOException: > java.io.IOException: > [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69, > > hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c65648691f614b2d8dd4b586c5923bfe] > overlaps with > [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69] > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2438) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) > Caused by: java.lang.IllegalArgumentException: > [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69, > > hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c65648691f614b2d8dd4b586c5923bfe] > overlaps with > [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69] > at > com.google.common.base.Preconditions.checkArgument(Preconditions.java:119) > at > org.apache.hadoop.hbase.regionserver.HStore.addToCompactingFiles(HStore.java:1824) > at > org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1798) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:415) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:388) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:317) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:306) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.compactRegion(RSRpcServices.java:1513) > at > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:23649) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380) > ... 3 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21905) TestFIFOCompactionPolicy is flaky
[ https://issues.apache.org/jira/browse/HBASE-21905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharath Vissapragada updated HBASE-21905: - Attachment: testFIFOCompactionPolicyExpiredEmptyHFiles-failure-log.txt > TestFIFOCompactionPolicy is flaky > - > > Key: HBASE-21905 > URL: https://issues.apache.org/jira/browse/HBASE-21905 > Project: HBase > Issue Type: Test > Components: test >Affects Versions: 1.5.0 >Reporter: Andrew Kyle Purtell >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Labels: branch-1 > Fix For: 1.7.0 > > Attachments: > org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy-output.txt, > testFIFOCompactionPolicyExpiredEmptyHFiles-failure-log.txt > > > java.lang.IllegalArgumentException , overlaps with > For example: > [ERROR] > testFIFOCompactionPolicyExpiredEmptyHFiles(org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy) > Time elapsed: 3.321 s <<< ERROR! > java.io.IOException: > java.io.IOException: > [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69, > > hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c65648691f614b2d8dd4b586c5923bfe] > overlaps with > [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69] > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2438) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) > Caused by: java.lang.IllegalArgumentException: > [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69, > > hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c65648691f614b2d8dd4b586c5923bfe] > overlaps with > [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69] > at > com.google.common.base.Preconditions.checkArgument(Preconditions.java:119) > at > org.apache.hadoop.hbase.regionserver.HStore.addToCompactingFiles(HStore.java:1824) > at > org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1798) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:415) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:388) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:317) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:306) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.compactRegion(RSRpcServices.java:1513) > at > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:23649) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380) > ... 3 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-21905) TestFIFOCompactionPolicy is flaky
[ https://issues.apache.org/jira/browse/HBASE-21905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120704#comment-17120704 ] Bharath Vissapragada commented on HBASE-21905: -- [~taklwu] Do you have a patch for this? If you don't have cycles to look into this, feel free to re-assign it to me. We are running into this issue internally on a branch-1 based fork. > TestFIFOCompactionPolicy is flaky > - > > Key: HBASE-21905 > URL: https://issues.apache.org/jira/browse/HBASE-21905 > Project: HBase > Issue Type: Test > Components: test >Affects Versions: 1.5.0 >Reporter: Andrew Kyle Purtell >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Labels: branch-1 > Fix For: 1.7.0 > > Attachments: > org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy-output.txt > > > java.lang.IllegalArgumentException , overlaps with > For example: > [ERROR] > testFIFOCompactionPolicyExpiredEmptyHFiles(org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy) > Time elapsed: 3.321 s <<< ERROR! > java.io.IOException: > java.io.IOException: > [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69, > > hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c65648691f614b2d8dd4b586c5923bfe] > overlaps with > [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69] > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2438) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) > Caused by: java.lang.IllegalArgumentException: > [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69, > > hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c65648691f614b2d8dd4b586c5923bfe] > overlaps with > [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69] > at > com.google.common.base.Preconditions.checkArgument(Preconditions.java:119) > at > org.apache.hadoop.hbase.regionserver.HStore.addToCompactingFiles(HStore.java:1824) > at > org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1798) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:415) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:388) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:317) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:306) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.compactRegion(RSRpcServices.java:1513) > at > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:23649) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380) > ... 3 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] bharathv opened a new pull request #1821: HBASE-24480: Deflake TestRSGroupsBasics#testClearDeadServers
bharathv opened a new pull request #1821: URL: https://github.com/apache/hbase/pull/1821 More details about the flakiness in the jira comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24454) BucketCache disabled instantly before error duration toleration is reached due to timing issue
[
https://issues.apache.org/jira/browse/HBASE-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120697#comment-17120697
]
Hudson commented on HBASE-24454:
Results for branch branch-2
[build #2686 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2686/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2686/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2686/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2686/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2686/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> BucketCache disabled instantly before error duration toleration is reached
> due to timing issue
> --
>
> Key: HBASE-24454
> URL: https://issues.apache.org/jira/browse/HBASE-24454
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 1.4.10
> Environment: We saw this in HBase 1.4.10 (EMR 5.28.1), but I believe
> the newest code still has this problem.
>Reporter: Jacob LeBlanc
>Assignee: Jacob LeBlanc
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.6
>
>
> We saw an instance where BucketCache was disabled after only two IO error
> were encountered at nearly the same time. The following shows all errors from
> a region server log for the 2020-05-26 17:00 hour. Notice that there are no
> other errors until the 17:14:50 at which time the BucketCache is disabled
> because it claims duration time has exceeded 6 ms:
> $ grep ERROR hbase-hbase-regionserver-ip-172-20-113-147.log.2020-05-26-17
> 2020-05-26 17:14:50,396 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: Failed reading block
> 9adfad3a603047cfa0c98b16da0b0974_13786 from bucket cache
> 2020-05-26 17:14:50,397 ERROR
> [regionserver/ip-172-20-113-147.us-west-2.compute.internal/172.20.113.147:16020-BucketCacheWriter-0]
> bucket.BucketCache: Failed syncing IO engine
> 2020-05-26 17:14:50,400 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: Failed reading block
> 9adfad3a603047cfa0c98b16da0b0974_13786 from bucket cache
> 2020-05-26 17:14:50,401 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: IO errors duration time has exceeded 6ms, disabling
> cache, please check your IOEngine
> The region server is very busy so it should be constantly getting successful
> reads and writes in the bucket cache (it is not as though there was some long
> ago error and then no successful IO to clear the error flag).
> We are running a busy EMR cluster backed by S3. A bucketcache getting
> disabled is a huge performance issue because all reads must go to S3.
> Looking at the code, I believe I've found a timing issue. Here is the code
> for checking and setting the ioErrorStartTime (taken from BucketCache.java):
>
> {code:java}
> /**
>* Check whether we tolerate IO error this time. If the duration of IOEngine
>* throwing errors exceeds ioErrorsDurationTimeTolerated, we will disable
> the
>* cache
>*/
> private void checkIOErrorIsTolerated() {
> long now = EnvironmentEdgeManager.currentTime();
> if (this.ioErrorStartTime > 0) {
> if (cacheEnabled && (now - ioErrorStartTime) >
> this.ioErrorsTolerationDuration) {
> LOG.error("IO errors duration time has exceeded " +
> ioErrorsTolerationDuration +
> "ms, disabling cache, please check your IOEngine");
> disableCache();
> }
> } else {
> this.ioErrorStartTime = now;
> }
> }
> {code}
>
> And here is the code for clearing the ioErrorStartTime when a successful read
> or write is done:
> {code:java}
> if (this.ioErrorStartTime > 0) {
> ioErrorStartTime = -1;
> }
> {code}
> Notice that that if ioErrorStartTime is set to -1 after the first if
> statement in checkIOErrorIsTolerated but before (now - ioErrorStartTime),
> then (now - ioErrorStartTime) will evaluate to (now + 1) resulting in the
> code thinking it has exceeded ioErrorsTolerationDuration.
--
This
[jira] [Commented] (HBASE-24475) Clean up the master thread name getting in SplitLogManager and AssignmentManager
[
https://issues.apache.org/jira/browse/HBASE-24475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120698#comment-17120698
]
Hudson commented on HBASE-24475:
Results for branch branch-2
[build #2686 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2686/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2686/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2686/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2686/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2686/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Clean up the master thread name getting in SplitLogManager and
> AssignmentManager
>
>
> Key: HBASE-24475
> URL: https://issues.apache.org/jira/browse/HBASE-24475
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.0
>
>
> As discussed in PR of HBASE-24451, seems just use
> master.getServerName().toShortString() is enough.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23340) hmaster /hbase/replication/rs session expired (hbase replication default value is true, we don't use ) causes logcleaner can not clean oldWALs, which resulits in old
[ https://issues.apache.org/jira/browse/HBASE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120694#comment-17120694 ] Chao Wang commented on HBASE-23340: --- Thanck you for this issue.I seem my code of 1.3.1 has merged this issue for HBASE-15234, but this issue still exist. > hmaster /hbase/replication/rs session expired (hbase replication default > value is true, we don't use ) causes logcleaner can not clean oldWALs, which > resulits in oldWALs too large (more than 2TB) > - > > Key: HBASE-23340 > URL: https://issues.apache.org/jira/browse/HBASE-23340 > Project: HBase > Issue Type: Improvement > Components: master >Reporter: jackylau >Assignee: jackylau >Priority: Major > Fix For: master > > Attachments: Snipaste_2019-11-21_10-39-25.png, > Snipaste_2019-11-21_14-10-36.png > > > hmaster /hbase/replication/rs session expired (hbase replication default > value is true, we don't use ) causes logcleaner can not clean oldWALs, which > resulits in oldWALs too large (more than 2TB). > !Snipaste_2019-11-21_10-39-25.png! > > !Snipaste_2019-11-21_14-10-36.png! > > we can solve it by following : > 1) increase the session timeout(but i think it is not a good idea. because we > do not know how long to set is suitable) > 2) close the hbase replication. It is not a good idea too, when our user uses > this feature > 3) we need add retry times, for example when it has already happened three > times, we set the ReplicationLogCleaner and SnapShotCleaner stop > that is all my ideas, i do not konw it is suitable, If it is suitable, could > i commit a PR? > Does anynode have a good idea. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24480) Deflake TestRSGroupsBasics#testClearDeadServers
[
https://issues.apache.org/jira/browse/HBASE-24480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120691#comment-17120691
]
Bharath Vissapragada commented on HBASE-24480:
--
I think the root cause is this. The problem happens when an empty list is
passed to postClearDeadServers hook...
{noformat}
public void postClearDeadServers(ObserverContext
ctx,
List servers, List notClearedServers)
throws IOException {
Set clearedServer = Sets.newHashSet();
for (ServerName server: servers) {
if (!notClearedServers.contains(server)) {
clearedServer.add(server.getAddress());
}
}
groupAdminServer.removeServers(clearedServer); <== clearedServer list is
empty
}
{noformat}
The cause of this is in clearDeadServers() RPC..
{noformat}
if (master.getServerManager().areDeadServersInProgress()) {
LOG.debug("Some dead server is still under processing, won't clear the
dead server list"); <===
response.addAllServerName(request.getServerNameList());
} else {
for (HBaseProtos.ServerName pbServer : request.getServerNameList()) {
if (!master.getServerManager().getDeadServers()
.removeDeadServer(ProtobufUtil.toServerName(pbServer))) {
response.addServerName(pbServer);
}
}
}
{noformat}
I could see the LOG.debug() in the logs. Its the same region server that was
stopped. Essentially there is a dead server that is being processed and hence
the current request was rejected. The fix is essentially the following
- Don't execute the post hook if no server is cleared
- Make the the test more robust to handle this RPC failure..
> Deflake TestRSGroupsBasics#testClearDeadServers
> ---
>
> Key: HBASE-24480
> URL: https://issues.apache.org/jira/browse/HBASE-24480
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Affects Versions: 2.3.0, 1.7.0
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Major
>
> Ran into this on our internal forks based on branch-1. It also applies to
> branch-2 but not master because the code has been re-implemented without
> co-proc due to HBASE-22514
> Running into this exception in the test run..
> {noformat}
> org.apache.hadoop.hbase.constraint.ConstraintException: The set of servers to
> remove cannot be null or empty.at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.removeServers(RSGroupAdminServer.java:391)
>at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint.postClearDeadServers(RSGroupAdminEndpoint.java:1175)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$104.call(MasterCoprocessorHost.java:1251)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1507)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postClearDeadServers(MasterCoprocessorHost.java:1247)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.clearDeadServers(MasterRpcServices.java:1167)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
>at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2421) at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:311)
>at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:291)"
>
> type="org.apache.hadoop.hbase.constraint.ConstraintException">org.apache.hadoop.hbase.constraint.ConstraintException:
>
> org.apache.hadoop.hbase.constraint.ConstraintException: The set of servers to
> remove cannot be null or empty.
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.removeServers(RSGroupAdminServer.java:391)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint.postClearDeadServers(RSGroupAdminEndpoint.java:1175)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$104.call(MasterCoprocessorHost.java:1251)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1507)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postClearDeadServers(MasterCoprocessorHost.java:1247)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.clearDeadServers(MasterRpcServices.java:1167)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2421)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:311)
> at
>
[GitHub] [hbase] Apache-HBase commented on pull request #1820: HBASE-24479: Deflake TestCompaction#testStopStartCompaction
Apache-HBase commented on pull request #1820: URL: https://github.com/apache/hbase/pull/1820#issuecomment-636572186 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 7m 3s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-1 Compile Tests _ | | +1 :green_heart: | mvninstall | 9m 53s | branch-1 passed | | +1 :green_heart: | compile | 0m 40s | branch-1 passed with JDK v1.8.0_252 | | +1 :green_heart: | compile | 0m 44s | branch-1 passed with JDK v1.7.0_262 | | +1 :green_heart: | checkstyle | 1m 39s | branch-1 passed | | +1 :green_heart: | shadedjars | 3m 4s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | branch-1 passed with JDK v1.8.0_252 | | +1 :green_heart: | javadoc | 0m 40s | branch-1 passed with JDK v1.7.0_262 | | +0 :ok: | spotbugs | 3m 1s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 0s | branch-1 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 57s | the patch passed | | +1 :green_heart: | compile | 0m 39s | the patch passed with JDK v1.8.0_252 | | +1 :green_heart: | javac | 0m 39s | the patch passed | | +1 :green_heart: | compile | 0m 47s | the patch passed with JDK v1.7.0_262 | | +1 :green_heart: | javac | 0m 47s | the patch passed | | -1 :x: | checkstyle | 1m 32s | hbase-server: The patch generated 1 new + 12 unchanged - 0 fixed = 13 total (was 12) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 2m 47s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 4m 40s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 0m 31s | the patch passed with JDK v1.8.0_252 | | +1 :green_heart: | javadoc | 0m 44s | the patch passed with JDK v1.7.0_262 | | +1 :green_heart: | findbugs | 2m 46s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 123m 17s | hbase-server in the patch passed. | | +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. | | | | 171m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1820/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1820 | | JIRA Issue | HBASE-24479 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux b17f2a4dc55b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1820/out/precommit/personality/provided.sh | | git revision | branch-1 / 3c13884 | | Default Java | 1.7.0_262 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_252 /usr/lib/jvm/zulu-7-amd64:1.7.0_262 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1820/1/artifact/out/diff-checkstyle-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1820/1/testReport/ | | Max. process+thread count | 4113 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1820/1/console | | versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24480) Deflake TestRSGroupsBasics#testClearDeadServers
Bharath Vissapragada created HBASE-24480:
Summary: Deflake TestRSGroupsBasics#testClearDeadServers
Key: HBASE-24480
URL: https://issues.apache.org/jira/browse/HBASE-24480
Project: HBase
Issue Type: Bug
Components: rsgroup
Affects Versions: 2.3.0, 1.7.0
Reporter: Bharath Vissapragada
Assignee: Bharath Vissapragada
Ran into this on our internal forks based on branch-1. It also applies to
branch-2 but not master because the code has been re-implemented without
co-proc due to HBASE-22514
Running into this exception in the test run..
{noformat}
org.apache.hadoop.hbase.constraint.ConstraintException: The set of servers to
remove cannot be null or empty. at
org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.removeServers(RSGroupAdminServer.java:391)
at
org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint.postClearDeadServers(RSGroupAdminEndpoint.java:1175)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost$104.call(MasterCoprocessorHost.java:1251)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1507)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost.postClearDeadServers(MasterCoprocessorHost.java:1247)
at
org.apache.hadoop.hbase.master.MasterRpcServices.clearDeadServers(MasterRpcServices.java:1167)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2421) at
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:311)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:291)"
type="org.apache.hadoop.hbase.constraint.ConstraintException">org.apache.hadoop.hbase.constraint.ConstraintException:
org.apache.hadoop.hbase.constraint.ConstraintException: The set of servers to
remove cannot be null or empty.
at
org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.removeServers(RSGroupAdminServer.java:391)
at
org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint.postClearDeadServers(RSGroupAdminEndpoint.java:1175)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost$104.call(MasterCoprocessorHost.java:1251)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1507)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost.postClearDeadServers(MasterCoprocessorHost.java:1247)
at
org.apache.hadoop.hbase.master.MasterRpcServices.clearDeadServers(MasterRpcServices.java:1167)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2421)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:311)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:291)
at
org.apache.hadoop.hbase.rsgroup.TestRSGroupsBasics.testClearDeadServers(TestRSGroupsBasics.java:215)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException:
org.apache.hadoop.hbase.constraint.ConstraintException: The set of servers to
remove cannot be null or empty.
at
org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.removeServers(RSGroupAdminServer.java:391)
at
org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint.postClearDeadServers(RSGroupAdminEndpoint.java:1175)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost$104.call(MasterCoprocessorHost.java:1251)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1507)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost.postClearDeadServers(MasterCoprocessorHost.java:1247)
at
org.apache.hadoop.hbase.master.MasterRpcServices.clearDeadServers(MasterRpcServices.java:1167)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2421)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:311)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:291)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24454) BucketCache disabled instantly before error duration toleration is reached due to timing issue
[
https://issues.apache.org/jira/browse/HBASE-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120655#comment-17120655
]
Hudson commented on HBASE-24454:
Results for branch branch-2.2
[build #881 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/881/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/881//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/881//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/881//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> BucketCache disabled instantly before error duration toleration is reached
> due to timing issue
> --
>
> Key: HBASE-24454
> URL: https://issues.apache.org/jira/browse/HBASE-24454
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 1.4.10
> Environment: We saw this in HBase 1.4.10 (EMR 5.28.1), but I believe
> the newest code still has this problem.
>Reporter: Jacob LeBlanc
>Assignee: Jacob LeBlanc
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.6
>
>
> We saw an instance where BucketCache was disabled after only two IO error
> were encountered at nearly the same time. The following shows all errors from
> a region server log for the 2020-05-26 17:00 hour. Notice that there are no
> other errors until the 17:14:50 at which time the BucketCache is disabled
> because it claims duration time has exceeded 6 ms:
> $ grep ERROR hbase-hbase-regionserver-ip-172-20-113-147.log.2020-05-26-17
> 2020-05-26 17:14:50,396 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: Failed reading block
> 9adfad3a603047cfa0c98b16da0b0974_13786 from bucket cache
> 2020-05-26 17:14:50,397 ERROR
> [regionserver/ip-172-20-113-147.us-west-2.compute.internal/172.20.113.147:16020-BucketCacheWriter-0]
> bucket.BucketCache: Failed syncing IO engine
> 2020-05-26 17:14:50,400 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: Failed reading block
> 9adfad3a603047cfa0c98b16da0b0974_13786 from bucket cache
> 2020-05-26 17:14:50,401 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: IO errors duration time has exceeded 6ms, disabling
> cache, please check your IOEngine
> The region server is very busy so it should be constantly getting successful
> reads and writes in the bucket cache (it is not as though there was some long
> ago error and then no successful IO to clear the error flag).
> We are running a busy EMR cluster backed by S3. A bucketcache getting
> disabled is a huge performance issue because all reads must go to S3.
> Looking at the code, I believe I've found a timing issue. Here is the code
> for checking and setting the ioErrorStartTime (taken from BucketCache.java):
>
> {code:java}
> /**
>* Check whether we tolerate IO error this time. If the duration of IOEngine
>* throwing errors exceeds ioErrorsDurationTimeTolerated, we will disable
> the
>* cache
>*/
> private void checkIOErrorIsTolerated() {
> long now = EnvironmentEdgeManager.currentTime();
> if (this.ioErrorStartTime > 0) {
> if (cacheEnabled && (now - ioErrorStartTime) >
> this.ioErrorsTolerationDuration) {
> LOG.error("IO errors duration time has exceeded " +
> ioErrorsTolerationDuration +
> "ms, disabling cache, please check your IOEngine");
> disableCache();
> }
> } else {
> this.ioErrorStartTime = now;
> }
> }
> {code}
>
> And here is the code for clearing the ioErrorStartTime when a successful read
> or write is done:
> {code:java}
> if (this.ioErrorStartTime > 0) {
> ioErrorStartTime = -1;
> }
> {code}
> Notice that that if ioErrorStartTime is set to -1 after the first if
> statement in checkIOErrorIsTolerated but before (now - ioErrorStartTime),
> then (now - ioErrorStartTime) will evaluate to (now + 1) resulting in the
> code thinking it has exceeded ioErrorsTolerationDuration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] mmpataki commented on a change in pull request #1814: HBASE-20904 Prometheus /metrics http endpoint for monitoring
mmpataki commented on a change in pull request #1814: URL: https://github.com/apache/hbase/pull/1814#discussion_r432997789 ## File path: hbase-common/src/main/resources/hbase-default.xml ## @@ -1727,6 +1727,15 @@ possible configurations would overwhelm and obscure the important. ThreadPool. + +hbase.http.enable.prometheus.servlets +false + + Enable prometheus servlets /prom and /prom2 for prometheus based monitoring. + /prom is based on new HBase metrics API and all metrics are not exported for now. + /prom2 is based on the old hadoop2 metrics API and has all the metrics. Review comment: @saintstack > Why not /prom and /prom-old ? Why have the old at all (maybe you want to backport this)? When /prom and /prom2, w/o close reading, users will think they need to read from /prom2 because 2 is a later number than no number? > > Also, can you say more on old vs new metrics. I'm not clear. Would be good to also clean up the description in here so clear to the casual reader. > > Thakns for adding the flag. That makes sense, I will change it to /prom-old or similar and fix the documentation of the flag. (Answers to other questions are in the next block) > Is 'new metrics' our use of 'hadoop metrics', a move we made years ago? What versions in release do the old way of metrics? Thanks. HBASE-9774 brought in hbase-native metrics collection for the coprocessors and a decision was made to to use this API to record all the other metrics as well (hbase-metrics-api / README.txt). I am referring to this as "new API'. Now all the work to consume this new API isn't done so we still have to depend on the "old", "hadoop2" based metric collection. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24479) Deflake TestCompaction#testStopStartCompaction
[
https://issues.apache.org/jira/browse/HBASE-24479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharath Vissapragada updated HBASE-24479:
-
Labels: compaction (was: )
> Deflake TestCompaction#testStopStartCompaction
> --
>
> Key: HBASE-24479
> URL: https://issues.apache.org/jira/browse/HBASE-24479
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.7.0
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Major
> Labels: compaction
>
> Saw this in our internal test runs.
> {noformat}
> org.apache.hadoop.hbase.regionserver.TestCompactionWithCoprocessor.testStopStartCompaction
> Failing for the past 1 build (Since Unstable#10 )
> Took 0.72 sec.
> add description
> Error Message
> expected:<1> but was:<0>
> Stacktrace
> java.lang.AssertionError: expected:<1> but was:<0>
> {noformat}
> The test asserts can be tightened to improve this. Will follow up with a
> patch shortly. HBASE-23899 added some debug logging to figure out why.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] bharathv commented on pull request #1820: HBASE-24479: Deflake TestCompaction#testStopStartCompaction
bharathv commented on pull request #1820: URL: https://github.com/apache/hbase/pull/1820#issuecomment-636543041 I have branch-1 checked out locally, so I submitted a PR for branch-1 now, will forward port this to other branches after this is merged. This applies to all branches AFAICT. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bharathv opened a new pull request #1820: HBASE-24479: Deflake TestCompaction#testStopStartCompaction
bharathv opened a new pull request #1820: URL: https://github.com/apache/hbase/pull/1820 Polling of active compaction count is racy. Tightened the asserts to be more reliable. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24479) Deflake TestCompaction#testStopStartCompaction
[
https://issues.apache.org/jira/browse/HBASE-24479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120648#comment-17120648
]
Bharath Vissapragada commented on HBASE-24479:
--
I think the test is racy.. (following snippet from branch-1)
{noformat}
public void testStopStartCompaction() throws IOException {
...
thread.requestCompaction(r, store, "test", Store.PRIORITY_USER, new
CompactionRequest(), null);
-> Add a sleep here and the test fails reliably <--
assertEquals(1, thread.getLongCompactions().getActiveCount() +
thread.getShortCompactions()
.getActiveCount());
}
{noformat}
The problem is that the compaction asynchronously finishes before we even poll
for the {{activeCount}}
> Deflake TestCompaction#testStopStartCompaction
> --
>
> Key: HBASE-24479
> URL: https://issues.apache.org/jira/browse/HBASE-24479
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.7.0
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Major
>
> Saw this in our internal test runs.
> {noformat}
> org.apache.hadoop.hbase.regionserver.TestCompactionWithCoprocessor.testStopStartCompaction
> Failing for the past 1 build (Since Unstable#10 )
> Took 0.72 sec.
> add description
> Error Message
> expected:<1> but was:<0>
> Stacktrace
> java.lang.AssertionError: expected:<1> but was:<0>
> {noformat}
> The test asserts can be tightened to improve this. Will follow up with a
> patch shortly. HBASE-23899 added some debug logging to figure out why.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24479) Deflake TestCompaction#testStopStartCompaction
Bharath Vissapragada created HBASE-24479:
Summary: Deflake TestCompaction#testStopStartCompaction
Key: HBASE-24479
URL: https://issues.apache.org/jira/browse/HBASE-24479
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.7.0
Reporter: Bharath Vissapragada
Assignee: Bharath Vissapragada
Saw this in our internal test runs.
{noformat}
org.apache.hadoop.hbase.regionserver.TestCompactionWithCoprocessor.testStopStartCompaction
Failing for the past 1 build (Since Unstable#10 )
Took 0.72 sec.
add description
Error Message
expected:<1> but was:<0>
Stacktrace
java.lang.AssertionError: expected:<1> but was:<0>
{noformat}
The test asserts can be tightened to improve this. Will follow up with a patch
shortly. HBASE-23899 added some debug logging to figure out why.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] bharathv merged pull request #1755: HBASE-24069 Provide an ExponentialBackOffPolicy sleep between failed …
bharathv merged pull request #1755: URL: https://github.com/apache/hbase/pull/1755 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-24477) Move ConfigurationObserver and related classes to hbase-common
[ https://issues.apache.org/jira/browse/HBASE-24477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharath Vissapragada resolved HBASE-24477. -- Fix Version/s: 2.4.0 1.7.0 2.3.0 3.0.0-alpha-1 Resolution: Fixed Pushed to branch-1/2/2.3/master. Appreciate the quick reviews, Stack/Viraj. > Move ConfigurationObserver and related classes to hbase-common > -- > > Key: HBASE-24477 > URL: https://issues.apache.org/jira/browse/HBASE-24477 > Project: HBase > Issue Type: Task > Components: conf >Reporter: Bharath Vissapragada >Assignee: Bharath Vissapragada >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.4.0 > > > It's a nice utility that all the modules can leverage. Putting it in common > makes it accessible. I want to use it in client for dynamic reconfiguration > of master addresses in HBASE-18095. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120590#comment-17120590
]
Danil Lipovoy edited comment on HBASE-23887 at 5/31/20, 9:18 PM:
-
All tests below have done on my home PC: _AMD Ryzen 7 2700X Eight-Core
Processor (3150 MHz, 16 threads)._
Logic of auto-scaling (see describe here):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation. Will be fine add more logic here.
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables (32 regions each):
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_requestdistribution=uniform_
Workload scenario "z":
_requestdistribution=zipfian_
Workload scenario "l":
_requestdistribution=latest_
Other parameters:
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_recordcount=100 (I just noticed this value is too small, I will provide
new tests with bigger value later)_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature:
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
!requests_100p.png!
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100 | no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 | start reading but
*count.limit* haven't reached.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 | *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction counter: 6, current caching DataBlock (%): 91
LruBlockCache: BlockCache evicted (MB): 7722, overhead (%):
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120565#comment-17120565
]
Danil Lipovoy edited comment on HBASE-23887 at 5/31/20, 8:44 PM:
-
Seems like our trouble with the servers for a long time and I've decided
install HBase on my home PC.
Another important point - I have done the algorithm, that I posted above (will
add changes to PR quite soon). It is good when numbers of reading requests are
changing. Looks like the new approach copes well with wide variety kind of
situation (a lot of tests in the next messages after answers).
1. I'm nor sure, but maybe it is because first few seconds, while BlockCache
is empty, my old version of realization prevented effective populating the BC.
I mean it was skipping blocks when eviction is not running - and a lot of
blocks could be cached but were lost. With the new approach the problems has
gone. For example:
This is when 100% of data caching (uniform distribution):
[OVERALL], RunTime(ms), 1506417
[OVERALL], Throughput(ops/sec), 33191.34077748724
[TOTAL_GCS_PS_Scavenge], Count, 8388
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 12146
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.8062840501667201
[TOTAL_GCS_PS_MarkSweep], Count, 1
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 22
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.0014604189942094387
[TOTAL_GCs], Count, 8389
[TOTAL_GC_TIME], Time(ms), 12168
[TOTAL_GC_TIME_%], Time(%), 0.8077444691609296
[READ], Operations, 5000
[READ], AverageLatency(us), 1503.45024378
[READ], MinLatency(us), 137
[READ], MaxLatency(us), 383999
[READ], 95thPercentileLatency(us), 2231
[READ], 99thPercentileLatency(us), 13503
[READ], Return=OK, 5000
The same table with the patch:
[OVERALL], RunTime(ms), 1073257
[OVERALL], Throughput(ops/sec), 46587.1641181935
[TOTAL_GCS_PS_Scavenge], Count, 7201
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 9799
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.9130152423883563
[TOTAL_GCS_PS_MarkSweep], Count, 1
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 23
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.002143009549436901
[TOTAL_GCs], Count, 7202
[TOTAL_GC_TIME], Time(ms), 9822
[TOTAL_GC_TIME_%], Time(%), 0.9151582519377931
[READ], Operations, 5000
[READ], AverageLatency(us), 1070.52889804
[READ], MinLatency(us), 142
[READ], MaxLatency(us), 327167
[READ], 95thPercentileLatency(us), 2071
[READ], 99thPercentileLatency(us), 6539
[READ], Return=OK, 5000
The same picture show all other test - you could see details below.
2.Looks like it could make negative effect if we try to use the feature if we
set *hbase.lru.cache.heavy.eviction.count.limit*=0 and
*hbase.lru.cache.heavy.eviction.mb.size.limit*=1 and doing sporadly short
reading the same data. I meant when size BC=3 and we read block 1,2,3,4,3,4 ...
4,3,2,1,2,1 ... 1,2,3,4,3,4... In this scenario better save all blocks. But
this parameters will skip blocks which we will need quite soon. My opinion - it
is extremely good for massive long-term reading on powerful servers. For short
reading small amount of date too small values of the parameters could be
pathological.
3. If I understand you correct - you meant that after compaction real blocks
offset changed. But when HFiles compacted anyway all blocks removed from BC too.
4.Now we have two parameters for tuning:
*hbase.lru.cache.heavy.eviction.count.limit* - it controls how soon we want to
see eviction rate reduce. If we know that our load pattern is only long term
reading, we can set it 0. It means if we are reading - it is for a long time.
But if we have some times short reading the same data and some times long-term
reading - we have to divide it by this parameter. For example we know - our
short reading used to about 1 min, we have to set the param about 10 and it
will enable the feature only for long time massive reading.
*hbase.lru.cache.heavy.eviction.mb.size.limit* - it lets to control when we
sure that GC will be suffer. For weak PC it could be about 50-100 MB. For
powerful servers 300-500 MB.
I added some useful information into logging:
{color:#871094}LOG{color}.info({color:#067d17}"BlockCache evicted (MB): {},
overhead (%) {}, " {color}+
{color:#067d17}"heavy eviction counter {}, " {color}+
{color:#067d17}"current caching DataBlock (%): {}"{color},
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount,
{color:#00}cache{color}.{color:#871094}cacheDataBlockPercent{color});
It will help to understand what kind of values we have and how to tune it.
5. I think it is pretty good idea. Give me time, please, to do tests and check
what will be.
Well, I will post information about the tests in the next message.
was (Author: pustota):
Seems like our trouble with the servers for a long time and I've decided
install HBase on my home PC.
Another important point - I have done the algorithm,
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120590#comment-17120590
]
Danil Lipovoy edited comment on HBASE-23887 at 5/31/20, 8:43 PM:
-
All tests below have done on my home PC: _AMD Ryzen 7 2700X Eight-Core
Processor (3150 MHz, 16 threads)._
Logic of auto-scaling (see describe here):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation. Will be fine add more logic here.
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables (32 regions each):
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_requestdistribution=uniform_
Workload scenario "z":
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_requestdistribution=zipfian_
Workload scenario "l":
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_requestdistribution=latest_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature:
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
!requests_100p.png!
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100 | no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 | start reading but
*count.limit* haven't reached.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 | *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction counter: 6, current caching DataBlock (%): 91
[GitHub] [hbase] Apache-HBase commented on pull request #1818: HBASE-24456 : Create ImmutableScan and use it for CustomizedScanInfoBuilder
Apache-HBase commented on pull request #1818: URL: https://github.com/apache/hbase/pull/1818#issuecomment-636525402 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 16s | master passed | | +1 :green_heart: | compile | 1m 29s | master passed | | +1 :green_heart: | shadedjars | 6m 21s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 1s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 53s | the patch passed | | +1 :green_heart: | compile | 1m 26s | the patch passed | | +1 :green_heart: | javac | 1m 26s | the patch passed | | +1 :green_heart: | shadedjars | 6m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 11s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 200m 10s | hbase-server in the patch passed. | | | | 230m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1818 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c22026e67a58 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/2/testReport/ | | Max. process+thread count | 3609 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1819: HBASE-24478 The regionInfo parameter for MasterProcedureScheduler#wai…
Apache-HBase commented on pull request #1819: URL: https://github.com/apache/hbase/pull/1819#issuecomment-636524347 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 40s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 24s | master passed | | +1 :green_heart: | compile | 1m 1s | master passed | | +1 :green_heart: | shadedjars | 6m 11s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 2s | the patch passed | | +1 :green_heart: | compile | 1m 1s | the patch passed | | +1 :green_heart: | javac | 1m 1s | the patch passed | | +1 :green_heart: | shadedjars | 6m 10s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 202m 17s | hbase-server in the patch passed. | | | | 229m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1819 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e56f30cbf0be 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/2/testReport/ | | Max. process+thread count | 3972 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120565#comment-17120565
]
Danil Lipovoy edited comment on HBASE-23887 at 5/31/20, 8:06 PM:
-
Seems like our trouble with the servers for a long time and I've decided
install HBase on my home PC.
Another important point - I have done the algorithm, that I posted above (will
add changes to PR quite soon). It is good when numbers of reading requests are
changing. Looks like the new approach copes well with wide variety kind of
situation (a lot of tests in the next messages after answers).
1. I'm nor sure, but maybe it is because first few seconds, while BlockCache
is empty, my old version of realization prevented effective populating the BC.
I mean it was skipping blocks when eviction is not running - and a lot of
blocks could be cached but were lost. With the new approach the problems has
gone. For example:
This is when 100% of data caching (uniform distribution):
[OVERALL], RunTime(ms), 1506417
[OVERALL], Throughput(ops/sec), 33191.34077748724
[TOTAL_GCS_PS_Scavenge], Count, 8388
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 12146
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.8062840501667201
[TOTAL_GCS_PS_MarkSweep], Count, 1
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 22
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.0014604189942094387
[TOTAL_GCs], Count, 8389
[TOTAL_GC_TIME], Time(ms), 12168
[TOTAL_GC_TIME_%], Time(%), 0.8077444691609296
[READ], Operations, 5000
[READ], AverageLatency(us), 1503.45024378
[READ], MinLatency(us), 137
[READ], MaxLatency(us), 383999
[READ], 95thPercentileLatency(us), 2231
[READ], 99thPercentileLatency(us), 13503
[READ], Return=OK, 5000
The same table with the patch:
[OVERALL], RunTime(ms), 1073257
[OVERALL], Throughput(ops/sec), 46587.1641181935
[TOTAL_GCS_PS_Scavenge], Count, 7201
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 9799
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.9130152423883563
[TOTAL_GCS_PS_MarkSweep], Count, 1
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 23
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.002143009549436901
[TOTAL_GCs], Count, 7202
[TOTAL_GC_TIME], Time(ms), 9822
[TOTAL_GC_TIME_%], Time(%), 0.9151582519377931
[READ], Operations, 5000
[READ], AverageLatency(us), 1070.52889804
[READ], MinLatency(us), 142
[READ], MaxLatency(us), 327167
[READ], 95thPercentileLatency(us), 2071
[READ], 99thPercentileLatency(us), 6539
[READ], Return=OK, 5000
The same picture all other test - you could see details below.
2.Looks like it could make negative effect if we try to use the feature if we
set *hbase.lru.cache.heavy.eviction.count.limit*=0 and
*hbase.lru.cache.heavy.eviction.mb.size.limit*=1 and doing sporadly short
reading the same data. I meant when size BC=3 and we read block 1,2,3,4,3,4 ...
4,3,2,1,2,1 ... 1,2,3,4,3,4... In this scenario better save all blocks. But
this parameters will skip blocks which we will need quite soon. My opinion - it
is extremely good for massive long-term reading on powerful servers. For short
reading small amount of date too small values of the parameters could be
pathological.
3. If I understand you correct - you meant that after compaction real blocks
offset changed. But when HFiles compacted anyway all blocks removed from BC too.
4.Now we have two parameters for tuning:
*hbase.lru.cache.heavy.eviction.count.limit* - it controls how soon we want to
see eviction rate reduce. If we know that our load pattern is only long term
reading, we can set it 0. It means if we are reading - it is for a long time.
But if we have some times short reading the same data and some times long-term
reading - we have to divide it by this parameter. For example we know - our
short reading used to about 1 min, we have to set the param about 10 and it
will enable the feature only for long time massive reading.
*hbase.lru.cache.heavy.eviction.mb.size.limit* - it lets to control when we
sure that GC will be suffer. For weak CPU it could be about 50-100 MB. For
powerful servers 300-500 MB.
I added some useful information into logging:
{color:#871094}LOG{color}.info({color:#067d17}"BlockCache evicted (MB): {},
overhead (%) {}, " {color}+
{color:#067d17}"heavy eviction counter {}, " {color}+
{color:#067d17}"current caching DataBlock (%): {}"{color},
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount,
{color:#00}cache{color}.{color:#871094}cacheDataBlockPercent{color});
It will help to understand what kind of values we have and how to tune it.
4. I think it is pretty good idea. Give me time, please, to do tests and check
what will be.
Well, I will post information about the tests in the next message.
was (Author: pustota):
Seems like our trouble with the servers for a long time and I've decided
install HBase on my home PC.
Another important point - I have done the algorithm, that I
[GitHub] [hbase] saintstack commented on a change in pull request #1814: HBASE-20904 Prometheus /metrics http endpoint for monitoring
saintstack commented on a change in pull request #1814: URL: https://github.com/apache/hbase/pull/1814#discussion_r432978537 ## File path: hbase-common/src/main/resources/hbase-default.xml ## @@ -1727,6 +1727,15 @@ possible configurations would overwhelm and obscure the important. ThreadPool. + +hbase.http.enable.prometheus.servlets +false + + Enable prometheus servlets /prom and /prom2 for prometheus based monitoring. + /prom is based on new HBase metrics API and all metrics are not exported for now. + /prom2 is based on the old hadoop2 metrics API and has all the metrics. Review comment: Is 'new metrics' our use of 'hadoop metrics', a move we made years ago? What versions in release do the old way of metrics? Thanks. ## File path: hbase-common/src/main/resources/hbase-default.xml ## @@ -1727,6 +1727,15 @@ possible configurations would overwhelm and obscure the important. ThreadPool. + +hbase.http.enable.prometheus.servlets +false + + Enable prometheus servlets /prom and /prom2 for prometheus based monitoring. + /prom is based on new HBase metrics API and all metrics are not exported for now. + /prom2 is based on the old hadoop2 metrics API and has all the metrics. Review comment: Why not /prom and /prom-old ? Why have the old at all (maybe you want to backport this)? When /prom and /prom2, w/o close reading, users will think they need to read from /prom2 because 2 is a later number than no number? Also, can you say more on old vs new metrics. I'm not clear. Would be good to also clean up the description in here so clear to the casual reader. Thakns for adding the flag. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #1814: HBASE-20904 Prometheus /metrics http endpoint for monitoring
saintstack commented on pull request #1814: URL: https://github.com/apache/hbase/pull/1814#issuecomment-636519732 Sorry you didn't response on mailing list. Yeah, i think it should be optional opt-in. We publish metrics on the /metrics servlet, we then also publish to jmx, we also ship metrics over the heartbeat to the master. Each formatting takes up resources -- mem, cpu -- so unless the prometheus custom interface is being consumed, lets not have the general service pay the price. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1783: HBASE-24436 The store file open and close thread pool should be share…
Apache-HBase commented on pull request #1783: URL: https://github.com/apache/hbase/pull/1783#issuecomment-636519452 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 30s | master passed | | +1 :green_heart: | compile | 0m 57s | master passed | | +1 :green_heart: | shadedjars | 5m 39s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 35s | the patch passed | | +1 :green_heart: | compile | 0m 58s | the patch passed | | +1 :green_heart: | javac | 0m 58s | the patch passed | | +1 :green_heart: | shadedjars | 6m 17s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 7m 40s | hbase-server in the patch failed. | | | | 31m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/7/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1783 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d4c29e38e5cf 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4f49a96258 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/7/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/7/testReport/ | | Max. process+thread count | 685 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/7/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1783: HBASE-24436 The store file open and close thread pool should be share…
Apache-HBase commented on pull request #1783: URL: https://github.com/apache/hbase/pull/1783#issuecomment-636519476 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 57s | master passed | | +1 :green_heart: | compile | 1m 2s | master passed | | +1 :green_heart: | shadedjars | 5m 46s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 38s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 1m 5s | the patch passed | | +1 :green_heart: | javac | 1m 5s | the patch passed | | +1 :green_heart: | shadedjars | 5m 47s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 7m 6s | hbase-server in the patch failed. | | | | 31m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/7/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1783 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e3fa9e94a0f2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4f49a96258 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/7/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/7/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/7/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/7/testReport/ | | Max. process+thread count | 600 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/7/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1783: HBASE-24436 The store file open and close thread pool should be share…
Apache-HBase commented on pull request #1783: URL: https://github.com/apache/hbase/pull/1783#issuecomment-636519538 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 40s | master passed | | +1 :green_heart: | checkstyle | 1m 10s | master passed | | +1 :green_heart: | spotbugs | 1m 57s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 21s | the patch passed | | +1 :green_heart: | checkstyle | 1m 8s | the patch passed | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 10s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 32m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/7/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1783 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux f4a9794bed25 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4f49a96258 | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/7/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1783: HBASE-24436 The store file open and close thread pool should be share…
Apache-HBase commented on pull request #1783: URL: https://github.com/apache/hbase/pull/1783#issuecomment-636515170 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 34s | master passed | | +1 :green_heart: | checkstyle | 1m 9s | master passed | | +1 :green_heart: | spotbugs | 2m 2s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 37s | the patch passed | | +1 :green_heart: | checkstyle | 1m 7s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 5s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 33m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1783 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 3428f925ef84 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4f49a96258 | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1818: HBASE-24456 : Create ImmutableScan and use it for CustomizedScanInfoBuilder
Apache-HBase commented on pull request #1818: URL: https://github.com/apache/hbase/pull/1818#issuecomment-636515189 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 13s | master passed | | +1 :green_heart: | compile | 1m 32s | master passed | | +1 :green_heart: | shadedjars | 5m 48s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 28s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 57s | the patch passed | | +1 :green_heart: | compile | 1m 31s | the patch passed | | +1 :green_heart: | javac | 1m 31s | the patch passed | | +1 :green_heart: | shadedjars | 5m 41s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 27s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 4s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 125m 11s | hbase-server in the patch passed. | | | | 154m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1818 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9826cd6383a2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/2/testReport/ | | Max. process+thread count | 4434 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1783: HBASE-24436 The store file open and close thread pool should be share…
Apache-HBase commented on pull request #1783: URL: https://github.com/apache/hbase/pull/1783#issuecomment-636515004 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 12s | master passed | | +1 :green_heart: | compile | 1m 2s | master passed | | +1 :green_heart: | shadedjars | 5m 43s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 42s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 0s | the patch passed | | +1 :green_heart: | compile | 1m 2s | the patch passed | | +1 :green_heart: | javac | 1m 2s | the patch passed | | +1 :green_heart: | shadedjars | 5m 44s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 7m 37s | hbase-server in the patch failed. | | | | 32m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1783 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5c7427244d85 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4f49a96258 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/6/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/6/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/6/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/6/testReport/ | | Max. process+thread count | 671 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1783: HBASE-24436 The store file open and close thread pool should be share…
Apache-HBase commented on pull request #1783: URL: https://github.com/apache/hbase/pull/1783#issuecomment-636514608 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 27s | master passed | | +1 :green_heart: | compile | 0m 56s | master passed | | +1 :green_heart: | shadedjars | 5m 29s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 21s | the patch passed | | +1 :green_heart: | compile | 0m 54s | the patch passed | | +1 :green_heart: | javac | 0m 54s | the patch passed | | +1 :green_heart: | shadedjars | 5m 30s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 6m 33s | hbase-server in the patch failed. | | | | 29m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1783 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 22acf384d259 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4f49a96258 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/6/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/6/testReport/ | | Max. process+thread count | 671 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bharathv merged pull request #1815: HBASE-24477: Move ConfigurationObserver and related classes to hbase-c…
bharathv merged pull request #1815: URL: https://github.com/apache/hbase/pull/1815 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120590#comment-17120590
]
Danil Lipovoy edited comment on HBASE-23887 at 5/31/20, 5:27 PM:
-
All tests below have done on my home PC: _AMD Ryzen 7 2700X Eight-Core
Processor (3150 MHz, 16 threads)._
Logic of autoscaling (see describe here):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (bytesFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation. Will be fine add more logic here.
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables (32 regions each):
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_requestdistribution=uniform_
Workload scenario "z":
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_requestdistribution=zipfian_
Workload scenario "l":
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_requestdistribution=latest_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature.
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
!requests_100p.png!
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100 | no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 | start reading but
*count.limit* haven't reach.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 | *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction counter: 6, current caching DataBlock (%): 91
[GitHub] [hbase] Apache-HBase commented on pull request #1818: HBASE-24456 : Create ImmutableScan and use it for CustomizedScanInfoB…
Apache-HBase commented on pull request #1818: URL: https://github.com/apache/hbase/pull/1818#issuecomment-636501303 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 16s | master passed | | +1 :green_heart: | checkstyle | 1m 46s | master passed | | +1 :green_heart: | spotbugs | 3m 25s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 53s | the patch passed | | -0 :warning: | checkstyle | 0m 29s | hbase-client: The patch generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 38s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 4m 4s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 43m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1818 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 063de7c0d041 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-client.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24454) BucketCache disabled instantly before error duration toleration is reached due to timing issue
[
https://issues.apache.org/jira/browse/HBASE-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120599#comment-17120599
]
Hudson commented on HBASE-24454:
Results for branch branch-2.3
[build #115 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/115/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/115/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/115/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/115/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/115/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> BucketCache disabled instantly before error duration toleration is reached
> due to timing issue
> --
>
> Key: HBASE-24454
> URL: https://issues.apache.org/jira/browse/HBASE-24454
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 1.4.10
> Environment: We saw this in HBase 1.4.10 (EMR 5.28.1), but I believe
> the newest code still has this problem.
>Reporter: Jacob LeBlanc
>Assignee: Jacob LeBlanc
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.6
>
>
> We saw an instance where BucketCache was disabled after only two IO error
> were encountered at nearly the same time. The following shows all errors from
> a region server log for the 2020-05-26 17:00 hour. Notice that there are no
> other errors until the 17:14:50 at which time the BucketCache is disabled
> because it claims duration time has exceeded 6 ms:
> $ grep ERROR hbase-hbase-regionserver-ip-172-20-113-147.log.2020-05-26-17
> 2020-05-26 17:14:50,396 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: Failed reading block
> 9adfad3a603047cfa0c98b16da0b0974_13786 from bucket cache
> 2020-05-26 17:14:50,397 ERROR
> [regionserver/ip-172-20-113-147.us-west-2.compute.internal/172.20.113.147:16020-BucketCacheWriter-0]
> bucket.BucketCache: Failed syncing IO engine
> 2020-05-26 17:14:50,400 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: Failed reading block
> 9adfad3a603047cfa0c98b16da0b0974_13786 from bucket cache
> 2020-05-26 17:14:50,401 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: IO errors duration time has exceeded 6ms, disabling
> cache, please check your IOEngine
> The region server is very busy so it should be constantly getting successful
> reads and writes in the bucket cache (it is not as though there was some long
> ago error and then no successful IO to clear the error flag).
> We are running a busy EMR cluster backed by S3. A bucketcache getting
> disabled is a huge performance issue because all reads must go to S3.
> Looking at the code, I believe I've found a timing issue. Here is the code
> for checking and setting the ioErrorStartTime (taken from BucketCache.java):
>
> {code:java}
> /**
>* Check whether we tolerate IO error this time. If the duration of IOEngine
>* throwing errors exceeds ioErrorsDurationTimeTolerated, we will disable
> the
>* cache
>*/
> private void checkIOErrorIsTolerated() {
> long now = EnvironmentEdgeManager.currentTime();
> if (this.ioErrorStartTime > 0) {
> if (cacheEnabled && (now - ioErrorStartTime) >
> this.ioErrorsTolerationDuration) {
> LOG.error("IO errors duration time has exceeded " +
> ioErrorsTolerationDuration +
> "ms, disabling cache, please check your IOEngine");
> disableCache();
> }
> } else {
> this.ioErrorStartTime = now;
> }
> }
> {code}
>
> And here is the code for clearing the ioErrorStartTime when a successful read
> or write is done:
> {code:java}
> if (this.ioErrorStartTime > 0) {
> ioErrorStartTime = -1;
> }
> {code}
> Notice that that if ioErrorStartTime is set to -1 after the first if
> statement in checkIOErrorIsTolerated but before (now - ioErrorStartTime),
> then (now - ioErrorStartTime) will evaluate to (now + 1) resulting in the
> code thinking it has exceeded ioErrorsTolerationDuration.
--
[jira] [Commented] (HBASE-24475) Clean up the master thread name getting in SplitLogManager and AssignmentManager
[
https://issues.apache.org/jira/browse/HBASE-24475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120600#comment-17120600
]
Hudson commented on HBASE-24475:
Results for branch branch-2.3
[build #115 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/115/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/115/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/115/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/115/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/115/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Clean up the master thread name getting in SplitLogManager and
> AssignmentManager
>
>
> Key: HBASE-24475
> URL: https://issues.apache.org/jira/browse/HBASE-24475
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.0
>
>
> As discussed in PR of HBASE-24451, seems just use
> master.getServerName().toShortString() is enough.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1819: HBASE-24478 The regionInfo parameter for MasterProcedureScheduler#wai…
Apache-HBase commented on pull request #1819: URL: https://github.com/apache/hbase/pull/1819#issuecomment-636500690 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 9s | master passed | | +1 :green_heart: | compile | 1m 15s | master passed | | +1 :green_heart: | shadedjars | 6m 36s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 46s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 48s | the patch passed | | +1 :green_heart: | compile | 1m 15s | the patch passed | | +1 :green_heart: | javac | 1m 15s | the patch passed | | +1 :green_heart: | shadedjars | 6m 49s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 50s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 12m 49s | hbase-server in the patch failed. | | | | 43m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1819 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8a2d486d8dab 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/2/testReport/ | | Max. process+thread count | 434 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1819: HBASE-24478 The regionInfo parameter for MasterProcedureScheduler#wai…
Apache-HBase commented on pull request #1819: URL: https://github.com/apache/hbase/pull/1819#issuecomment-636499260 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | master passed | | +1 :green_heart: | checkstyle | 1m 6s | master passed | | +1 :green_heart: | spotbugs | 1m 57s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 19s | the patch passed | | -0 :warning: | checkstyle | 1m 3s | hbase-server: The patch generated 2 new + 3 unchanged - 0 fixed = 5 total (was 3) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 8s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 8s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 32m 11s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1819 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 831f824b9ac2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120590#comment-17120590
]
Danil Lipovoy edited comment on HBASE-23887 at 5/31/20, 5:05 PM:
-
All tests below have done on my home PC: _AMD Ryzen 7 2700X Eight-Core
Processor (3150 MHz, 16 threads)._
Logic of autoscaling (see describe here):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (bytesFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables (32 regions each):
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_requestdistribution=uniform_
Workload scenario "z":
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_requestdistribution=zipfian_
Workload scenario "l":
_operationcount=50 000 000 (for tbl4 just 500 000 because there is batch 100)_
_readproportion=1_
_requestdistribution=latest_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature.
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
!requests_100p.png!
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100 | no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 | start reading but
*count.limit* haven't reach.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 | *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction counter: 6, current caching DataBlock (%): 91
LruBlockCache: BlockCache evicted
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120590#comment-17120590
]
Danil Lipovoy edited comment on HBASE-23887 at 5/31/20, 5:01 PM:
-
All tests below have done on: _AMD Ryzen 7 2700X Eight-Core Processor (3150
MHz, 16 threads)._
Logic of autoscaling (see describe here):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (bytesFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables:
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=uniform_
Workload scenario "z":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=zipfian_
Workload scenario "l":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=latest_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature.
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
!requests_100p.png!
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100 | no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 | start reading but
*count.limit* haven't reach.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 | *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction counter: 6, current caching DataBlock (%): 91
LruBlockCache: BlockCache evicted (MB): 7722, overhead (%): 3761, heavy
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120590#comment-17120590
]
Danil Lipovoy edited comment on HBASE-23887 at 5/31/20, 4:56 PM:
-
All tests below have done on: _AMD Ryzen 7 2700X Eight-Core Processor (3150
MHz, 16 threads)._
Logic of autoscaling (see describe here):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (bytesFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables:
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=uniform_
Workload scenario "z":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=zipfian_
Workload scenario "l":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=latest_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature.
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
!requests_100p.png!
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100 | no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 | start reading but
*count.limit* haven't reach.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 | *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction counter: 6, current caching DataBlock (%): 91
LruBlockCache: BlockCache evicted (MB): 7722, overhead (%): 3761, heavy
[jira] [Comment Edited] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120590#comment-17120590
]
Danil Lipovoy edited comment on HBASE-23887 at 5/31/20, 4:55 PM:
-
All tests below have done on: _AMD Ryzen 7 2700X Eight-Core Processor (3150
MHz, 16 threads)._
Logic of autoscaling (see describe here):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (bytesFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables:
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=uniform_
Workload scenario "z":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=zipfian_
Workload scenario "l":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=latest_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature.
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
!requests_100p.png!
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): 0, heavy eviction
counter: 0, current caching DataBlock (%): 100 | no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): 0, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): 0, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 | start reading but
*count.limit* haven't reach.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 | *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction counter: 6, current caching DataBlock (%): 91
LruBlockCache: BlockCache evicted (MB): 7722, overhead (%): 3761, heavy
eviction
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120591#comment-17120591
]
Danil Lipovoy commented on HBASE-23887:
---
All tests below have done on: _AMD Ryzen 7 2700X Eight-Core Processor (3150
MHz, 16 threads)._
Logic of autoscaling (see describe here):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (bytesFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables:
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=uniform_
Workload scenario "z":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=zipfian_
Workload scenario "l":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=latest_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature.
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100 <- no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 <- start reading but
*count.limit* haven't reach.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 <- *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction counter: 6, current caching DataBlock (%): 91
LruBlockCache: BlockCache evicted (MB): 7722, overhead (%): 3761, heavy
eviction counter: 7, current caching DataBlock (%): 88
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: requests_100p.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png, requests_100p.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Comment: was deleted
(was: All tests below have done on: _AMD Ryzen 7 2700X Eight-Core Processor
(3150 MHz, 16 threads)._
Logic of autoscaling (see describe here):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (bytesFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables:
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=uniform_
Workload scenario "z":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=zipfian_
Workload scenario "l":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=latest_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature.
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100 <- no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 <- start reading but
*count.limit* haven't reach.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 <- *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction counter: 6, current caching DataBlock (%): 91
LruBlockCache: BlockCache evicted (MB): 7722, overhead (%): 3761, heavy
eviction counter: 7, current caching DataBlock (%): 88
LruBlockCache:
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120590#comment-17120590
]
Danil Lipovoy commented on HBASE-23887:
---
All tests below have done on: _AMD Ryzen 7 2700X Eight-Core Processor (3150
MHz, 16 threads)._
Logic of autoscaling (see describe
[here|https://issues.apache.org/jira/browse/HBASE-23887?focusedCommentId=17110503=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17110503]):
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)
{
if (cacheDataBlockPercent != 100 && buf.getBlockType().isData()) {
if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) {
return;
}
}
...{code}
And how to calculate cacheDataBlockPercent is here:
{code:java}
public void run() {
...
LruBlockCache cache = this.cache.get();
if (cache == null) break;
bytesFreed = cache.evict();
long stopTime = System.currentTimeMillis(); // We need of control the time of
working cache.evict()
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (stopTime - startTime <= 1000 * 10 - 1) {
mbFreedSum += bytesFreed/1024/1024; // Now went less then 10 sec, just sum up
and thats all
} else {
freedDataOverheadPercent = (int) (mbFreedSum * 100 /
cache.heavyEvictionBytesSizeLimit) - 100;
if (mbFreedSum > cache.heavyEvictionBytesSizeLimit) {
heavyEvictionCount++;
if (heavyEvictionCount > cache.heavyEvictionCountLimit) {
if (freedDataOverheadPercent > 100) {
cache.cacheDataBlockPercent -= 3;
} else {
if (freedDataOverheadPercent > 50) {
cache.cacheDataBlockPercent -= 1;
} else {
if (freedDataOverheadPercent < 30) {
cache.cacheDataBlockPercent += 1;
}
}
}
}
} else {
if (bytesFreedSum > cache.heavyEvictionBytesSizeLimit * 0.5 &&
cache.cacheDataBlockPercent < 50) {
cache.cacheDataBlockPercent += 5; // It help prevent some premature
escape from accidental fluctuation
} else {
heavyEvictionCount = 0;
cache.cacheDataBlockPercent = 100;
}
}
LOG.info("BlockCache evicted (MB): {}, overhead (%): {}, " +
"heavy eviction counter: {}, " +
"current caching DataBlock (%): {}",
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount, cache.cacheDataBlockPercent);
mbFreedSum = 0;
startTime = stopTime;
}
{code}
I prepared 4 tables:
tbl1 - 200 mln records, 100 bytes each. Total size 30 Gb.
tbl2 - 20 mln records, 500 bytes each. Total size 10.4 Gb.
tbl3 - 100 mln records, 100 bytes each. Total size 15.4 Gb.
tbl4 - the same like tbl3 but I use it for testing work with batches
(batchSize=100)
Workload scenario "u":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=uniform_
Workload scenario "z":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=zipfian_
Workload scenario "l":
_operationcount=5000 (for tbl4 just 50 because there is batch 100)_
_readproportion=1_
_requestdistribution=latest_
Then I run all tables with all scenarios on original version (total 4*3=12
tests) and 12 with the feature.
*hbase.lru.cache.heavy.eviction.count.limit* = 3
*hbase.lru.cache.heavy.eviction.mb.size.limit* = 200
Performance results:
!requests_100p.png!
We could see that on the second graph lines have some a step at the begin. It
is because works auto scaling.
Let see the log of RegionServer:
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100 <- no load, do nothing
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 0, overhead (%): -100, heavy eviction
counter: 0, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 229, overhead (%): 14, heavy eviction
counter: 1, current caching DataBlock (%): 100 <- start reading but
*count.limit* haven't reach.
LruBlockCache: BlockCache evicted (MB): 6958, overhead (%): 3379, heavy
eviction counter: 2, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8117, overhead (%): 3958, heavy
eviction counter: 3, current caching DataBlock (%): 100
LruBlockCache: BlockCache evicted (MB): 8713, overhead (%): 4256, heavy
eviction counter: 4, current caching DataBlock (%): 97 <- *count.limit* have
reached, decrease on 3%
LruBlockCache: BlockCache evicted (MB): 8723, overhead (%): 4261, heavy
eviction counter: 5, current caching DataBlock (%): 94
LruBlockCache: BlockCache evicted (MB): 8318, overhead (%): 4059, heavy
eviction counter: 6, current
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: eviction_100p.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, eviction_100p.png, gc_100p.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1818: HBASE-24456 : Create ImmutableScan and use it for CustomizedScanInfoB…
Apache-HBase commented on pull request #1818: URL: https://github.com/apache/hbase/pull/1818#issuecomment-636495194 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 9s | master passed | | +1 :green_heart: | compile | 1m 27s | master passed | | +1 :green_heart: | shadedjars | 6m 10s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 1s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 59s | the patch passed | | +1 :green_heart: | compile | 1m 26s | the patch passed | | +1 :green_heart: | javac | 1m 26s | the patch passed | | +1 :green_heart: | shadedjars | 6m 13s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 2s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 10s | hbase-client in the patch passed. | | -1 :x: | unit | 225m 46s | hbase-server in the patch failed. | | | | 256m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1818 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 454e3f0db196 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/1/testReport/ | | Max. process+thread count | 2789 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1783: HBASE-24436 The store file open and close thread pool should be share…
Apache-HBase commented on pull request #1783: URL: https://github.com/apache/hbase/pull/1783#issuecomment-636495075 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 46s | master passed | | +1 :green_heart: | compile | 1m 15s | master passed | | +1 :green_heart: | shadedjars | 7m 21s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 21s | the patch passed | | +1 :green_heart: | compile | 1m 7s | the patch passed | | +1 :green_heart: | javac | 1m 7s | the patch passed | | +1 :green_heart: | shadedjars | 7m 26s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 6m 48s | hbase-server in the patch failed. | | | | 36m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1783 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1acaec710863 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/5/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/5/testReport/ | | Max. process+thread count | 668 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1783: HBASE-24436 The store file open and close thread pool should be share…
Apache-HBase commented on pull request #1783: URL: https://github.com/apache/hbase/pull/1783#issuecomment-636494911 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 41s | master passed | | +1 :green_heart: | checkstyle | 1m 13s | master passed | | +1 :green_heart: | spotbugs | 2m 4s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 41s | the patch passed | | -0 :warning: | checkstyle | 1m 14s | hbase-server: The patch generated 4 new + 191 unchanged - 0 fixed = 195 total (was 191) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 4s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 34m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1783 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 1b3e0fb81f94 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1819: HBASE-24478 The regionInfo parameter for MasterProcedureScheduler#wai…
Apache-HBase commented on pull request #1819: URL: https://github.com/apache/hbase/pull/1819#issuecomment-636494729 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 49s | master passed | | +1 :green_heart: | compile | 0m 55s | master passed | | +1 :green_heart: | shadedjars | 5m 40s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 29s | the patch passed | | +1 :green_heart: | compile | 0m 54s | the patch passed | | +1 :green_heart: | javac | 0m 54s | the patch passed | | +1 :green_heart: | shadedjars | 5m 36s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 138m 5s | hbase-server in the patch passed. | | | | 162m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1819 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8ea5bdd73bd8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/1/testReport/ | | Max. process+thread count | 4651 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1783: HBASE-24436 The store file open and close thread pool should be share…
Apache-HBase commented on pull request #1783: URL: https://github.com/apache/hbase/pull/1783#issuecomment-636494546 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 54s | master passed | | +1 :green_heart: | compile | 1m 5s | master passed | | +1 :green_heart: | shadedjars | 5m 49s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 39s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | the patch passed | | +1 :green_heart: | compile | 1m 5s | the patch passed | | +1 :green_heart: | javac | 1m 5s | the patch passed | | +1 :green_heart: | shadedjars | 5m 45s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 7m 8s | hbase-server in the patch failed. | | | | 31m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1783 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a8b587006e3c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/5/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/5/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/5/testReport/ | | Max. process+thread count | 639 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: gc_100p.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, gc_100p.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1818: HBASE-24456 : Create ImmutableScan and use it for CustomizedScanInfoB…
Apache-HBase commented on pull request #1818: URL: https://github.com/apache/hbase/pull/1818#issuecomment-636494325 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 54s | master passed | | +1 :green_heart: | compile | 1m 47s | master passed | | +1 :green_heart: | shadedjars | 6m 35s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 33s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 45s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 43s | the patch passed | | +1 :green_heart: | compile | 1m 44s | the patch passed | | +1 :green_heart: | javac | 1m 45s | the patch passed | | +1 :green_heart: | shadedjars | 6m 45s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 31s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 50s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 36s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 214m 0s | hbase-server in the patch passed. | | | | 249m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1818 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 07e23dc5591e 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/1/testReport/ | | Max. process+thread count | 3315 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1819: HBASE-24478 The regionInfo parameter for MasterProcedureScheduler#wai…
Apache-HBase commented on pull request #1819: URL: https://github.com/apache/hbase/pull/1819#issuecomment-636493877 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 15s | master passed | | +1 :green_heart: | compile | 1m 4s | master passed | | +1 :green_heart: | shadedjars | 5m 45s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 41s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 1m 2s | the patch passed | | +1 :green_heart: | javac | 1m 2s | the patch passed | | +1 :green_heart: | shadedjars | 5m 45s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 129m 40s | hbase-server in the patch failed. | | | | 155m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1819 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 65db66f6b8d3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/1/testReport/ | | Max. process+thread count | 3934 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: eviction_100p.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, eviction_100p.png, read_requests_100pBC_vs_23pBC.png,
> requests_100p.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: eviction_100p.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> eviction_100p.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1783: HBASE-24436 The store file open and close thread pool should be share…
Apache-HBase commented on pull request #1783: URL: https://github.com/apache/hbase/pull/1783#issuecomment-636490239 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 36s | master passed | | +1 :green_heart: | compile | 1m 4s | master passed | | +1 :green_heart: | shadedjars | 7m 19s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 47s | the patch passed | | +1 :green_heart: | compile | 1m 9s | the patch passed | | +1 :green_heart: | javac | 1m 9s | the patch passed | | +1 :green_heart: | shadedjars | 7m 31s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 7m 56s | hbase-server in the patch failed. | | | | 37m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1783 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 629327e74313 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/4/testReport/ | | Max. process+thread count | 682 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1783: HBASE-24436 The store file open and close thread pool should be share…
Apache-HBase commented on pull request #1783: URL: https://github.com/apache/hbase/pull/1783#issuecomment-636490011 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 54s | master passed | | +1 :green_heart: | checkstyle | 1m 12s | master passed | | +1 :green_heart: | spotbugs | 2m 6s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 41s | the patch passed | | -0 :warning: | checkstyle | 1m 13s | hbase-server: The patch generated 4 new + 191 unchanged - 0 fixed = 195 total (was 191) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 8s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 15s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 35m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1783 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 8aa61be511d0 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1783: HBASE-24436 The store file open and close thread pool should be share…
Apache-HBase commented on pull request #1783: URL: https://github.com/apache/hbase/pull/1783#issuecomment-636489567 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 16s | master passed | | +1 :green_heart: | compile | 1m 3s | master passed | | +1 :green_heart: | shadedjars | 5m 44s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 42s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 0s | the patch passed | | +1 :green_heart: | compile | 1m 3s | the patch passed | | +1 :green_heart: | javac | 1m 3s | the patch passed | | +1 :green_heart: | shadedjars | 5m 42s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 7m 9s | hbase-server in the patch failed. | | | | 32m 11s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1783 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a99ead93afea 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/4/testReport/ | | Max. process+thread count | 635 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1783/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: requests_100p.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24478) The regionInfo parameter for MasterProcedureScheduler#waitRegions and MasterProcedureScheduler#wakeRegions should be plural
[ https://issues.apache.org/jira/browse/HBASE-24478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] song XinCun updated HBASE-24478: Description: MasterProcedureScheduler#waitRegions and MasterProcedureScheduler#wakeRegions deal with a list of regions, so the variable name of region info should be plural (was: MasterProcedureScheduler#waitRegions deal with a list of regions, so the variable name of region info should be plural) > The regionInfo parameter for MasterProcedureScheduler#waitRegions and > MasterProcedureScheduler#wakeRegions should be plural > > > Key: HBASE-24478 > URL: https://issues.apache.org/jira/browse/HBASE-24478 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Affects Versions: 3.0.0-alpha-1 >Reporter: song XinCun >Assignee: song XinCun >Priority: Minor > > MasterProcedureScheduler#waitRegions and MasterProcedureScheduler#wakeRegions > deal with a list of regions, so the variable name of region info should be > plural -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24478) The regionInfo parameter for MasterProcedureScheduler#waitRegions and MasterProcedureScheduler#wakeRegions should be plural
[ https://issues.apache.org/jira/browse/HBASE-24478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] song XinCun updated HBASE-24478: Summary: The regionInfo parameter for MasterProcedureScheduler#waitRegions and MasterProcedureScheduler#wakeRegions should be plural (was: The regionInfo parameter for MasterProcedureScheduler#waitRegions should be plural ) > The regionInfo parameter for MasterProcedureScheduler#waitRegions and > MasterProcedureScheduler#wakeRegions should be plural > > > Key: HBASE-24478 > URL: https://issues.apache.org/jira/browse/HBASE-24478 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Affects Versions: 3.0.0-alpha-1 >Reporter: song XinCun >Assignee: song XinCun >Priority: Minor > > MasterProcedureScheduler#waitRegions deal with a list of regions, so the > variable name of region info should be plural -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120565#comment-17120565
]
Danil Lipovoy commented on HBASE-23887:
---
Seems like our trouble with the servers for a long time and I've decided
install HBase on my home PC.
Another important point - I have done the algorithm, that I posted above (will
add changes to PR quite soon). It is good when numbers of reading requests are
changing. Looks like the new approach copes well with wide variety kind of
situation (a lot of tests in the next messages after answers).
1. I'm nor sure, but maybe it is because few fist seconds, while BlockCache is
empty, my old version of realisation prevented effective populating the BC. I
mean it was skipping blocks when eviction is not running - and a lot of blocks
could be cached but were lost. With the new approach the probles has gone. For
example:
This is when 100% of data caching (uniform distribution):
[OVERALL], RunTime(ms), 1506417
[OVERALL], Throughput(ops/sec), 33191.34077748724
[TOTAL_GCS_PS_Scavenge], Count, 8388
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 12146
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.8062840501667201
[TOTAL_GCS_PS_MarkSweep], Count, 1
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 22
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.0014604189942094387
[TOTAL_GCs], Count, 8389
[TOTAL_GC_TIME], Time(ms), 12168
[TOTAL_GC_TIME_%], Time(%), 0.8077444691609296
[READ], Operations, 5000
[READ], AverageLatency(us), 1503.45024378
[READ], MinLatency(us), 137
[READ], MaxLatency(us), 383999
[READ], 95thPercentileLatency(us), 2231
[READ], 99thPercentileLatency(us), 13503
[READ], Return=OK, 5000
The same table with the patch:
[OVERALL], RunTime(ms), 1073257
[OVERALL], Throughput(ops/sec), 46587.1641181935
[TOTAL_GCS_PS_Scavenge], Count, 7201
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 9799
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.9130152423883563
[TOTAL_GCS_PS_MarkSweep], Count, 1
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 23
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.002143009549436901
[TOTAL_GCs], Count, 7202
[TOTAL_GC_TIME], Time(ms), 9822
[TOTAL_GC_TIME_%], Time(%), 0.9151582519377931
[READ], Operations, 5000
[READ], AverageLatency(us), 1070.52889804
[READ], MinLatency(us), 142
[READ], MaxLatency(us), 327167
[READ], 95thPercentileLatency(us), 2071
[READ], 99thPercentileLatency(us), 6539
[READ], Return=OK, 5000
The same picture all other test - you could see details below.
2.Looks like it could make negative effect if we try to use the feature if we
set *hbase.lru.cache.heavy.eviction.count.limit*=0 and
*hbase.lru.cache.heavy.eviction.mb.size.limit*=1 and doing sporadly short
reading the same data. I meant when size BC=3 and we read block 1,2,3,4,3,4 ...
4,3,2,1,2,1 ... 1,2,3,4,3,4... In this scenario better save all blocks. But
this parameters will skip blocks which we will need quite soon. My opinion - it
is extremely good for massive long-term reading on powerful servers. For short
reading small amount of date too small values of the parameters could be
pathological.
3. If I understand you correct - you meant that after compaction real blocks
offset changed. But when HFiles compacted anyway all blocks removed from BC too.
4.Now we have two parameters for tuning:
*hbase.lru.cache.heavy.eviction.count.limit* - it controls how soon we want to
see eviction rate reduce. If we know that our load pattern is only long term
reading, we can set it 0. It means if we are reading - it is for a long time.
But if we have some times short reading the same data and some times long-term
reading - we have to divide it by this parameter. For example we know - our
short reading used to about 1 min, we have to set the param about 10 and it
will enable the feature only for long time massive reading.
*hbase.lru.cache.heavy.eviction.mb.size.limit* **- it lets to control when we
sure that GC will be suffer. For weak CPU it could be about 50-100 MB. For
powerful servers 300-500 MB.
I added some useful information into logging:
{color:#871094}LOG{color}.info({color:#067d17}"BlockCache evicted (MB): {},
overhead (%) {}, " {color}+
{color:#067d17}"heavy eviction counter {}, " {color}+
{color:#067d17}"current caching DataBlock (%): {}"{color},
mbFreedSum, freedDataOverheadPercent,
heavyEvictionCount,
{color:#00}cache{color}.{color:#871094}cacheDataBlockPercent{color});
It will help to understand what kind of values we have and how to tune it.
4. I think it is pretty good idea. Give me time, please, to do tests and check
what will be.
Well, I will post information about the tests in the next message.
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue
[jira] [Commented] (HBASE-24454) BucketCache disabled instantly before error duration toleration is reached due to timing issue
[
https://issues.apache.org/jira/browse/HBASE-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120557#comment-17120557
]
Hudson commented on HBASE-24454:
Results for branch master
[build #1742 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1742/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1742/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1600//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1742/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1742/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> BucketCache disabled instantly before error duration toleration is reached
> due to timing issue
> --
>
> Key: HBASE-24454
> URL: https://issues.apache.org/jira/browse/HBASE-24454
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 1.4.10
> Environment: We saw this in HBase 1.4.10 (EMR 5.28.1), but I believe
> the newest code still has this problem.
>Reporter: Jacob LeBlanc
>Assignee: Jacob LeBlanc
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.6
>
>
> We saw an instance where BucketCache was disabled after only two IO error
> were encountered at nearly the same time. The following shows all errors from
> a region server log for the 2020-05-26 17:00 hour. Notice that there are no
> other errors until the 17:14:50 at which time the BucketCache is disabled
> because it claims duration time has exceeded 6 ms:
> $ grep ERROR hbase-hbase-regionserver-ip-172-20-113-147.log.2020-05-26-17
> 2020-05-26 17:14:50,396 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: Failed reading block
> 9adfad3a603047cfa0c98b16da0b0974_13786 from bucket cache
> 2020-05-26 17:14:50,397 ERROR
> [regionserver/ip-172-20-113-147.us-west-2.compute.internal/172.20.113.147:16020-BucketCacheWriter-0]
> bucket.BucketCache: Failed syncing IO engine
> 2020-05-26 17:14:50,400 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: Failed reading block
> 9adfad3a603047cfa0c98b16da0b0974_13786 from bucket cache
> 2020-05-26 17:14:50,401 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: IO errors duration time has exceeded 6ms, disabling
> cache, please check your IOEngine
> The region server is very busy so it should be constantly getting successful
> reads and writes in the bucket cache (it is not as though there was some long
> ago error and then no successful IO to clear the error flag).
> We are running a busy EMR cluster backed by S3. A bucketcache getting
> disabled is a huge performance issue because all reads must go to S3.
> Looking at the code, I believe I've found a timing issue. Here is the code
> for checking and setting the ioErrorStartTime (taken from BucketCache.java):
>
> {code:java}
> /**
>* Check whether we tolerate IO error this time. If the duration of IOEngine
>* throwing errors exceeds ioErrorsDurationTimeTolerated, we will disable
> the
>* cache
>*/
> private void checkIOErrorIsTolerated() {
> long now = EnvironmentEdgeManager.currentTime();
> if (this.ioErrorStartTime > 0) {
> if (cacheEnabled && (now - ioErrorStartTime) >
> this.ioErrorsTolerationDuration) {
> LOG.error("IO errors duration time has exceeded " +
> ioErrorsTolerationDuration +
> "ms, disabling cache, please check your IOEngine");
> disableCache();
> }
> } else {
> this.ioErrorStartTime = now;
> }
> }
> {code}
>
> And here is the code for clearing the ioErrorStartTime when a successful read
> or write is done:
> {code:java}
> if (this.ioErrorStartTime > 0) {
> ioErrorStartTime = -1;
> }
> {code}
> Notice that that if ioErrorStartTime is set to -1 after the first if
> statement in checkIOErrorIsTolerated but before (now - ioErrorStartTime),
> then (now - ioErrorStartTime) will evaluate to (now + 1) resulting in the
> code thinking it has exceeded ioErrorsTolerationDuration.
--
This message was sent by Atlassian
[jira] [Commented] (HBASE-24475) Clean up the master thread name getting in SplitLogManager and AssignmentManager
[
https://issues.apache.org/jira/browse/HBASE-24475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120558#comment-17120558
]
Hudson commented on HBASE-24475:
Results for branch master
[build #1742 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1742/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1742/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1600//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1742/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1742/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Clean up the master thread name getting in SplitLogManager and
> AssignmentManager
>
>
> Key: HBASE-24475
> URL: https://issues.apache.org/jira/browse/HBASE-24475
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.0
>
>
> As discussed in PR of HBASE-24451, seems just use
> master.getServerName().toShortString() is enough.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1819: HBASE-24478 The regionInfo parameter for MasterProcedureScheduler#wai…
Apache-HBase commented on pull request #1819: URL: https://github.com/apache/hbase/pull/1819#issuecomment-636478352 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 44s | master passed | | +1 :green_heart: | checkstyle | 1m 5s | master passed | | +1 :green_heart: | spotbugs | 2m 2s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 20s | the patch passed | | -0 :warning: | checkstyle | 1m 5s | hbase-server: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 6s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 34m 51s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1819 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 580cc6fef0f0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1819/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24478) The regionInfo parameter for MasterProcedureScheduler#waitRegions should be plural
[ https://issues.apache.org/jira/browse/HBASE-24478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120545#comment-17120545 ] song XinCun commented on HBASE-24478: - Same so for MasterProcedureScheduler#wakeRegions > The regionInfo parameter for MasterProcedureScheduler#waitRegions should be > plural > --- > > Key: HBASE-24478 > URL: https://issues.apache.org/jira/browse/HBASE-24478 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Affects Versions: 3.0.0-alpha-1 >Reporter: song XinCun >Assignee: song XinCun >Priority: Minor > > MasterProcedureScheduler#waitRegions deal with a list of regions, so the > variable name of region info should be plural -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1803: [HBASE-24468]Add region info when log meessages in HStore.
Apache-HBase commented on pull request #1803: URL: https://github.com/apache/hbase/pull/1803#issuecomment-636474521 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 19s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 5s | master passed | | +1 :green_heart: | compile | 0m 57s | master passed | | +1 :green_heart: | shadedjars | 6m 5s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 53s | the patch passed | | +1 :green_heart: | compile | 0m 58s | the patch passed | | +1 :green_heart: | javac | 0m 58s | the patch passed | | +1 :green_heart: | shadedjars | 6m 2s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 230m 15s | hbase-server in the patch failed. | | | | 256m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1803/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1803 | | JIRA Issue | HBASE-24468 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 329f3b3edb8c 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1803/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1803/3/testReport/ | | Max. process+thread count | 3824 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1803/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] songxincun opened a new pull request #1819: HBASE-24478 The regionInfo parameter for MasterProcedureScheduler#wai…
songxincun opened a new pull request #1819: URL: https://github.com/apache/hbase/pull/1819 …tRegions should be plural This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24478) The regionInfo parameter for MasterProcedureScheduler#waitRegions should be plural
song XinCun created HBASE-24478: --- Summary: The regionInfo parameter for MasterProcedureScheduler#waitRegions should be plural Key: HBASE-24478 URL: https://issues.apache.org/jira/browse/HBASE-24478 Project: HBase Issue Type: Improvement Components: proc-v2 Affects Versions: 3.0.0-alpha-1 Reporter: song XinCun Assignee: song XinCun MasterProcedureScheduler#waitRegions deal with a list of regions, so the variable name of region info should be plural -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1818: HBASE-24456 : Create ImmutableScan and use it for CustomizedScanInfoB…
Apache-HBase commented on pull request #1818: URL: https://github.com/apache/hbase/pull/1818#issuecomment-636467951 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 38s | master passed | | +1 :green_heart: | checkstyle | 1m 32s | master passed | | +1 :green_heart: | spotbugs | 2m 59s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 22s | the patch passed | | -0 :warning: | checkstyle | 0m 29s | hbase-client: The patch generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 4s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 3m 18s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | | | | 36m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1818 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux fdd4e76e21ad 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-client.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1818/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24456) Immutable Scan as unmodifiable subclass or wrapper of Scan
[ https://issues.apache.org/jira/browse/HBASE-24456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani updated HBASE-24456: - Status: Patch Available (was: In Progress) > Immutable Scan as unmodifiable subclass or wrapper of Scan > -- > > Key: HBASE-24456 > URL: https://issues.apache.org/jira/browse/HBASE-24456 > Project: HBase > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > > We should provide an ImmutableScan as subclass or wrapper of Scan which can > be used specifically by Coprocessors as read only Scan. This suggestion came > up while reviewing HBASE-24321 by multiple reviewers: > [https://github.com/apache/hbase/pull/1655] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-24456) Immutable Scan as unmodifiable subclass or wrapper of Scan
[ https://issues.apache.org/jira/browse/HBASE-24456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-24456 started by Viraj Jasani. > Immutable Scan as unmodifiable subclass or wrapper of Scan > -- > > Key: HBASE-24456 > URL: https://issues.apache.org/jira/browse/HBASE-24456 > Project: HBase > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > > We should provide an ImmutableScan as subclass or wrapper of Scan which can > be used specifically by Coprocessors as read only Scan. This suggestion came > up while reviewing HBASE-24321 by multiple reviewers: > [https://github.com/apache/hbase/pull/1655] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1803: [HBASE-24468]Add region info when log meessages in HStore.
Apache-HBase commented on pull request #1803: URL: https://github.com/apache/hbase/pull/1803#issuecomment-636462817 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 18s | master passed | | +1 :green_heart: | compile | 1m 4s | master passed | | +1 :green_heart: | shadedjars | 5m 51s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 41s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 59s | the patch passed | | +1 :green_heart: | compile | 1m 3s | the patch passed | | +1 :green_heart: | javac | 1m 3s | the patch passed | | +1 :green_heart: | shadedjars | 5m 47s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 128m 48s | hbase-server in the patch passed. | | | | 154m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1803/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1803 | | JIRA Issue | HBASE-24468 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b06a8b9c7ade 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b4a4debdd9 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1803/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1803/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1803/3/testReport/ | | Max. process+thread count | 4209 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1803/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani opened a new pull request #1818: HBASE-24456 : Create ImmutableScan and use it for CustomizedScanInfoB…
virajjasani opened a new pull request #1818: URL: https://github.com/apache/hbase/pull/1818 …uilder This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] songxincun commented on a change in pull request #1801: [HBASE-24441]CacheConfig details logged at Store open is not really u…
songxincun commented on a change in pull request #1801:
URL: https://github.com/apache/hbase/pull/1801#discussion_r432919519
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -379,6 +379,9 @@ private MemStore getMemstore() {
protected void createCacheConf(final ColumnFamilyDescriptor family) {
this.cacheConf = new CacheConfig(conf, family, region.getBlockCache(),
region.getRegionServicesForStores().getByteBuffAllocator());
+LOG.info("Created cacheConfig: " + this.getCacheConfig()
Review comment:
> Looks good. There is another jira where we discuss abt changing
HStore#toString() to add region info also there. Then u will need to change
this also.
But, HStore#toString() prints only short name of column family without any
attribute. I am not sure is it suitable to use here. @anoopsjohn Would you
give me some suggestions? Thanks a lot
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-24454) BucketCache disabled instantly before error duration toleration is reached due to timing issue
[
https://issues.apache.org/jira/browse/HBASE-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120507#comment-17120507
]
Hudson commented on HBASE-24454:
Results for branch branch-1
[build #1304 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1304/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1304//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1304//JDK7_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1304//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> BucketCache disabled instantly before error duration toleration is reached
> due to timing issue
> --
>
> Key: HBASE-24454
> URL: https://issues.apache.org/jira/browse/HBASE-24454
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 1.4.10
> Environment: We saw this in HBase 1.4.10 (EMR 5.28.1), but I believe
> the newest code still has this problem.
>Reporter: Jacob LeBlanc
>Assignee: Jacob LeBlanc
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.6
>
>
> We saw an instance where BucketCache was disabled after only two IO error
> were encountered at nearly the same time. The following shows all errors from
> a region server log for the 2020-05-26 17:00 hour. Notice that there are no
> other errors until the 17:14:50 at which time the BucketCache is disabled
> because it claims duration time has exceeded 6 ms:
> $ grep ERROR hbase-hbase-regionserver-ip-172-20-113-147.log.2020-05-26-17
> 2020-05-26 17:14:50,396 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: Failed reading block
> 9adfad3a603047cfa0c98b16da0b0974_13786 from bucket cache
> 2020-05-26 17:14:50,397 ERROR
> [regionserver/ip-172-20-113-147.us-west-2.compute.internal/172.20.113.147:16020-BucketCacheWriter-0]
> bucket.BucketCache: Failed syncing IO engine
> 2020-05-26 17:14:50,400 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: Failed reading block
> 9adfad3a603047cfa0c98b16da0b0974_13786 from bucket cache
> 2020-05-26 17:14:50,401 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: IO errors duration time has exceeded 6ms, disabling
> cache, please check your IOEngine
> The region server is very busy so it should be constantly getting successful
> reads and writes in the bucket cache (it is not as though there was some long
> ago error and then no successful IO to clear the error flag).
> We are running a busy EMR cluster backed by S3. A bucketcache getting
> disabled is a huge performance issue because all reads must go to S3.
> Looking at the code, I believe I've found a timing issue. Here is the code
> for checking and setting the ioErrorStartTime (taken from BucketCache.java):
>
> {code:java}
> /**
>* Check whether we tolerate IO error this time. If the duration of IOEngine
>* throwing errors exceeds ioErrorsDurationTimeTolerated, we will disable
> the
>* cache
>*/
> private void checkIOErrorIsTolerated() {
> long now = EnvironmentEdgeManager.currentTime();
> if (this.ioErrorStartTime > 0) {
> if (cacheEnabled && (now - ioErrorStartTime) >
> this.ioErrorsTolerationDuration) {
> LOG.error("IO errors duration time has exceeded " +
> ioErrorsTolerationDuration +
> "ms, disabling cache, please check your IOEngine");
> disableCache();
> }
> } else {
> this.ioErrorStartTime = now;
> }
> }
> {code}
>
> And here is the code for clearing the ioErrorStartTime when a successful read
> or write is done:
> {code:java}
> if (this.ioErrorStartTime > 0) {
> ioErrorStartTime = -1;
> }
> {code}
> Notice that that if ioErrorStartTime is set to -1 after the first if
> statement in checkIOErrorIsTolerated but before (now - ioErrorStartTime),
> then (now - ioErrorStartTime) will evaluate to (now + 1) resulting in the
> code thinking it has exceeded ioErrorsTolerationDuration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] songxincun commented on pull request #1803: [HBASE-24468]Add region info when log meessages in HStore.
songxincun commented on pull request #1803: URL: https://github.com/apache/hbase/pull/1803#issuecomment-636457128 @anoopsjohn Thanks for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Issue Comment Deleted] (HBASE-24441) CacheConfig details logged at Store open is not really useful
[
https://issues.apache.org/jira/browse/HBASE-24441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
song XinCun updated HBASE-24441:
Comment: was deleted
(was: Log after patch would look like:
{code:java}
2020-05-28 21:35:46,842 INFO [StoreOpener-3e735faebf71bb030f365fac2427f7bd-1]
hfile.CacheConfig: Created cacheConfig: cacheDataOnRead=true,
cacheDataOnWrite=true, cacheIndexesOnWrite=true, cacheBloomsOnWrite=true,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false for
family {NAME => 'cf', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false',
NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE',
CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'DIFF', TTL =>
'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW',
CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE =>
'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'SNAPPY',
BLOCKCACHE => 'true', BLOCKSIZE => '65536'} in region
table_100B_10,user5246,1590642270538.3e735faebf71bb030f365fac2427f7bd. with
blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@47c0732b
{code}
The region info which contains table and region name is added to the log.)
> CacheConfig details logged at Store open is not really useful
> -
>
> Key: HBASE-24441
> URL: https://issues.apache.org/jira/browse/HBASE-24441
> Project: HBase
> Issue Type: Improvement
> Components: logging, regionserver
>Affects Versions: 3.0.0-alpha-1
>Reporter: Anoop Sam John
>Assignee: song XinCun
>Priority: Minor
>
> CacheConfig constructor is logging 'this' object at INFO level. This log
> comes during Store open(As CacheConfig instance for that store is created).
> As the log is at CacheConfig only, we don't get to know this is for which
> region:store. So not really useful log.
> {code}
> blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@7bc02941,
> cacheDataOnRead=true, cacheDataOnWrite=true, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> {code}
> Also during every compaction also this logs keeps coming. This is because
> during compaction we create new CacheConfig based on the HStore level
> CacheConfig object. We can avoid this log with every compaction happening.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
