[GitHub] [hbase] Apache-HBase commented on pull request #2881: HBASE-25507 Leak of ESTABLISHED sockets when encountered "java.io.IOE…
Apache-HBase commented on pull request #2881: URL: https://github.com/apache/hbase/pull/2881#issuecomment-760256461 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 20s | master passed | | +1 :green_heart: | compile | 1m 6s | master passed | | +1 :green_heart: | shadedjars | 6m 42s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 1m 7s | the patch passed | | +1 :green_heart: | javac | 1m 7s | the patch passed | | +1 :green_heart: | shadedjars | 6m 40s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 133m 28s | hbase-server in the patch failed. | | | | 161m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2881/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2881 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b17c62ec1237 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3488c44a21 | | Default Java | AdoptOpenJDK-11.0.6+10 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2881/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2881/1/testReport/ | | Max. process+thread count | 4345 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2881/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-25505) ZK watcher threads are daemonized; reconsider
[ https://issues.apache.org/jira/browse/HBASE-25505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264952#comment-17264952 ] Duo Zhang edited comment on HBASE-25505 at 1/14/21, 3:18 PM: - I think I found a possible problem to lead to this situation, especially that [~larsh] confirmed that the not closed ZKWatcher is in ReplicationLogCleaner. Before HBASE-23340, ReplicationLogCleaner will create its own ZKWatcher, as we will close it in the stop method, which will be called in the cleanup method of CleanerChore. Then here comes the problem. The cleanup method of a ScheduledChore will only be called in the run method, so if you just call stop on the stopper instance which is passed to the ScheduledChore when creating it, everything will be fine. But in HMaster.stopChore, we use ChoreService.cancelChore to stop the ScheduledChore. So if the ScheduledChore has not been scheduled again after we set stopped to true for HMaster and before we call cancelChore(I even can not make sure setting stopped to true is happened before we call cancelChore...), the cleanup method will never be executed. And this is likely the case as the default schedule interval is 10 minutes... I added a UT in the uploaded patch to show that, calling cancelChore will not introduce a call to cleanup. I think a possible fix is to also call cleanup in the cancelChore method of ChoreService, just need to add a comment to say that the implementation should make sure that the method can be called multiple times without side effect. Thanks. was (Author: apache9): I think I found a possible problem to lead to this situation, especially that [~larsh] confirmed that the not closed ZKWatcher is in ReplicationLogCleaner. Before HBASE-23340, ReplicationLogCleaner will create its own ZKWatcher, as we will close it in the stop method, which will be called in the cleanup method of CleanerChore. Then here comes the problem. The cleanup method of a ScheduledChore will only be called in the run method, so if you just call stop on the stopper instance which is passed to the ScheduledChore when creating it, everything will be fine. But in HMaster.stopChore, we use ChoreService.cancelChore to stop the ScheduledChore. So if the ScheduledChore has been scheduled again after we set stopped to true for HMaster and before we call cancelChore(I even can not make sure setting stopped to true is happened before we call cancelChore...), the cleanup method will never be executed. And this is likely the case as the default schedule interval is 10 minutes... I added a UT in the uploaded patch to show that, calling cancelChore will not introduce a call to cleanup. I think a possible fix is to also call cleanup in the cancelChore method of ChoreService, just need to add a comment to say that the implementation should make sure that the method can be called multiple times without side effect. Thanks. > ZK watcher threads are daemonized; reconsider > - > > Key: HBASE-25505 > URL: https://issues.apache.org/jira/browse/HBASE-25505 > Project: HBase > Issue Type: Brainstorming >Reporter: Andrew Kyle Purtell >Priority: Major > Attachments: ScheduledChore.cleanup-not-called.diff > > > On HBASE-25279 there was some discussion and difference of opinion about > having ZK watcher pool threads be daemonized. This is not necessarily a > problem but should be reconsidered. > Daemon threads are subject to abrupt termination during JVM shutdown and > therefore may be interrupted before state changes are complete or resources > are released. > As long as ZK watchers are properly closed by shutdown logic the pool threads > will be terminated in a controlled manner and the JVM will exit. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25506) ServerManager affects MTTR of HMaster
[ https://issues.apache.org/jira/browse/HBASE-25506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Cui updated HBASE-25506: --- Affects Version/s: 3.0.0-alpha-1 > ServerManager affects MTTR of HMaster > - > > Key: HBASE-25506 > URL: https://issues.apache.org/jira/browse/HBASE-25506 > Project: HBase > Issue Type: Improvement > Components: MTTR >Affects Versions: 3.0.0-alpha-1 >Reporter: Bo Cui >Assignee: Bo Cui >Priority: Major > Attachments: image-2021-01-14-17-44-16-091.png, > image-2021-01-14-17-44-42-181.png > > > https://github.com/apache/hbase/blob/3488c44a21612aae1835fc3e91a4a12ed2abb8b7/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java#L925 > If a cluster has N+W regions, this > removeDeletedRegionFromLoadedFlushedSequenceIds takes a long time... > !image-2021-01-14-17-44-16-091.png! > !image-2021-01-14-17-44-42-181.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25506) ServerManager affects MTTR of HMaster
[ https://issues.apache.org/jira/browse/HBASE-25506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Cui reassigned HBASE-25506: -- Assignee: Bo Cui > ServerManager affects MTTR of HMaster > - > > Key: HBASE-25506 > URL: https://issues.apache.org/jira/browse/HBASE-25506 > Project: HBase > Issue Type: Improvement > Components: MTTR >Reporter: Bo Cui >Assignee: Bo Cui >Priority: Major > Attachments: image-2021-01-14-17-44-16-091.png, > image-2021-01-14-17-44-42-181.png > > > https://github.com/apache/hbase/blob/3488c44a21612aae1835fc3e91a4a12ed2abb8b7/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java#L925 > If a cluster has N+W regions, this > removeDeletedRegionFromLoadedFlushedSequenceIds takes a long time... > !image-2021-01-14-17-44-16-091.png! > !image-2021-01-14-17-44-42-181.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25506) ServerManager affects MTTR of HMaster
Bo Cui created HBASE-25506: -- Summary: ServerManager affects MTTR of HMaster Key: HBASE-25506 URL: https://issues.apache.org/jira/browse/HBASE-25506 Project: HBase Issue Type: Improvement Components: MTTR Reporter: Bo Cui Attachments: image-2021-01-14-17-44-16-091.png, image-2021-01-14-17-44-42-181.png https://github.com/apache/hbase/blob/3488c44a21612aae1835fc3e91a4a12ed2abb8b7/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java#L925 If a cluster has N+W regions, this removeDeletedRegionFromLoadedFlushedSequenceIds takes a long time... !image-2021-01-14-17-44-16-091.png! !image-2021-01-14-17-44-42-181.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] bsglz commented on pull request #2875: HBASE-25497 move_namespaces_rsgroup should change hbase.rsgroup.name …
bsglz commented on pull request #2875: URL: https://github.com/apache/hbase/pull/2875#issuecomment-760133381 Let's wait to see if any objection. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25329) Dump region hashes in logs for the regions that are stuck in transition for more than a configured amount of time
[
https://issues.apache.org/jira/browse/HBASE-25329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264818#comment-17264818
]
Hudson commented on HBASE-25329:
Results for branch master
[build #185 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Dump region hashes in logs for the regions that are stuck in transition for
> more than a configured amount of time
> -
>
> Key: HBASE-25329
> URL: https://issues.apache.org/jira/browse/HBASE-25329
> Project: HBase
> Issue Type: Improvement
>Reporter: Caroline
>Assignee: Caroline
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.1
>
> Attachments: HBASE-25329.branch-1.000.patch,
> HBASE-25329.branch-2.000.patch, HBASE-25329.master.000.patch
>
>
> We have metrics for number of RITs as well as number of RITs above a certain
> threshold, but we don't have any way of keeping track of the region hashes of
> those RITs. It would be beneficial to emit those region hashes as a metric,
> as well as log them, so that we don't accidentally lose this information for
> debugging the RIT at a later tiime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25449) 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml
[
https://issues.apache.org/jira/browse/HBASE-25449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264822#comment-17264822
]
Hudson commented on HBASE-25449:
Results for branch master
[build #185 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml
> -
>
> Key: HBASE-25449
> URL: https://issues.apache.org/jira/browse/HBASE-25449
> Project: HBase
> Issue Type: Improvement
> Components: conf
>Affects Versions: 2.0.1
>Reporter: shenshengli
>Assignee: shenshengli
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.3.5, 2.4.2
>
>
> I think this parameter is not suitable for in hbase-default.xml, because in
> this case, HDFS explicitly set to "dfs.client.read.shortcircuit=true", hbase
> rely on HDFS configuration, the parameters in hbase service still is
> false.Must be explicitly in hbase-site.xml is set to
> "dfs.client.read.shortcircuit=true" to take effect.The same goes for the
> parameter 'dfs.domain.socket.path', which is already covered in the specific
> change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25502) IntegrationTestMTTR fails with TableNotFoundException
[
https://issues.apache.org/jira/browse/HBASE-25502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264820#comment-17264820
]
Hudson commented on HBASE-25502:
Results for branch master
[build #185 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> IntegrationTestMTTR fails with TableNotFoundException
> -
>
> Key: HBASE-25502
> URL: https://issues.apache.org/jira/browse/HBASE-25502
> Project: HBase
> Issue Type: Test
> Components: integration tests
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.1, 2.3.5
>
>
> Admin.getDescriptor() fails beacuse table doesn't exist.
>
> {noformat}
> org.apache.hadoop.hbase.TableNotFoundException: IntegrationTestMTTR
> at java.lang.Thread.getStackTrace(Thread.java:1559)
> at
> org.apache.hadoop.hbase.util.FutureUtils.setStackTrace(FutureUtils.java:130)
> at
> org.apache.hadoop.hbase.util.FutureUtils.rethrow(FutureUtils.java:149)
> at org.apache.hadoop.hbase.util.FutureUtils.get(FutureUtils.java:172)
> at
> org.apache.hadoop.hbase.client.AdminOverAsyncAdmin.getDescriptor(AdminOverAsyncAdmin.java:163)
> at
> org.apache.hadoop.hbase.mttr.IntegrationTestMTTR.setupTables(IntegrationTestMTTR.java:236)
> at
> org.apache.hadoop.hbase.mttr.IntegrationTestMTTR.setUp(IntegrationTestMTTR.java:179)
>
> {noformat}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25503) HBase code download is failing on windows with invalid path error
[
https://issues.apache.org/jira/browse/HBASE-25503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264819#comment-17264819
]
Hudson commented on HBASE-25503:
Results for branch master
[build #185 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HBase code download is failing on windows with invalid path error
> -
>
> Key: HBASE-25503
> URL: https://issues.apache.org/jira/browse/HBASE-25503
> Project: HBase
> Issue Type: Bug
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.1
>
>
> git pull command is failing with "error: invalid path"
> {noformat}
> Host1 MINGW64 /d/hbase (master)
> $ git pull
> error: invalid path
> 'dev-support/design-docs/HBASE-18070-ROOT_hbase:meta_Region_Replicas.pdf'
> Updating 2e96a5b2d3..dfefff7e59{noformat}
> This problem is coming only on windows machines. Tried on windows 7 and
> windows 10 both have problem. Searched on net, seems no any easy solution.
> Problem is because file HBASE-18070-ROOT_hbase:meta_Region_Replicas.pdf has
> colon in its name.
> To fix the problem we should remove the colon : from file name
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] sunhelly opened a new pull request #2881: HBASE-25507 Leak of ESTABLISHED sockets when encountered "java.io.IOE…
sunhelly opened a new pull request #2881: URL: https://github.com/apache/hbase/pull/2881 …xception: Invalid HFile block magic" This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25505) ZK watcher threads are daemonized; reconsider
[ https://issues.apache.org/jira/browse/HBASE-25505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-25505: -- Attachment: ScheduledChore.cleanup-not-called.diff > ZK watcher threads are daemonized; reconsider > - > > Key: HBASE-25505 > URL: https://issues.apache.org/jira/browse/HBASE-25505 > Project: HBase > Issue Type: Brainstorming >Reporter: Andrew Kyle Purtell >Priority: Major > Attachments: ScheduledChore.cleanup-not-called.diff > > > On HBASE-25279 there was some discussion and difference of opinion about > having ZK watcher pool threads be daemonized. This is not necessarily a > problem but should be reconsidered. > Daemon threads are subject to abrupt termination during JVM shutdown and > therefore may be interrupted before state changes are complete or resources > are released. > As long as ZK watchers are properly closed by shutdown logic the pool threads > will be terminated in a controlled manner and the JVM will exit. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25495) fix comment error of admin.rb
[
https://issues.apache.org/jira/browse/HBASE-25495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264821#comment-17264821
]
Hudson commented on HBASE-25495:
Results for branch master
[build #185 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/185/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> fix comment error of admin.rb
> --
>
> Key: HBASE-25495
> URL: https://issues.apache.org/jira/browse/HBASE-25495
> Project: HBase
> Issue Type: Bug
>Reporter: xijiawen
>Assignee: xijiawen
>Priority: Minor
> Fix For: 3.0.0-alpha-1
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] sunhelly closed pull request #2881: HBASE-25507 Leak of ESTABLISHED sockets when encountered "java.io.IOE…
sunhelly closed pull request #2881: URL: https://github.com/apache/hbase/pull/2881 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264871#comment-17264871 ] Viraj Jasani commented on HBASE-25279: -- >From daemonizing threads viewpoint, it seems only ZKWatcher is missing in >PR#2407 (in comparison to PR#2196). I think we are good here. For the actual issue of missing closure of ZKWatcher, we can focus on HBASE-25505. > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.1 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25505) ZK watcher threads are daemonized; reconsider
[ https://issues.apache.org/jira/browse/HBASE-25505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264952#comment-17264952 ] Duo Zhang commented on HBASE-25505: --- I think I found a possible problem to lead to this situation, especially that [~larsh] confirmed that the not closed ZKWatcher is in ReplicationLogCleaner. Before HBASE-23340, ReplicationLogCleaner will create its own ZKWatcher, as we will close it in the stop method, which will be called in the cleanup method of CleanerChore. Then here comes the problem. The cleanup method of a ScheduledChore will only be called in the run method, so if you just call stop on the stopper instance which is passed to the ScheduledChore when creating it, everything will be fine. But in HMaster.stopChore, we use ChoreService.cancelChore to stop the ScheduledChore. So if the ScheduledChore has been scheduled again after we set stopped to true for HMaster and before we call cancelChore(I even can not make sure setting stopped to true is happened before we call cancelChore...), the cleanup method will never be executed. And this is likely the case as the default schedule interval is 10 minutes... I added a UT in the uploaded patch to show that, calling cancelChore will not introduce a call to cleanup. I think a possible fix is to also call cleanup in the cancelChore method of ChoreService, just need to add a comment to say that the implementation should make sure that the method can be called multiple times without side effect. Thanks. > ZK watcher threads are daemonized; reconsider > - > > Key: HBASE-25505 > URL: https://issues.apache.org/jira/browse/HBASE-25505 > Project: HBase > Issue Type: Brainstorming >Reporter: Andrew Kyle Purtell >Priority: Major > Attachments: ScheduledChore.cleanup-not-called.diff > > > On HBASE-25279 there was some discussion and difference of opinion about > having ZK watcher pool threads be daemonized. This is not necessarily a > problem but should be reconsidered. > Daemon threads are subject to abrupt termination during JVM shutdown and > therefore may be interrupted before state changes are complete or resources > are released. > As long as ZK watchers are properly closed by shutdown logic the pool threads > will be terminated in a controlled manner and the JVM will exit. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25507) Leak of ESTABLISHED sockets when encountered "java.io.IOException: Invalid HFile block magic"
Xiaolin Ha created HBASE-25507:
--
Summary: Leak of ESTABLISHED sockets when encountered
"java.io.IOException: Invalid HFile block magic"
Key: HBASE-25507
URL: https://issues.apache.org/jira/browse/HBASE-25507
Project: HBase
Issue Type: Improvement
Reporter: Xiaolin Ha
Assignee: Xiaolin Ha
Attachments: errorlogs.png,
increasing-of-established-sockets-image.png, problem-region-move-logs.png
Recently, we found socket leaks on our production cluster. The leaked sockets
are in ESTABLISHED state. We found this happened on RS who owned a particular
region from our analysis of metrics monitor and logs. RS without this region
works normally.
On the RS who owns the particular region, we found Exceptions as follows,
{code:java}
java.io.IOException: java.io.IOException: Could not seek
StoreFileScanner[org.apache.hadoop.hbase.ioo.HalfStoreFileReader$1@5388be2f,
cur=null] to key
org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFFamilyCell@25aa56fdat
org.apache.hadoop.hbase.regionserver.StoreScanner.parallelSeek(StoreScanner.java:1128)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:437)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:329)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:302)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.createScanner(Compactor.java:8806)
at
org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor$StripeInternalScannerFacctory.createScanner(StripeCompactor.java:82)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:316)
at
org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor.compact(StripeCompactor..java:120)
at
org.apache.hadoop.hbase.regionserver.compactions.StripeCompactionPolicy$SplitStripeCompacctionRequest.execute(StripeCompactionPolicy.java:662)
at
org.apache.hadoop.hbase.regionserver.StripeStoreEngine$StripeCompaction.compact(StripeStooreEngine.java:114)
at
org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1461)at
org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2121)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CommpactSplitThread.java:519)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplittThread.java:555)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)Caused by: java.io.IOException:
Could not seek
StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoreeFileReader$1@5388be2f,
cur=null] to key
org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFamilyCell@@25aa56fdat
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:229)
at
org.apache.hadoop.hbase.regionserver.handler.ParallelSeekHandler.process(ParallelSeekHanddler.java:56)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
... 3 moreCaused by: java.io.IOException: Invalid HFile block magic:
\x00\x00\x00\x00\x00\x00\x00\x00at
org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:159)at
org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:171)at
org.apache.hadoop.hbase.io.hfile.HFileBlock.createFromBuff(HFileBlock.java:333)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlockk.java:1753)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:15552)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:539)
at
org.apache.hadoop.hbase.io.hfile.HFileScannerImpl.readAndUpdateNewBlock(HFileScannerImpl..java:737)
at
org.apache.hadoop.hbase.io.hfile.HFileScannerImpl.seekTo(HFileScannerImpl.java:726)
at
org.apache.hadoop.hbase.io.HalfStoreFileReader$1.seekTo(HalfStoreFileReader.java:161)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.javaa:315)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:211)
... 5 more
{code}
The count of established sockets is always increasing, see picture,
!increasing-of-established-sockets-image.png!
!problem-region-move-logs.png!
!errorlogs.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25507) Leak of ESTABLISHED sockets when encountered "java.io.IOException: Invalid HFile block magic"
[
https://issues.apache.org/jira/browse/HBASE-25507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaolin Ha updated HBASE-25507:
---
Description:
Recently, we found socket leaks on our production cluster. The leaked sockets
are in ESTABLISHED state. We found this happened on RS who owned a particular
region from our analysis of metrics monitor and logs. RS without this region
works normally.
On the RS who owns the particular region, we found Exceptions as follows,
{code:java}
java.io.IOException: java.io.IOException: Could not seek
StoreFileScanner[org.apache.hadoop.hbase.ioo
.HalfStoreFileReader$1@5388be2f, cur=null] to key
org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFF
amilyCell@25aa56fd
at
org.apache.hadoop.hbase.regionserver.StoreScanner.parallelSeek(StoreScanner.java:1128)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:437)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:329)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:302)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.createScanner(Compactor.java:88
06)
at
org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor$StripeInternalScannerFacc
tory.createScanner(StripeCompactor.java:82)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:316)
at
org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor.compact(StripeCompactor..
java:120)
at
org.apache.hadoop.hbase.regionserver.compactions.StripeCompactionPolicy$SplitStripeCompacc
tionRequest.execute(StripeCompactionPolicy.java:662)
at
org.apache.hadoop.hbase.regionserver.StripeStoreEngine$StripeCompaction.compact(StripeStoo
reEngine.java:114)
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1461)
at
org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2121)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(Comm
pactSplitThread.java:519)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitt
Thread.java:555)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Could not seek
StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoree
FileReader$1@5388be2f, cur=null] to key
org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFamilyCell@@
25aa56fd
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:229)
at
org.apache.hadoop.hbase.regionserver.handler.ParallelSeekHandler.process(ParallelSeekHandd
ler.java:56)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
... 3 more
Caused by: java.io.IOException: Invalid HFile block magic:
\x00\x00\x00\x00\x00\x00\x00\x00
at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:159)
at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:171)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock.createFromBuff(HFileBlock.java:333)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlockk
.java:1753)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:155
52)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:539)
at
org.apache.hadoop.hbase.io.hfile.HFileScannerImpl.readAndUpdateNewBlock(HFileScannerImpl..
java:737)
at
org.apache.hadoop.hbase.io.hfile.HFileScannerImpl.seekTo(HFileScannerImpl.java:726)
at
org.apache.hadoop.hbase.io.HalfStoreFileReader$1.seekTo(HalfStoreFileReader.java:161)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.javaa
:315)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:211)
... 5 more
{code}
The count of established sockets is always increasing, see picture,
!increasing-of-established-sockets-image.png!
!problem-region-move-logs.png!
!errorlogs.png!
was:
Recently, we found socket leaks on our production cluster. The leaked sockets
are in ESTABLISHED state. We found this happened on RS who owned a particular
region from our analysis of metrics monitor and logs. RS without this region
works normally.
On the RS who owns the particular region, we found Exceptions as follows,
{code:java}
java.io.IOException: java.io.IOException: Could not seek
StoreFileScanner[org.apache.hadoop.hbase.ioo.HalfStoreFileReader$1@5388be2f,
cur=null] to key
[GitHub] [hbase] Apache-HBase commented on pull request #2881: HBASE-25507 Leak of ESTABLISHED sockets when encountered "java.io.IOE…
Apache-HBase commented on pull request #2881: URL: https://github.com/apache/hbase/pull/2881#issuecomment-760192592 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 12s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 51s | master passed | | +1 :green_heart: | compile | 3m 16s | master passed | | +1 :green_heart: | checkstyle | 1m 3s | master passed | | +1 :green_heart: | spotbugs | 2m 5s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 31s | the patch passed | | +1 :green_heart: | compile | 3m 14s | the patch passed | | +1 :green_heart: | javac | 3m 14s | the patch passed | | +1 :green_heart: | checkstyle | 1m 9s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 20m 38s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 47s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 54m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2881/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2881 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 756906154a8a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3488c44a21 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2881/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bsglz commented on pull request #2875: HBASE-25497 move_namespaces_rsgroup should change hbase.rsgroup.name …
bsglz commented on pull request #2875: URL: https://github.com/apache/hbase/pull/2875#issuecomment-760132522 LGTM. +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2881: HBASE-25507 Leak of ESTABLISHED sockets when encountered "java.io.IOE…
Apache-HBase commented on pull request #2881: URL: https://github.com/apache/hbase/pull/2881#issuecomment-760268387 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 12s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 56s | master passed | | +1 :green_heart: | compile | 1m 2s | master passed | | +1 :green_heart: | shadedjars | 6m 46s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 34s | the patch passed | | +1 :green_heart: | compile | 0m 59s | the patch passed | | +1 :green_heart: | javac | 0m 59s | the patch passed | | +1 :green_heart: | shadedjars | 6m 42s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 150m 34s | hbase-server in the patch passed. | | | | 180m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2881/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2881 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4318c5c5af3e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3488c44a21 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2881/1/testReport/ | | Max. process+thread count | 4131 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2881/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] christinefeng commented on a change in pull request #2866: HBASE-25478 - Implement retries when enabling tables in TestRegionReplicaReplicationEndpoint
christinefeng commented on a change in pull request #2866:
URL: https://github.com/apache/hbase/pull/2866#discussion_r557593057
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
##
@@ -487,4 +490,26 @@ private void testRegionReplicaReplicationIgnores(boolean
dropTable, boolean disa
connection.close();
}
}
+
+ private void createOrEnableTableWithRetries(TableDescriptor htd, boolean
createTableOperation) {
+// Helper function to run create/enable table operations with a retry
feature
+boolean continueToRetry = true;
+int tries = 0;
+while (continueToRetry && tries < 100) {
Review comment:
Good point, I will change it
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (HBASE-25356) HBaseAdmin#getRegion() needs to filter out non-regionName and non-encodedRegionName
[ https://issues.apache.org/jira/browse/HBASE-25356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun resolved HBASE-25356. -- Fix Version/s: 2.4.1 2.5.0 2.3.4 Resolution: Fixed > HBaseAdmin#getRegion() needs to filter out non-regionName and > non-encodedRegionName > --- > > Key: HBASE-25356 > URL: https://issues.apache.org/jira/browse/HBASE-25356 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.3.3, 2.4.0 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > Fix For: 2.3.4, 2.5.0, 2.4.1 > > > I was running shell command to major compact meta table. The implementation > is wrong because it tries to search the meta table with meta table name. This > also results in an unnecessary scan of meta table. > > majorCompactRegion() is calling HBaseAdmin#getRegion() which basically scan > meta table itself. > This command is being used by operator quite often, we need to correct it. > > This applies to split/flush command as well, which calls getRegion() with > tableName as an input. > > The solution is that getRegion() needs to filter out non-regionName and > non-encodedRegionName, this will save a query of meta table and a heavy scan > of meta table. If meta table size is large, the overhead is huge. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25083) make sure the next hbase 1.y release has Hadoop 2.10 as a minimum version
[ https://issues.apache.org/jira/browse/HBASE-25083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun updated HBASE-25083: - Fix Version/s: 2.3.4 > make sure the next hbase 1.y release has Hadoop 2.10 as a minimum version > - > > Key: HBASE-25083 > URL: https://issues.apache.org/jira/browse/HBASE-25083 > Project: HBase > Issue Type: Task > Components: documentation, hadoop2 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.7.0, 2.3.4 > > > Our reference guide list of prerequisites still has Hadoop 2.8 and 2.9 listed > for HBase 1 releases. > * [hadoop 2.8 is > EOM|https://lists.apache.org/thread.html/r348f7bc93a522f05b7cce78a911854d128a6b1b8bd8124bad4d06ce6%40%3Cuser.hadoop.apache.org%3E] > * [hadoop 2.9 is > EOM|https://lists.apache.org/thread.html/r16b14cce9504f7a9d228612c6b808e72d8dd20863c78be51a7e04ed5%40%3Cuser.hadoop.apache.org%3E] > The current list in the reference guide for HBase 1.6 is just the 1.5 list > copied. we should update it to remove 2.8 and 2.9 and make sure we're no > longer doing build/test based on those versions for branch-1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25425) Some notes on RawCell
[ https://issues.apache.org/jira/browse/HBASE-25425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun updated HBASE-25425: - Fix Version/s: 2.4.1 2.5.0 2.3.4 > Some notes on RawCell > - > > Key: HBASE-25425 > URL: https://issues.apache.org/jira/browse/HBASE-25425 > Project: HBase > Issue Type: Improvement >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Trivial > Fix For: 3.0.0-alpha-1, 2.3.4, 2.5.0, 2.4.1 > > > Add a few notes to RawCell on what it is. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25507) Leak of ESTABLISHED sockets when compaction encountered "java.io.IOException: Invalid HFile block magic"
[
https://issues.apache.org/jira/browse/HBASE-25507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaolin Ha updated HBASE-25507:
---
Summary: Leak of ESTABLISHED sockets when compaction encountered
"java.io.IOException: Invalid HFile block magic" (was: Leak of ESTABLISHED
sockets when encountered "java.io.IOException: Invalid HFile block magic")
> Leak of ESTABLISHED sockets when compaction encountered "java.io.IOException:
> Invalid HFile block magic"
>
>
> Key: HBASE-25507
> URL: https://issues.apache.org/jira/browse/HBASE-25507
> Project: HBase
> Issue Type: Improvement
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Attachments: errorlogs.png,
> increasing-of-established-sockets-image.png, problem-region-move-logs.png
>
>
> Recently, we found socket leaks on our production cluster. The leaked sockets
> are in ESTABLISHED state. We found this happened on RS who owned a particular
> region from our analysis of metrics monitor and logs. RS without this region
> works normally.
> On the RS who owns the particular region, we found Exceptions as follows,
> {code:java}
> java.io.IOException: java.io.IOException: Could not seek
> StoreFileScanner[org.apache.hadoop.hbase.ioo
> .HalfStoreFileReader$1@5388be2f, cur=null] to key
> org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFF
> amilyCell@25aa56fd
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.parallelSeek(StoreScanner.java:1128)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:437)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:302)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.createScanner(Compactor.java:88
> 06)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor$StripeInternalScannerFacc
> tory.createScanner(StripeCompactor.java:82)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:316)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor.compact(StripeCompactor..
> java:120)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactionPolicy$SplitStripeCompacc
> tionRequest.execute(StripeCompactionPolicy.java:662)
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreEngine$StripeCompaction.compact(StripeStoo
> reEngine.java:114)
> at
> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1461)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2121)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(Comm
> pactSplitThread.java:519)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitt
> Thread.java:555)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Could not seek
> StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoree
> FileReader$1@5388be2f, cur=null] to key
> org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFamilyCell@@
> 25aa56fd
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:229)
> at
> org.apache.hadoop.hbase.regionserver.handler.ParallelSeekHandler.process(ParallelSeekHandd
> ler.java:56)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
> ... 3 more
> Caused by: java.io.IOException: Invalid HFile block magic:
> \x00\x00\x00\x00\x00\x00\x00\x00
> at
> org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:159)
> at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:171)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.createFromBuff(HFileBlock.java:333)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlockk
> .java:1753)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:155
> 52)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:539)
> at
> org.apache.hadoop.hbase.io.hfile.HFileScannerImpl.readAndUpdateNewBlock(HFileScannerImpl..
> java:737)
> at
> org.apache.hadoop.hbase.io.hfile.HFileScannerImpl.seekTo(HFileScannerImpl.java:726)
> at
>
[jira] [Updated] (HBASE-25507) Leak of ESTABLISHED sockets when compaction encountered "java.io.IOException: Invalid HFile block magic"
[
https://issues.apache.org/jira/browse/HBASE-25507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaolin Ha updated HBASE-25507:
---
Component/s: Compaction
> Leak of ESTABLISHED sockets when compaction encountered "java.io.IOException:
> Invalid HFile block magic"
>
>
> Key: HBASE-25507
> URL: https://issues.apache.org/jira/browse/HBASE-25507
> Project: HBase
> Issue Type: Improvement
> Components: Compaction
>Affects Versions: 3.0.0-alpha-1, 2.4.1
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Attachments: errorlogs.png,
> increasing-of-established-sockets-image.png, problem-region-move-logs.png
>
>
> Recently, we found socket leaks on our production cluster. The leaked sockets
> are in ESTABLISHED state. We found this happened on RS who owned a particular
> region from our analysis of metrics monitor and logs. RS without this region
> works normally.
> On the RS who owns the particular region, we found Exceptions as follows,
> {code:java}
> java.io.IOException: java.io.IOException: Could not seek
> StoreFileScanner[org.apache.hadoop.hbase.ioo
> .HalfStoreFileReader$1@5388be2f, cur=null] to key
> org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFF
> amilyCell@25aa56fd
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.parallelSeek(StoreScanner.java:1128)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:437)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:302)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.createScanner(Compactor.java:88
> 06)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor$StripeInternalScannerFacc
> tory.createScanner(StripeCompactor.java:82)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:316)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor.compact(StripeCompactor..
> java:120)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactionPolicy$SplitStripeCompacc
> tionRequest.execute(StripeCompactionPolicy.java:662)
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreEngine$StripeCompaction.compact(StripeStoo
> reEngine.java:114)
> at
> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1461)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2121)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(Comm
> pactSplitThread.java:519)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitt
> Thread.java:555)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Could not seek
> StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoree
> FileReader$1@5388be2f, cur=null] to key
> org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFamilyCell@@
> 25aa56fd
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:229)
> at
> org.apache.hadoop.hbase.regionserver.handler.ParallelSeekHandler.process(ParallelSeekHandd
> ler.java:56)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
> ... 3 more
> Caused by: java.io.IOException: Invalid HFile block magic:
> \x00\x00\x00\x00\x00\x00\x00\x00
> at
> org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:159)
> at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:171)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.createFromBuff(HFileBlock.java:333)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlockk
> .java:1753)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:155
> 52)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:539)
> at
> org.apache.hadoop.hbase.io.hfile.HFileScannerImpl.readAndUpdateNewBlock(HFileScannerImpl..
> java:737)
> at
> org.apache.hadoop.hbase.io.hfile.HFileScannerImpl.seekTo(HFileScannerImpl.java:726)
> at
> org.apache.hadoop.hbase.io.HalfStoreFileReader$1.seekTo(HalfStoreFileReader.java:161)
> at
>
[jira] [Commented] (HBASE-25083) make sure the next hbase 1.y release has Hadoop 2.10 as a minimum version
[ https://issues.apache.org/jira/browse/HBASE-25083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265082#comment-17265082 ] Huaxiang Sun commented on HBASE-25083: -- Hi [~busbey], since the patch has been merged to all branches, I assume that this can be resolved. Please create a new followup if there is more work needed, thanks. > make sure the next hbase 1.y release has Hadoop 2.10 as a minimum version > - > > Key: HBASE-25083 > URL: https://issues.apache.org/jira/browse/HBASE-25083 > Project: HBase > Issue Type: Task > Components: documentation, hadoop2 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 3.0.0-alpha-1, 1.7.0, 2.3.4, 2.5.0, 2.4.1 > > > Our reference guide list of prerequisites still has Hadoop 2.8 and 2.9 listed > for HBase 1 releases. > * [hadoop 2.8 is > EOM|https://lists.apache.org/thread.html/r348f7bc93a522f05b7cce78a911854d128a6b1b8bd8124bad4d06ce6%40%3Cuser.hadoop.apache.org%3E] > * [hadoop 2.9 is > EOM|https://lists.apache.org/thread.html/r16b14cce9504f7a9d228612c6b808e72d8dd20863c78be51a7e04ed5%40%3Cuser.hadoop.apache.org%3E] > The current list in the reference guide for HBase 1.6 is just the 1.5 list > copied. we should update it to remove 2.8 and 2.9 and make sure we're no > longer doing build/test based on those versions for branch-1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25476) Enable error prone check in pre commit
[ https://issues.apache.org/jira/browse/HBASE-25476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun updated HBASE-25476: - Fix Version/s: (was: 2.3.5) 2.3.4 > Enable error prone check in pre commit > -- > > Key: HBASE-25476 > URL: https://issues.apache.org/jira/browse/HBASE-25476 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.4, 2.5.0, 2.4.1 > > > This has been a pain in the past as the output is not stable, but in > HBASE-25333, we found that some checks in error prone is really useful. > So here I think we need to find a way to try to enable it again in pre commit. > Maybe we could only check for ERROR level problems as it will cause > compilation error. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2861: HBASE-25455 Add trace support for HRegion read/write operation
Apache-HBase commented on pull request #2861: URL: https://github.com/apache/hbase/pull/2861#issuecomment-760287078 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 44s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-22120 Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 59s | HBASE-22120 passed | | +1 :green_heart: | compile | 6m 17s | HBASE-22120 passed | | +1 :green_heart: | checkstyle | 2m 25s | HBASE-22120 passed | | +1 :green_heart: | spotbugs | 4m 31s | HBASE-22120 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 20s | the patch passed | | +1 :green_heart: | compile | 5m 24s | the patch passed | | +1 :green_heart: | javac | 5m 24s | the patch passed | | +1 :green_heart: | checkstyle | 0m 25s | The patch passed checkstyle in hbase-common | | +1 :green_heart: | checkstyle | 0m 29s | The patch passed checkstyle in hbase-client | | +1 :green_heart: | checkstyle | 1m 12s | hbase-server: The patch generated 0 new + 159 unchanged - 2 fixed = 159 total (was 161) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 26s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 4m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. | | | | 62m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2861/8/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2861 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 99d6a052c1c4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-22120 / 1529dec9c3 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2861/8/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25083) make sure the next hbase 1.y release has Hadoop 2.10 as a minimum version
[ https://issues.apache.org/jira/browse/HBASE-25083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun updated HBASE-25083: - Fix Version/s: 3.0.0-alpha-1 > make sure the next hbase 1.y release has Hadoop 2.10 as a minimum version > - > > Key: HBASE-25083 > URL: https://issues.apache.org/jira/browse/HBASE-25083 > Project: HBase > Issue Type: Task > Components: documentation, hadoop2 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 3.0.0-alpha-1, 1.7.0, 2.3.4, 2.5.0, 2.4.1 > > > Our reference guide list of prerequisites still has Hadoop 2.8 and 2.9 listed > for HBase 1 releases. > * [hadoop 2.8 is > EOM|https://lists.apache.org/thread.html/r348f7bc93a522f05b7cce78a911854d128a6b1b8bd8124bad4d06ce6%40%3Cuser.hadoop.apache.org%3E] > * [hadoop 2.9 is > EOM|https://lists.apache.org/thread.html/r16b14cce9504f7a9d228612c6b808e72d8dd20863c78be51a7e04ed5%40%3Cuser.hadoop.apache.org%3E] > The current list in the reference guide for HBase 1.6 is just the 1.5 list > copied. we should update it to remove 2.8 and 2.9 and make sure we're no > longer doing build/test based on those versions for branch-1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25449) 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml
[
https://issues.apache.org/jira/browse/HBASE-25449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265106#comment-17265106
]
Hudson commented on HBASE-25449:
Results for branch branch-2
[build #158 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml
> -
>
> Key: HBASE-25449
> URL: https://issues.apache.org/jira/browse/HBASE-25449
> Project: HBase
> Issue Type: Improvement
> Components: conf
>Affects Versions: 2.0.1
>Reporter: shenshengli
>Assignee: shenshengli
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.3.5, 2.4.2
>
>
> I think this parameter is not suitable for in hbase-default.xml, because in
> this case, HDFS explicitly set to "dfs.client.read.shortcircuit=true", hbase
> rely on HDFS configuration, the parameters in hbase service still is
> false.Must be explicitly in hbase-site.xml is set to
> "dfs.client.read.shortcircuit=true" to take effect.The same goes for the
> parameter 'dfs.domain.socket.path', which is already covered in the specific
> change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25249) Adding StoreContext
[
https://issues.apache.org/jira/browse/HBASE-25249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265108#comment-17265108
]
Hudson commented on HBASE-25249:
Results for branch branch-2
[build #158 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Adding StoreContext
> ---
>
> Key: HBASE-25249
> URL: https://issues.apache.org/jira/browse/HBASE-25249
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha-1, 2.4.0, 2.3.4
>Reporter: Abhishek Khanna
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Adding StoreContext which contains the metadata about the HStore. This meta
> data can be used across the HFileWriter/Readers and other HStore consumers
> without the need of passing around the complete store and exposing its
> internals. This is a refactoring change which cleans up the existing code so
> that the following commits can leverage the context for a more maintainable
> code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25503) HBase code download is failing on windows with invalid path error
[
https://issues.apache.org/jira/browse/HBASE-25503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265104#comment-17265104
]
Hudson commented on HBASE-25503:
Results for branch branch-2
[build #158 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HBase code download is failing on windows with invalid path error
> -
>
> Key: HBASE-25503
> URL: https://issues.apache.org/jira/browse/HBASE-25503
> Project: HBase
> Issue Type: Bug
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.1
>
>
> git pull command is failing with "error: invalid path"
> {noformat}
> Host1 MINGW64 /d/hbase (master)
> $ git pull
> error: invalid path
> 'dev-support/design-docs/HBASE-18070-ROOT_hbase:meta_Region_Replicas.pdf'
> Updating 2e96a5b2d3..dfefff7e59{noformat}
> This problem is coming only on windows machines. Tried on windows 7 and
> windows 10 both have problem. Searched on net, seems no any easy solution.
> Problem is because file HBASE-18070-ROOT_hbase:meta_Region_Replicas.pdf has
> colon in its name.
> To fix the problem we should remove the colon : from file name
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25502) IntegrationTestMTTR fails with TableNotFoundException
[
https://issues.apache.org/jira/browse/HBASE-25502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264991#comment-17264991
]
Hudson commented on HBASE-25502:
Results for branch branch-2.3
[build #151 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/151/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/151/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/151/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/151/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/151/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> IntegrationTestMTTR fails with TableNotFoundException
> -
>
> Key: HBASE-25502
> URL: https://issues.apache.org/jira/browse/HBASE-25502
> Project: HBase
> Issue Type: Test
> Components: integration tests
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.1, 2.3.5
>
>
> Admin.getDescriptor() fails beacuse table doesn't exist.
>
> {noformat}
> org.apache.hadoop.hbase.TableNotFoundException: IntegrationTestMTTR
> at java.lang.Thread.getStackTrace(Thread.java:1559)
> at
> org.apache.hadoop.hbase.util.FutureUtils.setStackTrace(FutureUtils.java:130)
> at
> org.apache.hadoop.hbase.util.FutureUtils.rethrow(FutureUtils.java:149)
> at org.apache.hadoop.hbase.util.FutureUtils.get(FutureUtils.java:172)
> at
> org.apache.hadoop.hbase.client.AdminOverAsyncAdmin.getDescriptor(AdminOverAsyncAdmin.java:163)
> at
> org.apache.hadoop.hbase.mttr.IntegrationTestMTTR.setupTables(IntegrationTestMTTR.java:236)
> at
> org.apache.hadoop.hbase.mttr.IntegrationTestMTTR.setUp(IntegrationTestMTTR.java:179)
>
> {noformat}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] sunhelly opened a new pull request #2882: HBASE-25507 Leak of ESTABLISHED sockets when compaction encountered "…
sunhelly opened a new pull request #2882: URL: https://github.com/apache/hbase/pull/2882 …java.io.IOException: Invalid HFile block magic" This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25507) Leak of ESTABLISHED sockets when compaction encountered "java.io.IOException: Invalid HFile block magic"
[
https://issues.apache.org/jira/browse/HBASE-25507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaolin Ha updated HBASE-25507:
---
Affects Version/s: 3.0.0-alpha-1
2.4.1
> Leak of ESTABLISHED sockets when compaction encountered "java.io.IOException:
> Invalid HFile block magic"
>
>
> Key: HBASE-25507
> URL: https://issues.apache.org/jira/browse/HBASE-25507
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha-1, 2.4.1
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Attachments: errorlogs.png,
> increasing-of-established-sockets-image.png, problem-region-move-logs.png
>
>
> Recently, we found socket leaks on our production cluster. The leaked sockets
> are in ESTABLISHED state. We found this happened on RS who owned a particular
> region from our analysis of metrics monitor and logs. RS without this region
> works normally.
> On the RS who owns the particular region, we found Exceptions as follows,
> {code:java}
> java.io.IOException: java.io.IOException: Could not seek
> StoreFileScanner[org.apache.hadoop.hbase.ioo
> .HalfStoreFileReader$1@5388be2f, cur=null] to key
> org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFF
> amilyCell@25aa56fd
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.parallelSeek(StoreScanner.java:1128)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:437)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:302)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.createScanner(Compactor.java:88
> 06)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor$StripeInternalScannerFacc
> tory.createScanner(StripeCompactor.java:82)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:316)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor.compact(StripeCompactor..
> java:120)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactionPolicy$SplitStripeCompacc
> tionRequest.execute(StripeCompactionPolicy.java:662)
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreEngine$StripeCompaction.compact(StripeStoo
> reEngine.java:114)
> at
> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1461)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2121)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(Comm
> pactSplitThread.java:519)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitt
> Thread.java:555)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Could not seek
> StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoree
> FileReader$1@5388be2f, cur=null] to key
> org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFamilyCell@@
> 25aa56fd
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:229)
> at
> org.apache.hadoop.hbase.regionserver.handler.ParallelSeekHandler.process(ParallelSeekHandd
> ler.java:56)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
> ... 3 more
> Caused by: java.io.IOException: Invalid HFile block magic:
> \x00\x00\x00\x00\x00\x00\x00\x00
> at
> org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:159)
> at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:171)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.createFromBuff(HFileBlock.java:333)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlockk
> .java:1753)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:155
> 52)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:539)
> at
> org.apache.hadoop.hbase.io.hfile.HFileScannerImpl.readAndUpdateNewBlock(HFileScannerImpl..
> java:737)
> at
> org.apache.hadoop.hbase.io.hfile.HFileScannerImpl.seekTo(HFileScannerImpl.java:726)
> at
> org.apache.hadoop.hbase.io.HalfStoreFileReader$1.seekTo(HalfStoreFileReader.java:161)
> at
>
[GitHub] [hbase] Apache-HBase commented on pull request #2861: HBASE-25455 Add trace support for HRegion read/write operation
Apache-HBase commented on pull request #2861: URL: https://github.com/apache/hbase/pull/2861#issuecomment-760357647 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 44s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-22120 Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 5s | HBASE-22120 passed | | +1 :green_heart: | compile | 2m 2s | HBASE-22120 passed | | +1 :green_heart: | shadedjars | 7m 7s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 30s | HBASE-22120 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 2s | the patch passed | | +1 :green_heart: | compile | 2m 2s | the patch passed | | +1 :green_heart: | javac | 2m 2s | the patch passed | | +1 :green_heart: | shadedjars | 7m 1s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 31s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 45s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 9s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 140m 22s | hbase-server in the patch passed. | | | | 177m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2861/8/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2861 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c8650d505a19 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-22120 / 1529dec9c3 | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2861/8/testReport/ | | Max. process+thread count | 4215 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2861/8/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25083) make sure the next hbase 1.y release has Hadoop 2.10 as a minimum version
[ https://issues.apache.org/jira/browse/HBASE-25083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun updated HBASE-25083: - Fix Version/s: 2.5.0 > make sure the next hbase 1.y release has Hadoop 2.10 as a minimum version > - > > Key: HBASE-25083 > URL: https://issues.apache.org/jira/browse/HBASE-25083 > Project: HBase > Issue Type: Task > Components: documentation, hadoop2 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.7.0, 2.3.4, 2.5.0, 2.4.1 > > > Our reference guide list of prerequisites still has Hadoop 2.8 and 2.9 listed > for HBase 1 releases. > * [hadoop 2.8 is > EOM|https://lists.apache.org/thread.html/r348f7bc93a522f05b7cce78a911854d128a6b1b8bd8124bad4d06ce6%40%3Cuser.hadoop.apache.org%3E] > * [hadoop 2.9 is > EOM|https://lists.apache.org/thread.html/r16b14cce9504f7a9d228612c6b808e72d8dd20863c78be51a7e04ed5%40%3Cuser.hadoop.apache.org%3E] > The current list in the reference guide for HBase 1.6 is just the 1.5 list > copied. we should update it to remove 2.8 and 2.9 and make sure we're no > longer doing build/test based on those versions for branch-1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25083) make sure the next hbase 1.y release has Hadoop 2.10 as a minimum version
[ https://issues.apache.org/jira/browse/HBASE-25083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun updated HBASE-25083: - Fix Version/s: 2.4.1 > make sure the next hbase 1.y release has Hadoop 2.10 as a minimum version > - > > Key: HBASE-25083 > URL: https://issues.apache.org/jira/browse/HBASE-25083 > Project: HBase > Issue Type: Task > Components: documentation, hadoop2 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.7.0, 2.3.4, 2.4.1 > > > Our reference guide list of prerequisites still has Hadoop 2.8 and 2.9 listed > for HBase 1 releases. > * [hadoop 2.8 is > EOM|https://lists.apache.org/thread.html/r348f7bc93a522f05b7cce78a911854d128a6b1b8bd8124bad4d06ce6%40%3Cuser.hadoop.apache.org%3E] > * [hadoop 2.9 is > EOM|https://lists.apache.org/thread.html/r16b14cce9504f7a9d228612c6b808e72d8dd20863c78be51a7e04ed5%40%3Cuser.hadoop.apache.org%3E] > The current list in the reference guide for HBase 1.6 is just the 1.5 list > copied. we should update it to remove 2.8 and 2.9 and make sure we're no > longer doing build/test based on those versions for branch-1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23340) hmaster /hbase/replication/rs session expired (hbase replication default value is true, we don't use ) causes logcleaner can not clean oldWALs, which resulits in old
[
https://issues.apache.org/jira/browse/HBASE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265107#comment-17265107
]
Hudson commented on HBASE-23340:
Results for branch branch-2
[build #158 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> hmaster /hbase/replication/rs session expired (hbase replication default
> value is true, we don't use ) causes logcleaner can not clean oldWALs, which
> resulits in oldWALs too large (more than 2TB)
> -
>
> Key: HBASE-23340
> URL: https://issues.apache.org/jira/browse/HBASE-23340
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 3.0.0-alpha-1, 2.2.3
>Reporter: jackylau
>Assignee: Bo Cui
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
> Attachments: Snipaste_2019-11-21_10-39-25.png,
> Snipaste_2019-11-21_14-10-36.png
>
>
> hmaster /hbase/replication/rs session expired (hbase replication default
> value is true, we don't use ) causes logcleaner can not clean oldWALs, which
> resulits in oldWALs too large (more than 2TB).
> !Snipaste_2019-11-21_10-39-25.png!
>
> !Snipaste_2019-11-21_14-10-36.png!
>
> we can solve it by following :
> 1) increase the session timeout(but i think it is not a good idea. because we
> do not know how long to set is suitable)
> 2) close the hbase replication. It is not a good idea too, when our user uses
> this feature
> 3) we need add retry times, for example when it has already happened three
> times, we set the ReplicationLogCleaner and SnapShotCleaner stop
> that is all my ideas, i do not konw it is suitable, If it is suitable, could
> i commit a PR?
> Does anynode have a good idea.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25502) IntegrationTestMTTR fails with TableNotFoundException
[
https://issues.apache.org/jira/browse/HBASE-25502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265105#comment-17265105
]
Hudson commented on HBASE-25502:
Results for branch branch-2
[build #158 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/158/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> IntegrationTestMTTR fails with TableNotFoundException
> -
>
> Key: HBASE-25502
> URL: https://issues.apache.org/jira/browse/HBASE-25502
> Project: HBase
> Issue Type: Test
> Components: integration tests
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.1, 2.3.5
>
>
> Admin.getDescriptor() fails beacuse table doesn't exist.
>
> {noformat}
> org.apache.hadoop.hbase.TableNotFoundException: IntegrationTestMTTR
> at java.lang.Thread.getStackTrace(Thread.java:1559)
> at
> org.apache.hadoop.hbase.util.FutureUtils.setStackTrace(FutureUtils.java:130)
> at
> org.apache.hadoop.hbase.util.FutureUtils.rethrow(FutureUtils.java:149)
> at org.apache.hadoop.hbase.util.FutureUtils.get(FutureUtils.java:172)
> at
> org.apache.hadoop.hbase.client.AdminOverAsyncAdmin.getDescriptor(AdminOverAsyncAdmin.java:163)
> at
> org.apache.hadoop.hbase.mttr.IntegrationTestMTTR.setupTables(IntegrationTestMTTR.java:236)
> at
> org.apache.hadoop.hbase.mttr.IntegrationTestMTTR.setUp(IntegrationTestMTTR.java:179)
>
> {noformat}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25507) Leak of ESTABLISHED sockets when encountered "java.io.IOException: Invalid HFile block magic"
[
https://issues.apache.org/jira/browse/HBASE-25507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264993#comment-17264993
]
Xiaolin Ha edited comment on HBASE-25507 at 1/14/21, 4:19 PM:
--
When creating InternalScanner for Compactor, all selected files for this
compaction will be created Readers in the new StoreScanner, which is also
calling seek() for all the selected files internal. When the seek() of one or
more file throws Exception like upwards, the InternalScanner will not be
created, so the finally close() for the InternalScanner will not work. And
what's worse, even only one file seeks fail, all the selected file Readers
which are newly created for the compaction will not be closed. While the
compaction is running once by once, there are leaks of ESTABLISHED sockets.
was (Author: xiaolin ha):
When creating InternalScanner for Compactor, all selected files for this
compaction will be created new StoreScanner, which calling seek() for all the
selected files internal. When the seek() of one or more file throws Exception
like upwards, the InternalScanner will not be created, so the finally close()
will not work, and even only one file seeks fail, all the selected file Readers
which are newly created for the compaction will not be closed. While the
compaction is running once by once, there are leaks of ESTABLISHED sockets.
> Leak of ESTABLISHED sockets when encountered "java.io.IOException: Invalid
> HFile block magic"
> -
>
> Key: HBASE-25507
> URL: https://issues.apache.org/jira/browse/HBASE-25507
> Project: HBase
> Issue Type: Improvement
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Attachments: errorlogs.png,
> increasing-of-established-sockets-image.png, problem-region-move-logs.png
>
>
> Recently, we found socket leaks on our production cluster. The leaked sockets
> are in ESTABLISHED state. We found this happened on RS who owned a particular
> region from our analysis of metrics monitor and logs. RS without this region
> works normally.
> On the RS who owns the particular region, we found Exceptions as follows,
> {code:java}
> java.io.IOException: java.io.IOException: Could not seek
> StoreFileScanner[org.apache.hadoop.hbase.ioo
> .HalfStoreFileReader$1@5388be2f, cur=null] to key
> org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFF
> amilyCell@25aa56fd
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.parallelSeek(StoreScanner.java:1128)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:437)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:302)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.createScanner(Compactor.java:88
> 06)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor$StripeInternalScannerFacc
> tory.createScanner(StripeCompactor.java:82)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:316)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor.compact(StripeCompactor..
> java:120)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactionPolicy$SplitStripeCompacc
> tionRequest.execute(StripeCompactionPolicy.java:662)
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreEngine$StripeCompaction.compact(StripeStoo
> reEngine.java:114)
> at
> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1461)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2121)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(Comm
> pactSplitThread.java:519)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitt
> Thread.java:555)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Could not seek
> StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoree
> FileReader$1@5388be2f, cur=null] to key
> org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFamilyCell@@
> 25aa56fd
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:229)
> at
> org.apache.hadoop.hbase.regionserver.handler.ParallelSeekHandler.process(ParallelSeekHandd
> ler.java:56)
> at
>
[jira] [Created] (HBASE-25508) Add an example of using the thrift proxy in thrift-over-http mode
Sean Busbey created HBASE-25508: --- Summary: Add an example of using the thrift proxy in thrift-over-http mode Key: HBASE-25508 URL: https://issues.apache.org/jira/browse/HBASE-25508 Project: HBase Issue Type: Improvement Components: Thrift Reporter: Sean Busbey Assignee: Sean Busbey add an example that shows using the thrift proxy with over-http enabled -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-25508) Add an example of using the thrift proxy in thrift-over-http mode
[ https://issues.apache.org/jira/browse/HBASE-25508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-25508 started by Sean Busbey. --- > Add an example of using the thrift proxy in thrift-over-http mode > - > > Key: HBASE-25508 > URL: https://issues.apache.org/jira/browse/HBASE-25508 > Project: HBase > Issue Type: Improvement > Components: Thrift >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Minor > > add an example that shows using the thrift proxy with over-http enabled -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25083) make sure the next hbase 1.y release has Hadoop 2.10 as a minimum version
[ https://issues.apache.org/jira/browse/HBASE-25083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun resolved HBASE-25083. -- Resolution: Fixed > make sure the next hbase 1.y release has Hadoop 2.10 as a minimum version > - > > Key: HBASE-25083 > URL: https://issues.apache.org/jira/browse/HBASE-25083 > Project: HBase > Issue Type: Task > Components: documentation, hadoop2 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 3.0.0-alpha-1, 1.7.0, 2.3.4, 2.5.0, 2.4.1 > > > Our reference guide list of prerequisites still has Hadoop 2.8 and 2.9 listed > for HBase 1 releases. > * [hadoop 2.8 is > EOM|https://lists.apache.org/thread.html/r348f7bc93a522f05b7cce78a911854d128a6b1b8bd8124bad4d06ce6%40%3Cuser.hadoop.apache.org%3E] > * [hadoop 2.9 is > EOM|https://lists.apache.org/thread.html/r16b14cce9504f7a9d228612c6b808e72d8dd20863c78be51a7e04ed5%40%3Cuser.hadoop.apache.org%3E] > The current list in the reference guide for HBase 1.6 is just the 1.5 list > copied. we should update it to remove 2.8 and 2.9 and make sure we're no > longer doing build/test based on those versions for branch-1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on pull request #2858: HBASE-25482 Improve SimpleRegionNormalizer#getAverageRegionSizeMb
ndimiduk commented on pull request #2858: URL: https://github.com/apache/hbase/pull/2858#issuecomment-760385540 Any further comments/concerns @wchevreuil ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25449) 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml
[ https://issues.apache.org/jira/browse/HBASE-25449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun updated HBASE-25449: - Fix Version/s: (was: 2.3.5) 2.3.4 > 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml > - > > Key: HBASE-25449 > URL: https://issues.apache.org/jira/browse/HBASE-25449 > Project: HBase > Issue Type: Improvement > Components: conf >Affects Versions: 2.0.1 >Reporter: shenshengli >Assignee: shenshengli >Priority: Major > Fix For: 3.0.0-alpha-1, 2.2.7, 2.3.4, 2.5.0, 2.4.2 > > > I think this parameter is not suitable for in hbase-default.xml, because in > this case, HDFS explicitly set to "dfs.client.read.shortcircuit=true", hbase > rely on HDFS configuration, the parameters in hbase service still is > false.Must be explicitly in hbase-site.xml is set to > "dfs.client.read.shortcircuit=true" to take effect.The same goes for the > parameter 'dfs.domain.socket.path', which is already covered in the specific > change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25507) Leak of ESTABLISHED sockets when encountered "java.io.IOException: Invalid HFile block magic"
[
https://issues.apache.org/jira/browse/HBASE-25507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264993#comment-17264993
]
Xiaolin Ha commented on HBASE-25507:
When creating InternalScanner for Compactor, all selected files for this
compaction will be created new StoreScanner, which calling seek() for all the
selected files internal. When the seek() of one or more file throws Exception
like upwards, the InternalScanner will not be created, so the finally close()
will not work, and even only one file seeks fail, all the selected file Readers
which are newly created for the compaction will not be closed. While the
compaction is running once by once, there are leaks of ESTABLISHED sockets.
> Leak of ESTABLISHED sockets when encountered "java.io.IOException: Invalid
> HFile block magic"
> -
>
> Key: HBASE-25507
> URL: https://issues.apache.org/jira/browse/HBASE-25507
> Project: HBase
> Issue Type: Improvement
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Attachments: errorlogs.png,
> increasing-of-established-sockets-image.png, problem-region-move-logs.png
>
>
> Recently, we found socket leaks on our production cluster. The leaked sockets
> are in ESTABLISHED state. We found this happened on RS who owned a particular
> region from our analysis of metrics monitor and logs. RS without this region
> works normally.
> On the RS who owns the particular region, we found Exceptions as follows,
> {code:java}
> java.io.IOException: java.io.IOException: Could not seek
> StoreFileScanner[org.apache.hadoop.hbase.ioo
> .HalfStoreFileReader$1@5388be2f, cur=null] to key
> org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFF
> amilyCell@25aa56fd
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.parallelSeek(StoreScanner.java:1128)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:437)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:302)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.createScanner(Compactor.java:88
> 06)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor$StripeInternalScannerFacc
> tory.createScanner(StripeCompactor.java:82)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:316)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor.compact(StripeCompactor..
> java:120)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactionPolicy$SplitStripeCompacc
> tionRequest.execute(StripeCompactionPolicy.java:662)
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreEngine$StripeCompaction.compact(StripeStoo
> reEngine.java:114)
> at
> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1461)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2121)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(Comm
> pactSplitThread.java:519)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitt
> Thread.java:555)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Could not seek
> StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoree
> FileReader$1@5388be2f, cur=null] to key
> org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFamilyCell@@
> 25aa56fd
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:229)
> at
> org.apache.hadoop.hbase.regionserver.handler.ParallelSeekHandler.process(ParallelSeekHandd
> ler.java:56)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
> ... 3 more
> Caused by: java.io.IOException: Invalid HFile block magic:
> \x00\x00\x00\x00\x00\x00\x00\x00
> at
> org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:159)
> at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:171)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.createFromBuff(HFileBlock.java:333)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlockk
> .java:1753)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:155
> 52)
> at
>
[jira] [Commented] (HBASE-25449) 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml
[
https://issues.apache.org/jira/browse/HBASE-25449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264992#comment-17264992
]
Hudson commented on HBASE-25449:
Results for branch branch-2.3
[build #151 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/151/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/151/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/151/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/151/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/151/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml
> -
>
> Key: HBASE-25449
> URL: https://issues.apache.org/jira/browse/HBASE-25449
> Project: HBase
> Issue Type: Improvement
> Components: conf
>Affects Versions: 2.0.1
>Reporter: shenshengli
>Assignee: shenshengli
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.3.5, 2.4.2
>
>
> I think this parameter is not suitable for in hbase-default.xml, because in
> this case, HDFS explicitly set to "dfs.client.read.shortcircuit=true", hbase
> rely on HDFS configuration, the parameters in hbase service still is
> false.Must be explicitly in hbase-site.xml is set to
> "dfs.client.read.shortcircuit=true" to take effect.The same goes for the
> parameter 'dfs.domain.socket.path', which is already covered in the specific
> change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2882: HBASE-25507 Leak of ESTABLISHED sockets when compaction encountered "…
Apache-HBase commented on pull request #2882: URL: https://github.com/apache/hbase/pull/2882#issuecomment-760342630 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 50s | master passed | | +1 :green_heart: | compile | 3m 9s | master passed | | +1 :green_heart: | checkstyle | 1m 4s | master passed | | +1 :green_heart: | spotbugs | 2m 2s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 29s | the patch passed | | +1 :green_heart: | compile | 3m 1s | the patch passed | | +1 :green_heart: | javac | 3m 1s | the patch passed | | -0 :warning: | checkstyle | 1m 2s | hbase-server: The patch generated 1 new + 11 unchanged - 0 fixed = 12 total (was 11) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 8s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 9s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 45m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2882 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 8f189debdc19 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3488c44a21 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 95 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25502) IntegrationTestMTTR fails with TableNotFoundException
[
https://issues.apache.org/jira/browse/HBASE-25502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huaxiang Sun updated HBASE-25502:
-
Fix Version/s: (was: 2.3.5)
2.3.4
> IntegrationTestMTTR fails with TableNotFoundException
> -
>
> Key: HBASE-25502
> URL: https://issues.apache.org/jira/browse/HBASE-25502
> Project: HBase
> Issue Type: Test
> Components: integration tests
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.4, 2.4.1
>
>
> Admin.getDescriptor() fails beacuse table doesn't exist.
>
> {noformat}
> org.apache.hadoop.hbase.TableNotFoundException: IntegrationTestMTTR
> at java.lang.Thread.getStackTrace(Thread.java:1559)
> at
> org.apache.hadoop.hbase.util.FutureUtils.setStackTrace(FutureUtils.java:130)
> at
> org.apache.hadoop.hbase.util.FutureUtils.rethrow(FutureUtils.java:149)
> at org.apache.hadoop.hbase.util.FutureUtils.get(FutureUtils.java:172)
> at
> org.apache.hadoop.hbase.client.AdminOverAsyncAdmin.getDescriptor(AdminOverAsyncAdmin.java:163)
> at
> org.apache.hadoop.hbase.mttr.IntegrationTestMTTR.setupTables(IntegrationTestMTTR.java:236)
> at
> org.apache.hadoop.hbase.mttr.IntegrationTestMTTR.setUp(IntegrationTestMTTR.java:179)
>
> {noformat}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25508) Add a python example of using the thrift proxy in thrift-over-http mode
[ https://issues.apache.org/jira/browse/HBASE-25508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-25508: Summary: Add a python example of using the thrift proxy in thrift-over-http mode (was: Add an example of using the thrift proxy in thrift-over-http mode) > Add a python example of using the thrift proxy in thrift-over-http mode > --- > > Key: HBASE-25508 > URL: https://issues.apache.org/jira/browse/HBASE-25508 > Project: HBase > Issue Type: Improvement > Components: Thrift >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Minor > > add an example that shows using the thrift proxy with over-http enabled -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25505) ZK watcher threads are daemonized; reconsider
[ https://issues.apache.org/jira/browse/HBASE-25505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265543#comment-17265543 ] Bharath Vissapragada commented on HBASE-25505: -- Sure [~apurtell]. > ZK watcher threads are daemonized; reconsider > - > > Key: HBASE-25505 > URL: https://issues.apache.org/jira/browse/HBASE-25505 > Project: HBase > Issue Type: Brainstorming >Reporter: Andrew Kyle Purtell >Priority: Major > Attachments: ScheduledChore.cleanup-not-called.diff > > > On HBASE-25279 there was some discussion and difference of opinion about > having ZK watcher pool threads be daemonized. This is not necessarily a > problem but should be reconsidered. > Daemon threads are subject to abrupt termination during JVM shutdown and > therefore may be interrupted before state changes are complete or resources > are released. > As long as ZK watchers are properly closed by shutdown logic the pool threads > will be terminated in a controlled manner and the JVM will exit. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2866: HBASE-25478 - Implement retries when enabling tables in TestRegionReplicaReplicationEndpoint
Apache-HBase commented on pull request #2866: URL: https://github.com/apache/hbase/pull/2866#issuecomment-760549886 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 6s | master passed | | +1 :green_heart: | compile | 1m 6s | master passed | | +1 :green_heart: | shadedjars | 6m 37s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | the patch passed | | +1 :green_heart: | compile | 1m 7s | the patch passed | | +1 :green_heart: | javac | 1m 7s | the patch passed | | +1 :green_heart: | shadedjars | 6m 38s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 132m 52s | hbase-server in the patch failed. | | | | 160m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2866/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2866 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 79e68912b523 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3cc2468300 | | Default Java | AdoptOpenJDK-11.0.6+10 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2866/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2866/4/testReport/ | | Max. process+thread count | 4172 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2866/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25509) ChoreService.cancelChore will not call ScheduledChore.cleanup which may lead to resource leak
Duo Zhang created HBASE-25509: - Summary: ChoreService.cancelChore will not call ScheduledChore.cleanup which may lead to resource leak Key: HBASE-25509 URL: https://issues.apache.org/jira/browse/HBASE-25509 Project: HBase Issue Type: Sub-task Components: util Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25449) 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml
[ https://issues.apache.org/jira/browse/HBASE-25449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25449: Fix Version/s: (was: 2.4.2) 2.4.1 > 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml > - > > Key: HBASE-25449 > URL: https://issues.apache.org/jira/browse/HBASE-25449 > Project: HBase > Issue Type: Improvement > Components: conf >Affects Versions: 2.0.1 >Reporter: shenshengli >Assignee: shenshengli >Priority: Major > Fix For: 3.0.0-alpha-1, 2.2.7, 2.3.4, 2.5.0, 2.4.1 > > > I think this parameter is not suitable for in hbase-default.xml, because in > this case, HDFS explicitly set to "dfs.client.read.shortcircuit=true", hbase > rely on HDFS configuration, the parameters in hbase service still is > false.Must be explicitly in hbase-site.xml is set to > "dfs.client.read.shortcircuit=true" to take effect.The same goes for the > parameter 'dfs.domain.socket.path', which is already covered in the specific > change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25249) Adding StoreContext
[ https://issues.apache.org/jira/browse/HBASE-25249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265122#comment-17265122 ] Tak-Lon (Stephen) Wu commented on HBASE-25249: -- [~apurtell] Andrew, if we want to cherry-pick [2bb7beb448a92e39da4a39de9ec1af3b93328a03|https://github.com/apache/hbase/commit/2bb7beb448a92e39da4a39de9ec1af3b93328a03] to branch-2.4, do we need a new PR ? also I found it cherry-pick is clean > Adding StoreContext > --- > > Key: HBASE-25249 > URL: https://issues.apache.org/jira/browse/HBASE-25249 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-1, 2.4.0, 2.3.4 >Reporter: Abhishek Khanna >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > Adding StoreContext which contains the metadata about the HStore. This meta > data can be used across the HFileWriter/Readers and other HStore consumers > without the need of passing around the complete store and exposing its > internals. This is a refactoring change which cleans up the existing code so > that the following commits can leverage the context for a more maintainable > code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2861: HBASE-25455 Add trace support for HRegion read/write operation
Apache-HBase commented on pull request #2861: URL: https://github.com/apache/hbase/pull/2861#issuecomment-760403876 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-22120 Compile Tests _ | | +0 :ok: | mvndep | 0m 36s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 4s | HBASE-22120 passed | | +1 :green_heart: | compile | 1m 49s | HBASE-22120 passed | | +1 :green_heart: | shadedjars | 7m 37s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 18s | HBASE-22120 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 53s | the patch passed | | +1 :green_heart: | compile | 1m 46s | the patch passed | | +1 :green_heart: | javac | 1m 46s | the patch passed | | +1 :green_heart: | shadedjars | 7m 42s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 32s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 10s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 212m 3s | hbase-server in the patch passed. | | | | 248m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2861/8/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2861 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux fe8f512a3cce 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-22120 / 1529dec9c3 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2861/8/testReport/ | | Max. process+thread count | 3599 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2861/8/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-24972) Wait for connection attempt to succeed before performing operations on ZK
[
https://issues.apache.org/jira/browse/HBASE-24972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265148#comment-17265148
]
Prathyusha edited comment on HBASE-24972 at 1/14/21, 7:21 PM:
--
[~stack] Below is the stack trace of a failure incident we have seen -
Cause: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/table/SYSTEM.CATALOG
StackTrace:
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1337)
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354)
org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:625)
...
StackTraceId: 429763122
But yes, I see the retries in place where ever we are doing write operations.
[~sandeep.guggilam] These retries should suffice I guess. Any thoughts?
was (Author: prathyu6):
[~stack] Below is the stack trace of a failure incident we have seen -
Cause: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/table/SYSTEM.CATALOG
StackTrace:
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1337)
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354)
org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:625)
...
StackTraceId: 429763122
But yes, I see the retries in place where ever we are doing write operations.
[~sandeep.guggilam] These retries should suffice I guess. Any thoughts?
> Wait for connection attempt to succeed before performing operations on ZK
> -
>
> Key: HBASE-24972
> URL: https://issues.apache.org/jira/browse/HBASE-24972
> Project: HBase
> Issue Type: Bug
>Reporter: Sandeep Guggilam
>Assignee: Prathyusha
>Priority: Minor
>
> {color:#1d1c1d}Creating the connection with ZK is asynchronous and notified
> via the passed in watcher about the successful connection event. When we
> attempt any operations, we try to create a connection and then perform a
> read/write
> ({color}{color:#1d1c1d}[https://github.com/apache/hbase/blob/979edfe72046b2075adcc869c65ae820e6f3ec2d/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java#L323]{color}{color:#1d1c1d})
> without really waiting for the notification event
> ([https://github.com/apache/hbase/blob/979edfe72046b2075adcc869c65ae820e6f3ec2d/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKWatcher.java#L582)]{color}
>
> {color:#1d1c1d}It is possible we get ConnectionLoss errors when we perform
> operations on ZK without waiting for the connection attempt to succeed{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24972) Wait for connection attempt to succeed before performing operations on ZK
[
https://issues.apache.org/jira/browse/HBASE-24972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265148#comment-17265148
]
Prathyusha commented on HBASE-24972:
[~stack] Below is the stack trace of a failure incident we have seen -
Cause: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/table/SYSTEM.CATALOG
StackTrace:
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1337)
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354)
org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:625)
...
StackTraceId: 429763122
But yes, I see the retries in place where ever we are doing write operations.
[~sandeep.guggilam] These retries should suffice I guess. Any thoughts?
> Wait for connection attempt to succeed before performing operations on ZK
> -
>
> Key: HBASE-24972
> URL: https://issues.apache.org/jira/browse/HBASE-24972
> Project: HBase
> Issue Type: Bug
>Reporter: Sandeep Guggilam
>Assignee: Prathyusha
>Priority: Minor
>
> {color:#1d1c1d}Creating the connection with ZK is asynchronous and notified
> via the passed in watcher about the successful connection event. When we
> attempt any operations, we try to create a connection and then perform a
> read/write
> ({color}{color:#1d1c1d}[https://github.com/apache/hbase/blob/979edfe72046b2075adcc869c65ae820e6f3ec2d/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java#L323]{color}{color:#1d1c1d})
> without really waiting for the notification event
> ([https://github.com/apache/hbase/blob/979edfe72046b2075adcc869c65ae820e6f3ec2d/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKWatcher.java#L582)]{color}
>
> {color:#1d1c1d}It is possible we get ConnectionLoss errors when we perform
> operations on ZK without waiting for the connection attempt to succeed{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk merged pull request #2754: HBASE-25375 Provide a VM-based release environment
ndimiduk merged pull request #2754: URL: https://github.com/apache/hbase/pull/2754 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2237: HBASE-24833: Bootstrap should not delete the META table directory if …
Apache-HBase commented on pull request #2237: URL: https://github.com/apache/hbase/pull/2237#issuecomment-760552795 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 8s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 46s | branch-2 passed | | +1 :green_heart: | compile | 0m 59s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 17s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | branch-2 passed | | -0 :warning: | patch | 7m 10s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 14s | the patch passed | | +1 :green_heart: | compile | 0m 59s | the patch passed | | +1 :green_heart: | javac | 0m 59s | the patch passed | | +1 :green_heart: | shadedjars | 6m 1s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 138m 42s | hbase-server in the patch passed. | | | | 164m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2237/4/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2237 | | JIRA Issue | HBASE-24833 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e39ecde1ac07 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 2bb7beb448 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2237/4/testReport/ | | Max. process+thread count | 4342 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2237/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2866: HBASE-25478 - Implement retries when enabling tables in TestRegionReplicaReplicationEndpoint
Apache-HBase commented on pull request #2866: URL: https://github.com/apache/hbase/pull/2866#issuecomment-760553262 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 35s | master passed | | +1 :green_heart: | compile | 0m 58s | master passed | | +1 :green_heart: | shadedjars | 6m 32s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 31s | the patch passed | | +1 :green_heart: | compile | 0m 56s | the patch passed | | +1 :green_heart: | javac | 0m 56s | the patch passed | | +1 :green_heart: | shadedjars | 6m 42s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 144m 53s | hbase-server in the patch passed. | | | | 171m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2866/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2866 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ec9f2127848e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3cc2468300 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2866/4/testReport/ | | Max. process+thread count | 4221 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2866/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2237: HBASE-24833: Bootstrap should not delete the META table directory if …
Apache-HBase commented on pull request #2237: URL: https://github.com/apache/hbase/pull/2237#issuecomment-760575353 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 14s | Docker mode activated. | | -0 :warning: | yetus | 0m 8s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 46s | branch-2 passed | | +1 :green_heart: | compile | 1m 11s | branch-2 passed | | +1 :green_heart: | shadedjars | 7m 28s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | branch-2 passed | | -0 :warning: | patch | 8m 21s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 32s | the patch passed | | +1 :green_heart: | compile | 1m 12s | the patch passed | | +1 :green_heart: | javac | 1m 12s | the patch passed | | +1 :green_heart: | shadedjars | 7m 27s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 197m 39s | hbase-server in the patch passed. | | | | 228m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2237/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2237 | | JIRA Issue | HBASE-24833 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3d27205dc3dd 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 2bb7beb448 | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2237/4/testReport/ | | Max. process+thread count | 2576 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2237/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25505) ZK watcher threads are daemonized; reconsider
[ https://issues.apache.org/jira/browse/HBASE-25505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265138#comment-17265138 ] Bharath Vissapragada commented on HBASE-25505: -- [~larsh] Thanks for narrowing it down the to the ReplicationLogCleaner. [~zhangduo] That makes sense to me, nice find. Agree that the cleanup needs to be made idempotent to avoid any issues in teardown. You want to spin up a patch or you want me to do it (or if anyone else wants to own it)..? > ZK watcher threads are daemonized; reconsider > - > > Key: HBASE-25505 > URL: https://issues.apache.org/jira/browse/HBASE-25505 > Project: HBase > Issue Type: Brainstorming >Reporter: Andrew Kyle Purtell >Priority: Major > Attachments: ScheduledChore.cleanup-not-called.diff > > > On HBASE-25279 there was some discussion and difference of opinion about > having ZK watcher pool threads be daemonized. This is not necessarily a > problem but should be reconsidered. > Daemon threads are subject to abrupt termination during JVM shutdown and > therefore may be interrupted before state changes are complete or resources > are released. > As long as ZK watchers are properly closed by shutdown logic the pool threads > will be terminated in a controlled manner and the JVM will exit. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25249) Adding StoreContext
[ https://issues.apache.org/jira/browse/HBASE-25249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265136#comment-17265136 ] Tak-Lon (Stephen) Wu commented on HBASE-25249: -- thanks a lot !! > Adding StoreContext > --- > > Key: HBASE-25249 > URL: https://issues.apache.org/jira/browse/HBASE-25249 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-1, 2.4.0, 2.3.4 >Reporter: Abhishek Khanna >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.1 > > > Adding StoreContext which contains the metadata about the HStore. This meta > data can be used across the HFileWriter/Readers and other HStore consumers > without the need of passing around the complete store and exposing its > internals. This is a refactoring change which cleans up the existing code so > that the following commits can leverage the context for a more maintainable > code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2866: HBASE-25478 - Implement retries when enabling tables in TestRegionReplicaReplicationEndpoint
Apache-HBase commented on pull request #2866: URL: https://github.com/apache/hbase/pull/2866#issuecomment-760409579 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 6s | master passed | | +1 :green_heart: | compile | 4m 4s | master passed | | +1 :green_heart: | checkstyle | 1m 27s | master passed | | +1 :green_heart: | spotbugs | 2m 20s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 27s | the patch passed | | +1 :green_heart: | compile | 3m 54s | the patch passed | | +1 :green_heart: | javac | 3m 54s | the patch passed | | +1 :green_heart: | checkstyle | 1m 11s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 22m 24s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 0s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 18s | The patch does not generate ASF License warnings. | | | | 58m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2866/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2866 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 4f2eb2a2eedb 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3488c44a21 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2866/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25371) When openRegion fails during initial verification(before initializing and setting seq num), exception is observed during region close.
[ https://issues.apache.org/jira/browse/HBASE-25371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25371: Fix Version/s: (was: 2.4.1) 2.4.2 > When openRegion fails during initial verification(before initializing and > setting seq num), exception is observed during region close. > -- > > Key: HBASE-25371 > URL: https://issues.apache.org/jira/browse/HBASE-25371 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 2.2.3 >Reporter: Ajeet Rai >Assignee: Mohammad Arshad >Priority: Major > Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.3.5, 2.4.2 > > > When openRegion fails during initial verification(before initializing and > setting seq num), exception is observed during region close: > > 2020-12-03 16:34:47,133 ERROR > [RS_OPEN_REGION-regionserver/AA:16040-0] handler.OpenRegionHandler: > Failed open of > region=ns2:testtable4,15,1606912406234.cd386135276b7d3c57416df3666e4aea.2020-12-03 > 16:34:47,133 ERROR [RS_OPEN_REGION-regionserver/blrphispra01054:16040-0] > handler.OpenRegionHandler: Failed open of > region=ns2:testtable4,15,1606912406234.cd386135276b7d3c57416df3666e4aea.java.io.IOException: > The new max sequence id 1 is less than the old max sequence id 7134 at > org.apache.hadoop.hbase.wal.WALSplitUtil.writeRegionSequenceIdFile(WALSplitUtil.java:418) > at > org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1253) > at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1793) > at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1606) at > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1552) at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7522) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7467) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7439) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7397) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7348) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:286) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:111) > at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2866: HBASE-25478 - Implement retries when enabling tables in TestRegionReplicaReplicationEndpoint
Apache-HBase commented on pull request #2866: URL: https://github.com/apache/hbase/pull/2866#issuecomment-760500917 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 33s | master passed | | +1 :green_heart: | compile | 3m 12s | master passed | | +1 :green_heart: | checkstyle | 1m 2s | master passed | | +1 :green_heart: | spotbugs | 2m 5s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 37s | the patch passed | | +1 :green_heart: | compile | 3m 26s | the patch passed | | +1 :green_heart: | javac | 3m 26s | the patch passed | | +1 :green_heart: | checkstyle | 1m 5s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 15s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 46m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2866/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2866 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 5861cb6d5c01 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3cc2468300 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2866/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25277) postScannerFilterRow impacts Scan performance a lot in HBase 2.x
[ https://issues.apache.org/jira/browse/HBASE-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25277: Fix Version/s: (was: 2.4.0) 2.4.1 > postScannerFilterRow impacts Scan performance a lot in HBase 2.x > > > Key: HBASE-25277 > URL: https://issues.apache.org/jira/browse/HBASE-25277 > Project: HBase > Issue Type: Bug > Components: Coprocessors, scan >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Critical > Labels: perfomance, scanning > Fix For: 3.0.0-alpha-1, 2.2.7, 2.3.4, 2.5.0, 2.4.1 > > Attachments: test_report.png > > > In our test we observed Scan performance is degraded by more than 60% in > HBase-2.x as compared to 1.3.x. As per the flamegraph report, RS spent 31% > of the time in postScannerFilterRow, however the coprocessors > (AccessController, VisibilityController & ConstraintProcessor) does nothing > in that hook. > HBASE-14489 added the logic to avoid the call to postScannerFilterRow when > not needed which is not working as expected in HBase 2.x. AccessController, > VisibilityController & ConstraintProcessor override the postScannerFilterRow > with dummy (same as RegionObserver) implementation, so > RegionCoprocessorHost.hasCustomPostScannerFilterRow will be TRUE and call the > hook for all configured CPs while processing each row . Suppose we have > configured 5 region CPs and there are 1 M rows in table, then there will be 1 > * 5 M dummy call to postScannerFilterRow during whole table scan. > We need to remove postScannerFilterRow hook from these CPs as these are not > doing anything. > Another problem is in RegionCoprocessorHost.hasCustomPostScannerFilterRow > init logic, currently it is always TRUE even though we remove > postScannerFilterRow hook from AccessController, VisibilityController & > ConstraintProcessor, because we are finding postScannerFilterRow until it > is found (we look in configured CP's super class also) or clazz is NULL. > https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L301 > Supper class of Object (super class of AccessController) will be NULL, so > RegionCoprocessorHost.hasCustomPostScannerFilterRow will be set as TRUE > https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-17686) Improve Javadoc comments in Observer Interfaces
[ https://issues.apache.org/jira/browse/HBASE-17686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York reassigned HBASE-17686: - Assignee: (was: Zach York) > Improve Javadoc comments in Observer Interfaces > --- > > Key: HBASE-17686 > URL: https://issues.apache.org/jira/browse/HBASE-17686 > Project: HBase > Issue Type: Improvement > Components: Coprocessors, documentation >Affects Versions: 2.0.0 >Reporter: Zach York >Priority: Minor > > Based off of comments from https://issues.apache.org/jira/browse/HBASE-17312, > we should improve Javadoc comments in the Observer interfaces. This JIRA > includes adding @returns to clarify what is being returned (and why) and to > either improve @params/@throws or remove if there is no way to provide > meaningful information. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2866: HBASE-25478 - Implement retries when enabling tables in TestRegionReplicaReplicationEndpoint
Apache-HBase commented on pull request #2866: URL: https://github.com/apache/hbase/pull/2866#issuecomment-760475844 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 49s | master passed | | +1 :green_heart: | compile | 0m 57s | master passed | | +1 :green_heart: | shadedjars | 6m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 29s | the patch passed | | +1 :green_heart: | compile | 0m 57s | the patch passed | | +1 :green_heart: | javac | 0m 57s | the patch passed | | +1 :green_heart: | shadedjars | 6m 36s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 145m 4s | hbase-server in the patch passed. | | | | 171m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2866/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2866 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1b874eae77cc 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3488c44a21 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2866/3/testReport/ | | Max. process+thread count | 3899 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2866/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2883: HBASE-25492: Create table with rsgroup
Apache-HBase commented on pull request #2883: URL: https://github.com/apache/hbase/pull/2883#issuecomment-760490173 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 56s | branch-2 passed | | +1 :green_heart: | compile | 2m 45s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 10s | branch-2 passed | | +1 :green_heart: | spotbugs | 2m 33s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 14s | the patch passed | | +1 :green_heart: | compile | 2m 45s | the patch passed | | -0 :warning: | javac | 0m 53s | hbase-rsgroup generated 8 new + 128 unchanged - 0 fixed = 136 total (was 128) | | -0 :warning: | checkstyle | 0m 12s | hbase-rsgroup: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2) | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 48s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 56s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. | | | | 41m 51s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2883/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2883 | | JIRA Issue | HBASE-25492 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux a8e6edca2978 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 2bb7beb448 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2883/1/artifact/yetus-general-check/output/diff-compile-javac-hbase-rsgroup.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2883/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-rsgroup.txt | | Max. process+thread count | 96 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-client hbase-rsgroup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2883/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25449) 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml
[
https://issues.apache.org/jira/browse/HBASE-25449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265381#comment-17265381
]
Hudson commented on HBASE-25449:
Results for branch branch-2.2
[build #155 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/155/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/155//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/155//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/155//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml
> -
>
> Key: HBASE-25449
> URL: https://issues.apache.org/jira/browse/HBASE-25449
> Project: HBase
> Issue Type: Improvement
> Components: conf
>Affects Versions: 2.0.1
>Reporter: shenshengli
>Assignee: shenshengli
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.3.4, 2.5.0, 2.4.1
>
>
> I think this parameter is not suitable for in hbase-default.xml, because in
> this case, HDFS explicitly set to "dfs.client.read.shortcircuit=true", hbase
> rely on HDFS configuration, the parameters in hbase service still is
> false.Must be explicitly in hbase-site.xml is set to
> "dfs.client.read.shortcircuit=true" to take effect.The same goes for the
> parameter 'dfs.domain.socket.path', which is already covered in the specific
> change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-25249) Adding StoreContext
[ https://issues.apache.org/jira/browse/HBASE-25249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-25249. - Resolution: Fixed > Adding StoreContext > --- > > Key: HBASE-25249 > URL: https://issues.apache.org/jira/browse/HBASE-25249 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-1, 2.4.0, 2.3.4 >Reporter: Abhishek Khanna >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.1 > > > Adding StoreContext which contains the metadata about the HStore. This meta > data can be used across the HFileWriter/Readers and other HStore consumers > without the need of passing around the complete store and exposing its > internals. This is a refactoring change which cleans up the existing code so > that the following commits can leverage the context for a more maintainable > code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25249) Adding StoreContext
[ https://issues.apache.org/jira/browse/HBASE-25249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265131#comment-17265131 ] Andrew Kyle Purtell commented on HBASE-25249: - I just picked it for you because I'm making 2.4.1RC1 right now. > Adding StoreContext > --- > > Key: HBASE-25249 > URL: https://issues.apache.org/jira/browse/HBASE-25249 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-1, 2.4.0, 2.3.4 >Reporter: Abhishek Khanna >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > Adding StoreContext which contains the metadata about the HStore. This meta > data can be used across the HFileWriter/Readers and other HStore consumers > without the need of passing around the complete store and exposing its > internals. This is a refactoring change which cleans up the existing code so > that the following commits can leverage the context for a more maintainable > code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25249) Adding StoreContext
[ https://issues.apache.org/jira/browse/HBASE-25249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25249: Fix Version/s: 2.4.1 > Adding StoreContext > --- > > Key: HBASE-25249 > URL: https://issues.apache.org/jira/browse/HBASE-25249 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-1, 2.4.0, 2.3.4 >Reporter: Abhishek Khanna >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.1 > > > Adding StoreContext which contains the metadata about the HStore. This meta > data can be used across the HFileWriter/Readers and other HStore consumers > without the need of passing around the complete store and exposing its > internals. This is a refactoring change which cleans up the existing code so > that the following commits can leverage the context for a more maintainable > code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25375) Provide a VM-based release environment
[ https://issues.apache.org/jira/browse/HBASE-25375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-25375. -- Release Note: Adds a Vagrant project for running releases under `dev-support/release-vm`. This is intended to be an environment wherein the `create-release` scripts can be run (in docker mode). See the directory's readme for details. Resolution: Fixed > Provide a VM-based release environment > -- > > Key: HBASE-25375 > URL: https://issues.apache.org/jira/browse/HBASE-25375 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 3.0.0-alpha-1 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > Fix For: 3.0.0-alpha-1 > > > In order to make running releases more smooth and "clean-room" on my macOS > dev machine, I put together a Vagrant environment for running release builds. > In particular, docker IO on macOS is terribly slow, so slow that going > through the virtualization kernel extensions is an accelerant. Of course, > YMMV. I've used this on all of the 2.3.x release I've done. Maybe other RMs > like it, maybe they don't. I'll post up a patch as a point of reference. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2883: HBASE-25492: Create table with rsgroup
Apache-HBase commented on pull request #2883: URL: https://github.com/apache/hbase/pull/2883#issuecomment-760487958 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 11s | branch-2 passed | | +1 :green_heart: | compile | 1m 23s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 45s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 19s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 56s | the patch passed | | +1 :green_heart: | compile | 1m 26s | the patch passed | | +1 :green_heart: | javac | 1m 26s | the patch passed | | +1 :green_heart: | shadedjars | 6m 43s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 18s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 48s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 2m 26s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 2m 58s | hbase-rsgroup in the patch passed. | | | | 37m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2883/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2883 | | JIRA Issue | HBASE-25492 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 29426805e270 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 2bb7beb448 | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2883/1/testReport/ | | Max. process+thread count | 3753 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-client hbase-rsgroup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2883/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-25249) Adding StoreContext
[ https://issues.apache.org/jira/browse/HBASE-25249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265122#comment-17265122 ] Tak-Lon (Stephen) Wu edited comment on HBASE-25249 at 1/14/21, 6:47 PM: [~apurtell] Andrew, if we want to cherry-pick [2bb7beb448a92e39da4a39de9ec1af3b93328a03|https://github.com/apache/hbase/commit/2bb7beb448a92e39da4a39de9ec1af3b93328a03] to branch-2.4, do we need a new PR ? also I found the cherry-pick is clean was (Author: taklwu): [~apurtell] Andrew, if we want to cherry-pick [2bb7beb448a92e39da4a39de9ec1af3b93328a03|https://github.com/apache/hbase/commit/2bb7beb448a92e39da4a39de9ec1af3b93328a03] to branch-2.4, do we need a new PR ? also I found it cherry-pick is clean > Adding StoreContext > --- > > Key: HBASE-25249 > URL: https://issues.apache.org/jira/browse/HBASE-25249 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-1, 2.4.0, 2.3.4 >Reporter: Abhishek Khanna >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > Adding StoreContext which contains the metadata about the HStore. This meta > data can be used across the HFileWriter/Readers and other HStore consumers > without the need of passing around the complete store and exposing its > internals. This is a refactoring change which cleans up the existing code so > that the following commits can leverage the context for a more maintainable > code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25249) Adding StoreContext
[ https://issues.apache.org/jira/browse/HBASE-25249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265127#comment-17265127 ] Andrew Kyle Purtell commented on HBASE-25249: - Clean cherry pick is fine. > Adding StoreContext > --- > > Key: HBASE-25249 > URL: https://issues.apache.org/jira/browse/HBASE-25249 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-1, 2.4.0, 2.3.4 >Reporter: Abhishek Khanna >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > Adding StoreContext which contains the metadata about the HStore. This meta > data can be used across the HFileWriter/Readers and other HStore consumers > without the need of passing around the complete store and exposing its > internals. This is a refactoring change which cleans up the existing code so > that the following commits can leverage the context for a more maintainable > code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25505) ZK watcher threads are daemonized; reconsider
[ https://issues.apache.org/jira/browse/HBASE-25505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265146#comment-17265146 ] Andrew Kyle Purtell commented on HBASE-25505: - I think the bug fix discussed should be a subtask or a separate issue as it will fix a bug but not change the daemon status of the zk watcher pool threads (I assume, as they are not directly related). Otherwise sgtm > ZK watcher threads are daemonized; reconsider > - > > Key: HBASE-25505 > URL: https://issues.apache.org/jira/browse/HBASE-25505 > Project: HBase > Issue Type: Brainstorming >Reporter: Andrew Kyle Purtell >Priority: Major > Attachments: ScheduledChore.cleanup-not-called.diff > > > On HBASE-25279 there was some discussion and difference of opinion about > having ZK watcher pool threads be daemonized. This is not necessarily a > problem but should be reconsidered. > Daemon threads are subject to abrupt termination during JVM shutdown and > therefore may be interrupted before state changes are complete or resources > are released. > As long as ZK watchers are properly closed by shutdown logic the pool threads > will be terminated in a controlled manner and the JVM will exit. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-25371) When openRegion fails during initial verification(before initializing and setting seq num), exception is observed during region close.
[ https://issues.apache.org/jira/browse/HBASE-25371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun reopened HBASE-25371: -- I just found that this Jira has not been merged to branch-2, branch-2.3,branch-2.4 yet. Can you do backport and set the release version correctly? Thanks. > When openRegion fails during initial verification(before initializing and > setting seq num), exception is observed during region close. > -- > > Key: HBASE-25371 > URL: https://issues.apache.org/jira/browse/HBASE-25371 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 2.2.3 >Reporter: Ajeet Rai >Assignee: Mohammad Arshad >Priority: Major > Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.4.1, 2.3.5 > > > When openRegion fails during initial verification(before initializing and > setting seq num), exception is observed during region close: > > 2020-12-03 16:34:47,133 ERROR > [RS_OPEN_REGION-regionserver/AA:16040-0] handler.OpenRegionHandler: > Failed open of > region=ns2:testtable4,15,1606912406234.cd386135276b7d3c57416df3666e4aea.2020-12-03 > 16:34:47,133 ERROR [RS_OPEN_REGION-regionserver/blrphispra01054:16040-0] > handler.OpenRegionHandler: Failed open of > region=ns2:testtable4,15,1606912406234.cd386135276b7d3c57416df3666e4aea.java.io.IOException: > The new max sequence id 1 is less than the old max sequence id 7134 at > org.apache.hadoop.hbase.wal.WALSplitUtil.writeRegionSequenceIdFile(WALSplitUtil.java:418) > at > org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1253) > at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1793) > at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1606) at > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1552) at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7522) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7467) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7439) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7397) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7348) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:286) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:111) > at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25371) When openRegion fails during initial verification(before initializing and setting seq num), exception is observed during region close.
[ https://issues.apache.org/jira/browse/HBASE-25371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265206#comment-17265206 ] Huaxiang Sun commented on HBASE-25371: -- FYI, [~apurtell], I see you are preparing for 2.4.1. You may want to rekick RC as this Jira is included in the CHANGES.md, but it does not show up in the git. > When openRegion fails during initial verification(before initializing and > setting seq num), exception is observed during region close. > -- > > Key: HBASE-25371 > URL: https://issues.apache.org/jira/browse/HBASE-25371 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 2.2.3 >Reporter: Ajeet Rai >Assignee: Mohammad Arshad >Priority: Major > Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.4.1, 2.3.5 > > > When openRegion fails during initial verification(before initializing and > setting seq num), exception is observed during region close: > > 2020-12-03 16:34:47,133 ERROR > [RS_OPEN_REGION-regionserver/AA:16040-0] handler.OpenRegionHandler: > Failed open of > region=ns2:testtable4,15,1606912406234.cd386135276b7d3c57416df3666e4aea.2020-12-03 > 16:34:47,133 ERROR [RS_OPEN_REGION-regionserver/blrphispra01054:16040-0] > handler.OpenRegionHandler: Failed open of > region=ns2:testtable4,15,1606912406234.cd386135276b7d3c57416df3666e4aea.java.io.IOException: > The new max sequence id 1 is less than the old max sequence id 7134 at > org.apache.hadoop.hbase.wal.WALSplitUtil.writeRegionSequenceIdFile(WALSplitUtil.java:418) > at > org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1253) > at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1793) > at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1606) at > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1552) at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7522) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7467) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7439) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7397) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7348) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:286) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:111) > at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25371) When openRegion fails during initial verification(before initializing and setting seq num), exception is observed during region close.
[ https://issues.apache.org/jira/browse/HBASE-25371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265254#comment-17265254 ] Andrew Kyle Purtell commented on HBASE-25371: - Set fix version to 2.4.2. > When openRegion fails during initial verification(before initializing and > setting seq num), exception is observed during region close. > -- > > Key: HBASE-25371 > URL: https://issues.apache.org/jira/browse/HBASE-25371 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 2.2.3 >Reporter: Ajeet Rai >Assignee: Mohammad Arshad >Priority: Major > Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.3.5, 2.4.2 > > > When openRegion fails during initial verification(before initializing and > setting seq num), exception is observed during region close: > > 2020-12-03 16:34:47,133 ERROR > [RS_OPEN_REGION-regionserver/AA:16040-0] handler.OpenRegionHandler: > Failed open of > region=ns2:testtable4,15,1606912406234.cd386135276b7d3c57416df3666e4aea.2020-12-03 > 16:34:47,133 ERROR [RS_OPEN_REGION-regionserver/blrphispra01054:16040-0] > handler.OpenRegionHandler: Failed open of > region=ns2:testtable4,15,1606912406234.cd386135276b7d3c57416df3666e4aea.java.io.IOException: > The new max sequence id 1 is less than the old max sequence id 7134 at > org.apache.hadoop.hbase.wal.WALSplitUtil.writeRegionSequenceIdFile(WALSplitUtil.java:418) > at > org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1253) > at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1793) > at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1606) at > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1552) at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7522) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7467) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7439) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7397) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7348) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:286) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:111) > at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2237: HBASE-24833: Bootstrap should not delete the META table directory if …
Apache-HBase commented on pull request #2237: URL: https://github.com/apache/hbase/pull/2237#issuecomment-760501642 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 54s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 49s | branch-2 passed | | +1 :green_heart: | compile | 3m 11s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 10s | branch-2 passed | | +1 :green_heart: | spotbugs | 2m 0s | branch-2 passed | | -0 :warning: | patch | 2m 9s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 25s | the patch passed | | +1 :green_heart: | compile | 3m 27s | the patch passed | | +1 :green_heart: | javac | 3m 27s | the patch passed | | +1 :green_heart: | checkstyle | 1m 14s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 35s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 43m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2237/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2237 | | JIRA Issue | HBASE-24833 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 95de4940b43d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 2bb7beb448 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Max. process+thread count | 96 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2237/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-25249) Adding StoreContext
[ https://issues.apache.org/jira/browse/HBASE-25249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265122#comment-17265122 ] Tak-Lon (Stephen) Wu edited comment on HBASE-25249 at 1/14/21, 6:47 PM: [~apurtell] Andrew, if we want to cherry-pick the branch-2's commit [2bb7beb448a92e39da4a39de9ec1af3b93328a03|https://github.com/apache/hbase/commit/2bb7beb448a92e39da4a39de9ec1af3b93328a03] of this JIRA to branch-2.4, do we need a new PR ? also I found the cherry-pick is clean was (Author: taklwu): [~apurtell] Andrew, if we want to cherry-pick [2bb7beb448a92e39da4a39de9ec1af3b93328a03|https://github.com/apache/hbase/commit/2bb7beb448a92e39da4a39de9ec1af3b93328a03] to branch-2.4, do we need a new PR ? also I found the cherry-pick is clean > Adding StoreContext > --- > > Key: HBASE-25249 > URL: https://issues.apache.org/jira/browse/HBASE-25249 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-1, 2.4.0, 2.3.4 >Reporter: Abhishek Khanna >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > Adding StoreContext which contains the metadata about the HStore. This meta > data can be used across the HFileWriter/Readers and other HStore consumers > without the need of passing around the complete store and exposing its > internals. This is a refactoring change which cleans up the existing code so > that the following commits can leverage the context for a more maintainable > code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on pull request #2754: HBASE-25375 Provide a VM-based release environment
ndimiduk commented on pull request #2754: URL: https://github.com/apache/hbase/pull/2754#issuecomment-760414436 This is merged. Thanks for taking time with it! @Apache9 I included a signed-off-by line with your name on it because you did approve an early version of this. If you get back to this ticket and have some reservations, let me know and I'll revert the commit and continue the discussion. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2882: HBASE-25507 Leak of ESTABLISHED sockets when compaction encountered "…
Apache-HBase commented on pull request #2882: URL: https://github.com/apache/hbase/pull/2882#issuecomment-760414437 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 17s | master passed | | +1 :green_heart: | compile | 1m 10s | master passed | | +1 :green_heart: | shadedjars | 6m 44s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 1m 6s | the patch passed | | +1 :green_heart: | javac | 1m 6s | the patch passed | | +1 :green_heart: | shadedjars | 6m 42s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 135m 18s | hbase-server in the patch passed. | | | | 163m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2882 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 206f8dd1b782 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3488c44a21 | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/1/testReport/ | | Max. process+thread count | 3953 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-24972) Wait for connection attempt to succeed before performing operations on ZK
[
https://issues.apache.org/jira/browse/HBASE-24972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265148#comment-17265148
]
Prathyusha edited comment on HBASE-24972 at 1/14/21, 7:31 PM:
--
[~stack] Below is the stack trace of a failure incident we have seen -
Cause: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/table/SYSTEM.CATALOG
StackTrace:
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1337)
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354)
org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:625)
...
StackTraceId: 429763122
But yes, I see the retries in place where ever we are doing write operations.
[~sandeep.guggilam] These retries should suffice from what I see. Am I missing
something?
was (Author: prathyu6):
[~stack] Below is the stack trace of a failure incident we have seen -
Cause: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/table/SYSTEM.CATALOG
StackTrace:
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1337)
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354)
org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:625)
...
StackTraceId: 429763122
But yes, I see the retries in place where ever we are doing write operations.
[~sandeep.guggilam] These retries should suffice I guess. Any thoughts?
> Wait for connection attempt to succeed before performing operations on ZK
> -
>
> Key: HBASE-24972
> URL: https://issues.apache.org/jira/browse/HBASE-24972
> Project: HBase
> Issue Type: Bug
>Reporter: Sandeep Guggilam
>Assignee: Prathyusha
>Priority: Minor
>
> {color:#1d1c1d}Creating the connection with ZK is asynchronous and notified
> via the passed in watcher about the successful connection event. When we
> attempt any operations, we try to create a connection and then perform a
> read/write
> ({color}{color:#1d1c1d}[https://github.com/apache/hbase/blob/979edfe72046b2075adcc869c65ae820e6f3ec2d/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java#L323]{color}{color:#1d1c1d})
> without really waiting for the notification event
> ([https://github.com/apache/hbase/blob/979edfe72046b2075adcc869c65ae820e6f3ec2d/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKWatcher.java#L582)]{color}
>
> {color:#1d1c1d}It is possible we get ConnectionLoss errors when we perform
> operations on ZK without waiting for the connection attempt to succeed{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2882: HBASE-25507 Leak of ESTABLISHED sockets when compaction encountered "…
Apache-HBase commented on pull request #2882: URL: https://github.com/apache/hbase/pull/2882#issuecomment-760432715 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 42s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 50s | master passed | | +1 :green_heart: | compile | 0m 59s | master passed | | +1 :green_heart: | shadedjars | 8m 47s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 46s | the patch passed | | +1 :green_heart: | compile | 1m 13s | the patch passed | | +1 :green_heart: | javac | 1m 13s | the patch passed | | +1 :green_heart: | shadedjars | 8m 47s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 149m 5s | hbase-server in the patch passed. | | | | 182m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2882 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux cd8b5891ed59 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3488c44a21 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/1/testReport/ | | Max. process+thread count | 4631 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] z-york commented on pull request #2237: HBASE-24833: Bootstrap should not delete the META table directory if …
z-york commented on pull request #2237: URL: https://github.com/apache/hbase/pull/2237#issuecomment-760432544 I think we can agree to disagree on the automatic/idempotent IMP, we can propose a plan for that if I have more time to devote to it in the future. For now we can keep that as a patch. I think that this PR can still go in with throwing an exception instead of deleting meta (I don't see any reason to submit a separate PR, let's keep this discussion). I think we should fail instead of allowing an automatic delete of the meta directory (or at least have an option to fail and not delete) since we are continuing to add more metadata into meta and it will become more and more costly to rebuild. I have seen many cases where operators clear out ZK nodes + restart master to unblock some assignment issues, but admittedly that is on 1.x versions of HBase, I think the recovery options might be better in 2.x. We already have offline meta repair that I believe should be able to solve these issues if an exception is thrown. btw @saintstack for the double assigned meta scenario wouldn't the IO fencing/lease on the meta WAL handle that? Or will it try to write to a unique WAL each assignment? Just curious, not blocking this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25371) When openRegion fails during initial verification(before initializing and setting seq num), exception is observed during region close.
[ https://issues.apache.org/jira/browse/HBASE-25371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun updated HBASE-25371: - Fix Version/s: (was: 2.3.4) 2.3.5 > When openRegion fails during initial verification(before initializing and > setting seq num), exception is observed during region close. > -- > > Key: HBASE-25371 > URL: https://issues.apache.org/jira/browse/HBASE-25371 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 2.2.3 >Reporter: Ajeet Rai >Assignee: Mohammad Arshad >Priority: Major > Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.4.1, 2.3.5 > > > When openRegion fails during initial verification(before initializing and > setting seq num), exception is observed during region close: > > 2020-12-03 16:34:47,133 ERROR > [RS_OPEN_REGION-regionserver/AA:16040-0] handler.OpenRegionHandler: > Failed open of > region=ns2:testtable4,15,1606912406234.cd386135276b7d3c57416df3666e4aea.2020-12-03 > 16:34:47,133 ERROR [RS_OPEN_REGION-regionserver/blrphispra01054:16040-0] > handler.OpenRegionHandler: Failed open of > region=ns2:testtable4,15,1606912406234.cd386135276b7d3c57416df3666e4aea.java.io.IOException: > The new max sequence id 1 is less than the old max sequence id 7134 at > org.apache.hadoop.hbase.wal.WALSplitUtil.writeRegionSequenceIdFile(WALSplitUtil.java:418) > at > org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1253) > at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1793) > at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1606) at > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1552) at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7522) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7467) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7439) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7397) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7348) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:286) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:111) > at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25371) When openRegion fails during initial verification(before initializing and setting seq num), exception is observed during region close.
[ https://issues.apache.org/jira/browse/HBASE-25371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265204#comment-17265204 ] Huaxiang Sun commented on HBASE-25371: -- FYI [~vjasani], this Jira has been marked for 2.4.1 as well, but it is not merged yet. > When openRegion fails during initial verification(before initializing and > setting seq num), exception is observed during region close. > -- > > Key: HBASE-25371 > URL: https://issues.apache.org/jira/browse/HBASE-25371 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 2.2.3 >Reporter: Ajeet Rai >Assignee: Mohammad Arshad >Priority: Major > Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.4.1, 2.3.5 > > > When openRegion fails during initial verification(before initializing and > setting seq num), exception is observed during region close: > > 2020-12-03 16:34:47,133 ERROR > [RS_OPEN_REGION-regionserver/AA:16040-0] handler.OpenRegionHandler: > Failed open of > region=ns2:testtable4,15,1606912406234.cd386135276b7d3c57416df3666e4aea.2020-12-03 > 16:34:47,133 ERROR [RS_OPEN_REGION-regionserver/blrphispra01054:16040-0] > handler.OpenRegionHandler: Failed open of > region=ns2:testtable4,15,1606912406234.cd386135276b7d3c57416df3666e4aea.java.io.IOException: > The new max sequence id 1 is less than the old max sequence id 7134 at > org.apache.hadoop.hbase.wal.WALSplitUtil.writeRegionSequenceIdFile(WALSplitUtil.java:418) > at > org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1253) > at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1793) > at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1606) at > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1552) at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7522) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7467) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7439) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7397) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7348) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:286) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:111) > at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] arshadmohammad opened a new pull request #2883: HBASE-25492: Create table with rsgroup
arshadmohammad opened a new pull request #2883: URL: https://github.com/apache/hbase/pull/2883 Added getRegionServerGroup API in TableDescriptor Handled impacted RSGroup APIs moveTables, moveServersAndTables and renameRSGroup This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2866: HBASE-25478 - Implement retries when enabling tables in TestRegionReplicaReplicationEndpoint
Apache-HBase commented on pull request #2866: URL: https://github.com/apache/hbase/pull/2866#issuecomment-760471043 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 22s | master passed | | +1 :green_heart: | compile | 1m 10s | master passed | | +1 :green_heart: | shadedjars | 6m 56s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 15s | the patch passed | | +1 :green_heart: | compile | 1m 6s | the patch passed | | +1 :green_heart: | javac | 1m 6s | the patch passed | | +1 :green_heart: | shadedjars | 6m 58s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 133m 17s | hbase-server in the patch failed. | | | | 161m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2866/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2866 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f50f68b7630b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3488c44a21 | | Default Java | AdoptOpenJDK-11.0.6+10 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2866/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2866/3/testReport/ | | Max. process+thread count | 3857 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2866/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2883: HBASE-25492: Create table with rsgroup
Apache-HBase commented on pull request #2883: URL: https://github.com/apache/hbase/pull/2883#issuecomment-760486795 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 35s | branch-2 passed | | +1 :green_heart: | compile | 1m 16s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 58s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 10s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 21s | the patch passed | | +1 :green_heart: | compile | 1m 17s | the patch passed | | +1 :green_heart: | javac | 1m 17s | the patch passed | | +1 :green_heart: | shadedjars | 5m 58s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 7s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 23s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 2m 16s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 4m 18s | hbase-rsgroup in the patch passed. | | | | 35m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2883/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2883 | | JIRA Issue | HBASE-25492 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bb2361c49d7e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 2bb7beb448 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2883/1/testReport/ | | Max. process+thread count | 3252 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-client hbase-rsgroup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2883/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25510) Optimize TableName.valueOf from O(n) to O(1). We can get benefits when the number of tables in the cluster is greater than dozens
[
https://issues.apache.org/jira/browse/HBASE-25510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuobin zheng updated HBASE-25510:
--
Description:
Now, TableName.valueOf will try to find TableName Object in cache linearly(code
show as below). So it is too slow when we has thousands of tables on cluster.
{code:java}
// code placeholder
for (TableName tn : tableCache) {
if (Bytes.equals(tn.getQualifier(), qns) && Bytes.equals(tn.getNamespace(),
bns)) {
return tn;
}
}{code}
I try to store the object in the hash table, so it can look up more quickly.
code like this
{code:java}
// code placeholder
TableName oldTable = tableCache.get(nameAsStr);{code}
In our cluster which has tens thousands of tables. (Most of that is KYLIN
table).
We found that in the following two cases, the TableName.valueOf method will
severely restrict our performance.
Common premise: tens of thousands table in cluster
cause: TableName.valueOf with low performance. (because we need to traverse
all caches linearly)
Case1. Replication
premise1: one of table write with high qps, small value, Non-batch request.
cause too much wal entry
premise2: deserialize WAL Entry includes calling the TableName.valueOf method.
Cause: Replicat Stuck. A lot of WAL files pile up.
Case2. Active Master Start up
NamespaceStateManager init should init all RegionInfo, and regioninfo init will
call TableName.valueOf. It will cost some time if TableName.valueOf is slow.
was:
There are tens of thousands of tables on our cluster (Most of that is KYLIN
table).
We found that in the following two cases, the TableName.valueOf method will
severely restrict our performance.
Common premise: tens of thousands table in cluster
cause: TableName.valueOf with low performance. (because we need to traverse all
caches linearly)
Case1. Replication
premise: one of table write with high qps, small value, Non-batch request.
cause: There are too much wal entry in WAL. So we need to deserialize too many
WAL Entry which includes calling the TableName.valueOf method to instantiate
the TableName object.
Case2. Active Master Start up
> Optimize TableName.valueOf from O(n) to O(1). We can get benefits when the
> number of tables in the cluster is greater than dozens
> --
>
> Key: HBASE-25510
> URL: https://issues.apache.org/jira/browse/HBASE-25510
> Project: HBase
> Issue Type: Improvement
> Components: master, Replication
>Affects Versions: 1.2.12, 1.4.13, 2.4.1
>Reporter: zhuobin zheng
>Priority: Major
>
> Now, TableName.valueOf will try to find TableName Object in cache
> linearly(code show as below). So it is too slow when we has thousands of
> tables on cluster.
> {code:java}
> // code placeholder
> for (TableName tn : tableCache) {
> if (Bytes.equals(tn.getQualifier(), qns) && Bytes.equals(tn.getNamespace(),
> bns)) {
> return tn;
> }
> }{code}
> I try to store the object in the hash table, so it can look up more quickly.
> code like this
> {code:java}
> // code placeholder
> TableName oldTable = tableCache.get(nameAsStr);{code}
>
> In our cluster which has tens thousands of tables. (Most of that is KYLIN
> table).
> We found that in the following two cases, the TableName.valueOf method will
> severely restrict our performance.
>
> Common premise: tens of thousands table in cluster
> cause: TableName.valueOf with low performance. (because we need to traverse
> all caches linearly)
>
> Case1. Replication
> premise1: one of table write with high qps, small value, Non-batch request.
> cause too much wal entry
> premise2: deserialize WAL Entry includes calling the TableName.valueOf method.
> Cause: Replicat Stuck. A lot of WAL files pile up.
>
> Case2. Active Master Start up
> NamespaceStateManager init should init all RegionInfo, and regioninfo init
> will call TableName.valueOf. It will cost some time if TableName.valueOf is
> slow.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25511) After hbase-thirdparty 3.4.1 release, set next dev version
[ https://issues.apache.org/jira/browse/HBASE-25511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-25511: Fix Version/s: (was: thirdparty-3.5.0) Affects Version/s: (was: thirdparty-3.5.0) thirdparty-3.4.2 Status: Patch Available (was: Open) > After hbase-thirdparty 3.4.1 release, set next dev version > -- > > Key: HBASE-25511 > URL: https://issues.apache.org/jira/browse/HBASE-25511 > Project: HBase > Issue Type: Sub-task > Components: hbase-thirdparty >Affects Versions: thirdparty-3.4.2 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
