[GitHub] [hbase] virajjasani commented on issue #754: HBASE-22978 : Online slow response log
virajjasani commented on issue #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#issuecomment-586127846 > No, I do not thing it's good to maintain uniform print format across all client. Anything that is providing display functionality for an object will need control over how the content is formatted. > > I maintain that pretty-printed json does not belong in the object's `toString` method. The abundantly common use-case for `toString` is the logging infrastructure, which has different requirements from the shell's user interface. Please just use the Apache Commons Lang `ToStringBuilder` (I prefer it's `SHORT_PREFIX_STYLE`) for that purpose. > > If you would like the object to control it's own formatting as a convenience to a variety of clients (something I find dubious but why not), I have no problem with putting this concern behind a separate method in `SlowLogRecord`, such as `toFormattedJson`. My main objection to this approach is that this introduces a dependency into `SlowLogRecord`, what is otherwise a simple POJO, onto an unrelated provider of json representation. > > Also, you may not have noticed that the shell has it's own class for handling display, called `Formatter`, under `hbase-shell/src/main/ruby/shell/formatter.rb`. I haven't used it myself, but it seems purpose-built to handle the use-case you have. Thanks @ndimiduk I agree with your comment and updated the PR with recent commit. I have addressed all your concerns so far. Please review. Thank you This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22514) Move rsgroup feature into core of HBase
[
https://issues.apache.org/jira/browse/HBASE-22514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036704#comment-17036704
]
Hudson commented on HBASE-22514:
Results for branch HBASE-22514
[build #272 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/272/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/272//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/272//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/272//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Move rsgroup feature into core of HBase
> ---
>
> Key: HBASE-22514
> URL: https://issues.apache.org/jira/browse/HBASE-22514
> Project: HBase
> Issue Type: Umbrella
> Components: Admin, Client, rsgroup
>Reporter: Yechao Chen
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-22514.master.001.patch,
> image-2019-05-31-18-25-38-217.png
>
>
> The class RSGroupAdminClient is not public
> we need to use java api RSGroupAdminClient to manager RSG
> so RSGroupAdminClient should be public
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1170: Backport HBASE-23808 to branch-2.2
Apache-HBase commented on issue #1170: Backport HBASE-23808 to branch-2.2 URL: https://github.com/apache/hbase/pull/1170#issuecomment-586099123 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-2.2 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 48s | branch-2.2 passed | | +1 :green_heart: | compile | 0m 58s | branch-2.2 passed | | +1 :green_heart: | checkstyle | 1m 30s | branch-2.2 passed | | +1 :green_heart: | shadedjars | 4m 37s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | branch-2.2 passed | | +0 :ok: | spotbugs | 3m 58s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 55s | branch-2.2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 19s | the patch passed | | +1 :green_heart: | compile | 1m 0s | the patch passed | | +1 :green_heart: | javac | 1m 0s | the patch passed | | +1 :green_heart: | checkstyle | 1m 32s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 40s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 16m 40s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | | +1 :green_heart: | findbugs | 3m 34s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 330m 12s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. | | | | 391m 26s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.TestMobSnapshotCloneIndependence | | | hadoop.hbase.client.TestFromClientSide3 | | | hadoop.hbase.replication.TestReplicationKillSlaveRS | | | hadoop.hbase.client.TestFromClientSide | | | hadoop.hbase.client.TestCloneSnapshotFromClientNormal | | | hadoop.hbase.client.TestSnapshotTemporaryDirectoryWithRegionReplicas | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1170/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1170 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux bd5c87240cb3 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1170/out/precommit/personality/provided.sh | | git revision | branch-2.2 / 28f51d0802 | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1170/1/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1170/1/testReport/ | | Max. process+thread count | 4852 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1170/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1169: Backport HBASE-23808 to branch-2
Apache-HBase commented on issue #1169: Backport HBASE-23808 to branch-2 URL: https://github.com/apache/hbase/pull/1169#issuecomment-586087933 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 5s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 8m 53s | branch-2 passed | | +1 :green_heart: | compile | 1m 26s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 33s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 2s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 53s | branch-2 passed | | +0 :ok: | spotbugs | 4m 17s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 15s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 7m 42s | the patch passed | | +1 :green_heart: | compile | 1m 16s | the patch passed | | +1 :green_heart: | javac | 1m 16s | the patch passed | | +1 :green_heart: | checkstyle | 1m 32s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 5m 58s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 23m 54s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 49s | the patch passed | | +1 :green_heart: | findbugs | 4m 21s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 123m 28s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 52s | The patch does not generate ASF License warnings. | | | | 206m 24s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.replication.TestReplicationEndpoint | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1169/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1169 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 2dbf8afe815b 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1169/out/precommit/personality/provided.sh | | git revision | branch-2 / 5d77ff6bb6 | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1169/2/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1169/2/testReport/ | | Max. process+thread count | 7121 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1169/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23838) Adding debug logging to a few ExportSnapshot tests
[
https://issues.apache.org/jira/browse/HBASE-23838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036642#comment-17036642
]
Hudson commented on HBASE-23838:
Results for branch branch-2
[build #2472 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2472/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2472//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2472//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2472//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Adding debug logging to a few ExportSnapshot tests
> --
>
> Key: HBASE-23838
> URL: https://issues.apache.org/jira/browse/HBASE-23838
> Project: HBase
> Issue Type: Bug
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Trivial
> Fix For: 3.0.0, 2.3.0
>
> Attachments:
> 0001-HBASE-23838-Adding-debug-logging-to-a-few-ExportSnap.patch
>
>
> Bit of extra logging in tests to help me diagnose where they are going wrong.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on issue #1171: HBASE-23767 WIP do not merge
ndimiduk commented on issue #1171: HBASE-23767 WIP do not merge URL: https://github.com/apache/hbase/pull/1171#issuecomment-586047745 @busbey this mean anything to you? ``` /usr/bin/docker: Error response from daemon: unable to find user jenkins: no matching entries in passwd file. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1171: HBASE-23767 WIP do not merge
Apache-HBase commented on issue #1171: HBASE-23767 WIP do not merge URL: https://github.com/apache/hbase/pull/1171#issuecomment-586045992 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1171/3/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1171: HBASE-23767 WIP do not merge
Apache-HBase commented on issue #1171: HBASE-23767 WIP do not merge URL: https://github.com/apache/hbase/pull/1171#issuecomment-586044833 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1171/2/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on issue #1171: HBASE-23767 WIP do not merge
ndimiduk commented on issue #1171: HBASE-23767 WIP do not merge URL: https://github.com/apache/hbase/pull/1171#issuecomment-586044165 this seems bad. ``` [2020-02-14T00:47:22.988Z] WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. [2020-02-14T00:47:22.988Z] /usr/bin/docker: Error response from daemon: unable to find user jenkins: no matching entries in passwd file. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1171: HBASE-23767 WIP do not merge
Apache-HBase commented on issue #1171: HBASE-23767 WIP do not merge URL: https://github.com/apache/hbase/pull/1171#issuecomment-586042081 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1171/1/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk opened a new pull request #1171: HBASE-23767 WIP do not merge
ndimiduk opened a new pull request #1171: HBASE-23767 WIP do not merge URL: https://github.com/apache/hbase/pull/1171 Rebuild our Dockerfile with support for multiple JDK versions. Try to use it with Github/precommit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1169: Backport HBASE-23808 to branch-2
Apache-HBase commented on issue #1169: Backport HBASE-23808 to branch-2 URL: https://github.com/apache/hbase/pull/1169#issuecomment-586035626 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 27s | branch-2 passed | | +1 :green_heart: | compile | 0m 57s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 12s | branch-2 passed | | +1 :green_heart: | shadedjars | 4m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | branch-2 passed | | +0 :ok: | spotbugs | 3m 31s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 29s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 54s | the patch passed | | +1 :green_heart: | compile | 0m 55s | the patch passed | | +1 :green_heart: | javac | 0m 55s | the patch passed | | +1 :green_heart: | checkstyle | 1m 8s | the patch passed | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 18s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 15m 30s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | | +1 :green_heart: | findbugs | 3m 31s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 60m 21s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 42s | The patch does not generate ASF License warnings. | | | | 115m 29s | | | Reason | Tests | |---:|:--| | Unreaped Processes | hbase-server:12 | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1169/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1169 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 771cf8d7aaeb 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1169/out/precommit/personality/provided.sh | | git revision | branch-2 / 5d77ff6bb6 | | Default Java | 1.8.0_181 | | Unreaped Processes Log | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1169/1/artifact/out/patch-unit-hbase-server-reaper.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1169/1/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1169/1/testReport/ | | Max. process+thread count | 9594 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1169/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HBASE-23839) TestEntityLocks often fails in lower resource envs in testEntityLockTimeout
Mark Robert Miller created HBASE-23839: -- Summary: TestEntityLocks often fails in lower resource envs in testEntityLockTimeout Key: HBASE-23839 URL: https://issues.apache.org/jira/browse/HBASE-23839 Project: HBase Issue Type: Test Reporter: Mark Robert Miller The test waits for something to happen and if the computer is a little slow, it will fail on line 178. Doing a check against 3 instead 2x seems to help a lot to start. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1138: HBASE-22827 Expose multi-region merge in shell and Admin API
Apache-HBase commented on issue #1138: HBASE-22827 Expose multi-region merge in shell and Admin API URL: https://github.com/apache/hbase/pull/1138#issuecomment-586012230 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 36s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 17s | master passed | | +1 :green_heart: | compile | 0m 52s | master passed | | +1 :green_heart: | checkstyle | 0m 47s | master passed | | +1 :green_heart: | shadedjars | 4m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | | +0 :ok: | spotbugs | 1m 11s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 10s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 2s | the patch passed | | +1 :green_heart: | compile | 0m 51s | the patch passed | | +1 :green_heart: | javac | 0m 51s | the patch passed | | +1 :green_heart: | checkstyle | 0m 41s | the patch passed | | +1 :green_heart: | rubocop | 0m 10s | There were no new rubocop issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 5m 16s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 19m 31s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | | +1 :green_heart: | findbugs | 1m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 59s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 6m 56s | hbase-shell in the patch passed. | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 64m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1138/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1138 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile rubocop | | uname | Linux dffde7a7ab71 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1138/out/precommit/personality/provided.sh | | git revision | master / 488c2b2483 | | Default Java | 1.8.0_181 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1138/5/testReport/ | | Max. process+thread count | 4439 (vs. ulimit of 1) | | modules | C: hbase-client hbase-shell U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1138/5/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 rubocop=0.79.0 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] bharathv commented on a change in pull request #1167: HBASE-18095 Zookeeper-less client connection umbrella
bharathv commented on a change in pull request #1167: HBASE-18095 Zookeeper-less client connection umbrella URL: https://github.com/apache/hbase/pull/1167#discussion_r379155909 ## File path: src/main/asciidoc/_chapters/configuration.adoc ## @@ -563,38 +563,63 @@ Changes here will require a cluster restart for HBase to notice the change thoug If you are running HBase in standalone mode, you don't need to configure anything for your client to work provided that they are all on the same machine. -Since the HBase Master may move around, clients bootstrap by looking to ZooKeeper for current critical locations. -ZooKeeper is where all these values are kept. -Thus clients require the location of the ZooKeeper ensemble before they can do anything else. -Usually this ensemble location is kept out in the _hbase-site.xml_ and is picked up by the client from the `CLASSPATH`. +Starting release 3.0.0, the default connection registry has been switched to a master based implementation. Refer to <> for more details about +what a connection registry is and implications of this change. Depending on your HBase version, following is the expected minimal client configuration. -If you are configuring an IDE to run an HBase client, you should include the _conf/_ directory on your classpath so _hbase-site.xml_ settings can be found (or add _src/test/resources_ to pick up the hbase-site.xml used by tests). + Up until 2.x.y releases +In 2.x.y releases, the default connection registry was based on ZooKeeper as the source of truth. This means that the clients always looked up ZooKeeper znodes to fetch +the required metadata. For example, if an active master crashed and the a new master is elected, clients looked up the master znode to fetch +the active master address (similarly for meta locations). This meant that the clients needed to have access to ZooKeeper and need to know +the ZooKeeper ensemble information before they can do anything. This can be configured in the client configuration xml as follows: -For Java applications using Maven, including the hbase-shaded-client module is the recommended dependency when connecting to a cluster: [source,xml] - - org.apache.hbase - hbase-shaded-client - 2.0.0 - + + + + +hbase.zookeeper.quorum +example1,example2,example3 + Zookeeper ensemble information + + -A basic example _hbase-site.xml_ for client only may look as follows: + Starting 3.0.0 release + +The default implementation was switched to a master based connection registry. With this implementation, clients always contact the active or +stand-by master RPC end points to fetch the the connection registry information. This means that the clients should have access to the list of active and master +end points before they can do anything. This can be configured in the client configuration xml as follows: Review comment: > This specifically doesn't need to be all of the Masters configured for the cluster, right? Right. Added some notes on trouble shooting. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] bharathv commented on a change in pull request #1167: HBASE-18095 Zookeeper-less client connection umbrella
bharathv commented on a change in pull request #1167: HBASE-18095 Zookeeper-less client connection umbrella URL: https://github.com/apache/hbase/pull/1167#discussion_r379154192 ## File path: dev-support/design-docs/HBASE-18095-Zookeeper-less-client-connection-design.adoc ## @@ -0,0 +1,112 @@ + +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + += HBASE-18095: Zookeeper-less client connection + + +== Context +Currently, Zookeeper (ZK) lies in the critical code path of connection init. To set up a connection to a given HBase cluster, client relies on the zookeeper quorum configured in the client hbase-site.xml and attempts to fetch the following information. + +* ClusterID +* Active HMaster server name +* Meta table region locations + +ZK is deemed the source of truth since other processes that maintain the cluster state persist the changes to this data into ZK. So it is an obvious place to look at for clients to fetch the latest cluster state. However this comes with it’s own set of problems, some of them are below. + +* Timeouts and retry logic for ZK clients are managed separately from HBase configuration. This is more administration overhead for end users (example: multiple timeouts are to be configured for different types of RPCs. client->master, client->ZK etc.). This prevents HBase from having a single holistic timeout configuration that applies to any RPCs. +* If there is any issue with ZK (like connection overload / timeouts), the entire HBase service appears frozen and there is little visibility into it. +* Exposing zookeeper to all the clients can be risky since it can potentially be abused to DDOS. +* Embedded ZK client is bundled with hbase client jar as a dependency (with it’s log spew :-]). Review comment: Good point, added. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] bharathv commented on a change in pull request #1167: HBASE-18095 Zookeeper-less client connection umbrella
bharathv commented on a change in pull request #1167: HBASE-18095 Zookeeper-less client connection umbrella URL: https://github.com/apache/hbase/pull/1167#discussion_r379155014 ## File path: src/main/asciidoc/_chapters/architecture.adoc ## @@ -577,11 +644,24 @@ If the active Master loses its lease in ZooKeeper (or the Master shuts down), th [[master.runtime]] === Runtime Impact -A common dist-list question involves what happens to an HBase cluster when the Master goes down. +A common dist-list question involves what happens to an HBase cluster when the Master goes down. This information has changed staring 3.0.0. + + Up until releases 2.x.y Because the HBase client talks directly to the RegionServers, the cluster can still function in a "steady state". Additionally, per <>, `hbase:meta` exists as an HBase table and is not resident in the Master. However, the Master controls critical functions such as RegionServer failover and completing region splits. So while the cluster can still run for a short time without the Master, the Master should be restarted as soon as possible. + Staring release 3.0.0 +As mentioned in section <>, the default connection registry for clients is now based on master rpc end points. Hence the requirements for +masters' uptime are even tighter starting this release. + +- At least one active or stand by master is needed for a connection set up, unlike before when all the clients needed was a ZooKeeper ensemble. +- Master is now in critical path for read/write operations. For example, if the meta region bounces off to a different region server, clients +need master to fetch the new locations. Earlier this was done by fetching this information directly from ZooKeeper. +- Masters will now have higher connection load than before. So, the server side configuration might need adjustment depending on the load. + +Overall, the master uptime requirements starting with this version are even higher for the client operations to go through. Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] jatsakthi merged pull request #1138: HBASE-22827 Expose multi-region merge in shell and Admin API
jatsakthi merged pull request #1138: HBASE-22827 Expose multi-region merge in shell and Admin API URL: https://github.com/apache/hbase/pull/1138 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] jatsakthi commented on issue #1138: HBASE-22827 Expose multi-region merge in shell and Admin API
jatsakthi commented on issue #1138: HBASE-22827 Expose multi-region merge in shell and Admin API URL: https://github.com/apache/hbase/pull/1138#issuecomment-586005293 Let me push this in. Thanks for the reviews @virajjasani , @esteban , @joshelser This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk opened a new pull request #1170: Backport HBASE-23808 to branch-2.2
ndimiduk opened a new pull request #1170: Backport HBASE-23808 to branch-2.2 URL: https://github.com/apache/hbase/pull/1170 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk opened a new pull request #1169: Backport HBASE-23808 to branch-2
ndimiduk opened a new pull request #1169: Backport HBASE-23808 to branch-2 URL: https://github.com/apache/hbase/pull/1169 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (HBASE-23808) [Flakey Test] TestMasterShutdown#testMasterShutdownBeforeStartingAnyRegionServer
[ https://issues.apache.org/jira/browse/HBASE-23808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk reassigned HBASE-23808: Assignee: Nick Dimiduk > [Flakey Test] > TestMasterShutdown#testMasterShutdownBeforeStartingAnyRegionServer > > > Key: HBASE-23808 > URL: https://issues.apache.org/jira/browse/HBASE-23808 > Project: HBase > Issue Type: Test > Components: test >Affects Versions: 2.3.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > > Reproduces locally from time to time. Not much to go on here. Looks like the > test is trying to do some fancy HBase cluster initialization order on top of > a mini-cluster. Failure seems related to trying to start the HBase master > before HDFS is fully initialized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-23808) [Flakey Test] TestMasterShutdown#testMasterShutdownBeforeStartingAnyRegionServer
[ https://issues.apache.org/jira/browse/HBASE-23808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-23808 started by Nick Dimiduk. > [Flakey Test] > TestMasterShutdown#testMasterShutdownBeforeStartingAnyRegionServer > > > Key: HBASE-23808 > URL: https://issues.apache.org/jira/browse/HBASE-23808 > Project: HBase > Issue Type: Test > Components: test >Affects Versions: 2.3.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > > Reproduces locally from time to time. Not much to go on here. Looks like the > test is trying to do some fancy HBase cluster initialization order on top of > a mini-cluster. Failure seems related to trying to start the HBase master > before HDFS is fully initialized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23808) [Flakey Test] TestMasterShutdown#testMasterShutdownBeforeStartingAnyRegionServer
[ https://issues.apache.org/jira/browse/HBASE-23808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-23808: - Fix Version/s: 3.0.0 > [Flakey Test] > TestMasterShutdown#testMasterShutdownBeforeStartingAnyRegionServer > > > Key: HBASE-23808 > URL: https://issues.apache.org/jira/browse/HBASE-23808 > Project: HBase > Issue Type: Test > Components: test >Affects Versions: 2.3.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > Fix For: 3.0.0 > > > Reproduces locally from time to time. Not much to go on here. Looks like the > test is trying to do some fancy HBase cluster initialization order on top of > a mini-cluster. Failure seems related to trying to start the HBase master > before HDFS is fully initialized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] joshelser commented on issue #1138: HBASE-22827 Expose multi-region merge in shell and Admin API
joshelser commented on issue #1138: HBASE-22827 Expose multi-region merge in shell and Admin API URL: https://github.com/apache/hbase/pull/1138#issuecomment-585993547 Looks great. Ship it, sir. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk merged pull request #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
ndimiduk merged pull request #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore… URL: https://github.com/apache/hbase/pull/1141 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23838) Adding debug logging to a few ExportSnapshot tests
[ https://issues.apache.org/jira/browse/HBASE-23838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23838: -- Attachment: 0001-HBASE-23838-Adding-debug-logging-to-a-few-ExportSnap.patch > Adding debug logging to a few ExportSnapshot tests > -- > > Key: HBASE-23838 > URL: https://issues.apache.org/jira/browse/HBASE-23838 > Project: HBase > Issue Type: Bug >Reporter: Michael Stack >Priority: Trivial > Attachments: > 0001-HBASE-23838-Adding-debug-logging-to-a-few-ExportSnap.patch > > > Bit of extra logging in tests to help me diagnose where they are going wrong. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23838) Adding debug logging to a few ExportSnapshot tests
[ https://issues.apache.org/jira/browse/HBASE-23838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-23838. --- Fix Version/s: 2.3.0 3.0.0 Assignee: Michael Stack Resolution: Fixed > Adding debug logging to a few ExportSnapshot tests > -- > > Key: HBASE-23838 > URL: https://issues.apache.org/jira/browse/HBASE-23838 > Project: HBase > Issue Type: Bug >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Trivial > Fix For: 3.0.0, 2.3.0 > > Attachments: > 0001-HBASE-23838-Adding-debug-logging-to-a-few-ExportSnap.patch > > > Bit of extra logging in tests to help me diagnose where they are going wrong. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23838) Adding debug logging to a few ExportSnapshot tests
[ https://issues.apache.org/jira/browse/HBASE-23838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036534#comment-17036534 ] Michael Stack commented on HBASE-23838: --- What I pushed on branch-2 and master just now. > Adding debug logging to a few ExportSnapshot tests > -- > > Key: HBASE-23838 > URL: https://issues.apache.org/jira/browse/HBASE-23838 > Project: HBase > Issue Type: Bug >Reporter: Michael Stack >Priority: Trivial > Attachments: > 0001-HBASE-23838-Adding-debug-logging-to-a-few-ExportSnap.patch > > > Bit of extra logging in tests to help me diagnose where they are going wrong. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on issue #754: HBASE-22978 : Online slow response log
ndimiduk commented on issue #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#issuecomment-585986284 > I have tried to cover these points in my recent commit except for `need separation of concerns between SlowLog.toString and formatting of output in the shell`. The reason being shell is not able to use Gson as efficiently as keeping `Gson.toJson()` in toString of SlowLogRecord because we have some serialization filters (neither does shell have any good pretty print library that works with our POJO - AFAIK). shell or any client would receive `List` as output of `Admin.getSlowLogResponses()`. In a way, keeping `Gson.toJson()` within `toString()` would ensure all clients can print output in pretty print format. Do you think this is good to maintain uniform print format across all clients? No, I do not thing it's good to maintain uniform print format across all client. Anything that is providing display functionality for an object will need control over how the content is formatted. I maintain that pretty-printed json does not belong in the object's `toString` method. The abundantly common use-case for `toString` is the logging infrastructure, which has different requirements from the shell's user interface. Please just use the Apache Commons Lang `ToStringBuilder` (I prefer it's `SHORT_PREFIX_STYLE`) for that purpose. If you would like the object to control it's own formatting as a convenience to a variety of clients (something I find dubious but why not), I have no problem with putting this concern behind a separate method in `SlowLogRecord`, such as `toFormattedJson`. My main objection to this approach is that this introduces a dependency into `SlowLogRecord`, what is otherwise a simple POJO, onto an unrelated provider of json representation. Also, you may not have noticed that the shell has it's own class for handling display, called `Formatter`, under `hbase-shell/src/main/ruby/shell/formatter.rb`. I haven't used it myself, but it seems purpose-built to handle the use-case you have. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23562) [operator tools] Add a RegionsMerge tool that allows for merging multiple adjacent regions until a desired number of regions is reached.
[ https://issues.apache.org/jira/browse/HBASE-23562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036525#comment-17036525 ] Sakthi commented on HBASE-23562: FYI: HBASE-23826 [~wchevreuil] . Just wanted to keep you in loop as that jira is related to merging multiple regions as well. > [operator tools] Add a RegionsMerge tool that allows for merging multiple > adjacent regions until a desired number of regions is reached. > > > Key: HBASE-23562 > URL: https://issues.apache.org/jira/browse/HBASE-23562 > Project: HBase > Issue Type: New Feature >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > > There's been a few occasions where different customers had faced the need to > reduce the number of regions for specific tables, either due to mistakenly > pre-split or after a purge in table data. This jira is for adding a simple > merge tool that takes the table name and the desired number of regions, then > perform region merges until the total number of regions for the given table > reached the passed valued. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1168: HBASE-23837 Removed deprecated isLegalFamilyName(byte[]) from HColumnDescriptor
Apache-HBase commented on issue #1168: HBASE-23837 Removed deprecated isLegalFamilyName(byte[]) from HColumnDescriptor URL: https://github.com/apache/hbase/pull/1168#issuecomment-585985308 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 36s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 16s | master passed | | +1 :green_heart: | compile | 1m 25s | master passed | | +1 :green_heart: | checkstyle | 1m 39s | master passed | | +1 :green_heart: | shadedjars | 4m 42s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 3s | master passed | | +0 :ok: | spotbugs | 4m 21s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 5m 25s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 41s | the patch passed | | +1 :green_heart: | compile | 1m 27s | the patch passed | | +1 :green_heart: | javac | 1m 27s | the patch passed | | +1 :green_heart: | checkstyle | 0m 30s | hbase-client: The patch generated 0 new + 37 unchanged - 2 fixed = 37 total (was 39) | | +1 :green_heart: | checkstyle | 1m 5s | The patch passed checkstyle in hbase-server | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 48s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 16m 25s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 58s | the patch passed | | +1 :green_heart: | findbugs | 5m 44s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 53s | hbase-client in the patch passed. | | -1 :x: | unit | 64m 32s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 56s | The patch does not generate ASF License warnings. | | | | 130m 55s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.master.TestMasterShutdown | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1168/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1168 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 6cbb28789e24 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1168/out/precommit/personality/provided.sh | | git revision | master / 488c2b2483 | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1168/1/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1168/1/testReport/ | | Max. process+thread count | 7536 (vs. ulimit of 1) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1168/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23838) Adding debug logging to a few ExportSnapshot tests
[ https://issues.apache.org/jira/browse/HBASE-23838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23838: -- Summary: Adding debug logging to a few ExportSnapshot tests (was: Adding debug logging to a few tests) > Adding debug logging to a few ExportSnapshot tests > -- > > Key: HBASE-23838 > URL: https://issues.apache.org/jira/browse/HBASE-23838 > Project: HBase > Issue Type: Bug >Reporter: Michael Stack >Priority: Trivial > > Bit of extra logging in tests to help me diagnose where they are going wrong. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23838) Adding debug logging to a few tests
Michael Stack created HBASE-23838: - Summary: Adding debug logging to a few tests Key: HBASE-23838 URL: https://issues.apache.org/jira/browse/HBASE-23838 Project: HBase Issue Type: Bug Reporter: Michael Stack Bit of extra logging in tests to help me diagnose where they are going wrong. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22749) Distributed MOB compactions
[
https://issues.apache.org/jira/browse/HBASE-22749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036520#comment-17036520
]
HBase QA commented on HBASE-22749:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
2s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 23 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
25s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 6m
0s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
57s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 5m
2s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
15s{color} | {color:blue} branch/hbase-checkstyle no findbugs output file
(findbugsXml.xml) {color} |
| {color:orange}-0{color} | {color:orange} patch {color} | {color:orange} 5m
35s{color} | {color:orange} Used diff version of patch file. Binary files and
potentially other changes not applied. Please rebase and squash commits if
necessary. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
50s{color} | {color:green} root: The patch generated 0 new + 427 unchanged - 58
fixed = 427 total (was 485) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 6m
11s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
44s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
20m 22s{color} | {color:green} Patch does not cause any errors with Hadoop
2.8.5 2.9.2 or 3.1.2. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
16s{color} | {color:blue} hbase-checkstyle has no data from findbugs {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit
[GitHub] [hbase] Apache-HBase commented on issue #921: HBASE-22749: Distributed MOB compactions
Apache-HBase commented on issue #921: HBASE-22749: Distributed MOB compactions URL: https://github.com/apache/hbase/pull/921#issuecomment-585982399 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 2s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 23 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 37s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 19s | master passed | | +1 :green_heart: | compile | 2m 7s | master passed | | +1 :green_heart: | checkstyle | 2m 25s | master passed | | +0 :ok: | refguide | 6m 0s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 5m 57s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 34s | master passed | | +0 :ok: | spotbugs | 5m 2s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +0 :ok: | findbugs | 0m 15s | branch/hbase-checkstyle no findbugs output file (findbugsXml.xml) | | -0 :warning: | patch | 5m 35s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 6m 32s | the patch passed | | +1 :green_heart: | compile | 2m 27s | the patch passed | | +1 :green_heart: | javac | 2m 28s | the patch passed | | +1 :green_heart: | checkstyle | 2m 50s | root: The patch generated 0 new + 427 unchanged - 58 fixed = 427 total (was 485) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 3s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 6m 11s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 5m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 20m 22s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 1m 32s | the patch passed | | +0 :ok: | findbugs | 0m 16s | hbase-checkstyle has no data from findbugs | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 15s | hbase-checkstyle in the patch passed. | | +1 :green_heart: | unit | 1m 9s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 65m 30s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 1m 15s | hbase-it in the patch passed. | | +1 :green_heart: | asflicense | 2m 12s | The patch does not generate ASF License warnings. | | | | 163m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/9/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/921 | | JIRA Issue | HBASE-22749 | | Optional Tests | dupname asflicense checkstyle javac javadoc unit xml refguide spotbugs findbugs shadedjars hadoopcheck hbaseanti compile | | uname | Linux 594de1a42598 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-921/out/precommit/personality/provided.sh | | git revision | master / 488c2b2483 | | Default Java | 1.8.0_181 | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/9/artifact/out/branch-site/book.html | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/9/artifact/out/patch-site/book.html | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/9/testReport/ | | Max. process+thread count | 8198 (vs. ulimit of 1) | | modules | C: hbase-checkstyle hbase-common hbase-server hbase-it U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/9/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
[GitHub] [hbase] Apache-HBase commented on issue #1146: HBASE-23781 Removed deprecated createTableDescriptor(String) from HBaseTestingUtility
Apache-HBase commented on issue #1146: HBASE-23781 Removed deprecated createTableDescriptor(String) from HBaseTestingUtility URL: https://github.com/apache/hbase/pull/1146#issuecomment-585979683 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 14 new or modified test files. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 7m 0s | master passed | | +1 :green_heart: | compile | 1m 13s | master passed | | +1 :green_heart: | checkstyle | 1m 30s | master passed | | +1 :green_heart: | shadedjars | 6m 2s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | master passed | | +0 :ok: | spotbugs | 5m 2s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 59s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 45s | the patch passed | | +1 :green_heart: | compile | 1m 11s | the patch passed | | +1 :green_heart: | javac | 1m 11s | the patch passed | | +1 :green_heart: | checkstyle | 1m 31s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 6m 14s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 19m 57s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 43s | the patch passed | | +1 :green_heart: | findbugs | 4m 59s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 74m 5s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 40s | The patch does not generate ASF License warnings. | | | | 146m 38s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.TestClusterPortAssignment | | | hadoop.hbase.replication.TestSyncReplicationActive | | | hadoop.hbase.replication.TestSyncReplicationMoreLogsInLocalCopyToRemote | | | hadoop.hbase.replication.TestReplicationStatusSourceStartedTargetStoppedNoOps | | | hadoop.hbase.replication.TestRemoveFromSerialReplicationPeer | | | hadoop.hbase.replication.TestReplicationSmallTestsSync | | | hadoop.hbase.TestSplitMerge | | | hadoop.hbase.replication.TestSyncReplicationStandBy | | | hadoop.hbase.replication.TestReplicationKillMasterRS | | | hadoop.hbase.replication.TestSerialSyncReplication | | | hadoop.hbase.replication.TestSyncReplicationStandbyKillRS | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1146/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1146 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 61f6ebc4a7a1 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1146/out/precommit/personality/provided.sh | | git revision | master / 488c2b2483 | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1146/4/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1146/4/testReport/ | | Max. process+thread count | 7909 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1146/4/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] VladRodionov commented on a change in pull request #921: HBASE-22749: Distributed MOB compactions
VladRodionov commented on a change in pull request #921: HBASE-22749:
Distributed MOB compactions
URL: https://github.com/apache/hbase/pull/921#discussion_r379125214
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCleanerChore.java
##
@@ -0,0 +1,310 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.mob;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.TableDescriptors;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.HFileArchiver;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.master.HMaster;
+import org.apache.hadoop.hbase.regionserver.BloomType;
+import org.apache.hadoop.hbase.regionserver.HStoreFile;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.FSUtils;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * The class MobFileCleanerChore for running cleaner regularly to remove the
expired
+ * and obsolete (files which have no active references to) mob files.
+ */
+@SuppressWarnings("deprecation")
+@InterfaceAudience.Private
+public class MobFileCleanerChore extends ScheduledChore {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MobFileCleanerChore.class);
+ private final HMaster master;
+ private ExpiredMobFileCleaner cleaner;
+
+ static {
+Configuration.addDeprecation(MobConstants.DEPRECATED_MOB_CLEANER_PERIOD,
+ MobConstants.MOB_CLEANER_PERIOD);
+ }
+
+ public MobFileCleanerChore(HMaster master) {
+super(master.getServerName() + "-ExpiredMobFileCleanerChore", master,
+master.getConfiguration().getInt(MobConstants.MOB_CLEANER_PERIOD,
+ MobConstants.DEFAULT_MOB_CLEANER_PERIOD),
+master.getConfiguration().getInt(MobConstants.MOB_CLEANER_PERIOD,
+ MobConstants.DEFAULT_MOB_CLEANER_PERIOD),
+TimeUnit.SECONDS);
+this.master = master;
+cleaner = new ExpiredMobFileCleaner();
+cleaner.setConf(master.getConfiguration());
+checkObsoleteConfigurations();
+ }
+
+ private void checkObsoleteConfigurations() {
+Configuration conf = master.getConfiguration();
+
+if (conf.get("hbase.mob.compaction.mergeable.threshold") != null) {
+ LOG.warn("'hbase.mob.compaction.mergeable.threshold' is obsolete and not
used anymore.");
+}
+if (conf.get("hbase.mob.delfile.max.count") != null) {
+ LOG.warn("'hbase.mob.delfile.max.count' is obsolete and not used
anymore.");
+}
+if (conf.get("hbase.mob.compaction.threads.max") != null) {
+ LOG.warn("'hbase.mob.compaction.threads.max' is obsolete and not used
anymore.");
+}
+if (conf.get("hbase.mob.compaction.batch.size") != null) {
+ LOG.warn("'hbase.mob.compaction.batch.size' is obsolete and not used
anymore.");
+}
+ }
+
+ @VisibleForTesting
+ public MobFileCleanerChore() {
+this.master = null;
+ }
+
+ @Override
+ @edu.umd.cs.findbugs.annotations.SuppressWarnings(value =
"REC_CATCH_EXCEPTION",
+ justification = "Intentional")
+
+ protected void chore() {
+TableDescriptors htds = master.getTableDescriptors();
+
+Map map = null;
+try {
+ map = htds.getAll();
+} catch (IOException e) {
+
[GitHub] [hbase] Apache-HBase commented on issue #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
Apache-HBase commented on issue #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore… URL: https://github.com/apache/hbase/pull/1141#issuecomment-585976400 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 52s | master passed | | +1 :green_heart: | compile | 0m 56s | master passed | | +1 :green_heart: | checkstyle | 1m 11s | master passed | | +1 :green_heart: | shadedjars | 4m 45s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | | +0 :ok: | spotbugs | 4m 20s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 18s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 0s | the patch passed | | +1 :green_heart: | compile | 1m 0s | the patch passed | | +1 :green_heart: | javac | 1m 0s | the patch passed | | +1 :green_heart: | checkstyle | 1m 13s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 39s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 15m 48s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | | +1 :green_heart: | findbugs | 4m 26s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 64m 4s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 40s | The patch does not generate ASF License warnings. | | | | 122m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1141/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1141 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 7d8a8db7b4e7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1141/out/precommit/personality/provided.sh | | git revision | master / 488c2b2483 | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1141/6/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1141/6/testReport/ | | Max. process+thread count | 8559 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1141/6/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] VladRodionov commented on a change in pull request #921: HBASE-22749: Distributed MOB compactions
VladRodionov commented on a change in pull request #921: HBASE-22749:
Distributed MOB compactions
URL: https://github.com/apache/hbase/pull/921#discussion_r379122995
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCleanerChore.java
##
@@ -0,0 +1,310 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.mob;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.TableDescriptors;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.HFileArchiver;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.master.HMaster;
+import org.apache.hadoop.hbase.regionserver.BloomType;
+import org.apache.hadoop.hbase.regionserver.HStoreFile;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.FSUtils;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * The class MobFileCleanerChore for running cleaner regularly to remove the
expired
+ * and obsolete (files which have no active references to) mob files.
+ */
+@SuppressWarnings("deprecation")
+@InterfaceAudience.Private
+public class MobFileCleanerChore extends ScheduledChore {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MobFileCleanerChore.class);
+ private final HMaster master;
+ private ExpiredMobFileCleaner cleaner;
+
+ static {
+Configuration.addDeprecation(MobConstants.DEPRECATED_MOB_CLEANER_PERIOD,
+ MobConstants.MOB_CLEANER_PERIOD);
+ }
+
+ public MobFileCleanerChore(HMaster master) {
+super(master.getServerName() + "-ExpiredMobFileCleanerChore", master,
+master.getConfiguration().getInt(MobConstants.MOB_CLEANER_PERIOD,
+ MobConstants.DEFAULT_MOB_CLEANER_PERIOD),
+master.getConfiguration().getInt(MobConstants.MOB_CLEANER_PERIOD,
+ MobConstants.DEFAULT_MOB_CLEANER_PERIOD),
+TimeUnit.SECONDS);
+this.master = master;
+cleaner = new ExpiredMobFileCleaner();
+cleaner.setConf(master.getConfiguration());
+checkObsoleteConfigurations();
+ }
+
+ private void checkObsoleteConfigurations() {
+Configuration conf = master.getConfiguration();
+
+if (conf.get("hbase.mob.compaction.mergeable.threshold") != null) {
+ LOG.warn("'hbase.mob.compaction.mergeable.threshold' is obsolete and not
used anymore.");
+}
+if (conf.get("hbase.mob.delfile.max.count") != null) {
+ LOG.warn("'hbase.mob.delfile.max.count' is obsolete and not used
anymore.");
+}
+if (conf.get("hbase.mob.compaction.threads.max") != null) {
+ LOG.warn("'hbase.mob.compaction.threads.max' is obsolete and not used
anymore.");
+}
+if (conf.get("hbase.mob.compaction.batch.size") != null) {
+ LOG.warn("'hbase.mob.compaction.batch.size' is obsolete and not used
anymore.");
+}
+ }
+
+ @VisibleForTesting
+ public MobFileCleanerChore() {
+this.master = null;
+ }
+
+ @Override
+ @edu.umd.cs.findbugs.annotations.SuppressWarnings(value =
"REC_CATCH_EXCEPTION",
+ justification = "Intentional")
+
+ protected void chore() {
+TableDescriptors htds = master.getTableDescriptors();
+
+Map map = null;
+try {
+ map = htds.getAll();
+} catch (IOException e) {
+
[jira] [Updated] (HBASE-23826) [HBCK2] Expose Multi-Region merge feature via hbck2
[ https://issues.apache.org/jira/browse/HBASE-23826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sakthi updated HBASE-23826: --- Description: HBase Master can do multi-region merge from: 2.0.6,2.1.6,2.2.1,2.3.0,3.0.0 HBASE-22827 aims at exposing the feature via shell from: 2.3.0 & 3.0.0 Let's use this Jira to track, exposing the access to the multi-region merge feature from hbck2 for the versions where shell feature isn't available but the master supports the multi merge feature & hbck service supports mergeRegions api: i.e. 2.1.9+, 2.2.4+ and of course 2.3.0+,3.0.0+ where both shell/hbck2 would be able to do the task for us. was: HBase Master can do multi-region merge from: 2.0.6,2.1.6,2.2.1,2.3.0,3.0.0 HBASE-22827 aims at exposing the feature via shell from: 2.3.0 & 3.0.0 Let's use this Jira to track, exposing the access to the multi-region merge feature from hbck2 for the versions where shell feature isn't available but the master supports the feature: i.e. 2.0.6+,2.1.6+, 2.2.1+ > [HBCK2] Expose Multi-Region merge feature via hbck2 > --- > > Key: HBASE-23826 > URL: https://issues.apache.org/jira/browse/HBASE-23826 > Project: HBase > Issue Type: Task > Components: hbase-operator-tools >Reporter: Sakthi >Assignee: Sakthi >Priority: Major > Fix For: hbase-operator-tools-1.1.0 > > > HBase Master can do multi-region merge from: 2.0.6,2.1.6,2.2.1,2.3.0,3.0.0 > HBASE-22827 aims at exposing the feature via shell from: 2.3.0 & 3.0.0 > Let's use this Jira to track, exposing the access to the multi-region merge > feature from hbck2 for the versions where shell feature isn't available but > the master supports the multi merge feature & hbck service supports > mergeRegions api: i.e. 2.1.9+, 2.2.4+ and of course 2.3.0+,3.0.0+ where both > shell/hbck2 would be able to do the task for us. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-23826) [HBCK2] Expose Multi-Region merge feature via hbck2
[ https://issues.apache.org/jira/browse/HBASE-23826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036503#comment-17036503 ] Sakthi edited comment on HBASE-23826 at 2/13/20 9:11 PM: - Actually I just realized we can't make hbck2 access mergeRegions method from hbck service in any 2.0.x, 2.1.6-2.1.9 (i.e. if this change doesn't make into 2.1.9. else 2.1.8), 2.2.1-2.2.3. As HBASE-23827 would only be present from 3.0.0,2.3.0,2.2.4,2.1.10 was (Author: jatsakthi): Actually I just realized we can't make hbck2 access mergeRegions method from hbck service in any 2.0.x, 2.1.6-2.1.9 (i.e. if this change doesn't make into 2.1.9. else 2.1.8), 2.2.1-2.2.3. As HBASE-23827 would only be present from [3.0.0|https://issues.apache.org/jira/issues/?jql=project+%3D+HBASE+AND+fixVersion+%3D+3.0.0], [2.3.0|https://issues.apache.org/jira/issues/?jql=project+%3D+HBASE+AND+fixVersion+%3D+2.3.0], [2.2.4|https://issues.apache.org/jira/issues/?jql=project+%3D+HBASE+AND+fixVersion+%3D+2.2.4], [2.1.10|https://issues.apache.org/jira/issues/?jql=project+%3D+HBASE+AND+fixVersion+%3D+2.1.10] . > [HBCK2] Expose Multi-Region merge feature via hbck2 > --- > > Key: HBASE-23826 > URL: https://issues.apache.org/jira/browse/HBASE-23826 > Project: HBase > Issue Type: Task > Components: hbase-operator-tools >Reporter: Sakthi >Assignee: Sakthi >Priority: Major > Fix For: hbase-operator-tools-1.1.0 > > > HBase Master can do multi-region merge from: 2.0.6,2.1.6,2.2.1,2.3.0,3.0.0 > HBASE-22827 aims at exposing the feature via shell from: 2.3.0 & 3.0.0 > Let's use this Jira to track, exposing the access to the multi-region merge > feature from hbck2 for the versions where shell feature isn't available but > the master supports the feature: i.e. 2.0.6+,2.1.6+, 2.2.1+ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23826) [HBCK2] Expose Multi-Region merge feature via hbck2
[ https://issues.apache.org/jira/browse/HBASE-23826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036503#comment-17036503 ] Sakthi commented on HBASE-23826: Actually I just realized we can't make hbck2 access mergeRegions method from hbck service in any 2.0.x, 2.1.6-2.1.9 (i.e. if this change doesn't make into 2.1.9. else 2.1.8), 2.2.1-2.2.3. As HBASE-23827 would only be present from [3.0.0|https://issues.apache.org/jira/issues/?jql=project+%3D+HBASE+AND+fixVersion+%3D+3.0.0], [2.3.0|https://issues.apache.org/jira/issues/?jql=project+%3D+HBASE+AND+fixVersion+%3D+2.3.0], [2.2.4|https://issues.apache.org/jira/issues/?jql=project+%3D+HBASE+AND+fixVersion+%3D+2.2.4], [2.1.10|https://issues.apache.org/jira/issues/?jql=project+%3D+HBASE+AND+fixVersion+%3D+2.1.10] . > [HBCK2] Expose Multi-Region merge feature via hbck2 > --- > > Key: HBASE-23826 > URL: https://issues.apache.org/jira/browse/HBASE-23826 > Project: HBase > Issue Type: Task > Components: hbase-operator-tools >Reporter: Sakthi >Assignee: Sakthi >Priority: Major > Fix For: hbase-operator-tools-1.1.0 > > > HBase Master can do multi-region merge from: 2.0.6,2.1.6,2.2.1,2.3.0,3.0.0 > HBASE-22827 aims at exposing the feature via shell from: 2.3.0 & 3.0.0 > Let's use this Jira to track, exposing the access to the multi-region merge > feature from hbck2 for the versions where shell feature isn't available but > the master supports the feature: i.e. 2.0.6+,2.1.6+, 2.2.1+ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23802) Remove unnecessary Configuration instantiation in LossyAccounting
[
https://issues.apache.org/jira/browse/HBASE-23802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036492#comment-17036492
]
Hudson commented on HBASE-23802:
Results for branch branch-1.4
[build #1177 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/1177/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/1177//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/1177//JDK7_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/1177//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Remove unnecessary Configuration instantiation in LossyAccounting
> -
>
> Key: HBASE-23802
> URL: https://issues.apache.org/jira/browse/HBASE-23802
> Project: HBase
> Issue Type: Improvement
> Components: metrics
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 1.6.0, 1.4.13, 2.2.4, 2.1.10
>
>
> Mighty [~stack] pointed out over on HBASE-23801 that maybe {{LossyCounting}}
> doesn't need to be instantiating a {{Configuration}} instance in the first
> place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23602) TTL Before Which No Data is Purged
[ https://issues.apache.org/jira/browse/HBASE-23602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23602: Fix Version/s: (was: 1.6.0) 1.7.0 > TTL Before Which No Data is Purged > -- > > Key: HBASE-23602 > URL: https://issues.apache.org/jira/browse/HBASE-23602 > Project: HBase > Issue Type: New Feature >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 3.0.0, 1.7.0, 2.4.0 > > > HBase currently offers operators a choice. They can set > KEEP_DELETED_CELLS=true and VERSIONS to max value, plus no TTL, and they will > always have a complete history of all changes (but high storage costs and > penalties to read performance). Or they can have KEEP_DELETED_CELLS=false and > VERSIONS/TTL set to some reasonable values, but that means that major > compactions can destroy the ability to do a consistent snapshot read of any > prior time. (This limits the usefulness and correctness of, for example, > Phoenix's SCN lookback feature.) > I propose having a new TTL property to give a minimum age that an expired or > deleted Cell would have to achieve before it could be purged. (I see that > HBASE-10118 already does something similar for the delete markers > themselves.) > This would allow operators to have a consistent history for some finite > amount of recent time while still purging out the "long tail" of obsolete / > deleted versions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23768) Backport to 1.x
[ https://issues.apache.org/jira/browse/HBASE-23768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23768: Fix Version/s: (was: 1.6.0) 1.7.0 > Backport to 1.x > --- > > Key: HBASE-23768 > URL: https://issues.apache.org/jira/browse/HBASE-23768 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 1.7.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21776) Duplicate "Set storagePolicy" debug logging
[ https://issues.apache.org/jira/browse/HBASE-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-21776: Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > Duplicate "Set storagePolicy" debug logging > --- > > Key: HBASE-21776 > URL: https://issues.apache.org/jira/browse/HBASE-21776 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 1.5.0 >Reporter: Andrew Kyle Purtell >Assignee: Sandeep Pal >Priority: Trivial > Labels: branch-1 > Fix For: 1.6.0 > > Attachments: HBASE-21776-branch-1.001.patch > > > An example: > 2019-01-25 02:57:15,831 DEBUG > [regionserver/ip-172-31-13-83.us-west-2.compute.internal/172.31.13.83:8120-longCompactions-1548384634805] > util.CommonFSUtils: Set storagePolicy=HOT for > path=hdfs://ip-172-31-5-95.us-west-2.compute.internal:8020/hbase/data/default/IntegrationTestBigLinkedList/16899cdfdb4ab217208f4108ad0c4803/.tmp/meta > 2019-01-25 02:57:15,831 DEBUG > [regionserver/ip-172-31-13-83.us-west-2.compute.internal/172.31.13.83:8120-longCompactions-1548384634805] > util.CommonFSUtils: Set storagePolicy=HOT for > path=hdfs://ip-172-31-5-95.us-west-2.compute.internal:8020/hbase/data/default/IntegrationTestBigLinkedList/16899cdfdb4ab217208f4108ad0c4803/.tmp/meta > Seen most often during compactions. Ideally we only log once per directory > per flush or compaction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] apurtell merged pull request #856: HBASE-21776: Avoid duplicate calls to setStoragePolicy which shows up in debug Logging
apurtell merged pull request #856: HBASE-21776: Avoid duplicate calls to setStoragePolicy which shows up in debug Logging URL: https://github.com/apache/hbase/pull/856 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-22978:
Fix Version/s: (was: 1.6.0)
1.7.0
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, Operability, regionserver, shell
>Affects Versions: 3.0.0, 2.3.0, 1.5.1
>Reporter: Andrew Kyle Purtell
>Assignee: Viraj Jasani
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 1.7.0
>
> Attachments: Screen Shot 2019-10-19 at 2.31.59 AM.png, Screen Shot
> 2019-10-19 at 2.32.54 AM.png, Screen Shot 2019-10-19 at 2.34.11 AM.png,
> Screen Shot 2019-10-19 at 2.36.14 AM.png
>
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> Optionally persist new entries added to the ring buffer into one or more
> files in HDFS in a write-behind manner. If the HDFS writer blocks or falls
> behind and we are unable to persist an entry before it is overwritten, that
> is fine. Response too slow logging is best effort. If we can detect this make
> a note of it in the log file. Provide a tool for parsing, dumping, filtering,
> and pretty printing the slow logs written to HDFS. The tool and the shell can
> share and reuse some utility classes and methods for accomplishing that.
> —
> New shell commands:
> {{get_slow_responses [ ... , ] [ , \{
> } ]}}
> Retrieve, decode, and pretty print the contents of the too slow response ring
> buffer maintained by the given list of servers; or all servers in the cluster
> if no list is provided. Optionally provide a map of parameters for filtering
> as additional argument. The TABLE filter, which expects a string containing a
> table name, will include only entries pertaining to that table. The REGION
> filter, which expects a string containing an encoded region
[jira] [Commented] (HBASE-23349) Low refCount preventing archival of compacted away files
[
https://issues.apache.org/jira/browse/HBASE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036467#comment-17036467
]
Andrew Kyle Purtell commented on HBASE-23349:
-
We are releasing 1.6.0 now because of HBASE-23825, moving to 1.7.0.
> Low refCount preventing archival of compacted away files
>
>
> Key: HBASE-23349
> URL: https://issues.apache.org/jira/browse/HBASE-23349
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.3.0, 1.6.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.7.0
>
>
> We have observed that refCount on compacted away store files as low as 1 is
> prevent archival.
> {code:java}
> regionserver.HStore - Can't archive compacted file
> hdfs://{{root-dir}}/hbase/data/default/t1/12a9e1112e0371955b3db8d3ebb2d298/cf1/73b72f5ddfce4a34a9e01afe7b83c1f9
> because of either isCompactedAway=true or file has reference,
> isReferencedInReads=true, refCount=1, skipping for now.
> {code}
> We should come up with core code (run as part of discharger thread)
> gracefully resolve reader lock issue by resetting ongoing scanners to start
> pointing to new store files instead of compacted away store files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036470#comment-17036470
]
Andrew Kyle Purtell commented on HBASE-22978:
-
We are releasing 1.6.0 now because of HBASE-23825, moving to 1.7.0.
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, Operability, regionserver, shell
>Affects Versions: 3.0.0, 2.3.0, 1.5.1
>Reporter: Andrew Kyle Purtell
>Assignee: Viraj Jasani
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 1.7.0
>
> Attachments: Screen Shot 2019-10-19 at 2.31.59 AM.png, Screen Shot
> 2019-10-19 at 2.32.54 AM.png, Screen Shot 2019-10-19 at 2.34.11 AM.png,
> Screen Shot 2019-10-19 at 2.36.14 AM.png
>
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> Optionally persist new entries added to the ring buffer into one or more
> files in HDFS in a write-behind manner. If the HDFS writer blocks or falls
> behind and we are unable to persist an entry before it is overwritten, that
> is fine. Response too slow logging is best effort. If we can detect this make
> a note of it in the log file. Provide a tool for parsing, dumping, filtering,
> and pretty printing the slow logs written to HDFS. The tool and the shell can
> share and reuse some utility classes and methods for accomplishing that.
> —
> New shell commands:
> {{get_slow_responses [ ... , ] [ , \{
> } ]}}
> Retrieve, decode, and pretty print the contents of the too slow response ring
> buffer maintained by the given list of servers; or all servers in the cluster
> if no list is provided. Optionally provide a map of parameters for filtering
> as additional argument. The TABLE filter, which expects a string containing a
> table name, will include only entries pertaining to that table. The REGION
> filter,
[jira] [Updated] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22114: Fix Version/s: (was: 1.6.0) 1.7.0 > Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1 > --- > > Key: HBASE-22114 > URL: https://issues.apache.org/jira/browse/HBASE-22114 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 1.7.0 > > Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch, > HBASE-22114-branch-1.patch > > > HBASE-15560 introduces the TinyLFU cache policy for the blockcache. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE > 8+ type Optional, and has dependencies on libraries compiled with Java 8+ > bytecode. It could be backported to branch-1 but must be made optional both > at compile time and runtime, enabled by the 'build-with-jdk8' build profile. > The TinyLFU policy must go into its own build module. > The blockcache must be modified to load L1 implementation/policy dynamically > at startup by reflection if the policy is "TinyLFU" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22356) API to get hdfs block distribution from regionservers
[ https://issues.apache.org/jira/browse/HBASE-22356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22356: Fix Version/s: (was: 1.6.0) 2.4.0 1.7.0 > API to get hdfs block distribution from regionservers > - > > Key: HBASE-22356 > URL: https://issues.apache.org/jira/browse/HBASE-22356 > Project: HBase > Issue Type: Sub-task > Components: API, Balancer, regionserver >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Labels: balancer > Fix For: 3.0.0, 1.7.0, 2.4.0 > > Attachments: HBASE-22356.master.001.patch, > HBASE-22356.master.002.patch, HBASE-22356.master.003.patch > > > A RegionServer API has to be added which will return HDFSBlockDistribution > for all the regions it hosts. RS already has this info cached and updated > when flush/compaction happens. Master can query and get instead of hitting > the namenode and caching. The larger the cluster becomes, the more costly it > becomes to get this information and more stale the cached information becomes. > This jira is only to add the API to regionserver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23349) Low refCount preventing archival of compacted away files
[
https://issues.apache.org/jira/browse/HBASE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-23349:
Fix Version/s: (was: 1.6.0)
1.7.0
> Low refCount preventing archival of compacted away files
>
>
> Key: HBASE-23349
> URL: https://issues.apache.org/jira/browse/HBASE-23349
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.3.0, 1.6.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.7.0
>
>
> We have observed that refCount on compacted away store files as low as 1 is
> prevent archival.
> {code:java}
> regionserver.HStore - Can't archive compacted file
> hdfs://{{root-dir}}/hbase/data/default/t1/12a9e1112e0371955b3db8d3ebb2d298/cf1/73b72f5ddfce4a34a9e01afe7b83c1f9
> because of either isCompactedAway=true or file has reference,
> isReferencedInReads=true, refCount=1, skipping for now.
> {code}
> We should come up with core code (run as part of discharger thread)
> gracefully resolve reader lock issue by resetting ongoing scanners to start
> pointing to new store files instead of compacted away store files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-22126) TestBlocksRead is flaky
[ https://issues.apache.org/jira/browse/HBASE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22126: Fix Version/s: (was: 1.6.0) 1.7.0 > TestBlocksRead is flaky > --- > > Key: HBASE-22126 > URL: https://issues.apache.org/jira/browse/HBASE-22126 > Project: HBase > Issue Type: Test > Components: test >Affects Versions: 1.5.0 >Reporter: Andrew Kyle Purtell >Assignee: Sandeep Pal >Priority: Major > Labels: branch-1 > Fix For: 1.7.0 > > > TestBlocksRead does not fail when invoked by itself but is flaky when run as > part of the suite. > Some kind of race during setup. > [ERROR] > testBlocksStoredWhenCachingDisabled(org.apache.hadoop.hbase.regionserver.TestBlocksRead) > Time elapsed: 0.19 s <<< ERROR! > java.net.ConnectException: Call From $HOST/$IP to localhost:59658 failed on > connection exception: java.net.ConnectException: Connection refused; For more > details see: http://wiki.apache.org/hadoop/ConnectionRefused > at > org.apache.hadoop.hbase.regionserver.TestBlocksRead.initHRegion(TestBlocksRead.java:112) > at > org.apache.hadoop.hbase.regionserver.TestBlocksRead.testBlocksStoredWhenCachingDisabled(TestBlocksRead.java:389) > Caused by: java.net.ConnectException: Connection refused > at > org.apache.hadoop.hbase.regionserver.TestBlocksRead.initHRegion(TestBlocksRead.java:112) > at > org.apache.hadoop.hbase.regionserver.TestBlocksRead.testBlocksStoredWhenCachingDisabled(TestBlocksRead.java:389) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23220) Release 1.6.0
[ https://issues.apache.org/jira/browse/HBASE-23220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23220: Fix Version/s: (was: 1.6.0) > Release 1.6.0 > - > > Key: HBASE-23220 > URL: https://issues.apache.org/jira/browse/HBASE-23220 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 1.5.1 >Reporter: Sean Busbey >Assignee: Andrew Kyle Purtell >Priority: Major > > let's roll 1.6.0 to get HBASE-23174 out on recent branch-1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21140) Backport 'HBASE-21136 NPE in MetricsTableSourceImpl.updateFlushTime' to branch-1 . (and backport HBASE-15728 for branch-1)
[ https://issues.apache.org/jira/browse/HBASE-21140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-21140: Fix Version/s: (was: 1.6.0) 1.7.0 > Backport 'HBASE-21136 NPE in MetricsTableSourceImpl.updateFlushTime' to > branch-1 . (and backport HBASE-15728 for branch-1) > --- > > Key: HBASE-21140 > URL: https://issues.apache.org/jira/browse/HBASE-21140 > Project: HBase > Issue Type: Bug > Components: metrics >Reporter: Duo Zhang >Assignee: Xu Cang >Priority: Major > Fix For: 1.7.0 > > Attachments: > HBASE-21140.diff_against_cf198a65e8d704d28538c4c165a941b9e5bac678.branch-1.001.patch > > > There is no computeIfAbsent method on branch-1 as we still need to support > JDK7, so the fix will be different with branch-2+. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-17890) FuzzyRowFilter fail if unaligned support is false
[
https://issues.apache.org/jira/browse/HBASE-17890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-17890:
Fix Version/s: (was: 1.6.0)
1.7.0
> FuzzyRowFilter fail if unaligned support is false
> -
>
> Key: HBASE-17890
> URL: https://issues.apache.org/jira/browse/HBASE-17890
> Project: HBase
> Issue Type: Sub-task
> Components: util
>Affects Versions: 1.2.5, 2.0.0
>Reporter: Jerry He
>Assignee: Chia-Ping Tsai
>Priority: Major
> Fix For: 3.0.0, 1.7.0
>
> Attachments: HBASE-17890.v0.branch-1.patch, HBASE-17890.v0.patch,
> HBASE-17890.v1.branch-1.patch, HBASE-17890.v1.patch, HBASE-17890.v2.patch,
> HBASE-17890.v3.patch, HBASE-17890.v3.patch, HBASE-17890.v3.patch,
> HBASE-17890.v3.patch, HBASE-17890.v3.patch
>
>
> When unaligned support is false, FuzzyRow tests fail:
> {noformat}
> Failed tests:
> TestFuzzyRowAndColumnRangeFilter.Test:134->runTest:157->runScanner:186
> expected:<10> but was:<0>
> TestFuzzyRowFilter.testSatisfiesForward:81 expected: but was:
> TestFuzzyRowFilter.testSatisfiesReverse:121 expected: but
> was:
> TestFuzzyRowFilterEndToEnd.testEndToEnd:247->runTest1:278->runScanner:343
> expected:<6250> but was:<0>
> TestFuzzyRowFilterEndToEnd.testFilterList:385->runTest:417->runScanner:445
> expected:<5> but was:<0>
> TestFuzzyRowFilterEndToEnd.testHBASE14782:204 expected:<6> but was:<0>
> {noformat}
> This can be reproduced in the case described in HBASE-17869. Or on a platform
> really without unaligned support.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-18095) Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client
[ https://issues.apache.org/jira/browse/HBASE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036465#comment-17036465 ] Andrew Kyle Purtell commented on HBASE-18095: - We are releasing 1.6.0 now because of HBASE-23825. Moving to 1.7.0 > Provide an option for clients to find the server hosting META that does not > involve the ZooKeeper client > > > Key: HBASE-18095 > URL: https://issues.apache.org/jira/browse/HBASE-18095 > Project: HBase > Issue Type: New Feature > Components: Client >Reporter: Andrew Kyle Purtell >Assignee: Bharath Vissapragada >Priority: Major > Fix For: 3.0.0, 2.3.0, 1.7.0 > > Attachments: HBASE-18095.master-v1.patch, HBASE-18095.master-v2.patch > > > Clients are required to connect to ZooKeeper to find the location of the > regionserver hosting the meta table region. Site configuration provides the > client a list of ZK quorum peers and the client uses an embedded ZK client to > query meta location. Timeouts and retry behavior of this embedded ZK client > are managed orthogonally to HBase layer settings and in some cases the ZK > cannot manage what in theory the HBase client can, i.e. fail fast upon outage > or network partition. > We should consider new configuration settings that provide a list of > well-known master and backup master locations, and with this information the > client can contact any of the master processes directly. Any master in either > active or passive state will track meta location and respond to requests for > it with its cached last known location. If this location is stale, the client > can ask again with a flag set that requests the master refresh its location > cache and return the up-to-date location. Every client interaction with the > cluster thus uses only HBase RPC as transport, with appropriate settings > applied to the connection. The configuration toggle that enables this > alternative meta location lookup should be false by default. > This removes the requirement that HBase clients embed the ZK client and > contact the ZK service directly at the beginning of the connection lifecycle. > This has several benefits. ZK service need not be exposed to clients, and > their potential abuse, yet no benefit ZK provides the HBase server cluster is > compromised. Normalizing HBase client and ZK client timeout settings and > retry behavior - in some cases, impossible, i.e. for fail-fast - is no longer > necessary. > And, from [~ghelmling]: There is an additional complication here for > token-based authentication. When a delegation token is used for SASL > authentication, the client uses the cluster ID obtained from Zookeeper to > select the token identifier to use. So there would also need to be some > Zookeeper-less, unauthenticated way to obtain the cluster ID as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-18095) Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client
[ https://issues.apache.org/jira/browse/HBASE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-18095: Fix Version/s: (was: 1.6.0) 1.7.0 > Provide an option for clients to find the server hosting META that does not > involve the ZooKeeper client > > > Key: HBASE-18095 > URL: https://issues.apache.org/jira/browse/HBASE-18095 > Project: HBase > Issue Type: New Feature > Components: Client >Reporter: Andrew Kyle Purtell >Assignee: Bharath Vissapragada >Priority: Major > Fix For: 3.0.0, 2.3.0, 1.7.0 > > Attachments: HBASE-18095.master-v1.patch, HBASE-18095.master-v2.patch > > > Clients are required to connect to ZooKeeper to find the location of the > regionserver hosting the meta table region. Site configuration provides the > client a list of ZK quorum peers and the client uses an embedded ZK client to > query meta location. Timeouts and retry behavior of this embedded ZK client > are managed orthogonally to HBase layer settings and in some cases the ZK > cannot manage what in theory the HBase client can, i.e. fail fast upon outage > or network partition. > We should consider new configuration settings that provide a list of > well-known master and backup master locations, and with this information the > client can contact any of the master processes directly. Any master in either > active or passive state will track meta location and respond to requests for > it with its cached last known location. If this location is stale, the client > can ask again with a flag set that requests the master refresh its location > cache and return the up-to-date location. Every client interaction with the > cluster thus uses only HBase RPC as transport, with appropriate settings > applied to the connection. The configuration toggle that enables this > alternative meta location lookup should be false by default. > This removes the requirement that HBase clients embed the ZK client and > contact the ZK service directly at the beginning of the connection lifecycle. > This has several benefits. ZK service need not be exposed to clients, and > their potential abuse, yet no benefit ZK provides the HBase server cluster is > compromised. Normalizing HBase client and ZK client timeout settings and > retry behavior - in some cases, impossible, i.e. for fail-fast - is no longer > necessary. > And, from [~ghelmling]: There is an additional complication here for > token-based authentication. When a delegation token is used for SASL > authentication, the client uses the cluster ID obtained from Zookeeper to > select the token identifier to use. So there would also need to be some > Zookeeper-less, unauthenticated way to obtain the cluster ID as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-23220) Release 1.6.0
[ https://issues.apache.org/jira/browse/HBASE-23220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-23220 started by Andrew Kyle Purtell. --- > Release 1.6.0 > - > > Key: HBASE-23220 > URL: https://issues.apache.org/jira/browse/HBASE-23220 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 1.5.1 >Reporter: Sean Busbey >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 1.6.0 > > > let's roll 1.6.0 to get HBASE-23174 out on recent branch-1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] VladRodionov commented on a change in pull request #921: HBASE-22749: Distributed MOB compactions
VladRodionov commented on a change in pull request #921: HBASE-22749:
Distributed MOB compactions
URL: https://github.com/apache/hbase/pull/921#discussion_r379072432
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCleanerChore.java
##
@@ -0,0 +1,310 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.mob;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.TableDescriptors;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.HFileArchiver;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.master.HMaster;
+import org.apache.hadoop.hbase.regionserver.BloomType;
+import org.apache.hadoop.hbase.regionserver.HStoreFile;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.FSUtils;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * The class MobFileCleanerChore for running cleaner regularly to remove the
expired
+ * and obsolete (files which have no active references to) mob files.
+ */
+@SuppressWarnings("deprecation")
+@InterfaceAudience.Private
+public class MobFileCleanerChore extends ScheduledChore {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MobFileCleanerChore.class);
+ private final HMaster master;
+ private ExpiredMobFileCleaner cleaner;
+
+ static {
+Configuration.addDeprecation(MobConstants.DEPRECATED_MOB_CLEANER_PERIOD,
+ MobConstants.MOB_CLEANER_PERIOD);
+ }
+
+ public MobFileCleanerChore(HMaster master) {
+super(master.getServerName() + "-ExpiredMobFileCleanerChore", master,
+master.getConfiguration().getInt(MobConstants.MOB_CLEANER_PERIOD,
+ MobConstants.DEFAULT_MOB_CLEANER_PERIOD),
+master.getConfiguration().getInt(MobConstants.MOB_CLEANER_PERIOD,
+ MobConstants.DEFAULT_MOB_CLEANER_PERIOD),
+TimeUnit.SECONDS);
+this.master = master;
+cleaner = new ExpiredMobFileCleaner();
+cleaner.setConf(master.getConfiguration());
+checkObsoleteConfigurations();
+ }
+
+ private void checkObsoleteConfigurations() {
+Configuration conf = master.getConfiguration();
+
+if (conf.get("hbase.mob.compaction.mergeable.threshold") != null) {
+ LOG.warn("'hbase.mob.compaction.mergeable.threshold' is obsolete and not
used anymore.");
+}
+if (conf.get("hbase.mob.delfile.max.count") != null) {
+ LOG.warn("'hbase.mob.delfile.max.count' is obsolete and not used
anymore.");
+}
+if (conf.get("hbase.mob.compaction.threads.max") != null) {
+ LOG.warn("'hbase.mob.compaction.threads.max' is obsolete and not used
anymore.");
+}
+if (conf.get("hbase.mob.compaction.batch.size") != null) {
+ LOG.warn("'hbase.mob.compaction.batch.size' is obsolete and not used
anymore.");
+}
+ }
+
+ @VisibleForTesting
+ public MobFileCleanerChore() {
+this.master = null;
+ }
+
+ @Override
+ @edu.umd.cs.findbugs.annotations.SuppressWarnings(value =
"REC_CATCH_EXCEPTION",
+ justification = "Intentional")
+
+ protected void chore() {
+TableDescriptors htds = master.getTableDescriptors();
+
+Map map = null;
+try {
+ map = htds.getAll();
+} catch (IOException e) {
+
[jira] [Commented] (HBASE-21902) TestCompactionWithCoprocessor doesn't invoke coprocessor logic
[ https://issues.apache.org/jira/browse/HBASE-21902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036454#comment-17036454 ] david zhang commented on HBASE-21902: - IIRC throwing the RuntimeException just exposes the test passes (false-negative). It isn't intended as a fix. I can provide a fix if you'd like, I recall fixing this in my own branch. Just give me sometime to jog my memory. thanks for the comment Xu Cang. > TestCompactionWithCoprocessor doesn't invoke coprocessor logic > -- > > Key: HBASE-21902 > URL: https://issues.apache.org/jira/browse/HBASE-21902 > Project: HBase > Issue Type: Test > Components: test >Affects Versions: 2.0.4 >Reporter: david zhang >Priority: Minor > Attachments: hbase-test.patch > > > TestCompactionWithCoprocessor is designed to invoke NoOpScanPolicyObserver > which implements default behavior. in reality NoOpScanPolicyObserver isn't > instantiated, TestCompactionWithCoprocessor runs/passes trivially, without > increasing test coverage beyond TestCompaction. > See patch which passes TestCompactionWithCoprocessor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23837) Remove deprecated isLegalFamilyName(byte[]) from HColumnDescriptor
[
https://issues.apache.org/jira/browse/HBASE-23837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Hentschel updated HBASE-23837:
--
Status: Patch Available (was: In Progress)
> Remove deprecated isLegalFamilyName(byte[]) from HColumnDescriptor
> --
>
> Key: HBASE-23837
> URL: https://issues.apache.org/jira/browse/HBASE-23837
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Jan Hentschel
>Assignee: Jan Hentschel
>Priority: Minor
>
> {{isLegalFamilyName(byte[])}} in HColumnDescriptor was deprecated back in
> 2.0.0 and should be removed for 3.0.0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] HorizonNet opened a new pull request #1168: HBASE-23837 Removed deprecated isLegalFamilyName(byte[]) from HColumnDescriptor
HorizonNet opened a new pull request #1168: HBASE-23837 Removed deprecated isLegalFamilyName(byte[]) from HColumnDescriptor URL: https://github.com/apache/hbase/pull/1168 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HBASE-23837) Remove deprecated isLegalFamilyName(byte[]) from HColumnDescriptor
Jan Hentschel created HBASE-23837:
-
Summary: Remove deprecated isLegalFamilyName(byte[]) from
HColumnDescriptor
Key: HBASE-23837
URL: https://issues.apache.org/jira/browse/HBASE-23837
Project: HBase
Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Jan Hentschel
Assignee: Jan Hentschel
{{isLegalFamilyName(byte[])}} in HColumnDescriptor was deprecated back in 2.0.0
and should be removed for 3.0.0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HBASE-23837) Remove deprecated isLegalFamilyName(byte[]) from HColumnDescriptor
[
https://issues.apache.org/jira/browse/HBASE-23837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-23837 started by Jan Hentschel.
-
> Remove deprecated isLegalFamilyName(byte[]) from HColumnDescriptor
> --
>
> Key: HBASE-23837
> URL: https://issues.apache.org/jira/browse/HBASE-23837
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Jan Hentschel
>Assignee: Jan Hentschel
>Priority: Minor
>
> {{isLegalFamilyName(byte[])}} in HColumnDescriptor was deprecated back in
> 2.0.0 and should be removed for 3.0.0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #754: HBASE-22978 : Online slow response log
Apache-HBase commented on issue #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#issuecomment-585928190 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 38s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 13s | master passed | | +1 :green_heart: | compile | 3m 2s | master passed | | +1 :green_heart: | checkstyle | 2m 24s | master passed | | +0 :ok: | refguide | 4m 54s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 4m 40s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 10s | master passed | | +0 :ok: | spotbugs | 1m 23s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 23m 31s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 0s | the patch passed | | +1 :green_heart: | compile | 3m 3s | the patch passed | | +1 :green_heart: | cc | 3m 3s | the patch passed | | +1 :green_heart: | javac | 3m 3s | the patch passed | | -1 :x: | checkstyle | 2m 25s | root: The patch generated 3 new + 448 unchanged - 2 fixed = 451 total (was 450) | | +1 :green_heart: | rubocop | 0m 9s | There were no new rubocop issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 4m 46s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 4m 39s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 15m 51s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | hbaseprotoc | 8m 28s | the patch passed | | +1 :green_heart: | javadoc | 5m 18s | the patch passed | | +1 :green_heart: | findbugs | 24m 30s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 73m 34s | root in the patch failed. | | +1 :green_heart: | asflicense | 3m 47s | The patch does not generate ASF License warnings. | | | | 210m 6s | | | Reason | Tests | |---:|:--| | Unreaped Processes | root:4 | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/12/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/754 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile refguide xml cc hbaseprotoc prototool rubocop | | uname | Linux 78043d41d52c 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-754/out/precommit/personality/provided.sh | | git revision | master / 488c2b2483 | | Default Java | 1.8.0_181 | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/12/artifact/out/branch-site/book.html | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/12/artifact/out/diff-checkstyle-root.txt | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/12/artifact/out/patch-site/book.html | | Unreaped Processes Log | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/12/artifact/out/patch-unit-root-reaper.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/12/artifact/out/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/12/testReport/ | | Max. process+thread count | 8923 (vs. ulimit of 1) |
[jira] [Commented] (HBASE-18095) Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client
[
https://issues.apache.org/jira/browse/HBASE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036443#comment-17036443
]
Hudson commented on HBASE-18095:
Results for branch HBASE-18095/client-locate-meta-no-zookeeper
[build #74 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/74/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/74//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/74//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/74//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Provide an option for clients to find the server hosting META that does not
> involve the ZooKeeper client
>
>
> Key: HBASE-18095
> URL: https://issues.apache.org/jira/browse/HBASE-18095
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Reporter: Andrew Kyle Purtell
>Assignee: Bharath Vissapragada
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
> Attachments: HBASE-18095.master-v1.patch, HBASE-18095.master-v2.patch
>
>
> Clients are required to connect to ZooKeeper to find the location of the
> regionserver hosting the meta table region. Site configuration provides the
> client a list of ZK quorum peers and the client uses an embedded ZK client to
> query meta location. Timeouts and retry behavior of this embedded ZK client
> are managed orthogonally to HBase layer settings and in some cases the ZK
> cannot manage what in theory the HBase client can, i.e. fail fast upon outage
> or network partition.
> We should consider new configuration settings that provide a list of
> well-known master and backup master locations, and with this information the
> client can contact any of the master processes directly. Any master in either
> active or passive state will track meta location and respond to requests for
> it with its cached last known location. If this location is stale, the client
> can ask again with a flag set that requests the master refresh its location
> cache and return the up-to-date location. Every client interaction with the
> cluster thus uses only HBase RPC as transport, with appropriate settings
> applied to the connection. The configuration toggle that enables this
> alternative meta location lookup should be false by default.
> This removes the requirement that HBase clients embed the ZK client and
> contact the ZK service directly at the beginning of the connection lifecycle.
> This has several benefits. ZK service need not be exposed to clients, and
> their potential abuse, yet no benefit ZK provides the HBase server cluster is
> compromised. Normalizing HBase client and ZK client timeout settings and
> retry behavior - in some cases, impossible, i.e. for fail-fast - is no longer
> necessary.
> And, from [~ghelmling]: There is an additional complication here for
> token-based authentication. When a delegation token is used for SASL
> authentication, the client uses the cluster ID obtained from Zookeeper to
> select the token identifier to use. So there would also need to be some
> Zookeeper-less, unauthenticated way to obtain the cluster ID as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on issue #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
ndimiduk commented on issue #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore… URL: https://github.com/apache/hbase/pull/1141#issuecomment-585920811 Follow-on ticket is https://issues.apache.org/jira/browse/HBASE-23836 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23836) Race condition in Master startup can ignore shutdown RPC
[
https://issues.apache.org/jira/browse/HBASE-23836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-23836:
-
Affects Version/s: 2.3.0
> Race condition in Master startup can ignore shutdown RPC
>
>
> Key: HBASE-23836
> URL: https://issues.apache.org/jira/browse/HBASE-23836
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Nick Dimiduk
>Priority: Minor
>
> Identified as the root-cause of HBASE-23808. The master starts its RPC server
> first thing in its constructor, but does not initialize it's
> {{clusterTracker}} until later (both from the super constructor). Thus it is
> possible for a client to issue a {{shutdown}} RPC is received by a master
> before it has a clusterTracker instance. It is highly likely (guaranteed?) to
> also not be the active master at this point. In such a case, the {{shutdown}}
> is received and acknowledged back to the client, but is effectively ignored.
> After chatting with [~bharathv] on the PR, we believe this race becomes more
> likely (though still rare) after HBASE-18095. This is something to consider
> should we get around to a startup sequence refactor.
> Setting priority as "minor" because the condition is merely annoying; from
> what I can tell, a subsequent {{shutdown}} RPC to any master would result in
> the correct shutdown of the first.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] jatsakthi commented on a change in pull request #1138: HBASE-22827 Expose multi-region merge in shell and Admin API
jatsakthi commented on a change in pull request #1138: HBASE-22827 Expose multi-region merge in shell and Admin API URL: https://github.com/apache/hbase/pull/1138#discussion_r379055569 ## File path: hbase-shell/src/main/ruby/shell/commands/merge_region.rb ## @@ -31,18 +31,42 @@ def help TestTable,0094429456,1289497600452.527db22f95c8a9e0116f0cc13c680396. then the encoded region name portion is 527db22f95c8a9e0116f0cc13c680396 +You can either pass the list of regions as comma separated values or as an +array of regions as shown: + Examples: hbase> merge_region 'FULL_REGIONNAME', 'FULL_REGIONNAME' - hbase> merge_region 'FULL_REGIONNAME', 'FULL_REGIONNAME', true + hbase> merge_region 'FULL_REGIONNAME', 'FULL_REGIONNAME', 'FULL_REGIONNAME', ... + hbase> merge_region 'FULL_REGIONNAME', 'FULL_REGIONNAME', 'FULL_REGIONNAME', ..., true + + hbase> merge_region ['FULL_REGIONNAME', 'FULL_REGIONNAME'] + hbase> merge_region ['FULL_REGIONNAME', 'FULL_REGIONNAME', 'FULL_REGIONNAME', ...] + hbase> merge_region ['FULL_REGIONNAME', 'FULL_REGIONNAME', 'FULL_REGIONNAME', ..., true] hbase> merge_region 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME' - hbase> merge_region 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', true + hbase> merge_region 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', ... + hbase> merge_region 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', ..., true + + hbase> merge_region ['ENCODED_REGIONNAME', 'ENCODED_REGIONNAME'] + hbase> merge_region ['ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', ...] + hbase> merge_region ['ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', ..., true] EOF end - def command(region_a_name, region_b_name, force = 'false') -admin.merge_region(region_a_name, region_b_name, force) + def command(*args) +args = args.flatten.compact +args_len = args.length +raise(ArgumentError, 'Must pass atleast 2 regions to merge') unless args_len > 1 Review comment: thanks @joshelser let me make a quick change and push the new commit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #921: HBASE-22749: Distributed MOB compactions
busbey commented on a change in pull request #921: HBASE-22749: Distributed MOB
compactions
URL: https://github.com/apache/hbase/pull/921#discussion_r379050562
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
##
@@ -183,105 +271,184 @@ protected boolean performCompaction(FileDetails fd,
InternalScanner scanner, Cel
boolean hasMore;
Path path = MobUtils.getMobFamilyPath(conf, store.getTableName(),
store.getColumnFamilyName());
byte[] fileName = null;
-StoreFileWriter mobFileWriter = null, delFileWriter = null;
-long mobCells = 0, deleteMarkersCount = 0;
+StoreFileWriter mobFileWriter = null;
+/*
+ * mobCells are used only to decide if we need to commit or abort current
MOB output file.
+ */
+long mobCells = 0;
long cellsCountCompactedToMob = 0, cellsCountCompactedFromMob = 0;
long cellsSizeCompactedToMob = 0, cellsSizeCompactedFromMob = 0;
boolean finished = false;
+
ScannerContext scannerContext =
ScannerContext.newBuilder().setBatchLimit(compactionKVMax).build();
throughputController.start(compactionName);
-KeyValueScanner kvs = (scanner instanceof KeyValueScanner)?
(KeyValueScanner)scanner : null;
-long shippedCallSizeLimit = (long) numofFilesToCompact *
this.store.getColumnFamilyDescriptor().getBlocksize();
+KeyValueScanner kvs = (scanner instanceof KeyValueScanner) ?
(KeyValueScanner) scanner : null;
+long shippedCallSizeLimit =
+(long) numofFilesToCompact *
this.store.getColumnFamilyDescriptor().getBlocksize();
+
+Cell mobCell = null;
try {
- try {
-// If the mob file writer could not be created, directly write the
cell to the store file.
-mobFileWriter = mobStore.createWriterInTmp(new Date(fd.latestPutTs),
fd.maxKeyCount,
- compactionCompression, store.getRegionInfo().getStartKey(), true);
-fileName = Bytes.toBytes(mobFileWriter.getPath().getName());
- } catch (IOException e) {
-LOG.warn("Failed to create mob writer, "
- + "we will continue the compaction by writing MOB cells
directly in store files", e);
- }
- if (major) {
-try {
- delFileWriter = mobStore.createDelFileWriterInTmp(new
Date(fd.latestPutTs),
-fd.maxKeyCount, compactionCompression,
store.getRegionInfo().getStartKey());
-} catch (IOException e) {
- LOG.warn(
-"Failed to create del writer, "
-+ "we will continue the compaction by writing delete markers
directly in store files",
-e);
-}
- }
+
+ mobFileWriter = newMobWriter(fd);
+ fileName = Bytes.toBytes(mobFileWriter.getPath().getName());
+
do {
hasMore = scanner.next(cells, scannerContext);
-if (LOG.isDebugEnabled()) {
- now = EnvironmentEdgeManager.currentTime();
-}
+now = EnvironmentEdgeManager.currentTime();
for (Cell c : cells) {
- if (major && CellUtil.isDelete(c)) {
-if (MobUtils.isMobReferenceCell(c) || delFileWriter == null) {
- // Directly write it to a store file
- writer.append(c);
+ if (compactMOBs) {
+if (MobUtils.isMobReferenceCell(c)) {
+ String fName = MobUtils.getMobFileName(c);
+ Path pp = new Path(new Path(fs.getUri()), new Path(path, fName));
+
+ // Added to support migration
+ try {
+mobCell = mobStore.resolve(c, true, false).getCell();
+ } catch (FileNotFoundException fnfe) {
+if (discardMobMiss) {
+ LOG.error("Missing MOB cell: file={} not found cell={}",
fName, c);
+ continue;
+} else {
+ throw fnfe;
+}
+ }
+
+ if (discardMobMiss && mobCell.getValueLength() == 0) {
+LOG.error("Missing MOB cell value: file={} cell={}", pp,
mobCell);
+continue;
+ } else if (mobCell.getValueLength() == 0) {
+String errMsg = String.format("Found 0 length MOB cell in a
file=%s cell=%s",
+ fName, mobCell);
+throw new IOException(errMsg);
+ }
+
+ if (mobCell.getValueLength() > mobSizeThreshold) {
+// put the mob data back to the MOB store file
+PrivateCellUtil.setSequenceId(mobCell, c.getSequenceId());
+if (!ioOptimizedMode) {
+ mobFileWriter.append(mobCell);
+ mobCells++;
+ writer.append(
+MobUtils.createMobRefCell(mobCell, fileName,
this.mobStore.getRefCellTags()));
+} else {
+ // I/O optimized mode
+ // Check if MOB cell origin file size is
+ // greater than threshold
+
[GitHub] [hbase] ndimiduk commented on a change in pull request #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
ndimiduk commented on a change in pull request #1141: HBASE-23808 [Flakey Test]
TestMasterShutdown#testMasterShutdownBefore…
URL: https://github.com/apache/hbase/pull/1141#discussion_r379049688
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
##
@@ -30,26 +34,38 @@
import org.apache.hadoop.hbase.LocalHBaseCluster;
import org.apache.hadoop.hbase.MiniHBaseCluster;
import org.apache.hadoop.hbase.StartMiniClusterOption;
-import org.apache.hadoop.hbase.client.Admin;
-import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.Waiter;
+import org.apache.hadoop.hbase.client.AsyncConnection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.testclassification.LargeTests;
import org.apache.hadoop.hbase.testclassification.MasterTests;
import org.apache.hadoop.hbase.util.JVMClusterUtil.MasterThread;
+import org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient;
+import org.junit.Before;
import org.junit.ClassRule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import
org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
@Category({MasterTests.class, LargeTests.class})
public class TestMasterShutdown {
+ private static final Logger LOG =
LoggerFactory.getLogger(TestMasterShutdown.class);
@ClassRule
public static final HBaseClassTestRule CLASS_RULE =
HBaseClassTestRule.forClass(TestMasterShutdown.class);
- private static final Logger LOG =
LoggerFactory.getLogger(TestMasterShutdown.class);
+ private HBaseTestingUtility htu;
+
+ @Before
+ public void shutdownCluster() throws IOException {
+if (htu != null) {
Review comment:
I tried writing a test for `HBaseClassTestRule ` and the `Timeout` class it
uses to assert the behavior of a miniCluster in the face of the interrupt, but
that took me round in circles. Going to pass on that for the time being.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (HBASE-23836) Race condition in Master startup can ignore shutdown RPC
Nick Dimiduk created HBASE-23836:
Summary: Race condition in Master startup can ignore shutdown RPC
Key: HBASE-23836
URL: https://issues.apache.org/jira/browse/HBASE-23836
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 3.0.0
Reporter: Nick Dimiduk
Identified as the root-cause of HBASE-23808. The master starts its RPC server
first thing in its constructor, but does not initialize it's {{clusterTracker}}
until later (both from the super constructor). Thus it is possible for a client
to issue a {{shutdown}} RPC is received by a master before it has a
clusterTracker instance. It is highly likely (guaranteed?) to also not be the
active master at this point. In such a case, the {{shutdown}} is received and
acknowledged back to the client, but is effectively ignored.
After chatting with [~bharathv] on the PR, we believe this race becomes more
likely (though still rare) after HBASE-18095. This is something to consider
should we get around to a startup sequence refactor.
Setting priority as "minor" because the condition is merely annoying; from what
I can tell, a subsequent {{shutdown}} RPC to any master would result in the
correct shutdown of the first.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HBASE-23743) Release 1.4.13
[ https://issues.apache.org/jira/browse/HBASE-23743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-23743 started by Sakthi. -- > Release 1.4.13 > -- > > Key: HBASE-23743 > URL: https://issues.apache.org/jira/browse/HBASE-23743 > Project: HBase > Issue Type: Task > Components: build, community >Affects Versions: 1.4.13 >Reporter: Sean Busbey >Assignee: Sakthi >Priority: Major > Fix For: 1.4.13 > > > over due for 1.4.13. I'll try to use the release manager scripts this time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] joshelser commented on a change in pull request #1138: HBASE-22827 Expose multi-region merge in shell and Admin API
joshelser commented on a change in pull request #1138: HBASE-22827 Expose multi-region merge in shell and Admin API URL: https://github.com/apache/hbase/pull/1138#discussion_r379034894 ## File path: hbase-shell/src/main/ruby/shell/commands/merge_region.rb ## @@ -31,18 +31,42 @@ def help TestTable,0094429456,1289497600452.527db22f95c8a9e0116f0cc13c680396. then the encoded region name portion is 527db22f95c8a9e0116f0cc13c680396 +You can either pass the list of regions as comma separated values or as an +array of regions as shown: + Examples: hbase> merge_region 'FULL_REGIONNAME', 'FULL_REGIONNAME' - hbase> merge_region 'FULL_REGIONNAME', 'FULL_REGIONNAME', true + hbase> merge_region 'FULL_REGIONNAME', 'FULL_REGIONNAME', 'FULL_REGIONNAME', ... + hbase> merge_region 'FULL_REGIONNAME', 'FULL_REGIONNAME', 'FULL_REGIONNAME', ..., true + + hbase> merge_region ['FULL_REGIONNAME', 'FULL_REGIONNAME'] + hbase> merge_region ['FULL_REGIONNAME', 'FULL_REGIONNAME', 'FULL_REGIONNAME', ...] + hbase> merge_region ['FULL_REGIONNAME', 'FULL_REGIONNAME', 'FULL_REGIONNAME', ..., true] hbase> merge_region 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME' - hbase> merge_region 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', true + hbase> merge_region 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', ... + hbase> merge_region 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', ..., true + + hbase> merge_region ['ENCODED_REGIONNAME', 'ENCODED_REGIONNAME'] + hbase> merge_region ['ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', ...] + hbase> merge_region ['ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', ..., true] EOF end - def command(region_a_name, region_b_name, force = 'false') -admin.merge_region(region_a_name, region_b_name, force) + def command(*args) +args = args.flatten.compact +args_len = args.length +raise(ArgumentError, 'Must pass atleast 2 regions to merge') unless args_len > 1 Review comment: nit: "at least" This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on issue #754: HBASE-22978 : Online slow response log
virajjasani commented on issue #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#issuecomment-585829397 > Thanks for addressing all my concerns so far @virajjasani . I've tried to resolve all the inline discussions. Here's a summary of what I think remains outstanding. > > * log large responses (follow-on task)? > > * don’t assume “tooLarge” and “tooSlow” are mutually exclusive. > * replicate log entries into a file on HDFS or a new system table (follow-on task)? > * expose slow/large responses as a metric (follow-on task)? > * do we need to ship with thrift api parity? > * push filter and limit logic into the server, so all clients have the same experience > > * I think this requires adding a `TooSlowRequest` protobuf message with filter and limit parameters > * Maybe then `SlowLogPayload` is renamed to `SlowLogRecord` or something like that > * need separation of concerns between SlowLog.toString and formatting of output in the shell > * avoid adding to the surface area of HConstants > * proper interrupt handling within SlowLogEventHandler > * update configuration adoc to reference the feature adoc > * test cleanup > > * avoid sleeping for fixed time units, instead use waitFor > * maybe combine some methods containing repeated logic, introduce a helper method or two Thanks @ndimiduk for all your review. I have tried to cover these points in my recent commit except for `need separation of concerns between SlowLog.toString and formatting of output in the shell`. The reason being shell is not able to use Gson as efficiently as keeping `Gson.toJson()` in toString of SlowLogRecord because we have some serialization filters (neither does shell have any good pretty print library that works with our POJO - AFAIK). shell or any client would receive `List` as output of `Admin.getSlowLogResponses()`. In a way, keeping `Gson.toJson()` within `toString()` would ensure all clients can print output in pretty print format. Do you think this is good to maintain uniform print format across all clients? I have covered other points provided by you, please review. Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r378660761
##
File path:
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/client/ThriftAdmin.java
##
@@ -1157,6 +1158,18 @@ public boolean isSnapshotCleanupEnabled() {
throw new NotImplementedException("isSnapshotCleanupEnabled not supported
in ThriftAdmin");
}
+ @Override
+ public List getSlowLogResponses(final List
serverNames)
+ throws IOException {
+throw new NotImplementedException("getSlowLogResponses not supported in
ThriftAdmin");
Review comment:
I am not sure if it is deprecated but for at least 1 year, I have not seen
much of APIs getting implemented here, if you see, majority of APIs on this
file have "not supported in ThriftAdmin". Only earlier APIs like schema related
ops create, alter etc are implemented here.
Anyways, I am fine with taking this in follow up work, sir @ndimiduk . Your
call :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r378699546
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/SlowLogPayload.java
##
@@ -0,0 +1,318 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.client;
+
+import org.apache.hadoop.hbase.util.GsonUtil;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.apache.hbase.thirdparty.com.google.gson.Gson;
+import org.apache.hbase.thirdparty.com.google.gson.JsonObject;
+import org.apache.hbase.thirdparty.com.google.gson.JsonSerializer;
+
+/**
+ * SlowLog payload for client
+ */
+@InterfaceAudience.Private
+final public class SlowLogPayload {
+
+ private static final Gson GSON = GsonUtil.createGson()
+.setPrettyPrinting()
+.registerTypeAdapter(SlowLogPayload.class, (JsonSerializer)
+ (slowLogPayload, type, jsonSerializationContext) -> {
Review comment:
Moving to shaded GSON was decided due to never ending list of CVE fixes in
Jackson that used to keep hurting us and we had to keep making maintenance
releases for each active release branches
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #1167: Hbase 18095: Zookeeper-less client connection implementation
busbey commented on a change in pull request #1167: Hbase 18095: Zookeeper-less client connection implementation URL: https://github.com/apache/hbase/pull/1167#discussion_r378922404 ## File path: src/main/asciidoc/_chapters/configuration.adoc ## @@ -563,38 +563,63 @@ Changes here will require a cluster restart for HBase to notice the change thoug If you are running HBase in standalone mode, you don't need to configure anything for your client to work provided that they are all on the same machine. -Since the HBase Master may move around, clients bootstrap by looking to ZooKeeper for current critical locations. -ZooKeeper is where all these values are kept. -Thus clients require the location of the ZooKeeper ensemble before they can do anything else. -Usually this ensemble location is kept out in the _hbase-site.xml_ and is picked up by the client from the `CLASSPATH`. +Starting release 3.0.0, the default connection registry has been switched to a master based implementation. Refer to <> for more details about +what a connection registry is and implications of this change. Depending on your HBase version, following is the expected minimal client configuration. -If you are configuring an IDE to run an HBase client, you should include the _conf/_ directory on your classpath so _hbase-site.xml_ settings can be found (or add _src/test/resources_ to pick up the hbase-site.xml used by tests). + Up until 2.x.y releases +In 2.x.y releases, the default connection registry was based on ZooKeeper as the source of truth. This means that the clients always looked up ZooKeeper znodes to fetch +the required metadata. For example, if an active master crashed and the a new master is elected, clients looked up the master znode to fetch +the active master address (similarly for meta locations). This meant that the clients needed to have access to ZooKeeper and need to know +the ZooKeeper ensemble information before they can do anything. This can be configured in the client configuration xml as follows: -For Java applications using Maven, including the hbase-shaded-client module is the recommended dependency when connecting to a cluster: [source,xml] - - org.apache.hbase - hbase-shaded-client - 2.0.0 - + + + + +hbase.zookeeper.quorum +example1,example2,example3 + Zookeeper ensemble information + + -A basic example _hbase-site.xml_ for client only may look as follows: + Starting 3.0.0 release + +The default implementation was switched to a master based connection registry. With this implementation, clients always contact the active or +stand-by master RPC end points to fetch the the connection registry information. This means that the clients should have access to the list of active and master +end points before they can do anything. This can be configured in the client configuration xml as follows: Review comment: This specifically doesn't need to be _all_ of the Masters configured for the cluster, right? So if I wanted to hedge against clients ddosing masters to the point that I can't get an active master for the cluster I could e.g. only give them half of the Masters. Presuming that's the case, a brief entry in the troubleshooting section that basically says to do this in case clients are causing masters to die would help us get ahead of that concern. I say this as someone who has had to troubleshoot clusters where bad client behavior essentially meant every hbase operation hit ZK for meta and as a result said cluster would die. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #1167: Hbase 18095: Zookeeper-less client connection implementation
busbey commented on a change in pull request #1167: Hbase 18095: Zookeeper-less client connection implementation URL: https://github.com/apache/hbase/pull/1167#discussion_r378918970 ## File path: src/main/asciidoc/_chapters/architecture.adoc ## @@ -577,11 +644,24 @@ If the active Master loses its lease in ZooKeeper (or the Master shuts down), th [[master.runtime]] === Runtime Impact -A common dist-list question involves what happens to an HBase cluster when the Master goes down. +A common dist-list question involves what happens to an HBase cluster when the Master goes down. This information has changed staring 3.0.0. + + Up until releases 2.x.y Because the HBase client talks directly to the RegionServers, the cluster can still function in a "steady state". Additionally, per <>, `hbase:meta` exists as an HBase table and is not resident in the Master. However, the Master controls critical functions such as RegionServer failover and completing region splits. So while the cluster can still run for a short time without the Master, the Master should be restarted as soon as possible. + Staring release 3.0.0 +As mentioned in section <>, the default connection registry for clients is now based on master rpc end points. Hence the requirements for +masters' uptime are even tighter starting this release. + +- At least one active or stand by master is needed for a connection set up, unlike before when all the clients needed was a ZooKeeper ensemble. +- Master is now in critical path for read/write operations. For example, if the meta region bounces off to a different region server, clients +need master to fetch the new locations. Earlier this was done by fetching this information directly from ZooKeeper. +- Masters will now have higher connection load than before. So, the server side configuration might need adjustment depending on the load. + +Overall, the master uptime requirements starting with this version are even higher for the client operations to go through. Review comment: Should phrase this change in uptime needs of the master ensemble in terms of when the feature is enabled rather than in terms of hbase versions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #1167: Hbase 18095: Zookeeper-less client connection implementation
busbey commented on a change in pull request #1167: Hbase 18095: Zookeeper-less client connection implementation URL: https://github.com/apache/hbase/pull/1167#discussion_r378920122 ## File path: dev-support/design-docs/HBASE-18095-Zookeeper-less-client-connection-design.adoc ## @@ -0,0 +1,112 @@ + +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + += HBASE-18095: Zookeeper-less client connection + + +== Context +Currently, Zookeeper (ZK) lies in the critical code path of connection init. To set up a connection to a given HBase cluster, client relies on the zookeeper quorum configured in the client hbase-site.xml and attempts to fetch the following information. + +* ClusterID +* Active HMaster server name +* Meta table region locations + +ZK is deemed the source of truth since other processes that maintain the cluster state persist the changes to this data into ZK. So it is an obvious place to look at for clients to fetch the latest cluster state. However this comes with it’s own set of problems, some of them are below. + +* Timeouts and retry logic for ZK clients are managed separately from HBase configuration. This is more administration overhead for end users (example: multiple timeouts are to be configured for different types of RPCs. client->master, client->ZK etc.). This prevents HBase from having a single holistic timeout configuration that applies to any RPCs. +* If there is any issue with ZK (like connection overload / timeouts), the entire HBase service appears frozen and there is little visibility into it. +* Exposing zookeeper to all the clients can be risky since it can potentially be abused to DDOS. +* Embedded ZK client is bundled with hbase client jar as a dependency (with it’s log spew :-]). Review comment: Also clients need an additional set of config files in secure deployments because zk uses JAAS This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] busbey commented on issue #1167: Hbase 18095: Zookeeper-less client connection implementation
busbey commented on issue #1167: Hbase 18095: Zookeeper-less client connection implementation URL: https://github.com/apache/hbase/pull/1167#issuecomment-585803236 The docs changes are nice, thanks! The plan is for this to be opt-in but available on the next minor 2.y and 1.y releases right? If that's true then the sections that talk about the default in different versions should also clearly state how folks can do that opt-in. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23835) TestFromClientSide3 and subclasses often fail on testScanAfterDeletingSpecifiedRowV2.
[ https://issues.apache.org/jira/browse/HBASE-23835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Robert Miller updated HBASE-23835: --- Description: This test method fails a fair amount on me with something like: TestFromClientSide3WoUnsafe>TestFromClientSide3.testScanAfterDeletingSpecifiedRowV2:236 expected:<3> but was:<2> I had a hunch that it might be due to interference from other test methods running first so I tried changing the table name for just this method to be unique - still fails. However, when I just run testScanAfterDeletingSpecifiedRowV2 on it's own without the methods, it does not seem to fail so far. was: This test method fails a fair amount on me with something like: TestFromClientSide3WoUnsafe>TestFromClientSide3.testScanAfterDeletingSpecifiedRowV2:236 expected:<3> but was:<2> I had a hunch that it might be due to interference from other test methods running first so I tried changing the table name for just this method to be unique - still fails. However, when I just run > TestFromClientSide3 and subclasses often fail on > testScanAfterDeletingSpecifiedRowV2. > - > > Key: HBASE-23835 > URL: https://issues.apache.org/jira/browse/HBASE-23835 > Project: HBase > Issue Type: Test >Affects Versions: master >Reporter: Mark Robert Miller >Priority: Major > > This test method fails a fair amount on me with something like: > TestFromClientSide3WoUnsafe>TestFromClientSide3.testScanAfterDeletingSpecifiedRowV2:236 > expected:<3> but was:<2> > I had a hunch that it might be due to interference from other test methods > running first so I tried changing the table name for just this method to be > unique - still fails. > However, when I just run testScanAfterDeletingSpecifiedRowV2 on it's own > without the methods, it does not seem to fail so far. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23835) TestFromClientSide3 and subclasses often fail on testScanAfterDeletingSpecifiedRowV2.
[ https://issues.apache.org/jira/browse/HBASE-23835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Robert Miller updated HBASE-23835: --- Description: This test method fails a fair amount on me with something like: TestFromClientSide3WoUnsafe>TestFromClientSide3.testScanAfterDeletingSpecifiedRowV2:236 expected:<3> but was:<2> I had a hunch that it might be due to interference from other test methods running first so I tried changing the table name for just this method to be unique - still fails. However, when I just run was: This test method fails a fair amount on me with something like: > TestFromClientSide3 and subclasses often fail on > testScanAfterDeletingSpecifiedRowV2. > - > > Key: HBASE-23835 > URL: https://issues.apache.org/jira/browse/HBASE-23835 > Project: HBase > Issue Type: Test >Affects Versions: master >Reporter: Mark Robert Miller >Priority: Major > > This test method fails a fair amount on me with something like: > TestFromClientSide3WoUnsafe>TestFromClientSide3.testScanAfterDeletingSpecifiedRowV2:236 > expected:<3> but was:<2> > I had a hunch that it might be due to interference from other test methods > running first so I tried changing the table name for just this method to be > unique - still fails. > However, when I just run -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23835) TestFromClientSide3 and subclasses often fail on testScanAfterDeletingSpecifiedRowV2.
Mark Robert Miller created HBASE-23835: -- Summary: TestFromClientSide3 and subclasses often fail on testScanAfterDeletingSpecifiedRowV2. Key: HBASE-23835 URL: https://issues.apache.org/jira/browse/HBASE-23835 Project: HBase Issue Type: Test Affects Versions: master Reporter: Mark Robert Miller This test method fails a fair amount on me with something like: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23816) [Flakey Test] TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91) Wrong FS!
[
https://issues.apache.org/jira/browse/HBASE-23816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036135#comment-17036135
]
Hudson commented on HBASE-23816:
Results for branch master
[build #1628 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1628/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1628//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1628//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1628//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> [Flakey Test]
> TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91)
> Wrong FS!
>
>
> Key: HBASE-23816
> URL: https://issues.apache.org/jira/browse/HBASE-23816
> Project: HBase
> Issue Type: Sub-task
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: Screen Shot 2020-02-11 at 9.00.06 AM.png, Screen Shot
> 2020-02-12 at 7.12.41 PM.png
>
>
> Can't repro locally w/o running with lots of parallelism. Follow-on from
> parent issue.
> Here is error:
> {code}
> [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 115.741 s <<< FAILURE! - in
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster
> [ERROR]
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState
> Time elapsed: 74.459 s <<< ERROR!
> java.lang.IllegalArgumentException: Wrong FS:
> file:/Users/stack/checkouts/hbase.apache.git/hbase-mapreduce/target/test-data/fad01338-65ef-e973-2412-05de114016fe/.hbase-snapshot/tableWithRefsV2,
> expected: hdfs://localhost:58051
>at
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster.testSnapshotWithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:107)
>at
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91)
> {code}
> With v1 had just run find which does same thing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1167: Hbase 18095: Zookeeper-less client connection implementation
Apache-HBase commented on issue #1167: Hbase 18095: Zookeeper-less client connection implementation URL: https://github.com/apache/hbase/pull/1167#issuecomment-585629650 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 4s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 96 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 37s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 20s | master passed | | +1 :green_heart: | compile | 3m 7s | master passed | | +1 :green_heart: | checkstyle | 2m 39s | master passed | | +0 :ok: | refguide | 4m 57s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 4m 44s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 27s | master passed | | +0 :ok: | spotbugs | 1m 18s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 24m 47s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 1s | the patch passed | | +1 :green_heart: | compile | 3m 6s | the patch passed | | +1 :green_heart: | cc | 3m 6s | the patch passed | | +1 :green_heart: | javac | 3m 6s | the patch passed | | -1 :x: | checkstyle | 2m 31s | root: The patch generated 1 new + 753 unchanged - 96 fixed = 754 total (was 849) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 4m 44s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 4m 42s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 16m 14s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | hbaseprotoc | 8m 44s | the patch passed | | +1 :green_heart: | javadoc | 5m 39s | the patch passed | | +1 :green_heart: | findbugs | 26m 51s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 75m 46s | root in the patch failed. | | +1 :green_heart: | asflicense | 3m 43s | The patch does not generate ASF License warnings. | | | | 219m 13s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.TestMobRestoreSnapshotFromClientAfterSplittingRegions | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1167/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1167 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool refguide | | uname | Linux 0b182b130902 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1167/out/precommit/personality/provided.sh | | git revision | master / 488c2b2483 | | Default Java | 1.8.0_181 | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1167/2/artifact/out/branch-site/book.html | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1167/2/artifact/out/diff-checkstyle-root.txt | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1167/2/artifact/out/patch-site/book.html | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1167/2/artifact/out/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1167/2/testReport/ | | Max. process+thread count | 8337 (vs. ulimit of 1) | | modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-zookeeper hbase-server hbase-mapreduce hbase-thrift . U: . | | Console output |
