[GitHub] [hbase] Apache-HBase commented on pull request #2071: HBASE-24743 Reject to add a peer which replicate to itself earlier
Apache-HBase commented on pull request #2071: URL: https://github.com/apache/hbase/pull/2071#issuecomment-659164148 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 46s | master passed | | +1 :green_heart: | checkstyle | 1m 13s | master passed | | +1 :green_heart: | spotbugs | 2m 3s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 46s | the patch passed | | -0 :warning: | checkstyle | 1m 12s | hbase-server: The patch generated 1 new + 114 unchanged - 0 fixed = 115 total (was 114) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 9s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 35m 13s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2071/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2071 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 50420a6cf4e1 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2505c7760d | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2071/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2071/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158891#comment-17158891 ] Michael Stack commented on HBASE-11288: --- I put up a discussion doc on distributing ROOT load: [https://docs.google.com/document/d/12B1BOMLx_esFycILdHmIYh0NKOzWUJ1I9XuvL75Euug/edit?usp=sharing] TL;DR asks if concern for ROOT loading is the proper focus; rather should we be solving hotspotting/meta access prioritization/etc for the general case. If ROOT loading is critical, has some starts on possible implementations to get the discussion going. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24723) Distributing ROOT Region load
[ https://issues.apache.org/jira/browse/HBASE-24723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158884#comment-17158884 ] Michael Stack commented on HBASE-24723: --- Discussion doc. Looks at distributing ROOT Region load. > Distributing ROOT Region load > - > > Key: HBASE-24723 > URL: https://issues.apache.org/jira/browse/HBASE-24723 > Project: HBase > Issue Type: Bug >Reporter: Michael Stack >Priority: Major > > HBASE-11288 'Splittable Meta' progress keeps getting hung up on how to cache > the 'root' table – the first table in the hierarchy of lookups locating > user-data in a Table Region. The 'root' table is not splittable by design so > its load can not be spread about the cluster by splitting the hot Region as > we would usually do. > This issue is about listing out requirements, sketching possible approaches, > and then getting agreement ahead of implementation as subtasks. > In particular, an approach that would work whether the Region to be cached is > master-based or hosted by a RegionServer (see design attached to HBASE-11288 > for elaboration) would be preferred since then we can separate discussion of > cache from the HBASE-11288 'split meta' discussion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2071: HBASE-24743 Reject to add a peer which replicate to itself earlier
Apache-HBase commented on pull request #2071: URL: https://github.com/apache/hbase/pull/2071#issuecomment-659153641 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 46s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 17s | master passed | | +1 :green_heart: | compile | 1m 26s | master passed | | +1 :green_heart: | shadedjars | 7m 46s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 49s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 45s | the patch passed | | +1 :green_heart: | compile | 1m 33s | the patch passed | | +1 :green_heart: | javac | 1m 33s | the patch passed | | +1 :green_heart: | shadedjars | 7m 47s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 51s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 228m 50s | hbase-server in the patch failed. | | | | 263m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2071/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2071 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c7903439634c 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2505c7760d | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2071/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2071/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2071/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2071/2/testReport/ | | Max. process+thread count | 2910 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2071/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2071: HBASE-24743 Reject to add a peer which replicate to itself earlier
Apache-HBase commented on pull request #2071: URL: https://github.com/apache/hbase/pull/2071#issuecomment-659152126 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 47s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 39s | master passed | | +1 :green_heart: | compile | 1m 9s | master passed | | +1 :green_heart: | shadedjars | 6m 49s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 38s | the patch passed | | +1 :green_heart: | compile | 1m 12s | the patch passed | | +1 :green_heart: | javac | 1m 12s | the patch passed | | +1 :green_heart: | shadedjars | 6m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 227m 26s | hbase-server in the patch failed. | | | | 257m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2071/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2071 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5fcf1debfa1f 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2505c7760d | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2071/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2071/2/testReport/ | | Max. process+thread count | 3433 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2071/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2077: HBASE-24684 Fetch ReplicationSink servers list from HMaster instead o…
Apache-HBase commented on pull request #2077: URL: https://github.com/apache/hbase/pull/2077#issuecomment-659147815 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | prototool | 0m 1s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-24666 Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 38s | HBASE-24666 passed | | +1 :green_heart: | checkstyle | 2m 42s | HBASE-24666 passed | | +1 :green_heart: | spotbugs | 7m 32s | HBASE-24666 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 22s | the patch passed | | -0 :warning: | checkstyle | 0m 28s | hbase-client: The patch generated 2 new + 38 unchanged - 0 fixed = 40 total (was 38) | | -0 :warning: | checkstyle | 1m 8s | hbase-server: The patch generated 1 new + 182 unchanged - 2 fixed = 183 total (was 184) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 7s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | hbaseprotoc | 2m 26s | the patch passed | | +1 :green_heart: | spotbugs | 8m 9s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 50s | The patch does not generate ASF License warnings. | | | | 51m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2077 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle cc hbaseprotoc prototool | | uname | Linux f8e7bbbf35c0 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24666 / 9e8c930feb | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-client.txt | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2077/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2073: HBASE-24467 Backport HBASE-23963: Split TestFromClientSide; it takes …
Apache-HBase commented on pull request #2073: URL: https://github.com/apache/hbase/pull/2073#issuecomment-659147610 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ branch-2.2 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 42s | branch-2.2 passed | | +1 :green_heart: | compile | 1m 21s | branch-2.2 passed | | +1 :green_heart: | checkstyle | 1m 54s | branch-2.2 passed | | +1 :green_heart: | shadedjars | 4m 27s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | branch-2.2 passed | | +0 :ok: | spotbugs | 3m 47s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 40s | branch-2.2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 13s | the patch passed | | +1 :green_heart: | compile | 1m 30s | the patch passed | | +1 :green_heart: | javac | 1m 30s | the patch passed | | +1 :green_heart: | checkstyle | 2m 0s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 32s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 28m 33s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | javadoc | 0m 58s | the patch passed | | +1 :green_heart: | findbugs | 4m 48s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 0s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 162m 1s | hbase-server in the patch passed. | | +1 :green_heart: | asflicense | 0m 49s | The patch does not generate ASF License warnings. | | | | 240m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2073/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2073 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 1aefff662035 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-2073/out/precommit/personality/provided.sh | | git revision | branch-2.2 / f44336d277 | | Default Java | 1.8.0_181 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2073/2/testReport/ | | Max. process+thread count | 4878 (vs. ulimit of 1) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2073/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2077: HBASE-24684 Fetch ReplicationSink servers list from HMaster instead o…
infraio commented on a change in pull request #2077:
URL: https://github.com/apache/hbase/pull/2077#discussion_r455496591
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java
##
@@ -148,27 +149,30 @@ public boolean isAborted() {
return false;
}
+ /**
+ * Get the connection to peer cluster
+ * @return connection to peer cluster
+ * @throws IOException
+ */
+ protected synchronized Connection getPeerConnection() throws IOException {
+if (peerConnection == null) {
+ peerConnection =
ConnectionFactory.createConnection(ctx.getConfiguration());
+}
+return peerConnection;
+ }
+
/**
* Get the list of all the region servers from the specified peer
- * @param zkw zk connection to use
* @return list of region server addresses or an empty list if the slave is
unavailable
*/
- protected static List fetchSlavesAddresses(ZKWatcher zkw)
Review comment:
Should be compatible with the old ZK impl. Cannot remove them directly?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1935: HBASE-22146 SpaceQuotaViolationPolicy Disable is not working in Names…
Apache-HBase commented on pull request #1935: URL: https://github.com/apache/hbase/pull/1935#issuecomment-659140171 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 10m 51s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 5s | master passed | | +1 :green_heart: | compile | 1m 21s | master passed | | +1 :green_heart: | shadedjars | 6m 3s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 47s | the patch passed | | +1 :green_heart: | compile | 1m 21s | the patch passed | | +1 :green_heart: | javac | 1m 21s | the patch passed | | +1 :green_heart: | shadedjars | 6m 3s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 4s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 8s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 219m 55s | hbase-server in the patch passed. | | | | 259m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1935/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1935 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8818ecb214cb 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2505c7760d | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1935/4/testReport/ | | Max. process+thread count | 2903 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1935/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1935: HBASE-22146 SpaceQuotaViolationPolicy Disable is not working in Names…
Apache-HBase commented on pull request #1935: URL: https://github.com/apache/hbase/pull/1935#issuecomment-659138591 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 44s | master passed | | +1 :green_heart: | compile | 1m 37s | master passed | | +1 :green_heart: | shadedjars | 6m 25s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 27s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 40s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 32s | the patch passed | | +1 :green_heart: | compile | 1m 38s | the patch passed | | +1 :green_heart: | javac | 1m 38s | the patch passed | | +1 :green_heart: | shadedjars | 6m 31s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 26s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 41s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 20s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 220m 39s | hbase-server in the patch passed. | | | | 253m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1935/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1935 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8bb9f794a9cf 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2505c7760d | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1935/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1935/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1935/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1935/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1935/4/testReport/ | | Max. process+thread count | 3197 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1935/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ddupg opened a new pull request #2077: HBASE-24684 Fetch ReplicationSink servers list from HMaster instead o…
ddupg opened a new pull request #2077: URL: https://github.com/apache/hbase/pull/2077 …f ZooKeeper This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24298) Reduce cpu load of locating region especially in batch mode.
[ https://issues.apache.org/jira/browse/HBASE-24298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] star updated HBASE-24298: - Attachment: HBASE-24298.patch > Reduce cpu load of locating region especially in batch mode. > > > Key: HBASE-24298 > URL: https://issues.apache.org/jira/browse/HBASE-24298 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.12 >Reporter: star >Assignee: star >Priority: Major > Fix For: 1.2.12 > > Attachments: HBASE-24298.patch, locating region.svg > > > Binary search is used to speedup the process of locating region. It is > already fast enough, while cpu of HBASE client becomes the bottleneck when > doing TCSB benchmark. We can make the process of locating region faster to > reduce cpu load in some special cases , which however is our common case in > production environment. It is the case: > 1. Predefined splits in uniform distribution. > > 2. Load data in batch mode. > The optimization is very simple, just to contract range of binary search. > Initially, record all startIndex and endIndex of first or two bytes of keys. > When a region key comes, find the contracted startIndex and endIndex of the > key. Then return to normal binary search process with the specified > startIndex and endIndex. > Then we can ideally reduce cpu to 1/8 with 1 byte or 1/16 with 2 bytes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24298) Reduce cpu load of locating region especially in batch mode.
[ https://issues.apache.org/jira/browse/HBASE-24298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] star updated HBASE-24298: - Fix Version/s: 1.2.12 Affects Version/s: 1.2.12 Status: Patch Available (was: Open) > Reduce cpu load of locating region especially in batch mode. > > > Key: HBASE-24298 > URL: https://issues.apache.org/jira/browse/HBASE-24298 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.12 >Reporter: star >Assignee: star >Priority: Major > Fix For: 1.2.12 > > Attachments: HBASE-24298.patch, locating region.svg > > > Binary search is used to speedup the process of locating region. It is > already fast enough, while cpu of HBASE client becomes the bottleneck when > doing TCSB benchmark. We can make the process of locating region faster to > reduce cpu load in some special cases , which however is our common case in > production environment. It is the case: > 1. Predefined splits in uniform distribution. > > 2. Load data in batch mode. > The optimization is very simple, just to contract range of binary search. > Initially, record all startIndex and endIndex of first or two bytes of keys. > When a region key comes, find the contracted startIndex and endIndex of the > key. Then return to normal binary search process with the specified > startIndex and endIndex. > Then we can ideally reduce cpu to 1/8 with 1 byte or 1/16 with 2 bytes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24298) Reduce cpu load of locating region especially in batch mode.
[ https://issues.apache.org/jira/browse/HBASE-24298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] star updated HBASE-24298: - Attachment: (was: HBASE-24298.patch) > Reduce cpu load of locating region especially in batch mode. > > > Key: HBASE-24298 > URL: https://issues.apache.org/jira/browse/HBASE-24298 > Project: HBase > Issue Type: Bug >Reporter: star >Assignee: star >Priority: Major > Attachments: locating region.svg > > > Binary search is used to speedup the process of locating region. It is > already fast enough, while cpu of HBASE client becomes the bottleneck when > doing TCSB benchmark. We can make the process of locating region faster to > reduce cpu load in some special cases , which however is our common case in > production environment. It is the case: > 1. Predefined splits in uniform distribution. > > 2. Load data in batch mode. > The optimization is very simple, just to contract range of binary search. > Initially, record all startIndex and endIndex of first or two bytes of keys. > When a region key comes, find the contracted startIndex and endIndex of the > key. Then return to normal binary search process with the specified > startIndex and endIndex. > Then we can ideally reduce cpu to 1/8 with 1 byte or 1/16 with 2 bytes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24298) Reduce cpu load of locating region especially in batch mode.
[
https://issues.apache.org/jira/browse/HBASE-24298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158851#comment-17158851
]
star edited comment on HBASE-24298 at 7/16/20, 3:08 AM:
A prototype implementation for review. It is a configurable feature in hbase
client. You can activate the optimization per table by setting properties as
following.
{code:java}
// table 'user' is optimized with ByteCopyOnWriteArrayMap
conf.setString("hbase.client.meta.cache.optimize.user", "4,2");{code}
Hbase client will cache meta data of table 'user', and locate regions in an
optimized way. "4,2" means skipping 4 bytes and using followed 2 bytes to
contract the ranger of binary search.
The optimization reduce 50% of cpu time in a simple performance test case from
our production environment TestByteCopyOnWriteMaps#testLowerKeyPerformance.
was (Author: starphin):
A prototype implementation for review. It is a configurable feature in hbase
client. You can activate the optimization per table by setting properties as
following.
{code:java}
// table 'user' is optimized with ByteCopyOnWriteArrayMap
conf.setString("hbase.client.meta.cache.optimize.user", "4,2");{code}
Hbase client will cache meta data of table 'user', and locate regions in an
optimized way. "4,2" means skipping 4 bytes and using followed 2 bytes to
contract the ranger of binary search.
> Reduce cpu load of locating region especially in batch mode.
>
>
> Key: HBASE-24298
> URL: https://issues.apache.org/jira/browse/HBASE-24298
> Project: HBase
> Issue Type: Bug
>Reporter: star
>Assignee: star
>Priority: Major
> Attachments: HBASE-24298.patch, locating region.svg
>
>
> Binary search is used to speedup the process of locating region. It is
> already fast enough, while cpu of HBASE client becomes the bottleneck when
> doing TCSB benchmark. We can make the process of locating region faster to
> reduce cpu load in some special cases , which however is our common case in
> production environment. It is the case:
> 1. Predefined splits in uniform distribution.
>
> 2. Load data in batch mode.
> The optimization is very simple, just to contract range of binary search.
> Initially, record all startIndex and endIndex of first or two bytes of keys.
> When a region key comes, find the contracted startIndex and endIndex of the
> key. Then return to normal binary search process with the specified
> startIndex and endIndex.
> Then we can ideally reduce cpu to 1/8 with 1 byte or 1/16 with 2 bytes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Reidddddd commented on pull request #2063: HBASE-24578 [WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count
Reidd commented on pull request #2063: URL: https://github.com/apache/hbase/pull/2063#issuecomment-659127335 By adding a space or something to trigger the QA, if it works, you can change back This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Reidddddd commented on pull request #2063: HBASE-24578 [WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count
Reidd commented on pull request #2063: URL: https://github.com/apache/hbase/pull/2063#issuecomment-659127024 @WenFeiYi you can try again This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24298) Reduce cpu load of locating region especially in batch mode.
[
https://issues.apache.org/jira/browse/HBASE-24298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158851#comment-17158851
]
star commented on HBASE-24298:
--
A prototype implementation for review. It is a configurable feature in hbase
client. You can activate the optimization per table by setting properties as
following.
{code:java}
// table 'user' is optimized with ByteCopyOnWriteArrayMap
conf.setString("hbase.client.meta.cache.optimize.user", "4,2");{code}
Hbase client will cache meta data of table 'user', and locate regions in an
optimized way. "4,2" means skipping 4 bytes and using followed 2 bytes to
contract the ranger of binary search.
> Reduce cpu load of locating region especially in batch mode.
>
>
> Key: HBASE-24298
> URL: https://issues.apache.org/jira/browse/HBASE-24298
> Project: HBase
> Issue Type: Bug
>Reporter: star
>Assignee: star
>Priority: Major
> Attachments: HBASE-24298.patch, locating region.svg
>
>
> Binary search is used to speedup the process of locating region. It is
> already fast enough, while cpu of HBASE client becomes the bottleneck when
> doing TCSB benchmark. We can make the process of locating region faster to
> reduce cpu load in some special cases , which however is our common case in
> production environment. It is the case:
> 1. Predefined splits in uniform distribution.
>
> 2. Load data in batch mode.
> The optimization is very simple, just to contract range of binary search.
> Initially, record all startIndex and endIndex of first or two bytes of keys.
> When a region key comes, find the contracted startIndex and endIndex of the
> key. Then return to normal binary search process with the specified
> startIndex and endIndex.
> Then we can ideally reduce cpu to 1/8 with 1 byte or 1/16 with 2 bytes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Reidddddd merged pull request #2065: HBASE-24739 [Build] branch-1's build seems broken because of pylint
Reidd merged pull request #2065: URL: https://github.com/apache/hbase/pull/2065 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24615) MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the distribution for last bucket.
[
https://issues.apache.org/jira/browse/HBASE-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158847#comment-17158847
]
Hudson commented on HBASE-24615:
Results for branch branch-2.3
[build #184 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/184/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/184/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/182/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/184/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/182/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the
> distribution for last bucket.
>
>
> Key: HBASE-24615
> URL: https://issues.apache.org/jira/browse/HBASE-24615
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.3.0, master, 1.3.7, 2.2.6
>Reporter: Rushabh Shah
>Assignee: wenfeiyi666
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.6
>
>
> We are not processing the distribution for last bucket.
> https://github.com/apache/hbase/blob/master/hbase-hadoop-compat/src/main/java/org/apache/hadoop/metrics2/lib/MutableRangeHistogram.java#L70
> {code:java}
> public void updateSnapshotRangeMetrics(MetricsRecordBuilder
> metricsRecordBuilder,
> Snapshot snapshot) {
> long priorRange = 0;
> long cumNum = 0;
> final long[] ranges = getRanges();
> final String rangeType = getRangeType();
> for (int i = 0; i < ranges.length - 1; i++) { -> The bug lies
> here. We are not processing last bucket.
> long val = snapshot.getCountAtOrBelow(ranges[i]);
> if (val - cumNum > 0) {
> metricsRecordBuilder.addCounter(
> Interns.info(name + "_" + rangeType + "_" + priorRange + "-" +
> ranges[i], desc),
> val - cumNum);
> }
> priorRange = ranges[i];
> cumNum = val;
> }
> long val = snapshot.getCount();
> if (val - cumNum > 0) {
> metricsRecordBuilder.addCounter(
> Interns.info(name + "_" + rangeType + "_" + ranges[ranges.length -
> 1] + "-inf", desc),
> val - cumNum);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24721) rename_rsgroup overwriting the existing rsgroup.
[
https://issues.apache.org/jira/browse/HBASE-24721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158848#comment-17158848
]
Hudson commented on HBASE-24721:
Results for branch branch-2.3
[build #184 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/184/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/184/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/182/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/184/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/182/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> rename_rsgroup overwriting the existing rsgroup.
>
>
> Key: HBASE-24721
> URL: https://issues.apache.org/jira/browse/HBASE-24721
> Project: HBase
> Issue Type: Bug
>Reporter: chiranjeevi
>Assignee: Mohammad Arshad
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.6
>
>
> rename_rsgroup overwriting the current rsgroup.
> Steps:
> 1)add_rsgroup 'RSG1' and 'RSG2'
> 2)move_servers_rsgroup 'RSG1',['server1:port']
> 3)rename_rsgroup 'RSG1','RSG2'
> After performing step3 RSG1 overwriting to RSG2 and region servers added in
> RSG1 are not available now.
> Ideally system should show error message Group already exists: RSG2
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24298) Reduce cpu load of locating region especially in batch mode.
[ https://issues.apache.org/jira/browse/HBASE-24298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] star updated HBASE-24298: - Attachment: HBASE-24298.patch > Reduce cpu load of locating region especially in batch mode. > > > Key: HBASE-24298 > URL: https://issues.apache.org/jira/browse/HBASE-24298 > Project: HBase > Issue Type: Bug >Reporter: star >Assignee: star >Priority: Major > Attachments: HBASE-24298.patch, locating region.svg > > > Binary search is used to speedup the process of locating region. It is > already fast enough, while cpu of HBASE client becomes the bottleneck when > doing TCSB benchmark. We can make the process of locating region faster to > reduce cpu load in some special cases , which however is our common case in > production environment. It is the case: > 1. Predefined splits in uniform distribution. > > 2. Load data in batch mode. > The optimization is very simple, just to contract range of binary search. > Initially, record all startIndex and endIndex of first or two bytes of keys. > When a region key comes, find the contracted startIndex and endIndex of the > key. Then return to normal binary search process with the specified > startIndex and endIndex. > Then we can ideally reduce cpu to 1/8 with 1 byte or 1/16 with 2 bytes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] bharathv commented on a change in pull request #2076: HBASE-24740 Enable journal logging for HBase snapshot operation
bharathv commented on a change in pull request #2076:
URL: https://github.com/apache/hbase/pull/2076#discussion_r455459886
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
##
@@ -624,14 +623,16 @@ private void takeSnapshotInternal(SnapshotDescription
snapshot) throws IOExcepti
AssignmentManager assignmentMgr = master.getAssignmentManager();
if (assignmentMgr.getTableStateManager().isTableState(snapshotTable,
ZooKeeperProtos.Table.State.ENABLED)) {
- LOG.debug("Table enabled, starting distributed snapshot.");
+ LOG.debug("Table enabled, starting distributed snapshot for "
Review comment:
branch-1 doesn't support parameterized logging, so wrap with
LOG.isDebugEnabled() checks? Saves string construction when debug logging isn't
enabled. (multiple places).
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java
##
@@ -343,6 +343,13 @@ private void load() throws IOException {
}
}
+ /**
+ * Sets the status task for monitoring all the subtasks for Snapshot
operation
+ */
+ public void setMonitoredTask(MonitoredTask statusTask) {
Review comment:
Any reason to have a separate setter instead of passing it via create() ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2071: HBASE-24743 Reject to add a peer which replicate to itself earlier
infraio commented on a change in pull request #2071:
URL: https://github.com/apache/hbase/pull/2071#discussion_r455463816
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationPeerManager.java
##
@@ -501,6 +511,32 @@ private void checkClusterKey(String clusterKey) throws
DoNotRetryIOException {
}
}
+ private void checkClusterId(String clusterKey) throws DoNotRetryIOException {
+String peerClusterId = "";
+try {
+ // Create the peer cluster config for get peer cluster id
+ Configuration peerConf = HBaseConfiguration.createClusterConf(conf,
clusterKey);
+ ZKWatcher zkWatcher =
+new ZKWatcher(peerConf, this + "check-peer-cluster-key", new
Abortable() {
+ @Override public void abort(String why, Throwable e) {
+ }
+
+ @Override public boolean isAborted() {
+return false;
+ }
+});
+ peerClusterId = ZKClusterId.readClusterIdZNode(zkWatcher);
+} catch (IOException | KeeperException e) {
+ throw new DoNotRetryIOException("Can't get peerClusterId for
clusterKey=" + clusterKey, e);
+}
+// In rare case, zookeeper setting may be messed up. That leads to the
incorrect
+// peerClusterId value, which is the same as the source clusterId
+if (clusterId.equals(peerClusterId)) {
Review comment:
The peerClusterId was not initialized in the constructor method of
ReplicationEndpoint. So ReplicationEndpoint#getPeerUUID() will get null here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2071: HBASE-24743 Reject to add a peer which replicate to itself earlier
infraio commented on a change in pull request #2071:
URL: https://github.com/apache/hbase/pull/2071#discussion_r455463882
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationPeerManager.java
##
@@ -501,6 +511,32 @@ private void checkClusterKey(String clusterKey) throws
DoNotRetryIOException {
}
}
+ private void checkClusterId(String clusterKey) throws DoNotRetryIOException {
+String peerClusterId = "";
+try {
+ // Create the peer cluster config for get peer cluster id
+ Configuration peerConf = HBaseConfiguration.createClusterConf(conf,
clusterKey);
+ ZKWatcher zkWatcher =
+new ZKWatcher(peerConf, this + "check-peer-cluster-key", new
Abortable() {
+ @Override public void abort(String why, Throwable e) {
+ }
+
+ @Override public boolean isAborted() {
+return false;
+ }
+});
+ peerClusterId = ZKClusterId.readClusterIdZNode(zkWatcher);
+} catch (IOException | KeeperException e) {
+ throw new DoNotRetryIOException("Can't get peerClusterId for
clusterKey=" + clusterKey, e);
+}
Review comment:
Good catch. Let me fix this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2071: HBASE-24743 Reject to add a peer which replicate to itself earlier
infraio commented on a change in pull request #2071:
URL: https://github.com/apache/hbase/pull/2071#discussion_r455463412
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationPeerManager.java
##
@@ -337,25 +347,21 @@ public void removeAllQueuesAndHFileRefs(String peerId)
throws ReplicationExcepti
private void checkPeerConfig(ReplicationPeerConfig peerConfig) throws
DoNotRetryIOException {
String replicationEndpointImpl = peerConfig.getReplicationEndpointImpl();
-boolean checkClusterKey = true;
+ReplicationEndpoint endpoint = null;
if (!StringUtils.isBlank(replicationEndpointImpl)) {
- // try creating a instance
- ReplicationEndpoint endpoint;
try {
+// try creating a instance
endpoint = Class.forName(replicationEndpointImpl)
.asSubclass(ReplicationEndpoint.class).getDeclaredConstructor().newInstance();
} catch (Throwable e) {
throw new DoNotRetryIOException(
"Can not instantiate configured replication endpoint class=" +
replicationEndpointImpl,
e);
}
- // do not check cluster key if we are not
HBaseInterClusterReplicationEndpoint
- if (!(endpoint instanceof HBaseInterClusterReplicationEndpoint)) {
-checkClusterKey = false;
- }
}
-if (checkClusterKey) {
- checkClusterKey(peerConfig.getClusterKey());
+// Default is HBaseInterClusterReplicationEndpoint and only it need to
check cluster key
+if (endpoint == null || endpoint instanceof
HBaseInterClusterReplicationEndpoint) {
Review comment:
For default case, it didn't new a ReplicationEndpoint. There are a check
!StringUtils.isBlank(replicationEndpointImpl).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2065: HBASE-24739 [Build] branch-1's build seems broken because of pylint
Apache-HBase commented on pull request #2065: URL: https://github.com/apache/hbase/pull/2065#issuecomment-659106884 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 52s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | hadolint | 0m 0s | hadolint was not available. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-1 Compile Tests _ | | +0 :ok: | mvndep | 2m 26s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for patch | | +1 :green_heart: | shellcheck | 0m 0s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? | | | | 10m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2065/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2065 | | Optional Tests | dupname asflicense hadolint shellcheck shelldocs | | uname | Linux 7f9abd6e7c0c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-2065/out/precommit/personality/provided.sh | | git revision | branch-1 / f5080eb | | Max. process+thread count | 41 (vs. ulimit of 1) | | modules | C: U: | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2065/4/console | | versions | git=1.9.1 maven=3.0.5 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #2064: HBASE-24735: Refactor ReplicationSourceManager: move logPositionAndCl…
bharathv commented on a change in pull request #2064:
URL: https://github.com/apache/hbase/pull/2064#discussion_r455459283
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
##
@@ -926,7 +799,8 @@ public void join() {
* @return list of all normal sources
*/
public List getSources() {
-return new ArrayList<>(this.sources.values());
+return this.sources.values().stream().filter(source ->
source.isSourceActive())
Review comment:
ok
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
##
@@ -139,17 +144,16 @@
* Instantiation method used by region servers
* @param conf configuration to use
* @param fs file system to use
- * @param manager replication manager to ping to
* @param server the server for this region server
* @param queueId the id of our replication queue
* @param clusterId unique UUID for the cluster
* @param metrics metrics for replication source
*/
@Override
- public void init(Configuration conf, FileSystem fs, ReplicationSourceManager
manager,
+ public void init(Configuration conf, FileSystem fs, Path walDir,
ReplicationSourceManager manager,
Review comment:
Oh, I see, didn't realize you didn't change the recovered source.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #2071: HBASE-24743 Reject to add a peer which replicate to itself earlier
bharathv commented on a change in pull request #2071:
URL: https://github.com/apache/hbase/pull/2071#discussion_r455457366
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationPeerManager.java
##
@@ -337,26 +347,26 @@ public void removeAllQueuesAndHFileRefs(String peerId)
throws ReplicationExcepti
private void checkPeerConfig(ReplicationPeerConfig peerConfig) throws
DoNotRetryIOException {
String replicationEndpointImpl = peerConfig.getReplicationEndpointImpl();
-boolean checkClusterKey = true;
+ReplicationEndpoint endpoint = null;
if (!StringUtils.isBlank(replicationEndpointImpl)) {
- // try creating a instance
- ReplicationEndpoint endpoint;
try {
+// try creating a instance
endpoint = Class.forName(replicationEndpointImpl)
.asSubclass(ReplicationEndpoint.class).getDeclaredConstructor().newInstance();
} catch (Throwable e) {
throw new DoNotRetryIOException(
"Can not instantiate configured replication endpoint class=" +
replicationEndpointImpl,
e);
}
- // do not check cluster key if we are not
HBaseInterClusterReplicationEndpoint
- if (!(endpoint instanceof HBaseInterClusterReplicationEndpoint)) {
-checkClusterKey = false;
- }
}
-if (checkClusterKey) {
+// Default is HBaseInterClusterReplicationEndpoint and only it need to
check cluster key
+if (endpoint == null || endpoint instanceof
HBaseInterClusterReplicationEndpoint) {
checkClusterKey(peerConfig.getClusterKey());
}
+// Default is HBaseInterClusterReplicationEndpoint which cannot replicate
to same cluster
+if (endpoint == null || !endpoint.canReplicateToSameCluster()) {
Review comment:
nit: same question as above..null check on endpoint.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationPeerManager.java
##
@@ -493,12 +499,35 @@ private void
checkConfiguredWALEntryFilters(ReplicationPeerConfig peerConfig)
}
}
- private void checkClusterKey(String clusterKey) throws DoNotRetryIOException
{
+ private void checkClusterKeyForHBaseInterClusterReplicationEndpoint(String
clusterKey)
+throws DoNotRetryIOException {
try {
ZKConfig.validateClusterKey(clusterKey);
} catch (IOException e) {
throw new DoNotRetryIOException("Invalid cluster key: " + clusterKey, e);
}
+String peerClusterId = "";
Review comment:
oh sorry my bad, I misread the code.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationPeerManager.java
##
@@ -337,26 +347,26 @@ public void removeAllQueuesAndHFileRefs(String peerId)
throws ReplicationExcepti
private void checkPeerConfig(ReplicationPeerConfig peerConfig) throws
DoNotRetryIOException {
String replicationEndpointImpl = peerConfig.getReplicationEndpointImpl();
-boolean checkClusterKey = true;
+ReplicationEndpoint endpoint = null;
Review comment:
nit = null is redundant.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationPeerManager.java
##
@@ -501,6 +511,32 @@ private void checkClusterKey(String clusterKey) throws
DoNotRetryIOException {
}
}
+ private void checkClusterId(String clusterKey) throws DoNotRetryIOException {
+String peerClusterId = "";
+try {
+ // Create the peer cluster config for get peer cluster id
+ Configuration peerConf = HBaseConfiguration.createClusterConf(conf,
clusterKey);
+ ZKWatcher zkWatcher =
+new ZKWatcher(peerConf, this + "check-peer-cluster-key", new
Abortable() {
+ @Override public void abort(String why, Throwable e) {
+ }
+
+ @Override public boolean isAborted() {
+return false;
+ }
+});
+ peerClusterId = ZKClusterId.readClusterIdZNode(zkWatcher);
+} catch (IOException | KeeperException e) {
+ throw new DoNotRetryIOException("Can't get peerClusterId for
clusterKey=" + clusterKey, e);
+}
Review comment:
We want to close() the zkWatcher? Otherwise there is a connection leak?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationPeerManager.java
##
@@ -501,6 +511,32 @@ private void checkClusterKey(String clusterKey) throws
DoNotRetryIOException {
}
}
+ private void checkClusterId(String clusterKey) throws DoNotRetryIOException {
+String peerClusterId = "";
+try {
+ // Create the peer cluster config for get peer cluster id
+ Configuration peerConf = HBaseConfiguration.createClusterConf(conf,
clusterKey);
+ ZKWatcher zkWatcher =
+new ZKWatcher(peerConf, this + "check-peer-cluster-key", new
Abortable() {
+ @Override public void abort(String why, Throwable e) {
+
[GitHub] [hbase] Apache-HBase commented on pull request #2065: HBASE-24739 [Build] branch-1's build seems broken because of pylint
Apache-HBase commented on pull request #2065: URL: https://github.com/apache/hbase/pull/2065#issuecomment-659104362 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2065/4/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24739) [Build] branch-1's build seems broken because of pylint
[ https://issues.apache.org/jira/browse/HBASE-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-24739: -- Priority: Blocker (was: Major) > [Build] branch-1's build seems broken because of pylint > --- > > Key: HBASE-24739 > URL: https://issues.apache.org/jira/browse/HBASE-24739 > Project: HBase > Issue Type: Bug >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24721) rename_rsgroup overwriting the existing rsgroup.
[ https://issues.apache.org/jira/browse/HBASE-24721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158817#comment-17158817 ] Guanghao Zhang commented on HBASE-24721: +1 for branch-2.2. Let me cherry-pick it. > rename_rsgroup overwriting the existing rsgroup. > > > Key: HBASE-24721 > URL: https://issues.apache.org/jira/browse/HBASE-24721 > Project: HBase > Issue Type: Bug >Reporter: chiranjeevi >Assignee: Mohammad Arshad >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0 > > > rename_rsgroup overwriting the current rsgroup. > Steps: > 1)add_rsgroup 'RSG1' and 'RSG2' > 2)move_servers_rsgroup 'RSG1',['server1:port'] > 3)rename_rsgroup 'RSG1','RSG2' > After performing step3 RSG1 overwriting to RSG2 and region servers added in > RSG1 are not available now. > Ideally system should show error message Group already exists: RSG2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24721) rename_rsgroup overwriting the existing rsgroup.
[ https://issues.apache.org/jira/browse/HBASE-24721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-24721: --- Fix Version/s: 2.2.6 > rename_rsgroup overwriting the existing rsgroup. > > > Key: HBASE-24721 > URL: https://issues.apache.org/jira/browse/HBASE-24721 > Project: HBase > Issue Type: Bug >Reporter: chiranjeevi >Assignee: Mohammad Arshad >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.6 > > > rename_rsgroup overwriting the current rsgroup. > Steps: > 1)add_rsgroup 'RSG1' and 'RSG2' > 2)move_servers_rsgroup 'RSG1',['server1:port'] > 3)rename_rsgroup 'RSG1','RSG2' > After performing step3 RSG1 overwriting to RSG2 and region servers added in > RSG1 are not available now. > Ideally system should show error message Group already exists: RSG2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] infraio commented on a change in pull request #2064: HBASE-24735: Refactor ReplicationSourceManager: move logPositionAndCl…
infraio commented on a change in pull request #2064:
URL: https://github.com/apache/hbase/pull/2064#discussion_r455448586
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
##
@@ -926,7 +799,8 @@ public void join() {
* @return list of all normal sources
*/
public List getSources() {
-return new ArrayList<>(this.sources.values());
+return this.sources.values().stream().filter(source ->
source.isSourceActive())
Review comment:
This is for the ut TestMasterReplication#testLoopedReplication. Will
remove this after HBASE-24743.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2064: HBASE-24735: Refactor ReplicationSourceManager: move logPositionAndCl…
infraio commented on a change in pull request #2064: URL: https://github.com/apache/hbase/pull/2064#discussion_r455448274 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java ## @@ -139,17 +144,16 @@ * Instantiation method used by region servers * @param conf configuration to use * @param fs file system to use - * @param manager replication manager to ping to * @param server the server for this region server * @param queueId the id of our replication queue * @param clusterId unique UUID for the cluster * @param metrics metrics for replication source */ @Override - public void init(Configuration conf, FileSystem fs, ReplicationSourceManager manager, + public void init(Configuration conf, FileSystem fs, Path walDir, ReplicationSourceManager manager, Review comment: There are one typre ReplicationSource which called RecoveredReplicationSource, which replicate the WAL queues from a dead RegionServer. For RecoveredReplicationSource, the replication stream will finished when all WALs was replicated because the dead server cannot generate new WAL. Here need manager because RecoveredReplicationSource need to finish itself and clear the related reference from ReplicationSourceManager. When move to ReplicationServer future, need a ReplicationServer instead of this manager. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24615) MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the distribution for last bucket.
[
https://issues.apache.org/jira/browse/HBASE-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-24615:
---
Fix Version/s: 2.2.6
> MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the
> distribution for last bucket.
>
>
> Key: HBASE-24615
> URL: https://issues.apache.org/jira/browse/HBASE-24615
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.3.0, master, 1.3.7, 2.2.6
>Reporter: Rushabh Shah
>Assignee: wenfeiyi666
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.6
>
>
> We are not processing the distribution for last bucket.
> https://github.com/apache/hbase/blob/master/hbase-hadoop-compat/src/main/java/org/apache/hadoop/metrics2/lib/MutableRangeHistogram.java#L70
> {code:java}
> public void updateSnapshotRangeMetrics(MetricsRecordBuilder
> metricsRecordBuilder,
> Snapshot snapshot) {
> long priorRange = 0;
> long cumNum = 0;
> final long[] ranges = getRanges();
> final String rangeType = getRangeType();
> for (int i = 0; i < ranges.length - 1; i++) { -> The bug lies
> here. We are not processing last bucket.
> long val = snapshot.getCountAtOrBelow(ranges[i]);
> if (val - cumNum > 0) {
> metricsRecordBuilder.addCounter(
> Interns.info(name + "_" + rangeType + "_" + priorRange + "-" +
> ranges[i], desc),
> val - cumNum);
> }
> priorRange = ranges[i];
> cumNum = val;
> }
> long val = snapshot.getCount();
> if (val - cumNum > 0) {
> metricsRecordBuilder.addCounter(
> Interns.info(name + "_" + rangeType + "_" + ranges[ranges.length -
> 1] + "-inf", desc),
> val - cumNum);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2071: HBASE-24743 Reject to add a peer which replicate to itself earlier
Apache-HBase commented on pull request #2071: URL: https://github.com/apache/hbase/pull/2071#issuecomment-659090238 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 8s | master passed | | +1 :green_heart: | checkstyle | 1m 15s | master passed | | +1 :green_heart: | spotbugs | 2m 9s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 45s | the patch passed | | +1 :green_heart: | checkstyle | 1m 13s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 17s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 36m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2071/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2071 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux e7d783199938 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2505c7760d | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2071/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2076: HBASE-24740 Enable journal logging for HBase snapshot operation
Apache-HBase commented on pull request #2076: URL: https://github.com/apache/hbase/pull/2076#issuecomment-659089050 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | docker | 0m 7s | Docker failed to build yetus/hbase:7cf97016b9. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/2076 | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2076/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] sguggilam commented on pull request #2076: HBASE-24740 Enable journal logging for HBase snapshot operation
sguggilam commented on pull request #2076: URL: https://github.com/apache/hbase/pull/2076#issuecomment-659087056 @apurtell @bharathv @virajjasani Can you please review ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] sguggilam commented on pull request #2076: HBASE-24740 Enable journal logging for HBase snapshot operation
sguggilam commented on pull request #2076: URL: https://github.com/apache/hbase/pull/2076#issuecomment-659086947 Sample Journal entry: Table snapshot journal : Running FLUSH table snapshot test_snapshot C_M_SNAPSHOT_TABLE on table test at 1594859523584 Consolidate snapshot: test_snapshot at 1594859523691 (+107 ms) Loading Region manifests for test_snapshot at 1594859523692 (+1 ms) Writing data manifest for test_snapshot at 1594859523693 (+1 ms) Verifying snapshot: test_snapshot at 1594859523706 (+13 ms) Snapshot test_snapshot of table test completed at 1594859523710 (+4 ms) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24615) MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the distribution for last bucket.
[
https://issues.apache.org/jira/browse/HBASE-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158810#comment-17158810
]
Guanghao Zhang commented on HBASE-24615:
{quote}The snippet you shared is for overflow bucket (for the count that falls
outside of ranges[length-1])
In this jira we fixed the case where we are not updating the count between the
range: ranges[length-2] and ranges[length-1]
{quote}
Got it. +1 for branch-2.2. Let me cherry-pick it.
> MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the
> distribution for last bucket.
>
>
> Key: HBASE-24615
> URL: https://issues.apache.org/jira/browse/HBASE-24615
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.3.0, master, 1.3.7, 2.2.6
>Reporter: Rushabh Shah
>Assignee: wenfeiyi666
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0
>
>
> We are not processing the distribution for last bucket.
> https://github.com/apache/hbase/blob/master/hbase-hadoop-compat/src/main/java/org/apache/hadoop/metrics2/lib/MutableRangeHistogram.java#L70
> {code:java}
> public void updateSnapshotRangeMetrics(MetricsRecordBuilder
> metricsRecordBuilder,
> Snapshot snapshot) {
> long priorRange = 0;
> long cumNum = 0;
> final long[] ranges = getRanges();
> final String rangeType = getRangeType();
> for (int i = 0; i < ranges.length - 1; i++) { -> The bug lies
> here. We are not processing last bucket.
> long val = snapshot.getCountAtOrBelow(ranges[i]);
> if (val - cumNum > 0) {
> metricsRecordBuilder.addCounter(
> Interns.info(name + "_" + rangeType + "_" + priorRange + "-" +
> ranges[i], desc),
> val - cumNum);
> }
> priorRange = ranges[i];
> cumNum = val;
> }
> long val = snapshot.getCount();
> if (val - cumNum > 0) {
> metricsRecordBuilder.addCounter(
> Interns.info(name + "_" + rangeType + "_" + ranges[ranges.length -
> 1] + "-inf", desc),
> val - cumNum);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] sguggilam opened a new pull request #2076: HBASE-24740 Enable journal logging for HBase snapshot operation
sguggilam opened a new pull request #2076: URL: https://github.com/apache/hbase/pull/2076 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24615) MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the distribution for last bucket.
[
https://issues.apache.org/jira/browse/HBASE-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158807#comment-17158807
]
Rushabh Shah commented on HBASE-24615:
--
[~zghao] The snippet you shared is for overflow bucket (for the count that
falls outside of ranges[length-1])
In this jira we fixed the case where we are not updating the count between the
range: ranges[length-2] and ranges[length-1]
Hope it makes sense.
> MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the
> distribution for last bucket.
>
>
> Key: HBASE-24615
> URL: https://issues.apache.org/jira/browse/HBASE-24615
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.3.0, master, 1.3.7, 2.2.6
>Reporter: Rushabh Shah
>Assignee: wenfeiyi666
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0
>
>
> We are not processing the distribution for last bucket.
> https://github.com/apache/hbase/blob/master/hbase-hadoop-compat/src/main/java/org/apache/hadoop/metrics2/lib/MutableRangeHistogram.java#L70
> {code:java}
> public void updateSnapshotRangeMetrics(MetricsRecordBuilder
> metricsRecordBuilder,
> Snapshot snapshot) {
> long priorRange = 0;
> long cumNum = 0;
> final long[] ranges = getRanges();
> final String rangeType = getRangeType();
> for (int i = 0; i < ranges.length - 1; i++) { -> The bug lies
> here. We are not processing last bucket.
> long val = snapshot.getCountAtOrBelow(ranges[i]);
> if (val - cumNum > 0) {
> metricsRecordBuilder.addCounter(
> Interns.info(name + "_" + rangeType + "_" + priorRange + "-" +
> ranges[i], desc),
> val - cumNum);
> }
> priorRange = ranges[i];
> cumNum = val;
> }
> long val = snapshot.getCount();
> if (val - cumNum > 0) {
> metricsRecordBuilder.addCounter(
> Interns.info(name + "_" + rangeType + "_" + ranges[ranges.length -
> 1] + "-inf", desc),
> val - cumNum);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24615) MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the distribution for last bucket.
[
https://issues.apache.org/jira/browse/HBASE-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158799#comment-17158799
]
Guanghao Zhang commented on HBASE-24615:
{code:java}
long val = snapshot.getCount();
if (val - cumNum > 0) {
metricsRecordBuilder.addCounter(
Interns.info(name + "_" + rangeType + "_" + ranges[ranges.length - 1]
+ "-inf", desc),
val - cumNum);
}
{code}
I am not fully understand the fix. I thought the last bucket was handled in
previous code, too?
> MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the
> distribution for last bucket.
>
>
> Key: HBASE-24615
> URL: https://issues.apache.org/jira/browse/HBASE-24615
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.3.0, master, 1.3.7, 2.2.6
>Reporter: Rushabh Shah
>Assignee: wenfeiyi666
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0
>
>
> We are not processing the distribution for last bucket.
> https://github.com/apache/hbase/blob/master/hbase-hadoop-compat/src/main/java/org/apache/hadoop/metrics2/lib/MutableRangeHistogram.java#L70
> {code:java}
> public void updateSnapshotRangeMetrics(MetricsRecordBuilder
> metricsRecordBuilder,
> Snapshot snapshot) {
> long priorRange = 0;
> long cumNum = 0;
> final long[] ranges = getRanges();
> final String rangeType = getRangeType();
> for (int i = 0; i < ranges.length - 1; i++) { -> The bug lies
> here. We are not processing last bucket.
> long val = snapshot.getCountAtOrBelow(ranges[i]);
> if (val - cumNum > 0) {
> metricsRecordBuilder.addCounter(
> Interns.info(name + "_" + rangeType + "_" + priorRange + "-" +
> ranges[i], desc),
> val - cumNum);
> }
> priorRange = ranges[i];
> cumNum = val;
> }
> long val = snapshot.getCount();
> if (val - cumNum > 0) {
> metricsRecordBuilder.addCounter(
> Interns.info(name + "_" + rangeType + "_" + ranges[ranges.length -
> 1] + "-inf", desc),
> val - cumNum);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24705) MetaFixer#fixHoles() does not include the case for read replicas (i.e, replica regions are not created)
[
https://issues.apache.org/jira/browse/HBASE-24705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158795#comment-17158795
]
Hudson commented on HBASE-24705:
Results for branch branch-2
[build #2746 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2746/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2746/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2746/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2746/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2746/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> MetaFixer#fixHoles() does not include the case for read replicas (i.e,
> replica regions are not created)
> ---
>
> Key: HBASE-24705
> URL: https://issues.apache.org/jira/browse/HBASE-24705
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 2.3.0
>Reporter: Huaxiang Sun
>Assignee: Huaxiang Sun
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24721) rename_rsgroup overwriting the existing rsgroup.
[
https://issues.apache.org/jira/browse/HBASE-24721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158797#comment-17158797
]
Hudson commented on HBASE-24721:
Results for branch branch-2
[build #2746 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2746/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2746/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2746/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2746/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2746/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> rename_rsgroup overwriting the existing rsgroup.
>
>
> Key: HBASE-24721
> URL: https://issues.apache.org/jira/browse/HBASE-24721
> Project: HBase
> Issue Type: Bug
>Reporter: chiranjeevi
>Assignee: Mohammad Arshad
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0
>
>
> rename_rsgroup overwriting the current rsgroup.
> Steps:
> 1)add_rsgroup 'RSG1' and 'RSG2'
> 2)move_servers_rsgroup 'RSG1',['server1:port']
> 3)rename_rsgroup 'RSG1','RSG2'
> After performing step3 RSG1 overwriting to RSG2 and region servers added in
> RSG1 are not available now.
> Ideally system should show error message Group already exists: RSG2
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24615) MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the distribution for last bucket.
[
https://issues.apache.org/jira/browse/HBASE-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158796#comment-17158796
]
Hudson commented on HBASE-24615:
Results for branch branch-2
[build #2746 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2746/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2746/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2746/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2746/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2746/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the
> distribution for last bucket.
>
>
> Key: HBASE-24615
> URL: https://issues.apache.org/jira/browse/HBASE-24615
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.3.0, master, 1.3.7, 2.2.6
>Reporter: Rushabh Shah
>Assignee: wenfeiyi666
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0
>
>
> We are not processing the distribution for last bucket.
> https://github.com/apache/hbase/blob/master/hbase-hadoop-compat/src/main/java/org/apache/hadoop/metrics2/lib/MutableRangeHistogram.java#L70
> {code:java}
> public void updateSnapshotRangeMetrics(MetricsRecordBuilder
> metricsRecordBuilder,
> Snapshot snapshot) {
> long priorRange = 0;
> long cumNum = 0;
> final long[] ranges = getRanges();
> final String rangeType = getRangeType();
> for (int i = 0; i < ranges.length - 1; i++) { -> The bug lies
> here. We are not processing last bucket.
> long val = snapshot.getCountAtOrBelow(ranges[i]);
> if (val - cumNum > 0) {
> metricsRecordBuilder.addCounter(
> Interns.info(name + "_" + rangeType + "_" + priorRange + "-" +
> ranges[i], desc),
> val - cumNum);
> }
> priorRange = ranges[i];
> cumNum = val;
> }
> long val = snapshot.getCount();
> if (val - cumNum > 0) {
> metricsRecordBuilder.addCounter(
> Interns.info(name + "_" + rangeType + "_" + ranges[ranges.length -
> 1] + "-inf", desc),
> val - cumNum);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] infraio commented on a change in pull request #2071: HBASE-24743 Reject to add a peer which replicate to itself earlier
infraio commented on a change in pull request #2071:
URL: https://github.com/apache/hbase/pull/2071#discussion_r455434266
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationPeerManager.java
##
@@ -493,12 +499,35 @@ private void
checkConfiguredWALEntryFilters(ReplicationPeerConfig peerConfig)
}
}
- private void checkClusterKey(String clusterKey) throws DoNotRetryIOException
{
+ private void checkClusterKeyForHBaseInterClusterReplicationEndpoint(String
clusterKey)
+throws DoNotRetryIOException {
try {
ZKConfig.validateClusterKey(clusterKey);
} catch (IOException e) {
throw new DoNotRetryIOException("Invalid cluster key: " + clusterKey, e);
}
+String peerClusterId = "";
+try {
+ Configuration otherConf = HBaseConfiguration.createClusterConf(conf,
clusterKey);
+ ZKWatcher zkWatcher =
+new ZKWatcher(otherConf, this + "check-peer-cluster-key", new
Abortable() {
+ @Override public void abort(String why, Throwable e) {
+ }
+
+ @Override public boolean isAborted() {
+return false;
+ }
+});
+ peerClusterId = ZKClusterId.readClusterIdZNode(zkWatcher);
Review comment:
See the changed code in
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1935: HBASE-22146 SpaceQuotaViolationPolicy Disable is not working in Names…
Apache-HBase commented on pull request #1935: URL: https://github.com/apache/hbase/pull/1935#issuecomment-659077939 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 19s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 3s | master passed | | +1 :green_heart: | checkstyle | 1m 38s | master passed | | +1 :green_heart: | spotbugs | 3m 9s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 45s | the patch passed | | -0 :warning: | checkstyle | 1m 11s | hbase-server: The patch generated 1 new + 6 unchanged - 0 fixed = 7 total (was 6) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 28s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 3m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 40m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1935/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1935 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 13c674ccd6f9 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2505c7760d | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1935/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1935/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2075: HBASE-24742 Improve performance of SKIP vs SEEK logic.
Apache-HBase commented on pull request #2075: URL: https://github.com/apache/hbase/pull/2075#issuecomment-659077993 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | docker | 0m 9s | Docker failed to build yetus/hbase:7cf97016b9. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/2075 | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2075/2/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] lhofhansl edited a comment on pull request #2075: HBASE-24742 Improve performance of SKIP vs SEEK logic.
lhofhansl edited a comment on pull request #2075: URL: https://github.com/apache/hbase/pull/2075#issuecomment-659076398 In unit tests StoreScanner.store might be null. Fixed. Thanks @bharathv for pointing that out. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] lhofhansl commented on pull request #2075: HBASE-24742 Improve performance of SKIP vs SEEK logic.
lhofhansl commented on pull request #2075: URL: https://github.com/apache/hbase/pull/2075#issuecomment-659076398 In unit tests StoreScanner.store might be null. Fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2071: HBASE-24743 Reject to add a peer which replicate to itself earlier
infraio commented on a change in pull request #2071:
URL: https://github.com/apache/hbase/pull/2071#discussion_r455430870
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationPeerManager.java
##
@@ -337,25 +347,21 @@ public void removeAllQueuesAndHFileRefs(String peerId)
throws ReplicationExcepti
private void checkPeerConfig(ReplicationPeerConfig peerConfig) throws
DoNotRetryIOException {
String replicationEndpointImpl = peerConfig.getReplicationEndpointImpl();
-boolean checkClusterKey = true;
+ReplicationEndpoint endpoint = null;
if (!StringUtils.isBlank(replicationEndpointImpl)) {
- // try creating a instance
- ReplicationEndpoint endpoint;
try {
+// try creating a instance
endpoint = Class.forName(replicationEndpointImpl)
.asSubclass(ReplicationEndpoint.class).getDeclaredConstructor().newInstance();
} catch (Throwable e) {
throw new DoNotRetryIOException(
"Can not instantiate configured replication endpoint class=" +
replicationEndpointImpl,
e);
}
- // do not check cluster key if we are not
HBaseInterClusterReplicationEndpoint
- if (!(endpoint instanceof HBaseInterClusterReplicationEndpoint)) {
-checkClusterKey = false;
- }
}
-if (checkClusterKey) {
- checkClusterKey(peerConfig.getClusterKey());
+// Default is HBaseInterClusterReplicationEndpoint and only it need to
check cluster key
+if (endpoint == null || endpoint instanceof
HBaseInterClusterReplicationEndpoint) {
Review comment:
canReplicateToSameCluster() was added onlt for
HBaseInterClusterReplicationEndpoint. But checked the code again, it was added
to ReplicationEndpoint interface now and it is IA.Replication. So maybe
override by user's plugglable ReplicationEndpoiint. Let me update the code to
use canReplicateToSameCluster() again.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2071: HBASE-24743 Reject to add a peer which replicate to itself earlier
infraio commented on a change in pull request #2071:
URL: https://github.com/apache/hbase/pull/2071#discussion_r455427724
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationPeerManager.java
##
@@ -493,12 +499,35 @@ private void
checkConfiguredWALEntryFilters(ReplicationPeerConfig peerConfig)
}
}
- private void checkClusterKey(String clusterKey) throws DoNotRetryIOException
{
+ private void checkClusterKeyForHBaseInterClusterReplicationEndpoint(String
clusterKey)
+throws DoNotRetryIOException {
try {
ZKConfig.validateClusterKey(clusterKey);
} catch (IOException e) {
throw new DoNotRetryIOException("Invalid cluster key: " + clusterKey, e);
}
+String peerClusterId = "";
Review comment:
No, this fetch the peer cluster id.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2071: HBASE-24743 Reject to add a peer which replicate to itself earlier
infraio commented on a change in pull request #2071:
URL: https://github.com/apache/hbase/pull/2071#discussion_r455427867
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationPeerManager.java
##
@@ -493,12 +499,35 @@ private void
checkConfiguredWALEntryFilters(ReplicationPeerConfig peerConfig)
}
}
- private void checkClusterKey(String clusterKey) throws DoNotRetryIOException
{
+ private void checkClusterKeyForHBaseInterClusterReplicationEndpoint(String
clusterKey)
+throws DoNotRetryIOException {
try {
ZKConfig.validateClusterKey(clusterKey);
} catch (IOException e) {
throw new DoNotRetryIOException("Invalid cluster key: " + clusterKey, e);
}
+String peerClusterId = "";
+try {
+ Configuration otherConf = HBaseConfiguration.createClusterConf(conf,
clusterKey);
+ ZKWatcher zkWatcher =
+new ZKWatcher(otherConf, this + "check-peer-cluster-key", new
Abortable() {
+ @Override public void abort(String why, Throwable e) {
+ }
+
+ @Override public boolean isAborted() {
+return false;
+ }
+});
+ peerClusterId = ZKClusterId.readClusterIdZNode(zkWatcher);
Review comment:
Not right. This fetch peer cluster id.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-24689) Generate CHANGES.md and RELEASENOTES.md for 2.2.6
[
https://issues.apache.org/jira/browse/HBASE-24689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158787#comment-17158787
]
Hudson commented on HBASE-24689:
Results for branch branch-2.2
[build #913 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/913/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/913//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/913//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/913//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Generate CHANGES.md and RELEASENOTES.md for 2.2.6
> -
>
> Key: HBASE-24689
> URL: https://issues.apache.org/jira/browse/HBASE-24689
> Project: HBase
> Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.2.6
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24690) Set version to 2.2.6 in branch-2.2 for first RC of 2.2.6
[
https://issues.apache.org/jira/browse/HBASE-24690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158788#comment-17158788
]
Hudson commented on HBASE-24690:
Results for branch branch-2.2
[build #913 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/913/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/913//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/913//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/913//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Set version to 2.2.6 in branch-2.2 for first RC of 2.2.6
>
>
> Key: HBASE-24690
> URL: https://issues.apache.org/jira/browse/HBASE-24690
> Project: HBase
> Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] bharathv commented on a change in pull request #2064: HBASE-24735: Refactor ReplicationSourceManager: move logPositionAndCl…
bharathv commented on a change in pull request #2064:
URL: https://github.com/apache/hbase/pull/2064#discussion_r455417117
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
##
@@ -192,4 +188,8 @@ default boolean isSyncReplication() {
default boolean isRecovered() {
return false;
}
+
+ void logPosition(WALEntryBatch entryBatch);
Review comment:
nit: method javadoc
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
##
@@ -192,4 +188,8 @@ default boolean isSyncReplication() {
default boolean isRecovered() {
return false;
}
+
+ void logPosition(WALEntryBatch entryBatch);
+
+ void cleanOldLogs(String walName, boolean inclusive);
Review comment:
nit: method javadoc
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
##
@@ -729,4 +728,143 @@ void removeWorker(ReplicationSourceShipper worker) {
private String logPeerId(){
return "[Source for peer " + this.getPeer().getId() + "]:";
}
+
+ @VisibleForTesting
+ public void logPosition(WALEntryBatch entryBatch) {
+String fileName = entryBatch.getLastWalPath().getName();
+interruptOrAbortWhenFail(() -> this.queueStorage
+ .setWALPosition(server.getServerName(), getQueueId(), fileName,
+entryBatch.getLastWalPosition(), entryBatch.getLastSeqIds()));
+ }
+
+ /**
Review comment:
move this to the interface?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
##
@@ -139,17 +144,16 @@
* Instantiation method used by region servers
* @param conf configuration to use
* @param fs file system to use
- * @param manager replication manager to ping to
* @param server the server for this region server
* @param queueId the id of our replication queue
* @param clusterId unique UUID for the cluster
* @param metrics metrics for replication source
*/
@Override
- public void init(Configuration conf, FileSystem fs, ReplicationSourceManager
manager,
+ public void init(Configuration conf, FileSystem fs, Path walDir,
ReplicationSourceManager manager,
Review comment:
You don't need the manager in the c'tor anymore?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
##
@@ -926,7 +799,8 @@ public void join() {
* @return list of all normal sources
*/
public List getSources() {
-return new ArrayList<>(this.sources.values());
+return this.sources.values().stream().filter(source ->
source.isSourceActive())
Review comment:
curious why this change?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryBatch.java
##
@@ -30,8 +30,7 @@
/**
* Holds a batch of WAL entries to replicate, along with some statistics
*/
-@InterfaceAudience.Private
-class WALEntryBatch {
+@InterfaceAudience.Private public class WALEntryBatch {
Review comment:
I think this is not needed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] lhofhansl commented on pull request #2065: HBASE-24739 [Build] branch-1's build seems broken because of pylint
lhofhansl commented on pull request #2065: URL: https://github.com/apache/hbase/pull/2065#issuecomment-659059419 Let's get it in? It's holding up other changes for branch-1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #2071: HBASE-24743 Reject to add a peer which replicate to itself earlier
bharathv commented on a change in pull request #2071:
URL: https://github.com/apache/hbase/pull/2071#discussion_r455409660
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationPeerManager.java
##
@@ -493,12 +499,35 @@ private void
checkConfiguredWALEntryFilters(ReplicationPeerConfig peerConfig)
}
}
- private void checkClusterKey(String clusterKey) throws DoNotRetryIOException
{
+ private void checkClusterKeyForHBaseInterClusterReplicationEndpoint(String
clusterKey)
+throws DoNotRetryIOException {
try {
ZKConfig.validateClusterKey(clusterKey);
} catch (IOException e) {
throw new DoNotRetryIOException("Invalid cluster key: " + clusterKey, e);
}
+String peerClusterId = "";
+try {
+ Configuration otherConf = HBaseConfiguration.createClusterConf(conf,
clusterKey);
+ ZKWatcher zkWatcher =
+new ZKWatcher(otherConf, this + "check-peer-cluster-key", new
Abortable() {
+ @Override public void abort(String why, Throwable e) {
+ }
+
+ @Override public boolean isAborted() {
+return false;
+ }
+});
+ peerClusterId = ZKClusterId.readClusterIdZNode(zkWatcher);
Review comment:
ReplicationPeerManager is colocated with master, right? you can use
master.getClusterId() without doing all this? (that is cached, so saves a ZK
round trip).
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationPeerManager.java
##
@@ -493,12 +499,35 @@ private void
checkConfiguredWALEntryFilters(ReplicationPeerConfig peerConfig)
}
}
- private void checkClusterKey(String clusterKey) throws DoNotRetryIOException
{
+ private void checkClusterKeyForHBaseInterClusterReplicationEndpoint(String
clusterKey)
+throws DoNotRetryIOException {
try {
ZKConfig.validateClusterKey(clusterKey);
} catch (IOException e) {
throw new DoNotRetryIOException("Invalid cluster key: " + clusterKey, e);
}
+String peerClusterId = "";
Review comment:
You are fetching the currentClusterId right? Why name it as
peerClusterId? I think the argument to this method is peerClusterKey.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationPeerManager.java
##
@@ -337,25 +347,21 @@ public void removeAllQueuesAndHFileRefs(String peerId)
throws ReplicationExcepti
private void checkPeerConfig(ReplicationPeerConfig peerConfig) throws
DoNotRetryIOException {
String replicationEndpointImpl = peerConfig.getReplicationEndpointImpl();
-boolean checkClusterKey = true;
+ReplicationEndpoint endpoint = null;
if (!StringUtils.isBlank(replicationEndpointImpl)) {
- // try creating a instance
- ReplicationEndpoint endpoint;
try {
+// try creating a instance
endpoint = Class.forName(replicationEndpointImpl)
.asSubclass(ReplicationEndpoint.class).getDeclaredConstructor().newInstance();
} catch (Throwable e) {
throw new DoNotRetryIOException(
"Can not instantiate configured replication endpoint class=" +
replicationEndpointImpl,
e);
}
- // do not check cluster key if we are not
HBaseInterClusterReplicationEndpoint
- if (!(endpoint instanceof HBaseInterClusterReplicationEndpoint)) {
-checkClusterKey = false;
- }
}
-if (checkClusterKey) {
- checkClusterKey(peerConfig.getClusterKey());
+// Default is HBaseInterClusterReplicationEndpoint and only it need to
check cluster key
+if (endpoint == null || endpoint instanceof
HBaseInterClusterReplicationEndpoint) {
Review comment:
Replication#canReplicateToSameCluster() isn't it supposed to do this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158755#comment-17158755
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #16 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/16/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/16/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/16/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/16/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] bharathv commented on pull request #2075: HBASE-24742 Improve performance of SKIP vs SEEK logic.
bharathv commented on pull request #2075: URL: https://github.com/apache/hbase/pull/2075#issuecomment-65804 > Just a question: The speed is inline with 1.3? Its within 5-10% bounds based on my preliminary analysis (on my local box, **if this optimization kicks in**). Will have to do a detailed cluster based perf run to dig deeper. As discussed in the jira, there is scope for further optimization for general use cases (specifically when versions=1, arbitrary bloom filters types etc) but that might need a bigger refactor of code. My point being that we cannot generally compare with 1.3 until we fix those issues too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24115) Relocate test-only REST "client" from src/ to test/ and mark Private
[ https://issues.apache.org/jira/browse/HBASE-24115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158646#comment-17158646 ] Pankaj Kumar commented on HBASE-24115: -- [~apurtell] Sorry for coming late on this, I thought these are REST java client interfaces to perform create/delete table and put/get/Scan operations. We have some use cases too where applications are using RemoteAdmin/RemoteHTable APIs. Pardon me, I might have missed the background of these interfaces. > Relocate test-only REST "client" from src/ to test/ and mark Private > > > Key: HBASE-24115 > URL: https://issues.apache.org/jira/browse/HBASE-24115 > Project: HBase > Issue Type: Test > Components: REST, security >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0, 1.3.7, 1.7.0, 2.1.10, 1.4.14, 2.2.5 > > > Relocate test-only REST "client" from src/ to test/ and annotate as Private. > The classes o.a.h.h.rest.Remote* were developed to facilitate REST unit tests > and incorrectly committed to src/ . > Although this "breaks" compatibility by moving public classes to test jar and > marking them private, no attention has been paid to these classes with > respect to performance, convenience, or security. Consensus from various > discussions over the years is to move them to test/ as was intent of the > original committer, but misplaced by the same individual. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] virajjasani commented on pull request #2075: HBASE-24742 Improve performance of SKIP vs SEEK logic.
virajjasani commented on pull request #2075: URL: https://github.com/apache/hbase/pull/2075#issuecomment-658953647 > lgtm. My quick test shows that this shaves off a compare for tables without ROWCOL bfs. Thats a nice speed up. Just a question: The speed is inline with 1.3? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on a change in pull request #2065: HBASE-24739 [Build] branch-1's build seems broken because of pylint
virajjasani commented on a change in pull request #2065: URL: https://github.com/apache/hbase/pull/2065#discussion_r455280866 ## File path: dev-support/docker/Dockerfile ## @@ -114,7 +115,7 @@ ENV FINDBUGS_HOME /opt/findbugs # Install pylint -RUN pip install pylint==1.9.2 +# RUN pip install pylint==1.9.5 Review comment: nit: remove entire comment This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24459) Move the locateMeta logic from AsyncMetaRegionTableLocator to ConnectionRegistry
[ https://issues.apache.org/jira/browse/HBASE-24459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158625#comment-17158625 ] Viraj Jasani commented on HBASE-24459: -- Few questions: If we move locateMeta to ConnectionRegistry, MasterRegistry can call MasterProtos.locateMetaRegion(), however, what about ZKConnectionRegistry? Are we expecting default registry to be MasterRegistry with splittable meta work (skimmed through design doc, maybe I missed something related to this)? I think mostly no because many internal cluster connections still do require ZKConnection right? I hope the intention of this Jira is for clients to hedge the requests in a random order to avoid hot-spotting a single HMaster, hence using Master Registry. Also, with splittable meta, we have no chance of ZK locating meta regions. Is that correct? > Move the locateMeta logic from AsyncMetaRegionTableLocator to > ConnectionRegistry > > > Key: HBASE-24459 > URL: https://issues.apache.org/jira/browse/HBASE-24459 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Priority: Major > > Now the related code is only in AsyncMetaRegionTableLocator, we could make > the actually implementation pluggable, so we do not always need to go to the > active master. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] busbey commented on pull request #2065: HBASE-24739 [Build] branch-1's build seems broken because of pylint
busbey commented on pull request #2065: URL: https://github.com/apache/hbase/pull/2065#issuecomment-658930444 seems reasonable. delete the whole "install pylint via pip" section? maybe put up a test commit on this PR that touches a python file so we can ensure pylint works? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #2015: HBASE-24568 do-release need not wait for tag
busbey commented on pull request #2015: URL: https://github.com/apache/hbase/pull/2015#issuecomment-658928752 I mean using `git ls-remote` in place of our current calls to `check_for_tag`; since the latter just calls the former. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2056: HBASE-24710 Incorrect checksum calculation in saveVersion.sh
Apache-HBase commented on pull request #2056: URL: https://github.com/apache/hbase/pull/2056#issuecomment-658893173 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2056: HBASE-24710 Incorrect checksum calculation in saveVersion.sh
Apache-HBase commented on pull request #2056: URL: https://github.com/apache/hbase/pull/2056#issuecomment-658893345 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | | +1 :green_heart: | shellcheck | 0m 0s | The patch generated 0 new + 0 unchanged - 10 fixed = 0 total (was 10) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 1m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2056/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2056 | | Optional Tests | dupname asflicense shellcheck shelldocs | | uname | Linux 2b002e7bec9d 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2505c7760d | | Max. process+thread count | 42 (vs. ulimit of 12500) | | modules | C: hbase-common U: hbase-common | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2056/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-22046) add precommit check for hbase native client
[ https://issues.apache.org/jira/browse/HBASE-22046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharath Vissapragada updated HBASE-22046: - Fix Version/s: master Resolution: Fixed Status: Resolved (was: Patch Available) > add precommit check for hbase native client > --- > > Key: HBASE-22046 > URL: https://issues.apache.org/jira/browse/HBASE-22046 > Project: HBase > Issue Type: Sub-task >Reporter: Junhong Xu >Assignee: Bharath Vissapragada >Priority: Minor > Fix For: master > > Attachments: HBASE-22046.HBASE-14850.v01.patch, > HBASE-22046.HBASE-14850.v02.patch, HBASE-22046.HBASE-14850.v03.patch > > > The UT and other checks for hbase cpp client haven't been added into the > Hadoop QA pipeline -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] bharathv commented on pull request #2075: HBASE-24742 Improve performance of SKIP vs SEEK logic.
bharathv commented on pull request #2075: URL: https://github.com/apache/hbase/pull/2075#issuecomment-658879791 Looks like branch-1 pre-commit is broken. @Reidd has a patch at #2065. Let me take a look at that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-20904) Prometheus /metrics http endpoint for monitoring integration
[ https://issues.apache.org/jira/browse/HBASE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158524#comment-17158524 ] Nick Dimiduk commented on HBASE-20904: -- Apologies for arriving late to the discussion, it appears a fair bit of development has already happened. However, I'm concerned -- why would we build this ourselves when there exists [a tool provided by the prometheus project|https://github.com/prometheus/jmx_exporter] for exporting JMX beans? I have not tried it, but I have had very good success using [a tool for exporting JMX beans over http/json|https://jolokia.org] that appears to function identically. Why not avoid writing all the code and instead document use of the existing agent? > Prometheus /metrics http endpoint for monitoring integration > > > Key: HBASE-20904 > URL: https://issues.apache.org/jira/browse/HBASE-20904 > Project: HBase > Issue Type: New Feature > Components: metrics, monitoring >Reporter: Hari Sekhon >Assignee: Madhusoodan >Priority: Major > > Feature Request to add Prometheus /metrics http endpoint for monitoring > integration: > [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E] > Prometheus metrics format for that endpoint: > [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24526) Deadlock executing assign meta procedure
[
https://issues.apache.org/jira/browse/HBASE-24526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158522#comment-17158522
]
Huaxiang Sun commented on HBASE-24526:
--
Thanks [~zhangduo] for the detail explain. We traced further, the issue is not
at the master side. The issue is at RS side, where RS receives the openMeta
RPC, but is not able to process it. More logging is added at RS code, and still
working on reproducing.
> Deadlock executing assign meta procedure
>
>
> Key: HBASE-24526
> URL: https://issues.apache.org/jira/browse/HBASE-24526
> Project: HBase
> Issue Type: Bug
> Components: proc-v2, Region Assignment
>Affects Versions: 2.3.0
>Reporter: Nick Dimiduk
>Priority: Critical
>
> I have what appears to be a deadlock while assigning meta. During recovery,
> master creates the assign procedure for meta, and immediately marks meta as
> assigned in zookeeper. It then creates the subprocedure to open meta on the
> target region. However, the PEWorker pool is full of procedures that are
> stuck, I think because their calls to update meta are going nowhere. For what
> it's worth, the balancer is running concurrently, and has calculated a plan
> size of 41.
> From the master log,
> {noformat}
> 2020-06-06 00:34:07,314 INFO
> org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure:
> Starting pid=17802, ppid=17801,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true;
> TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN;
> state=OPEN, location=null; forceNewPlan=true, retain=false
> 2020-06-06 00:34:07,465 INFO
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator: Setting hbase:meta
> (replicaId=0) location in ZooKeeper as
> hbasedn139.example.com,16020,1591403576247
> 2020-06-06 00:34:07,466 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Initialized
> subprocedures=[{pid=17803, ppid=17802, state=RUNNABLE;
> org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}]
> {noformat}
> {{pid=17803}} is not mentioned again. hbasedn139 never receives an
> {{openRegion}} RPC.
> Meanwhile, additional procedures are scheduled and picked up by workers, each
> getting "stuck". I see log lines for all 16 PEWorker threads, saying that
> they are stuck.
> {noformat}
> 2020-06-06 00:34:07,961 INFO
> org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler: Took xlock
> for pid=17804, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE;
> TransitRegionStateProcedure table=IntegrationTestBigLinkedList,
> region=54f4f6c0e921e6d25e6043cba79c09aa, REOPEN/MOVE
> 2020-06-06 00:34:07,961 INFO
> org.apache.hadoop.hbase.master.assignment.RegionStateStore: pid=17804
> updating hbase:meta row=54f4f6c0e921e6d25e6043cba79c09aa,
> regionState=CLOSING, regionLocation=hbasedn046.example.com,16020,1591402383956
> ...
> 2020-06-06 00:34:22,295 WARN
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Worker stuck
> PEWorker-16(pid=17804), run time 14.3340 sec
> ...
> 2020-06-06 00:34:27,295 WARN
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Worker stuck
> PEWorker-16(pid=17804), run time 19.3340 sec
> ...
> {noformat}
> The cluster stays in this state, with PEWorker thread stuck for upwards of 15
> minutes. Eventually master starts logging
> {noformat}
> 2020-06-06 00:50:18,033 INFO
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl: Call exception,
> tries=30, retries=31, started=970072 ms ago, cancelled=false, msg=Call queue
> is full on hbasedn139.example.com,16020,1591403576247, too many items queued
> ?, details=row
> 'IntegrationTestBigLinkedList,,1591398987965.54f4f6c0e921e6d25e6043cba79c09aa.'
> on table 'hbase:meta' at region=hbase:meta,,1.
> 1588230740, hostname=hbasedn139.example.com,16020,1591403576247, seqNum=-1,
> see https://s.apache.org/timeout
> {noformat}
> The master never recovers on its own.
> I'm not sure how common this condition might be. This popped after about 20
> total hours of running ITBLL with ServerKillingMonkey.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] petersomogyi commented on a change in pull request #2056: HBASE-24710 Incorrect checksum calculation in saveVersion.sh
petersomogyi commented on a change in pull request #2056:
URL: https://github.com/apache/hbase/pull/2056#discussion_r455189133
##
File path: hbase-common/src/saveVersion.sh
##
@@ -30,34 +30,34 @@ nativeOutputDirectory="$2/native/utils/"
pushd .
cd ..
-user=`whoami | sed -n -e 's/\\\//p'`
+user=$(whoami | sed -n -e "s/\\\//p")
if [ "$user" == "" ]
then
- user=`whoami`
+ user=$(whoami)
fi
-date=`date`
-cwd=`pwd`
+date=$(date)
+cwd=$(pwd)
if [ -d .svn ]; then
- revision=`(svn info | sed -n -e 's/Last Changed Rev: \(.*\)/\1/p') || true`
- url=`(svn info | sed -n -e 's/^URL: \(.*\)/\1/p') || true`
+ revision=$( (svn info | sed -n -e "s/Last Changed Rev: \(.*\)/\1/p") || true)
+ url=$( (svn info | sed -n -e 's/^URL: \(.*\)/\1/p') || true)
elif [ -d .git ]; then
- revision=`git log -1 --no-show-signature --pretty=format:"%H" || true`
- hostname=`hostname`
+ revision=$(git log -1 --no-show-signature --pretty=format:"%H" || true)
+ hostname=$(hostname)
url="git://${hostname}${cwd}"
fi
if [ -z "${revision}" ]; then
echo "[WARN] revision info is empty! either we're not in VCS or VCS commands
failed." >&2
revision="Unknown"
url="file://$cwd"
fi
-if ! [ -x "$(command -v openssl)" ]; then
+if ! [ -x "$(command -v openssl)" ]; then
if ! [ -x "$(command -v gpg)" ]; then
srcChecksum="Unknown"
else
-srcChecksum=`find hbase-*/src/main/ | grep -e "\.java" -e "\.proto" |
LC_ALL=C sort | xargs gpg --print-md sha512 | gpg --print-md sha512 | cut -d '
' -f 1`
Review comment:
Based on history it was used for `md5` and `md5sum` commands but it
wasn't removed when SHA512 was introduced in saveVersion.sh.
```
root@8224f4c69e6f:/# echo "foo" | md5sum
d3b07384d113edec49eaa6238ad5ff00 -
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] petersomogyi commented on a change in pull request #2056: HBASE-24710 Incorrect checksum calculation in saveVersion.sh
petersomogyi commented on a change in pull request #2056:
URL: https://github.com/apache/hbase/pull/2056#discussion_r455187162
##
File path: hbase-common/src/saveVersion.sh
##
@@ -30,34 +30,34 @@ nativeOutputDirectory="$2/native/utils/"
pushd .
cd ..
-user=`whoami | sed -n -e 's/\\\//p'`
+user=$(whoami | sed -n -e "s/\\\//p")
if [ "$user" == "" ]
then
- user=`whoami`
+ user=$(whoami)
fi
-date=`date`
-cwd=`pwd`
+date=$(date)
+cwd=$(pwd)
if [ -d .svn ]; then
- revision=`(svn info | sed -n -e 's/Last Changed Rev: \(.*\)/\1/p') || true`
- url=`(svn info | sed -n -e 's/^URL: \(.*\)/\1/p') || true`
+ revision=$( (svn info | sed -n -e "s/Last Changed Rev: \(.*\)/\1/p") || true)
+ url=$( (svn info | sed -n -e 's/^URL: \(.*\)/\1/p') || true)
elif [ -d .git ]; then
- revision=`git log -1 --no-show-signature --pretty=format:"%H" || true`
- hostname=`hostname`
+ revision=$(git log -1 --no-show-signature --pretty=format:"%H" || true)
+ hostname=$(hostname)
url="git://${hostname}${cwd}"
fi
if [ -z "${revision}" ]; then
echo "[WARN] revision info is empty! either we're not in VCS or VCS commands
failed." >&2
revision="Unknown"
url="file://$cwd"
fi
-if ! [ -x "$(command -v openssl)" ]; then
+if ! [ -x "$(command -v openssl)" ]; then
if ! [ -x "$(command -v gpg)" ]; then
srcChecksum="Unknown"
else
-srcChecksum=`find hbase-*/src/main/ | grep -e "\.java" -e "\.proto" |
LC_ALL=C sort | xargs gpg --print-md sha512 | gpg --print-md sha512 | cut -d '
' -f 1`
+srcChecksum=$(find hbase-*/src/main/ | grep -e "\.java" -e "\.proto" |
LC_ALL=C sort xargs gpg --print-md sha512 | gpg --print-md sha512 | tr '\n' ' '
| sed 's/[[:space:]]*//g')
Review comment:
Yes, dropping `|` is a bug, let me fix it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2075: HBASE-24742 Improve performance of SKIP vs SEEK logic.
Apache-HBase commented on pull request #2075: URL: https://github.com/apache/hbase/pull/2075#issuecomment-658872759 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | docker | 5m 27s | Docker failed to build yetus/hbase:7cf97016b9. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/2075 | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2075/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24376) MergeNormalizer is merging non-adjacent regions and causing region overlaps/holes.
[
https://issues.apache.org/jira/browse/HBASE-24376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158517#comment-17158517
]
Huaxiang Sun commented on HBASE-24376:
--
Hi [~ruslan.sabitov], last time I checked, for branch-1, it is ok. In your
case, these two regions are overlapping with each other, so it is ok. It seems
that there is already overlaps which means the table is not at a consistent
state.
> MergeNormalizer is merging non-adjacent regions and causing region
> overlaps/holes.
> --
>
> Key: HBASE-24376
> URL: https://issues.apache.org/jira/browse/HBASE-24376
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 2.3.0
>Reporter: Huaxiang Sun
>Assignee: Huaxiang Sun
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.3.0
>
>
> Currently, we found normalizer was merging regions which are non-adjacent, it
> will cause inconsistencies in the cluster.
> {code:java}
> 439055 2020-05-08 17:47:09,814 INFO
> org.apache.hadoop.hbase.master.normalizer.MergeNormalizationPlan: Executing
> merging normalization plan: MergeNormalizationPlan{firstRegion={ENCODED =>
> 47fe236a5e3649ded95cb64ad0c08492, NAME =>
> 'TABLE,\x03\x01\x05\x01\x04\x02,1554838974870.47fe236a5e3649ded95cb64ad
> 0c08492.', STARTKEY => '\x03\x01\x05\x01\x04\x02', ENDKEY =>
> '\x03\x01\x05\x01\x04\x02\x01\x02\x02201904082200\x00\x00\x03Mac\x00\x00\x00\x00\x00\x00\x00\x00\x00iMac13,1\x00\x00\x00\x00\x00\x049.3-14E260\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x05'},
> secondRegion={ENCODED => 0c0f2aa67f4329d5c4 8ba0320f173d31, NAME =>
> 'TABLE,\x03\x01\x05\x02\x01\x01,1554830735526.0c0f2aa67f4329d5c48ba0320f173d31.',
> STARTKEY => '\x03\x01\x05\x02\x01\x01', ENDKEY =>
> '\x03\x01\x05\x02\x01\x02'}}
> 439056 2020-05-08 17:47:11,438 INFO org.apache.hadoop.hbase.ScheduledChore:
> CatalogJanitor-*:16000 average execution time: 1676219193 ns.
> 439057 2020-05-08 17:47:11,730 INFO org.apache.hadoop.hbase.master.HMaster:
> Client=null/null merge regions [47fe236a5e3649ded95cb64ad0c08492],
> [0c0f2aa67f4329d5c48ba0320f173d31]
> {code}
>
> The root cause is that getMergeNormalizationPlan() uses a list of regionInfo
> which is ordered by regionName. regionName does not necessary guarantee the
> order of STARTKEY (let's say 'aa1', 'aa1!', in order of regionName, it will
> be 'aa1!' followed by 'aa1'. This will result in normalizer merging
> non-adjacent regions into one and creates overlaps. This is not an issue in
> branch-1 as the list is already ordered by RegionInfo.COMPARATOR in
> normalizer.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on a change in pull request #2056: HBASE-24710 Incorrect checksum calculation in saveVersion.sh
ndimiduk commented on a change in pull request #2056:
URL: https://github.com/apache/hbase/pull/2056#discussion_r455181612
##
File path: hbase-common/src/saveVersion.sh
##
@@ -30,34 +30,34 @@ nativeOutputDirectory="$2/native/utils/"
pushd .
cd ..
-user=`whoami | sed -n -e 's/\\\//p'`
+user=$(whoami | sed -n -e "s/\\\//p")
if [ "$user" == "" ]
then
- user=`whoami`
+ user=$(whoami)
fi
-date=`date`
-cwd=`pwd`
+date=$(date)
+cwd=$(pwd)
if [ -d .svn ]; then
- revision=`(svn info | sed -n -e 's/Last Changed Rev: \(.*\)/\1/p') || true`
- url=`(svn info | sed -n -e 's/^URL: \(.*\)/\1/p') || true`
+ revision=$( (svn info | sed -n -e "s/Last Changed Rev: \(.*\)/\1/p") || true)
+ url=$( (svn info | sed -n -e 's/^URL: \(.*\)/\1/p') || true)
elif [ -d .git ]; then
- revision=`git log -1 --no-show-signature --pretty=format:"%H" || true`
- hostname=`hostname`
+ revision=$(git log -1 --no-show-signature --pretty=format:"%H" || true)
+ hostname=$(hostname)
url="git://${hostname}${cwd}"
fi
if [ -z "${revision}" ]; then
echo "[WARN] revision info is empty! either we're not in VCS or VCS commands
failed." >&2
revision="Unknown"
url="file://$cwd"
fi
-if ! [ -x "$(command -v openssl)" ]; then
+if ! [ -x "$(command -v openssl)" ]; then
if ! [ -x "$(command -v gpg)" ]; then
srcChecksum="Unknown"
else
-srcChecksum=`find hbase-*/src/main/ | grep -e "\.java" -e "\.proto" |
LC_ALL=C sort | xargs gpg --print-md sha512 | gpg --print-md sha512 | cut -d '
' -f 1`
+srcChecksum=$(find hbase-*/src/main/ | grep -e "\.java" -e "\.proto" |
LC_ALL=C sort xargs gpg --print-md sha512 | gpg --print-md sha512 | tr '\n' ' '
| sed 's/[[:space:]]*//g')
fi
else
- srcChecksum=`find hbase-*/src/main/ | grep -e "\.java" -e "\.proto" |
LC_ALL=C sort | xargs openssl dgst -sha512 | openssl dgst -sha512 | cut -d ' '
-f 1`
+ srcChecksum=$(find hbase-*/src/main/ | grep -e "\.java" -e "\.proto" |
LC_ALL=C sort xargs openssl dgst -sha512 | openssl dgst -sha512 | sed 's/^.*
//')
Review comment:
Oh, this is the example from your comment on JIRA, a difference between
the mac and linux versions of `openssl dgst`. Disregard.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #2056: HBASE-24710 Incorrect checksum calculation in saveVersion.sh
ndimiduk commented on a change in pull request #2056:
URL: https://github.com/apache/hbase/pull/2056#discussion_r455177715
##
File path: hbase-common/src/saveVersion.sh
##
@@ -30,34 +30,34 @@ nativeOutputDirectory="$2/native/utils/"
pushd .
cd ..
-user=`whoami | sed -n -e 's/\\\//p'`
+user=$(whoami | sed -n -e "s/\\\//p")
if [ "$user" == "" ]
then
- user=`whoami`
+ user=$(whoami)
fi
-date=`date`
-cwd=`pwd`
+date=$(date)
+cwd=$(pwd)
if [ -d .svn ]; then
- revision=`(svn info | sed -n -e 's/Last Changed Rev: \(.*\)/\1/p') || true`
- url=`(svn info | sed -n -e 's/^URL: \(.*\)/\1/p') || true`
+ revision=$( (svn info | sed -n -e "s/Last Changed Rev: \(.*\)/\1/p") || true)
+ url=$( (svn info | sed -n -e 's/^URL: \(.*\)/\1/p') || true)
elif [ -d .git ]; then
- revision=`git log -1 --no-show-signature --pretty=format:"%H" || true`
- hostname=`hostname`
+ revision=$(git log -1 --no-show-signature --pretty=format:"%H" || true)
+ hostname=$(hostname)
url="git://${hostname}${cwd}"
fi
if [ -z "${revision}" ]; then
echo "[WARN] revision info is empty! either we're not in VCS or VCS commands
failed." >&2
revision="Unknown"
url="file://$cwd"
fi
-if ! [ -x "$(command -v openssl)" ]; then
+if ! [ -x "$(command -v openssl)" ]; then
if ! [ -x "$(command -v gpg)" ]; then
srcChecksum="Unknown"
else
-srcChecksum=`find hbase-*/src/main/ | grep -e "\.java" -e "\.proto" |
LC_ALL=C sort | xargs gpg --print-md sha512 | gpg --print-md sha512 | cut -d '
' -f 1`
Review comment:
Previous use of `cut` here at the end is surprising. It took only the
first block...
##
File path: hbase-common/src/saveVersion.sh
##
@@ -30,34 +30,34 @@ nativeOutputDirectory="$2/native/utils/"
pushd .
cd ..
-user=`whoami | sed -n -e 's/\\\//p'`
+user=$(whoami | sed -n -e "s/\\\//p")
if [ "$user" == "" ]
then
- user=`whoami`
+ user=$(whoami)
fi
-date=`date`
-cwd=`pwd`
+date=$(date)
+cwd=$(pwd)
if [ -d .svn ]; then
- revision=`(svn info | sed -n -e 's/Last Changed Rev: \(.*\)/\1/p') || true`
- url=`(svn info | sed -n -e 's/^URL: \(.*\)/\1/p') || true`
+ revision=$( (svn info | sed -n -e "s/Last Changed Rev: \(.*\)/\1/p") || true)
+ url=$( (svn info | sed -n -e 's/^URL: \(.*\)/\1/p') || true)
elif [ -d .git ]; then
- revision=`git log -1 --no-show-signature --pretty=format:"%H" || true`
- hostname=`hostname`
+ revision=$(git log -1 --no-show-signature --pretty=format:"%H" || true)
+ hostname=$(hostname)
url="git://${hostname}${cwd}"
fi
if [ -z "${revision}" ]; then
echo "[WARN] revision info is empty! either we're not in VCS or VCS commands
failed." >&2
revision="Unknown"
url="file://$cwd"
fi
-if ! [ -x "$(command -v openssl)" ]; then
+if ! [ -x "$(command -v openssl)" ]; then
if ! [ -x "$(command -v gpg)" ]; then
srcChecksum="Unknown"
else
-srcChecksum=`find hbase-*/src/main/ | grep -e "\.java" -e "\.proto" |
LC_ALL=C sort | xargs gpg --print-md sha512 | gpg --print-md sha512 | cut -d '
' -f 1`
+srcChecksum=$(find hbase-*/src/main/ | grep -e "\.java" -e "\.proto" |
LC_ALL=C sort xargs gpg --print-md sha512 | gpg --print-md sha512 | tr '\n' ' '
| sed 's/[[:space:]]*//g')
Review comment:
I think dropping the `|` between `sort` and `xargs` is a bug. I believe
this calculates the sha512 of the sort error message, not the sha512 of the
sha512s of the contents of all the files.
On a mac anyway, I get
```
$ find hbase-common/src/main/ | grep -e "\.java" | LC_ALL=C sort xargs gpg
--print-md sha512
sort: unrecognized option `--print-md'
Usage: sort [-bcCdfigMmnrsuz] [-kPOS1[,POS2] ... ] [+POS1 [-POS2]] [-S
memsize] [-T tmpdir] [-t separator] [-o outfile] [--batch-size size]
[--files0-from file] [--heapsort] [--mergesort] [--radixsort] [--qsort]
[--mmap] [--parallel thread_no] [--human-numeric-sort] [--version-sort]
[--random-sort [--random-source file]] [--compress-program program] [file ...]
$ find hbase-common/src/main/ | grep -e "\.java" | LC_ALL=C sort xargs
openssl dgst -sha512
sort: invalid option -- a
Usage: sort [-bcCdfigMmnrsuz] [-kPOS1[,POS2] ... ] [+POS1 [-POS2]] [-S
memsize] [-T tmpdir] [-t separator] [-o outfile] [--batch-size size]
[--files0-from file] [--heapsort] [--mergesort] [--radixsort] [--qsort]
[--mmap] [--parallel thread_no] [--human-numeric-sort] [--version-sort]
[--random-sort [--random-source file]] [--compress-program program] [file ...]
```
##
File path: hbase-common/src/saveVersion.sh
##
@@ -30,34 +30,34 @@ nativeOutputDirectory="$2/native/utils/"
pushd .
cd ..
-user=`whoami | sed -n -e 's/\\\//p'`
+user=$(whoami | sed -n -e "s/\\\//p")
if [ "$user" == "" ]
then
- user=`whoami`
+ user=$(whoami)
fi
-date=`date`
-cwd=`pwd`
+date=$(date)
+cwd=$(pwd)
if [ -d .svn ]; then
- revision=`(svn info | sed -n -e 's/Last Changed Rev: \(.*\)/\1/p') || true`
- url=`(svn
[jira] [Commented] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[ https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158497#comment-17158497 ] Lars Hofhansl commented on HBASE-24742: --- Created a PR for observation #1 above. > Improve performance of SKIP vs SEEK logic > - > > Key: HBASE-24742 > URL: https://issues.apache.org/jira/browse/HBASE-24742 > Project: HBase > Issue Type: Bug > Components: Performance, regionserver >Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: hbase-1.6-regression-flame-graph.png, > hbase-24742-branch-1.txt > > > In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% > slowdown in scanning scenarios. > We tracked it back to HBASE-17958 and HBASE-19863. > Both add comparisons to one of the tightest HBase has. > [~bharathv] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] lhofhansl opened a new pull request #2075: HBASE-24742 Improve performance of SKIP vs SEEK logic.
lhofhansl opened a new pull request #2075: URL: https://github.com/apache/hbase/pull/2075 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2074: HBASE-24719: Renaming invalid rsgroup throws NPE instead of proper er…
Apache-HBase commented on pull request #2074: URL: https://github.com/apache/hbase/pull/2074#issuecomment-658824068 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 18s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | branch-2 passed | | +1 :green_heart: | checkstyle | 0m 13s | branch-2 passed | | +1 :green_heart: | spotbugs | 0m 42s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 35s | the patch passed | | +1 :green_heart: | checkstyle | 0m 12s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 45s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 0m 50s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 31m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2074/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2074 | | JIRA Issue | HBASE-24719 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 842528546d1e 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / ad826a43a2 | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-rsgroup U: hbase-rsgroup | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2074/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2074: HBASE-24719: Renaming invalid rsgroup throws NPE instead of proper er…
Apache-HBase commented on pull request #2074: URL: https://github.com/apache/hbase/pull/2074#issuecomment-658822344 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 42s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 34s | branch-2 passed | | +1 :green_heart: | compile | 0m 29s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 5s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 25s | hbase-rsgroup in branch-2 failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed | | +1 :green_heart: | javac | 0m 28s | the patch passed | | +1 :green_heart: | shadedjars | 6m 39s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 23s | hbase-rsgroup in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 1s | hbase-rsgroup in the patch passed. | | | | 28m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2074/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2074 | | JIRA Issue | HBASE-24719 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 341361684e60 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / ad826a43a2 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2074/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-rsgroup.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2074/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-rsgroup.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2074/1/testReport/ | | Max. process+thread count | 3862 (vs. ulimit of 12500) | | modules | C: hbase-rsgroup U: hbase-rsgroup | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2074/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2074: HBASE-24719: Renaming invalid rsgroup throws NPE instead of proper er…
Apache-HBase commented on pull request #2074: URL: https://github.com/apache/hbase/pull/2074#issuecomment-658819998 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 42s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 46s | branch-2 passed | | +1 :green_heart: | compile | 0m 23s | branch-2 passed | | +1 :green_heart: | shadedjars | 4m 58s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 22s | the patch passed | | +1 :green_heart: | compile | 0m 23s | the patch passed | | +1 :green_heart: | javac | 0m 23s | the patch passed | | +1 :green_heart: | shadedjars | 5m 3s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 7s | hbase-rsgroup in the patch passed. | | | | 23m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2074/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2074 | | JIRA Issue | HBASE-24719 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d2baaaf50ea8 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / ad826a43a2 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2074/1/testReport/ | | Max. process+thread count | 3580 (vs. ulimit of 12500) | | modules | C: hbase-rsgroup U: hbase-rsgroup | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2074/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24705) MetaFixer#fixHoles() does not include the case for read replicas (i.e, replica regions are not created)
[
https://issues.apache.org/jira/browse/HBASE-24705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158229#comment-17158229
]
Hudson commented on HBASE-24705:
Results for branch branch-2.3
[build #183 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/183/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/183/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/183/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/183/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/183/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> MetaFixer#fixHoles() does not include the case for read replicas (i.e,
> replica regions are not created)
> ---
>
> Key: HBASE-24705
> URL: https://issues.apache.org/jira/browse/HBASE-24705
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 2.3.0
>Reporter: Huaxiang Sun
>Assignee: Huaxiang Sun
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] arshadmohammad opened a new pull request #2074: HBASE-24719: Renaming invalid rsgroup throws NPE instead of proper er…
arshadmohammad opened a new pull request #2074: URL: https://github.com/apache/hbase/pull/2074 …ror message This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24719) Renaming invalid rsgroup throws NPE instead of proper error message
[ https://issues.apache.org/jira/browse/HBASE-24719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Arshad updated HBASE-24719: Summary: Renaming invalid rsgroup throws NPE instead of proper error message (was: Renaming invalid rsgroup throwing NPE instead of proper error message ) > Renaming invalid rsgroup throws NPE instead of proper error message > > > Key: HBASE-24719 > URL: https://issues.apache.org/jira/browse/HBASE-24719 > Project: HBase > Issue Type: Bug > Environment: HBASE_2.2.3 > >Reporter: chiranjeevi >Assignee: Mohammad Arshad >Priority: Major > > Renaming an rsgroup which is not available in the cluster throwing > IOException. > Create rsgroup > add_rsgroup 'RSG1' > rename_rsgroup 'RSG2','RSG3' > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2073: HBASE-24467 Backport HBASE-23963: Split TestFromClientSide; it takes …
Apache-HBase commented on pull request #2073: URL: https://github.com/apache/hbase/pull/2073#issuecomment-658801828 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ branch-2.2 Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 39s | branch-2.2 passed | | +1 :green_heart: | compile | 1m 24s | branch-2.2 passed | | +1 :green_heart: | checkstyle | 1m 58s | branch-2.2 passed | | +1 :green_heart: | shadedjars | 4m 55s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 8s | branch-2.2 passed | | +0 :ok: | spotbugs | 4m 17s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 5m 16s | branch-2.2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 26s | the patch passed | | +1 :green_heart: | compile | 1m 35s | the patch passed | | +1 :green_heart: | javac | 1m 35s | the patch passed | | -1 :x: | checkstyle | 1m 36s | hbase-server: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 41s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 29m 38s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | javadoc | 0m 57s | the patch passed | | +1 :green_heart: | findbugs | 4m 52s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 54s | hbase-common in the patch passed. | | -1 :x: | unit | 180m 11s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 54s | The patch does not generate ASF License warnings. | | | | 263m 6s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.TestZooKeeper | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2073/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2073 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 93fd568892d2 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-2073/out/precommit/personality/provided.sh | | git revision | branch-2.2 / f44336d277 | | Default Java | 1.8.0_181 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2073/1/artifact/out/diff-checkstyle-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2073/1/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2073/1/testReport/ | | Max. process+thread count | 4691 (vs. ulimit of 1) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2073/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158180#comment-17158180 ] Duo Zhang commented on HBASE-11288: --- I found a version in one of my local repo and pushed it back to HBASE-11288. Please take a look to see if it is the newest one. Sorry again. [~toffer]. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24742) Improve performance of SKIP vs SEEK logic
[
https://issues.apache.org/jira/browse/HBASE-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158167#comment-17158167
]
Duo Zhang commented on HBASE-24742:
---
{quote}
I'm not sure how we can avoid passing the fake keys up, since it is designed to
handle things and the "upper" heap. At least not without a lot refactoring.
{quote}
We need a big refactoring for this change, I believe. Will open a brainstorm
issue later, as I'm a bit busy these days. Make summary for the first half and
make plan for the second half...
> Improve performance of SKIP vs SEEK logic
> -
>
> Key: HBASE-24742
> URL: https://issues.apache.org/jira/browse/HBASE-24742
> Project: HBase
> Issue Type: Bug
> Components: Performance, regionserver
>Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Major
> Attachments: hbase-1.6-regression-flame-graph.png,
> hbase-24742-branch-1.txt
>
>
> In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30%
> slowdown in scanning scenarios.
> We tracked it back to HBASE-17958 and HBASE-19863.
> Both add comparisons to one of the tightest HBase has.
> [~bharathv]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24721) rename_rsgroup overwriting the existing rsgroup.
[ https://issues.apache.org/jira/browse/HBASE-24721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158155#comment-17158155 ] Viraj Jasani commented on HBASE-24721: -- For now, closing the Jira with patch landing on master, branch-2, 2.3, branch-1. If I get the permission for branch-2.2, will reopen the Jira and update fix versions after applying the patch. > rename_rsgroup overwriting the existing rsgroup. > > > Key: HBASE-24721 > URL: https://issues.apache.org/jira/browse/HBASE-24721 > Project: HBase > Issue Type: Bug >Reporter: chiranjeevi >Assignee: Mohammad Arshad >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.6 > > > rename_rsgroup overwriting the current rsgroup. > Steps: > 1)add_rsgroup 'RSG1' and 'RSG2' > 2)move_servers_rsgroup 'RSG1',['server1:port'] > 3)rename_rsgroup 'RSG1','RSG2' > After performing step3 RSG1 overwriting to RSG2 and region servers added in > RSG1 are not available now. > Ideally system should show error message Group already exists: RSG2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24721) rename_rsgroup overwriting the existing rsgroup.
[ https://issues.apache.org/jira/browse/HBASE-24721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani resolved HBASE-24721. -- Fix Version/s: (was: 2.2.6) Hadoop Flags: Reviewed Resolution: Fixed > rename_rsgroup overwriting the existing rsgroup. > > > Key: HBASE-24721 > URL: https://issues.apache.org/jira/browse/HBASE-24721 > Project: HBase > Issue Type: Bug >Reporter: chiranjeevi >Assignee: Mohammad Arshad >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0 > > > rename_rsgroup overwriting the current rsgroup. > Steps: > 1)add_rsgroup 'RSG1' and 'RSG2' > 2)move_servers_rsgroup 'RSG1',['server1:port'] > 3)rename_rsgroup 'RSG1','RSG2' > After performing step3 RSG1 overwriting to RSG2 and region servers added in > RSG1 are not available now. > Ideally system should show error message Group already exists: RSG2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158152#comment-17158152
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288
[build #2 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288/2/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288/2/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288/2/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288/2/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24721) rename_rsgroup overwriting the existing rsgroup.
[ https://issues.apache.org/jira/browse/HBASE-24721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158141#comment-17158141 ] Viraj Jasani commented on HBASE-24721: -- [~zghao] can we commit this to branch-2.2? Thanks > rename_rsgroup overwriting the existing rsgroup. > > > Key: HBASE-24721 > URL: https://issues.apache.org/jira/browse/HBASE-24721 > Project: HBase > Issue Type: Bug >Reporter: chiranjeevi >Assignee: Mohammad Arshad >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.6 > > > rename_rsgroup overwriting the current rsgroup. > Steps: > 1)add_rsgroup 'RSG1' and 'RSG2' > 2)move_servers_rsgroup 'RSG1',['server1:port'] > 3)rename_rsgroup 'RSG1','RSG2' > After performing step3 RSG1 overwriting to RSG2 and region servers added in > RSG1 are not available now. > Ideally system should show error message Group already exists: RSG2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] virajjasani closed pull request #2066: HBASE-24721: rename_rsgroup overwriting the existing rsgroup.
virajjasani closed pull request #2066: URL: https://github.com/apache/hbase/pull/2066 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24720) Meta replicas not cleaned when disabled
[
https://issues.apache.org/jira/browse/HBASE-24720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158112#comment-17158112
]
Hudson commented on HBASE-24720:
Results for branch master
[build #1786 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1786/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1786/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1698/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1786/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1786/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Meta replicas not cleaned when disabled
> ---
>
> Key: HBASE-24720
> URL: https://issues.apache.org/jira/browse/HBASE-24720
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.4.0, 2.2.5
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0, 2.2.6
>
>
> The assignMetaReplicas method works kinda like this:
> {code:java}
> void assignMetaReplicas(){
> if (numReplicas <= 1) return;
> //create if needed then assign meta replicas
> unassignExcessMetaReplica(numReplicas);
> }
> {code}
> Now this unassignExcessMetaReplica method is the one that gets rid of the
> replicas we no longer need. It closes them and deletes their zNode.
> Unfortunately this only happens if we decreased the replica number. If we
> disabled it, by setting the replica number to 1 assignMetaReplicas returns
> instantly without cleaning up the no longer needed replicas resulting in
> replicas lingering around.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24721) rename_rsgroup overwriting the existing rsgroup.
[
https://issues.apache.org/jira/browse/HBASE-24721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158111#comment-17158111
]
Hudson commented on HBASE-24721:
Results for branch master
[build #1786 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1786/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1786/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1698/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1786/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1786/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> rename_rsgroup overwriting the existing rsgroup.
>
>
> Key: HBASE-24721
> URL: https://issues.apache.org/jira/browse/HBASE-24721
> Project: HBase
> Issue Type: Bug
>Reporter: chiranjeevi
>Assignee: Mohammad Arshad
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.6
>
>
> rename_rsgroup overwriting the current rsgroup.
> Steps:
> 1)add_rsgroup 'RSG1' and 'RSG2'
> 2)move_servers_rsgroup 'RSG1',['server1:port']
> 3)rename_rsgroup 'RSG1','RSG2'
> After performing step3 RSG1 overwriting to RSG2 and region servers added in
> RSG1 are not available now.
> Ideally system should show error message Group already exists: RSG2
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24581) Skip compaction request/check for replica regions at the early stage.
[
https://issues.apache.org/jira/browse/HBASE-24581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158118#comment-17158118
]
Hudson commented on HBASE-24581:
Results for branch master
[build #1786 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1786/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1786/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1698/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1786/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1786/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Skip compaction request/check for replica regions at the early stage.
> -
>
> Key: HBASE-24581
> URL: https://issues.apache.org/jira/browse/HBASE-24581
> Project: HBase
> Issue Type: Improvement
> Components: read replicas
>Affects Versions: 2.3.0
>Reporter: Huaxiang Sun
>Assignee: Huaxiang Sun
>Priority: Major
>
> I found that in certain cases replica regions can trigger compaction, one
> example as follows, need to check all places to avoid compaction for replica
> regions.
> {code:java}
> @Override
> public void postOpenDeployTasks(final PostOpenDeployContext context) throws
> IOException {
> HRegion r = context.getRegion();
> long openProcId = context.getOpenProcId();
> long masterSystemTime = context.getMasterSystemTime();
> rpcServices.checkOpen();
> LOG.info("Post open deploy tasks for {}, openProcId={},
> masterSystemTime={}",
> r.getRegionInfo().getRegionNameAsString(), openProcId, masterSystemTime);
> // Do checks to see if we need to compact (references or too many files)
> // TODO: SHX, do not do this for replica regions? Otherwise, it is going to
> lost data locality for primary regions.
> for (HStore s : r.stores.values()) {
> if (s.hasReferences() || s.needsCompaction()) {
> this.compactSplitThread.requestSystemCompaction(r, s, "Opening Region");
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24705) MetaFixer#fixHoles() does not include the case for read replicas (i.e, replica regions are not created)
[
https://issues.apache.org/jira/browse/HBASE-24705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158117#comment-17158117
]
Hudson commented on HBASE-24705:
Results for branch master
[build #1786 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1786/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1786/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1698/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1786/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1786/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> MetaFixer#fixHoles() does not include the case for read replicas (i.e,
> replica regions are not created)
> ---
>
> Key: HBASE-24705
> URL: https://issues.apache.org/jira/browse/HBASE-24705
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 2.3.0
>Reporter: Huaxiang Sun
>Assignee: Huaxiang Sun
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
