[GitHub] [hbase] Apache-HBase commented on pull request #1719: HBASE-24370 Avoid aggressive MergeRegion and GCMultipleMergedRegionsP…
Apache-HBase commented on pull request #1719: URL: https://github.com/apache/hbase/pull/1719#issuecomment-629963430 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 4s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 20s | master passed | | +1 :green_heart: | compile | 1m 2s | master passed | | +1 :green_heart: | shadedjars | 6m 9s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 57s | the patch passed | | +1 :green_heart: | compile | 1m 1s | the patch passed | | +1 :green_heart: | javac | 1m 1s | the patch passed | | +1 :green_heart: | shadedjars | 6m 8s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 9m 17s | hbase-server in the patch failed. | | | | 38m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1719/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1719 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 760d6d8d3b79 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6ce1e140ee | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1719/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1719/3/testReport/ | | Max. process+thread count | 729 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1719/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24393) release scripts should allow for creation of initial CHANGES/RELEASE on new major version.
[
https://issues.apache.org/jira/browse/HBASE-24393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-24393:
Status: Patch Available (was: In Progress)
> release scripts should allow for creation of initial CHANGES/RELEASE on new
> major version.
> --
>
> Key: HBASE-24393
> URL: https://issues.apache.org/jira/browse/HBASE-24393
> Project: HBase
> Issue Type: Bug
> Components: community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
>
> Right now if you try to run the release scripts on a new major version, e.g.
> one without a RELEASENOTES.md and CHANGES.md things fail:
> {code}
> /opt/hbase-rm/output
> + sed -i -e '/^## Release 3.0.0-alpha-1/,/^## Release/ {//!d; /^## Release
> 3.0.0-alpha-1/d;}' /opt/hbase-rm/output/hbase/CHANGES.md
> sed: can't read /opt/hbase-rm/output/hbase/CHANGES.md: No such file or
> directory
> + true
> + sed -i -e '/^# HBASE 3.0.0-alpha-1 Release Notes/,/^# HBASE/{//!d; /^#
> HBASE 3.0.0-alpha-1 Release Notes/d;}'
> /opt/hbase-rm/output/hbase/RELEASENOTES.md
> sed: can't read /opt/hbase-rm/output/hbase/RELEASENOTES.md: No such file or
> directory
> + true
> + sed -i -e '/## Release/,$!d' CHANGELOG.3.0.0-alpha-1.md
> + sed -i -e '/DO NOT REMOVE/r CHANGELOG.3.0.0-alpha-1.md'
> /opt/hbase-rm/output/hbase/CHANGES.md
> sed: can't read /opt/hbase-rm/output/hbase/CHANGES.md: No such file or
> directory
> + cleanup
> {code}
> instead we should just create the new files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] busbey commented on pull request #1728: HBASE-24393 release scripts should allow for creation of initial CHANGES/RELEASE on new major version.
busbey commented on pull request #1728: URL: https://github.com/apache/hbase/pull/1728#issuecomment-629962889 could you check this out @mattf-apache? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] busbey opened a new pull request #1728: HBASE-24393 release scripts should allow for creation of initial CHANGES/RELEASE on new major version.
busbey opened a new pull request #1728: URL: https://github.com/apache/hbase/pull/1728 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #1719: HBASE-24370 Avoid aggressive MergeRegion and GCMultipleMergedRegionsP…
huaxiangsun commented on a change in pull request #1719:
URL: https://github.com/apache/hbase/pull/1719#discussion_r426371166
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi2.java
##
@@ -161,20 +163,39 @@ public void testMergeRegions() throws Exception {
.getTableHRegionLocations(metaTable, tableName).get();
RegionInfo regionA;
RegionInfo regionB;
+RegionInfo regionC;
+RegionInfo mergedChildRegion = null;
// merge with full name
assertEquals(3, regionLocations.size());
regionA = regionLocations.get(0).getRegion();
regionB = regionLocations.get(1).getRegion();
+regionC = regionLocations.get(2).getRegion();
admin.mergeRegions(regionA.getRegionName(), regionB.getRegionName(),
false).get();
regionLocations = AsyncMetaTableAccessor
.getTableHRegionLocations(metaTable, tableName).get();
+
assertEquals(2, regionLocations.size());
+for (HRegionLocation rl : regionLocations) {
+ if (regionC.compareTo(rl.getRegion()) != 0) {
+mergedChildRegion = rl.getRegion();
+break;
+ }
+}
+
+assertTrue(mergedChildRegion != null);
Review comment:
Thanks @HorizonNet, let me address the comments and push another update.
For some reason, the second push did not trigger unittest runs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1722: HBASE-24359 Optionally ignore edits for deleted CFs for replication
Apache-HBase commented on pull request #1722: URL: https://github.com/apache/hbase/pull/1722#issuecomment-629940630 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 34s | master passed | | +1 :green_heart: | checkstyle | 1m 28s | master passed | | +1 :green_heart: | spotbugs | 2m 39s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 20s | the patch passed | | -0 :warning: | checkstyle | 1m 4s | hbase-server: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 6s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 35m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1722/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1722 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 7342334f4384 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6ce1e140ee | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1722/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1722/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ddupg commented on pull request #1722: HBASE-24359 Optionally ignore edits for deleted CFs for replication
ddupg commented on pull request #1722: URL: https://github.com/apache/hbase/pull/1722#issuecomment-629932872 Thanks @HorizonNet for reviewing. I've fix it as comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24152) Add ServerSsdLocalityCostFunction to StochasticLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109784#comment-17109784
]
Zheng Wang commented on HBASE-24152:
{quote}Why can't the ssd aspect be factored into the general server weight;
e.g. why a replica that is on ssd can't be considered 'more' local that a
replica on hdd.
{quote}
I think the locality should be the actual ratio, or else the users will be
confused on it.
Anyway, will back if i find a better way.
Thanks a lot.
> Add ServerSsdLocalityCostFunction to StochasticLoadBalancer
> ---
>
> Key: HBASE-24152
> URL: https://issues.apache.org/jira/browse/HBASE-24152
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
>
> When use ONE_SSD storagy policy, or ALL_SSD but has not enough SSD, there
> will be some hdfs blocks on DISK and others on SSD,so it is reasonable to
> consider the locality of ssd for StochasticLoadBalancer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24377) MemStoreFlusher throw NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-24377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109775#comment-17109775
]
Lijin Bin edited comment on HBASE-24377 at 5/18/20, 2:34 AM:
-
Push to maser/branch-2/branch-2.3/branch-2.2, [~anoop.hbase] [~ram_krish]
[~zhangduo] thanks for the review.
was (Author: aoxiang):
Push to maser/branch-2/branch-2.3/branch-2.2,
[~anoop.hbase][~ram_krish][~zhangduo] thanks for the review.
> MemStoreFlusher throw NullPointerException
> --
>
> Key: HBASE-24377
> URL: https://issues.apache.org/jira/browse/HBASE-24377
> Project: HBase
> Issue Type: Bug
>Reporter: Lijin Bin
>Assignee: Lijin Bin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.5
>
>
> {code}
> 2020-02-26 10:15:33,585 ERROR [MemStoreFlusher.2]
> regionserver.MemStoreFlusher: Cache flusher failed for entry null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:158)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$700(MemStoreFlusher.java:68)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:346)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Need to setFlushType first before use it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24377) MemStoreFlusher throw NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-24377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109775#comment-17109775
]
Lijin Bin commented on HBASE-24377:
---
Push to maser/branch-2/branch-2.3/branch-2.2,
[~anoop.hbase][~ram_krish][~zhangduo] thanks for the review.
> MemStoreFlusher throw NullPointerException
> --
>
> Key: HBASE-24377
> URL: https://issues.apache.org/jira/browse/HBASE-24377
> Project: HBase
> Issue Type: Bug
>Reporter: Lijin Bin
>Assignee: Lijin Bin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.5
>
>
> {code}
> 2020-02-26 10:15:33,585 ERROR [MemStoreFlusher.2]
> regionserver.MemStoreFlusher: Cache flusher failed for entry null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:158)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$700(MemStoreFlusher.java:68)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:346)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Need to setFlushType first before use it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-24377) MemStoreFlusher throw NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-24377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lijin Bin reassigned HBASE-24377:
-
Assignee: Lijin Bin
> MemStoreFlusher throw NullPointerException
> --
>
> Key: HBASE-24377
> URL: https://issues.apache.org/jira/browse/HBASE-24377
> Project: HBase
> Issue Type: Bug
>Reporter: Lijin Bin
>Assignee: Lijin Bin
>Priority: Major
>
> {code}
> 2020-02-26 10:15:33,585 ERROR [MemStoreFlusher.2]
> regionserver.MemStoreFlusher: Cache flusher failed for entry null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:158)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$700(MemStoreFlusher.java:68)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:346)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Need to setFlushType first before use it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24377) MemStoreFlusher throw NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-24377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lijin Bin resolved HBASE-24377.
---
Resolution: Fixed
> MemStoreFlusher throw NullPointerException
> --
>
> Key: HBASE-24377
> URL: https://issues.apache.org/jira/browse/HBASE-24377
> Project: HBase
> Issue Type: Bug
>Reporter: Lijin Bin
>Assignee: Lijin Bin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.5
>
>
> {code}
> 2020-02-26 10:15:33,585 ERROR [MemStoreFlusher.2]
> regionserver.MemStoreFlusher: Cache flusher failed for entry null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:158)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$700(MemStoreFlusher.java:68)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:346)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Need to setFlushType first before use it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24377) MemStoreFlusher throw NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-24377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lijin Bin updated HBASE-24377:
--
Fix Version/s: 2.2.5
2.3.0
3.0.0-alpha-1
> MemStoreFlusher throw NullPointerException
> --
>
> Key: HBASE-24377
> URL: https://issues.apache.org/jira/browse/HBASE-24377
> Project: HBase
> Issue Type: Bug
>Reporter: Lijin Bin
>Assignee: Lijin Bin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.5
>
>
> {code}
> 2020-02-26 10:15:33,585 ERROR [MemStoreFlusher.2]
> regionserver.MemStoreFlusher: Cache flusher failed for entry null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:158)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$700(MemStoreFlusher.java:68)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:346)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Need to setFlushType first before use it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1727: HBASE-17756 We should have better introspection of HFiles
Apache-HBase commented on pull request #1727: URL: https://github.com/apache/hbase/pull/1727#issuecomment-629907638 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 4s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 58s | master passed | | +1 :green_heart: | checkstyle | 2m 18s | master passed | | +1 :green_heart: | spotbugs | 11m 45s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 44s | the patch passed | | +1 :green_heart: | checkstyle | 2m 18s | root: The patch generated 0 new + 0 unchanged - 11 fixed = 0 total (was 11) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 3s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 12m 13s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | -1 :x: | spotbugs | 2m 17s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | -1 :x: | spotbugs | 9m 40s | root generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 58m 36s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-server | | | BigDecimal constructed from 0.01 in new org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter$1() At HFilePrettyPrinter.java:new org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter$1() At HFilePrettyPrinter.java:[line 141] | | FindBugs | module:root | | | BigDecimal constructed from 0.01 in new org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter$1() At HFilePrettyPrinter.java:new org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter$1() At HFilePrettyPrinter.java:[line 141] | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1727/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1727 | | Optional Tests | dupname asflicense hadoopcheck xml spotbugs hbaseanti checkstyle | | uname | Linux 3147f953892c 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | spotbugs | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1727/1/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html | | spotbugs | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1727/1/artifact/yetus-general-check/output/new-spotbugs-root.html | | Max. process+thread count | 122 (vs. ulimit of 12500) | | modules | C: hbase-server . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1727/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1727: HBASE-17756 We should have better introspection of HFiles
Apache-HBase commented on pull request #1727: URL: https://github.com/apache/hbase/pull/1727#issuecomment-629905915 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 4s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 4s | master passed | | +1 :green_heart: | compile | 3m 8s | master passed | | +1 :green_heart: | shadedjars | 6m 38s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 15s | root in master failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 27s | the patch passed | | +1 :green_heart: | compile | 3m 3s | the patch passed | | +1 :green_heart: | javac | 3m 3s | the patch passed | | +1 :green_heart: | shadedjars | 6m 27s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 0m 14s | root in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 15m 18s | root in the patch failed. | | | | 51m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1727/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1727 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5f010055db7d 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1727/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1727/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1727/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1727/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1727/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1727/1/testReport/ | | Max. process+thread count | 708 (vs. ulimit of 12500) | | modules | C: hbase-server . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1727/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] binlijin merged pull request #1721: HBASE-24377 MemStoreFlusher throw NullPointerException
binlijin merged pull request #1721: URL: https://github.com/apache/hbase/pull/1721 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24135) TableStateNotFoundException happends when table creation if rsgroup is enable
[
https://issues.apache.org/jira/browse/HBASE-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109748#comment-17109748
]
Hudson commented on HBASE-24135:
Results for branch branch-2
[build #2665 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2665/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2665/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2665/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2665/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2665/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> TableStateNotFoundException happends when table creation if rsgroup is enable
> -
>
> Key: HBASE-24135
> URL: https://issues.apache.org/jira/browse/HBASE-24135
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.3
>Reporter: Ajeet Rai
>Assignee: Sun Xin
>Priority: Minor
> Fix For: 2.3.0, 2.2.5
>
>
> IF RS group is enabled then Unable to get table state error is thrown while
> creating a new table
>
> Stpes:
> 1: Make sure RS group feature is enabled
> 2: Create a table say usertable2000
> 3: Check master log and observe below exception is thrown
> org.apache.hadoop.hbase.master.TableStateManager$TableStateNotFoundException:
> usertable2000
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24388) Introduce a 'local root region' at master side to store the location of meta table
[ https://issues.apache.org/jira/browse/HBASE-24388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109747#comment-17109747 ] Duo Zhang commented on HBASE-24388: --- I think at beginning we can only query the active master, but this can be changed later. For querying meta location on zk through masters, the masters just act as a cache for zk, so I think backup masters could also act as a cache for the active master. Can do this in later patches. > Introduce a 'local root region' at master side to store the location of meta > table > -- > > Key: HBASE-24388 > URL: https://issues.apache.org/jira/browse/HBASE-24388 > Project: HBase > Issue Type: Sub-task > Components: meta, Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24381) The Size metrics in Master Webui is wrong if the size is 0
[ https://issues.apache.org/jira/browse/HBASE-24381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-24381. Fix Version/s: 2.2.5 2.3.0 3.0.0-alpha-1 Resolution: Fixed Pushed to branch-2.2+. Thanks [~DeanZ] for contributing. > The Size metrics in Master Webui is wrong if the size is 0 > -- > > Key: HBASE-24381 > URL: https://issues.apache.org/jira/browse/HBASE-24381 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: 2.2.4 >Reporter: Baiqiang Zhao >Assignee: Baiqiang Zhao >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.5 > > Attachments: master-webui-size-wrong.png > > > As shown in attachment, there is no storefiles on the last RS, but the > StoreFile Size is as large as the previous RS. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-17756) We should have better introspection of HFiles
[ https://issues.apache.org/jira/browse/HBASE-17756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109723#comment-17109723 ] Michael Stack commented on HBASE-17756: --- If you want to take it up from here [~shahrs87], I'd suggest: * run the linked Performance Evaluation Test Tool for HFiles to see if the sketches slow writing; if it doesn't slow the writing, then lets make a subtask to commit recording of sketches at write time (any other things we should sketch?). * Then lets commit this to pretty printer. If you have ideas for how to make it prettier, just say * Then maybe make a new issue for the Region and Table Pretty Printers. > We should have better introspection of HFiles > - > > Key: HBASE-17756 > URL: https://issues.apache.org/jira/browse/HBASE-17756 > Project: HBase > Issue Type: Brainstorming > Components: HFile >Reporter: Esteban Gutierrez >Assignee: Rushabh Shah >Priority: Major > > [~saint@gmail.com] was suggesting to use DataSketches > (https://datasketches.github.io) in order to write additional statistics to > the HFiles. This could be used to improve our split decisions, > troubleshooting or potentially do other interesting analysis without having > to perform full table scans. The statistics could be stored as part of the > HFile but we could initially improve the visibility of the data by adding > some statistics to HFilePrettyPrinter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] infraio merged pull request #1723: HBASE-24381 The Size metrics in Master Webui is wrong if the size is 0
infraio merged pull request #1723: URL: https://github.com/apache/hbase/pull/1723 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-17756) We should have better introspection of HFiles
[ https://issues.apache.org/jira/browse/HBASE-17756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109719#comment-17109719 ] Michael Stack edited comment on HBASE-17756 at 5/18/20, 1:13 AM: - Pushed writing of key and value sketches. Of mild interest. Changed the pretty printer to dump out 100 quantiles on size with 50/75th, and 95th. Changed writer to use the sketches figuring averages. TODO: measure to see if this slows down writing. I think we need a RegionPrettyPrinter, like HFilePrettyPrinter, only it reads the Region row-wise accumulating row size and column count sketches. It would optionally run HFilePrettyPrinter per file in the Region summing the per file key and value sketches. It would write out stuff like min/max and then quantiles for row size, row count, key sizees, and value sizes. Would be easy to graph perhaps producing graph files. A TablePrettyPrinter would run a RegionPrettyPrinter per Region in the Table. was (Author: stack): Pushed writing of key and value sketches. Of mild interest. Changed the pretty printer to dump out 100 quantiles on size with 50/75th, and 95th. Changed writer to use the sketches figuring averages. TODO: measure to see if this slows down writing. I think we need a RegionPrettyPrinter, like HFilePrettyPrinter, only it reads the Region row-wise accumulating row size and column count sketches. It would optionally run HFilePrettyPrinter per file in the Region summing the per file key and value sketches. It would write out stuff like min/max and then quantiles for row size, row count, key sizees, and value sizes. Would be easy to graph perhaps producing graph files. > We should have better introspection of HFiles > - > > Key: HBASE-17756 > URL: https://issues.apache.org/jira/browse/HBASE-17756 > Project: HBase > Issue Type: Brainstorming > Components: HFile >Reporter: Esteban Gutierrez >Assignee: Rushabh Shah >Priority: Major > > [~saint@gmail.com] was suggesting to use DataSketches > (https://datasketches.github.io) in order to write additional statistics to > the HFiles. This could be used to improve our split decisions, > troubleshooting or potentially do other interesting analysis without having > to perform full table scans. The statistics could be stored as part of the > HFile but we could initially improve the visibility of the data by adding > some statistics to HFilePrettyPrinter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-17756) We should have better introspection of HFiles
[ https://issues.apache.org/jira/browse/HBASE-17756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109719#comment-17109719 ] Michael Stack commented on HBASE-17756: --- Pushed writing of key and value sketches. Of mild interest. Changed the pretty printer to dump out 100 quantiles on size with 50/75th, and 95th. Changed writer to use the sketches figuring averages. TODO: measure to see if this slows down writing. I think we need a RegionPrettyPrinter, like HFilePrettyPrinter, only it reads the Region row-wise accumulating row size and column count sketches. It would optionally run HFilePrettyPrinter per file in the Region summing the per file key and value sketches. It would write out stuff like min/max and then quantiles for row size, row count, key sizees, and value sizes. Would be easy to graph perhaps producing graph files. > We should have better introspection of HFiles > - > > Key: HBASE-17756 > URL: https://issues.apache.org/jira/browse/HBASE-17756 > Project: HBase > Issue Type: Brainstorming > Components: HFile >Reporter: Esteban Gutierrez >Assignee: Rushabh Shah >Priority: Major > > [~saint@gmail.com] was suggesting to use DataSketches > (https://datasketches.github.io) in order to write additional statistics to > the HFiles. This could be used to improve our split decisions, > troubleshooting or potentially do other interesting analysis without having > to perform full table scans. The statistics could be stored as part of the > HFile but we could initially improve the visibility of the data by adding > some statistics to HFilePrettyPrinter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1727: HBASE-17756 We should have better introspection of HFiles
Apache-HBase commented on pull request #1727: URL: https://github.com/apache/hbase/pull/1727#issuecomment-629893972 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | docker | 0m 1s | Docker command '/usr/bin/docker' is too old ( < 17.0). | | Subsystem | Report/Notes | |--:|:-| | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1727/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack opened a new pull request #1727: HBASE-17756 We should have better introspection of HFiles
saintstack opened a new pull request #1727: URL: https://github.com/apache/hbase/pull/1727 TODO: Check cost; Run HFilePE. hbase-server/pom.xml Add dependency on data sketches. hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java Print out key and value quantiles. hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java Accumulate sketches. Use sketches for counts and average key and value size. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1726: HBASE-24296 install yetus as a part of building the rm docker image.
Apache-HBase commented on pull request #1726: URL: https://github.com/apache/hbase/pull/1726#issuecomment-629892413 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1726: HBASE-24296 install yetus as a part of building the rm docker image.
Apache-HBase commented on pull request #1726: URL: https://github.com/apache/hbase/pull/1726#issuecomment-629892031 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | docker | 0m 1s | Docker command '/usr/bin/docker' is too old ( < 17.0). | | Subsystem | Report/Notes | |--:|:-| | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1726/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work started] (HBASE-24393) release scripts should allow for creation of initial CHANGES/RELEASE on new major version.
[
https://issues.apache.org/jira/browse/HBASE-24393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-24393 started by Sean Busbey.
---
> release scripts should allow for creation of initial CHANGES/RELEASE on new
> major version.
> --
>
> Key: HBASE-24393
> URL: https://issues.apache.org/jira/browse/HBASE-24393
> Project: HBase
> Issue Type: Bug
> Components: community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
>
> Right now if you try to run the release scripts on a new major version, e.g.
> one without a RELEASENOTES.md and CHANGES.md things fail:
> {code}
> /opt/hbase-rm/output
> + sed -i -e '/^## Release 3.0.0-alpha-1/,/^## Release/ {//!d; /^## Release
> 3.0.0-alpha-1/d;}' /opt/hbase-rm/output/hbase/CHANGES.md
> sed: can't read /opt/hbase-rm/output/hbase/CHANGES.md: No such file or
> directory
> + true
> + sed -i -e '/^# HBASE 3.0.0-alpha-1 Release Notes/,/^# HBASE/{//!d; /^#
> HBASE 3.0.0-alpha-1 Release Notes/d;}'
> /opt/hbase-rm/output/hbase/RELEASENOTES.md
> sed: can't read /opt/hbase-rm/output/hbase/RELEASENOTES.md: No such file or
> directory
> + true
> + sed -i -e '/## Release/,$!d' CHANGELOG.3.0.0-alpha-1.md
> + sed -i -e '/DO NOT REMOVE/r CHANGELOG.3.0.0-alpha-1.md'
> /opt/hbase-rm/output/hbase/CHANGES.md
> sed: can't read /opt/hbase-rm/output/hbase/CHANGES.md: No such file or
> directory
> + cleanup
> {code}
> instead we should just create the new files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24393) release scripts should allow for creation of initial CHANGES/RELEASE on new major version.
Sean Busbey created HBASE-24393:
---
Summary: release scripts should allow for creation of initial
CHANGES/RELEASE on new major version.
Key: HBASE-24393
URL: https://issues.apache.org/jira/browse/HBASE-24393
Project: HBase
Issue Type: Bug
Components: community
Reporter: Sean Busbey
Assignee: Sean Busbey
Right now if you try to run the release scripts on a new major version, e.g.
one without a RELEASENOTES.md and CHANGES.md things fail:
{code}
/opt/hbase-rm/output
+ sed -i -e '/^## Release 3.0.0-alpha-1/,/^## Release/ {//!d; /^## Release
3.0.0-alpha-1/d;}' /opt/hbase-rm/output/hbase/CHANGES.md
sed: can't read /opt/hbase-rm/output/hbase/CHANGES.md: No such file or directory
+ true
+ sed -i -e '/^# HBASE 3.0.0-alpha-1 Release Notes/,/^# HBASE/{//!d; /^# HBASE
3.0.0-alpha-1 Release Notes/d;}' /opt/hbase-rm/output/hbase/RELEASENOTES.md
sed: can't read /opt/hbase-rm/output/hbase/RELEASENOTES.md: No such file or
directory
+ true
+ sed -i -e '/## Release/,$!d' CHANGELOG.3.0.0-alpha-1.md
+ sed -i -e '/DO NOT REMOVE/r CHANGELOG.3.0.0-alpha-1.md'
/opt/hbase-rm/output/hbase/CHANGES.md
sed: can't read /opt/hbase-rm/output/hbase/CHANGES.md: No such file or directory
+ cleanup
{code}
instead we should just create the new files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24296) release scripts should install yetus via the docker build
[ https://issues.apache.org/jira/browse/HBASE-24296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-24296: Status: Patch Available (was: In Progress) > release scripts should install yetus via the docker build > - > > Key: HBASE-24296 > URL: https://issues.apache.org/jira/browse/HBASE-24296 > Project: HBase > Issue Type: Improvement > Components: community, scripts >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > right now we have to download yetus on each release run. we should be able to > point at a local install instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] busbey opened a new pull request #1726: HBASE-24296 install yetus as a part of building the rm docker image.
busbey opened a new pull request #1726: URL: https://github.com/apache/hbase/pull/1726 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #1726: HBASE-24296 install yetus as a part of building the rm docker image.
busbey commented on pull request #1726: URL: https://github.com/apache/hbase/pull/1726#issuecomment-629890495 mind taking a look @mattf-apache ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
busbey commented on pull request #1620: URL: https://github.com/apache/hbase/pull/1620#issuecomment-629885166 shellcheck and whitespace complaints should be easy enough to clean up. will get to it after I get PRs up for all the commits on the branch that are not this particular jira. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
busbey commented on a change in pull request #1620: URL: https://github.com/apache/hbase/pull/1620#discussion_r426321691 ## File path: dev-support/create-release/release-build.sh ## @@ -79,17 +78,7 @@ if [[ -z "$ASF_PASSWORD" ]]; then stty -echo && printf "ASF password: " && read ASF_PASSWORD && printf '\n' && stty echo fi -# Read in the GPG passphrase -if [[ -z "$GPG_PASSPHRASE" ]]; then - echo 'The environment variable GPG_PASSPHRASE is not set. Enter the passphrase to' - echo 'unlock the GPG signing key that will be used to sign the release!' - echo - stty -echo && printf "GPG passphrase: " && read GPG_PASSPHRASE && printf '\n' && stty echo - export GPG_PASSPHRASE - export GPG_TTY=$(tty) -fi - -for env in ASF_USERNAME GPG_PASSPHRASE GPG_KEY; do Review comment: yep! no more GPG_PASSPHRASE because that should get specified by you to your local gpg-agent. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
busbey commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r426321658
##
File path: dev-support/create-release/do-release-docker.sh
##
@@ -151,21 +200,53 @@ GIT_NAME=$GIT_NAME
GIT_EMAIL=$GIT_EMAIL
GPG_KEY=$GPG_KEY
ASF_PASSWORD=$ASF_PASSWORD
-GPG_PASSPHRASE=$GPG_PASSPHRASE
RELEASE_STEP=$RELEASE_STEP
RELEASE_STEP=$RELEASE_STEP
API_DIFF_TAG=$API_DIFF_TAG
EOF
-JAVA_VOL=
+JAVA_MOUNT=()
if [ -n "$JAVA" ]; then
echo "JAVA_HOME=/opt/hbase-java" >> "$ENVFILE"
- JAVA_VOL="--volume $JAVA:/opt/hbase-java"
+ JAVA_MOUNT=(--mount "type=bind,src=${JAVA},dest=/opt/hbase-java,readonly")
+fi
+
+GPG_PROXY_MOUNT=()
+if [ "${HOST_OS}" == "DARWIN" ]; then
+ GPG_PROXY_MOUNT=(--mount
"type=volume,src=gpgagent,dst=/home/${USER}/.gnupg/")
+ echo "Setting up GPG agent proxy container needed on OS X."
+ echo " we should clean this up for you. If that fails the container ID
is below and in " \
+ "gpg-proxy.cid"
+ #TODO the key pair used should be configurable
+ docker run --rm -p 6:22 \
+ --detach --cidfile "${WORKDIR}/gpg-proxy.cid" \
+ --mount \
+
"type=bind,src=${HOME}/.ssh/id_rsa.pub,dst=/home/${USER}/.ssh/authorized_keys,readonly"
\
+ "${GPG_PROXY_MOUNT[@]}" \
+ "org.apache.hbase/gpg-agent-proxy:${IMGTAG}"
+ echo "Launching ssh reverse tunnel from the container to gpg agent."
+ echo " we should clean this up for you. If that fails the PID is in
gpg-proxy.ssh.pid"
+ ssh -p 6 -R "/home/${USER}/.gnupg/S.gpg-agent:$(gpgconf --list-dir
agent-extra-socket)" \
+ -i "${HOME}/.ssh/id_rsa" -N -n localhost &
+ echo $! > "${WORKDIR}/gpg-proxy.ssh.pid"
+else
+ # TODO this presumes we are still trying to make a local gpg-agent available
to the container.
+ # add an option so that we can run the buid on a remote machine and get
the forwarded
+ # gpg-agent in the container. Should look like the side-car container
mount above.
+ # it is important not to do that for a local linux agent because we
only want the container
+ # to get access to the restricted extra socket on our local gpg-agent.
+ GPG_PROXY_MOUNT=(--mount \
+ "type=bind,src=$(gpgconf --list-dir
agent-extra-socket),dst=/home/${USER}/.gnupg/S.gpg-agent")
Review comment:
yeah. I think I can make this less complicated by giving docs on how to
proxy your gpg-agent to a remote host. gonna work through that next now that I
have local execution on my mac working.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
busbey commented on a change in pull request #1620:
URL: https://github.com/apache/hbase/pull/1620#discussion_r426321582
##
File path: dev-support/create-release/do-release-docker.sh
##
@@ -102,12 +102,26 @@ if [ -d "$WORKDIR/output" ]; then
fi
fi
+if [ -f "${WORKDIR}/gpg-proxy.ssh.pid" ] || \
+ [ -f "${WORKDIR}/gpg-proxy.cid" ] || \
+ [ -f "${WORKDIR}/release.cid" ]; then
+ read -r -p "container/pid files from prior run exists. Overwrite and
continue? [y/n] " ANSWER
+ if [ "$ANSWER" != "y" ]; then
+error "Exiting."
+ fi
+fi
+
cd "$WORKDIR"
rm -rf "$WORKDIR/output"
+rm -rf "${WORKDIR}/gpg-proxy.ssh.pid" "${WORKDIR}/gpg-proxy.cid"
"${WORKDIR}/release.cid"
Review comment:
so the clean up function should take care of them, but something can go
wrong such that they don't get handled.
since docker will fail loudly if the passed container id file path exists,
this ensures we have a smooth path back to running again even if something goes
wrong.
also the latest version of the WIP has an option to purposefully leave those
files in place at the end of execution in case someone needs to debug what the
containers are doing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
busbey commented on pull request #1620: URL: https://github.com/apache/hbase/pull/1620#issuecomment-629884049 sorry for the lag here. missed that I had comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1725: HBASE-24297 release scripts should be able to use an existing project clone
Apache-HBase commented on pull request #1725: URL: https://github.com/apache/hbase/pull/1725#issuecomment-629877190 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 1s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch | | -0 :warning: | shellcheck | 0m 3s | The patch generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? | | | | 2m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1725/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1725 | | Optional Tests | dupname asflicense shellcheck shelldocs | | uname | Linux 72d599c81e71 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | shellcheck | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1725/1/artifact/yetus-general-check/output/diff-patch-shellcheck.txt | | Max. process+thread count | 48 (vs. ulimit of 12500) | | modules | C: U: | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1725/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1725: HBASE-24297 release scripts should be able to use an existing project clone
Apache-HBase commented on pull request #1725: URL: https://github.com/apache/hbase/pull/1725#issuecomment-629877289 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24297) release scripts should be able to use an existing project checkout
[ https://issues.apache.org/jira/browse/HBASE-24297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-24297: Status: Patch Available (was: Open) > release scripts should be able to use an existing project checkout > -- > > Key: HBASE-24297 > URL: https://issues.apache.org/jira/browse/HBASE-24297 > Project: HBase > Issue Type: Improvement > Components: community, scripts >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > doing a full new clone is expensive, especially for the main repo. we should > be able to optionally point at an existing clone. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-24297) release scripts should be able to use an existing project checkout
[ https://issues.apache.org/jira/browse/HBASE-24297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reassigned HBASE-24297: --- Assignee: Sean Busbey > release scripts should be able to use an existing project checkout > -- > > Key: HBASE-24297 > URL: https://issues.apache.org/jira/browse/HBASE-24297 > Project: HBase > Issue Type: Improvement > Components: community, scripts >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > doing a full new clone is expensive, especially for the main repo. we should > be able to optionally point at an existing clone. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] busbey opened a new pull request #1725: HBASE-24297 release scripts should be able to use an existing project clone
busbey opened a new pull request #1725: URL: https://github.com/apache/hbase/pull/1725 Adds the ability to specify the remote git repo for the release scripts, including a local clone. * adds a optional `-r [repo]` arg * if the passed repo is on the local filesystem, creates a container mount * git clone operations consolidated into a function * when cloning a local repo configure git to share objects with the local repo instead of copying * when cloning a local repo in a container configure the clone to have a remote that will work back on the host. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #1725: HBASE-24297 release scripts should be able to use an existing project clone
busbey commented on pull request #1725: URL: https://github.com/apache/hbase/pull/1725#issuecomment-629875827 @mattf-apache mind taking a look? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24135) TableStateNotFoundException happends when table creation if rsgroup is enable
[
https://issues.apache.org/jira/browse/HBASE-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109698#comment-17109698
]
Hudson commented on HBASE-24135:
Results for branch branch-2.2
[build #866 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/866/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/866//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/866//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/866//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> TableStateNotFoundException happends when table creation if rsgroup is enable
> -
>
> Key: HBASE-24135
> URL: https://issues.apache.org/jira/browse/HBASE-24135
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.3
>Reporter: Ajeet Rai
>Assignee: Sun Xin
>Priority: Minor
> Fix For: 2.3.0, 2.2.5
>
>
> IF RS group is enabled then Unable to get table state error is thrown while
> creating a new table
>
> Stpes:
> 1: Make sure RS group feature is enabled
> 2: Create a table say usertable2000
> 3: Check master log and observe below exception is thrown
> org.apache.hadoop.hbase.master.TableStateManager$TableStateNotFoundException:
> usertable2000
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109694#comment-17109694 ] Andrew Kyle Purtell commented on HBASE-11288: - I agree a feature branch would be good. I came by the PR twice last week and both times had to move on because it was too much to look at... > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Apache-HBase commented on pull request #1620: URL: https://github.com/apache/hbase/pull/1620#issuecomment-629861697 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 7s | Maven dependency ordering for patch | | +1 :green_heart: | hadolint | 0m 1s | There were no new hadolint issues. | | -0 :warning: | shellcheck | 0m 4s | The patch generated 6 new + 5 unchanged - 0 fixed = 11 total (was 5) | | -0 :warning: | whitespace | 0m 0s | The patch 6 line(s) with tabs. | ||| _ Other Tests _ | | +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? | | | | 3m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1620 | | Optional Tests | dupname asflicense shellcheck shelldocs hadolint | | uname | Linux 27abd214f267 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | shellcheck | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/5/artifact/yetus-general-check/output/diff-patch-shellcheck.txt | | whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/5/artifact/yetus-general-check/output/whitespace-tabs.txt | | Max. process+thread count | 52 (vs. ulimit of 12500) | | modules | C: U: | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 hadolint=1.17.5-0-g443423c | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Apache-HBase commented on pull request #1620: URL: https://github.com/apache/hbase/pull/1620#issuecomment-629861490 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 7s | Maven dependency ordering for patch | ||| _ Other Tests _ | | | | 1m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1620 | | Optional Tests | | | uname | Linux 726fd3d0fe94 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Max. process+thread count | 52 (vs. ulimit of 12500) | | modules | C: U: | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1620: HBASE-23339 Release scripts should not need to write out a copy of gpg key material - WIP do not merge
Apache-HBase commented on pull request #1620: URL: https://github.com/apache/hbase/pull/1620#issuecomment-629861601 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch | ||| _ Other Tests _ | | | | 2m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1620 | | Optional Tests | | | uname | Linux daa0ec5d60ed 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Max. process+thread count | 45 (vs. ulimit of 12500) | | modules | C: U: | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1620/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1681: HBASE-23938 : System table hbase:slowlog to store complete slow/large…
Apache-HBase commented on pull request #1681: URL: https://github.com/apache/hbase/pull/1681#issuecomment-629840560 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 56s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 58s | master passed | | +1 :green_heart: | compile | 1m 49s | master passed | | +1 :green_heart: | shadedjars | 6m 23s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 26s | master passed | | -0 :warning: | patch | 8m 18s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 50s | the patch passed | | +1 :green_heart: | compile | 1m 51s | the patch passed | | +1 :green_heart: | javac | 1m 51s | the patch passed | | +1 :green_heart: | shadedjars | 5m 59s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 28s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 5s | hbase-client in the patch passed. | | -1 :x: | unit | 192m 59s | hbase-server in the patch failed. | | | | 227m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/7/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1681 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1a3dfdc3c0d4 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/7/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/7/testReport/ | | Max. process+thread count | 3273 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/7/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24388) Introduce a 'local root region' at master side to store the location of meta table
[ https://issues.apache.org/jira/browse/HBASE-24388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109588#comment-17109588 ] Anoop Sam John commented on HBASE-24388: In master branch, now we have MasterRegistry where the META location is available at every HM (Active, Standby). Now when we have the root region which will be having the META regions info, if that is available only in Active HM, that will against this other solution's direction (?). Just raised for a brain storming. > Introduce a 'local root region' at master side to store the location of meta > table > -- > > Key: HBASE-24388 > URL: https://issues.apache.org/jira/browse/HBASE-24388 > Project: HBase > Issue Type: Sub-task > Components: meta, Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1681: HBASE-23938 : System table hbase:slowlog to store complete slow/large…
Apache-HBase commented on pull request #1681: URL: https://github.com/apache/hbase/pull/1681#issuecomment-629830668 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 55s | master passed | | +1 :green_heart: | compile | 1m 55s | master passed | | +1 :green_heart: | shadedjars | 5m 46s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 17s | hbase-common in master failed. | | -0 :warning: | javadoc | 0m 26s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 38s | hbase-server in master failed. | | -0 :warning: | patch | 7m 42s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 0s | the patch passed | | +1 :green_heart: | compile | 1m 53s | the patch passed | | +1 :green_heart: | javac | 1m 53s | the patch passed | | +1 :green_heart: | shadedjars | 5m 41s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 18s | hbase-common in the patch failed. | | -0 :warning: | javadoc | 0m 25s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 41s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 34s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 3s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 122m 9s | hbase-server in the patch passed. | | | | 154m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/7/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1681 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4e972a110fc8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/7/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/7/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/7/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/7/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/7/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/7/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/7/testReport/ | | Max. process+thread count | 4510 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/7/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1708: HBASE-19577 Use log4j2 instead of log4j for logging
Apache-HBase commented on pull request #1708: URL: https://github.com/apache/hbase/pull/1708#issuecomment-629824778 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 6s | master passed | | +1 :green_heart: | compile | 2m 44s | master passed | | +1 :green_heart: | shadedjars | 5m 43s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 16s | hbase-logging in master failed. | | -0 :warning: | javadoc | 0m 15s | root in master failed. | | -0 :warning: | javadoc | 0m 15s | hbase-client-project in master failed. | | -0 :warning: | javadoc | 0m 15s | hbase-shaded-client-project in master failed. | | -0 :warning: | javadoc | 0m 16s | hbase-asyncfs in master failed. | | -0 :warning: | javadoc | 0m 18s | hbase-backup in master failed. | | -0 :warning: | javadoc | 0m 16s | hbase-balancer in master failed. | | -0 :warning: | javadoc | 0m 24s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 16s | hbase-common in master failed. | | -0 :warning: | javadoc | 0m 17s | hbase-examples in master failed. | | -0 :warning: | javadoc | 0m 18s | hbase-hadoop-compat in master failed. | | -0 :warning: | javadoc | 0m 16s | hbase-hbtop in master failed. | | -0 :warning: | javadoc | 0m 16s | hbase-http in master failed. | | -0 :warning: | javadoc | 0m 18s | hbase-mapreduce in master failed. | | -0 :warning: | javadoc | 0m 15s | hbase-metrics in master failed. | | -0 :warning: | javadoc | 0m 15s | hbase-metrics-api in master failed. | | -0 :warning: | javadoc | 0m 17s | hbase-procedure in master failed. | | -0 :warning: | javadoc | 0m 15s | hbase-replication in master failed. | | -0 :warning: | javadoc | 0m 18s | hbase-rest in master failed. | | -0 :warning: | javadoc | 0m 38s | hbase-server in master failed. | | -0 :warning: | javadoc | 0m 49s | hbase-thrift in master failed. | | -0 :warning: | javadoc | 0m 15s | hbase-zookeeper in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 7s | the patch passed | | +1 :green_heart: | compile | 2m 52s | the patch passed | | +1 :green_heart: | javac | 2m 52s | the patch passed | | +1 :green_heart: | shadedjars | 6m 43s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 14s | hbase-logging in the patch failed. | | -0 :warning: | javadoc | 0m 17s | hbase-common in the patch failed. | | -0 :warning: | javadoc | 0m 15s | hbase-metrics-api in the patch failed. | | -0 :warning: | javadoc | 0m 15s | hbase-metrics in the patch failed. | | -0 :warning: | javadoc | 0m 19s | hbase-hadoop-compat in the patch failed. | | -0 :warning: | javadoc | 0m 25s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 15s | hbase-zookeeper in the patch failed. | | -0 :warning: | javadoc | 0m 16s | hbase-replication in the patch failed. | | -0 :warning: | javadoc | 0m 16s | hbase-balancer in the patch failed. | | -0 :warning: | javadoc | 0m 16s | hbase-http in the patch failed. | | -0 :warning: | javadoc | 0m 17s | hbase-asyncfs in the patch failed. | | -0 :warning: | javadoc | 0m 17s | hbase-procedure in the patch failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 0m 20s | hbase-mapreduce in the patch failed. | | -0 :warning: | javadoc | 0m 49s | hbase-thrift in the patch failed. | | -0 :warning: | javadoc | 0m 19s | hbase-backup in the patch failed. | | -0 :warning: | javadoc | 0m 20s | hbase-rest in the patch failed. | | -0 :warning: | javadoc | 0m 17s | hbase-examples in the patch failed. | | -0 :warning: | javadoc | 0m 16s | hbase-hbtop in the patch failed. | | -0 :warning: | javadoc | 0m 16s | hbase-client-project in the patch failed. | | -0 :warning: | javadoc | 0m 14s | hbase-shaded-client-project in the patch failed. | | -0 :warning: | javadoc | 0m 15s | root in the patch failed. | ||| _ Other Tests _ | | +1
[jira] [Created] (HBASE-24392) Make sure we can rollingupgrade from a non-splittable meta to splittable meta
Duo Zhang created HBASE-24392: - Summary: Make sure we can rollingupgrade from a non-splittable meta to splittable meta Key: HBASE-24392 URL: https://issues.apache.org/jira/browse/HBASE-24392 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang Maybe we should set a DisabledSplitPolicy on existing meta table to prevent it being split, so we could still store the location of meta on zk so old client can still communicate with us. And later once all the old clients are also upgraded, we remove the policy from meta table to make it splittable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1681: HBASE-23938 : System table hbase:slowlog to store complete slow/large…
Apache-HBase commented on pull request #1681: URL: https://github.com/apache/hbase/pull/1681#issuecomment-629814579 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 19s | master passed | | +1 :green_heart: | checkstyle | 2m 0s | master passed | | +1 :green_heart: | spotbugs | 3m 31s | master passed | | -0 :warning: | patch | 2m 18s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 20s | the patch passed | | -0 :warning: | checkstyle | 0m 28s | hbase-client: The patch generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) | | -0 :warning: | checkstyle | 1m 7s | hbase-server: The patch generated 4 new + 150 unchanged - 0 fixed = 154 total (was 150) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 9s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 4m 7s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. | | | | 38m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/7/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1681 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 5c077c7d2c96 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/7/artifact/yetus-general-check/output/diff-checkstyle-hbase-client.txt | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/7/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1681/7/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24391) Implement meta split
Duo Zhang created HBASE-24391: - Summary: Implement meta split Key: HBASE-24391 URL: https://issues.apache.org/jira/browse/HBASE-24391 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang Need to modify Split/Merge procedures to deal with meta, and also remove all the hard coded non-splittable check for meta region. And we also need to think of the split policy for meta table, i.e, whether it is safe tosplit at any point. For safety, maybe we could first use a special split policy to make sure that all records for a table are in the same region. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24390) Remove RegionInfoBuilder.FIRST_META_REGIONINFO
Duo Zhang created HBASE-24390: - Summary: Remove RegionInfoBuilder.FIRST_META_REGIONINFO Key: HBASE-24390 URL: https://issues.apache.org/jira/browse/HBASE-24390 Project: HBase Issue Type: Sub-task Components: meta Reporter: Duo Zhang This is very strong assumption that there is only one meta region, we should cleanup all its usage and remove it from our code base. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24135) TableStateNotFoundException happends when table creation if rsgroup is enable
[ https://issues.apache.org/jira/browse/HBASE-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-24135. --- Hadoop Flags: Reviewed Resolution: Fixed Pushed to branch-2, branch-2.3 and branch-2.2. Thanks [~Ddupg] for contributing and [~binlijin] for reviewing. > TableStateNotFoundException happends when table creation if rsgroup is enable > - > > Key: HBASE-24135 > URL: https://issues.apache.org/jira/browse/HBASE-24135 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.3 >Reporter: Ajeet Rai >Assignee: Sun Xin >Priority: Minor > Fix For: 2.3.0, 2.2.5 > > > IF RS group is enabled then Unable to get table state error is thrown while > creating a new table > > Stpes: > 1: Make sure RS group feature is enabled > 2: Create a table say usertable2000 > 3: Check master log and observe below exception is thrown > org.apache.hadoop.hbase.master.TableStateManager$TableStateNotFoundException: > usertable2000 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24389) Introduce a new master rpc service to locate meta region through root region
Duo Zhang created HBASE-24389: - Summary: Introduce a new master rpc service to locate meta region through root region Key: HBASE-24389 URL: https://issues.apache.org/jira/browse/HBASE-24389 Project: HBase Issue Type: Sub-task Components: master, meta Reporter: Duo Zhang And also reimlement the client logic to use the new api to locate meta region. In general, this will remove the assumption at client side that there is only one meta region. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] virajjasani commented on a change in pull request #1724: HBASE-24386 TableSnapshotScanner support scan limit
virajjasani commented on a change in pull request #1724:
URL: https://github.com/apache/hbase/pull/1724#discussion_r426269014
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTableSnapshotScanner.java
##
@@ -190,6 +195,36 @@ public void testNoDuplicateResultsWhenSplitting() throws
Exception {
}
}
+
+ @Test
+ public void testScanLimit() throws Exception {
+setupCluster();
+final TableName tableName = TableName.valueOf(name.getMethodName());
+final String snapshotName = tableName + "Snapshot";
+try {
+ createTableAndSnapshot(UTIL, tableName, snapshotName, 50);
+ Path restoreDir = UTIL.getDataTestDirOnTestFS(snapshotName);
+ Scan scan = new Scan().withStartRow(bbb).setLimit(100); // limit the scan
+
+ TableSnapshotScanner scanner =
+ new TableSnapshotScanner(UTIL.getConfiguration(), restoreDir,
snapshotName, scan);
+ int count = 0;
+ while (true) {
+Result result = scanner.next();
+if (result == null) {
+ break;
+}
+count++;
+ }
+ Assert.assertEquals(100, count);
+ scanner.close();
Review comment:
Can we move `scanner.close()` in finally? We can definte
`TableSnapshotScanner scanner=null` outside of try {}, and in finally, we can
check if scanner is not null, then close it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on a change in pull request #1681: HBASE-23938 : System table hbase:slowlog to store complete slow/large…
virajjasani commented on a change in pull request #1681:
URL: https://github.com/apache/hbase/pull/1681#discussion_r426264941
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/slowlog/LogEventHandler.java
##
@@ -53,12 +55,28 @@
class LogEventHandler implements EventHandler {
private static final Logger LOG =
LoggerFactory.getLogger(LogEventHandler.class);
+ private static final int SYS_TABLE_QUEUE_SIZE = 1000;
- private final Queue queue;
+ private final Queue queueForRingBuffer;
+ private final Queue queueForSysTable;
+ private final boolean isSlowLogTableEnabled;
- LogEventHandler(int eventCount) {
+ private Configuration configuration;
+
+ private static final ReentrantLock LOCK = new ReentrantLock();
+
+ LogEventHandler(int eventCount, boolean isSlowLogTableEnabled, Configuration
conf) {
+this.configuration = conf;
EvictingQueue evictingQueue =
EvictingQueue.create(eventCount);
-queue = Queues.synchronizedQueue(evictingQueue);
+queueForRingBuffer = Queues.synchronizedQueue(evictingQueue);
+this.isSlowLogTableEnabled = isSlowLogTableEnabled;
+if (isSlowLogTableEnabled) {
+ EvictingQueue evictingQueueForTable =
EvictingQueue.create(
Review comment:
One Q is literally managed by user only and it's purpose is to server
online slowlogs from memory. Another Q is for cron to insert records in system
table in batch, hence both should not be same. Even if user clears the Q, the
one intended is in-memory ring buffer Q. If user has opted for system table,
the other Q is totally managed internally and not upto user to manage. User
might also opt for lower size for config
`hbase.regionserver.slowlog.ringbuffer.size` to say 100/50. But there is no
guarantee that within 10 min of cron run, the no of slow RPC calls would be
less than 50, and hence better to have another Q for systable maintenance.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on a change in pull request #1681: HBASE-23938 : System table hbase:slowlog to store complete slow/large…
virajjasani commented on a change in pull request #1681:
URL: https://github.com/apache/hbase/pull/1681#discussion_r426263602
##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
##
@@ -1532,6 +1532,16 @@
"hbase.regionserver.slowlog.buffer.enabled";
public static final boolean DEFAULT_ONLINE_LOG_PROVIDER_ENABLED = false;
+ /** The slowlog info family as a string*/
+ private static final String SLOWLOG_INFO_FAMILY_STR = "info";
+
+ /** The slowlog info family */
+ public static final byte [] SLOWLOG_INFO_FAMILY =
Bytes.toBytes(SLOWLOG_INFO_FAMILY_STR);
+
+ public static final String SLOW_LOG_SYS_TABLE_ENABLED_KEY =
+"hbase.regionserver.slowlog.systable.enabled";
+ public static final boolean DEFAULT_SLOW_LOG_SYS_TABLE_ENABLED_KEY = false;
Review comment:
Yes, this is region name that comes as part of req param. We do record
it, but in ringbuffer and systable, we record the full name, whereas in
RpcServer log, we record truncated name. The truncation was one of the first
reasons behind the idea of online ringbuffer implementation.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/slowlog/LogEventHandler.java
##
@@ -53,12 +55,28 @@
class LogEventHandler implements EventHandler {
private static final Logger LOG =
LoggerFactory.getLogger(LogEventHandler.class);
+ private static final int SYS_TABLE_QUEUE_SIZE = 1000;
Review comment:
That will conflict with the existing one. We have
`hbase.regionserver.slowlog.ringbuffer.size` but this is only related to in
memory ring buffer and has nothing to do with system table. If you believe we
can increase this size, we can do it. It is rough estimate that within 10 min,
we can have slow/large logs count within 1000.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/slowlog/LogEventHandler.java
##
@@ -53,12 +55,28 @@
class LogEventHandler implements EventHandler {
private static final Logger LOG =
LoggerFactory.getLogger(LogEventHandler.class);
+ private static final int SYS_TABLE_QUEUE_SIZE = 1000;
- private final Queue queue;
+ private final Queue queueForRingBuffer;
+ private final Queue queueForSysTable;
+ private final boolean isSlowLogTableEnabled;
- LogEventHandler(int eventCount) {
+ private Configuration configuration;
+
+ private static final ReentrantLock LOCK = new ReentrantLock();
+
+ LogEventHandler(int eventCount, boolean isSlowLogTableEnabled, Configuration
conf) {
+this.configuration = conf;
EvictingQueue evictingQueue =
EvictingQueue.create(eventCount);
-queue = Queues.synchronizedQueue(evictingQueue);
+queueForRingBuffer = Queues.synchronizedQueue(evictingQueue);
+this.isSlowLogTableEnabled = isSlowLogTableEnabled;
+if (isSlowLogTableEnabled) {
+ EvictingQueue evictingQueueForTable =
EvictingQueue.create(
Review comment:
One Q is literally managed by user only and it's purpose is to server
online slowlogs from memory. Another Q is for cron to insert records in system
table, hence both should not be same. Even if user clears the Q, the one
intended is in-memory ring buffer Q. If user has opted for system table, the
other Q is totally managed internally and not upto user to manage. User might
also opt for lower size for config `hbase.regionserver.slowlog.ringbuffer.size`
to say 100/50. But there is no guarantee that within 10 min of cron run, the no
of slow RPC calls would be less than 50, and hence better to have another Q for
systable maintenance.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/slowlog/LogEventHandler.java
##
@@ -160,7 +183,7 @@ boolean clearSlowLogs() {
if (LOG.isDebugEnabled()) {
LOG.debug("Received request to clean up online slowlog buffer..");
}
-queue.clear();
+queueForRingBuffer.clear();
Review comment:
Yes, this is intended since another Q is quite internal for systable
insertion by cron.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/slowlog/LogEventHandler.java
##
@@ -129,7 +147,12 @@ public void onEvent(RingBufferEnvelope event, long
sequence, boolean endOfBatch)
.setType(type)
.setUserName(userName)
.build();
-queue.add(slowLogPayload);
+queueForRingBuffer.add(slowLogPayload);
Review comment:
Actually this is for in-memory ring buffer and has nothing to do with
below check. `isSlowLogTableEnabled` is for `queueForSysTable` only. If the
execution comes till this point, that means we do at least need to create
in-memory ring buffer and that one is `queueForRingBuffer` (this is previous
feature).
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/slowlog/SlowLogTableAccessor.java
##
@@ -0,0 +1,140 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation
[GitHub] [hbase] Apache-HBase commented on pull request #1724: HBASE-24386 TableSnapshotScanner support scan limit
Apache-HBase commented on pull request #1724: URL: https://github.com/apache/hbase/pull/1724#issuecomment-629802754 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 57s | master passed | | +1 :green_heart: | compile | 0m 55s | master passed | | +1 :green_heart: | shadedjars | 5m 33s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | compile | 0m 55s | the patch passed | | +1 :green_heart: | javac | 0m 55s | the patch passed | | +1 :green_heart: | shadedjars | 5m 34s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 131m 54s | hbase-server in the patch passed. | | | | 156m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1724 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4bc9505232db 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/3/testReport/ | | Max. process+thread count | 4154 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1724: HBASE-24386 TableSnapshotScanner support scan limit
Apache-HBase commented on pull request #1724: URL: https://github.com/apache/hbase/pull/1724#issuecomment-629801738 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 55s | master passed | | +1 :green_heart: | compile | 1m 4s | master passed | | +1 :green_heart: | shadedjars | 5m 48s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 40s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 1s | the patch passed | | +1 :green_heart: | compile | 1m 4s | the patch passed | | +1 :green_heart: | javac | 1m 4s | the patch passed | | +1 :green_heart: | shadedjars | 5m 44s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 38s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 123m 28s | hbase-server in the patch passed. | | | | 149m 1s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1724 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6983bff6d8b9 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/3/testReport/ | | Max. process+thread count | 4106 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24387) TableSnapshotInputFormatImpl support scan limit on each InputSplit
[ https://issues.apache.org/jira/browse/HBASE-24387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-24387: - Description: In some scenario , We want to scan limited rows on each InputSplit for sampling data extraction (was: In some scenario , We want to scan limited rows on each InputSplit for sampling verification) > TableSnapshotInputFormatImpl support scan limit on each InputSplit > -- > > Key: HBASE-24387 > URL: https://issues.apache.org/jira/browse/HBASE-24387 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: niuyulin >Assignee: niuyulin >Priority: Major > > In some scenario , We want to scan limited rows on each InputSplit for > sampling data extraction -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-24388) Introduce a 'local root region' at master side to store the location of meta table
[ https://issues.apache.org/jira/browse/HBASE-24388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-24388 started by Duo Zhang. - > Introduce a 'local root region' at master side to store the location of meta > table > -- > > Key: HBASE-24388 > URL: https://issues.apache.org/jira/browse/HBASE-24388 > Project: HBase > Issue Type: Sub-task > Components: meta, Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-24388) Introduce a 'local root region' at master side to store the location of meta table
[ https://issues.apache.org/jira/browse/HBASE-24388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-24388: - Assignee: Duo Zhang > Introduce a 'local root region' at master side to store the location of meta > table > -- > > Key: HBASE-24388 > URL: https://issues.apache.org/jira/browse/HBASE-24388 > Project: HBase > Issue Type: Sub-task > Components: meta, Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24388) Introduce a 'local root region' at master side to store the location of meta table
Duo Zhang created HBASE-24388: - Summary: Introduce a 'local root region' at master side to store the location of meta table Key: HBASE-24388 URL: https://issues.apache.org/jira/browse/HBASE-24388 Project: HBase Issue Type: Sub-task Components: meta, Region Assignment Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-11288: -- Component/s: meta > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-11288: -- Parent: (was: HBASE-11165) Issue Type: Umbrella (was: Sub-task) > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109469#comment-17109469 ] Duo Zhang commented on HBASE-11288: --- The PR is too big for reviewing... Anyway I see lots of code for dealing with root region, for example, init root region, and wait root region to be assigned, and also extra stages for SCP to split the log for root region first. If we use the 'local root region', these are not needed any more. If no big concerns, I will create sub tasks here to implement the feature on a feature branch. Thanks [~toffer] for the nice write up. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24135) TableStateNotFoundException happends when table creation if rsgroup is enable
[ https://issues.apache.org/jira/browse/HBASE-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-24135: -- Fix Version/s: 2.2.5 2.3.0 > TableStateNotFoundException happends when table creation if rsgroup is enable > - > > Key: HBASE-24135 > URL: https://issues.apache.org/jira/browse/HBASE-24135 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.3 >Reporter: Ajeet Rai >Assignee: Sun Xin >Priority: Minor > Fix For: 2.3.0, 2.2.5 > > > IF RS group is enabled then Unable to get table state error is thrown while > creating a new table > > Stpes: > 1: Make sure RS group feature is enabled > 2: Create a table say usertable2000 > 3: Check master log and observe below exception is thrown > org.apache.hadoop.hbase.master.TableStateManager$TableStateNotFoundException: > usertable2000 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1708: HBASE-19577 Use log4j2 instead of log4j for logging
Apache-HBase commented on pull request #1708: URL: https://github.com/apache/hbase/pull/1708#issuecomment-629791993 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 merged pull request #1550: HBASE-24135 TableStateNotFoundException happends when table creation if rsgroup is enable
Apache9 merged pull request #1550: URL: https://github.com/apache/hbase/pull/1550 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1724: HBASE-24386 TableSnapshotScanner support scan limit
Apache-HBase commented on pull request #1724: URL: https://github.com/apache/hbase/pull/1724#issuecomment-629784676 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 20s | master passed | | +1 :green_heart: | checkstyle | 1m 3s | master passed | | +1 :green_heart: | spotbugs | 1m 55s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 19s | the patch passed | | +1 :green_heart: | checkstyle | 1m 2s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 7s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 7s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 31m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1724 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 3f1dcd192e91 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] nyl3532016 commented on a change in pull request #1724: HBASE-24386 TableSnapshotScanner support scan limit
nyl3532016 commented on a change in pull request #1724:
URL: https://github.com/apache/hbase/pull/1724#discussion_r426246522
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTableSnapshotScanner.java
##
@@ -190,6 +190,36 @@ public void testNoDuplicateResultsWhenSplitting() throws
Exception {
}
}
+
+ @Test
+ public void testScanLimit() throws Exception {
+setupCluster();
+TableName tableName = TableName.valueOf("testScanLimit");
Review comment:
yes, I will amend it as you said
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1723: HBASE-24381 The Size metrics in Master Webui is wrong if the size is 0
Apache-HBase commented on pull request #1723: URL: https://github.com/apache/hbase/pull/1723#issuecomment-629780060 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 44s | master passed | | -0 :warning: | javadoc | 0m 44s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 29s | the patch passed | | -0 :warning: | javadoc | 0m 42s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 194m 2s | hbase-server in the patch passed. | | | | 209m 13s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1723/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1723 | | Optional Tests | javac javadoc unit | | uname | Linux e8bb7018ca21 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1723/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1723/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1723/2/testReport/ | | Max. process+thread count | 3351 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1723/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1721: HBASE-24377 MemStoreFlusher throw NullPointerException
Apache-HBase commented on pull request #1721: URL: https://github.com/apache/hbase/pull/1721#issuecomment-629779985 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 38s | master passed | | +1 :green_heart: | compile | 1m 8s | master passed | | +1 :green_heart: | shadedjars | 6m 18s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 42s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 31s | the patch passed | | +1 :green_heart: | compile | 1m 8s | the patch passed | | +1 :green_heart: | javac | 1m 8s | the patch passed | | +1 :green_heart: | shadedjars | 6m 16s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 180m 12s | hbase-server in the patch passed. | | | | 208m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1721 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2f904b4e3b83 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/testReport/ | | Max. process+thread count | 3283 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on a change in pull request #1681: HBASE-23938 : System table hbase:slowlog to store complete slow/large…
anoopsjohn commented on a change in pull request #1681:
URL: https://github.com/apache/hbase/pull/1681#discussion_r426227621
##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
##
@@ -1532,6 +1532,16 @@
"hbase.regionserver.slowlog.buffer.enabled";
public static final boolean DEFAULT_ONLINE_LOG_PROVIDER_ENABLED = false;
+ /** The slowlog info family as a string*/
+ private static final String SLOWLOG_INFO_FAMILY_STR = "info";
+
+ /** The slowlog info family */
+ public static final byte [] SLOWLOG_INFO_FAMILY =
Bytes.toBytes(SLOWLOG_INFO_FAMILY_STR);
+
+ public static final String SLOW_LOG_SYS_TABLE_ENABLED_KEY =
+"hbase.regionserver.slowlog.systable.enabled";
+ public static final boolean DEFAULT_SLOW_LOG_SYS_TABLE_ENABLED_KEY = false;
Review comment:
Are we not logging the region's encoded name? This is the region name
which comes as part of the req param:
##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java
##
@@ -83,6 +83,12 @@
public static final TableName META_TABLE_NAME =
valueOf(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME_STR, "meta");
+ /** hbase:slowlog table name - can be enabled
+ * with config - hbase.regionserver.slowlog.systable.enabled
+ */
+ public static final TableName SLOW_LOG_TABLE_NAME =
Review comment:
Are we exposing this table name to customer? This is a public class. If
we expose the name for user to create queries, we might have to expose the
column names also? But those can not be done in this class. Any thinking of
having a new Public class if needed to expose? If you dont want to expose this
table name, we should not keep this in TableName public class
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/slowlog/LogEventHandler.java
##
@@ -53,12 +55,28 @@
class LogEventHandler implements EventHandler {
private static final Logger LOG =
LoggerFactory.getLogger(LogEventHandler.class);
+ private static final int SYS_TABLE_QUEUE_SIZE = 1000;
- private final Queue queue;
+ private final Queue queueForRingBuffer;
+ private final Queue queueForSysTable;
+ private final boolean isSlowLogTableEnabled;
- LogEventHandler(int eventCount) {
+ private Configuration configuration;
+
+ private static final ReentrantLock LOCK = new ReentrantLock();
+
+ LogEventHandler(int eventCount, boolean isSlowLogTableEnabled, Configuration
conf) {
+this.configuration = conf;
EvictingQueue evictingQueue =
EvictingQueue.create(eventCount);
-queue = Queues.synchronizedQueue(evictingQueue);
+queueForRingBuffer = Queues.synchronizedQueue(evictingQueue);
+this.isSlowLogTableEnabled = isSlowLogTableEnabled;
+if (isSlowLogTableEnabled) {
+ EvictingQueue evictingQueueForTable =
EvictingQueue.create(
Review comment:
When system table logging is enabled, we have 2 queue. This new one
will always have 1000 max size where as the 'queueForRingBuffer' will have a
default of 256 only. Anyways we will occupy more heap memory upto 1000
messages. Why we should reduce the max numbers in queueForRingBuffer?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/slowlog/LogEventHandler.java
##
@@ -53,12 +55,28 @@
class LogEventHandler implements EventHandler {
private static final Logger LOG =
LoggerFactory.getLogger(LogEventHandler.class);
+ private static final int SYS_TABLE_QUEUE_SIZE = 1000;
- private final Queue queue;
+ private final Queue queueForRingBuffer;
+ private final Queue queueForSysTable;
+ private final boolean isSlowLogTableEnabled;
- LogEventHandler(int eventCount) {
+ private Configuration configuration;
+
+ private static final ReentrantLock LOCK = new ReentrantLock();
+
+ LogEventHandler(int eventCount, boolean isSlowLogTableEnabled, Configuration
conf) {
+this.configuration = conf;
EvictingQueue evictingQueue =
EvictingQueue.create(eventCount);
-queue = Queues.synchronizedQueue(evictingQueue);
+queueForRingBuffer = Queues.synchronizedQueue(evictingQueue);
+this.isSlowLogTableEnabled = isSlowLogTableEnabled;
+if (isSlowLogTableEnabled) {
+ EvictingQueue evictingQueueForTable =
EvictingQueue.create(
Review comment:
Can we have a way to avoid this 2 queues but keep it single?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/slowlog/LogEventHandler.java
##
@@ -53,12 +55,28 @@
class LogEventHandler implements EventHandler {
private static final Logger LOG =
LoggerFactory.getLogger(LogEventHandler.class);
+ private static final int SYS_TABLE_QUEUE_SIZE = 1000;
Review comment:
You want a config for this like
'hbase.regionserver.slowlog.ringbuffer.size'?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/slowlog/SlowLogTableAccessor.java
##
[GitHub] [hbase] Apache-HBase commented on pull request #1721: HBASE-24377 MemStoreFlusher throw NullPointerException
Apache-HBase commented on pull request #1721: URL: https://github.com/apache/hbase/pull/1721#issuecomment-629774449 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 44s | master passed | | +1 :green_heart: | compile | 0m 55s | master passed | | +1 :green_heart: | shadedjars | 5m 41s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 22s | the patch passed | | +1 :green_heart: | compile | 0m 54s | the patch passed | | +1 :green_heart: | javac | 0m 54s | the patch passed | | +1 :green_heart: | shadedjars | 5m 32s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 135m 1s | hbase-server in the patch failed. | | | | 159m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1721 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8a6be9747578 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/testReport/ | | Max. process+thread count | 4841 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] HorizonNet commented on a change in pull request #1719: HBASE-24370 Avoid aggressive MergeRegion and GCMultipleMergedRegionsP…
HorizonNet commented on a change in pull request #1719:
URL: https://github.com/apache/hbase/pull/1719#discussion_r426242742
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi2.java
##
@@ -161,20 +163,39 @@ public void testMergeRegions() throws Exception {
.getTableHRegionLocations(metaTable, tableName).get();
RegionInfo regionA;
RegionInfo regionB;
+RegionInfo regionC;
+RegionInfo mergedChildRegion = null;
// merge with full name
assertEquals(3, regionLocations.size());
regionA = regionLocations.get(0).getRegion();
regionB = regionLocations.get(1).getRegion();
+regionC = regionLocations.get(2).getRegion();
admin.mergeRegions(regionA.getRegionName(), regionB.getRegionName(),
false).get();
regionLocations = AsyncMetaTableAccessor
.getTableHRegionLocations(metaTable, tableName).get();
+
assertEquals(2, regionLocations.size());
+for (HRegionLocation rl : regionLocations) {
+ if (regionC.compareTo(rl.getRegion()) != 0) {
+mergedChildRegion = rl.getRegion();
+break;
+ }
+}
+
+assertTrue(mergedChildRegion != null);
Review comment:
Ditto
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin1.java
##
@@ -530,25 +532,42 @@ public void testMergeRegions() throws Exception {
List tableRegions;
RegionInfo regionA;
RegionInfo regionB;
+ RegionInfo regionC;
+ RegionInfo mergedChildRegion = null;
// merge with full name
tableRegions = ADMIN.getRegions(tableName);
assertEquals(3, ADMIN.getRegions(tableName).size());
regionA = tableRegions.get(0);
regionB = tableRegions.get(1);
+ regionC = tableRegions.get(2);
// TODO convert this to version that is synchronous (See HBASE-16668)
- ADMIN.mergeRegionsAsync(regionA.getRegionName(),
regionB.getRegionName(), false).get(60,
-TimeUnit.SECONDS);
+ ADMIN.mergeRegionsAsync(regionA.getRegionName(), regionB.getRegionName(),
+false).get(60, TimeUnit.SECONDS);
- assertEquals(2, ADMIN.getRegions(tableName).size());
-
- // merge with encoded name
tableRegions = ADMIN.getRegions(tableName);
- regionA = tableRegions.get(0);
- regionB = tableRegions.get(1);
+
+ assertEquals(2, tableRegions.size());
+ for (RegionInfo ri : tableRegions) {
+if (regionC.compareTo(ri) != 0) {
+ mergedChildRegion = ri;
+ break;
+}
+ }
+
+ assertTrue(mergedChildRegion != null);
Review comment:
Use `assertNotNull` instead.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] HorizonNet commented on a change in pull request #1722: HBASE-24359 Optionally ignore edits for deleted CFs for replication
HorizonNet commented on a change in pull request #1722:
URL: https://github.com/apache/hbase/pull/1722#discussion_r426241594
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEditsDroppedWithDroppedTable.java
##
@@ -0,0 +1,251 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.replication;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.fail;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.HColumnDescriptor;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HTableDescriptor;
+import org.apache.hadoop.hbase.NamespaceDescriptor;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import
org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint;
+import org.apache.hadoop.hbase.testclassification.LargeTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.JVMClusterUtil;
+import org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Category({ LargeTests.class })
+public class TestReplicationEditsDroppedWithDroppedTable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(TestReplicationEditsDroppedWithDroppedTable.class);
+
+ private static Configuration conf1 = HBaseConfiguration.create();
+ private static Configuration conf2 = HBaseConfiguration.create();
+
+ protected static HBaseTestingUtility utility1;
+ protected static HBaseTestingUtility utility2;
+
+ private static Admin admin1;
+ private static Admin admin2;
+
+ private static final String namespace = "NS";
+ private static final TableName NORMAL_TABLE =
TableName.valueOf("normal-table");
+ private static final TableName DROPPED_TABLE =
TableName.valueOf("dropped-table");
+ private static final TableName DROPPED_NS_TABLE =
TableName.valueOf("NS:dropped-table");
+ private static final byte[] ROW = Bytes.toBytes("row");
+ private static final byte[] FAMILY = Bytes.toBytes("f");
+ private static final byte[] QUALIFIER = Bytes.toBytes("q");
+ private static final byte[] VALUE = Bytes.toBytes("value");
+
+ private static final String PEER_ID = "1";
+ private static final long SLEEP_TIME = 1000;
+ private static final int NB_RETRIES = 10;
+
+ @BeforeClass
+ public static void setUpBeforeClass() throws Exception {
+// Set true to filter replication edits for dropped table
+
conf1.setBoolean(HBaseInterClusterReplicationEndpoint.REPLICATION_DROP_ON_DELETED_TABLE_KEY,
true);
+conf1.set(HConstants.ZOOKEEPER_ZNODE_PARENT, "/1");
+conf1.setInt("replication.source.nb.capacity", 1);
+utility1 = new HBaseTestingUtility(conf1);
+utility1.startMiniZKCluster();
+MiniZooKeeperCluster miniZK = utility1.getZkCluster();
+conf1 = utility1.getConfiguration();
+
+conf2 = HBaseConfiguration.create(conf1);
+conf2.set(HConstants.ZOOKEEPER_ZNODE_PARENT, "/2");
+utility2 = new HBaseTestingUtility(conf2);
+utility2.setZkCluster(miniZK);
+
+utility1.startMiniCluster(1);
+utility2.startMiniCluster(1);
+
+admin1 = utility1.getAdmin();
+admin2 = utility2.getAdmin();
+
+NamespaceDescriptor nsDesc = NamespaceDescriptor.create(namespace).build();
+admin1.createNamespace(nsDesc);
+admin2.createNamespace(nsDesc);
+ }
+
+ @AfterClass
+ public static void tearDownAfterClass() throws Exception {
+utility2.shutdownMiniCluster();
+utility1.shutdownMiniCluster();
+ }
+
+ @Before
+ public void setup() throws Exception {
+// Roll log
+for (JVMClusterUtil.RegionServerThread r :
[GitHub] [hbase] Apache-HBase commented on pull request #1723: HBASE-24381 The Size metrics in Master Webui is wrong if the size is 0
Apache-HBase commented on pull request #1723: URL: https://github.com/apache/hbase/pull/1723#issuecomment-629772811 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 47s | master passed | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 25s | the patch passed | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 132m 29s | hbase-server in the patch passed. | | | | 143m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1723/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1723 | | Optional Tests | javac javadoc unit | | uname | Linux 7affbdb92822 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1723/2/testReport/ | | Max. process+thread count | 4881 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1723/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] HorizonNet commented on a change in pull request #1724: HBASE-24386 TableSnapshotScanner support scan limit
HorizonNet commented on a change in pull request #1724:
URL: https://github.com/apache/hbase/pull/1724#discussion_r426240189
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTableSnapshotScanner.java
##
@@ -190,6 +190,36 @@ public void testNoDuplicateResultsWhenSplitting() throws
Exception {
}
}
+
+ @Test
+ public void testScanLimit() throws Exception {
+setupCluster();
+TableName tableName = TableName.valueOf("testScanLimit");
Review comment:
Please introduce a rule for the table name
```
@Rule
public TestName name = new TestName();
```
And use it like this
```
final TableName tableName = TableName.valueOf(name.getMethodName());
```
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTableSnapshotScanner.java
##
@@ -190,6 +190,36 @@ public void testNoDuplicateResultsWhenSplitting() throws
Exception {
}
}
+
+ @Test
+ public void testScanLimit() throws Exception {
+setupCluster();
+TableName tableName = TableName.valueOf("testScanLimit");
+String snapshotName = "testScanLimitSnapshot";
+try {
+ createTableAndSnapshot(UTIL, tableName, snapshotName, 50);
+ Path restoreDir = UTIL.getDataTestDirOnTestFS(snapshotName);
+ Scan scan = new Scan().withStartRow(bbb).setLimit(100); // limit the scan
+
+ TableSnapshotScanner scanner =
+ new TableSnapshotScanner(UTIL.getConfiguration(), restoreDir,
snapshotName, scan);
+ int count = 0;
+ while (true) {
+Result result = scanner.next();
+if (result == null) {
+ break;
+}
+count++;
+ }
+ assert (100 == count);
Review comment:
I would prefer `assertEquals` in that case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on a change in pull request #1724: HBASE-24386 TableSnapshotScanner support scan limit
virajjasani commented on a change in pull request #1724:
URL: https://github.com/apache/hbase/pull/1724#discussion_r426234257
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTableSnapshotScanner.java
##
@@ -190,6 +190,36 @@ public void testNoDuplicateResultsWhenSplitting() throws
Exception {
}
}
+
+ @Test
+ public void testScanLimit() throws Exception {
+setupCluster();
+TableName tableName = TableName.valueOf("testScanLimit");
+String snapshotName = "testScanLimitSnapshot";
+try {
+ createTableAndSnapshot(UTIL, tableName, snapshotName, 50);
+ Path restoreDir = UTIL.getDataTestDirOnTestFS(snapshotName);
+ Scan scan = new Scan().withStartRow(bbb).setLimit(100); // limit the scan
+
+ TableSnapshotScanner scanner =
+ new TableSnapshotScanner(UTIL.getConfiguration(), restoreDir,
snapshotName, scan);
+ int count = 0;
+ while (true) {
+Result result = scanner.next();
+if (result == null) {
+ break;
+}
+count++;
+ }
+ assert (100 == count);
Review comment:
Please use assertTrue or assertEquals.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1721: HBASE-24377 MemStoreFlusher throw NullPointerException
Apache-HBase commented on pull request #1721: URL: https://github.com/apache/hbase/pull/1721#issuecomment-629760775 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 54s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 22s | master passed | | +1 :green_heart: | checkstyle | 1m 14s | master passed | | +1 :green_heart: | spotbugs | 2m 14s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 51s | the patch passed | | +1 :green_heart: | checkstyle | 1m 10s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 36s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 49s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 38m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1721 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux ac4b874fe124 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1721/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1724: HBASE-24386 TableSnapshotScanner support scan limit
Apache-HBase commented on pull request #1724: URL: https://github.com/apache/hbase/pull/1724#issuecomment-629758639 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 45s | master passed | | +1 :green_heart: | compile | 0m 56s | master passed | | +1 :green_heart: | shadedjars | 5m 33s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 27s | the patch passed | | +1 :green_heart: | compile | 0m 55s | the patch passed | | +1 :green_heart: | javac | 0m 55s | the patch passed | | +1 :green_heart: | shadedjars | 5m 29s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 133m 9s | hbase-server in the patch passed. | | | | 157m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1724 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0bfe6fa3cd48 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/2/testReport/ | | Max. process+thread count | 3899 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1724: HBASE-24386 TableSnapshotScanner support scan limit
Apache-HBase commented on pull request #1724: URL: https://github.com/apache/hbase/pull/1724#issuecomment-629757992 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 20s | master passed | | +1 :green_heart: | compile | 1m 3s | master passed | | +1 :green_heart: | shadedjars | 5m 42s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 42s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | the patch passed | | +1 :green_heart: | compile | 1m 2s | the patch passed | | +1 :green_heart: | javac | 1m 2s | the patch passed | | +1 :green_heart: | shadedjars | 5m 42s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 124m 33s | hbase-server in the patch passed. | | | | 150m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1724 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f6df5e6c998a 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/2/testReport/ | | Max. process+thread count | 4099 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1724/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1723: HBASE-24381 The Size metrics in Master Webui is wrong if the size is 0
Apache-HBase commented on pull request #1723: URL: https://github.com/apache/hbase/pull/1723#issuecomment-629757274 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 2m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1723/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1723 | | Optional Tests | dupname asflicense | | uname | Linux 58e8e0d976d6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 15627bb722 | | Max. process+thread count | 47 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1723/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ZhaoBQ commented on pull request #1723: HBASE-24381 The Size metrics in Master Webui is wrong if the size is 0
ZhaoBQ commented on pull request #1723: URL: https://github.com/apache/hbase/pull/1723#issuecomment-629757233 @HorizonNet Indentation fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] binlijin commented on pull request #1721: HBASE-24377 MemStoreFlusher throw NullPointerException
binlijin commented on pull request #1721: URL: https://github.com/apache/hbase/pull/1721#issuecomment-629757074 @Apache9 @anoopsjohn @ramkrish86 ok, remove the flushType field. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24377) MemStoreFlusher throw NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-24377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109372#comment-17109372
]
Lijin Bin commented on HBASE-24377:
---
OK, get it.
> MemStoreFlusher throw NullPointerException
> --
>
> Key: HBASE-24377
> URL: https://issues.apache.org/jira/browse/HBASE-24377
> Project: HBase
> Issue Type: Bug
>Reporter: Lijin Bin
>Priority: Major
>
> {code}
> 2020-02-26 10:15:33,585 ERROR [MemStoreFlusher.2]
> regionserver.MemStoreFlusher: Cache flusher failed for entry null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:158)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$700(MemStoreFlusher.java:68)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:346)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Need to setFlushType first before use it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23806) Provide a much faster and efficient alternate option to maven and surefire for running tests.
[ https://issues.apache.org/jira/browse/HBASE-23806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109348#comment-17109348 ] Mark Robert Miller commented on HBASE-23806: I’m done with my project. I was originally going to try and kind of document and share my trail, but it did not really pan out. My intention was that once the test suite was seen at another level, it might just be too tempting to take some benefit from that example of what can be done. Really though, the main benefit was for me, for general knowledge and so that I could understand how to do a couple things in the code with confidence. > Provide a much faster and efficient alternate option to maven and surefire > for running tests. > - > > Key: HBASE-23806 > URL: https://issues.apache.org/jira/browse/HBASE-23806 > Project: HBase > Issue Type: Wish >Reporter: Mark Robert Miller >Priority: Minor > > With HBASE-23795, the hope is to drive tests with maven and surefire much > closer to their potential. > That will still leave a lot of room for improvement. > For those that have some nice hardware and a need for speed, we can blow > right past maven+surefire. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23806) Provide a much faster and efficient alternate option to maven and surefire for running tests.
[ https://issues.apache.org/jira/browse/HBASE-23806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Robert Miller resolved HBASE-23806. Resolution: Won't Fix > Provide a much faster and efficient alternate option to maven and surefire > for running tests. > - > > Key: HBASE-23806 > URL: https://issues.apache.org/jira/browse/HBASE-23806 > Project: HBase > Issue Type: Wish >Reporter: Mark Robert Miller >Priority: Minor > > With HBASE-23795, the hope is to drive tests with maven and surefire much > closer to their potential. > That will still leave a lot of room for improvement. > For those that have some nice hardware and a need for speed, we can blow > right past maven+surefire. -- This message was sent by Atlassian Jira (v8.3.4#803005)
