[GitHub] [hbase] bharathv opened a new pull request #3345: HBASE-25932: Ensure replication reads the trailer bytes from WAL. (#3…
bharathv opened a new pull request #3345: URL: https://github.com/apache/hbase/pull/3345 …332) This bug was exposed by the test from HBASE-25924. Since this wal implementations close the wal asynchronously, replication can potentially miss the trailer bytes. (see jira comment for detailed analysis). While this is not a correctness problem (since trailer does not have any entry data), it erroneously bumps a metric that is used to track skipped bytes in WAL resulting in false alarms which is something we should avoid. Reviewed-by: Rushabh Shah Signed-off-by: Viraj Jasani Signed-off-by Anoop Sam John (cherry picked from commit b04c3c77860fe430308a6bfd7dd8aebe1ad3510c) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
bharathv commented on a change in pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#discussion_r642797633
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestWALEntryStream.java
##
@@ -530,7 +557,8 @@ public void testReplicationSourceWALReaderWrongPosition()
throws Exception {
@Override
public boolean evaluate() throws Exception {
-return fs.getFileStatus(walPath).getLen() > 0;
+return fs.getFileStatus(walPath).getLen() > 0 &&
+((AbstractFSWAL) log).getInflightWALCloseCount() == 0;
Review comment:
We don't wait for 5s and check. We wait for the condition to pass within
5s or timeout, this is the javadoc, think the test is doing what you are saying
(unless I misunderstood what you are saying)..
```
/**
* Waits up to the duration equal to the specified timeout multiplied by
the
* {@link #getWaitForRatio(Configuration)} for the given {@link Predicate}
to become
* true, failing the test if the timeout is reached and the
Predicate is still
* false.
*
* @param conf the configuration
* @param timeout the timeout in milliseconds to wait for the predicate.
* @param predicate the predicate to evaluate.
* @return the effective wait, in milli-seconds until the predicate
becomes true or
* wait is interrupted otherwise -1 when times out
*/
```
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestWALEntryStream.java
##
@@ -103,6 +107,27 @@
private static final NavigableMap scopes = getScopes();
private final String fakeWalGroupId = "fake-wal-group-id";
+ /**
+ * Test helper that waits until a non-null entry is available in the stream
next or times out.
+ */
+ private static class WALEntryStreamWithRetries extends WALEntryStream {
Review comment:
Addressed in #3344
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bharathv opened a new pull request #3344: HBASE-25932 addendum: Add test comments.
bharathv opened a new pull request #3344: URL: https://github.com/apache/hbase/pull/3344 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-13126) Move HBaseTestingUtility and associated support classes into hbase-testing-utility module
[ https://issues.apache.org/jira/browse/HBASE-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354817#comment-17354817 ] Duo Zhang commented on HBASE-13126: --- Plan to pick this up recently. But in general, I do not think the original HBTU can be moved to hbase-testing-util, as we need to use it write UT for hbase-server module. But we could introduce a new interface/class in hbase-testing-util module for end users and then mark HBTU and related classes as IA.Private. If no big concerns, will start working with this direction. Thanks. > Move HBaseTestingUtility and associated support classes into > hbase-testing-utility module > - > > Key: HBASE-13126 > URL: https://issues.apache.org/jira/browse/HBASE-13126 > Project: HBase > Issue Type: Task > Components: API >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 3.0.0-alpha-2 > > > Over in the review for HBASE-12972, [~enis] mentioned that one of the HBTU > methods wasn't intended for public consumption. > Can we build a list of such methods across the API, appropriately annotate > them for 2.0, and deprecate them in earlier versions with a warning that > they're going to be restricted? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3339: HBASE-25947 Backport 'HBASE-25894 Improve the performance for region …
Apache-HBase commented on pull request #3339: URL: https://github.com/apache/hbase/pull/3339#issuecomment-851817009 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 16s | branch-2.4 passed | | +1 :green_heart: | compile | 0m 59s | branch-2.4 passed | | +1 :green_heart: | shadedjars | 6m 26s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 18s | the patch passed | | +1 :green_heart: | compile | 0m 59s | the patch passed | | +1 :green_heart: | javac | 0m 59s | the patch passed | | +1 :green_heart: | shadedjars | 5m 59s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 149m 23s | hbase-server in the patch passed. | | | | 174m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3339/5/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3339 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 88e9bdc40601 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / 1972261826 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3339/5/testReport/ | | Max. process+thread count | 4126 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3339/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3339: HBASE-25947 Backport 'HBASE-25894 Improve the performance for region …
Apache-HBase commented on pull request #3339: URL: https://github.com/apache/hbase/pull/3339#issuecomment-851814471 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 58s | branch-2.4 passed | | +1 :green_heart: | compile | 1m 7s | branch-2.4 passed | | +1 :green_heart: | shadedjars | 6m 44s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 57s | the patch passed | | +1 :green_heart: | compile | 1m 6s | the patch passed | | +1 :green_heart: | javac | 1m 6s | the patch passed | | +1 :green_heart: | shadedjars | 6m 48s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 142m 31s | hbase-server in the patch failed. | | | | 170m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3339/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3339 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 07725cc91ae2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / 1972261826 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3339/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3339/5/testReport/ | | Max. process+thread count | 4073 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3339/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bharathv commented on pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
bharathv commented on pull request #3332: URL: https://github.com/apache/hbase/pull/3332#issuecomment-851811848 Oops, I thought I had a +1 from you @anoopsjohn and I merged the change. Let me open a follow up patch to address your comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bharathv merged pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
bharathv merged pull request #3332: URL: https://github.com/apache/hbase/pull/3332 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on a change in pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
anoopsjohn commented on a change in pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#discussion_r642784905
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestWALEntryStream.java
##
@@ -530,7 +557,8 @@ public void testReplicationSourceWALReaderWrongPosition()
throws Exception {
@Override
public boolean evaluate() throws Exception {
-return fs.getFileStatus(walPath).getLen() > 0;
+return fs.getFileStatus(walPath).getLen() > 0 &&
+((AbstractFSWAL) log).getInflightWALCloseCount() == 0;
Review comment:
We wait for 5 sec and check. Should be wait for
getInflightWALCloseCount to be zero before coming here? Or will that be a
better guarantee for WAL close has happened where we come here for FS.getLen()
check?
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestWALEntryStream.java
##
@@ -103,6 +107,27 @@
private static final NavigableMap scopes = getScopes();
private final String fakeWalGroupId = "fake-wal-group-id";
+ /**
+ * Test helper that waits until a non-null entry is available in the stream
next or times out.
+ */
+ private static class WALEntryStreamWithRetries extends WALEntryStream {
Review comment:
Can u also pls explain why we use this special subclass here for the
tests? (as comments) will be easy to read and digest later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
bharathv commented on a change in pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#discussion_r642784660
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
##
@@ -1190,9 +1201,18 @@ public OptionalLong getLogFileSizeIfBeingWritten(Path
path) {
try {
Path currentPath = getOldPath();
if (path.equals(currentPath)) {
+// Currently active path.
W writer = this.writer;
return writer != null ? OptionalLong.of(writer.getSyncedLength()) :
OptionalLong.empty();
} else {
+W temp = inflightWALClosures.get(path.getName());
+if (temp != null) {
Review comment:
Thanks for taking a look @anoopsjohn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on a change in pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
anoopsjohn commented on a change in pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#discussion_r642782022
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
##
@@ -1190,9 +1201,18 @@ public OptionalLong getLogFileSizeIfBeingWritten(Path
path) {
try {
Path currentPath = getOldPath();
if (path.equals(currentPath)) {
+// Currently active path.
W writer = this.writer;
return writer != null ? OptionalLong.of(writer.getSyncedLength()) :
OptionalLong.empty();
} else {
+W temp = inflightWALClosures.get(path.getName());
+if (temp != null) {
Review comment:
Thanks for checking and confirming. Looks good then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
bharathv commented on a change in pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#discussion_r642781222
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
##
@@ -1190,9 +1201,18 @@ public OptionalLong getLogFileSizeIfBeingWritten(Path
path) {
try {
Path currentPath = getOldPath();
if (path.equals(currentPath)) {
+// Currently active path.
W writer = this.writer;
return writer != null ? OptionalLong.of(writer.getSyncedLength()) :
OptionalLong.empty();
} else {
+W temp = inflightWALClosures.get(path.getName());
+if (temp != null) {
Review comment:
Thanks for clarifying.
> What we are in effect doing is, when the WAL file is not yet closed, we
get the length from writer which tracks the synced length. Once the WAL file is
closed,
Correct.
> Now with the async close, it is possible that when this method execution
is at Line 1208, the close is not over but before we return from this method,
the WAL file is really closed
Correct
> Still we will return a value and system thinks WAL file is still active
one and undergo related logic. It may be very much fine in those places. But I
just wanted to make sure we double check.
Yes this is transparently handled by WalEntryStream. If what you said
happens (which is possible), WALEntryStream thinks that the file is not fully
done yet (in that loop iteration) and in the next attempt when it does the same
check again to read more bytes, it sees that the file is already done and
closed and either reads or terminates accordingly.
Relevant code links
https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java#L283
(the entry read is null, so it just loops again)
https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java#L197
(this is for looping again, in the subsequent next() call, this fill will be
seen as closed and it enters the if block in L180)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Reopened] (HBASE-25773) TestSnapshotScannerHDFSAclController.setupBeforeClass is flaky
[ https://issues.apache.org/jira/browse/HBASE-25773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaolin Ha reopened HBASE-25773: It still fails at the same position. I'll dig more. https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3325/3/testReport/org.apache.hadoop.hbase.security.access/TestSnapshotScannerHDFSAclController/precommit_checks___yetus_jdk8_Hadoop3_checks__/ > TestSnapshotScannerHDFSAclController.setupBeforeClass is flaky > -- > > Key: HBASE-25773 > URL: https://issues.apache.org/jira/browse/HBASE-25773 > Project: HBase > Issue Type: Improvement >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > [https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3140/2/testReport/org.apache.hadoop.hbase.security.access/TestSnapshotScannerHDFSAclController/precommit_checks___yetus_jdk8_Hadoop3_checks__/] > SnapshotScannerHDFSAclController.postStartMaster alters hbase:acl to add a > new cf "m", but > `TestSnapshotScannerHDFSAclController.setupBeforeClass(TestSnapshotScannerHDFSAclController.java:101)` > fails before the disable and enable hbase:acl complete. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3339: HBASE-25947 Backport 'HBASE-25894 Improve the performance for region …
Apache-HBase commented on pull request #3339: URL: https://github.com/apache/hbase/pull/3339#issuecomment-851775381 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 20s | branch-2.4 passed | | +1 :green_heart: | compile | 3m 20s | branch-2.4 passed | | +1 :green_heart: | checkstyle | 1m 10s | branch-2.4 passed | | +1 :green_heart: | spotbugs | 2m 14s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 30s | the patch passed | | +1 :green_heart: | compile | 3m 24s | the patch passed | | +1 :green_heart: | javac | 3m 24s | the patch passed | | +1 :green_heart: | checkstyle | 1m 11s | hbase-server: The patch generated 0 new + 57 unchanged - 5 fixed = 57 total (was 62) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 12s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 48m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3339/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3339 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux f207270eb690 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / 1972261826 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3339/5/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25891) Remove dependence storing WAL filenames for backup
[
https://issues.apache.org/jira/browse/HBASE-25891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354766#comment-17354766
]
Anoop Sam John commented on HBASE-25891:
Sounds good. Thanks.
> Remove dependence storing WAL filenames for backup
> --
>
> Key: HBASE-25891
> URL: https://issues.apache.org/jira/browse/HBASE-25891
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Affects Versions: 3.0.0-alpha-1
>Reporter: Mallikarjun
>Assignee: Mallikarjun
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> Currently WAL logs are stored in `backup:system` meta table
> {code:java}
> // code placeholder
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:backupId,
> timestamp=1622003479895, value=backup_1622003358258
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:file,
> timestamp=1622003479895,
> value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621996160175
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:root,
> timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:backupId,
> timestamp=1622003479895, value=backup_1622003358258
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:file,
> timestamp=1622003479895,
> value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621999760280
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:root,
> timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
> {code}
>
> This has several problems
> # Ever growing rows of wal's sourced for incremental backup is maintained
> and never cleaned up.
> # Unnecessary to have wal log listed for performing incremental backup or
> for performing logcleaner.
> # No support for rsgroup. Hence all rsgroups which doesn't have backup
> enabled tables, WAL's are retained forever.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] anoopsjohn commented on a change in pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
anoopsjohn commented on a change in pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#discussion_r642750975
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
##
@@ -1190,9 +1201,18 @@ public OptionalLong getLogFileSizeIfBeingWritten(Path
path) {
try {
Path currentPath = getOldPath();
if (path.equals(currentPath)) {
+// Currently active path.
W writer = this.writer;
return writer != null ? OptionalLong.of(writer.getSyncedLength()) :
OptionalLong.empty();
} else {
+W temp = inflightWALClosures.get(path.getName());
+if (temp != null) {
Review comment:
Sorry for not making clear.
What we are in effect doing is, when the WAL file is not yet closed, we get
the length from writer which tracks the synced length. Once the WAL file is
closed, (we know that from the output of this method) there is different logic.
(I have not seen that full area though). Now with the async close, it is
possible that when this method execution is at Line 1208, the close is not over
but before we return from this method, the WAL file is really closed. Still
we will return a value and system thinks WAL file is still active one and
undergo related logic. It may be very much fine in those places. But I just
wanted to make sure we double check.
With sync close, this method and the writer close all were having a common
lock. (rollWriterLock). The entire rollWriter(boolean force) is under that
lock and cur writer close was within it.
Hope am making it clear now (?)
Again it may be very much fine only. But just can see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on a change in pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
anoopsjohn commented on a change in pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#discussion_r642750975
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
##
@@ -1190,9 +1201,18 @@ public OptionalLong getLogFileSizeIfBeingWritten(Path
path) {
try {
Path currentPath = getOldPath();
if (path.equals(currentPath)) {
+// Currently active path.
W writer = this.writer;
return writer != null ? OptionalLong.of(writer.getSyncedLength()) :
OptionalLong.empty();
} else {
+W temp = inflightWALClosures.get(path.getName());
+if (temp != null) {
Review comment:
Sorry for not making clear.
What we are in effect doing is, when the WAL file is not yet closed, we get
the length from writer which tracks the synced length. Once the WAL file is
closed, (we know that from the output of this method) there is different logic.
(I have not seen that full area though). Now with the async close, it is
possible that when this method execution is at Line 1208, the close is not over
but we return from here, the WAL file is really closed. Still we will return
a value and system thinks WAL file is still active one and undergo related
logic. It may be very much fine in those places. But I just wanted to make we
double check.
With sync close, this method and the writer close all were having a common
lock. (rollWriterLock). The entire rollWriter(boolean force) is under that
lock and cur writer close was within it.
Hope am making it clear now?
Again it may be very much fine only. But just can see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-23634) Enable "Split WAL to HFile" by default
[ https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354752#comment-17354752 ] Anoop Sam John commented on HBASE-23634: There were some issues around this. Not remembering all. But one was related to half done splits. Not sure we have a separate Jira for that. Let me read through this Jira fully. Thanks for reminder here Duo. > Enable "Split WAL to HFile" by default > -- > > Key: HBASE-23634 > URL: https://issues.apache.org/jira/browse/HBASE-23634 > Project: HBase > Issue Type: Task >Affects Versions: 3.0.0-alpha-1, 2.3.0 >Reporter: Guanghao Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25920) Support Hadoop 3.3.1
[ https://issues.apache.org/jira/browse/HBASE-25920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354751#comment-17354751 ] Wei-Chiu Chuang commented on HBASE-25920: - bq. Wei-Chiu Chuang yeah these looks like changes to transitive dependencies. You're reverting all these for a patch release? BouncyCastle was committed in 3.3.1 only so no impact. The lz4 and snappy codec change is a little too much for a patch release imo > Support Hadoop 3.3.1 > > > Key: HBASE-25920 > URL: https://issues.apache.org/jira/browse/HBASE-25920 > Project: HBase > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > The Hadoop 3.3.1 is a big release, quite different from 3.3.0. > File this jira to track the support for Hadoop 3.3.1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25830) HBaseCluster support CompactionServer for UTs
[ https://issues.apache.org/jira/browse/HBASE-25830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yulin Niu resolved HBASE-25830. --- Resolution: Fixed > HBaseCluster support CompactionServer for UTs > - > > Key: HBASE-25830 > URL: https://issues.apache.org/jira/browse/HBASE-25830 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Yulin Niu >Assignee: Yulin Niu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25830) HBaseCluster support CompactionServer for UTs
[ https://issues.apache.org/jira/browse/HBASE-25830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354748#comment-17354748 ] Yulin Niu commented on HBASE-25830: --- push to branch HBASE-25714, thanks [~zhangduo] and [~stack] 's reviewing > HBaseCluster support CompactionServer for UTs > - > > Key: HBASE-25830 > URL: https://issues.apache.org/jira/browse/HBASE-25830 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Yulin Niu >Assignee: Yulin Niu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] nyl3532016 merged pull request #3266: HBASE-25830 HBaseCluster support CompactionServer for UTs
nyl3532016 merged pull request #3266: URL: https://github.com/apache/hbase/pull/3266 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25942) Get rid of null regioninfo in wrapped connection exceptions
[
https://issues.apache.org/jira/browse/HBASE-25942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354741#comment-17354741
]
Hudson commented on HBASE-25942:
Results for branch branch-2.4
[build #131 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/131/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/131/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/131/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/131/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/131/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Get rid of null regioninfo in wrapped connection exceptions
> ---
>
> Key: HBASE-25942
> URL: https://issues.apache.org/jira/browse/HBASE-25942
> Project: HBase
> Issue Type: Improvement
> Components: logging
>Affects Versions: 3.0.0-alpha-1, 2.5.0, 2.4.4
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Trivial
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4
>
>
> I was debugging something and noticed this,
> {noformat}
> org.apache.hbase.thirdparty.com.google.protobuf.ServiceException:
> java.io.IOException: Call to address=localhost:59142null failed on local
> exception: java.io.IOException: Connection reset by peer
> {noformat}
> `null` there is regioninfo and because this call has nothing set on it, it
> was confusing where that null is coming from, added recently via HBASE-25735.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25941) TestRESTServerSSL fails because of jdk bug
[
https://issues.apache.org/jira/browse/HBASE-25941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354743#comment-17354743
]
Hudson commented on HBASE-25941:
Results for branch branch-2.4
[build #131 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/131/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/131/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/131/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/131/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/131/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> TestRESTServerSSL fails because of jdk bug
> --
>
> Key: HBASE-25941
> URL: https://issues.apache.org/jira/browse/HBASE-25941
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: Michael Stack
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.6, 2.4.4
>
>
> [~weijing329] identified issue in TestRESTServerSSL when using jdk8 292+. It
> came up in comment in the parent issue. I verified it fails for me using jdk8
> v292. Here is the failure
> ```[INFO] Running org.apache.hadoop.hbase.rest.TestRESTServerSSL
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.4 s
> <<< FAILURE! - in org.apache.hadoop.hbase.rest.TestRESTServerSSL
> [ERROR] org.apache.hadoop.hbase.rest.TestRESTServerSSL Time elapsed: 1.387 s
> <<< ERROR!
> java.security.NoSuchAlgorithmException: unrecognized algorithm name:
> PBEWithSHA1AndDESede
> at
> org.apache.hadoop.hbase.rest.TestRESTServerSSL.beforeClass(TestRESTServerSSL.java:74)```
> For workaround, see https://github.com/bcgit/bc-java/issues/941
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25903) ReadOnlyZKClient APIs - CompletableFuture.get() calls can cause threads to hang forver when ZK client create throws Non IOException
[
https://issues.apache.org/jira/browse/HBASE-25903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354744#comment-17354744
]
Hudson commented on HBASE-25903:
Results for branch branch-2.4
[build #131 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/131/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/131/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/131/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/131/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/131/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> ReadOnlyZKClient APIs - CompletableFuture.get() calls can cause threads to
> hang forver when ZK client create throws Non IOException
> ---
>
> Key: HBASE-25903
> URL: https://issues.apache.org/jira/browse/HBASE-25903
> Project: HBase
> Issue Type: Bug
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4
>
>
> This is applicable for zk client versions which is not having fix for
> ZOOKEEPER-2184.
> Now we are on zookeeper 3.5.7 on active 2.x branches. Still its better to
> handle this case in our code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on pull request #3339: HBASE-25947 Backport 'HBASE-25894 Improve the performance for region …
Apache9 commented on pull request #3339: URL: https://github.com/apache/hbase/pull/3339#issuecomment-851750997 OK, I think this is the problem. 2021-05-31 16:15:08,227 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41365] balancer.StochasticLoadBalancer(458): start StochasticLoadBalancer.balancer, initCost=NaN, functionCost=RegionCountSkewCostFunction : (500.0, 0.0); PrimaryRegionCountSkewCostFunction : (not needed); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0, 0.0); RackLocalityCostFunction : (15.0, 0.0); TableSkewCostFunction : (35.0, 1.0); RegionReplicaHostCostFunction : (not needed); RegionReplicaRackCostFunction : (not needed); ReadRequestCostFunction : (5.0, NaN); WriteRequestCostFunction : (5.0, NaN); MemStoreSizeCostFunction : (5.0, NaN); StoreFileCostFunction : (5.0, 0.0); computedMaxSteps: 9600 The region load related cost functions all return NaN, which makes the balancer broken. Let me dig more. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Apache-HBase commented on pull request #3332: URL: https://github.com/apache/hbase/pull/3332#issuecomment-851745942 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 2s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 23s | master passed | | +1 :green_heart: | compile | 1m 3s | master passed | | +1 :green_heart: | shadedjars | 8m 59s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 59s | the patch passed | | +1 :green_heart: | compile | 1m 3s | the patch passed | | +1 :green_heart: | javac | 1m 3s | the patch passed | | +1 :green_heart: | shadedjars | 8m 59s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 209m 53s | hbase-server in the patch passed. | | | | 242m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3332 | | JIRA Issue | HBASE-25932 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux fd6e094938e7 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 06c6e06803 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/5/testReport/ | | Max. process+thread count | 3086 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #3339: HBASE-25947 Backport 'HBASE-25894 Improve the performance for region …
Apache9 commented on pull request #3339: URL: https://github.com/apache/hbase/pull/3339#issuecomment-851740950 Not sure why jdk11 always pass but jdk8 always fail, it is TestAsyncTableGetMultiThreaded. Let me check the logs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Apache-HBase commented on pull request #3332: URL: https://github.com/apache/hbase/pull/3332#issuecomment-851727851 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 32s | master passed | | +1 :green_heart: | compile | 1m 12s | master passed | | +1 :green_heart: | shadedjars | 8m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 16s | the patch passed | | +1 :green_heart: | compile | 1m 11s | the patch passed | | +1 :green_heart: | javac | 1m 11s | the patch passed | | +1 :green_heart: | shadedjars | 8m 18s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 136m 22s | hbase-server in the patch passed. | | | | 168m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3332 | | JIRA Issue | HBASE-25932 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1ef13b1b7d55 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 06c6e06803 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/5/testReport/ | | Max. process+thread count | 4258 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Apache-HBase commented on pull request #3332: URL: https://github.com/apache/hbase/pull/3332#issuecomment-851704613 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 6s | master passed | | +1 :green_heart: | compile | 3m 54s | master passed | | +1 :green_heart: | checkstyle | 1m 17s | master passed | | +1 :green_heart: | spotbugs | 2m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | the patch passed | | +1 :green_heart: | compile | 3m 22s | the patch passed | | +1 :green_heart: | javac | 3m 22s | the patch passed | | -0 :warning: | checkstyle | 1m 9s | hbase-server: The patch generated 2 new + 11 unchanged - 0 fixed = 13 total (was 11) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 20m 6s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 53m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3332 | | JIRA Issue | HBASE-25932 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 76d596f17c0d 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 06c6e06803 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/5/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3336: HBASE-25920 Support Hadoop 3.3.1
Apache-HBase commented on pull request #3336: URL: https://github.com/apache/hbase/pull/3336#issuecomment-851696617 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 23s | master passed | | +1 :green_heart: | compile | 3m 4s | master passed | | +1 :green_heart: | shadedjars | 9m 45s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 6m 55s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 1m 30s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 14s | the patch passed | | +1 :green_heart: | compile | 2m 52s | the patch passed | | +1 :green_heart: | javac | 2m 52s | the patch passed | | +1 :green_heart: | shadedjars | 10m 5s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 20s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 369m 51s | root in the patch failed. | | | | 420m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3336 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 09feb8c2c977 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 1ccba10847 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/5/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/5/testReport/ | | Max. process+thread count | 2739 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
bharathv commented on a change in pull request #3332:
URL: https://github.com/apache/hbase/pull/3332#discussion_r642682623
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
##
@@ -1190,9 +1201,18 @@ public OptionalLong getLogFileSizeIfBeingWritten(Path
path) {
try {
Path currentPath = getOldPath();
if (path.equals(currentPath)) {
+// Currently active path.
W writer = this.writer;
return writer != null ? OptionalLong.of(writer.getSyncedLength()) :
OptionalLong.empty();
} else {
+W temp = inflightWALClosures.get(path.getName());
+if (temp != null) {
Review comment:
Not sure what you mean, we are returning a long from getSyncedLength()
under the same lock (we are not returning the reference temp), actual method
getSyncedLength() is thread-safe, if thats what you are asking..
Also, both shutdown() / rollWriter are under the scope of rollWriterLock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3336: HBASE-25920 Support Hadoop 3.3.1
Apache-HBase commented on pull request #3336: URL: https://github.com/apache/hbase/pull/3336#issuecomment-851639270 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 41s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 50s | master passed | | +1 :green_heart: | compile | 3m 58s | master passed | | +1 :green_heart: | shadedjars | 10m 42s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 22s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 33s | the patch passed | | +1 :green_heart: | compile | 3m 31s | the patch passed | | +1 :green_heart: | javac | 3m 31s | the patch passed | | +1 :green_heart: | shadedjars | 9m 18s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 198m 35s | root in the patch passed. | | | | 251m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3336 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 35e142ebdd9b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 1ccba10847 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/5/testReport/ | | Max. process+thread count | 6617 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-25902) 1.x to 2.3.x upgrade does not work; you must install an hbase2 that is earlier than hbase-2.3.0 first
[ https://issues.apache.org/jira/browse/HBASE-25902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354629#comment-17354629 ] Andrew Kyle Purtell edited comment on HBASE-25902 at 5/31/21, 6:55 PM: --- Our documented procedure for 1.x to 2.x upgrade is this: 1. Upgrade one RS. Meta will move to it. Wait for meta to move. (The HBase 1 master guarantees the reassignment of meta to the newest version of RS.) 2. Rolling upgrade the rest of the RS. 3. Rolling upgrade the masters. If there is an meta schema issue that would crash a master it must be handled in step 1, when the first 2.x regionserver goes live and is assigned meta. What we need to do here at least is support the documented upgrade procedure from 1.x to 2.x for all versions of 2.x. Step 1 gives us a chance to modify meta if we need to. If this requires a regionserver to change meta schema, then that’s what it has to do, because where else will we get the opportunity? Refer to the 1.x to 2.x upgrade section of the online book. was (Author: apurtell): Our documented procedure for 1.x to 2.x upgrade is this: 1. Upgrade one RS. Meta will move to it. Wait for meta to move. (The HBase 1 master guarantees the reassignment of meta to the newest version of RS.) 2. Rolling upgrade the rest of the RS. 3. Rolling upgrade the masters. If there is an meta schema issue that would crash a master it must be handled in step 1, when the first 2.x regionserver goes live and is assigned meta. > 1.x to 2.3.x upgrade does not work; you must install an hbase2 that is > earlier than hbase-2.3.0 first > - > > Key: HBASE-25902 > URL: https://issues.apache.org/jira/browse/HBASE-25902 > Project: HBase > Issue Type: Bug > Components: meta, Operability >Affects Versions: 2.3.0, 2.4.0 >Reporter: Michael Stack >Priority: Critical > Attachments: NoSuchColumnFamilyException.png > > > Making note of this issue in case others run into it. At my place of employ, > we tried to upgrade a cluster that was an hbase-1.2.x version to an > hbase-2.3.5 but it failed because meta didn't have the 'table' column family. > Up to 2.3.0, hbase:meta was hardcoded. HBASE-12035 added the 'table' CF for > hbase-2.0.0. HBASE-23782 (2.3.0) undid hardcoding of the hbase:meta schema; > i.e. reading hbase:meta schema from the filesystem. The hbase:meta schema is > only created on initial install. If an upgrade over existing data, the > hbase-1 hbase:meta will not be suitable for hbase-2.3.x context as it will be > missing columnfamilies needed to run (HBASE-23055 made it so hbase:meta could > be altered (2.3.0) but probably of no use since Master won't come up). > It would be a nice-to-have if a user could go from hbase1 to hbase.2.3.0 w/o > having to first install an hbase2 that is earlier than 2.3.0 but needs to be > demand before we would work on it; meantime, install an intermediate hbase2 > version before going to hbase-2.3.0+ if coming from hbase-1.x -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25902) 1.x to 2.3.x upgrade does not work; you must install an hbase2 that is earlier than hbase-2.3.0 first
[ https://issues.apache.org/jira/browse/HBASE-25902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354629#comment-17354629 ] Andrew Kyle Purtell commented on HBASE-25902: - Our documented procedure for 1.x to 2.x upgrade is this: 1. Upgrade one RS. Meta will move to it. Wait for meta to move. (The HBase 1 master guarantees the reassignment of meta to the newest version of RS.) 2. Rolling upgrade the rest of the RS. 3. Rolling upgrade the masters. If there is an meta schema issue that would crash a master it must be handled in step 1, when the first 2.x regionserver goes live and is assigned meta. > 1.x to 2.3.x upgrade does not work; you must install an hbase2 that is > earlier than hbase-2.3.0 first > - > > Key: HBASE-25902 > URL: https://issues.apache.org/jira/browse/HBASE-25902 > Project: HBase > Issue Type: Bug > Components: meta, Operability >Affects Versions: 2.3.0, 2.4.0 >Reporter: Michael Stack >Priority: Critical > Attachments: NoSuchColumnFamilyException.png > > > Making note of this issue in case others run into it. At my place of employ, > we tried to upgrade a cluster that was an hbase-1.2.x version to an > hbase-2.3.5 but it failed because meta didn't have the 'table' column family. > Up to 2.3.0, hbase:meta was hardcoded. HBASE-12035 added the 'table' CF for > hbase-2.0.0. HBASE-23782 (2.3.0) undid hardcoding of the hbase:meta schema; > i.e. reading hbase:meta schema from the filesystem. The hbase:meta schema is > only created on initial install. If an upgrade over existing data, the > hbase-1 hbase:meta will not be suitable for hbase-2.3.x context as it will be > missing columnfamilies needed to run (HBASE-23055 made it so hbase:meta could > be altered (2.3.0) but probably of no use since Master won't come up). > It would be a nice-to-have if a user could go from hbase1 to hbase.2.3.0 w/o > having to first install an hbase2 that is earlier than 2.3.0 but needs to be > demand before we would work on it; meantime, install an intermediate hbase2 > version before going to hbase-2.3.0+ if coming from hbase-1.x -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25903) ReadOnlyZKClient APIs - CompletableFuture.get() calls can cause threads to hang forver when ZK client create throws Non IOException
[
https://issues.apache.org/jira/browse/HBASE-25903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354613#comment-17354613
]
Hudson commented on HBASE-25903:
Results for branch branch-2
[build #266 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> ReadOnlyZKClient APIs - CompletableFuture.get() calls can cause threads to
> hang forver when ZK client create throws Non IOException
> ---
>
> Key: HBASE-25903
> URL: https://issues.apache.org/jira/browse/HBASE-25903
> Project: HBase
> Issue Type: Bug
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4
>
>
> This is applicable for zk client versions which is not having fix for
> ZOOKEEPER-2184.
> Now we are on zookeeper 3.5.7 on active 2.x branches. Still its better to
> handle this case in our code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25941) TestRESTServerSSL fails because of jdk bug
[
https://issues.apache.org/jira/browse/HBASE-25941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354611#comment-17354611
]
Hudson commented on HBASE-25941:
Results for branch branch-2
[build #266 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> TestRESTServerSSL fails because of jdk bug
> --
>
> Key: HBASE-25941
> URL: https://issues.apache.org/jira/browse/HBASE-25941
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: Michael Stack
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.6, 2.4.4
>
>
> [~weijing329] identified issue in TestRESTServerSSL when using jdk8 292+. It
> came up in comment in the parent issue. I verified it fails for me using jdk8
> v292. Here is the failure
> ```[INFO] Running org.apache.hadoop.hbase.rest.TestRESTServerSSL
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.4 s
> <<< FAILURE! - in org.apache.hadoop.hbase.rest.TestRESTServerSSL
> [ERROR] org.apache.hadoop.hbase.rest.TestRESTServerSSL Time elapsed: 1.387 s
> <<< ERROR!
> java.security.NoSuchAlgorithmException: unrecognized algorithm name:
> PBEWithSHA1AndDESede
> at
> org.apache.hadoop.hbase.rest.TestRESTServerSSL.beforeClass(TestRESTServerSSL.java:74)```
> For workaround, see https://github.com/bcgit/bc-java/issues/941
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25942) Get rid of null regioninfo in wrapped connection exceptions
[
https://issues.apache.org/jira/browse/HBASE-25942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354610#comment-17354610
]
Hudson commented on HBASE-25942:
Results for branch branch-2
[build #266 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Get rid of null regioninfo in wrapped connection exceptions
> ---
>
> Key: HBASE-25942
> URL: https://issues.apache.org/jira/browse/HBASE-25942
> Project: HBase
> Issue Type: Improvement
> Components: logging
>Affects Versions: 3.0.0-alpha-1, 2.5.0, 2.4.4
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Trivial
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4
>
>
> I was debugging something and noticed this,
> {noformat}
> org.apache.hbase.thirdparty.com.google.protobuf.ServiceException:
> java.io.IOException: Call to address=localhost:59142null failed on local
> exception: java.io.IOException: Connection reset by peer
> {noformat}
> `null` there is regioninfo and because this call has nothing set on it, it
> was confusing where that null is coming from, added recently via HBASE-25735.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25948) Remove deprecated ZK command 'rmr' in hbase-cleanup.sh
[
https://issues.apache.org/jira/browse/HBASE-25948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354612#comment-17354612
]
Hudson commented on HBASE-25948:
Results for branch branch-2
[build #266 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/266/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove deprecated ZK command 'rmr' in hbase-cleanup.sh
> --
>
> Key: HBASE-25948
> URL: https://issues.apache.org/jira/browse/HBASE-25948
> Project: HBase
> Issue Type: Task
> Components: scripts
>Affects Versions: 3.0.0-alpha-1, 2.5.0
>Reporter: mokai
>Assignee: mokai
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Deprecated command 'rmr' was removed from ZooKeeper since 3.6.0(see:
> ZOOKEEPER-3411).
> It's better to use 'deleteall' instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Apache-HBase commented on pull request #3332: URL: https://github.com/apache/hbase/pull/3332#issuecomment-851618311 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 11s | master passed | | +1 :green_heart: | compile | 1m 3s | master passed | | +1 :green_heart: | shadedjars | 9m 5s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 5s | the patch passed | | +1 :green_heart: | compile | 1m 3s | the patch passed | | +1 :green_heart: | javac | 1m 3s | the patch passed | | -1 :x: | shadedjars | 7m 10s | patch has 10 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 212m 11s | hbase-server in the patch failed. | | | | 243m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3332 | | JIRA Issue | HBASE-25932 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5b3a055d4b7c 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 1ccba10847 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/testReport/ | | Max. process+thread count | 3373 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3339: HBASE-25947 Backport 'HBASE-25894 Improve the performance for region …
Apache-HBase commented on pull request #3339: URL: https://github.com/apache/hbase/pull/3339#issuecomment-851612011 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 13s | branch-2.4 passed | | +1 :green_heart: | compile | 0m 59s | branch-2.4 passed | | +1 :green_heart: | shadedjars | 5m 59s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 20s | the patch passed | | +1 :green_heart: | compile | 0m 57s | the patch passed | | +1 :green_heart: | javac | 0m 57s | the patch passed | | +1 :green_heart: | shadedjars | 5m 58s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 140m 10s | hbase-server in the patch failed. | | | | 164m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3339/4/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3339 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 96d2b6c4f96e 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / 1972261826 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3339/4/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3339/4/testReport/ | | Max. process+thread count | 4106 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3339/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3339: HBASE-25947 Backport 'HBASE-25894 Improve the performance for region …
Apache-HBase commented on pull request #3339: URL: https://github.com/apache/hbase/pull/3339#issuecomment-851610050 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 55s | branch-2.4 passed | | +1 :green_heart: | compile | 1m 8s | branch-2.4 passed | | +1 :green_heart: | shadedjars | 6m 52s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 59s | the patch passed | | +1 :green_heart: | compile | 1m 7s | the patch passed | | +1 :green_heart: | javac | 1m 7s | the patch passed | | +1 :green_heart: | shadedjars | 6m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 130m 53s | hbase-server in the patch passed. | | | | 158m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3339/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3339 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4c3cccaba9f8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / 1972261826 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3339/4/testReport/ | | Max. process+thread count | 4330 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3339/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Apache-HBase commented on pull request #3332: URL: https://github.com/apache/hbase/pull/3332#issuecomment-851592364 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 15s | master passed | | +1 :green_heart: | compile | 1m 10s | master passed | | +1 :green_heart: | shadedjars | 8m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 17s | the patch passed | | +1 :green_heart: | compile | 1m 11s | the patch passed | | +1 :green_heart: | javac | 1m 11s | the patch passed | | -1 :x: | shadedjars | 6m 36s | patch has 10 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 136m 54s | hbase-server in the patch failed. | | | | 166m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3332 | | JIRA Issue | HBASE-25932 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 05e8daa0e069 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 1ccba10847 | | Default Java | AdoptOpenJDK-11.0.10+9 | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/testReport/ | | Max. process+thread count | 4335 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-14887) Use Orientated Abstractions
[ https://issues.apache.org/jira/browse/HBASE-14887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-14887: -- Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > Use Orientated Abstractions > --- > > Key: HBASE-14887 > URL: https://issues.apache.org/jira/browse/HBASE-14887 > Project: HBase > Issue Type: Umbrella > Components: API >Affects Versions: 2.0.0 >Reporter: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-2 > > > This umbrella issue is about adding clean server-side "user orientated" > internal APIs that are about 'doing things' (as [~benlau] says over in > HBASE-14439 > "New/Improved Filesystem Abstractions") such as creating and deleting tables, > etc. Initially, we'll retrofit in a ClusterSchema Interface/API that cleans > up Master internals and abstracts away implementation detail around the > manipulation of namespace and tables. > This project operates at a level above the undertaking described in > HBASE-7806 "Isolate the FileSystem calls" and related issues, but has the > same intent of making it so we can the more easily swap in alternate > implementations and representations; e.g. changing filesystem layout under > the covers. It is a complement to HBASE-7806 and related lower-level > undertakings (look out for the gold spike when this top-down vector crosses > the bottom-up endeavor). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21946) Replace the byte[] pread by ByteBuffer pread in HFileBlock reading once HDFS-3246 prepared
[ https://issues.apache.org/jira/browse/HBASE-21946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21946: -- Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > Replace the byte[] pread by ByteBuffer pread in HFileBlock reading once > HDFS-3246 prepared > -- > > Key: HBASE-21946 > URL: https://issues.apache.org/jira/browse/HBASE-21946 > Project: HBase > Issue Type: Improvement > Components: Offheaping >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Critical > Fix For: 2.5.0, 3.0.0-alpha-2 > > Attachments: HBASE-21946.HBASE-21879.v01.patch, > HBASE-21946.HBASE-21879.v02.patch, HBASE-21946.HBASE-21879.v03.patch, > HBASE-21946.HBASE-21879.v04.patch > > > [~stakiar] is working on HDFS-3246, so now we have to keep the byte[] pread > in HFileBlock reading. Once it get resolved, we can upgrade the hadoop > version and do the replacement. > I think it will be a great p999 latency improvement in 100% Get case, anyway > file a issue address this firstly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-19515) Region server left in online servers list forever if it went down after registering to master and before creating ephemeral node
[ https://issues.apache.org/jira/browse/HBASE-19515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19515: -- Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > Region server left in online servers list forever if it went down after > registering to master and before creating ephemeral node > > > Key: HBASE-19515 > URL: https://issues.apache.org/jira/browse/HBASE-19515 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Michael Stack >Priority: Critical > Fix For: 3.0.0-alpha-2 > > > This one is interesting. It was supposedly fixed long time ago back in > HBASE-9593 (The issue has same subject as this one) but there was a problem > w/ the fix reported later, post-commit, long after the issue was closed. The > 'fix' was registering ephemeral node in ZK BEFORE reporting in to the Master > for the first time. The problem w/ this approach is that the Master tells the > RS what name it should use reporting in. If we register in ZK before we talk > to the Master, the name in ZK and the one the RS ends up using could deviate. > In hbase2, we do the right thing registering the ephemeral node after we > report to the Master. So, the issue reported in HBASE-9593, that a RS that > dies between reporting to master and registering up in ZK, stays registered > at the Master for ever is back; we'll keep trying to assign it regions. Its a > real problem. > That hbase2 has this issue has been suppressed up until now. The test that > was written for HBASE-9593, TestRSKilledWhenInitializing, is a good test but > a little sloppy. It puts up two RSs aborting one only after registering at > the Master before posting to ZK. That leaves one healthy server up. It is > hosting hbase:meta. This is enough for the test to bluster through. The only > assign it does is namespace table. It goes to the hbase:meta server. If the > test created a new table and did roundrobin, it'd fail. > After HBASE-18946, where we do round robin on table create -- a desirable > attribute -- via the balancer so all is kosher, the test > TestRSKilledWhenInitializing now starts to fail because we chose the hobbled > server most of the time. > So, this issue is about fixing the original issue properly for hbase2. We > don't have a timeout on assign in AMv2, not yet, that might be the fix, or > perhaps a double report before we online a server with the second report > coming in after ZK goes up (or we stop doing ephemeral nodes for RS up in ZK > and just rely on heartbeats). > Making this a critical issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3336: HBASE-25920 Support Hadoop 3.3.1
Apache-HBase commented on pull request #3336: URL: https://github.com/apache/hbase/pull/3336#issuecomment-851576210 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 36s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 58s | master passed | | +1 :green_heart: | compile | 10m 50s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 1m 38s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 11s | the patch passed | | +1 :green_heart: | compile | 10m 50s | the patch passed | | +1 :green_heart: | javac | 10m 50s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 7s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 24m 23s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 52s | The patch does not generate ASF License warnings. | | | | 70m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3336 | | Optional Tests | dupname asflicense javac hadoopcheck xml compile | | uname | Linux 38b41c846bb5 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 1ccba10847 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 126 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-19307) Ignore the case of string when lookup the enum property from configuration
[ https://issues.apache.org/jira/browse/HBASE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19307: -- Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > Ignore the case of string when lookup the enum property from configuration > -- > > Key: HBASE-19307 > URL: https://issues.apache.org/jira/browse/HBASE-19307 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Sergey Soldatov >Priority: Critical > Fix For: 3.0.0-alpha-2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-18799) Revisit the AccessControl
[ https://issues.apache.org/jira/browse/HBASE-18799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-18799: -- Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > Revisit the AccessControl > - > > Key: HBASE-18799 > URL: https://issues.apache.org/jira/browse/HBASE-18799 > Project: HBase > Issue Type: Task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 3.0.0-alpha-2 > > > We are doing the gardening task for coprocessor. The hooks come and go, but > we should assure our access control is fine. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-18638) There are version-related dirty data caused by delete/ttl
[ https://issues.apache.org/jira/browse/HBASE-18638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-18638: -- Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > There are version-related dirty data caused by delete/ttl > - > > Key: HBASE-18638 > URL: https://issues.apache.org/jira/browse/HBASE-18638 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 2.0.0-alpha-1 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 3.0.0-alpha-2 > > Attachments: HBASE-18638-ut.patch, HBASE-18638-ut.patch > > > |put_0(t0)| > |put_1(t1)| <-- the latest cell > If we call get, the put_1 will return. That is good. > If we call get after a delete, the put_0 will return. That is weird. The > put_0 is old data, and it should be dropped in flush. For client, put_0 > should not exist after the put_1 happen. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-18228) HBCK improvements
[ https://issues.apache.org/jira/browse/HBASE-18228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-18228: -- Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > HBCK improvements > - > > Key: HBASE-18228 > URL: https://issues.apache.org/jira/browse/HBASE-18228 > Project: HBase > Issue Type: Sub-task > Components: hbck >Reporter: Lars Hofhansl >Priority: Critical > Fix For: 3.0.0-alpha-2 > > Attachments: HBASE-18228.branch-1.3.patch > > > We just had a prod issue and running HBCK the way we did actually causes more > problems. > In part HBCK did stuff we did not expect, in part we had little visibility > into what HBCK was doing, and in part the logging was confusing. > I'm proposing 2 improvements: > 1. A dry-run mode. Run, and just list what would have been done. > 2. An interactive mode. Run, and for each action request Y/N user input. So > that a user can opt-out of stuff. > [~jmhsieh], FYI -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-17360) Add REGION/ROW/NONE consistency level for scan
[ https://issues.apache.org/jira/browse/HBASE-17360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17360: -- Fix Version/s: (was: 3.0.0-alpha-1) > Add REGION/ROW/NONE consistency level for scan > -- > > Key: HBASE-17360 > URL: https://issues.apache.org/jira/browse/HBASE-17360 > Project: HBase > Issue Type: Sub-task > Components: Client, Scanners >Reporter: Duo Zhang >Priority: Major > > As discussed in HBASE-17177, we can record a compacted mvcc for each > storefile, and use an inmemory data structure to track it. If we get a scan > request with an mvcc in place, we can use this value to determine if we can > guarantee the given consistency level. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-16525) [2.0] Cell timestamps can be assigned out of order with sequenceId
[
https://issues.apache.org/jira/browse/HBASE-16525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354538#comment-17354538
]
Duo Zhang commented on HBASE-16525:
---
I think the problem is still there. Assign timestamp and assign sequence
id/mvcc are separated steps without lock protection.
> [2.0] Cell timestamps can be assigned out of order with sequenceId
> --
>
> Key: HBASE-16525
> URL: https://issues.apache.org/jira/browse/HBASE-16525
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Priority: Critical
> Fix For: 3.0.0-alpha-2
>
>
> While working on something else, noticed that in 2.0, we can end up with
> assigning timestamps out of order of sequenceId for the same row, thus ending
> with a case that a "later" mutation will not be visible due to an earlier
> mutation with a lower sequenceId. This can happen only in 2.0 code bases
> where we have the read-write lock based rowlocks. In 1.x, due to row locks
> being exclusive, we always order the cell timestamps in order of sequenceIds.
> In HRegion.doMiniBatchMutate(), step 2 is to assign cell timestamps:
> {code}
> // STEP 2. Update any LATEST_TIMESTAMP timestamps
> // We should record the timestamp only after we have acquired the
> rowLock,
> // otherwise, newer puts/deletes are not guaranteed to have a newer
> timestamp
> {code}
> If two transactions that modify the same row starts concurrently, and t1
> executes step 2 first, while t2 gets the sequenceId first, we can end up with
> two transactions where t1 has higher timestamp but lower seqId, and t2 lower
> timestamp but higher seqId.
> Not sure how big a problem is this. One can say that the "order" of
> transactions is the order they execute step 2 (assign cell timestamps) rather
> than assign sequenceIds.
> [~saint@gmail.com] , [~eclark] FYI.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-17177) Compaction can break the region/row level atomic when scan even if we pass mvcc to client
[ https://issues.apache.org/jira/browse/HBASE-17177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17177: -- Fix Version/s: (was: 3.0.0-alpha-1) > Compaction can break the region/row level atomic when scan even if we pass > mvcc to client > - > > Key: HBASE-17177 > URL: https://issues.apache.org/jira/browse/HBASE-17177 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Duo Zhang >Priority: Critical > > We know that major compaction will actually delete the cells which are > deleted by a delete marker. In order to give a consistent view for a scan, we > need to use a map to track the read points for all scanners for a region, and > the smallest one will be used for a compaction. For all delete markers whose > mvcc is greater than this value, we will not use it to delete other cells. > And the problem for a scan restart after region move is that, the new RS does > not have the information of the scanners opened at the old RS before the > client sends scan requests to the new RS which means the read points map is > incomplete and the smallest read point maybe greater than the correct value. > So if a major compaction happens at that time, it may delete some cells which > should be kept. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-16525) [2.0] Cell timestamps can be assigned out of order with sequenceId
[
https://issues.apache.org/jira/browse/HBASE-16525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-16525:
--
Fix Version/s: (was: 3.0.0-alpha-1)
3.0.0-alpha-2
> [2.0] Cell timestamps can be assigned out of order with sequenceId
> --
>
> Key: HBASE-16525
> URL: https://issues.apache.org/jira/browse/HBASE-16525
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Priority: Critical
> Fix For: 3.0.0-alpha-2
>
>
> While working on something else, noticed that in 2.0, we can end up with
> assigning timestamps out of order of sequenceId for the same row, thus ending
> with a case that a "later" mutation will not be visible due to an earlier
> mutation with a lower sequenceId. This can happen only in 2.0 code bases
> where we have the read-write lock based rowlocks. In 1.x, due to row locks
> being exclusive, we always order the cell timestamps in order of sequenceIds.
> In HRegion.doMiniBatchMutate(), step 2 is to assign cell timestamps:
> {code}
> // STEP 2. Update any LATEST_TIMESTAMP timestamps
> // We should record the timestamp only after we have acquired the
> rowLock,
> // otherwise, newer puts/deletes are not guaranteed to have a newer
> timestamp
> {code}
> If two transactions that modify the same row starts concurrently, and t1
> executes step 2 first, while t2 gets the sequenceId first, we can end up with
> two transactions where t1 has higher timestamp but lower seqId, and t2 lower
> timestamp but higher seqId.
> Not sure how big a problem is this. One can say that the "order" of
> transactions is the order they execute step 2 (assign cell timestamps) rather
> than assign sequenceIds.
> [~saint@gmail.com] , [~eclark] FYI.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-16025) Cache table state to reduce load on META
[ https://issues.apache.org/jira/browse/HBASE-16025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16025: -- Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > Cache table state to reduce load on META > > > Key: HBASE-16025 > URL: https://issues.apache.org/jira/browse/HBASE-16025 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 3.0.0-alpha-2 > > > HBASE-12035 moved keeping table enabled/disabled state from ZooKeeper into > hbase:meta. When we retry operations on the client, we check table state in > order to return a specific message if the table is disabled. This means that > in master we will be going back to meta for every retry, even if a region's > location has not changed. This is going to cause performance issues when a > cluster is already loaded, ie. in cases where regionservers may be returning > CallQueueTooBigException. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-16313) Docs on hbase-spark connector need to include examples of actually submitting jobs
[ https://issues.apache.org/jira/browse/HBASE-16313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16313: -- Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > Docs on hbase-spark connector need to include examples of actually submitting > jobs > -- > > Key: HBASE-16313 > URL: https://issues.apache.org/jira/browse/HBASE-16313 > Project: HBase > Issue Type: Improvement > Components: documentation, spark >Reporter: Sean Busbey >Priority: Critical > Fix For: 3.0.0-alpha-2 > > > Currently the docs have some code examples, but they don't include any > mention of actually submitting jobs. Esp what folks should or should not > worry about wrt classpath. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-8075) HRegion preClose hook is getting called multiple times for single close
[
https://issues.apache.org/jira/browse/HBASE-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354533#comment-17354533
]
Duo Zhang commented on HBASE-8075:
--
Any updates here?
Seems we still have 3 places to call preClose CP hook...
> HRegion preClose hook is getting called multiple times for single close
> ---
>
> Key: HBASE-8075
> URL: https://issues.apache.org/jira/browse/HBASE-8075
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 1.3.0, 1.2.1, 1.1.4, 0.98.19, 2.0.0
>Reporter: rajeshbabu
>Assignee: Stephen Yuan Jiang
>Priority: Critical
> Fix For: 3.0.0-alpha-1
>
> Attachments: HBASE-8075.v1-master.patch
>
>
> 1) HRegionServer#closeRegion(final RpcController controller,
> final CloseRegionRequest request)
> {code}
> // Can be null if we're calling close on a region that's not online
> final HRegion region = this.getFromOnlineRegions(encodedRegionName);
> if ((region != null) && (region .getCoprocessorHost() != null)) {
> region.getCoprocessorHost().preClose(false);
> }
> {code}
> 2) HRegionServer#closeRegion(String encodedName, final boolean abort,
> final boolean zk, final int versionOfClosingNode, final ServerName sn)
> {code}
> if ((actualRegion != null) && (actualRegion.getCoprocessorHost() !=
> null)) {
> try {
> actualRegion.getCoprocessorHost().preClose(false);
> } catch (IOException exp) {
> LOG.warn("Unable to close region: the coprocessor launched an error
> ", exp);
> return false;
> }
> }
> {code}
> 3) HRegion# private List doClose(
> final boolean abort, MonitoredTask status)
> {code}
> if (coprocessorHost != null) {
> status.setStatus("Running coprocessor pre-close hooks");
> this.coprocessorHost.preClose(abort);
> }
> {code}
> IMO 3rd one is enough and remaining two are not needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-13126) Move HBaseTestingUtility and associated support classes into hbase-testing-utility module
[ https://issues.apache.org/jira/browse/HBASE-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-13126: -- Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > Move HBaseTestingUtility and associated support classes into > hbase-testing-utility module > - > > Key: HBASE-13126 > URL: https://issues.apache.org/jira/browse/HBASE-13126 > Project: HBase > Issue Type: Task > Components: API >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 3.0.0-alpha-2 > > > Over in the review for HBASE-12972, [~enis] mentioned that one of the HBTU > methods wasn't intended for public consumption. > Can we build a list of such methods across the API, appropriately annotate > them for 2.0, and deprecate them in earlier versions with a warning that > they're going to be restricted? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-6189) Snappy build instructions are out of date
[ https://issues.apache.org/jira/browse/HBASE-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354528#comment-17354528 ] Duo Zhang commented on HBASE-6189: -- We just need a documentation update here? No? > Snappy build instructions are out of date > - > > Key: HBASE-6189 > URL: https://issues.apache.org/jira/browse/HBASE-6189 > Project: HBase > Issue Type: Bug > Components: build, documentation, Performance >Affects Versions: 0.92.1, 0.94.0, 0.95.2 >Reporter: Dave Revell >Assignee: Sean Busbey >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > In the ref guide (http://hbase.apache.org/book.html#build.snappy), it says to > build snappy by passing -Dsnappy. Something's wrong here, because: > 1. this causes the build to fail because the hadoop-snappy tar artifact can't > be found by maven > 2. the snappy classes are already included in hadoop 1.0, so using the snappy > profile is unnecessary > It would be great if someone who knows when/why to use the snappy profile > could fix the instructions (and fix the POM if necessary). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21873) region can be assigned to 2 servers due to a timed-out call or an unknown exception
[
https://issues.apache.org/jira/browse/HBASE-21873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-21873:
--
Priority: Major (was: Blocker)
> region can be assigned to 2 servers due to a timed-out call or an unknown
> exception
> ---
>
> Key: HBASE-21873
> URL: https://issues.apache.org/jira/browse/HBASE-21873
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.2.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
> Attachments: HBASE-21862-forUT.patch, HBASE-21862-v1.patch,
> HBASE-21862-v2.patch, HBASE-21862.patch
>
>
> It's a classic bug, sort of... the call times out to open the region, but RS
> actually processes it alright. It could also happen if the response didn't
> make it back due to a network issue.
> As a result region is opened on two servers.
> There are some mitigations possible to narrow down the race window.
> 1) Don't process expired open calls, fail them. Won't help for network issues.
> 2) Don't ignore invalid RS state, kill it (YouAreDead exception) - but that
> will require fixing other network races where master kills RS, which would
> require adding state versioning to the protocol.
> The fundamental fix though would require either
> 1) an unknown failure from open to ascertain the state of the region from the
> server. Again, this would probably require protocol changes to make sure we
> ascertain the region is not opened, and also that the
> already-failed-on-master open is NOT going to be processed if it's some queue
> or even in transit on the network (via a nonce-like mechanism)?
> 2) some form of a distributed lock per region, e.g. in ZK
> 3) some form of 2PC? but the participant list cannot be determined in a
> manner that's both scalable and guaranteed correct. Theoretically it could be
> all RSes.
> {noformat}
> 2019-02-08 03:21:31,715 INFO [PEWorker-7]
> procedure.MasterProcedureScheduler: Took xlock for pid=260626, ppid=260595,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false;
> TransitRegionStateProcedure table=table,
> region=d0214809147e43dc6870005742d5d204, ASSIGN
> 2019-02-08 03:21:31,758 INFO [PEWorker-7]
> assignment.TransitRegionStateProcedure: Starting pid=260626, ppid=260595,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true;
> TransitRegionStateProcedure table=table,
> region=d0214809147e43dc6870005742d5d204, ASSIGN; rit=OPEN,
> location=server1,17020,1549567999303; forceNewPlan=false, retain=true
> 2019-02-08 03:21:31,984 INFO [PEWorker-13] assignment.RegionStateStore:
> pid=260626 updating hbase:meta row=d0214809147e43dc6870005742d5d204,
> regionState=OPENING, regionLocation=server1,17020,1549623714617
> 2019-02-08 03:22:32,552 WARN [RSProcedureDispatcher-pool4-t3451]
> assignment.RegionRemoteProcedureBase: The remote operation pid=260637,
> ppid=260626, state=RUNNABLE, hasLock=false;
> org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure for region ...
> to server server1,17020,1549623714617 failed
> java.io.IOException: Call to server1/...:17020 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=27191,
> waitTime=60145, rpcTimeout=6^M
> at
> org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:185)^M
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:391)^M
> ...
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=27191,
> waitTime=60145, rpcTimeout=6^M
> at
> org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:200)^M
> ... 4 more^M
> {noformat}
> RS:
> {noformat}
> hbase-regionserver.log:2019-02-08 03:22:41,131 INFO
> [RS_OPEN_REGION-regionserver/server1:17020-2] handler.AssignRegionHandler:
> Open ...d0214809147e43dc6870005742d5d204.
> ...
> hbase-regionserver.log:2019-02-08 03:25:44,751 INFO
> [RS_OPEN_REGION-regionserver/server1:17020-2] handler.AssignRegionHandler:
> Opened ...d0214809147e43dc6870005742d5d204.
> {noformat}
> Retry:
> {noformat}
> 2019-02-08 03:22:32,967 INFO [PEWorker-6]
> assignment.TransitRegionStateProcedure: Retry=1 of max=2147483647;
> pid=260626, ppid=260595,
> state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true;
> TransitRegionStateProcedure table=table,
> region=d0214809147e43dc6870005742d5d204, ASSIGN; rit=OPENING,
> location=server1,17020,1549623714617
> 2019-02-08 03:22:33,084 INFO [PEWorker-6]
> assignment.TransitRegionStateProcedure: Starting pid=260626, ppid=260595,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true;
> TransitRegionStateProcedure table=table,
[jira] [Updated] (HBASE-21873) region can be assigned to 2 servers due to a timed-out call or an unknown exception
[
https://issues.apache.org/jira/browse/HBASE-21873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-21873:
--
Fix Version/s: (was: 3.0.0-alpha-1)
3.0.0-alpha-2
> region can be assigned to 2 servers due to a timed-out call or an unknown
> exception
> ---
>
> Key: HBASE-21873
> URL: https://issues.apache.org/jira/browse/HBASE-21873
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.2.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Fix For: 3.0.0-alpha-2
>
> Attachments: HBASE-21862-forUT.patch, HBASE-21862-v1.patch,
> HBASE-21862-v2.patch, HBASE-21862.patch
>
>
> It's a classic bug, sort of... the call times out to open the region, but RS
> actually processes it alright. It could also happen if the response didn't
> make it back due to a network issue.
> As a result region is opened on two servers.
> There are some mitigations possible to narrow down the race window.
> 1) Don't process expired open calls, fail them. Won't help for network issues.
> 2) Don't ignore invalid RS state, kill it (YouAreDead exception) - but that
> will require fixing other network races where master kills RS, which would
> require adding state versioning to the protocol.
> The fundamental fix though would require either
> 1) an unknown failure from open to ascertain the state of the region from the
> server. Again, this would probably require protocol changes to make sure we
> ascertain the region is not opened, and also that the
> already-failed-on-master open is NOT going to be processed if it's some queue
> or even in transit on the network (via a nonce-like mechanism)?
> 2) some form of a distributed lock per region, e.g. in ZK
> 3) some form of 2PC? but the participant list cannot be determined in a
> manner that's both scalable and guaranteed correct. Theoretically it could be
> all RSes.
> {noformat}
> 2019-02-08 03:21:31,715 INFO [PEWorker-7]
> procedure.MasterProcedureScheduler: Took xlock for pid=260626, ppid=260595,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false;
> TransitRegionStateProcedure table=table,
> region=d0214809147e43dc6870005742d5d204, ASSIGN
> 2019-02-08 03:21:31,758 INFO [PEWorker-7]
> assignment.TransitRegionStateProcedure: Starting pid=260626, ppid=260595,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true;
> TransitRegionStateProcedure table=table,
> region=d0214809147e43dc6870005742d5d204, ASSIGN; rit=OPEN,
> location=server1,17020,1549567999303; forceNewPlan=false, retain=true
> 2019-02-08 03:21:31,984 INFO [PEWorker-13] assignment.RegionStateStore:
> pid=260626 updating hbase:meta row=d0214809147e43dc6870005742d5d204,
> regionState=OPENING, regionLocation=server1,17020,1549623714617
> 2019-02-08 03:22:32,552 WARN [RSProcedureDispatcher-pool4-t3451]
> assignment.RegionRemoteProcedureBase: The remote operation pid=260637,
> ppid=260626, state=RUNNABLE, hasLock=false;
> org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure for region ...
> to server server1,17020,1549623714617 failed
> java.io.IOException: Call to server1/...:17020 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=27191,
> waitTime=60145, rpcTimeout=6^M
> at
> org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:185)^M
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:391)^M
> ...
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=27191,
> waitTime=60145, rpcTimeout=6^M
> at
> org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:200)^M
> ... 4 more^M
> {noformat}
> RS:
> {noformat}
> hbase-regionserver.log:2019-02-08 03:22:41,131 INFO
> [RS_OPEN_REGION-regionserver/server1:17020-2] handler.AssignRegionHandler:
> Open ...d0214809147e43dc6870005742d5d204.
> ...
> hbase-regionserver.log:2019-02-08 03:25:44,751 INFO
> [RS_OPEN_REGION-regionserver/server1:17020-2] handler.AssignRegionHandler:
> Opened ...d0214809147e43dc6870005742d5d204.
> {noformat}
> Retry:
> {noformat}
> 2019-02-08 03:22:32,967 INFO [PEWorker-6]
> assignment.TransitRegionStateProcedure: Retry=1 of max=2147483647;
> pid=260626, ppid=260595,
> state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true;
> TransitRegionStateProcedure table=table,
> region=d0214809147e43dc6870005742d5d204, ASSIGN; rit=OPENING,
> location=server1,17020,1549623714617
> 2019-02-08 03:22:33,084 INFO [PEWorker-6]
> assignment.TransitRegionStateProcedure: Starting pid=260626, ppid=260595,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true;
[jira] [Updated] (HBASE-13740) Stop using Hadoop private interfaces
[ https://issues.apache.org/jira/browse/HBASE-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-13740: -- Priority: Major (was: Blocker) > Stop using Hadoop private interfaces > > > Key: HBASE-13740 > URL: https://issues.apache.org/jira/browse/HBASE-13740 > Project: HBase > Issue Type: Umbrella >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Priority: Major > Fix For: 3.0.0-alpha-1 > > > Now that we are push downstream folks to stay off of our private interfaces, > we should provide a good example by doing the same with Hadoop. > Things to do in this umbrella > * We need a good way to check; manual inspection is untenable > * For anything where Hadoop isn't maintaining an isolated API (i.e. they > include a non-org.apache.hadoop or jvm class), we should just rip things out > * For the rest we'll need to determine if we ask for upgrading things to > LimitedPrivate(HBase) or Public -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-13740) Stop using Hadoop private interfaces
[ https://issues.apache.org/jira/browse/HBASE-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-13740: -- Fix Version/s: (was: 3.0.0-alpha-1) 3.0.0-alpha-2 > Stop using Hadoop private interfaces > > > Key: HBASE-13740 > URL: https://issues.apache.org/jira/browse/HBASE-13740 > Project: HBase > Issue Type: Umbrella >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Priority: Major > Fix For: 3.0.0-alpha-2 > > > Now that we are push downstream folks to stay off of our private interfaces, > we should provide a good example by doing the same with Hadoop. > Things to do in this umbrella > * We need a good way to check; manual inspection is untenable > * For anything where Hadoop isn't maintaining an isolated API (i.e. they > include a non-org.apache.hadoop or jvm class), we should just rip things out > * For the rest we'll need to determine if we ask for upgrading things to > LimitedPrivate(HBase) or Public -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23634) Enable "Split WAL to HFile" by default
[ https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354525#comment-17354525 ] Duo Zhang commented on HBASE-23634: --- Any updates here? > Enable "Split WAL to HFile" by default > -- > > Key: HBASE-23634 > URL: https://issues.apache.org/jira/browse/HBASE-23634 > Project: HBase > Issue Type: Task >Affects Versions: 3.0.0-alpha-1, 2.3.0 >Reporter: Guanghao Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3339: HBASE-25947 Backport 'HBASE-25894 Improve the performance for region …
Apache-HBase commented on pull request #3339: URL: https://github.com/apache/hbase/pull/3339#issuecomment-851566777 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 26s | branch-2.4 passed | | +1 :green_heart: | compile | 3m 25s | branch-2.4 passed | | +1 :green_heart: | checkstyle | 1m 12s | branch-2.4 passed | | +1 :green_heart: | spotbugs | 2m 6s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 30s | the patch passed | | +1 :green_heart: | compile | 3m 20s | the patch passed | | +1 :green_heart: | javac | 3m 20s | the patch passed | | +1 :green_heart: | checkstyle | 1m 10s | hbase-server: The patch generated 0 new + 57 unchanged - 5 fixed = 57 total (was 62) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 26s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 47m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3339/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3339 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 3ebf380f57d6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / 1972261826 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3339/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25649) Complete the work on moving all the balancer related classes to hbase-balancer module
[ https://issues.apache.org/jira/browse/HBASE-25649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25649. --- Hadoop Flags: Reviewed Resolution: Fixed All sub tasks are done. Created HBASE-25953 for rs group related migration. Resolve. > Complete the work on moving all the balancer related classes to > hbase-balancer module > - > > Key: HBASE-25649 > URL: https://issues.apache.org/jira/browse/HBASE-25649 > Project: HBase > Issue Type: Umbrella > Components: Balancer >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1 > > > This is the follow up issue of HBASE-23933, where we set up the new > hbase-balancer module and moved several classes into it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25649) Complete the work on moving all the balancer related classes to hbase-balancer module
[ https://issues.apache.org/jira/browse/HBASE-25649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-25649: -- Release Note: Introduced a ClusterInfoProvider as a bridge for LoadBalancer implementation to get cluster information and operate on the cluster, so the LoadBalancer implementation does not need to depend on HMaster/MasterServices directly. So then we could move most LoadBalancer related code to hbase-balancer module. MasterClusterInfoProvider is the actual implementation class for ClusterInfoProvider. Moved most unit tests for load balancer to hbase-balancer module, unless it requires starting a mini cluster. RSGroupBasedLoadBalancer and related code are still in hbase-server module. As we still have lots of migration and compatibility code, which makes it deeply tied togather with HMaster/MasterServices. We could try to move it again in 4.0.0, when we have purged the migration and compatibility code. > Complete the work on moving all the balancer related classes to > hbase-balancer module > - > > Key: HBASE-25649 > URL: https://issues.apache.org/jira/browse/HBASE-25649 > Project: HBase > Issue Type: Umbrella > Components: Balancer >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1 > > > This is the follow up issue of HBASE-23933, where we set up the new > hbase-balancer module and moved several classes into it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22889) hbase-shaded-client for Hbase 2.x does not include all the classes it needs at runtime
[
https://issues.apache.org/jira/browse/HBASE-22889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354511#comment-17354511
]
Oleg Muravskiy commented on HBASE-22889:
The same is true for shaded
`org.apache.hadoop.hbase.shaded.org.codehaus.jackson.*`.
hbase-shaded-client uses `org.apache.hadoop.conf.Configuration`, which imports
two classes from `org.apache.hadoop.hbase.shaded.org.codehaus.jackson` package,
but this package is not present in hbase-shaded-client artefact.
I would also raise the priority of this issue, as currently it isn't possible
to only depend on hbase-shaded-client at runtime.
> hbase-shaded-client for Hbase 2.x does not include all the classes it needs
> at runtime
> --
>
> Key: HBASE-22889
> URL: https://issues.apache.org/jira/browse/HBASE-22889
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Ismaël Mejía
>Priority: Minor
>
> When running a simple Job to read data from HBase 2 I got this exception, so
> it seems not all needed classes are included in `hbase-shaded-client`
> {quote}java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/shaded/org/mortbay/jetty/nio/SelectChannelConnector
> at com.example.hbase.HBaseIOTest.beforeClass(HBaseIOTest.java:79)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.shaded.org.mortbay.jetty.nio.SelectChannelConnector
> at com.example.hbase.HBaseIOTest.beforeClass(HBaseIOTest.java:79{quote}
> If I replace the dependency with just `hbase-client` everything works
> correctly.
> I created an external repo to test this in case someone may want to take a
> look.
> https://github.com/iemejia/hbase-shaded-server-test/blob/f644a7b9850d416f19093c717b73dd8c1048c553/pom.xml#L55
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25649) Complete the work on moving all the balancer related classes to hbase-balancer module
[ https://issues.apache.org/jira/browse/HBASE-25649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354509#comment-17354509 ] Duo Zhang commented on HBASE-25649: --- After reviewing the code for rs group again, I do not think it is worth to force move it to hbase-balancer for now. We have a lot of code for migrating and keeping compatibility, which use a lot of internal stuff of HMaster, so it is not easy to move it hbase-balancer module. We could try to do this again in 4.0.0, when we can remove the migration and compatibility code. Filed HBASE-25953 for this. > Complete the work on moving all the balancer related classes to > hbase-balancer module > - > > Key: HBASE-25649 > URL: https://issues.apache.org/jira/browse/HBASE-25649 > Project: HBase > Issue Type: Umbrella > Components: Balancer >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1 > > > This is the follow up issue of HBASE-23933, where we set up the new > hbase-balancer module and moved several classes into it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25649) Complete the work on moving all the balancer related classes to hbase-balancer module
[ https://issues.apache.org/jira/browse/HBASE-25649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-25649: -- Fix Version/s: 3.0.0-alpha-1 > Complete the work on moving all the balancer related classes to > hbase-balancer module > - > > Key: HBASE-25649 > URL: https://issues.apache.org/jira/browse/HBASE-25649 > Project: HBase > Issue Type: Umbrella > Components: Balancer >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1 > > > This is the follow up issue of HBASE-23933, where we set up the new > hbase-balancer module and moved several classes into it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25917) Move RSGroupBasedLoadBalancer to hbase-balancer
[ https://issues.apache.org/jira/browse/HBASE-25917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-25917: -- Parent Issue: HBASE-25953 (was: HBASE-25649) > Move RSGroupBasedLoadBalancer to hbase-balancer > --- > > Key: HBASE-25917 > URL: https://issues.apache.org/jira/browse/HBASE-25917 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25649) Complete the work on moving all the balancer related classes to hbase-balancer module
[ https://issues.apache.org/jira/browse/HBASE-25649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-25649: - Assignee: Duo Zhang > Complete the work on moving all the balancer related classes to > hbase-balancer module > - > > Key: HBASE-25649 > URL: https://issues.apache.org/jira/browse/HBASE-25649 > Project: HBase > Issue Type: Umbrella > Components: Balancer >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > This is the follow up issue of HBASE-23933, where we set up the new > hbase-balancer module and moved several classes into it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25949) Move RSGroupInfoManager to hbase-balancer
[ https://issues.apache.org/jira/browse/HBASE-25949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-25949: -- Parent Issue: HBASE-25953 (was: HBASE-25649) > Move RSGroupInfoManager to hbase-balancer > - > > Key: HBASE-25949 > URL: https://issues.apache.org/jira/browse/HBASE-25949 > Project: HBase > Issue Type: Sub-task > Components: Balancer, rsgroup >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25953) Move rs group related balancer code to hbase-balancer module
Duo Zhang created HBASE-25953: - Summary: Move rs group related balancer code to hbase-balancer module Key: HBASE-25953 URL: https://issues.apache.org/jira/browse/HBASE-25953 Project: HBase Issue Type: Umbrella Components: Balancer, rsgroup Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25916) Move FavoredNodeLoadBalancer to hbase-balancer module
[ https://issues.apache.org/jira/browse/HBASE-25916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25916. --- Fix Version/s: 3.0.0-alpha-1 Hadoop Flags: Reviewed Assignee: Duo Zhang Resolution: Fixed Merged to master. Thanks [~niuyulin] for reviewing. > Move FavoredNodeLoadBalancer to hbase-balancer module > - > > Key: HBASE-25916 > URL: https://issues.apache.org/jira/browse/HBASE-25916 > Project: HBase > Issue Type: Sub-task > Components: Balancer, FavoredNodes >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 merged pull request #3327: HBASE-25916 Move FavoredNodeLoadBalancer to hbase-balancer module
Apache9 merged pull request #3327: URL: https://github.com/apache/hbase/pull/3327 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #3336: HBASE-25920 Support Hadoop 3.3.1
saintstack commented on pull request #3336: URL: https://github.com/apache/hbase/pull/3336#issuecomment-851540643 TestMergeTableRegionsWhileRSCrash failed this time. Rerun. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3332: HBASE-25932: Ensure replication reads the trailer bytes from WAL.
Apache-HBase commented on pull request #3332: URL: https://github.com/apache/hbase/pull/3332#issuecomment-851537971 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 21s | master passed | | +1 :green_heart: | compile | 3m 26s | master passed | | +1 :green_heart: | checkstyle | 1m 9s | master passed | | +1 :green_heart: | spotbugs | 2m 12s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 1s | the patch passed | | +1 :green_heart: | compile | 3m 20s | the patch passed | | +1 :green_heart: | javac | 3m 20s | the patch passed | | -0 :warning: | checkstyle | 1m 9s | hbase-server: The patch generated 2 new + 11 unchanged - 0 fixed = 13 total (was 11) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 58s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 21s | the patch passed | ||| _ Other Tests _ | | -1 :x: | asflicense | 0m 13s | The patch generated 1 ASF License warnings. | | | | 51m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3332 | | JIRA Issue | HBASE-25932 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux bf344310ac48 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 1ccba10847 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | asflicense | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/artifact/yetus-general-check/output/patch-asflicense-problems.txt | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3332/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on a change in pull request #3319: HBASE-25914 Provide slow/large logs on RegionServer UI
virajjasani commented on a change in pull request #3319:
URL: https://github.com/apache/hbase/pull/3319#discussion_r642508139
##
File path:
hbase-server/src/main/resources/hbase-webapps/regionserver/rsNamedQueueLog.jsp
##
@@ -0,0 +1,157 @@
+<%--
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+--%>
+<%@ page contentType="text/html;charset=UTF-8"
+ import="java.util.Date"
+ import="java.util.List"
+ import="java.util.Arrays"
+ import="java.util.HashSet"
+ import="org.apache.hadoop.conf.Configuration"
+ import="org.apache.hadoop.hbase.client.Admin"
+ import="org.apache.hadoop.hbase.client.SnapshotDescription"
+ import="org.apache.hadoop.hbase.http.InfoServer"
+ import="org.apache.hadoop.hbase.master.HMaster"
+ import="org.apache.hadoop.hbase.snapshot.SnapshotInfo"
+ import="org.apache.hadoop.util.StringUtils"
+ import="org.apache.hadoop.hbase.TableName"
+ import="org.apache.hadoop.hbase.client.ServerType"
+ import="org.apache.hadoop.hbase.client.LogEntry"
+ import="org.apache.hadoop.hbase.regionserver.HRegionServer"
+ import="org.apache.hadoop.hbase.client.OnlineLogRecord"
+ import="org.apache.hadoop.hbase.HConstants"
+ import="org.apache.hadoop.hbase.ServerName"
+%>
+<%
+ HRegionServer rs = (HRegionServer)
getServletContext().getAttribute(HRegionServer.REGIONSERVER);
+ Configuration conf = rs.getConfiguration();
+ List slowLogs = null;
+ List largeLogs = null;
+
+ if(rs.isOnline()) {
+try (Admin rsAdmin = rs.getConnection().getAdmin()) {
+ slowLogs = (List)(List)rsAdmin.getLogEntries(new
HashSet(Arrays.asList(rs.getServerName())), "SLOW_LOG",
ServerType.REGION_SERVER, HConstants.DEFAULT_SLOW_LOG_RING_BUFFER_SIZE, null);
Review comment:
@GeorryHuang Thanks, yes you can try to explore if you are able to
directly fetch records from RegionServer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase-thirdparty] Apache-HBase commented on pull request #54: HBASE-25946 [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.m…
Apache-HBase commented on pull request #54: URL: https://github.com/apache/hbase-thirdparty/pull/54#issuecomment-851507819 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 1m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-54/1/artifact/yetus-precommit-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase-thirdparty/pull/54 | | Optional Tests | dupname asflicense markdownlint | | uname | Linux 8e1d38c52a15 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 GNU/Linux | | Build tool | maven | | git revision | master / 9fa54d4 | | Max. process+thread count | 28 (vs. ulimit of 1000) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-54/1/console | | versions | git=2.20.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25952) [hbase-thirdparty] Put up 3.5.1RC0
Duo Zhang created HBASE-25952: - Summary: [hbase-thirdparty] Put up 3.5.1RC0 Key: HBASE-25952 URL: https://issues.apache.org/jira/browse/HBASE-25952 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase-thirdparty] Apache9 opened a new pull request #54: HBASE-25946 [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.m…
Apache9 opened a new pull request #54: URL: https://github.com/apache/hbase-thirdparty/pull/54 …d for 3.5.1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25946) [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.5.1
[ https://issues.apache.org/jira/browse/HBASE-25946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25946. --- Resolution: Fixed For generating CHANGES.md. > [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.5.1 > > > Key: HBASE-25946 > URL: https://issues.apache.org/jira/browse/HBASE-25946 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: thirdparty-3.5.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-25946) [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.5.1
[ https://issues.apache.org/jira/browse/HBASE-25946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-25946: --- > [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.5.1 > > > Key: HBASE-25946 > URL: https://issues.apache.org/jira/browse/HBASE-25946 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: thirdparty-3.5.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25863) [hbase-thirdparty] Shade javax.ws.rs package for use with shaded Jersey
[ https://issues.apache.org/jira/browse/HBASE-25863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-25863: -- Fix Version/s: (was: thirdparty-3.5.1) thirdparty-4.0.0 > [hbase-thirdparty] Shade javax.ws.rs package for use with shaded Jersey > --- > > Key: HBASE-25863 > URL: https://issues.apache.org/jira/browse/HBASE-25863 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > Fix For: thirdparty-4.0.0 > > > From a comment on the parent ticket, > bq. Jersey is an implementation of the specifications that define that > package and its subcomponents. Thus, to use a shaded version of Jersey, one > must also shade usage of this package as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25946) [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.5.1
[ https://issues.apache.org/jira/browse/HBASE-25946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-25946: - Assignee: Duo Zhang > [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.5.1 > > > Key: HBASE-25946 > URL: https://issues.apache.org/jira/browse/HBASE-25946 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: thirdparty-3.5.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25945) [hbase-thirdparty] Set version as 3.5.1 in prep for first RC
[ https://issues.apache.org/jira/browse/HBASE-25945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25945. --- Hadoop Flags: Reviewed Assignee: Duo Zhang Resolution: Fixed Merged to master. Thanks [~psomogyi] for reviewing. > [hbase-thirdparty] Set version as 3.5.1 in prep for first RC > > > Key: HBASE-25945 > URL: https://issues.apache.org/jira/browse/HBASE-25945 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: thirdparty-3.5.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase-thirdparty] Apache9 merged pull request #53: HBASE-25945 [hbase-thirdparty] Set version as 3.5.1 in prep for first RC
Apache9 merged pull request #53: URL: https://github.com/apache/hbase-thirdparty/pull/53 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25939) Move more tests code for StochasticLoadBalancer to hbase-balancer module
[
https://issues.apache.org/jira/browse/HBASE-25939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354469#comment-17354469
]
Hudson commented on HBASE-25939:
Results for branch master
[build #310 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/310/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/310/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/310/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/310/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Move more tests code for StochasticLoadBalancer to hbase-balancer module
>
>
> Key: HBASE-25939
> URL: https://issues.apache.org/jira/browse/HBASE-25939
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, test
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25951) affect version is editable
Ankur Tagra created HBASE-25951: --- Summary: affect version is editable Key: HBASE-25951 URL: https://issues.apache.org/jira/browse/HBASE-25951 Project: HBase Issue Type: Bug Affects Versions: 1.4.12 Reporter: Ankur Tagra Fix For: 1.4.14 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25891) Remove dependence storing WAL filenames for backup
[
https://issues.apache.org/jira/browse/HBASE-25891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354430#comment-17354430
]
Mallikarjun commented on HBASE-25891:
-
[~anoop.hbase] Not there completely. Let me put down the details and share it.
> Remove dependence storing WAL filenames for backup
> --
>
> Key: HBASE-25891
> URL: https://issues.apache.org/jira/browse/HBASE-25891
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Affects Versions: 3.0.0-alpha-1
>Reporter: Mallikarjun
>Assignee: Mallikarjun
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> Currently WAL logs are stored in `backup:system` meta table
> {code:java}
> // code placeholder
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:backupId,
> timestamp=1622003479895, value=backup_1622003358258
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:file,
> timestamp=1622003479895,
> value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621996160175
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:root,
> timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:backupId,
> timestamp=1622003479895, value=backup_1622003358258
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:file,
> timestamp=1622003479895,
> value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621999760280
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:root,
> timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
> {code}
>
> This has several problems
> # Ever growing rows of wal's sourced for incremental backup is maintained
> and never cleaned up.
> # Unnecessary to have wal log listed for performing incremental backup or
> for performing logcleaner.
> # No support for rsgroup. Hence all rsgroups which doesn't have backup
> enabled tables, WAL's are retained forever.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25914) Provide slow/large logs on RegionServer UI
[ https://issues.apache.org/jira/browse/HBASE-25914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354390#comment-17354390 ] Viraj Jasani commented on HBASE-25914: -- Default size is fine. Thanks > Provide slow/large logs on RegionServer UI > -- > > Key: HBASE-25914 > URL: https://issues.apache.org/jira/browse/HBASE-25914 > Project: HBase > Issue Type: Improvement > Components: regionserver, UI >Affects Versions: 3.0.0-alpha-1, 2.5.0 >Reporter: Zhuoyue Huang >Assignee: Zhuoyue Huang >Priority: Major > Attachments: callDetails.png, largeLog.png, slowLog.png > > > Pulling slow/large log from in-memory queues on RegionServer then display > details info in RegionServer status UI -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25914) Provide slow/large logs on RegionServer UI
[
https://issues.apache.org/jira/browse/HBASE-25914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354389#comment-17354389
]
Zhuoyue Huang commented on HBASE-25914:
---
{quote} In the current implementation we directly use the max queue size as
param to retrieve all log records
{quote}
Not max size, but default size
> Provide slow/large logs on RegionServer UI
> --
>
> Key: HBASE-25914
> URL: https://issues.apache.org/jira/browse/HBASE-25914
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, UI
>Affects Versions: 3.0.0-alpha-1, 2.5.0
>Reporter: Zhuoyue Huang
>Assignee: Zhuoyue Huang
>Priority: Major
> Attachments: callDetails.png, largeLog.png, slowLog.png
>
>
> Pulling slow/large log from in-memory queues on RegionServer then display
> details info in RegionServer status UI
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25914) Provide slow/large logs on RegionServer UI
[
https://issues.apache.org/jira/browse/HBASE-25914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354386#comment-17354386
]
Zhuoyue Huang commented on HBASE-25914:
---
{quote}In the past, folks have been worried about showing full RPC payload in
logs... privacy/security concerns. The full display of rpc payload should
probably be an opt-in switch.{quote}
Agree! 'hbase.regionserver.slowlog.buffer.enabled' already set whether the RS
should record 'Slow Logs/Large Logs'. I think we can use this conf to decide
whether UI should display the so-called 'Slow Logs' page in UI. any other
suggestions?
> Provide slow/large logs on RegionServer UI
> --
>
> Key: HBASE-25914
> URL: https://issues.apache.org/jira/browse/HBASE-25914
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, UI
>Affects Versions: 3.0.0-alpha-1, 2.5.0
>Reporter: Zhuoyue Huang
>Assignee: Zhuoyue Huang
>Priority: Major
> Attachments: callDetails.png, largeLog.png, slowLog.png
>
>
> Pulling slow/large log from in-memory queues on RegionServer then display
> details info in RegionServer status UI
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25914) Provide slow/large logs on RegionServer UI
[
https://issues.apache.org/jira/browse/HBASE-25914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354385#comment-17354385
]
Zhuoyue Huang commented on HBASE-25914:
---
{quote}I think so and hopefully it will be scrollable. [Zhuoyue
Huang|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=GeorryHuang]
is my assumption correct? As API supports limit param, let's have a drop down
on UI to limit no of recent records in UI?{quote}
[~vjasani] Yes, you are right. But should we really need a drop-down on UI? In
the current implementation we directly use the max queue size as param to
retrieve all log records(Now, using admin API to retrieve RS log entries maybe
not a good way.)
Using page buttons to switch pages, and each page showing fixed-line records is
another good way(just like the UI of user tables page)?
> Provide slow/large logs on RegionServer UI
> --
>
> Key: HBASE-25914
> URL: https://issues.apache.org/jira/browse/HBASE-25914
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, UI
>Affects Versions: 3.0.0-alpha-1, 2.5.0
>Reporter: Zhuoyue Huang
>Assignee: Zhuoyue Huang
>Priority: Major
> Attachments: callDetails.png, largeLog.png, slowLog.png
>
>
> Pulling slow/large log from in-memory queues on RegionServer then display
> details info in RegionServer status UI
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] tomscut commented on pull request #3325: HBASE-25934 Add username for RegionScannerHolder
tomscut commented on pull request #3325: URL: https://github.com/apache/hbase/pull/3325#issuecomment-851417232 Hi @saintstack , those failed unit tests look unrelated to the change, and they work fine locally. org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor5.testAppendWithoutWAL[1] org.apache.hadoop.hbase.replication.multiwal.TestReplicationEndpointWithMultipleWAL org.apache.hadoop.hbase.security.access.TestSnapshotScannerHDFSAclController.(?) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25891) Remove dependence storing WAL filenames for backup
[
https://issues.apache.org/jira/browse/HBASE-25891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354381#comment-17354381
]
Anoop Sam John commented on HBASE-25891:
Is there a new design doc somewhere? Instead of system table, where/how the
WAL files are being tracked? When the cleaning is possible finally (for WAL
files). When a full backup is taken next? Ya a detailed doc for backup (with
the changes now you are doing) would be a great start. May be its already there
but am missing.
> Remove dependence storing WAL filenames for backup
> --
>
> Key: HBASE-25891
> URL: https://issues.apache.org/jira/browse/HBASE-25891
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Affects Versions: 3.0.0-alpha-1
>Reporter: Mallikarjun
>Assignee: Mallikarjun
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> Currently WAL logs are stored in `backup:system` meta table
> {code:java}
> // code placeholder
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:backupId,
> timestamp=1622003479895, value=backup_1622003358258
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:file,
> timestamp=1622003479895,
> value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621996160175
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:root,
> timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:backupId,
> timestamp=1622003479895, value=backup_1622003358258
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:file,
> timestamp=1622003479895,
> value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621999760280
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:root,
> timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
> {code}
>
> This has several problems
> # Ever growing rows of wal's sourced for incremental backup is maintained
> and never cleaned up.
> # Unnecessary to have wal log listed for performing incremental backup or
> for performing logcleaner.
> # No support for rsgroup. Hence all rsgroups which doesn't have backup
> enabled tables, WAL's are retained forever.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25914) Provide slow/large logs on RegionServer UI
[
https://issues.apache.org/jira/browse/HBASE-25914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354378#comment-17354378
]
Zhuoyue Huang commented on HBASE-25914:
---
{quote}I'd expect some text on the page explaining what is being
displayed,{quote}
{quote}Would suggest a version of your explanation be added to the head of the
'Named Queue Log' tab to explain what it is.{quote}
[~stack] Ok, going to add explanation.
{quote}My concern w/ 'Name Queue Log' is that it indeed is generic, so generic,
the operator will be confounded by what it is they are looking at; the tab
needs explanatory text and a better name I'd suggest (HBase has many queues.
'Name'? 'Log' is usually a file on disk ).{quote}
{quote}On naming, instead of 'Name Queue Log' , which exposes the
implementation, something like 'Detail' or 'Operation Detail' with explanation
on the page of why the particular set of samples are displayed and of what the
samples are? Perhaps others have better input here. They are not 'Large Logs'
or 'Slow Logs', they are slow queries or requests or rpcs? And ditto for large,
no?{quote}
Let me think about how to name it. Thanks for your suggestion.
> Provide slow/large logs on RegionServer UI
> --
>
> Key: HBASE-25914
> URL: https://issues.apache.org/jira/browse/HBASE-25914
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, UI
>Affects Versions: 3.0.0-alpha-1, 2.5.0
>Reporter: Zhuoyue Huang
>Assignee: Zhuoyue Huang
>Priority: Major
> Attachments: callDetails.png, largeLog.png, slowLog.png
>
>
> Pulling slow/large log from in-memory queues on RegionServer then display
> details info in RegionServer status UI
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3336: HBASE-25920 Support Hadoop 3.3.1
Apache-HBase commented on pull request #3336: URL: https://github.com/apache/hbase/pull/3336#issuecomment-851412284 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 36s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 9s | master passed | | +1 :green_heart: | compile | 3m 31s | master passed | | +1 :green_heart: | shadedjars | 11m 30s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 23s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 1m 39s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 52s | the patch passed | | +1 :green_heart: | compile | 3m 40s | the patch passed | | +1 :green_heart: | javac | 3m 40s | the patch passed | | +1 :green_heart: | shadedjars | 12m 22s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 30s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 307m 37s | root in the patch failed. | | | | 364m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3336 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8db6cca3af82 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d164314489 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/4/testReport/ | | Max. process+thread count | 3869 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
