[GitHub] [hbase] Apache-HBase commented on pull request #3188: HBASE-25799 add clusterReadRequests and clusterWriteRequests jmx
Apache-HBase commented on pull request #3188: URL: https://github.com/apache/hbase/pull/3188#issuecomment-852725921 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 1s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 44s | master passed | | +1 :green_heart: | compile | 6m 3s | master passed | | +1 :green_heart: | checkstyle | 1m 55s | master passed | | +1 :green_heart: | spotbugs | 7m 24s | master passed | | -0 :warning: | patch | 2m 33s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 36s | the patch passed | | +1 :green_heart: | compile | 6m 0s | the patch passed | | +1 :green_heart: | cc | 6m 0s | the patch passed | | +1 :green_heart: | javac | 6m 0s | the patch passed | | -0 :warning: | checkstyle | 0m 27s | hbase-client: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) | | -0 :warning: | checkstyle | 1m 3s | hbase-server: The patch generated 2 new + 53 unchanged - 0 fixed = 55 total (was 53) | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 12s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | hbaseprotoc | 2m 22s | the patch passed | | +1 :green_heart: | spotbugs | 8m 1s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 49s | The patch does not generate ASF License warnings. | | | | 70m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3188/10/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3188 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool | | uname | Linux d5661e02c126 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 335305e0cf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3188/10/artifact/yetus-general-check/output/diff-checkstyle-hbase-client.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3188/10/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-hadoop-compat hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3188/10/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25960) Build includes unshaded netty .so; clashes w/ downstreamers who would use a different version of netty
[
https://issues.apache.org/jira/browse/HBASE-25960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355475#comment-17355475
]
Michael Stack commented on HBASE-25960:
---
If I check the thirdparty libs, I see that 3.2.0 has the shaded lib only. 3.3.0
is where we include shaded and non-shaded Perhaps it something about how
these libs are built?
> Build includes unshaded netty .so; clashes w/ downstreamers who would use a
> different version of netty
> --
>
> Key: HBASE-25960
> URL: https://issues.apache.org/jira/browse/HBASE-25960
> Project: HBase
> Issue Type: Bug
> Components: build
>Reporter: Michael Stack
>Priority: Major
>
> A coworker was trying to use hbase client in a fat application that uses a
> different netty version to what hbase uses internally. Their app would fail
> to launch because it kept bumping into an incompatible netty .so lib. Here
> are the unshaded netty .so's we bundle looking at hbase-2.4.1...:
> ./lib/hbase-shaded-netty-3.4.1.jar has:
> {code}
> META-INF/native/libnetty_transport_native_epoll_aarch_64.so
> META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_64.so
> META-INF/native/libnetty_transport_native_epoll_x86_64.so
> {code}
> (HBASE-25959 should fix the non-relocation of
> libnetty_transport_native_epoll_aarch_64).
> ./lib/shaded-clients/hbase-shaded-client-byo-hadoop-2.4.1.1-apple.jar has the
> same three .sos as does
> ./lib/shaded-clients/hbase-shaded-mapreduce-2.4.1.1-apple.jar
> and ./lib/shaded-clients/hbase-shaded-client-2.4.1.1-apple.jar
> We even bundle ./lib/netty-all-4.1.17.Final.jar which unsurprisingly has the
> netty .sos in it.
> Looking at published builds of hbase-thirdparty, I see that these too include
> the above trio of .sos... The hbase-shaded-netty includes them in 3.4.1
> https://repo1.maven.org/maven2/org/apache/hbase/thirdparty/hbase-shaded-netty/3.4.1/
> as does 3.5.0.
> I just tried running a build of hbase-thirdparty and it does NOT include the
> extras
> META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_aarch_64.so
> META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_64.so
> (it has the fix for aarch included... when I built)
> Here is link to the snapshot I made:
> https://repository.apache.org/content/repositories/orgapachehbase-1451/org/apache/hbase/thirdparty/hbase-shaded-netty/3.5.1-stack4/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase-operator-tools] anoopsjohn commented on a change in pull request #88: HBASE-25921 Fix Wrong FileSystem when running `filesystem` on non-HDFS storage
anoopsjohn commented on a change in pull request #88:
URL:
https://github.com/apache/hbase-operator-tools/pull/88#discussion_r643642855
##
File path: hbase-hbck2/src/main/java/org/apache/hbase/hbck1/HBaseFsck.java
##
@@ -1927,7 +1931,7 @@ public boolean rebuildMeta() throws IOException,
InterruptedException {
HBaseTestingUtility.closeRegionAndWAL(meta);
// Clean out the WAL we created and used here.
LOG.info("Deleting {}, result={}", waldir,
- CommonFSUtils.delete(FileSystem.get(getConf()), waldir, true));
+ HBCKFsUtils.delete(waldir.getFileSystem(getConf()), waldir, true));
Review comment:
Not on this patch. But here is an issue. The actual delete() call
happening within a Log statement. If INFO level not enabled, the delete wont
get called. Can we fix that also here pls?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (HBASE-25960) Build includes unshaded netty .so; clashes w/ downstreamers who would use a different version of netty
Michael Stack created HBASE-25960:
-
Summary: Build includes unshaded netty .so; clashes w/
downstreamers who would use a different version of netty
Key: HBASE-25960
URL: https://issues.apache.org/jira/browse/HBASE-25960
Project: HBase
Issue Type: Bug
Components: build
Reporter: Michael Stack
A coworker was trying to use hbase client in a fat application that uses a
different netty version to what hbase uses internally. Their app would fail to
launch because it kept bumping into an incompatible netty .so lib. Here are the
unshaded netty .so's we bundle looking at hbase-2.4.1...:
./lib/hbase-shaded-netty-3.4.1.jar has:
{code}
META-INF/native/libnetty_transport_native_epoll_aarch_64.so
META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_64.so
META-INF/native/libnetty_transport_native_epoll_x86_64.so
{code}
(HBASE-25959 should fix the non-relocation of
libnetty_transport_native_epoll_aarch_64).
./lib/shaded-clients/hbase-shaded-client-byo-hadoop-2.4.1.1-apple.jar has the
same three .sos as does
./lib/shaded-clients/hbase-shaded-mapreduce-2.4.1.1-apple.jar
and ./lib/shaded-clients/hbase-shaded-client-2.4.1.1-apple.jar
We even bundle ./lib/netty-all-4.1.17.Final.jar which unsurprisingly has the
netty .sos in it.
Looking at published builds of hbase-thirdparty, I see that these too include
the above trio of .sos... The hbase-shaded-netty includes them in 3.4.1
https://repo1.maven.org/maven2/org/apache/hbase/thirdparty/hbase-shaded-netty/3.4.1/
as does 3.5.0.
I just tried running a build of hbase-thirdparty and it does NOT include the
extras
META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_aarch_64.so
META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_64.so
(it has the fix for aarch included... when I built)
Here is link to the snapshot I made:
https://repository.apache.org/content/repositories/orgapachehbase-1451/org/apache/hbase/thirdparty/hbase-shaded-netty/3.5.1-stack4/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (HBASE-24975) HBASE shell update_config doesn't work for master node
[ https://issues.apache.org/jira/browse/HBASE-24975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tomscut updated HBASE-24975: Comment: was deleted (was: It works fine with version 2.3.2. 1 update hbase-site.xml. 2 update_config 'hostname,16000' .) > HBASE shell update_config doesn't work for master node > -- > > Key: HBASE-24975 > URL: https://issues.apache.org/jira/browse/HBASE-24975 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.3.1 >Reporter: Clara Xiong >Priority: Critical > > The helper message says: > Reload a subset of configuration on server 'servername' where servername is > host, port plus startcode. For example: > host187.example.com,60020,1289493121758 > See http://hbase.apache.org/book.html#dyn_config for more details. Here is how > you would run the command in the hbase shell: > hbase> update_config 'servername' > > But the command doesn't update config on master. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24975) HBASE shell update_config doesn't work for master node
[ https://issues.apache.org/jira/browse/HBASE-24975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17353121#comment-17353121 ] tomscut edited comment on HBASE-24975 at 6/2/21, 4:14 AM: -- It works fine with version 2.3.2. 1 update hbase-site.xml. 2 update_config 'hostname,16000' . was (Author: tomscut): It works fine with version 2.3.2. 1 update hbase-site.xml. 2 update_config 'hostname,16000' . > HBASE shell update_config doesn't work for master node > -- > > Key: HBASE-24975 > URL: https://issues.apache.org/jira/browse/HBASE-24975 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.3.1 >Reporter: Clara Xiong >Priority: Critical > > The helper message says: > Reload a subset of configuration on server 'servername' where servername is > host, port plus startcode. For example: > host187.example.com,60020,1289493121758 > See http://hbase.apache.org/book.html#dyn_config for more details. Here is how > you would run the command in the hbase shell: > hbase> update_config 'servername' > > But the command doesn't update config on master. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25952) [hbase-thirdparty] Put up 3.5.1RC0
[ https://issues.apache.org/jira/browse/HBASE-25952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-25952: - Assignee: Duo Zhang > [hbase-thirdparty] Put up 3.5.1RC0 > -- > > Key: HBASE-25952 > URL: https://issues.apache.org/jira/browse/HBASE-25952 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-25952) [hbase-thirdparty] Put up 3.5.1RC0
[ https://issues.apache.org/jira/browse/HBASE-25952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-25952 started by Duo Zhang. - > [hbase-thirdparty] Put up 3.5.1RC0 > -- > > Key: HBASE-25952 > URL: https://issues.apache.org/jira/browse/HBASE-25952 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25946) [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.5.1
[ https://issues.apache.org/jira/browse/HBASE-25946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25946. --- Hadoop Flags: Reviewed Resolution: Fixed > [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.5.1 > > > Key: HBASE-25946 > URL: https://issues.apache.org/jira/browse/HBASE-25946 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: thirdparty-3.5.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase-thirdparty] Apache9 merged pull request #54: HBASE-25946 [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.m…
Apache9 merged pull request #54: URL: https://github.com/apache/hbase-thirdparty/pull/54 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-thirdparty] Apache-HBase commented on pull request #54: HBASE-25946 [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.m…
Apache-HBase commented on pull request #54: URL: https://github.com/apache/hbase-thirdparty/pull/54#issuecomment-852674545 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 7s | The patch does not generate ASF License warnings. | | | | 0m 51s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-54/2/artifact/yetus-precommit-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase-thirdparty/pull/54 | | Optional Tests | dupname asflicense markdownlint | | uname | Linux fa98e3f1395b 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 GNU/Linux | | Build tool | maven | | git revision | master / 2e9e6ea | | Max. process+thread count | 28 (vs. ulimit of 1000) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-54/2/console | | versions | git=2.20.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on a change in pull request #3347: HBASE-25955 Setting NAMESPACES when adding a replication peer doesn't…
sandeepvinayak commented on a change in pull request #3347:
URL: https://github.com/apache/hbase/pull/3347#discussion_r643612020
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java
##
@@ -396,11 +447,11 @@ public void testMetricsSourceBaseSourcePassThrough() {
// after calling #setAgeOfLastShippedOpByTable
boolean containsRandomNewTable = source.getSingleSourceSourceByTable()
.containsKey("RandomNewTable");
-Assert.assertEquals(false, containsRandomNewTable);
+assertEquals(false, containsRandomNewTable);
Review comment:
nit: `assertFalse(containsRandomNewTable)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on a change in pull request #3347: HBASE-25955 Setting NAMESPACES when adding a replication peer doesn't…
sandeepvinayak commented on a change in pull request #3347:
URL: https://github.com/apache/hbase/pull/3347#discussion_r643611884
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java
##
@@ -396,11 +447,11 @@ public void testMetricsSourceBaseSourcePassThrough() {
// after calling #setAgeOfLastShippedOpByTable
boolean containsRandomNewTable = source.getSingleSourceSourceByTable()
.containsKey("RandomNewTable");
-Assert.assertEquals(false, containsRandomNewTable);
+assertEquals(false, containsRandomNewTable);
source.updateTableLevelMetrics(createWALEntriesWithSize("RandomNewTable"));
containsRandomNewTable = source.getSingleSourceSourceByTable()
.containsKey("RandomNewTable");
-Assert.assertEquals(true, containsRandomNewTable);
+assertEquals(true, containsRandomNewTable);
Review comment:
nit: `assertTrue(containsRandomNewTable)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on a change in pull request #3347: HBASE-25955 Setting NAMESPACES when adding a replication peer doesn't…
sandeepvinayak commented on a change in pull request #3347:
URL: https://github.com/apache/hbase/pull/3347#discussion_r643611673
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java
##
@@ -296,6 +300,53 @@ public boolean evaluate() throws Exception {
hbaseAdmin.removeReplicationPeer("testWALEntryFilterFromReplicationEndpoint");
}
+ @Test
+ public void testNamespacesMutualExclusiveScopesWALEntryFilter() throws
Exception {
+Set namespaces = new HashSet();
+namespaces.add("default");
+ReplicationPeerConfig rpc = ReplicationPeerConfig.newBuilder()
+ .setClusterKey(ZKConfig.getZooKeeperClusterKey(CONF1))
+
.setReplicationEndpointImpl(SelfWrappedReplicationEndpointForTest.class.getName())
+ .setReplicateAllUserTables(false)
+ // sets namespaces
+ .setNamespaces(namespaces).build();
+
hbaseAdmin.addReplicationPeer("testNamespacesMutualExclusiveScopesWALEntryFilter",
rpc);
Review comment:
mind making `testNamespacesMutualExclusiveScopesWALEntryFilter` a
constant string in the test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on a change in pull request #3347: HBASE-25955 Setting NAMESPACES when adding a replication peer doesn't…
sandeepvinayak commented on a change in pull request #3347:
URL: https://github.com/apache/hbase/pull/3347#discussion_r643610707
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/BaseReplicationEndpoint.java
##
@@ -103,6 +103,12 @@ protected WALEntryFilter getScopeWALEntryFilter() {
/** Returns a WALEntryFilter for checking replication per table and CF.
Subclasses can
* return null if they don't want this filter */
protected WALEntryFilter getNamespaceTableCfWALEntryFilter() {
+//If none of the below sets are defined, there's no reason to create this
filter
+if(ctx.getPeerConfig().getNamespaces()==null &&
ctx.getPeerConfig().getTableCFsMap()==null
Review comment:
Should we handle the empty values as well?
`Strings.isNullOrEmpty(strVar)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3188: HBASE-25799 add clusterReadRequests and clusterWriteRequests jmx
Apache-HBase commented on pull request #3188: URL: https://github.com/apache/hbase/pull/3188#issuecomment-852663613 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25932) TestWALEntryStream#testCleanClosedWALs test is failing.
[
https://issues.apache.org/jira/browse/HBASE-25932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355449#comment-17355449
]
Hudson commented on HBASE-25932:
Results for branch branch-2.4
[build #132 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/132/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/132/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/132/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/132/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/132/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> TestWALEntryStream#testCleanClosedWALs test is failing.
> ---
>
> Key: HBASE-25932
> URL: https://issues.apache.org/jira/browse/HBASE-25932
> Project: HBase
> Issue Type: Bug
> Components: metrics, Replication, wal
>Affects Versions: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4
>Reporter: Rushabh Shah
>Assignee: Bharath Vissapragada
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4
>
> Attachments: HBASE-25932-test-approach.patch
>
>
> We are seeing the following test failure.
> TestWALEntryStream#testCleanClosedWALs
> This test was added in HBASE-25924. I don't think the test failure has
> anything to do with the patch in HBASE-25924.
> Before HBASE-25924, we were *not* monitoring _uncleanlyClosedWAL_ metric. In
> all the branches, we were not parsing the wal trailer when we close the wal
> reader inside ReplicationSourceWALReader thread. The root cause was when we
> add active WAL to ReplicationSourceWALReader, we cache the file size when the
> wal was being actively written and once the wal was closed and replicated and
> removed from WALEntryStream, we did reset the ProtobufLogReader object but we
> didn't update the length of the wal and that was causing EOF errors since it
> can't find the WALTrailer with the stale wal file length.
> The fix applied nicely to branch-1 since we use FSHlog implementation which
> closes the WAL file sychronously.
> But in branch-2 and master, we use _AsyncFSWAL_ implementation and the
> closing of wal file is done asynchronously (as the name suggests). This is
> causing the test to fail. Below is the test.
> {code:java}
> @Test
> public void testCleanClosedWALs() throws Exception {
> try (WALEntryStream entryStream = new WALEntryStream(
> logQueue, CONF, 0, log, null, logQueue.getMetrics(), fakeWalGroupId)) {
> assertEquals(0, logQueue.getMetrics().getUncleanlyClosedWALs());
> appendToLogAndSync();
> assertNotNull(entryStream.next());
> log.rollWriter(); ===> This does an asynchronous close of wal.
> appendToLogAndSync();
> assertNotNull(entryStream.next());
> assertEquals(0, logQueue.getMetrics().getUncleanlyClosedWALs());
> }
> }
> {code}
> In the above code, when we roll writer, we don't close the old wal file
> immediately so the ReplicationReader thread is not able to get the updated
> wal file size and that is throwing EOF errors.
> If I add a sleep of few milliseconds (1 ms in my local env) between
> rollWriter and appendToLogAndSync statement then the test passes but this is
> *not* a proper fix since we are working around the race between
> ReplicationSourceWALReaderThread and closing of WAL file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25959) Relocate libnetty_transport_native_epoll_aarch_64.so in hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-25959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355447#comment-17355447 ] Michael Stack commented on HBASE-25959: --- Thanks for fixing version [~zhangduo] > Relocate libnetty_transport_native_epoll_aarch_64.so in hbase-thirdparty > > > Key: HBASE-25959 > URL: https://issues.apache.org/jira/browse/HBASE-25959 > Project: HBase > Issue Type: Bug > Components: hbase-thirdparty >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Fix For: thirdparty-3.5.1 > > Attachments: > 0001-HBASE-25959-Relocate-libnetty_transport_native_epoll.patch > > > Minor item I came across while trying to figure where all the netty > native_epoll .so instances are coming from when I look at an hbase release. > We've relocated the x86 lib but not the aarch_64... Minor item. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-25946) [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.5.1
[ https://issues.apache.org/jira/browse/HBASE-25946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-25946: --- > [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.5.1 > > > Key: HBASE-25946 > URL: https://issues.apache.org/jira/browse/HBASE-25946 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: thirdparty-3.5.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25946) [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.5.1
[ https://issues.apache.org/jira/browse/HBASE-25946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25946. --- Resolution: Fixed > [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.5.1 > > > Key: HBASE-25946 > URL: https://issues.apache.org/jira/browse/HBASE-25946 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: thirdparty-3.5.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-7386) Investigate providing some supervisor support for znode deletion
[ https://issues.apache.org/jira/browse/HBASE-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-7386: - Fix Version/s: (was: 3.0.0-alpha-2) > Investigate providing some supervisor support for znode deletion > > > Key: HBASE-7386 > URL: https://issues.apache.org/jira/browse/HBASE-7386 > Project: HBase > Issue Type: Task > Components: master, regionserver, scripts >Reporter: Gregory Chanan >Assignee: Michael Stack >Priority: Blocker > Attachments: HBASE-7386-bin-v2.patch, HBASE-7386-bin-v3.patch, > HBASE-7386-bin.patch, HBASE-7386-conf-v2.patch, HBASE-7386-conf-v3.patch, > HBASE-7386-conf.patch, HBASE-7386-master-00.patch, > HBASE-7386-master-01.patch, HBASE-7386-src.patch, HBASE-7386-v0.patch, > supervisordconfigs-v0.patch > > > There a couple of JIRAs for deleting the znode on a process failure: > HBASE-5844 (RS) > HBASE-5926 (Master) > which are pretty neat; on process failure, they delete the znode of the > underlying process so HBase can recover faster. > These JIRAs were implemented via the startup scripts; i.e. the script hangs > around and waits for the process to exit, then deletes the znode. > There are a few problems associated with this approach, as listed in the > below JIRAs: > 1) Hides startup output in script > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 2) two hbase processes listed per launched daemon > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 3) Not run by a real supervisor > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 4) Weird output after kill -9 actual process in standalone mode > https://issues.apache.org/jira/browse/HBASE-5926?focusedCommentId=13506801=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506801 > 5) Can kill existing RS if called again > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 6) Hides stdout/stderr[6] > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13506832=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506832 > I suspect running in via something like supervisor.d can solve these issues > if we provide the right support. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25959) Relocate libnetty_transport_native_epoll_aarch_64.so in hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-25959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-25959: -- Fix Version/s: (was: hbase-thirdparty-3.5.1) thirdparty-3.5.1 > Relocate libnetty_transport_native_epoll_aarch_64.so in hbase-thirdparty > > > Key: HBASE-25959 > URL: https://issues.apache.org/jira/browse/HBASE-25959 > Project: HBase > Issue Type: Bug > Components: hbase-thirdparty >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Fix For: thirdparty-3.5.1 > > Attachments: > 0001-HBASE-25959-Relocate-libnetty_transport_native_epoll.patch > > > Minor item I came across while trying to figure where all the netty > native_epoll .so instances are coming from when I look at an hbase release. > We've relocated the x86 lib but not the aarch_64... Minor item. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25855) [hbase-thirdparty] Fix typo in jersey relocation path
[ https://issues.apache.org/jira/browse/HBASE-25855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-25855: -- Fix Version/s: (was: thirdparty-4.0.0) thirdparty-3.5.1 Hadoop Flags: Incompatible change,Reviewed Release Note: We have a typo in our jersey relocation path, where we use 'jersery' instead of 'jersey'. This is a minor change, but indeed an incompatible change. But since hbase-thirdparty is only used by hbase, it is not a big problem for end users. We will update the thridparty dependency for all active branches. > [hbase-thirdparty] Fix typo in jersey relocation path > - > > Key: HBASE-25855 > URL: https://issues.apache.org/jira/browse/HBASE-25855 > Project: HBase > Issue Type: Task > Components: hbase-thirdparty >Affects Versions: thirdparty-3.4.1 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: thirdparty-3.5.1 > > > We shade to "jersery" instead of "jersey". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase-thirdparty] Apache9 commented on pull request #54: HBASE-25946 [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.m…
Apache9 commented on pull request #54: URL: https://github.com/apache/hbase-thirdparty/pull/54#issuecomment-852641580 I think it is a bug. And since thirdparty is only used by HBase itself, I think we should include this in 3.5.1 release and it breaks something, we need to fix it. Let me change the fix version for this issue and regenerate the CHANGES and RELEASENOTES. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-19701) Close without justification following succesful open
[
https://issues.apache.org/jira/browse/HBASE-19701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19701:
--
Fix Version/s: (was: 3.0.0-alpha-2)
> Close without justification following succesful open
>
>
> Key: HBASE-19701
> URL: https://issues.apache.org/jira/browse/HBASE-19701
> Project: HBase
> Issue Type: Bug
>Reporter: Michael Stack
>Priority: Critical
>
> [~jmspaggi] conjured an interesting condition where we close a region soon
> after open WITHOUT seemingly saying why (It looks like Master is asking for
> region CLOSE but that is not clear looking at RegionServer log).
> Here is log snippet from https://pastebin.com/0r76Y6ap (in case the pastebin
> evaporates)
> {code}
>
> 2017-12-31 09:54:20,864 INFO
> [PostOpenDeployTasks:f49f3cbb7f3db4cf96c7eb3b0cf83869]
> regionserver.HRegionServer: Post open deploy tasks for
> TestTable,0408944640,1505391191559.f49f3cbb7f3db4cf96c7eb3b0cf83869.
> 2017-12-31 09:54:20,870 INFO
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1]
> regionserver.CompactingMemStore: Setting in-memory flush size threshold to
> 13421772 and immutable segments index to be of type CHUNK_MAP
> 2017-12-31 09:54:20,870 INFO
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1] regionserver.HStore:
> Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore
> 2017-12-31 09:54:20,870 INFO
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1] hfile.CacheConfig: Created
> cacheConfig for info: blockCache=LruBlockCache{blockCount=0,
> currentSize=2454760, freeSize=3347745560, maxSize=3350200320,
> heapSize=2454760, minSize=3182690304, minFactor=0.95, multiSize=1591345152,
> multiFactor=0.5, singleSize=795672576, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2017-12-31 09:54:20,872 INFO
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1]
> compactions.CompactionConfiguration: size [134217728, 9223372036854775807,
> 9223372036854775807); files [3, 10); ratio 1,20; off-peak ratio 5,00;
> throttle point 2684354560; major period 60480, major jitter 0,50, min
> locality to compact 0,00; tiered compaction: max_age 9223372036854775807,
> incoming window min 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2017-12-31 09:54:20,903 INFO
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1]
> regionserver.CompactingMemStore: Setting in-memory flush size threshold to
> 13421772 and immutable segments index to be of type CHUNK_MAP
> 2017-12-31 09:54:20,904 INFO
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1] regionserver.HStore:
> Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore
> 2017-12-31 09:54:20,904 INFO
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1] hfile.CacheConfig: Created
> cacheConfig for info: blockCache=LruBlockCache{blockCount=0,
> currentSize=2454760, freeSize=3347745560, maxSize=3350200320,
> heapSize=2454760, minSize=3182690304, minFactor=0.95, multiSize=1591345152,
> multiFactor=0.5, singleSize=795672576, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2017-12-31 09:54:20,905 INFO
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1]
> compactions.CompactionConfiguration: size [134217728, 9223372036854775807,
> 9223372036854775807); files [3, 10); ratio 1,20; off-peak ratio 5,00;
> throttle point 2684354560; major period 60480, major jitter 0,50, min
> locality to compact 0,00; tiered compaction: max_age 9223372036854775807,
> incoming window min 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2017-12-31 09:54:20,929 INFO [RS_OPEN_REGION-node1:16020-1]
> regionserver.HRegion: Setting FlushNonSloppyStoresFirstPolicy for the
> region=TestTable,0262144000,1505391191559.166b9c45d7724f72fd126adb4445d6ec.
> 2017-12-31 09:54:20,956 INFO [RS_OPEN_REGION-node1:16020-0]
> regionserver.HRegion: Setting FlushNonSloppyStoresFirstPolicy for the
> region=TestTable,0188743680,1505391191559.330f09f4a0eaf26811c320fbf1b14e70.
> 2017-12-31 09:54:20,991 INFO
[GitHub] [hbase-thirdparty] Apache9 commented on pull request #54: HBASE-25946 [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.m…
Apache9 commented on pull request #54: URL: https://github.com/apache/hbase-thirdparty/pull/54#issuecomment-852640453 > The 3.5.1 RC is aimed at releasing stuff currently in the master branch? That branch includes HBASE-25855 which I don't see in the release notes; I thought it was a breaking change? Let me check. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3276: HBASE-25894 Improve the performance for region load and region count related cost functions
Apache9 commented on a change in pull request #3276:
URL: https://github.com/apache/hbase/pull/3276#discussion_r643588134
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/DoubleArrayCost.java
##
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import java.util.function.Consumer;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * A helper class to compute a scaled cost using
+ * {@link
org.apache.commons.math3.stat.descriptive.DescriptiveStatistics#DescriptiveStatistics()}.
+ * It assumes that this is a zero sum set of costs. It assumes that the worst
case possible is all
+ * of the elements in one region server and the rest having 0.
+ */
+@InterfaceAudience.Private
+final class DoubleArrayCost {
+
+ private double[] costs;
+
+ // computeCost call is expensive so we use this flag to indicate whether we
need to recalculate
+ // the cost by calling computeCost
+ private boolean costsChanged;
+
+ private double cost;
+
+ void prepare(int length) {
+if (costs == null || costs.length != length) {
+ costs = new double[length];
+}
+ }
+
+ void setCosts(Consumer consumer) {
Review comment:
Do you have any suggestion on the naming? fillCosts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (HBASE-25959) Relocate libnetty_transport_native_epoll_aarch_64.so in hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-25959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-25959. --- Fix Version/s: hbase-thirdparty-3.5.1 Assignee: Michael Stack Resolution: Fixed Pushed the one-liner. > Relocate libnetty_transport_native_epoll_aarch_64.so in hbase-thirdparty > > > Key: HBASE-25959 > URL: https://issues.apache.org/jira/browse/HBASE-25959 > Project: HBase > Issue Type: Bug > Components: hbase-thirdparty >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Fix For: hbase-thirdparty-3.5.1 > > Attachments: > 0001-HBASE-25959-Relocate-libnetty_transport_native_epoll.patch > > > Minor item I came across while trying to figure where all the netty > native_epoll .so instances are coming from when I look at an hbase release. > We've relocated the x86 lib but not the aarch_64... Minor item. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25959) Relocate libnetty_transport_native_epoll_aarch_64.so in hbase-thirdparty
Michael Stack created HBASE-25959: - Summary: Relocate libnetty_transport_native_epoll_aarch_64.so in hbase-thirdparty Key: HBASE-25959 URL: https://issues.apache.org/jira/browse/HBASE-25959 Project: HBase Issue Type: Bug Components: hbase-thirdparty Reporter: Michael Stack Minor item I came across while trying to figure where all the netty native_epoll .so instances are coming from when I look at an hbase release. We've relocated the x86 lib but not the aarch_64... Minor item. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25959) Relocate libnetty_transport_native_epoll_aarch_64.so in hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-25959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-25959: -- Attachment: 0001-HBASE-25959-Relocate-libnetty_transport_native_epoll.patch > Relocate libnetty_transport_native_epoll_aarch_64.so in hbase-thirdparty > > > Key: HBASE-25959 > URL: https://issues.apache.org/jira/browse/HBASE-25959 > Project: HBase > Issue Type: Bug > Components: hbase-thirdparty >Reporter: Michael Stack >Priority: Minor > Attachments: > 0001-HBASE-25959-Relocate-libnetty_transport_native_epoll.patch > > > Minor item I came across while trying to figure where all the netty > native_epoll .so instances are coming from when I look at an hbase release. > We've relocated the x86 lib but not the aarch_64... Minor item. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25911) Fix uses of System.currentTimeMillis (should be EnvironmentEdgeManager.currentTime)
[ https://issues.apache.org/jira/browse/HBASE-25911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25911: Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > Fix uses of System.currentTimeMillis (should be > EnvironmentEdgeManager.currentTime) > --- > > Key: HBASE-25911 > URL: https://issues.apache.org/jira/browse/HBASE-25911 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.5.0 > > > We introduced EnvironmentEdgeManager a long time ago as a way to inject > alternate clocks (gettimeofday() aka System.currentTimeMillis()) for unit > tests. In order for this to be effective, all callers that would otherwise > use System.currentTimeMillis() must call EnvironmentEdgeManager.currentTime() > instead, except obviously the implementors of EnvironmentEdge. > It's common for contributors to be unaware of this practice and reviewers > might not catch it. > It will be much more important to have EnvironmentEdgeManager in use where > expected once we have EnvironmentEdge also providing a monotonic clock > source. (See parent.) > On another subtask I will introduce a build enforcer that bans > System.currentTimeMillis() except where annotated to allow it. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25896) Implement a Region Visualization on Master WebUI
[
https://issues.apache.org/jira/browse/HBASE-25896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355364#comment-17355364
]
Nick Dimiduk commented on HBASE-25896:
--
bq. For adding minified javascript to the repo I would be extremely surprised
if anything automated caught it. I think that section of the repo gets excluded
from the clumsy tooling we have.
The minified JS files do not include any license header comments. All three
vega-* dependencies are on GitHub as {{BSD 3-Clause "New" or "Revised"
License}}, i.e., https://github.com/vega/vega/blob/v5.19.1/LICENSE. I add this
license to our NOTICE.txt? I don't see anything specific for jquery other than
excludes in the rat plugin configuration.
> Implement a Region Visualization on Master WebUI
>
>
> Key: HBASE-25896
> URL: https://issues.apache.org/jira/browse/HBASE-25896
> Project: HBase
> Issue Type: Sub-task
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
>
> Demonstrate end-to-end delivery of an interesting visualization.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25792) Filter out o.a.hadoop.thirdparty building shaded jars
[
https://issues.apache.org/jira/browse/HBASE-25792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355358#comment-17355358
]
Michael Stack commented on HBASE-25792:
---
[~ndimiduk]
bq. Accidental push to branch-2.3 ?
Yes. Reverting. Thanks.
> Filter out o.a.hadoop.thirdparty building shaded jars
> -
>
> Key: HBASE-25792
> URL: https://issues.apache.org/jira/browse/HBASE-25792
> Project: HBase
> Issue Type: Bug
> Components: shading
>Affects Versions: 3.0.0-alpha-1, 2.5.0, 2.4.3
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Hadoop 3.3.1 (unreleased currently) shades guava. The shaded guava then trips
> the check in our shading that tries to exclude hadoop bits from the fat jars
> we build.
> For the issue to trigger, need to build against tip of hadoop branch-3.3. You
> then get this complaint:
> {code}
> [INFO] --- exec-maven-plugin:1.6.0:exec (check-jar-contents) @
> hbase-shaded-check-invariants ---
> [ERROR] Found artifact with unexpected contents:
> '/Users/stack/.m2/repository/org/apache/hbase/hbase-shaded-mapreduce/2.3.6-SNAPSHOT/hbase-shaded-mapreduce-2.3.6-SNAPSHOT.jar'
> Please check the following and either correct the build or update
> the allowed list with reasoning.
> org/apache/hadoop/thirdparty/
> org/apache/hadoop/thirdparty/com/
> org/apache/hadoop/thirdparty/com/google/
> org/apache/hadoop/thirdparty/com/google/common/
> org/apache/hadoop/thirdparty/com/google/common/annotations/
> org/apache/hadoop/thirdparty/com/google/common/annotations/Beta.class
>
> org/apache/hadoop/thirdparty/com/google/common/annotations/GwtCompatible.class
>
> org/apache/hadoop/thirdparty/com/google/common/annotations/GwtIncompatible.class
>
> org/apache/hadoop/thirdparty/com/google/common/annotations/VisibleForTesting.class
> org/apache/hadoop/thirdparty/com/google/common/base/
> org/apache/hadoop/thirdparty/com/google/common/base/Absent.class
>
> org/apache/hadoop/thirdparty/com/google/common/base/AbstractIterator$1.class
>
> org/apache/hadoop/thirdparty/com/google/common/base/AbstractIterator$State.class
> org/apache/hadoop/thirdparty/com/google/common/base/AbstractIterator.class
> org/apache/hadoop/thirdparty/com/google/common/base/Ascii.class
> org/apache/hadoop/thirdparty/com/google/common/base/CaseFormat$1.class
> org/apache/hadoop/thirdparty/com/google/common/base/CaseFormat$2.class
> org/apache/hadoop/thirdparty/com/google/common/base/CaseFormat$3.class
> org/apache/hadoop/thirdparty/com/google/common/base/CaseFormat$4.class
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25941) TestRESTServerSSL fails because of jdk bug
[ https://issues.apache.org/jira/browse/HBASE-25941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-25941: - Fix Version/s: 2.5.0 > TestRESTServerSSL fails because of jdk bug > -- > > Key: HBASE-25941 > URL: https://issues.apache.org/jira/browse/HBASE-25941 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Michael Stack >Assignee: Wei-Chiu Chuang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4 > > > [~weijing329] identified issue in TestRESTServerSSL when using jdk8 292+. It > came up in comment in the parent issue. I verified it fails for me using jdk8 > v292. Here is the failure > ```[INFO] Running org.apache.hadoop.hbase.rest.TestRESTServerSSL > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.4 s > <<< FAILURE! - in org.apache.hadoop.hbase.rest.TestRESTServerSSL > [ERROR] org.apache.hadoop.hbase.rest.TestRESTServerSSL Time elapsed: 1.387 s > <<< ERROR! > java.security.NoSuchAlgorithmException: unrecognized algorithm name: > PBEWithSHA1AndDESede > at > org.apache.hadoop.hbase.rest.TestRESTServerSSL.beforeClass(TestRESTServerSSL.java:74)``` > For workaround, see https://github.com/bcgit/bc-java/issues/941 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25792) Filter out o.a.hadoop.thirdparty building shaded jars
[
https://issues.apache.org/jira/browse/HBASE-25792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355354#comment-17355354
]
Nick Dimiduk commented on HBASE-25792:
--
[~stack] Accidental push to branch-2.3 ?
> Filter out o.a.hadoop.thirdparty building shaded jars
> -
>
> Key: HBASE-25792
> URL: https://issues.apache.org/jira/browse/HBASE-25792
> Project: HBase
> Issue Type: Bug
> Components: shading
>Affects Versions: 3.0.0-alpha-1, 2.5.0, 2.4.3
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Hadoop 3.3.1 (unreleased currently) shades guava. The shaded guava then trips
> the check in our shading that tries to exclude hadoop bits from the fat jars
> we build.
> For the issue to trigger, need to build against tip of hadoop branch-3.3. You
> then get this complaint:
> {code}
> [INFO] --- exec-maven-plugin:1.6.0:exec (check-jar-contents) @
> hbase-shaded-check-invariants ---
> [ERROR] Found artifact with unexpected contents:
> '/Users/stack/.m2/repository/org/apache/hbase/hbase-shaded-mapreduce/2.3.6-SNAPSHOT/hbase-shaded-mapreduce-2.3.6-SNAPSHOT.jar'
> Please check the following and either correct the build or update
> the allowed list with reasoning.
> org/apache/hadoop/thirdparty/
> org/apache/hadoop/thirdparty/com/
> org/apache/hadoop/thirdparty/com/google/
> org/apache/hadoop/thirdparty/com/google/common/
> org/apache/hadoop/thirdparty/com/google/common/annotations/
> org/apache/hadoop/thirdparty/com/google/common/annotations/Beta.class
>
> org/apache/hadoop/thirdparty/com/google/common/annotations/GwtCompatible.class
>
> org/apache/hadoop/thirdparty/com/google/common/annotations/GwtIncompatible.class
>
> org/apache/hadoop/thirdparty/com/google/common/annotations/VisibleForTesting.class
> org/apache/hadoop/thirdparty/com/google/common/base/
> org/apache/hadoop/thirdparty/com/google/common/base/Absent.class
>
> org/apache/hadoop/thirdparty/com/google/common/base/AbstractIterator$1.class
>
> org/apache/hadoop/thirdparty/com/google/common/base/AbstractIterator$State.class
> org/apache/hadoop/thirdparty/com/google/common/base/AbstractIterator.class
> org/apache/hadoop/thirdparty/com/google/common/base/Ascii.class
> org/apache/hadoop/thirdparty/com/google/common/base/CaseFormat$1.class
> org/apache/hadoop/thirdparty/com/google/common/base/CaseFormat$2.class
> org/apache/hadoop/thirdparty/com/google/common/base/CaseFormat$3.class
> org/apache/hadoop/thirdparty/com/google/common/base/CaseFormat$4.class
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3336: HBASE-25920 Support Hadoop 3.3.1
Apache-HBase commented on pull request #3336: URL: https://github.com/apache/hbase/pull/3336#issuecomment-852501705 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 37s | master passed | | +1 :green_heart: | compile | 2m 35s | master passed | | +1 :green_heart: | shadedjars | 8m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 25s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 42s | the patch passed | | +1 :green_heart: | compile | 2m 41s | the patch passed | | +1 :green_heart: | javac | 2m 41s | the patch passed | | +1 :green_heart: | shadedjars | 8m 55s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 26s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 324m 5s | root in the patch failed. | | | | 365m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3336 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux aec9f86c4c8a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4fb0861214 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/6/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/6/testReport/ | | Max. process+thread count | 3651 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/6/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25896) Implement a Region Visualization on Master WebUI
[
https://issues.apache.org/jira/browse/HBASE-25896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355346#comment-17355346
]
Nick Dimiduk commented on HBASE-25896:
--
rat check from my branch doesn't make mention of any of my JS additions,
{noformat}
Files with unapproved licenses:
hbase-protocol/target/maven-archiver/pom.properties
hbase-hadoop2-compat/target/test-classes/mrapp-generated-classpath
hbase-hadoop2-compat/target/maven-archiver/pom.properties
server_metrics.json
hbase-prefix-tree/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/inputFiles.lst
hbase-prefix-tree/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/createdFiles.lst
hbase-prefix-tree/target/maven-status/maven-compiler-plugin/compile/default-compile/inputFiles.lst
hbase-prefix-tree/target/maven-status/maven-compiler-plugin/compile/default-compile/createdFiles.lst
hbase-prefix-tree/target/maven-archiver/pom.properties
hbase-rsgroup/target/protoc-dependencies/b47b1ebc2b25ee58f549160423c8fb61/google/protobuf/timestamp.proto
hbase-rsgroup/target/protoc-dependencies/b47b1ebc2b25ee58f549160423c8fb61/google/protobuf/field_mask.proto
hbase-rsgroup/target/protoc-dependencies/b47b1ebc2b25ee58f549160423c8fb61/google/protobuf/api.proto
hbase-rsgroup/target/protoc-dependencies/b47b1ebc2b25ee58f549160423c8fb61/google/protobuf/duration.proto
hbase-rsgroup/target/protoc-dependencies/b47b1ebc2b25ee58f549160423c8fb61/google/protobuf/struct.proto
hbase-rsgroup/target/protoc-dependencies/b47b1ebc2b25ee58f549160423c8fb61/google/protobuf/wrappers.proto
hbase-rsgroup/target/protoc-dependencies/b47b1ebc2b25ee58f549160423c8fb61/google/protobuf/source_context.proto
hbase-rsgroup/target/protoc-dependencies/b47b1ebc2b25ee58f549160423c8fb61/google/protobuf/any.proto
hbase-rsgroup/target/protoc-dependencies/b47b1ebc2b25ee58f549160423c8fb61/google/protobuf/type.proto
hbase-rsgroup/target/protoc-dependencies/b47b1ebc2b25ee58f549160423c8fb61/google/protobuf/empty.proto
hbase-rsgroup/target/protoc-dependencies/b47b1ebc2b25ee58f549160423c8fb61/google/protobuf/compiler/plugin.proto
hbase-rsgroup/target/protoc-dependencies/b47b1ebc2b25ee58f549160423c8fb61/google/protobuf/descriptor.proto
hbase-rsgroup/target/maven-archiver/pom.properties
{noformat}
> Implement a Region Visualization on Master WebUI
>
>
> Key: HBASE-25896
> URL: https://issues.apache.org/jira/browse/HBASE-25896
> Project: HBase
> Issue Type: Sub-task
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
>
> Demonstrate end-to-end delivery of an interesting visualization.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3302: HBASE-25911 Replace calls to System.currentTimeMillis with EnvironmentEdgeManager.currentTime
Apache-HBase commented on pull request #3302: URL: https://github.com/apache/hbase/pull/3302#issuecomment-852462312 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 27s | master passed | | +1 :green_heart: | compile | 7m 59s | master passed | | +1 :green_heart: | shadedjars | 11m 1s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 48s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 48s | the patch passed | | +1 :green_heart: | compile | 8m 3s | the patch passed | | +1 :green_heart: | javac | 8m 3s | the patch passed | | +1 :green_heart: | shadedjars | 11m 6s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 6m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 5s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 28s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 52s | hbase-zookeeper in the patch passed. | | +1 :green_heart: | unit | 7m 23s | hbase-balancer in the patch passed. | | +1 :green_heart: | unit | 0m 54s | hbase-http in the patch passed. | | +1 :green_heart: | unit | 2m 13s | hbase-procedure in the patch passed. | | +1 :green_heart: | unit | 160m 29s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 10m 30s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | unit | 7m 44s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 3m 4s | hbase-endpoint in the patch passed. | | +1 :green_heart: | unit | 10m 10s | hbase-backup in the patch passed. | | +1 :green_heart: | unit | 1m 19s | hbase-it in the patch passed. | | +1 :green_heart: | unit | 4m 15s | hbase-rest in the patch passed. | | +1 :green_heart: | unit | 2m 10s | hbase-examples in the patch passed. | | +1 :green_heart: | unit | 0m 52s | hbase-hbtop in the patch passed. | | | | 281m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3302 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 00bacfb605a0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / e9f595307a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/5/testReport/ | | Max. process+thread count | 5132 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-zookeeper hbase-balancer hbase-http hbase-procedure hbase-server hbase-mapreduce hbase-thrift hbase-endpoint hbase-backup hbase-it hbase-rest hbase-examples hbase-hbtop U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3325: HBASE-25934 Add username for RegionScannerHolder
Apache-HBase commented on pull request #3325: URL: https://github.com/apache/hbase/pull/3325#issuecomment-852460543 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 18s | master passed | | +1 :green_heart: | compile | 1m 2s | master passed | | +1 :green_heart: | shadedjars | 9m 1s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 2s | the patch passed | | +1 :green_heart: | compile | 1m 3s | the patch passed | | +1 :green_heart: | javac | 1m 3s | the patch passed | | +1 :green_heart: | shadedjars | 9m 7s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 210m 5s | hbase-server in the patch failed. | | | | 242m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3325/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3325 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5ab70eebe42a 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 335305e0cf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3325/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3325/4/testReport/ | | Max. process+thread count | 3205 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3325/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3302: HBASE-25911 Replace calls to System.currentTimeMillis with EnvironmentEdgeManager.currentTime
Apache-HBase commented on pull request #3302: URL: https://github.com/apache/hbase/pull/3302#issuecomment-852459174 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 14s | master passed | | +1 :green_heart: | compile | 8m 25s | master passed | | +1 :green_heart: | shadedjars | 9m 24s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 56s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 56s | the patch passed | | +1 :green_heart: | compile | 8m 15s | the patch passed | | +1 :green_heart: | javac | 8m 15s | the patch passed | | +1 :green_heart: | shadedjars | 9m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 56s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 58s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 30s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 48s | hbase-zookeeper in the patch passed. | | +1 :green_heart: | unit | 7m 6s | hbase-balancer in the patch passed. | | +1 :green_heart: | unit | 0m 54s | hbase-http in the patch passed. | | +1 :green_heart: | unit | 1m 56s | hbase-procedure in the patch passed. | | +1 :green_heart: | unit | 156m 50s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 12m 22s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | unit | 7m 25s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 2m 58s | hbase-endpoint in the patch passed. | | +1 :green_heart: | unit | 8m 58s | hbase-backup in the patch passed. | | +1 :green_heart: | unit | 1m 17s | hbase-it in the patch passed. | | +1 :green_heart: | unit | 3m 37s | hbase-rest in the patch passed. | | +1 :green_heart: | unit | 2m 3s | hbase-examples in the patch passed. | | +1 :green_heart: | unit | 0m 59s | hbase-hbtop in the patch passed. | | | | 275m 33s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3302 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bfe9cd1a1b5c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / e9f595307a | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/5/testReport/ | | Max. process+thread count | 4842 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-zookeeper hbase-balancer hbase-http hbase-procedure hbase-server hbase-mapreduce hbase-thrift hbase-endpoint hbase-backup hbase-it hbase-rest hbase-examples hbase-hbtop U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3333: Backport "HBASE-25651 NORMALIZER_TARGET_REGION_SIZE needs a unit in its name (#3063)" to branch-2
Apache-HBase commented on pull request #: URL: https://github.com/apache/hbase/pull/#issuecomment-852449621 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 13s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 47s | branch-2 passed | | +1 :green_heart: | compile | 0m 54s | branch-2 passed | | +1 :green_heart: | shadedjars | 7m 32s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 29s | the patch passed | | +1 :green_heart: | compile | 0m 53s | the patch passed | | +1 :green_heart: | javac | 0m 53s | the patch passed | | +1 :green_heart: | shadedjars | 7m 31s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 53s | hbase-client in the patch passed. | | -1 :x: | unit | 8m 17s | hbase-shell in the patch failed. | | | | 41m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/ | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 90a8269d1f84 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / d4285be5c1 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-shell.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-/2/testReport/ | | Max. process+thread count | 1856 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3333: Backport "HBASE-25651 NORMALIZER_TARGET_REGION_SIZE needs a unit in its name (#3063)" to branch-2
Apache-HBase commented on pull request #: URL: https://github.com/apache/hbase/pull/#issuecomment-852446978 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 55s | branch-2 passed | | +1 :green_heart: | compile | 1m 27s | branch-2 passed | | +1 :green_heart: | checkstyle | 0m 44s | branch-2 passed | | +1 :green_heart: | spotbugs | 1m 14s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 45s | the patch passed | | +1 :green_heart: | compile | 1m 27s | the patch passed | | +1 :green_heart: | javac | 1m 27s | the patch passed | | +1 :green_heart: | checkstyle | 0m 44s | the patch passed | | -0 :warning: | rubocop | 0m 17s | The patch generated 7 new + 610 unchanged - 5 fixed = 617 total (was 615) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 56s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 1m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | | | | 37m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/ | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile rubocop | | uname | Linux a7aed9ff5e8f 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / d4285be5c1 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | rubocop | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-/2/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 86 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 rubocop=0.80.0 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3333: Backport "HBASE-25651 NORMALIZER_TARGET_REGION_SIZE needs a unit in its name (#3063)" to branch-2
Apache-HBase commented on pull request #: URL: https://github.com/apache/hbase/pull/#issuecomment-852445001 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 33s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 29s | branch-2 passed | | +1 :green_heart: | compile | 0m 51s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 0s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 23s | the patch passed | | +1 :green_heart: | compile | 0m 50s | the patch passed | | +1 :green_heart: | javac | 0m 50s | the patch passed | | +1 :green_heart: | shadedjars | 6m 1s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 38s | hbase-client in the patch passed. | | -1 :x: | unit | 6m 56s | hbase-shell in the patch failed. | | | | 34m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/ | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0b084f366f75 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / d4285be5c1 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-/2/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-shell.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-/2/testReport/ | | Max. process+thread count | 2416 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25896) Implement a Region Visualization on Master WebUI
[ https://issues.apache.org/jira/browse/HBASE-25896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355333#comment-17355333 ] Sean Busbey commented on HBASE-25896: - A quick look at our rat-plugin settings suggests that the tooling will at least catch if the javascript doesn't include some kind of license text the tool recognizes. That won't cover maintaining our LICENSE/NOTICE files, but it would at least flag if some incompatible license header was present in the javascript itself. > Implement a Region Visualization on Master WebUI > > > Key: HBASE-25896 > URL: https://issues.apache.org/jira/browse/HBASE-25896 > Project: HBase > Issue Type: Sub-task >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > > Demonstrate end-to-end delivery of an interesting visualization. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25896) Implement a Region Visualization on Master WebUI
[ https://issues.apache.org/jira/browse/HBASE-25896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355332#comment-17355332 ] Sean Busbey commented on HBASE-25896: - For adding minified javascript to the repo I would be extremely surprised if anything automated caught it. I think that section of the repo gets excluded from the clumsy tooling we have. > Implement a Region Visualization on Master WebUI > > > Key: HBASE-25896 > URL: https://issues.apache.org/jira/browse/HBASE-25896 > Project: HBase > Issue Type: Sub-task >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > > Demonstrate end-to-end delivery of an interesting visualization. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25958) Implement a Region Size histogram
Nick Dimiduk created HBASE-25958: Summary: Implement a Region Size histogram Key: HBASE-25958 URL: https://issues.apache.org/jira/browse/HBASE-25958 Project: HBase Issue Type: Sub-task Components: master, UI Reporter: Nick Dimiduk Assignee: Nick Dimiduk Add a new view that displays a histogram of region size. Make it so that the visualization can by selected, rather than showing multiple charts at once. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25896) Implement a Region Visualization on Master WebUI
[ https://issues.apache.org/jira/browse/HBASE-25896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355331#comment-17355331 ] Nick Dimiduk commented on HBASE-25896: -- bq. Please make sure that prior to merge you... Ah thank you for the guidance. Should I expect some build process to fail if I get this wrong? > Implement a Region Visualization on Master WebUI > > > Key: HBASE-25896 > URL: https://issues.apache.org/jira/browse/HBASE-25896 > Project: HBase > Issue Type: Sub-task >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > > Demonstrate end-to-end delivery of an interesting visualization. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25895) Implement a Cluster Metrics JSON endpoint
[
https://issues.apache.org/jira/browse/HBASE-25895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355330#comment-17355330
]
Nick Dimiduk commented on HBASE-25895:
--
bq. How does this API definition square with our downstream facing
compatibility and versioning promises?
That depends on how we use this. For now, I'd like these endpoints to be
considered as internal/private, so that we can experiment and develop with
them. Eventually, one day, if this method of UI development catches on, it
would be nice to reconcile these APIs with those exposed in the REST Gateway.
bq. If that’s the case we should note it somewhere more prominent because I
don’t think downstream users are likely to notice.
Do you have a proposal here? There isn't a way to broadcast intended audience
from a webservice... unless we do something terrible. Is a release note on the
JIRA good enough?
bq. Pardon, my infoserver knowledge is a bit rusty.
I have no yet performed an audit of what information is exposed through this
interface, vs. what is already available on webUIs, vs. what is already
available to an authenticated user via the usual client API. What's exposed
here is a subset of what's available via client APIs, and I haven't noticed any
restrictions on {{Admin}} beyond authentication.
bq. ... but “unprivileged” means that they won’t need to be in any particular
access group. Is that correct?
That is consistent with my understanding. TBH, our security filters are pretty
clumsy compared to what Jersey offers. For now, I'm implementing within the
existing security structures we have in place. If Jersey turns out to work well
for us, I'd like to make an effort to overhaul the access filters so that they
work more in line with the Jersey model.
>From the javadoc strings on these methods,
{noformat}
/**
* Adds a servlet in the server that any user can access. This method differs
from
* {@link #addPrivilegedServlet(String, ServletHolder)} in that any
authenticated user
* can interact with the servlet added by this method.
* @param pathSpec The path spec for the servlet
* @param holder The servlet holder
*/
public void addUnprivilegedServlet(String pathSpec, ServletHolder holder) {
{noformat}
{noformat}
/**
* Adds a servlet in the server that only administrators can access. This
method differs from
* {@link #addUnprivilegedServlet(String, ServletHolder)} in that only those
* authenticated user who are identified as administrators can interact with
the servlet added by
* this method.
*/
public void addPrivilegedServlet(String pathSpec, ServletHolder holder) {
{noformat}
bq. We should note this expectation on the package-info for the cluster_metrics
package and the class javadocs for the ClusterMetrics api implementation.
Sure, I can add warning notes for developers in more placed in the new code.
bq. If there is some way we can proactively prevent the addition of REST verbs
that change state that would also make me feel more at ease with the permissive
access.
I'm not aware of anything like this. It might be possible to craft an enforcer
rule that fails in the presence of PUT/POST/DELETE annotation verbs. This would
be much more strict that anything we have today pertaining to our existing
servlets. But still, this wouldn't block changing state per se, only block
accepting these types of HTTP requests. Nothing stops a dev from implementing
something that changes state via GET query parameters, for example.
> Implement a Cluster Metrics JSON endpoint
> -
>
> Key: HBASE-25895
> URL: https://issues.apache.org/jira/browse/HBASE-25895
> Project: HBase
> Issue Type: Sub-task
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
>
> In order to build a dynamic visualization functionality, we need access to
> the underlying dataset. I think we can do a lot by exposing the
> ClusterMetrics already available via the {{o.a.h.h.client.Admin}} interface.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] wchevreuil commented on pull request #3347: HBASE-25955 Setting NAMESPACES when adding a replication peer doesn't…
wchevreuil commented on pull request #3347: URL: https://github.com/apache/hbase/pull/3347#issuecomment-852424919 Looking this UT failure on this TestRemoveFromSerialReplicationPeer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #3339: HBASE-25947 Backport 'HBASE-25894 Improve the performance for region …
ndimiduk commented on pull request #3339: URL: https://github.com/apache/hbase/pull/3339#issuecomment-852416465 @Apache9 Good by me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #3276: HBASE-25894 Improve the performance for region load and region count related cost functions
ndimiduk commented on a change in pull request #3276:
URL: https://github.com/apache/hbase/pull/3276#discussion_r643448466
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/DoubleArrayCost.java
##
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import java.util.function.Consumer;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * A helper class to compute a scaled cost using
+ * {@link
org.apache.commons.math3.stat.descriptive.DescriptiveStatistics#DescriptiveStatistics()}.
+ * It assumes that this is a zero sum set of costs. It assumes that the worst
case possible is all
+ * of the elements in one region server and the rest having 0.
+ */
+@InterfaceAudience.Private
+final class DoubleArrayCost {
+
+ private double[] costs;
+
+ // computeCost call is expensive so we use this flag to indicate whether we
need to recalculate
+ // the cost by calling computeCost
+ private boolean costsChanged;
+
+ private double cost;
+
+ void prepare(int length) {
+if (costs == null || costs.length != length) {
+ costs = new double[length];
+}
+ }
+
+ void setCosts(Consumer consumer) {
Review comment:
nit: having a `setFoo` method that is not a simple POJO field assignment
is surprising!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3336: HBASE-25920 Support Hadoop 3.3.1
Apache-HBase commented on pull request #3336: URL: https://github.com/apache/hbase/pull/3336#issuecomment-852409444 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 5s | master passed | | +1 :green_heart: | compile | 3m 0s | master passed | | +1 :green_heart: | shadedjars | 8m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 13s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 12s | the patch passed | | +1 :green_heart: | compile | 3m 3s | the patch passed | | +1 :green_heart: | javac | 3m 3s | the patch passed | | +1 :green_heart: | shadedjars | 8m 12s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 15s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 192m 25s | root in the patch passed. | | | | 235m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3336 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux cd3322a6fb88 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4fb0861214 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/6/testReport/ | | Max. process+thread count | 6316 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/6/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-operator-tools] Apache-HBase commented on pull request #88: HBASE-25921 Fix Wrong FileSystem when running `filesystem` on non-HDFS storage
Apache-HBase commented on pull request #88: URL: https://github.com/apache/hbase-operator-tools/pull/88#issuecomment-852375874 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 44s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | spotbugs | 0m 0s | spotbugs executables are not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 49s | master passed | | +1 :green_heart: | compile | 0m 9s | master passed | | +1 :green_heart: | checkstyle | 0m 7s | master passed | | +1 :green_heart: | javadoc | 0m 7s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 11s | the patch passed | | +1 :green_heart: | compile | 0m 9s | the patch passed | | +1 :green_heart: | javac | 0m 9s | the patch passed | | +1 :green_heart: | checkstyle | 0m 4s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | javadoc | 0m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 5m 9s | hbase-hbck2 in the patch passed. | | +1 :green_heart: | asflicense | 0m 6s | The patch does not generate ASF License warnings. | | | | 7m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-88/5/artifact/yetus-precommit-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase-operator-tools/pull/88 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile | | uname | Linux 168872e2fccf 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 GNU/Linux | | Build tool | maven | | git revision | master / 33d4c31 | | Default Java | Oracle Corporation-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-88/5/testReport/ | | Max. process+thread count | 957 (vs. ulimit of 5000) | | modules | C: hbase-hbck2 U: hbase-hbck2 | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-Operator-Tools-PreCommit/job/PR-88/5/console | | versions | git=2.20.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-operator-tools] taklwu commented on a change in pull request #88: HBASE-25921 Fix Wrong FileSystem when running `filesystem` on non-HDFS storage
taklwu commented on a change in pull request #88:
URL:
https://github.com/apache/hbase-operator-tools/pull/88#discussion_r643404148
##
File path: hbase-hbck2/src/main/java/org/apache/hbase/HBCKFsUtils.java
##
@@ -107,6 +107,15 @@ public static Path getRootDir(final Configuration c)
throws IOException {
return p.makeQualified(fs.getUri(), fs.getWorkingDirectory());
}
+ /**
+ * @param conf must not be null
+ * @return Returns the filesystem of the hbase rootdir.
+ * @throws IOException from underlying FileSystem
+ */
+ public static FileSystem getCurrentFileSystem(Configuration conf) throws
IOException {
Review comment:
thanks @anoopsjohn , I renamed that to `getRootDirFileSystem`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3302: HBASE-25911 Replace calls to System.currentTimeMillis with EnvironmentEdgeManager.currentTime
Apache-HBase commented on pull request #3302: URL: https://github.com/apache/hbase/pull/3302#issuecomment-852358769 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 5s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 40s | master passed | | +1 :green_heart: | compile | 12m 21s | master passed | | +1 :green_heart: | checkstyle | 6m 10s | master passed | | +1 :green_heart: | spotbugs | 11m 24s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 37s | the patch passed | | +1 :green_heart: | compile | 12m 19s | the patch passed | | +1 :green_heart: | javac | 12m 19s | the patch passed | | -0 :warning: | checkstyle | 1m 35s | hbase-server: The patch generated 1 new + 1257 unchanged - 6 fixed = 1258 total (was 1263) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 2s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 14m 30s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 3m 1s | The patch does not generate ASF License warnings. | | | | 105m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3302 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 58d98f4ec49e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / e9f595307a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-zookeeper hbase-balancer hbase-http hbase-procedure hbase-server hbase-mapreduce hbase-thrift hbase-endpoint hbase-backup hbase-it hbase-rest hbase-examples hbase-hbtop U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/5/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3325: HBASE-25934 Add username for RegionScannerHolder
Apache-HBase commented on pull request #3325: URL: https://github.com/apache/hbase/pull/3325#issuecomment-852347289 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 12s | master passed | | +1 :green_heart: | compile | 3m 21s | master passed | | +1 :green_heart: | checkstyle | 1m 11s | master passed | | +1 :green_heart: | spotbugs | 2m 13s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 2s | the patch passed | | +1 :green_heart: | compile | 3m 18s | the patch passed | | +1 :green_heart: | javac | 3m 18s | the patch passed | | +1 :green_heart: | checkstyle | 1m 11s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 53s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 51m 13s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3325/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3325 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux e9014d017c28 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 335305e0cf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3325/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3347: HBASE-25955 Setting NAMESPACES when adding a replication peer doesn't…
Apache-HBase commented on pull request #3347: URL: https://github.com/apache/hbase/pull/3347#issuecomment-852330045 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-7386) Investigate providing some supervisor support for znode deletion
[ https://issues.apache.org/jira/browse/HBASE-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-7386: - Resolution: Won't Fix Status: Resolved (was: Patch Available) No progress on old feature. Resolving. Reopen if interest. > Investigate providing some supervisor support for znode deletion > > > Key: HBASE-7386 > URL: https://issues.apache.org/jira/browse/HBASE-7386 > Project: HBase > Issue Type: Task > Components: master, regionserver, scripts >Reporter: Gregory Chanan >Assignee: Michael Stack >Priority: Blocker > Fix For: 3.0.0-alpha-2 > > Attachments: HBASE-7386-bin-v2.patch, HBASE-7386-bin-v3.patch, > HBASE-7386-bin.patch, HBASE-7386-conf-v2.patch, HBASE-7386-conf-v3.patch, > HBASE-7386-conf.patch, HBASE-7386-master-00.patch, > HBASE-7386-master-01.patch, HBASE-7386-src.patch, HBASE-7386-v0.patch, > supervisordconfigs-v0.patch > > > There a couple of JIRAs for deleting the znode on a process failure: > HBASE-5844 (RS) > HBASE-5926 (Master) > which are pretty neat; on process failure, they delete the znode of the > underlying process so HBase can recover faster. > These JIRAs were implemented via the startup scripts; i.e. the script hangs > around and waits for the process to exit, then deletes the znode. > There are a few problems associated with this approach, as listed in the > below JIRAs: > 1) Hides startup output in script > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 2) two hbase processes listed per launched daemon > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 3) Not run by a real supervisor > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 4) Weird output after kill -9 actual process in standalone mode > https://issues.apache.org/jira/browse/HBASE-5926?focusedCommentId=13506801=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506801 > 5) Can kill existing RS if called again > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 6) Hides stdout/stderr[6] > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13506832=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506832 > I suspect running in via something like supervisor.d can solve these issues > if we provide the right support. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3336: HBASE-25920 Support Hadoop 3.3.1
Apache-HBase commented on pull request #3336: URL: https://github.com/apache/hbase/pull/3336#issuecomment-852292720 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 13s | master passed | | +1 :green_heart: | compile | 8m 49s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 1m 32s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 10s | the patch passed | | +1 :green_heart: | compile | 8m 47s | the patch passed | | +1 :green_heart: | javac | 8m 47s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 4s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 21m 26s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 47s | The patch does not generate ASF License warnings. | | | | 59m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3336 | | Optional Tests | dupname asflicense javac hadoopcheck xml compile | | uname | Linux 02845fec8de4 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4fb0861214 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 126 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3336/6/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell merged pull request #3302: HBASE-25911 Replace calls to System.currentTimeMillis with EnvironmentEdgeManager.currentTime
apurtell merged pull request #3302: URL: https://github.com/apache/hbase/pull/3302 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-thirdparty] busbey commented on pull request #54: HBASE-25946 [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.m…
busbey commented on pull request #54: URL: https://github.com/apache/hbase-thirdparty/pull/54#issuecomment-852286370 The 3.5.1 RC is aimed at releasing stuff currently in the master branch? That branch includes HBASE-25855 which I don't see in the release notes; I thought it was a breaking change? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25924) Seeing a spike in uncleanlyClosedWALs metric.
[
https://issues.apache.org/jira/browse/HBASE-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharath Vissapragada resolved HBASE-25924.
--
Resolution: Fixed
> Seeing a spike in uncleanlyClosedWALs metric.
> -
>
> Key: HBASE-25924
> URL: https://issues.apache.org/jira/browse/HBASE-25924
> Project: HBase
> Issue Type: Bug
> Components: Replication, wal
>Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.5.0, 2.4.4
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4, 1.7.1
>
>
> Getting the following log line in all of our production clusters when
> WALEntryStream is dequeuing WAL file.
> {noformat}
> 2021-05-02 04:01:30,437 DEBUG [04901996] regionserver.WALEntryStream -
> Reached the end of WAL file hdfs://. It was not closed
> cleanly, so we did not parse 8 bytes of data. This is normally ok.
> {noformat}
> The 8 bytes are usually the trailer serialized size (SIZE_OF_INT (4bytes) +
> "LAWP" (4 bytes) = 8 bytes)
> While dequeue'ing the WAL file from WALEntryStream, we reset the reader here.
> [WALEntryStream|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java#L199-L221]
> {code:java}
> private void tryAdvanceEntry() throws IOException {
> if (checkReader()) {
> readNextEntryAndSetPosition();
> if (currentEntry == null) { // no more entries in this log file - see
> if log was rolled
> if (logQueue.getQueue(walGroupId).size() > 1) { // log was rolled
> // Before dequeueing, we should always get one more attempt at
> reading.
> // This is in case more entries came in after we opened the reader,
> // and a new log was enqueued while we were reading. See HBASE-6758
> resetReader(); ---> HERE
> readNextEntryAndSetPosition();
> if (currentEntry == null) {
> if (checkAllBytesParsed()) { // now we're certain we're done with
> this log file
> dequeueCurrentLog();
> if (openNextLog()) {
> readNextEntryAndSetPosition();
> }
> }
> }
> } // no other logs, we've simply hit the end of the current open log.
> Do nothing
> }
> }
> // do nothing if we don't have a WAL Reader (e.g. if there's no logs in
> queue)
> }
> {code}
> In resetReader, we call the following methods, WALEntryStream#resetReader
> > ProtobufLogReader#reset ---> ProtobufLogReader#initInternal.
> In ProtobufLogReader#initInternal, we try to create the whole reader object
> from scratch to see if any new data has been written.
> We reset all the fields of ProtobufLogReader except for ReaderBase#fileLength.
> We calculate whether trailer is present or not depending on fileLength.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25932) TestWALEntryStream#testCleanClosedWALs test is failing.
[
https://issues.apache.org/jira/browse/HBASE-25932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharath Vissapragada updated HBASE-25932:
-
Component/s: wal
metrics
> TestWALEntryStream#testCleanClosedWALs test is failing.
> ---
>
> Key: HBASE-25932
> URL: https://issues.apache.org/jira/browse/HBASE-25932
> Project: HBase
> Issue Type: Bug
> Components: metrics, Replication, wal
>Affects Versions: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4
>Reporter: Rushabh Shah
>Assignee: Bharath Vissapragada
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4
>
> Attachments: HBASE-25932-test-approach.patch
>
>
> We are seeing the following test failure.
> TestWALEntryStream#testCleanClosedWALs
> This test was added in HBASE-25924. I don't think the test failure has
> anything to do with the patch in HBASE-25924.
> Before HBASE-25924, we were *not* monitoring _uncleanlyClosedWAL_ metric. In
> all the branches, we were not parsing the wal trailer when we close the wal
> reader inside ReplicationSourceWALReader thread. The root cause was when we
> add active WAL to ReplicationSourceWALReader, we cache the file size when the
> wal was being actively written and once the wal was closed and replicated and
> removed from WALEntryStream, we did reset the ProtobufLogReader object but we
> didn't update the length of the wal and that was causing EOF errors since it
> can't find the WALTrailer with the stale wal file length.
> The fix applied nicely to branch-1 since we use FSHlog implementation which
> closes the WAL file sychronously.
> But in branch-2 and master, we use _AsyncFSWAL_ implementation and the
> closing of wal file is done asynchronously (as the name suggests). This is
> causing the test to fail. Below is the test.
> {code:java}
> @Test
> public void testCleanClosedWALs() throws Exception {
> try (WALEntryStream entryStream = new WALEntryStream(
> logQueue, CONF, 0, log, null, logQueue.getMetrics(), fakeWalGroupId)) {
> assertEquals(0, logQueue.getMetrics().getUncleanlyClosedWALs());
> appendToLogAndSync();
> assertNotNull(entryStream.next());
> log.rollWriter(); ===> This does an asynchronous close of wal.
> appendToLogAndSync();
> assertNotNull(entryStream.next());
> assertEquals(0, logQueue.getMetrics().getUncleanlyClosedWALs());
> }
> }
> {code}
> In the above code, when we roll writer, we don't close the old wal file
> immediately so the ReplicationReader thread is not able to get the updated
> wal file size and that is throwing EOF errors.
> If I add a sleep of few milliseconds (1 ms in my local env) between
> rollWriter and appendToLogAndSync statement then the test passes but this is
> *not* a proper fix since we are working around the race between
> ReplicationSourceWALReaderThread and closing of WAL file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25924) Seeing a spike in uncleanlyClosedWALs metric.
[
https://issues.apache.org/jira/browse/HBASE-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharath Vissapragada updated HBASE-25924:
-
Fix Version/s: (was: 2.3.6)
> Seeing a spike in uncleanlyClosedWALs metric.
> -
>
> Key: HBASE-25924
> URL: https://issues.apache.org/jira/browse/HBASE-25924
> Project: HBase
> Issue Type: Bug
> Components: Replication, wal
>Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.5.0, 2.4.4
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4, 1.7.1
>
>
> Getting the following log line in all of our production clusters when
> WALEntryStream is dequeuing WAL file.
> {noformat}
> 2021-05-02 04:01:30,437 DEBUG [04901996] regionserver.WALEntryStream -
> Reached the end of WAL file hdfs://. It was not closed
> cleanly, so we did not parse 8 bytes of data. This is normally ok.
> {noformat}
> The 8 bytes are usually the trailer serialized size (SIZE_OF_INT (4bytes) +
> "LAWP" (4 bytes) = 8 bytes)
> While dequeue'ing the WAL file from WALEntryStream, we reset the reader here.
> [WALEntryStream|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java#L199-L221]
> {code:java}
> private void tryAdvanceEntry() throws IOException {
> if (checkReader()) {
> readNextEntryAndSetPosition();
> if (currentEntry == null) { // no more entries in this log file - see
> if log was rolled
> if (logQueue.getQueue(walGroupId).size() > 1) { // log was rolled
> // Before dequeueing, we should always get one more attempt at
> reading.
> // This is in case more entries came in after we opened the reader,
> // and a new log was enqueued while we were reading. See HBASE-6758
> resetReader(); ---> HERE
> readNextEntryAndSetPosition();
> if (currentEntry == null) {
> if (checkAllBytesParsed()) { // now we're certain we're done with
> this log file
> dequeueCurrentLog();
> if (openNextLog()) {
> readNextEntryAndSetPosition();
> }
> }
> }
> } // no other logs, we've simply hit the end of the current open log.
> Do nothing
> }
> }
> // do nothing if we don't have a WAL Reader (e.g. if there's no logs in
> queue)
> }
> {code}
> In resetReader, we call the following methods, WALEntryStream#resetReader
> > ProtobufLogReader#reset ---> ProtobufLogReader#initInternal.
> In ProtobufLogReader#initInternal, we try to create the whole reader object
> from scratch to see if any new data has been written.
> We reset all the fields of ProtobufLogReader except for ReaderBase#fileLength.
> We calculate whether trailer is present or not depending on fileLength.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25924) Seeing a spike in uncleanlyClosedWALs metric.
[
https://issues.apache.org/jira/browse/HBASE-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355221#comment-17355221
]
Bharath Vissapragada commented on HBASE-25924:
--
Thanks, opened HBASE-25957 as a subtask.
> Seeing a spike in uncleanlyClosedWALs metric.
> -
>
> Key: HBASE-25924
> URL: https://issues.apache.org/jira/browse/HBASE-25924
> Project: HBase
> Issue Type: Bug
> Components: Replication, wal
>Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.5.0, 2.4.4
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4, 1.7.1
>
>
> Getting the following log line in all of our production clusters when
> WALEntryStream is dequeuing WAL file.
> {noformat}
> 2021-05-02 04:01:30,437 DEBUG [04901996] regionserver.WALEntryStream -
> Reached the end of WAL file hdfs://. It was not closed
> cleanly, so we did not parse 8 bytes of data. This is normally ok.
> {noformat}
> The 8 bytes are usually the trailer serialized size (SIZE_OF_INT (4bytes) +
> "LAWP" (4 bytes) = 8 bytes)
> While dequeue'ing the WAL file from WALEntryStream, we reset the reader here.
> [WALEntryStream|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java#L199-L221]
> {code:java}
> private void tryAdvanceEntry() throws IOException {
> if (checkReader()) {
> readNextEntryAndSetPosition();
> if (currentEntry == null) { // no more entries in this log file - see
> if log was rolled
> if (logQueue.getQueue(walGroupId).size() > 1) { // log was rolled
> // Before dequeueing, we should always get one more attempt at
> reading.
> // This is in case more entries came in after we opened the reader,
> // and a new log was enqueued while we were reading. See HBASE-6758
> resetReader(); ---> HERE
> readNextEntryAndSetPosition();
> if (currentEntry == null) {
> if (checkAllBytesParsed()) { // now we're certain we're done with
> this log file
> dequeueCurrentLog();
> if (openNextLog()) {
> readNextEntryAndSetPosition();
> }
> }
> }
> } // no other logs, we've simply hit the end of the current open log.
> Do nothing
> }
> }
> // do nothing if we don't have a WAL Reader (e.g. if there's no logs in
> queue)
> }
> {code}
> In resetReader, we call the following methods, WALEntryStream#resetReader
> > ProtobufLogReader#reset ---> ProtobufLogReader#initInternal.
> In ProtobufLogReader#initInternal, we try to create the whole reader object
> from scratch to see if any new data has been written.
> We reset all the fields of ProtobufLogReader except for ReaderBase#fileLength.
> We calculate whether trailer is present or not depending on fileLength.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25957) Back port HBASE-25924 and its related patches to branch-2.3
[ https://issues.apache.org/jira/browse/HBASE-25957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355219#comment-17355219 ] Bharath Vissapragada commented on HBASE-25957: -- [~rushabh.shah] FYI, if you have any free cycles. > Back port HBASE-25924 and its related patches to branch-2.3 > --- > > Key: HBASE-25957 > URL: https://issues.apache.org/jira/browse/HBASE-25957 > Project: HBase > Issue Type: Sub-task > Components: metrics, Replication, wal >Affects Versions: 2.3.6 >Reporter: Bharath Vissapragada >Priority: Major > > The branch has diverged a bit and the patches are not applying clean. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25957) Back port HBASE-25924 and its related patches to branch-2.3
[ https://issues.apache.org/jira/browse/HBASE-25957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharath Vissapragada updated HBASE-25957: - Component/s: wal Replication metrics > Back port HBASE-25924 and its related patches to branch-2.3 > --- > > Key: HBASE-25957 > URL: https://issues.apache.org/jira/browse/HBASE-25957 > Project: HBase > Issue Type: Sub-task > Components: metrics, Replication, wal >Affects Versions: 2.3.6 >Reporter: Bharath Vissapragada >Priority: Major > > The branch has diverged a bit and the patches are not applying clean. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25957) Back port HBASE-25924 and its related patches to branch-2.3
Bharath Vissapragada created HBASE-25957: Summary: Back port HBASE-25924 and its related patches to branch-2.3 Key: HBASE-25957 URL: https://issues.apache.org/jira/browse/HBASE-25957 Project: HBase Issue Type: Sub-task Affects Versions: 2.3.6 Reporter: Bharath Vissapragada The branch has diverged a bit and the patches are not applying clean. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25956) Add 2.4.3 to download page
[ https://issues.apache.org/jira/browse/HBASE-25956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-25956. - Resolution: Fixed > Add 2.4.3 to download page > -- > > Key: HBASE-25956 > URL: https://issues.apache.org/jira/browse/HBASE-25956 > Project: HBase > Issue Type: Bug >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Minor > Fix For: 2.4.3 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25956) Add 2.4.3 to download page
Andrew Kyle Purtell created HBASE-25956: --- Summary: Add 2.4.3 to download page Key: HBASE-25956 URL: https://issues.apache.org/jira/browse/HBASE-25956 Project: HBase Issue Type: Bug Reporter: Andrew Kyle Purtell Assignee: Andrew Kyle Purtell Fix For: 2.4.3 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23634) Enable "Split WAL to HFile" by default
[ https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355215#comment-17355215 ] Michael Stack commented on HBASE-23634: --- This feature is unfinished w/ open questions still. > Enable "Split WAL to HFile" by default > -- > > Key: HBASE-23634 > URL: https://issues.apache.org/jira/browse/HBASE-23634 > Project: HBase > Issue Type: Task >Affects Versions: 3.0.0-alpha-1, 2.3.0 >Reporter: Guanghao Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-19701) Close without justification following succesful open
[
https://issues.apache.org/jira/browse/HBASE-19701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack resolved HBASE-19701.
---
Resolution: Cannot Reproduce
Closing as 'Cannot Reporduce' ... haven't seen it since original filing... May
still be there but no work done on this item... Will open new one if seen again.
> Close without justification following succesful open
>
>
> Key: HBASE-19701
> URL: https://issues.apache.org/jira/browse/HBASE-19701
> Project: HBase
> Issue Type: Bug
>Reporter: Michael Stack
>Priority: Critical
> Fix For: 3.0.0-alpha-2
>
>
> [~jmspaggi] conjured an interesting condition where we close a region soon
> after open WITHOUT seemingly saying why (It looks like Master is asking for
> region CLOSE but that is not clear looking at RegionServer log).
> Here is log snippet from https://pastebin.com/0r76Y6ap (in case the pastebin
> evaporates)
> {code}
>
> 2017-12-31 09:54:20,864 INFO
> [PostOpenDeployTasks:f49f3cbb7f3db4cf96c7eb3b0cf83869]
> regionserver.HRegionServer: Post open deploy tasks for
> TestTable,0408944640,1505391191559.f49f3cbb7f3db4cf96c7eb3b0cf83869.
> 2017-12-31 09:54:20,870 INFO
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1]
> regionserver.CompactingMemStore: Setting in-memory flush size threshold to
> 13421772 and immutable segments index to be of type CHUNK_MAP
> 2017-12-31 09:54:20,870 INFO
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1] regionserver.HStore:
> Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore
> 2017-12-31 09:54:20,870 INFO
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1] hfile.CacheConfig: Created
> cacheConfig for info: blockCache=LruBlockCache{blockCount=0,
> currentSize=2454760, freeSize=3347745560, maxSize=3350200320,
> heapSize=2454760, minSize=3182690304, minFactor=0.95, multiSize=1591345152,
> multiFactor=0.5, singleSize=795672576, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2017-12-31 09:54:20,872 INFO
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1]
> compactions.CompactionConfiguration: size [134217728, 9223372036854775807,
> 9223372036854775807); files [3, 10); ratio 1,20; off-peak ratio 5,00;
> throttle point 2684354560; major period 60480, major jitter 0,50, min
> locality to compact 0,00; tiered compaction: max_age 9223372036854775807,
> incoming window min 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2017-12-31 09:54:20,903 INFO
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1]
> regionserver.CompactingMemStore: Setting in-memory flush size threshold to
> 13421772 and immutable segments index to be of type CHUNK_MAP
> 2017-12-31 09:54:20,904 INFO
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1] regionserver.HStore:
> Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore
> 2017-12-31 09:54:20,904 INFO
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1] hfile.CacheConfig: Created
> cacheConfig for info: blockCache=LruBlockCache{blockCount=0,
> currentSize=2454760, freeSize=3347745560, maxSize=3350200320,
> heapSize=2454760, minSize=3182690304, minFactor=0.95, multiSize=1591345152,
> multiFactor=0.5, singleSize=795672576, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2017-12-31 09:54:20,905 INFO
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1]
> compactions.CompactionConfiguration: size [134217728, 9223372036854775807,
> 9223372036854775807); files [3, 10); ratio 1,20; off-peak ratio 5,00;
> throttle point 2684354560; major period 60480, major jitter 0,50, min
> locality to compact 0,00; tiered compaction: max_age 9223372036854775807,
> incoming window min 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2017-12-31 09:54:20,929 INFO [RS_OPEN_REGION-node1:16020-1]
> regionserver.HRegion: Setting FlushNonSloppyStoresFirstPolicy for the
> region=TestTable,0262144000,1505391191559.166b9c45d7724f72fd126adb4445d6ec.
> 2017-12-31 09:54:20,956 INFO [RS_OPEN_REGION-node1:16020-0]
>
[jira] [Comment Edited] (HBASE-25924) Seeing a spike in uncleanlyClosedWALs metric.
[
https://issues.apache.org/jira/browse/HBASE-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355208#comment-17355208
]
Andrew Kyle Purtell edited comment on HBASE-25924 at 6/1/21, 4:29 PM:
--
bq. Whats the general guidance on back porting to branch-2.3?
[~bharathv]
It is a live branch that we are still releasing from, so should receive all
relevant bug fixes -- and this issue is relevant according to that criteria --
and changes that are meaningful for cross-branch compatibility (i.e. impacting
an upgrade from 1.x, or impacting an upgrade to 2.4 or later).
bq. That branch has diverged quite a bit and this patch doesn't apply cleanly.
It's fine to resolve this issue without a 2.3 fix version and open a subtask or
another jira for a backport to 2.3, especially if a new PR is advisable due to
divergence.
was (Author: apurtell):
bq. Whats the general guidance on back porting to branch-2.3?
[~bharathv]
It is a live branch that we are still releasing from, so should receive all
relevant bug fixes -- and this issue is relevant according to that criteria --
and changes that are meaningful for cross-branch compatibility (i.e. impacting
an upgrade from 1.x, or impacting an upgrade to 2.4 or later).
bq. That branch has diverged quite a bit and this patch doesn't apply cleanly.
It's fine to resolve this issue without a 2.3 fix version and open a subtask or
another jira for a backport to 2.3.
> Seeing a spike in uncleanlyClosedWALs metric.
> -
>
> Key: HBASE-25924
> URL: https://issues.apache.org/jira/browse/HBASE-25924
> Project: HBase
> Issue Type: Bug
> Components: Replication, wal
>Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.5.0, 2.4.4
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4, 1.7.1
>
>
> Getting the following log line in all of our production clusters when
> WALEntryStream is dequeuing WAL file.
> {noformat}
> 2021-05-02 04:01:30,437 DEBUG [04901996] regionserver.WALEntryStream -
> Reached the end of WAL file hdfs://. It was not closed
> cleanly, so we did not parse 8 bytes of data. This is normally ok.
> {noformat}
> The 8 bytes are usually the trailer serialized size (SIZE_OF_INT (4bytes) +
> "LAWP" (4 bytes) = 8 bytes)
> While dequeue'ing the WAL file from WALEntryStream, we reset the reader here.
> [WALEntryStream|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java#L199-L221]
> {code:java}
> private void tryAdvanceEntry() throws IOException {
> if (checkReader()) {
> readNextEntryAndSetPosition();
> if (currentEntry == null) { // no more entries in this log file - see
> if log was rolled
> if (logQueue.getQueue(walGroupId).size() > 1) { // log was rolled
> // Before dequeueing, we should always get one more attempt at
> reading.
> // This is in case more entries came in after we opened the reader,
> // and a new log was enqueued while we were reading. See HBASE-6758
> resetReader(); ---> HERE
> readNextEntryAndSetPosition();
> if (currentEntry == null) {
> if (checkAllBytesParsed()) { // now we're certain we're done with
> this log file
> dequeueCurrentLog();
> if (openNextLog()) {
> readNextEntryAndSetPosition();
> }
> }
> }
> } // no other logs, we've simply hit the end of the current open log.
> Do nothing
> }
> }
> // do nothing if we don't have a WAL Reader (e.g. if there's no logs in
> queue)
> }
> {code}
> In resetReader, we call the following methods, WALEntryStream#resetReader
> > ProtobufLogReader#reset ---> ProtobufLogReader#initInternal.
> In ProtobufLogReader#initInternal, we try to create the whole reader object
> from scratch to see if any new data has been written.
> We reset all the fields of ProtobufLogReader except for ReaderBase#fileLength.
> We calculate whether trailer is present or not depending on fileLength.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25924) Seeing a spike in uncleanlyClosedWALs metric.
[
https://issues.apache.org/jira/browse/HBASE-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355208#comment-17355208
]
Andrew Kyle Purtell edited comment on HBASE-25924 at 6/1/21, 4:28 PM:
--
bq. Whats the general guidance on back porting to branch-2.3?
[~bharathv]
It is a live branch that we are still releasing from, so should receive all
relevant bug fixes -- and this issue is relevant according to that criteria --
and changes that are meaningful for cross-branch compatibility (i.e. impacting
an upgrade from 1.x, or impacting an upgrade to 2.4 or later).
bq. That branch has diverged quite a bit and this patch doesn't apply cleanly.
It's fine to resolve this issue without a 2.3 fix version and open a subtask or
another jira for a backport to 2.3.
was (Author: apurtell):
bq. Whats the general guidance on back porting to branch-2.3?
[~bharathv]
It is a live branch that we are still releasing from, so should receive all
relevant bug fixes and changes that are meaningful for cross-branch
compatibility (i.e. impacting an upgrade from 1.x, or impacting an upgrade to
2.4 or later).
bq. That branch has diverged quite a bit and this patch doesn't apply cleanly.
It's fine to resolve this issue without a 2.3 fix version and open a subtask or
another jira for a backport to 2.3.
> Seeing a spike in uncleanlyClosedWALs metric.
> -
>
> Key: HBASE-25924
> URL: https://issues.apache.org/jira/browse/HBASE-25924
> Project: HBase
> Issue Type: Bug
> Components: Replication, wal
>Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.5.0, 2.4.4
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4, 1.7.1
>
>
> Getting the following log line in all of our production clusters when
> WALEntryStream is dequeuing WAL file.
> {noformat}
> 2021-05-02 04:01:30,437 DEBUG [04901996] regionserver.WALEntryStream -
> Reached the end of WAL file hdfs://. It was not closed
> cleanly, so we did not parse 8 bytes of data. This is normally ok.
> {noformat}
> The 8 bytes are usually the trailer serialized size (SIZE_OF_INT (4bytes) +
> "LAWP" (4 bytes) = 8 bytes)
> While dequeue'ing the WAL file from WALEntryStream, we reset the reader here.
> [WALEntryStream|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java#L199-L221]
> {code:java}
> private void tryAdvanceEntry() throws IOException {
> if (checkReader()) {
> readNextEntryAndSetPosition();
> if (currentEntry == null) { // no more entries in this log file - see
> if log was rolled
> if (logQueue.getQueue(walGroupId).size() > 1) { // log was rolled
> // Before dequeueing, we should always get one more attempt at
> reading.
> // This is in case more entries came in after we opened the reader,
> // and a new log was enqueued while we were reading. See HBASE-6758
> resetReader(); ---> HERE
> readNextEntryAndSetPosition();
> if (currentEntry == null) {
> if (checkAllBytesParsed()) { // now we're certain we're done with
> this log file
> dequeueCurrentLog();
> if (openNextLog()) {
> readNextEntryAndSetPosition();
> }
> }
> }
> } // no other logs, we've simply hit the end of the current open log.
> Do nothing
> }
> }
> // do nothing if we don't have a WAL Reader (e.g. if there's no logs in
> queue)
> }
> {code}
> In resetReader, we call the following methods, WALEntryStream#resetReader
> > ProtobufLogReader#reset ---> ProtobufLogReader#initInternal.
> In ProtobufLogReader#initInternal, we try to create the whole reader object
> from scratch to see if any new data has been written.
> We reset all the fields of ProtobufLogReader except for ReaderBase#fileLength.
> We calculate whether trailer is present or not depending on fileLength.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25924) Seeing a spike in uncleanlyClosedWALs metric.
[
https://issues.apache.org/jira/browse/HBASE-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355208#comment-17355208
]
Andrew Kyle Purtell edited comment on HBASE-25924 at 6/1/21, 4:27 PM:
--
bq. Whats the general guidance on back porting to branch-2.3?
[~bharathv]
It is a live branch that we are still releasing from, so should receive all
relevant bug fixes and changes that are meaningful for cross-branch
compatibility (i.e. impacting an upgrade from 1.x, or impacting an upgrade to
2.4 or later).
bq. That branch has diverged quite a bit and this patch doesn't apply cleanly.
It's fine to resolve this issue without a 2.3 fix version and open a subtask or
another jira for a backport to 2.3.
was (Author: apurtell):
bq. Whats the general guidance on back porting to branch-2.3?
It is a live branch that we are still releasing from, so should receive all
relevant bug fixes and changes that are meaningful for cross-branch
compatibility (i.e. impacting an upgrade from 1.x, or impacting an upgrade to
2.4 or later).
bq. That branch has diverged quite a bit and this patch doesn't apply cleanly.
It's fine to resolve this issue without a 2.3 fix version and open a subtask or
another jira for a backport to 2.3.
> Seeing a spike in uncleanlyClosedWALs metric.
> -
>
> Key: HBASE-25924
> URL: https://issues.apache.org/jira/browse/HBASE-25924
> Project: HBase
> Issue Type: Bug
> Components: Replication, wal
>Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.5.0, 2.4.4
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4, 1.7.1
>
>
> Getting the following log line in all of our production clusters when
> WALEntryStream is dequeuing WAL file.
> {noformat}
> 2021-05-02 04:01:30,437 DEBUG [04901996] regionserver.WALEntryStream -
> Reached the end of WAL file hdfs://. It was not closed
> cleanly, so we did not parse 8 bytes of data. This is normally ok.
> {noformat}
> The 8 bytes are usually the trailer serialized size (SIZE_OF_INT (4bytes) +
> "LAWP" (4 bytes) = 8 bytes)
> While dequeue'ing the WAL file from WALEntryStream, we reset the reader here.
> [WALEntryStream|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java#L199-L221]
> {code:java}
> private void tryAdvanceEntry() throws IOException {
> if (checkReader()) {
> readNextEntryAndSetPosition();
> if (currentEntry == null) { // no more entries in this log file - see
> if log was rolled
> if (logQueue.getQueue(walGroupId).size() > 1) { // log was rolled
> // Before dequeueing, we should always get one more attempt at
> reading.
> // This is in case more entries came in after we opened the reader,
> // and a new log was enqueued while we were reading. See HBASE-6758
> resetReader(); ---> HERE
> readNextEntryAndSetPosition();
> if (currentEntry == null) {
> if (checkAllBytesParsed()) { // now we're certain we're done with
> this log file
> dequeueCurrentLog();
> if (openNextLog()) {
> readNextEntryAndSetPosition();
> }
> }
> }
> } // no other logs, we've simply hit the end of the current open log.
> Do nothing
> }
> }
> // do nothing if we don't have a WAL Reader (e.g. if there's no logs in
> queue)
> }
> {code}
> In resetReader, we call the following methods, WALEntryStream#resetReader
> > ProtobufLogReader#reset ---> ProtobufLogReader#initInternal.
> In ProtobufLogReader#initInternal, we try to create the whole reader object
> from scratch to see if any new data has been written.
> We reset all the fields of ProtobufLogReader except for ReaderBase#fileLength.
> We calculate whether trailer is present or not depending on fileLength.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25924) Seeing a spike in uncleanlyClosedWALs metric.
[
https://issues.apache.org/jira/browse/HBASE-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355208#comment-17355208
]
Andrew Kyle Purtell commented on HBASE-25924:
-
bq. Whats the general guidance on back porting to branch-2.3?
It is a live branch that we are still releasing from, so should receive all
relevant bug fixes and changes that are meaningful for cross-branch
compatibility (i.e. impacting an upgrade from 1.x, or impacting an upgrade to
2.4 or later).
bq. That branch has diverged quite a bit and this patch doesn't apply cleanly.
It's fine to resolve this issue without a 2.3 fix version and open a subtask or
another jira for a backport to 2.3.
> Seeing a spike in uncleanlyClosedWALs metric.
> -
>
> Key: HBASE-25924
> URL: https://issues.apache.org/jira/browse/HBASE-25924
> Project: HBase
> Issue Type: Bug
> Components: Replication, wal
>Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.5.0, 2.4.4
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4, 1.7.1
>
>
> Getting the following log line in all of our production clusters when
> WALEntryStream is dequeuing WAL file.
> {noformat}
> 2021-05-02 04:01:30,437 DEBUG [04901996] regionserver.WALEntryStream -
> Reached the end of WAL file hdfs://. It was not closed
> cleanly, so we did not parse 8 bytes of data. This is normally ok.
> {noformat}
> The 8 bytes are usually the trailer serialized size (SIZE_OF_INT (4bytes) +
> "LAWP" (4 bytes) = 8 bytes)
> While dequeue'ing the WAL file from WALEntryStream, we reset the reader here.
> [WALEntryStream|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java#L199-L221]
> {code:java}
> private void tryAdvanceEntry() throws IOException {
> if (checkReader()) {
> readNextEntryAndSetPosition();
> if (currentEntry == null) { // no more entries in this log file - see
> if log was rolled
> if (logQueue.getQueue(walGroupId).size() > 1) { // log was rolled
> // Before dequeueing, we should always get one more attempt at
> reading.
> // This is in case more entries came in after we opened the reader,
> // and a new log was enqueued while we were reading. See HBASE-6758
> resetReader(); ---> HERE
> readNextEntryAndSetPosition();
> if (currentEntry == null) {
> if (checkAllBytesParsed()) { // now we're certain we're done with
> this log file
> dequeueCurrentLog();
> if (openNextLog()) {
> readNextEntryAndSetPosition();
> }
> }
> }
> } // no other logs, we've simply hit the end of the current open log.
> Do nothing
> }
> }
> // do nothing if we don't have a WAL Reader (e.g. if there's no logs in
> queue)
> }
> {code}
> In resetReader, we call the following methods, WALEntryStream#resetReader
> > ProtobufLogReader#reset ---> ProtobufLogReader#initInternal.
> In ProtobufLogReader#initInternal, we try to create the whole reader object
> from scratch to see if any new data has been written.
> We reset all the fields of ProtobufLogReader except for ReaderBase#fileLength.
> We calculate whether trailer is present or not depending on fileLength.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] saintstack commented on pull request #3336: HBASE-25920 Support Hadoop 3.3.1
saintstack commented on pull request #3336: URL: https://github.com/apache/hbase/pull/3336#issuecomment-852246361 Retry. TestLogRolling and WAL on EC... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-13126) Move HBaseTestingUtility and associated support classes into hbase-testing-utility module
[ https://issues.apache.org/jira/browse/HBASE-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355191#comment-17355191 ] Sean Busbey commented on HBASE-13126: - That's a good idea on making this incrementally better Duo. > Move HBaseTestingUtility and associated support classes into > hbase-testing-utility module > - > > Key: HBASE-13126 > URL: https://issues.apache.org/jira/browse/HBASE-13126 > Project: HBase > Issue Type: Task > Components: API >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 3.0.0-alpha-2 > > > Over in the review for HBASE-12972, [~enis] mentioned that one of the HBTU > methods wasn't intended for public consumption. > Can we build a list of such methods across the API, appropriately annotate > them for 2.0, and deprecate them in earlier versions with a warning that > they're going to be restricted? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3347: HBASE-25955 Setting NAMESPACES when adding a replication peer doesn't…
Apache-HBase commented on pull request #3347: URL: https://github.com/apache/hbase/pull/3347#issuecomment-852232792 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 37s | master passed | | +1 :green_heart: | compile | 3m 11s | master passed | | +1 :green_heart: | checkstyle | 1m 3s | master passed | | +1 :green_heart: | spotbugs | 2m 1s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 39s | the patch passed | | +1 :green_heart: | compile | 3m 14s | the patch passed | | +1 :green_heart: | javac | 3m 14s | the patch passed | | -0 :warning: | checkstyle | 1m 1s | hbase-server: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 1s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 12s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 46m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3347/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3347 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux da07db58ccb6 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b04c3c7786 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3347/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3347/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25924) Seeing a spike in uncleanlyClosedWALs metric.
[
https://issues.apache.org/jira/browse/HBASE-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355178#comment-17355178
]
Bharath Vissapragada commented on HBASE-25924:
--
HBASE-25932 is now committed to master/branch-2/branch2.4.
[~apurtell] Whats the general guidance on back porting to branch-2.3? That
branch has diverged quite a bit and this patch doesn't apply cleanly.
> Seeing a spike in uncleanlyClosedWALs metric.
> -
>
> Key: HBASE-25924
> URL: https://issues.apache.org/jira/browse/HBASE-25924
> Project: HBase
> Issue Type: Bug
> Components: Replication, wal
>Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.5.0, 2.4.4
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4, 1.7.1
>
>
> Getting the following log line in all of our production clusters when
> WALEntryStream is dequeuing WAL file.
> {noformat}
> 2021-05-02 04:01:30,437 DEBUG [04901996] regionserver.WALEntryStream -
> Reached the end of WAL file hdfs://. It was not closed
> cleanly, so we did not parse 8 bytes of data. This is normally ok.
> {noformat}
> The 8 bytes are usually the trailer serialized size (SIZE_OF_INT (4bytes) +
> "LAWP" (4 bytes) = 8 bytes)
> While dequeue'ing the WAL file from WALEntryStream, we reset the reader here.
> [WALEntryStream|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java#L199-L221]
> {code:java}
> private void tryAdvanceEntry() throws IOException {
> if (checkReader()) {
> readNextEntryAndSetPosition();
> if (currentEntry == null) { // no more entries in this log file - see
> if log was rolled
> if (logQueue.getQueue(walGroupId).size() > 1) { // log was rolled
> // Before dequeueing, we should always get one more attempt at
> reading.
> // This is in case more entries came in after we opened the reader,
> // and a new log was enqueued while we were reading. See HBASE-6758
> resetReader(); ---> HERE
> readNextEntryAndSetPosition();
> if (currentEntry == null) {
> if (checkAllBytesParsed()) { // now we're certain we're done with
> this log file
> dequeueCurrentLog();
> if (openNextLog()) {
> readNextEntryAndSetPosition();
> }
> }
> }
> } // no other logs, we've simply hit the end of the current open log.
> Do nothing
> }
> }
> // do nothing if we don't have a WAL Reader (e.g. if there's no logs in
> queue)
> }
> {code}
> In resetReader, we call the following methods, WALEntryStream#resetReader
> > ProtobufLogReader#reset ---> ProtobufLogReader#initInternal.
> In ProtobufLogReader#initInternal, we try to create the whole reader object
> from scratch to see if any new data has been written.
> We reset all the fields of ProtobufLogReader except for ReaderBase#fileLength.
> We calculate whether trailer is present or not depending on fileLength.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25932) TestWALEntryStream#testCleanClosedWALs test is failing.
[
https://issues.apache.org/jira/browse/HBASE-25932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharath Vissapragada updated HBASE-25932:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> TestWALEntryStream#testCleanClosedWALs test is failing.
> ---
>
> Key: HBASE-25932
> URL: https://issues.apache.org/jira/browse/HBASE-25932
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4
>Reporter: Rushabh Shah
>Assignee: Bharath Vissapragada
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4
>
> Attachments: HBASE-25932-test-approach.patch
>
>
> We are seeing the following test failure.
> TestWALEntryStream#testCleanClosedWALs
> This test was added in HBASE-25924. I don't think the test failure has
> anything to do with the patch in HBASE-25924.
> Before HBASE-25924, we were *not* monitoring _uncleanlyClosedWAL_ metric. In
> all the branches, we were not parsing the wal trailer when we close the wal
> reader inside ReplicationSourceWALReader thread. The root cause was when we
> add active WAL to ReplicationSourceWALReader, we cache the file size when the
> wal was being actively written and once the wal was closed and replicated and
> removed from WALEntryStream, we did reset the ProtobufLogReader object but we
> didn't update the length of the wal and that was causing EOF errors since it
> can't find the WALTrailer with the stale wal file length.
> The fix applied nicely to branch-1 since we use FSHlog implementation which
> closes the WAL file sychronously.
> But in branch-2 and master, we use _AsyncFSWAL_ implementation and the
> closing of wal file is done asynchronously (as the name suggests). This is
> causing the test to fail. Below is the test.
> {code:java}
> @Test
> public void testCleanClosedWALs() throws Exception {
> try (WALEntryStream entryStream = new WALEntryStream(
> logQueue, CONF, 0, log, null, logQueue.getMetrics(), fakeWalGroupId)) {
> assertEquals(0, logQueue.getMetrics().getUncleanlyClosedWALs());
> appendToLogAndSync();
> assertNotNull(entryStream.next());
> log.rollWriter(); ===> This does an asynchronous close of wal.
> appendToLogAndSync();
> assertNotNull(entryStream.next());
> assertEquals(0, logQueue.getMetrics().getUncleanlyClosedWALs());
> }
> }
> {code}
> In the above code, when we roll writer, we don't close the old wal file
> immediately so the ReplicationReader thread is not able to get the updated
> wal file size and that is throwing EOF errors.
> If I add a sleep of few milliseconds (1 ms in my local env) between
> rollWriter and appendToLogAndSync statement then the test passes but this is
> *not* a proper fix since we are working around the race between
> ReplicationSourceWALReaderThread and closing of WAL file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25932) TestWALEntryStream#testCleanClosedWALs test is failing.
[
https://issues.apache.org/jira/browse/HBASE-25932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharath Vissapragada updated HBASE-25932:
-
Fix Version/s: 2.4.4
2.5.0
3.0.0-alpha-1
> TestWALEntryStream#testCleanClosedWALs test is failing.
> ---
>
> Key: HBASE-25932
> URL: https://issues.apache.org/jira/browse/HBASE-25932
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4
>Reporter: Rushabh Shah
>Assignee: Bharath Vissapragada
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4
>
> Attachments: HBASE-25932-test-approach.patch
>
>
> We are seeing the following test failure.
> TestWALEntryStream#testCleanClosedWALs
> This test was added in HBASE-25924. I don't think the test failure has
> anything to do with the patch in HBASE-25924.
> Before HBASE-25924, we were *not* monitoring _uncleanlyClosedWAL_ metric. In
> all the branches, we were not parsing the wal trailer when we close the wal
> reader inside ReplicationSourceWALReader thread. The root cause was when we
> add active WAL to ReplicationSourceWALReader, we cache the file size when the
> wal was being actively written and once the wal was closed and replicated and
> removed from WALEntryStream, we did reset the ProtobufLogReader object but we
> didn't update the length of the wal and that was causing EOF errors since it
> can't find the WALTrailer with the stale wal file length.
> The fix applied nicely to branch-1 since we use FSHlog implementation which
> closes the WAL file sychronously.
> But in branch-2 and master, we use _AsyncFSWAL_ implementation and the
> closing of wal file is done asynchronously (as the name suggests). This is
> causing the test to fail. Below is the test.
> {code:java}
> @Test
> public void testCleanClosedWALs() throws Exception {
> try (WALEntryStream entryStream = new WALEntryStream(
> logQueue, CONF, 0, log, null, logQueue.getMetrics(), fakeWalGroupId)) {
> assertEquals(0, logQueue.getMetrics().getUncleanlyClosedWALs());
> appendToLogAndSync();
> assertNotNull(entryStream.next());
> log.rollWriter(); ===> This does an asynchronous close of wal.
> appendToLogAndSync();
> assertNotNull(entryStream.next());
> assertEquals(0, logQueue.getMetrics().getUncleanlyClosedWALs());
> }
> }
> {code}
> In the above code, when we roll writer, we don't close the old wal file
> immediately so the ReplicationReader thread is not able to get the updated
> wal file size and that is throwing EOF errors.
> If I add a sleep of few milliseconds (1 ms in my local env) between
> rollWriter and appendToLogAndSync statement then the test passes but this is
> *not* a proper fix since we are working around the race between
> ReplicationSourceWALReaderThread and closing of WAL file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] bharathv merged pull request #3346: HBASE-25932: Ensure replication reads the trailer bytes from WAL. (#3…
bharathv merged pull request #3346: URL: https://github.com/apache/hbase/pull/3346 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bharathv merged pull request #3345: HBASE-25932: Ensure replication reads the trailer bytes from WAL. (#3…
bharathv merged pull request #3345: URL: https://github.com/apache/hbase/pull/3345 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bharathv merged pull request #3344: HBASE-25932 addendum: Add test comments.
bharathv merged pull request #3344: URL: https://github.com/apache/hbase/pull/3344 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25916) Move FavoredNodeLoadBalancer to hbase-balancer module
[
https://issues.apache.org/jira/browse/HBASE-25916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355111#comment-17355111
]
Hudson commented on HBASE-25916:
Results for branch master
[build #311 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Move FavoredNodeLoadBalancer to hbase-balancer module
> -
>
> Key: HBASE-25916
> URL: https://issues.apache.org/jira/browse/HBASE-25916
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, FavoredNodes
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25948) Remove deprecated ZK command 'rmr' in hbase-cleanup.sh
[
https://issues.apache.org/jira/browse/HBASE-25948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355109#comment-17355109
]
Hudson commented on HBASE-25948:
Results for branch master
[build #311 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove deprecated ZK command 'rmr' in hbase-cleanup.sh
> --
>
> Key: HBASE-25948
> URL: https://issues.apache.org/jira/browse/HBASE-25948
> Project: HBase
> Issue Type: Task
> Components: scripts
>Affects Versions: 3.0.0-alpha-1, 2.5.0
>Reporter: mokai
>Assignee: mokai
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Deprecated command 'rmr' was removed from ZooKeeper since 3.6.0(see:
> ZOOKEEPER-3411).
> It's better to use 'deleteall' instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25903) ReadOnlyZKClient APIs - CompletableFuture.get() calls can cause threads to hang forver when ZK client create throws Non IOException
[
https://issues.apache.org/jira/browse/HBASE-25903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355110#comment-17355110
]
Hudson commented on HBASE-25903:
Results for branch master
[build #311 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> ReadOnlyZKClient APIs - CompletableFuture.get() calls can cause threads to
> hang forver when ZK client create throws Non IOException
> ---
>
> Key: HBASE-25903
> URL: https://issues.apache.org/jira/browse/HBASE-25903
> Project: HBase
> Issue Type: Bug
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4
>
>
> This is applicable for zk client versions which is not having fix for
> ZOOKEEPER-2184.
> Now we are on zookeeper 3.5.7 on active 2.x branches. Still its better to
> handle this case in our code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25941) TestRESTServerSSL fails because of jdk bug
[
https://issues.apache.org/jira/browse/HBASE-25941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355108#comment-17355108
]
Hudson commented on HBASE-25941:
Results for branch master
[build #311 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> TestRESTServerSSL fails because of jdk bug
> --
>
> Key: HBASE-25941
> URL: https://issues.apache.org/jira/browse/HBASE-25941
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: Michael Stack
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.6, 2.4.4
>
>
> [~weijing329] identified issue in TestRESTServerSSL when using jdk8 292+. It
> came up in comment in the parent issue. I verified it fails for me using jdk8
> v292. Here is the failure
> ```[INFO] Running org.apache.hadoop.hbase.rest.TestRESTServerSSL
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.4 s
> <<< FAILURE! - in org.apache.hadoop.hbase.rest.TestRESTServerSSL
> [ERROR] org.apache.hadoop.hbase.rest.TestRESTServerSSL Time elapsed: 1.387 s
> <<< ERROR!
> java.security.NoSuchAlgorithmException: unrecognized algorithm name:
> PBEWithSHA1AndDESede
> at
> org.apache.hadoop.hbase.rest.TestRESTServerSSL.beforeClass(TestRESTServerSSL.java:74)```
> For workaround, see https://github.com/bcgit/bc-java/issues/941
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25942) Get rid of null regioninfo in wrapped connection exceptions
[
https://issues.apache.org/jira/browse/HBASE-25942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355107#comment-17355107
]
Hudson commented on HBASE-25942:
Results for branch master
[build #311 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/311/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Get rid of null regioninfo in wrapped connection exceptions
> ---
>
> Key: HBASE-25942
> URL: https://issues.apache.org/jira/browse/HBASE-25942
> Project: HBase
> Issue Type: Improvement
> Components: logging
>Affects Versions: 3.0.0-alpha-1, 2.5.0, 2.4.4
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Trivial
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4
>
>
> I was debugging something and noticed this,
> {noformat}
> org.apache.hbase.thirdparty.com.google.protobuf.ServiceException:
> java.io.IOException: Call to address=localhost:59142null failed on local
> exception: java.io.IOException: Connection reset by peer
> {noformat}
> `null` there is regioninfo and because this call has nothing set on it, it
> was confusing where that null is coming from, added recently via HBASE-25735.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25553) It is better for ReplicationTracker.getListOfRegionServers to return ServerName instead of String
[ https://issues.apache.org/jira/browse/HBASE-25553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-25553: --- Issue Type: Improvement (was: Umbrella) > It is better for ReplicationTracker.getListOfRegionServers to return > ServerName instead of String > - > > Key: HBASE-25553 > URL: https://issues.apache.org/jira/browse/HBASE-25553 > Project: HBase > Issue Type: Improvement >Reporter: Sun Xin >Assignee: Sun Xin >Priority: Major > Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.3.5, 2.4.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (HBASE-24373) Implement JvmMetrics in HBase instead of using the one in hadoop
[ https://issues.apache.org/jira/browse/HBASE-24373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-24373: --- Comment: was deleted (was: Applied in master only right [~zhangduo]?) > Implement JvmMetrics in HBase instead of using the one in hadoop > > > Key: HBASE-24373 > URL: https://issues.apache.org/jira/browse/HBASE-24373 > Project: HBase > Issue Type: Sub-task > Components: logging, metrics >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > The JvmMetrics from hadoop is hard coded to use log4j. > Although we do not make use of the ability, it still prevents us to ban the > log4j dependencies completely. > So let's implement the JvmMetrics by our own, based on log4j2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24373) Implement JvmMetrics in HBase instead of using the one in hadoop
[ https://issues.apache.org/jira/browse/HBASE-24373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355062#comment-17355062 ] Anoop Sam John commented on HBASE-24373: Applied in master only right [~zhangduo]? > Implement JvmMetrics in HBase instead of using the one in hadoop > > > Key: HBASE-24373 > URL: https://issues.apache.org/jira/browse/HBASE-24373 > Project: HBase > Issue Type: Sub-task > Components: logging, metrics >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > The JvmMetrics from hadoop is hard coded to use log4j. > Although we do not make use of the ability, it still prevents us to ban the > log4j dependencies completely. > So let's implement the JvmMetrics by our own, based on log4j2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24734) Wrong comparator opening Region when 'split-to-WAL' enabled.
[
https://issues.apache.org/jira/browse/HBASE-24734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355058#comment-17355058
]
Anoop Sam John commented on HBASE-24734:
When WAL split happens for META region, we will hit this issue right?
bq. we should at least fail bulk load if trying to bulk load meta
So we should not use split to HFile for META table's WALs? [~stack]
> Wrong comparator opening Region when 'split-to-WAL' enabled.
>
>
> Key: HBASE-24734
> URL: https://issues.apache.org/jira/browse/HBASE-24734
> Project: HBase
> Issue Type: Sub-task
> Components: HFile, MTTR
>Reporter: Michael Stack
>Priority: Major
>
> Came across this when we were testing the 'split-to-hfile' feature running
> ITBLL:
>
> {code:java}
> 2020-07-10 10:16:49,983 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closing region hbase:meta,,1.15882307402020-07-10 10:16:49,997 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: Closed
> hbase:meta,,1.15882307402020-07-10 10:16:49,998 WARN
> org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler: Fatal error
> occurred while opening region hbase:meta,,1.1588230740,
> aborting...java.lang.IllegalArgumentException: Invalid range:
> IntegrationTestBigLinkedList,,1594350463222.8f89e01a5245e79946e22d8a8ab4698b.
> >
> IntegrationTestBigLinkedList,\x10\x02J\xA1,1594349535271.be24dc276f686e6dcc7fb9d3f91c8387.
> at
> org.apache.hadoop.hbase.client.RegionInfoBuilder$MutableRegionInfo.containsRange(RegionInfoBuilder.java:300)
> at
> org.apache.hadoop.hbase.regionserver.HStore.tryCommitRecoveredHFile(HStore.java:)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.loadRecoveredHFilesIfAny(HRegion.java:5442)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:1010)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:950)
>at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7490)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7448)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7424)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7382)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7333)
> at
> org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler.process(AssignRegionHandler.java:135)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)2020-07-10
> 10:16:50,005 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: *
> ABORTING region server hbasedn149.example.org,16020,1594375563853: Failed to
> open region hbase:meta,,1.1588230740 and can not recover
> *java.lang.IllegalArgumentException: Invalid range:
> IntegrationTestBigLinkedList,,1594350463222.8f89e01a5245e79946e22d8a8ab4698b.
> >
> IntegrationTestBigLinkedList,\x10\x02J\xA1,1594349535271.be24dc276f686e6dcc7fb9d3f91c8387.
> {code}
> Seems basic case of wrong comparator. Below passes if I use the meta
> comparator
> {code:java}
> @Test
> public void testBinaryKeys() throws Exception {
> Set set = new TreeSet<>(CellComparatorImpl.COMPARATOR);
> final byte [] fam = Bytes.toBytes("col");
> final byte [] qf = Bytes.toBytes("umn");
> final byte [] nb = new byte[0];
> Cell [] keys = {
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,\u\u,2"), fam, qf, 2,
> nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,\u0001,3"), fam, qf, 3, nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,,1"), fam, qf, 1, nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,\u1000,5"), fam, qf, 5, nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,a,4"), fam, qf, 4, nb)),
> createByteBufferKeyValueFromKeyValue(
> new KeyValue(Bytes.toBytes("a,a,0"), fam, qf, 0, nb)),
> };
> // Add to set with bad comparator
> Collections.addAll(set, keys);
> // This will output the keys incorrectly.
> boolean assertion = false;
> int count = 0;
> try {
> for (Cell k: set) {
> assertTrue("count=" + count + ", " + k.toString(), count++ ==
> k.getTimestamp());
> }
> } catch (AssertionError e) {
>
[GitHub] [hbase] Apache-HBase commented on pull request #3347: HBASE-25955 Setting NAMESPACES when adding a replication peer doesn't…
Apache-HBase commented on pull request #3347: URL: https://github.com/apache/hbase/pull/3347#issuecomment-852036886 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 59s | master passed | | +1 :green_heart: | compile | 3m 11s | master passed | | +1 :green_heart: | checkstyle | 1m 4s | master passed | | +1 :green_heart: | spotbugs | 2m 7s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 39s | the patch passed | | +1 :green_heart: | compile | 3m 8s | the patch passed | | +1 :green_heart: | javac | 3m 8s | the patch passed | | -0 :warning: | checkstyle | 1m 3s | hbase-server: The patch generated 4 new + 3 unchanged - 0 fixed = 7 total (was 3) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 6s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 46m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3347/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3347 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 1fdd50d910f4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b04c3c7786 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3347/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 95 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3347/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3347: HBASE-25955 Setting NAMESPACES when adding a replication peer doesn't…
Apache-HBase commented on pull request #3347: URL: https://github.com/apache/hbase/pull/3347#issuecomment-852033219 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 31s | master passed | | +1 :green_heart: | compile | 1m 11s | master passed | | +1 :green_heart: | shadedjars | 8m 12s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 15s | the patch passed | | +1 :green_heart: | compile | 1m 14s | the patch passed | | +1 :green_heart: | javac | 1m 14s | the patch passed | | +1 :green_heart: | shadedjars | 8m 13s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 10m 29s | hbase-server in the patch failed. | | | | 41m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3347/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3347 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3cc9f126ce05 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b04c3c7786 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3347/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3347/1/testReport/ | | Max. process+thread count | 762 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3347/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3347: HBASE-25955 Setting NAMESPACES when adding a replication peer doesn't…
Apache-HBase commented on pull request #3347: URL: https://github.com/apache/hbase/pull/3347#issuecomment-852031351 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 55s | master passed | | +1 :green_heart: | compile | 0m 59s | master passed | | +1 :green_heart: | shadedjars | 8m 18s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | the patch passed | | +1 :green_heart: | compile | 1m 1s | the patch passed | | +1 :green_heart: | javac | 1m 1s | the patch passed | | +1 :green_heart: | shadedjars | 8m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 8m 47s | hbase-server in the patch failed. | | | | 38m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3347/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3347 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9243bc609785 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b04c3c7786 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3347/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3347/1/testReport/ | | Max. process+thread count | 844 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3347/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25951) affect version is editable
[ https://issues.apache.org/jira/browse/HBASE-25951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-25951. --- Resolution: Invalid > affect version is editable > -- > > Key: HBASE-25951 > URL: https://issues.apache.org/jira/browse/HBASE-25951 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.12 >Reporter: Ankur Tagra >Priority: Trivial > Fix For: 1.4.14 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25391) Flush directly into data directory, skip rename when committing flush
[
https://issues.apache.org/jira/browse/HBASE-25391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil reassigned HBASE-25391:
Assignee: Wellington Chevreuil
> Flush directly into data directory, skip rename when committing flush
> -
>
> Key: HBASE-25391
> URL: https://issues.apache.org/jira/browse/HBASE-25391
> Project: HBase
> Issue Type: Sub-task
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Wellington Chevreuil
>Priority: Major
>
> {color:#00}When flushing memstore snapshot to HFile, we write it directly
> to the data directory.{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25955) Setting NAMESPACES when adding a replication peer doesn't have any effect
[ https://issues.apache.org/jira/browse/HBASE-25955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-25955: - Affects Version/s: 2.5.0 2.3.5 2.4.3 > Setting NAMESPACES when adding a replication peer doesn't have any effect > -- > > Key: HBASE-25955 > URL: https://issues.apache.org/jira/browse/HBASE-25955 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 3.0.0-alpha-1, 2.5.0, 2.3.5, 2.4.3 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > > A common ask from our customers is for an easy way to enable replication for > all tables within a namespace, but we had noticed that setting *NAMESPACES* > option when adding a peer has no effect on enabling replication for the > defined namespaces tables, it still replicates only entries where the *CF > REPLICATION_SCOPE* is set to '1'. > The problem is due to the chaining of *ScopeWALEntryFilter* and > *NamespaceTableCfWALEntryFilter* together. This is currently done on > *BaseReplicationEndpoint* class, but these two filters are not compatible, as > *ScopeWALEntryFilter.filterCells* logic filters out cells from CFs where > REPLICATION_SCOPE property is set to 0, before > *NamespaceTableCfWALEntryFilter.filterCells* can have a chance to apply it's > own logic, based on namespace definition. > This PR changes the logic described above, making *ScopeWALEntryFilter* and > *NamespaceTableCfWALEntryFilter* mutual exclusive, allowing for entries from > all tables within the defined namespaces to be replicated without needing to > have *REPLICATION_SCOPE* set to true. -- This message was sent by Atlassian Jira (v8.3.4#803005)
