[GitHub] [hbase] Apache-HBase commented on pull request #4136: HBASE-26245 Store region server list in master local region
Apache-HBase commented on pull request #4136: URL: https://github.com/apache/hbase/pull/4136#issuecomment-1051706296 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 12s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 23s | master passed | | +1 :green_heart: | compile | 1m 56s | master passed | | +1 :green_heart: | checkstyle | 0m 37s | master passed | | +1 :green_heart: | spotbugs | 1m 15s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 23s | the patch passed | | +1 :green_heart: | compile | 1m 57s | the patch passed | | +1 :green_heart: | javac | 1m 57s | the patch passed | | +1 :green_heart: | checkstyle | 0m 37s | hbase-server: The patch generated 0 new + 93 unchanged - 3 fixed = 93 total (was 96) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 27s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotbugs | 1m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 9s | The patch does not generate ASF License warnings. | | | | 30m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4136/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4136 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 6b31510abe0e 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5dc663ea38 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 60 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4136/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-26774) [hbase-thirdparty] Set version as 4.1.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/HBASE-26774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26774. --- Fix Version/s: (was: thirdparty-4.1.0) Assignee: (was: Duo Zhang) Resolution: Duplicate > [hbase-thirdparty] Set version as 4.1.0-SNAPSHOT > > > Key: HBASE-26774 > URL: https://issues.apache.org/jira/browse/HBASE-26774 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26591) [hbase-thirdparty] Make a 4.0.1 release for hbase-thirdparty
[
https://issues.apache.org/jira/browse/HBASE-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang resolved HBASE-26591.
---
Resolution: Fixed
> [hbase-thirdparty] Make a 4.0.1 release for hbase-thirdparty
>
>
> Key: HBASE-26591
> URL: https://issues.apache.org/jira/browse/HBASE-26591
> Project: HBase
> Issue Type: Umbrella
> Components: community, thirdparty
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Copy the content in mailing list here
> {noformat}
> Oh, when backporting to branch-2, I found that there is a serious problem
> with this release.
> We do not include the proto files in protobuf-java jar, I think this is a
> side effect of HBASE-26496.
> And on the master branch, hbase-shaded-protobuf also pulls in protobuf-java
> 3.19.1 dependency so everything is fine. But for branch-2, we will need to
> depend on protobuf 2.5.0 then the compilation is broken.
> Anyway, we should include the proto files in hbase-shaded-protobuf, and also
> we should not pull in the original protobuf-java dependency.
> Let me fix this and do a 4.0.1 release.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26774) [hbase-thirdparty] Set version as 4.1.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/HBASE-26774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26774: -- Component/s: thirdparty > [hbase-thirdparty] Set version as 4.1.0-SNAPSHOT > > > Key: HBASE-26774 > URL: https://issues.apache.org/jira/browse/HBASE-26774 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HBASE-26774) [hbase-thirdparty] Set version as 4.1.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/HBASE-26774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-26774: - Assignee: Duo Zhang > [hbase-thirdparty] Set version as 4.1.0-SNAPSHOT > > > Key: HBASE-26774 > URL: https://issues.apache.org/jira/browse/HBASE-26774 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: thirdparty-4.1.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26774) [hbase-thirdparty] Set version as 4.1.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/HBASE-26774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26774: -- Fix Version/s: thirdparty-4.1.0 > [hbase-thirdparty] Set version as 4.1.0-SNAPSHOT > > > Key: HBASE-26774 > URL: https://issues.apache.org/jira/browse/HBASE-26774 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Priority: Major > Fix For: thirdparty-4.1.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26774) [hbase-thirdparty] Set version as 4.1.0-SNAPSHOT
Duo Zhang created HBASE-26774: - Summary: [hbase-thirdparty] Set version as 4.1.0-SNAPSHOT Key: HBASE-26774 URL: https://issues.apache.org/jira/browse/HBASE-26774 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Reopened] (HBASE-26591) [hbase-thirdparty] Make a 4.0.1 release for hbase-thirdparty
[
https://issues.apache.org/jira/browse/HBASE-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang reopened HBASE-26591:
---
Forgot to bump the version in pom...
> [hbase-thirdparty] Make a 4.0.1 release for hbase-thirdparty
>
>
> Key: HBASE-26591
> URL: https://issues.apache.org/jira/browse/HBASE-26591
> Project: HBase
> Issue Type: Umbrella
> Components: community, thirdparty
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Copy the content in mailing list here
> {noformat}
> Oh, when backporting to branch-2, I found that there is a serious problem
> with this release.
> We do not include the proto files in protobuf-java jar, I think this is a
> side effect of HBASE-26496.
> And on the master branch, hbase-shaded-protobuf also pulls in protobuf-java
> 3.19.1 dependency so everything is fine. But for branch-2, we will need to
> depend on protobuf 2.5.0 then the compilation is broken.
> Anyway, we should include the proto files in hbase-shaded-protobuf, and also
> we should not pull in the original protobuf-java dependency.
> Let me fix this and do a 4.0.1 release.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] Apache9 commented on pull request #4119: HBASE-26762: Un-Deprecate and improve documentation for Scan#setRowPrefixFilter
Apache9 commented on pull request #4119: URL: https://github.com/apache/hbase/pull/4119#issuecomment-1051649220 Oh, one minor thing, I think we could also cherry-pick this change to branch-2 and branch-2.5, so let's change the javadoc from 'since 3.0.0' to 'since 2.5.0'? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4115: Introduce a Snapshot Procedure
Apache-HBase commented on pull request #4115: URL: https://github.com/apache/hbase/pull/4115#issuecomment-1051649210 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 4s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 22s | master passed | | +1 :green_heart: | compile | 3m 21s | master passed | | +1 :green_heart: | checkstyle | 1m 1s | master passed | | +1 :green_heart: | spotbugs | 4m 26s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 24s | the patch passed | | +1 :green_heart: | compile | 3m 24s | the patch passed | | +1 :green_heart: | cc | 3m 24s | the patch passed | | +1 :green_heart: | javac | 3m 24s | the patch passed | | -0 :warning: | checkstyle | 0m 37s | hbase-server: The patch generated 1 new + 105 unchanged - 3 fixed = 106 total (was 108) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 19s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | hbaseprotoc | 1m 19s | the patch passed | | +1 :green_heart: | spotbugs | 4m 46s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 42m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4115/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4115 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool | | uname | Linux ddf41b59447b 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5dc663ea38 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4115/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 60 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4115/5/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26773) Introduce a hbase-thirdparty-unsafe module to remove the direct references of Unsafe in our main code base
[ https://issues.apache.org/jira/browse/HBASE-26773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26773: -- Summary: Introduce a hbase-thirdparty-unsafe module to remove the direct references of Unsafe in our main code base (was: Introduce a hbase-thirdparty-unsafe module to remove the direct references of Unsafe in our main repo) > Introduce a hbase-thirdparty-unsafe module to remove the direct references of > Unsafe in our main code base > -- > > Key: HBASE-26773 > URL: https://issues.apache.org/jira/browse/HBASE-26773 > Project: HBase > Issue Type: Improvement > Components: compatibility, util >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > For java 9+, sun.misc.Unsafe is in the module jdk.unsupported. When compiling > with java 11 --release 8, these symbols will not be exported and lead to > compile errors. > There is a jdk issue > https://bugs.openjdk.java.net/browse/JDK-8214165 > But seems the jdk team does not want to fix it. So if we want to move up to > java 11 but still keep the java 8 compatibility, a possible choice is to > remove the direct reference of sun.misc.Unsafe so at compile time there will > be no problem, and at runtime we export the jdk.unsupported so there will be > no problem too. > For more details, please see the discussion of this thread on the dev mailing > list. > https://lists.apache.org/thread/w5lrxkhswlonj09xf9hcwgvck3nsjdfx -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26773) Introduce a hbase-thirdparty-unsafe module to remove the direct references of Unsafe in our main repo
[ https://issues.apache.org/jira/browse/HBASE-26773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26773: -- Summary: Introduce a hbase-thirdparty-unsafe module to remove the direct references of Unsafe in our main repo (was: Use Netty PlatformDependent to remove the direct references of Unsafe) > Introduce a hbase-thirdparty-unsafe module to remove the direct references of > Unsafe in our main repo > - > > Key: HBASE-26773 > URL: https://issues.apache.org/jira/browse/HBASE-26773 > Project: HBase > Issue Type: Improvement > Components: compatibility, util >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > For java 9+, sun.misc.Unsafe is in the module jdk.unsupported. When compiling > with java 11 --release 8, these symbols will not be exported and lead to > compile errors. > There is a jdk issue > https://bugs.openjdk.java.net/browse/JDK-8214165 > But seems the jdk team does not want to fix it. So if we want to move up to > java 11 but still keep the java 8 compatibility, a possible choice is to > remove the direct reference of sun.misc.Unsafe so at compile time there will > be no problem, and at runtime we export the jdk.unsupported so there will be > no problem too. > For more details, please see the discussion of this thread on the dev mailing > list. > https://lists.apache.org/thread/w5lrxkhswlonj09xf9hcwgvck3nsjdfx -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26773) Introduce a hbase-thirdparty-unsafe module to remove the direct references of Unsafe in our main code base
[ https://issues.apache.org/jira/browse/HBASE-26773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26773: -- Component/s: thirdparty (was: util) > Introduce a hbase-thirdparty-unsafe module to remove the direct references of > Unsafe in our main code base > -- > > Key: HBASE-26773 > URL: https://issues.apache.org/jira/browse/HBASE-26773 > Project: HBase > Issue Type: Improvement > Components: compatibility, thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > For java 9+, sun.misc.Unsafe is in the module jdk.unsupported. When compiling > with java 11 --release 8, these symbols will not be exported and lead to > compile errors. > There is a jdk issue > https://bugs.openjdk.java.net/browse/JDK-8214165 > But seems the jdk team does not want to fix it. So if we want to move up to > java 11 but still keep the java 8 compatibility, a possible choice is to > remove the direct reference of sun.misc.Unsafe so at compile time there will > be no problem, and at runtime we export the jdk.unsupported so there will be > no problem too. > For more details, please see the discussion of this thread on the dev mailing > list. > https://lists.apache.org/thread/w5lrxkhswlonj09xf9hcwgvck3nsjdfx -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase-connectors] busbey commented on pull request #93: HBASE-22338 LICENSE file only contains Apache 2.0
busbey commented on pull request #93:
URL: https://github.com/apache/hbase-connectors/pull/93#issuecomment-1051639162
we're missing the part of the main project tooling that checks for problems
with the generated license:
```
(base) sbusbey@seans-mbp hbase-connectors-1.0.1-SNAPSHOT % grep -B 2
"^ERROR:" LICENSE.txt
This product includes Streaming API for XML licensed under the GNU General
Public Library.
ERROR: Please check this License for acceptability here:
```
stax-api is dual licensed GPL and CDDL. We'll also need to specify that
we're picking CDDL.
The generated NOTICE is missing the build year:
```
(base) sbusbey@seans-mbp hbase-connectors-1.0.1-SNAPSHOT % head -n 3
NOTICE.txt
Apache HBase Connectors - Assembly
Copyright 2018-${copyright-end-year} The Apache Software Foundation
```
It looks like this is because we need to activate the
`buildnumber-maven-plugin` in the `hbase-connectors-assembly` module. we only
define the version/configs in plugin management on the top level pom; we never
actually tell the plugin to run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26773) Use Netty PlatformDependent to remove the direct references of Unsafe
[ https://issues.apache.org/jira/browse/HBASE-26773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17498398#comment-17498398 ] Duo Zhang commented on HBASE-26773: --- The result of the first try is not very good: 1. In ClassSize, we need to use ARRAY_OBJECT_INDEX_SCALE and ARRAY_BYTE_INDEX_SCALE, which are not exposed by netty's PlatformDependent. 2. In Bytes and ByteBufferUtils, we uses the raw Unsafe instead of UnsafeAccess. I think this is for performance as we will calculate the offset on the fly, and then use it as the final offset directly. Use UnsafeAccess will introduce an extra calculation so may impact performance. So now, I prefer we provide a hbase-thirdparty-unsafe module in hbase-thirdparty, to expose what we want with Unsafe, and compile it with jdk8, to keep compatibility. Let me change the title. > Use Netty PlatformDependent to remove the direct references of Unsafe > - > > Key: HBASE-26773 > URL: https://issues.apache.org/jira/browse/HBASE-26773 > Project: HBase > Issue Type: Improvement > Components: compatibility, util >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > For java 9+, sun.misc.Unsafe is in the module jdk.unsupported. When compiling > with java 11 --release 8, these symbols will not be exported and lead to > compile errors. > There is a jdk issue > https://bugs.openjdk.java.net/browse/JDK-8214165 > But seems the jdk team does not want to fix it. So if we want to move up to > java 11 but still keep the java 8 compatibility, a possible choice is to > remove the direct reference of sun.misc.Unsafe so at compile time there will > be no problem, and at runtime we export the jdk.unsupported so there will be > no problem too. > For more details, please see the discussion of this thread on the dev mailing > list. > https://lists.apache.org/thread/w5lrxkhswlonj09xf9hcwgvck3nsjdfx -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] frostruan commented on a change in pull request #4115: Introduce a Snapshot Procedure
frostruan commented on a change in pull request #4115:
URL: https://github.com/apache/hbase/pull/4115#discussion_r814902123
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
##
@@ -1214,11 +1358,27 @@ public void initialize(MasterServices master,
MetricsMaster metricsMaster) throw
this.coordinator = new ProcedureCoordinator(comms, tpool, timeoutMillis,
wakeFrequency);
this.executorService = master.getExecutorService();
+this.verifyWorkerAssigner = new WorkerAssigner(master,
+ conf.getInt("hbase.snapshot.verify.task.max", 3),
+ new ProcedureEvent<>("snapshot-verify-worker-assigning"));
+restoreUnfinishedSnapshotProcedure();
+restoreWorkers();
resetTempDir();
snapshotHandlerChoreCleanerTask =
scheduleThreadPool.scheduleAtFixedRate(this::cleanupSentinels, 10, 10,
TimeUnit.SECONDS);
}
+ private void restoreUnfinishedSnapshotProcedure() {
Review comment:
Yes. I also think that it is the most elegant way to restore the procid
in afterReplay method, but because of the startup order of master, there will
be a NPE problem if we do that.
We can see the startup order in HMaster#finishActiveMasterInitialization().
1. we replay procedures in the method HMaster#createProcedureExecutor()
2. we create the snapshot manager instance in the method
HMaster#initializeZKBasedSystemTrackers()
3. we start the procedure executor in the method
HMaster#startServiceThreads()
Since these three methods are executed in the same thread, so I think it's
safe here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] frostruan commented on a change in pull request #4115: Introduce a Snapshot Procedure
frostruan commented on a change in pull request #4115:
URL: https://github.com/apache/hbase/pull/4115#discussion_r814902668
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
##
@@ -1264,4 +1424,55 @@ private SnapshotDescription
toSnapshotDescription(ProcedureDescription desc)
builder.setType(SnapshotDescription.Type.FLUSH);
return builder.build();
}
+
+ public void registerSnapshotProcedure(SnapshotDescription snapshot, long
procId) {
+snapshotToProcIdMap.put(snapshot, procId);
+LOG.debug("register snapshot={}, snapshot procedure id = {}",
+ ClientSnapshotDescriptionUtils.toString(snapshot), procId);
+ }
+
+ public void unregisterSnapshotProcedure(SnapshotDescription snapshot, long
procId) {
+snapshotToProcIdMap.remove(snapshot, procId);
+LOG.debug("unregister snapshot={}, snapshot procedure id = {}",
+ ClientSnapshotDescriptionUtils.toString(snapshot), procId);
+ }
+
+ public boolean snapshotProcedureEnabled() {
+return master.getConfiguration()
+ .getBoolean(SNAPSHOT_PROCEDURE_ENABLED,
SNAPSHOT_PROCEDURE_ENABLED_DEFAULT);
+ }
+
+ public ServerName acquireSnapshotVerifyWorker(SnapshotVerifyProcedure
procedure)
+ throws ProcedureSuspendedException {
+Optional worker = verifyWorkerAssigner.acquire();
+if (worker.isPresent()) {
+ LOG.debug("{} Acquired verify snapshot worker={}", procedure,
worker.get());
+ return worker.get();
+}
+verifyWorkerAssigner.suspend(procedure);
+throw new ProcedureSuspendedException();
+ }
+
+ public void releaseSnapshotVerifyWorker(SnapshotVerifyProcedure procedure,
+ ServerName worker, MasterProcedureScheduler scheduler) {
+LOG.debug("{} Release verify snapshot worker={}", procedure, worker);
+verifyWorkerAssigner.release(worker);
+verifyWorkerAssigner.wake(scheduler);
+ }
+
+ private void restoreWorkers() {
+master.getMasterProcedureExecutor().getActiveProceduresNoCopy().stream()
+ .filter(p -> p instanceof SnapshotVerifyProcedure)
+ .map(p -> (SnapshotVerifyProcedure) p)
+ .filter(p -> !p.isFinished())
+ .filter(p -> p.getServerName() != null)
+ .forEach(p -> {
+verifyWorkerAssigner.addUsedWorker(p.getServerName());
Review comment:
I think it's harmless that we have dead servers in the WorkAssigner
based on this consideration:
1. If the remote server is dead, there will be a
FailedRemoteDispatchException thrown when we dispatch remote procedures. The
procedure-v2 framework is designed to be able to handle this. The upper layer
procedure can notice that and reschedule a new procedure, or the procedure can
retry itself.
2. When acquire a new worker from WorkAssigner, we will always choose from
the online server list. The dead server will not be able to become a candidate
again.
3. If there are too many dead servers in the WorkerAssigner, we can restart
the master to clear it. This is a very lightweight operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-26773) Use Netty PlatformDependent to remove the direct references of Unsafe
[ https://issues.apache.org/jira/browse/HBASE-26773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26773: -- Component/s: compatibility > Use Netty PlatformDependent to remove the direct references of Unsafe > - > > Key: HBASE-26773 > URL: https://issues.apache.org/jira/browse/HBASE-26773 > Project: HBase > Issue Type: Improvement > Components: compatibility, util >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > For java 9+, sun.misc.Unsafe is in the module jdk.unsupported. When compiling > with java 11 --release 8, these symbols will not be exported and lead to > compile errors. > There is a jdk issue > https://bugs.openjdk.java.net/browse/JDK-8214165 > But seems the jdk team does not want to fix it. So if we want to move up to > java 11 but still keep the java 8 compatibility, a possible choice is to > remove the direct reference of sun.misc.Unsafe so at compile time there will > be no problem, and at runtime we export the jdk.unsupported so there will be > no problem too. > For more details, please see the discussion of this thread on the dev mailing > list. > https://lists.apache.org/thread/w5lrxkhswlonj09xf9hcwgvck3nsjdfx -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HBASE-26773) Use Netty PlatformDependent to remove the direct references of Unsafe
[ https://issues.apache.org/jira/browse/HBASE-26773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-26773: - Assignee: Duo Zhang > Use Netty PlatformDependent to remove the direct references of Unsafe > - > > Key: HBASE-26773 > URL: https://issues.apache.org/jira/browse/HBASE-26773 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > For java 9+, sun.misc.Unsafe is in the module jdk.unsupported. When compiling > with java 11 --release 8, these symbols will not be exported and lead to > compile errors. > There is a jdk issue > https://bugs.openjdk.java.net/browse/JDK-8214165 > But seems the jdk team does not want to fix it. So if we want to move up to > java 11 but still keep the java 8 compatibility, a possible choice is to > remove the direct reference of sun.misc.Unsafe so at compile time there will > be no problem, and at runtime we export the jdk.unsupported so there will be > no problem too. > For more details, please see the discussion of this thread on the dev mailing > list. > https://lists.apache.org/thread/w5lrxkhswlonj09xf9hcwgvck3nsjdfx -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26773) Use Netty PlatformDependent to remove the direct references of Unsafe
[ https://issues.apache.org/jira/browse/HBASE-26773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26773: -- Component/s: util > Use Netty PlatformDependent to remove the direct references of Unsafe > - > > Key: HBASE-26773 > URL: https://issues.apache.org/jira/browse/HBASE-26773 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > For java 9+, sun.misc.Unsafe is in the module jdk.unsupported. When compiling > with java 11 --release 8, these symbols will not be exported and lead to > compile errors. > There is a jdk issue > https://bugs.openjdk.java.net/browse/JDK-8214165 > But seems the jdk team does not want to fix it. So if we want to move up to > java 11 but still keep the java 8 compatibility, a possible choice is to > remove the direct reference of sun.misc.Unsafe so at compile time there will > be no problem, and at runtime we export the jdk.unsupported so there will be > no problem too. > For more details, please see the discussion of this thread on the dev mailing > list. > https://lists.apache.org/thread/w5lrxkhswlonj09xf9hcwgvck3nsjdfx -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work started] (HBASE-26773) Use Netty PlatformDependent to remove the direct references of Unsafe
[ https://issues.apache.org/jira/browse/HBASE-26773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-26773 started by Duo Zhang. - > Use Netty PlatformDependent to remove the direct references of Unsafe > - > > Key: HBASE-26773 > URL: https://issues.apache.org/jira/browse/HBASE-26773 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > For java 9+, sun.misc.Unsafe is in the module jdk.unsupported. When compiling > with java 11 --release 8, these symbols will not be exported and lead to > compile errors. > There is a jdk issue > https://bugs.openjdk.java.net/browse/JDK-8214165 > But seems the jdk team does not want to fix it. So if we want to move up to > java 11 but still keep the java 8 compatibility, a possible choice is to > remove the direct reference of sun.misc.Unsafe so at compile time there will > be no problem, and at runtime we export the jdk.unsupported so there will be > no problem too. > For more details, please see the discussion of this thread on the dev mailing > list. > https://lists.apache.org/thread/w5lrxkhswlonj09xf9hcwgvck3nsjdfx -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26773) Use Netty PlatformDependent to remove the direct references of Unsafe
Duo Zhang created HBASE-26773: - Summary: Use Netty PlatformDependent to remove the direct references of Unsafe Key: HBASE-26773 URL: https://issues.apache.org/jira/browse/HBASE-26773 Project: HBase Issue Type: Improvement Reporter: Duo Zhang For java 9+, sun.misc.Unsafe is in the module jdk.unsupported. When compiling with java 11 --release 8, these symbols will not be exported and lead to compile errors. There is a jdk issue https://bugs.openjdk.java.net/browse/JDK-8214165 But seems the jdk team does not want to fix it. So if we want to move up to java 11 but still keep the java 8 compatibility, a possible choice is to remove the direct reference of sun.misc.Unsafe so at compile time there will be no problem, and at runtime we export the jdk.unsupported so there will be no problem too. For more details, please see the discussion of this thread on the dev mailing list. https://lists.apache.org/thread/w5lrxkhswlonj09xf9hcwgvck3nsjdfx -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #4125: HBASE-26666 Add native TLS encryption support to RPC server/client
Apache-HBase commented on pull request #4125: URL: https://github.com/apache/hbase/pull/4125#issuecomment-1051313422 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 23s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 22s | master passed | | +1 :green_heart: | compile | 5m 40s | master passed | | +1 :green_heart: | checkstyle | 1m 9s | master passed | | +1 :green_heart: | spotbugs | 9m 26s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 23s | the patch passed | | +1 :green_heart: | compile | 5m 43s | the patch passed | | -0 :warning: | javac | 5m 43s | root generated 22 new + 1625 unchanged - 0 fixed = 1647 total (was 1625) | | +1 :green_heart: | checkstyle | 1m 8s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 11m 11s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotbugs | 10m 15s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. | | | | 56m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4125 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile xml | | uname | Linux 7b57d1c694b0 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5dc663ea38 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/3/artifact/yetus-general-check/output/diff-compile-javac-root.txt | | Max. process+thread count | 88 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4125: HBASE-26666 Add native TLS encryption support to RPC server/client
Apache-HBase commented on pull request #4125: URL: https://github.com/apache/hbase/pull/4125#issuecomment-1051298102 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 10s | master passed | | +1 :green_heart: | compile | 1m 59s | master passed | | +1 :green_heart: | shadedjars | 5m 23s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 40s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 56s | the patch passed | | +1 :green_heart: | compile | 2m 0s | the patch passed | | +1 :green_heart: | javac | 2m 0s | the patch passed | | +1 :green_heart: | shadedjars | 5m 21s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 41s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 5m 45s | root in the patch failed. | | | | 34m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4125 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 92a6207747c1 5.4.0-1047-aws #49~18.04.1-Ubuntu SMP Wed Apr 28 23:08:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5dc663ea38 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/3/testReport/ | | Max. process+thread count | 353 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4125: HBASE-26666 Add native TLS encryption support to RPC server/client
Apache-HBase commented on pull request #4125: URL: https://github.com/apache/hbase/pull/4125#issuecomment-1051295047 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 22s | master passed | | +1 :green_heart: | compile | 1m 43s | master passed | | +1 :green_heart: | shadedjars | 4m 37s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 19s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 23s | the patch passed | | +1 :green_heart: | compile | 1m 42s | the patch passed | | +1 :green_heart: | javac | 1m 42s | the patch passed | | +1 :green_heart: | shadedjars | 4m 40s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 17s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 5m 15s | root in the patch failed. | | | | 29m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4125 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 81c9edbd6b41 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5dc663ea38 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/3/testReport/ | | Max. process+thread count | 334 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4127: HBASE-26768 Avoid unnecessary replication suspending in RegionReplica…
Apache-HBase commented on pull request #4127: URL: https://github.com/apache/hbase/pull/4127#issuecomment-1051283946 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 27s | master passed | | +1 :green_heart: | compile | 0m 38s | master passed | | +1 :green_heart: | shadedjars | 4m 42s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 29s | the patch passed | | +1 :green_heart: | compile | 0m 39s | the patch passed | | +1 :green_heart: | javac | 0m 39s | the patch passed | | +1 :green_heart: | shadedjars | 4m 43s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 353m 39s | hbase-server in the patch passed. | | | | 372m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4127/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4127 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux dbfc7d21bb40 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5dc663ea38 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4127/4/testReport/ | | Max. process+thread count | 1701 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4127/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4127: HBASE-26768 Avoid unnecessary replication suspending in RegionReplica…
Apache-HBase commented on pull request #4127: URL: https://github.com/apache/hbase/pull/4127#issuecomment-1051267547 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 2s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 49s | master passed | | +1 :green_heart: | compile | 0m 46s | master passed | | +1 :green_heart: | shadedjars | 4m 44s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 50s | the patch passed | | +1 :green_heart: | compile | 0m 47s | the patch passed | | +1 :green_heart: | javac | 0m 47s | the patch passed | | +1 :green_heart: | shadedjars | 4m 41s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 331m 21s | hbase-server in the patch passed. | | | | 351m 16s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4127/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4127 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ad30a5a66fa7 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5dc663ea38 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4127/4/testReport/ | | Max. process+thread count | 1780 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4127/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-26707) Reduce number of renames during bulkload
[ https://issues.apache.org/jira/browse/HBASE-26707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil resolved HBASE-26707. -- Resolution: Fixed Thanks for the contribution, [~bszabolcs] ! > Reduce number of renames during bulkload > > > Key: HBASE-26707 > URL: https://issues.apache.org/jira/browse/HBASE-26707 > Project: HBase > Issue Type: Sub-task >Reporter: Szabolcs Bukros >Assignee: Szabolcs Bukros >Priority: Major > > Make sure we only do a single rename operation during bulkload when > StoreEngine does not require the the use of tmp directories. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26707) Reduce number of renames during bulkload
[ https://issues.apache.org/jira/browse/HBASE-26707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17498292#comment-17498292 ] Wellington Chevreuil commented on HBASE-26707: -- Merged second PR to branch-2. > Reduce number of renames during bulkload > > > Key: HBASE-26707 > URL: https://issues.apache.org/jira/browse/HBASE-26707 > Project: HBase > Issue Type: Sub-task >Reporter: Szabolcs Bukros >Assignee: Szabolcs Bukros >Priority: Major > > Make sure we only do a single rename operation during bulkload when > StoreEngine does not require the the use of tmp directories. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] wchevreuil merged pull request #4122: HBASE-26707: Reduce number of renames during bulkload (#4066)
wchevreuil merged pull request #4122: URL: https://github.com/apache/hbase/pull/4122 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #4137: HBASE-26772 Shell suspended in background
busbey commented on pull request #4137: URL: https://github.com/apache/hbase/pull/4137#issuecomment-1051149855 Change looks reasonable to me. Holding off on an approval until I can test run things locally. Feel free to proceed if someone else comes along before I get to finish. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26765) Minor refactor of async scanning code
[
https://issues.apache.org/jira/browse/HBASE-26765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17498271#comment-17498271
]
Hudson commented on HBASE-26765:
Results for branch branch-2.5
[build #51 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/51/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/51/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/51/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/51/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/51/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Minor refactor of async scanning code
> -
>
> Key: HBASE-26765
> URL: https://issues.apache.org/jira/browse/HBASE-26765
> Project: HBase
> Issue Type: Sub-task
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
>
> Some small changes to internal/private classes implementing async scan will
> make the code easier to read and reason about.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26691) Replacing log4j with reload4j for branch-2.x
[ https://issues.apache.org/jira/browse/HBASE-26691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17498266#comment-17498266 ] Andrew Kyle Purtell commented on HBASE-26691: - Approved the PR with some minor comments. Apologies for the delay in review. > Replacing log4j with reload4j for branch-2.x > > > Key: HBASE-26691 > URL: https://issues.apache.org/jira/browse/HBASE-26691 > Project: HBase > Issue Type: Task > Components: logging >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.5.0, 2.6.0 > > > There are several new CVEs for log4j1 now. > As it is not suitable to upgrade to log4j2 for 2.x releases, let's replace > the log4j1 dependencies with reload4j. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26691) Replacing log4j with reload4j for branch-2.x
[ https://issues.apache.org/jira/browse/HBASE-26691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-26691: Fix Version/s: 2.6.0 (was: 2.4.10) Status: Patch Available (was: Open) > Replacing log4j with reload4j for branch-2.x > > > Key: HBASE-26691 > URL: https://issues.apache.org/jira/browse/HBASE-26691 > Project: HBase > Issue Type: Task > Components: logging >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.5.0, 2.6.0 > > > There are several new CVEs for log4j1 now. > As it is not suitable to upgrade to log4j2 for 2.x releases, let's replace > the log4j1 dependencies with reload4j. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] apurtell commented on a change in pull request #4050: HBASE-26691 Replacing log4j with reload4j for branch-2.x
apurtell commented on a change in pull request #4050:
URL: https://github.com/apache/hbase/pull/4050#discussion_r815018868
##
File path: hbase-mapreduce/pom.xml
##
@@ -322,100 +340,6 @@
-
Review comment:
Oh, I see, some extra refactoring?
##
File path: hbase-mapreduce/pom.xml
##
@@ -220,6 +220,24 @@
compile
true
+
Review comment:
Is this related?
##
File path: hbase-shaded/hbase-shaded-testing-util/pom.xml
##
@@ -37,43 +37,8 @@
org.apache.hadoop
hadoop-common
-${hadoop.version}
test-jar
compile
-
Review comment:
I don't have a strong opinion about this so will approve the PR but it
would be better to stick with just the changes that are implied by the
description _[HBASE-26691 Replacing log4j with reload4j for branch-2.x]_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26765) Minor refactor of async scanning code
[
https://issues.apache.org/jira/browse/HBASE-26765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17498258#comment-17498258
]
Hudson commented on HBASE-26765:
Results for branch master
[build #521 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/521/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/521/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/521/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/521/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Minor refactor of async scanning code
> -
>
> Key: HBASE-26765
> URL: https://issues.apache.org/jira/browse/HBASE-26765
> Project: HBase
> Issue Type: Sub-task
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
>
> Some small changes to internal/private classes implementing async scan will
> make the code easier to read and reason about.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] frostruan commented on a change in pull request #4115: Introduce a Snapshot Procedure
frostruan commented on a change in pull request #4115:
URL: https://github.com/apache/hbase/pull/4115#discussion_r814901349
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
##
@@ -418,22 +471,33 @@ public boolean isSnapshotDone(SnapshotDescription
expected) throws IOException {
* Currently we have a limitation only allowing a single snapshot per table
at a time. Also we
* don't allow snapshot with the same name.
* @param snapshot description of the snapshot being checked.
+ * @param checkTable check if the table is already taking a snapshot.
Review comment:
This parameter is used for different snapshot backends. For
zk-coordinated snapshot, we need to check if another zk-coordinated snapshot on
the same table is in progress, but for the snapshot procedure, it is
unnecessary.
The basic rule is that,
1. Two zk-coordinated snapshots on the same table cannot be taken at the
same time.
2. Two snapshot procedures on the same table can run at the same time.
3. A zk-coordinated snapshot and a proc-v2 based procedure on the same table
can be taken at the same time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] frostruan commented on a change in pull request #4115: Introduce a Snapshot Procedure
frostruan commented on a change in pull request #4115:
URL: https://github.com/apache/hbase/pull/4115#discussion_r814902123
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
##
@@ -1214,11 +1358,27 @@ public void initialize(MasterServices master,
MetricsMaster metricsMaster) throw
this.coordinator = new ProcedureCoordinator(comms, tpool, timeoutMillis,
wakeFrequency);
this.executorService = master.getExecutorService();
+this.verifyWorkerAssigner = new WorkerAssigner(master,
+ conf.getInt("hbase.snapshot.verify.task.max", 3),
+ new ProcedureEvent<>("snapshot-verify-worker-assigning"));
+restoreUnfinishedSnapshotProcedure();
+restoreWorkers();
resetTempDir();
snapshotHandlerChoreCleanerTask =
scheduleThreadPool.scheduleAtFixedRate(this::cleanupSentinels, 10, 10,
TimeUnit.SECONDS);
}
+ private void restoreUnfinishedSnapshotProcedure() {
Review comment:
Yes. I also think that it is the most elegant way to restore the procid
in afterReplay method, but because of the startup order of master, there will
be a NPE problem if we do that.
We can see the startup order in HMaster#finishActiveMasterInitialization().
1. we replay procedures in the method HMaster#createProcedureExecutor()
2. we create the snapshot manager instance in the method
HMaster#initializeZKBasedSystemTrackers()
3. we start the procedure executor in the method
HMaster#startServiceThreads()
Since these three method are executed in the same thread, so I think it's
safe here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] frostruan commented on pull request #4115: Introduce a Snapshot Procedure
frostruan commented on pull request #4115: URL: https://github.com/apache/hbase/pull/4115#issuecomment-1050994730 Really thanks for your great suggestions and help, Duo. I will address it ASAP. @Apache9 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] frostruan commented on a change in pull request #4115: Introduce a Snapshot Procedure
frostruan commented on a change in pull request #4115:
URL: https://github.com/apache/hbase/pull/4115#discussion_r814902922
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/SnapshotProcedure.java
##
@@ -0,0 +1,471 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.master.procedure;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionReplicaUtil;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher;
+import org.apache.hadoop.hbase.master.MasterCoprocessorHost;
+import org.apache.hadoop.hbase.master.MetricsSnapshot;
+import org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure;
+import org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure;
+import org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier;
+import org.apache.hadoop.hbase.master.snapshot.SnapshotManager;
+import org.apache.hadoop.hbase.mob.MobUtils;
+import org.apache.hadoop.hbase.monitoring.MonitoredTask;
+import org.apache.hadoop.hbase.monitoring.TaskMonitor;
+import org.apache.hadoop.hbase.procedure2.Procedure;
+import org.apache.hadoop.hbase.procedure2.ProcedureStateSerializer;
+import org.apache.hadoop.hbase.procedure2.ProcedureSuspendedException;
+import org.apache.hadoop.hbase.procedure2.ProcedureUtil;
+import org.apache.hadoop.hbase.procedure2.ProcedureYieldException;
+import org.apache.hadoop.hbase.snapshot.ClientSnapshotDescriptionUtils;
+import org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException;
+import org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils;
+import org.apache.hadoop.hbase.snapshot.SnapshotManifest;
+import org.apache.hadoop.hbase.util.CommonFSUtils;
+import org.apache.hadoop.hbase.util.ModifyRegionUtils;
+import org.apache.hadoop.hbase.util.RetryCounter;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotProcedureStateData;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos.ProcedureState;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription;
+
+/**
+ * A procedure used to take snapshot on tables.
+ */
+@InterfaceAudience.Private
+public class SnapshotProcedure
+extends AbstractStateMachineTableProcedure {
+ private static final Logger LOG =
LoggerFactory.getLogger(SnapshotProcedure.class);
+ private final MetricsSnapshot metricsSnapshot = new MetricsSnapshot();
+
+ private Configuration conf;
+ private SnapshotDescription snapshot;
+ private Path rootDir;
+ private Path snapshotDir;
+ private Path workingDir;
+ private FileSystem workingDirFS;
+ private FileSystem rootFs;
+ private TableName snapshotTable;
+ private MonitoredTask status;
+ private SnapshotManifest snapshotManifest;
+ private TableDescriptor htd;
+
+ private RetryCounter retryCounter;
+
+ public SnapshotProcedure() { }
+
+ public SnapshotProcedure(final MasterProcedureEnv env, final
SnapshotDescription snapshot) {
+super(env);
+this.snapshot = snapshot;
+ }
+
+ @Override
+ public TableName getTableName() {
+return TableName.valueOf(snapshot.getTable());
+ }
+
+ @Override
+ public TableOperationType getTableOperationType() {
+return TableOperationType.SNAPSHOT;
+ }
+
+ @Override
+ protected LockState acquireLock(MasterProcedureEnv env) {
+//
[GitHub] [hbase] frostruan commented on a change in pull request #4115: Introduce a Snapshot Procedure
frostruan commented on a change in pull request #4115:
URL: https://github.com/apache/hbase/pull/4115#discussion_r814902668
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
##
@@ -1264,4 +1424,55 @@ private SnapshotDescription
toSnapshotDescription(ProcedureDescription desc)
builder.setType(SnapshotDescription.Type.FLUSH);
return builder.build();
}
+
+ public void registerSnapshotProcedure(SnapshotDescription snapshot, long
procId) {
+snapshotToProcIdMap.put(snapshot, procId);
+LOG.debug("register snapshot={}, snapshot procedure id = {}",
+ ClientSnapshotDescriptionUtils.toString(snapshot), procId);
+ }
+
+ public void unregisterSnapshotProcedure(SnapshotDescription snapshot, long
procId) {
+snapshotToProcIdMap.remove(snapshot, procId);
+LOG.debug("unregister snapshot={}, snapshot procedure id = {}",
+ ClientSnapshotDescriptionUtils.toString(snapshot), procId);
+ }
+
+ public boolean snapshotProcedureEnabled() {
+return master.getConfiguration()
+ .getBoolean(SNAPSHOT_PROCEDURE_ENABLED,
SNAPSHOT_PROCEDURE_ENABLED_DEFAULT);
+ }
+
+ public ServerName acquireSnapshotVerifyWorker(SnapshotVerifyProcedure
procedure)
+ throws ProcedureSuspendedException {
+Optional worker = verifyWorkerAssigner.acquire();
+if (worker.isPresent()) {
+ LOG.debug("{} Acquired verify snapshot worker={}", procedure,
worker.get());
+ return worker.get();
+}
+verifyWorkerAssigner.suspend(procedure);
+throw new ProcedureSuspendedException();
+ }
+
+ public void releaseSnapshotVerifyWorker(SnapshotVerifyProcedure procedure,
+ ServerName worker, MasterProcedureScheduler scheduler) {
+LOG.debug("{} Release verify snapshot worker={}", procedure, worker);
+verifyWorkerAssigner.release(worker);
+verifyWorkerAssigner.wake(scheduler);
+ }
+
+ private void restoreWorkers() {
+master.getMasterProcedureExecutor().getActiveProceduresNoCopy().stream()
+ .filter(p -> p instanceof SnapshotVerifyProcedure)
+ .map(p -> (SnapshotVerifyProcedure) p)
+ .filter(p -> !p.isFinished())
+ .filter(p -> p.getServerName() != null)
+ .forEach(p -> {
+verifyWorkerAssigner.addUsedWorker(p.getServerName());
Review comment:
I think it's harmless that we have dead servers in the WorkAssigner
based on this consideration:
1. If the remote server is dead, there will be a
FailedRemoteDispatchException thrown when we dispatch remote procedures. The
procedure-v2 framework is designed to be able to handle this. The upper layer
procedure can notice that and reschedule a new procedure, or the procedure can
retry itself.
2. When acquire a new worker from WorkAssigner, we will always choose from
the online server list. The dead server will not be able to become the
candidate.
3. If there are too many dead servers in the WorkerAssigner, we can restart
the master to clear it. This is a very lightweight operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] frostruan commented on a change in pull request #4115: Introduce a Snapshot Procedure
frostruan commented on a change in pull request #4115:
URL: https://github.com/apache/hbase/pull/4115#discussion_r814902123
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
##
@@ -1214,11 +1358,27 @@ public void initialize(MasterServices master,
MetricsMaster metricsMaster) throw
this.coordinator = new ProcedureCoordinator(comms, tpool, timeoutMillis,
wakeFrequency);
this.executorService = master.getExecutorService();
+this.verifyWorkerAssigner = new WorkerAssigner(master,
+ conf.getInt("hbase.snapshot.verify.task.max", 3),
+ new ProcedureEvent<>("snapshot-verify-worker-assigning"));
+restoreUnfinishedSnapshotProcedure();
+restoreWorkers();
resetTempDir();
snapshotHandlerChoreCleanerTask =
scheduleThreadPool.scheduleAtFixedRate(this::cleanupSentinels, 10, 10,
TimeUnit.SECONDS);
}
+ private void restoreUnfinishedSnapshotProcedure() {
Review comment:
Yes. I also think that it is the most elegant way to restore the procid
in afterReplay method, but because of the startup order of master, there will
be a NPE problem if we do that.
We can see the startup order in HMaster#finishActiveMasterInitialization().
1. we replay procedures in the method HMaster#createProcedureExecutor()
2. we create the snapshot manager instance in the method
HMaster#initializeZKBasedSystemTrackers()
3. we start the procedure executor in the method
HMaster#startServiceThreads()
Since these three method are executed in the same thread, so I think it's
safe here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] frostruan commented on a change in pull request #4115: Introduce a Snapshot Procedure
frostruan commented on a change in pull request #4115:
URL: https://github.com/apache/hbase/pull/4115#discussion_r814901349
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
##
@@ -418,22 +471,33 @@ public boolean isSnapshotDone(SnapshotDescription
expected) throws IOException {
* Currently we have a limitation only allowing a single snapshot per table
at a time. Also we
* don't allow snapshot with the same name.
* @param snapshot description of the snapshot being checked.
+ * @param checkTable check if the table is already taking a snapshot.
Review comment:
This parameter is used for different snapshot backends. For
zk-coordinated snapshot, we need to check if another zk-coordinated snapshot on
the same table is in progress, but for the snapshot procedure, it is
unnecessary.
The basic rule is that,
1. Two zk-coordinated snapshots cannot be taken at the same time.
2. Two snapshot procedures can run at the same time.
3. A zk-coordinated snapshot and a proc-v2 based procedure on the same table
can be taken at the same time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] frostruan commented on a change in pull request #4115: Introduce a Snapshot Procedure
frostruan commented on a change in pull request #4115:
URL: https://github.com/apache/hbase/pull/4115#discussion_r814900911
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/SnapshotRegionProcedure.java
##
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.master.procedure;
+
+import com.google.errorprone.annotations.RestrictedApi;
+import java.io.IOException;
+import java.util.Optional;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.master.assignment.RegionStateNode;
+import org.apache.hadoop.hbase.master.assignment.RegionStates;
+import org.apache.hadoop.hbase.master.assignment.ServerState;
+import org.apache.hadoop.hbase.procedure2.FailedRemoteDispatchException;
+import org.apache.hadoop.hbase.procedure2.Procedure;
+import org.apache.hadoop.hbase.procedure2.ProcedureEvent;
+import org.apache.hadoop.hbase.procedure2.ProcedureStateSerializer;
+import org.apache.hadoop.hbase.procedure2.ProcedureSuspendedException;
+import org.apache.hadoop.hbase.procedure2.ProcedureUtil;
+import org.apache.hadoop.hbase.procedure2.ProcedureYieldException;
+import
org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.RemoteOperation;
+import
org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.RemoteProcedure;
+import org.apache.hadoop.hbase.procedure2.RemoteProcedureException;
+import org.apache.hadoop.hbase.regionserver.SnapshotRegionCallable;
+import org.apache.hadoop.hbase.util.RetryCounter;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotRegionProcedureStateData;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos.ProcedureState;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription;
+
+/**
+ * A remote procedure which is used to send region snapshot request to region
server.
+ * The basic logic of SnapshotRegionProcedure is similar like {@link
ServerRemoteProcedure},
+ * only with a little difference, when {@link FailedRemoteDispatchException}
was thrown,
+ * SnapshotRegionProcedure will sleep some time and continue retrying until
success.
+ */
+@InterfaceAudience.Private
+public class SnapshotRegionProcedure extends Procedure
+implements TableProcedureInterface, RemoteProcedure {
+ private static final Logger LOG =
LoggerFactory.getLogger(SnapshotRegionProcedure.class);
+
+ private SnapshotDescription snapshot;
+ private ProcedureEvent event;
+ private RegionInfo region;
+ private boolean dispatched;
+ private boolean succ;
+ private RetryCounter retryCounter;
+
+ public SnapshotRegionProcedure() {
+ }
+
+ public SnapshotRegionProcedure(SnapshotDescription snapshot, RegionInfo
region) {
+this.snapshot = snapshot;
+this.region = region;
+ }
+
+ @Override
+ protected LockState acquireLock(final MasterProcedureEnv env) {
+if (env.getProcedureScheduler().waitRegions(this, getTableName(), region))
{
+ return LockState.LOCK_EVENT_WAIT;
+}
+return LockState.LOCK_ACQUIRED;
+ }
+
+ @Override
+ protected void releaseLock(final MasterProcedureEnv env) {
+env.getProcedureScheduler().wakeRegions(this, getTableName(), region);
+ }
+
+ @Override
+ protected boolean holdLock(MasterProcedureEnv env) {
+return false;
+ }
+
+ @Override
+ public Optional remoteCallBuild(MasterProcedureEnv env,
ServerName serverName) {
+return Optional.of(new RSProcedureDispatcher.ServerOperation(this,
getProcId(),
+ SnapshotRegionCallable.class,
MasterProcedureProtos.SnapshotRegionParameter.newBuilder()
+
.setRegion(ProtobufUtil.toRegionInfo(region)).setSnapshot(snapshot).build().toByteArray()));
+ }
+
+ @Override
+ public void
[GitHub] [hbase] frostruan commented on a change in pull request #4115: Introduce a Snapshot Procedure
frostruan commented on a change in pull request #4115:
URL: https://github.com/apache/hbase/pull/4115#discussion_r814900752
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/SnapshotRegionProcedure.java
##
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.master.procedure;
+
+import com.google.errorprone.annotations.RestrictedApi;
+import java.io.IOException;
+import java.util.Optional;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.master.assignment.RegionStateNode;
+import org.apache.hadoop.hbase.master.assignment.RegionStates;
+import org.apache.hadoop.hbase.master.assignment.ServerState;
+import org.apache.hadoop.hbase.procedure2.FailedRemoteDispatchException;
+import org.apache.hadoop.hbase.procedure2.Procedure;
+import org.apache.hadoop.hbase.procedure2.ProcedureEvent;
+import org.apache.hadoop.hbase.procedure2.ProcedureStateSerializer;
+import org.apache.hadoop.hbase.procedure2.ProcedureSuspendedException;
+import org.apache.hadoop.hbase.procedure2.ProcedureUtil;
+import org.apache.hadoop.hbase.procedure2.ProcedureYieldException;
+import
org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.RemoteOperation;
+import
org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.RemoteProcedure;
+import org.apache.hadoop.hbase.procedure2.RemoteProcedureException;
+import org.apache.hadoop.hbase.regionserver.SnapshotRegionCallable;
+import org.apache.hadoop.hbase.util.RetryCounter;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotRegionProcedureStateData;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos.ProcedureState;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription;
+
+/**
+ * A remote procedure which is used to send region snapshot request to region
server.
+ * The basic logic of SnapshotRegionProcedure is similar like {@link
ServerRemoteProcedure},
+ * only with a little difference, when {@link FailedRemoteDispatchException}
was thrown,
+ * SnapshotRegionProcedure will sleep some time and continue retrying until
success.
+ */
+@InterfaceAudience.Private
+public class SnapshotRegionProcedure extends Procedure
+implements TableProcedureInterface, RemoteProcedure {
+ private static final Logger LOG =
LoggerFactory.getLogger(SnapshotRegionProcedure.class);
+
+ private SnapshotDescription snapshot;
+ private ProcedureEvent event;
+ private RegionInfo region;
+ private boolean dispatched;
+ private boolean succ;
+ private RetryCounter retryCounter;
+
+ public SnapshotRegionProcedure() {
+ }
+
+ public SnapshotRegionProcedure(SnapshotDescription snapshot, RegionInfo
region) {
+this.snapshot = snapshot;
+this.region = region;
+ }
+
+ @Override
+ protected LockState acquireLock(final MasterProcedureEnv env) {
+if (env.getProcedureScheduler().waitRegions(this, getTableName(), region))
{
+ return LockState.LOCK_EVENT_WAIT;
+}
+return LockState.LOCK_ACQUIRED;
+ }
+
+ @Override
+ protected void releaseLock(final MasterProcedureEnv env) {
+env.getProcedureScheduler().wakeRegions(this, getTableName(), region);
+ }
+
+ @Override
+ protected boolean holdLock(MasterProcedureEnv env) {
+return false;
+ }
+
+ @Override
+ public Optional remoteCallBuild(MasterProcedureEnv env,
ServerName serverName) {
+return Optional.of(new RSProcedureDispatcher.ServerOperation(this,
getProcId(),
+ SnapshotRegionCallable.class,
MasterProcedureProtos.SnapshotRegionParameter.newBuilder()
+
.setRegion(ProtobufUtil.toRegionInfo(region)).setSnapshot(snapshot).build().toByteArray()));
+ }
+
+ @Override
+ public void
[GitHub] [hbase] frostruan commented on a change in pull request #4115: Introduce a Snapshot Procedure
frostruan commented on a change in pull request #4115:
URL: https://github.com/apache/hbase/pull/4115#discussion_r814900450
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/SnapshotProcedure.java
##
@@ -0,0 +1,471 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.master.procedure;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionReplicaUtil;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher;
+import org.apache.hadoop.hbase.master.MasterCoprocessorHost;
+import org.apache.hadoop.hbase.master.MetricsSnapshot;
+import org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure;
+import org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure;
+import org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier;
+import org.apache.hadoop.hbase.master.snapshot.SnapshotManager;
+import org.apache.hadoop.hbase.mob.MobUtils;
+import org.apache.hadoop.hbase.monitoring.MonitoredTask;
+import org.apache.hadoop.hbase.monitoring.TaskMonitor;
+import org.apache.hadoop.hbase.procedure2.Procedure;
+import org.apache.hadoop.hbase.procedure2.ProcedureStateSerializer;
+import org.apache.hadoop.hbase.procedure2.ProcedureSuspendedException;
+import org.apache.hadoop.hbase.procedure2.ProcedureUtil;
+import org.apache.hadoop.hbase.procedure2.ProcedureYieldException;
+import org.apache.hadoop.hbase.snapshot.ClientSnapshotDescriptionUtils;
+import org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException;
+import org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils;
+import org.apache.hadoop.hbase.snapshot.SnapshotManifest;
+import org.apache.hadoop.hbase.util.CommonFSUtils;
+import org.apache.hadoop.hbase.util.ModifyRegionUtils;
+import org.apache.hadoop.hbase.util.RetryCounter;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotProcedureStateData;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos.ProcedureState;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription;
+
+/**
+ * A procedure used to take snapshot on tables.
+ */
+@InterfaceAudience.Private
+public class SnapshotProcedure
+extends AbstractStateMachineTableProcedure {
+ private static final Logger LOG =
LoggerFactory.getLogger(SnapshotProcedure.class);
+ private final MetricsSnapshot metricsSnapshot = new MetricsSnapshot();
+
+ private Configuration conf;
+ private SnapshotDescription snapshot;
+ private Path rootDir;
+ private Path snapshotDir;
+ private Path workingDir;
+ private FileSystem workingDirFS;
+ private FileSystem rootFs;
+ private TableName snapshotTable;
+ private MonitoredTask status;
+ private SnapshotManifest snapshotManifest;
+ private TableDescriptor htd;
+
+ private RetryCounter retryCounter;
+
+ public SnapshotProcedure() { }
+
+ public SnapshotProcedure(final MasterProcedureEnv env, final
SnapshotDescription snapshot) {
+super(env);
+this.snapshot = snapshot;
+ }
+
+ @Override
+ public TableName getTableName() {
+return TableName.valueOf(snapshot.getTable());
+ }
+
+ @Override
+ public TableOperationType getTableOperationType() {
+return TableOperationType.SNAPSHOT;
+ }
+
+ @Override
+ protected LockState acquireLock(MasterProcedureEnv env) {
+//
[GitHub] [hbase] Apache-HBase commented on pull request #4127: HBASE-26768 Avoid unnecessary replication suspending in RegionReplica…
Apache-HBase commented on pull request #4127: URL: https://github.com/apache/hbase/pull/4127#issuecomment-1050966144 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 19s | master passed | | +1 :green_heart: | compile | 1m 50s | master passed | | +1 :green_heart: | checkstyle | 0m 35s | master passed | | +1 :green_heart: | spotbugs | 1m 11s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 24s | the patch passed | | +1 :green_heart: | compile | 1m 49s | the patch passed | | +1 :green_heart: | javac | 1m 49s | the patch passed | | -0 :warning: | checkstyle | 0m 34s | hbase-server: The patch generated 3 new + 140 unchanged - 0 fixed = 143 total (was 140) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 42s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotbugs | 1m 18s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 8s | The patch does not generate ASF License warnings. | | | | 29m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4127/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4127 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 3efdec3001bc 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5dc663ea38 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4127/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 60 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4127/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4125: HBASE-26666 Add native TLS encryption support to RPC server/client
Apache-HBase commented on pull request #4125: URL: https://github.com/apache/hbase/pull/4125#issuecomment-1050941551 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 14s | master passed | | +1 :green_heart: | compile | 1m 14s | master passed | | +1 :green_heart: | shadedjars | 5m 30s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 53s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 54s | the patch passed | | +1 :green_heart: | compile | 1m 14s | the patch passed | | +1 :green_heart: | javac | 1m 14s | the patch passed | | +1 :green_heart: | shadedjars | 5m 18s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 52s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 32s | hbase-common in the patch passed. | | -1 :x: | unit | 1m 18s | hbase-client in the patch failed. | | -1 :x: | unit | 18m 13s | hbase-server in the patch failed. | | | | 45m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4125 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5bbd9175c252 5.4.0-1047-aws #49~18.04.1-Ubuntu SMP Wed Apr 28 23:08:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5dc663ea38 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-client.txt | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/2/testReport/ | | Max. process+thread count | 526 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4125: HBASE-26666 Add native TLS encryption support to RPC server/client
Apache-HBase commented on pull request #4125: URL: https://github.com/apache/hbase/pull/4125#issuecomment-1050936087 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 36s | master passed | | +1 :green_heart: | compile | 1m 14s | master passed | | +1 :green_heart: | shadedjars | 4m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 55s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 24s | the patch passed | | +1 :green_heart: | compile | 1m 13s | the patch passed | | +1 :green_heart: | javac | 1m 14s | the patch passed | | +1 :green_heart: | shadedjars | 4m 37s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 52s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 10s | hbase-common in the patch passed. | | -1 :x: | unit | 1m 15s | hbase-client in the patch failed. | | -1 :x: | unit | 15m 12s | hbase-server in the patch failed. | | | | 39m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4125 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux df3f56db775e 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5dc663ea38 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-client.txt | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/2/testReport/ | | Max. process+thread count | 540 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4125: HBASE-26666 Add native TLS encryption support to RPC server/client
Apache-HBase commented on pull request #4125: URL: https://github.com/apache/hbase/pull/4125#issuecomment-1050933135 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 24s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 33s | master passed | | +1 :green_heart: | compile | 2m 59s | master passed | | +1 :green_heart: | checkstyle | 1m 7s | master passed | | +1 :green_heart: | spotbugs | 2m 29s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 22s | the patch passed | | +1 :green_heart: | compile | 2m 58s | the patch passed | | +1 :green_heart: | javac | 2m 58s | the patch passed | | +1 :green_heart: | checkstyle | 1m 6s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 16s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotbugs | 2m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. | | | | 35m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4125 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux e1f71fa51324 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5dc663ea38 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 60 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4125/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4127: HBASE-26768 Avoid unnecessary replication suspending in RegionReplica…
Apache-HBase commented on pull request #4127: URL: https://github.com/apache/hbase/pull/4127#issuecomment-1050927550 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 51s | master passed | | +1 :green_heart: | compile | 0m 47s | master passed | | +1 :green_heart: | shadedjars | 4m 47s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 48s | the patch passed | | +1 :green_heart: | compile | 0m 46s | the patch passed | | +1 :green_heart: | javac | 0m 46s | the patch passed | | +1 :green_heart: | shadedjars | 4m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 338m 57s | hbase-server in the patch passed. | | | | 359m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4127/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4127 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 538efa4aeb25 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5dc663ea38 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4127/3/testReport/ | | Max. process+thread count | 1778 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4127/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4127: HBASE-26768 Avoid unnecessary replication suspending in RegionReplica…
Apache-HBase commented on pull request #4127: URL: https://github.com/apache/hbase/pull/4127#issuecomment-1050925135 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 13s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | -1 :x: | mvninstall | 4m 29s | root in master failed. | | +1 :green_heart: | compile | 0m 42s | master passed | | +1 :green_heart: | shadedjars | 5m 43s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 30s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 58s | the patch passed | | +1 :green_heart: | compile | 0m 44s | the patch passed | | +1 :green_heart: | javac | 0m 44s | the patch passed | | +1 :green_heart: | shadedjars | 5m 27s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 332m 57s | hbase-server in the patch passed. | | | | 356m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4127/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4127 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 7f5a33aaa48f 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5dc663ea38 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4127/3/artifact/yetus-jdk8-hadoop3-check/output/branch-mvninstall-root.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4127/3/testReport/ | | Max. process+thread count | 1580 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4127/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #4115: Introduce a Snapshot Procedure
Apache9 commented on a change in pull request #4115:
URL: https://github.com/apache/hbase/pull/4115#discussion_r814824572
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
##
@@ -181,6 +196,12 @@
// snapshot using Procedure-V2.
private Map restoreTableToProcIdMap = new HashMap<>();
+ // SnapshotDescription -> SnapshotProcId
+ private final ConcurrentHashMap
Review comment:
So this is trick here.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/SnapshotProcedure.java
##
@@ -0,0 +1,471 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.master.procedure;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionReplicaUtil;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher;
+import org.apache.hadoop.hbase.master.MasterCoprocessorHost;
+import org.apache.hadoop.hbase.master.MetricsSnapshot;
+import org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure;
+import org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure;
+import org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier;
+import org.apache.hadoop.hbase.master.snapshot.SnapshotManager;
+import org.apache.hadoop.hbase.mob.MobUtils;
+import org.apache.hadoop.hbase.monitoring.MonitoredTask;
+import org.apache.hadoop.hbase.monitoring.TaskMonitor;
+import org.apache.hadoop.hbase.procedure2.Procedure;
+import org.apache.hadoop.hbase.procedure2.ProcedureStateSerializer;
+import org.apache.hadoop.hbase.procedure2.ProcedureSuspendedException;
+import org.apache.hadoop.hbase.procedure2.ProcedureUtil;
+import org.apache.hadoop.hbase.procedure2.ProcedureYieldException;
+import org.apache.hadoop.hbase.snapshot.ClientSnapshotDescriptionUtils;
+import org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException;
+import org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils;
+import org.apache.hadoop.hbase.snapshot.SnapshotManifest;
+import org.apache.hadoop.hbase.util.CommonFSUtils;
+import org.apache.hadoop.hbase.util.ModifyRegionUtils;
+import org.apache.hadoop.hbase.util.RetryCounter;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotProcedureStateData;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos.ProcedureState;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription;
+
+/**
+ * A procedure used to take snapshot on tables.
+ */
+@InterfaceAudience.Private
+public class SnapshotProcedure
+extends AbstractStateMachineTableProcedure {
+ private static final Logger LOG =
LoggerFactory.getLogger(SnapshotProcedure.class);
+ private final MetricsSnapshot metricsSnapshot = new MetricsSnapshot();
+
+ private Configuration conf;
+ private SnapshotDescription snapshot;
+ private Path rootDir;
+ private Path snapshotDir;
+ private Path workingDir;
+ private FileSystem workingDirFS;
+ private FileSystem rootFs;
+ private TableName snapshotTable;
+ private MonitoredTask status;
+ private SnapshotManifest snapshotManifest;
+ private TableDescriptor htd;
+
+ private RetryCounter retryCounter;
+
+ public SnapshotProcedure() { }
+
+ public SnapshotProcedure(final MasterProcedureEnv env, final
SnapshotDescription snapshot) {
+
[GitHub] [hbase] Apache9 commented on pull request #4126: HBASE-26759 Fix orphaned `RegionScanner.close` spans
Apache9 commented on pull request #4126: URL: https://github.com/apache/hbase/pull/4126#issuecomment-1050893500 I mean we have a span here https://github.com/apache/hbase/blob/5dc663ea388d64f9d583a9e8151da6b8d403b0fd/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/ServerRpcConnection.java#L634 I think it will be the parent span for the spans at server side, and seems it has not been ended properly? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #4127: HBASE-26768 Avoid unnecessary replication suspending in RegionReplica…
Apache9 commented on a change in pull request #4127:
URL: https://github.com/apache/hbase/pull/4127#discussion_r814795220
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/regionreplication/RegionReplicationSink.java
##
@@ -448,4 +458,35 @@ public void waitUntilStopped() throws InterruptedException
{
}
}
}
+
+ @RestrictedApi(explanation = "Should only be called in tests", link = "",
+ allowedOnPath = ".*/src/test/.*")
+ RegionReplicationFlushRequester getFlushRequester() {
+return this.flushRequester;
+ }
+
+ @RestrictedApi(explanation = "Should only be called in tests", link = "",
+ allowedOnPath = ".*/src/test/.*")
+ IntHashSet getFailedReplicas() {
+synchronized (entries) {
+ return this.failedReplicas;
+}
+ }
+
+ @RestrictedApi(explanation = "Should only be called in tests", link = "",
+ allowedOnPath = ".*/src/test/.*")
+ boolean isSending() {
+synchronized (entries) {
+ return this.sending;
+}
+ }
+
+ @RestrictedApi(explanation = "Should only be called in tests", link = "",
+ allowedOnPath = ".*/src/test/.*")
+ Queue getEntries() {
+synchronized (entries) {
Review comment:
We just return the entries out so I do not get the point on
'synchronized (entries)' here...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26770) HBase client does not honor UserGroupInformation.doAs
[
https://issues.apache.org/jira/browse/HBASE-26770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17498143#comment-17498143
]
Bryan Beaudreault commented on HBASE-26770:
---
Ok, thanks for the input. I think this sounds like too big of a change. I
personally feel like from a credentialing perspective we should be able to use
the connection's user despite sending along a proxy user, because we can sort
of assume that if the connection's user is impersonating another user it should
be ok to access all the same caches. But that could be my bias based on how we
use hbase, where auth is scoped to a process for us. Even if we could agree on
that, we'd have the problem of needing to refactor how connection headers are
sent. Alternatively we could make the cache key change you mentioned, but then
we have the problem of keeping many more connection rpc channels open. On a
moderately sized HBase cluster of 350 regionservers, a proxy API serving
requests for 100 upstream callers may go from 350 sockets to 35k sockets. A
ConnectionCache has similar problems (worse, since each client gets its own
meta cache so we'd increase load on meta during failure scenarios).
Unless someone has a novel idea here, I'll probably go back to the drawing
board of my original problem, which was to send proxy user information along to
regionservers for QoS purposes. The hope was that UGI.doAs could handle that
for us, but seems like not based on the overhead involved with
user-per-connection.
> HBase client does not honor UserGroupInformation.doAs
> -
>
> Key: HBASE-26770
> URL: https://issues.apache.org/jira/browse/HBASE-26770
> Project: HBase
> Issue Type: Bug
>Reporter: Bryan Beaudreault
>Priority: Major
>
> Despite passing necessary UserInformation to the RegionServer, which does
> authorize the request, the async and block clients do not work correctly with
> the following access pattern:
> {code:java}
> Connection connection = ConnectionFactory.createConnection();
> Table table = connection.getTable(name);
> UserGroupInformation proxy = UserGroupInformation.createProxyUser(
> "testUser",
> UserGroupInformation.getCurrentUser()
> );
> Result result = proxy.doAs(() -> table.get(get));{code}
> In this case, you would expect the get to be executed as "testUser", but
> instead it is executed as whichever user created the initial connection. This
> can be verified by checking the security logger on the RegionServer side.
> The reason for this is we stash the current User onto the actual
> ConnectionImplementation, and we pass that through all calls in the stack
> when executing an RPC. I think the appropriate way would be to replace usage
> of this stashed User with a call to UserGroupInformation.getCurrentUser() in
> RpcConnection, where sasl is negotiated and headers generated.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #4137: HBASE-26772 Shell suspended in background
Apache-HBase commented on pull request #4137: URL: https://github.com/apache/hbase/pull/4137#issuecomment-1050685227 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 6s | master passed | | +1 :green_heart: | javadoc | 0m 12s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 49s | the patch passed | | +1 :green_heart: | javadoc | 0m 11s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 9m 44s | hbase-shell in the patch passed. | | | | 17m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4137/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4137 | | Optional Tests | javac javadoc unit | | uname | Linux 57f0c3660a38 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5dc663ea38 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4137/2/testReport/ | | Max. process+thread count | 849 (vs. ulimit of 3) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4137/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4137: HBASE-26772 Shell suspended in background
Apache-HBase commented on pull request #4137: URL: https://github.com/apache/hbase/pull/4137#issuecomment-1050684161 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 4s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 38s | master passed | | +1 :green_heart: | javadoc | 0m 11s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 27s | the patch passed | | +1 :green_heart: | javadoc | 0m 11s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 8m 39s | hbase-shell in the patch passed. | | | | 15m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4137/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4137 | | Optional Tests | javac javadoc unit | | uname | Linux 45a78edd3ab1 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5dc663ea38 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4137/2/testReport/ | | Max. process+thread count | 794 (vs. ulimit of 3) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4137/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4137: HBASE-26772 Shell suspended in background
Apache-HBase commented on pull request #4137: URL: https://github.com/apache/hbase/pull/4137#issuecomment-1050677455 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 41s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 22s | the patch passed | | -0 :warning: | rubocop | 0m 5s | The patch generated 3 new + 47 unchanged - 1 fixed = 50 total (was 48) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 7m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4137/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4137 | | Optional Tests | dupname asflicense javac rubocop | | uname | Linux c06bb1738b1b 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5dc663ea38 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | rubocop | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4137/2/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 45 (vs. ulimit of 3) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4137/2/console | | versions | git=2.17.1 maven=3.6.3 rubocop=0.80.0 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4133: HBASE-26622 Update error-prone to 2.10
Apache-HBase commented on pull request #4133: URL: https://github.com/apache/hbase/pull/4133#issuecomment-1050625056 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 13s | Docker mode activated. | | -0 :warning: | yetus | 2m 49s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 36s | master passed | | +1 :green_heart: | compile | 1m 40s | master passed | | +1 :green_heart: | shadedjars | 4m 36s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 0s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 21s | the patch passed | | +1 :green_heart: | compile | 1m 39s | the patch passed | | +1 :green_heart: | javac | 1m 39s | the patch passed | | +1 :green_heart: | shadedjars | 4m 38s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 2s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 472m 38s | root in the patch failed. | | | | 503m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4133/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4133 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux cf14d2298919 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5dc663ea38 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4133/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4133/2/testReport/ | | Max. process+thread count | 1719 (vs. ulimit of 3) | | modules | C: hbase-build-configuration hbase-common hbase-asyncfs hbase-server hbase-mapreduce hbase-thrift hbase-endpoint . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4133/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
