[jira] [Resolved] (HBASE-26027) The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException
[
https://issues.apache.org/jira/browse/HBASE-26027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Wang resolved HBASE-26027.
Resolution: Fixed
> The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone
> caused by ArrayStoreException
> -
>
> Key: HBASE-26027
> URL: https://issues.apache.org/jira/browse/HBASE-26027
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.2.7, 2.5.0, 2.3.5, 2.4.4
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
> Fix For: 2.6.0
>
>
> The batch api of HTable contains a param named results to store result or
> exception, its type is Object[].
> If user pass an array with other type, eg:
> org.apache.hadoop.hbase.client.Result, and if we need to put an exception
> into it by some reason, then the ArrayStoreException will occur in
> AsyncRequestFutureImpl.updateResult, then the
> AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
> AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
> actionsInProgress again and again, forever.
> It is better to add an cutoff calculated by operationTimeout, instead of only
> depend on the value of actionsInProgress.
> BTW, this issue only for 2.x, since 3.x the implement has refactored.
> How to reproduce:
> 1: add sleep in RSRpcServices.multi to mock slow response
> {code:java}
> try {
> Thread.sleep(2000);
> } catch (InterruptedException e) {
> e.printStackTrace();
> }
> {code}
> 2: set time out in config
> {code:java}
> conf.set("hbase.rpc.timeout","2000");
> conf.set("hbase.client.operation.timeout","6000");
> {code}
> 3: call batch api
> {code:java}
> Table table = HbaseUtil.getTable("test");
> byte[] cf = Bytes.toBytes("f");
> byte[] c = Bytes.toBytes("c1");
> List gets = new ArrayList<>();
> for (int i = 0; i < 10; i++) {
> byte[] rk = Bytes.toBytes("rk-" + i);
> Get get = new Get(rk);
> get.addColumn(cf, c);
> gets.add(get);
> }
> Result[] results = new Result[gets.size()];
> table.batch(gets, results);
> {code}
> The log will looks like below:
> {code:java}
> [ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 -
> id=1 error for test processing localhost,16020,1624343786295
> java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10 actions to
> finish on table:
> [INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10 actions to
> finish on table: test
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
Re: [NOTICE] Apache log4j2 security vulnerability
Thanks for sharing! I found another post [2] that said how to perform such an attack. Should we have a JIRA and keep tracking the solution for it? [2] https://www.lunasec.io/docs/blog/log4j-zero-day/ -Stephen On Thu, Dec 9, 2021 at 8:09 PM 张铎(Duo Zhang) wrote: > See this PR > > https://github.com/apache/logging-log4j2/pull/608 > > Although the final 2.15.0 release for log4j2 has not been published yet, at > least on the Chinese internet the details and how to make use of > this vulnerability has already been public[1]. > > HBase 3.0.0-alpha-1 is affected, so once 2.15.0 is out, we will push a > 3.0.0-alpha-2 release out soon. And for those who already use HBase > 3.0.0-alpha-1, please consider using the following ways to disable JNDI > > Add '-Dlog4j2.formatMsgNoLookups=true' when starting JVM > Add 'log4j2.formatMsgNoLookups=True' to config file > 'export FORMAT_MESSAGES_PATTERN_DISABLE_LOOKUPS=true' before starting JVM > > Thanks. > > 1. https://nosec.org/home/detail/4917.html >
Re: [VOTE] The second release condidate for hbase-thirdparty 4.0.0 is available for download
With 4 binding +1s, no -1, the vote passes. Let me push out the release. Thanks all for voting! 张铎(Duo Zhang) 于2021年12月10日周五 13:40写道： > Here is my +1, I've been testing it by opening a PR against hbase master > branch and it works. > > See https://github.com/apache/hbase/pull/3910 > > Yu Li 于2021年12月10日周五 09:37写道： > >> +1 >> >> Checked the diff between 3.5.1 and 4.0.0-rc1: OK ( >> https://github.com/apache/hbase-thirdparty/compare/rel/3.5.1...4.0.0RC1) >> Checked release note and changes: OK >> Checked sums and signatures: OK >> Maven clean install from source (1.8.0_121): OK >> - Minor: I tried to build a tarball from source following README but >> failed >> with "No assembly descriptors found" error >> Checked the jars in the staging repo: OK >> >> btw, I haven't followed up for a while and could anyone kindly let me know >> where to find this fancy hbase-vote.sh script, so next time I could also >> try it out? Thanks :-) >> >> Best Regards, >> Yu >> >> >> On Fri, 10 Dec 2021 at 05:46, Nick Dimiduk wrote: >> >> > +1 >> > >> > I've used the hbase-vote.sh script to evaluate this artifact and >> there's a >> > problem in the final `run_tests` , executed after `build_from_source`. >> > >> > * Signature: ok >> > * Checksum : ok >> > * Rat check (11.0.11): ok >> > - mvn clean apache-rat:check >> > * Built from source (11.0.11): ok >> > - mvn clean install -DskipTests >> > * Unit tests pass (11.0.11): failed >> > - mvn package -P runAllTests >> -Dsurefire.rerunFailingTestsCount=3 >> > >> > [WARNING] The requested profile "runAllTests" could not be activated >> > because it does not exist. >> > [ERROR] Failed to execute goal >> > org.apache.maven.plugins:maven-shade-plugin:3.2.4:shade (default) on >> > project hbase-shaded-jackson-jaxrs-json-provider: Error creating shaded >> > jar: duplicate entry: >> > META-INF/services/org.apache.hbase.thirdparty.javax.ws.rs >> > .ext.MessageBodyWriter >> > >> > Manually running `mvn clean package` within the source tarball, we >> succeed. >> > >> > I have also triggered a PR build of HBASE-25864 / PR#3243 that uses this >> > RC. The tests are still running. >> > >> > >> https://ci-hadoop.apache.org/blue/organizations/jenkins/HBase%2FHBase-PreCommit-GitHub-PR/detail/PR-3243/5/pipeline/ >> > >> > On Tue, Dec 7, 2021 at 6:02 PM 张铎(Duo Zhang) >> > wrote: >> > >> > > Ah, Thanks Nick for explaining and thanks Andrew for testing. >> > > >> > > We still need one more +1 to close this vote. >> > > >> > > Andrew Purtell 于2021年12月7日周二 05:50写道： >> > > >> > > > Ok, change my vote to +1 (binding). The hbase-thirdparty build and >> > > > artifacts are good. >> > > > >> > > > > On Dec 6, 2021, at 1:18 PM, Nick Dimiduk >> > wrote: >> > > > > >> > > > > On Mon, Dec 6, 2021 at 11:49 AM Andrew Purtell < >> apurt...@apache.org >> > > >> > > > wrote: >> > > > > >> > > > >> -1 (binding) >> > > > >> >> > > > >> Checked sums and signature, ok >> > > > >> RAT check passed, ok >> > > > >> Built from source, ok >> > > > >> Built HEAD of master (d9315fa043) with >> > > -Dhbase-thirdparty.version=4.0.0, >> > > > >> hbase-http module tests fail >> > > > >> >> > > > > >> > > > > Adoption of this dependency will require changes to master. I had >> > > posted >> > > > > necessary changes on https://github.com/apache/hbase/pull/3243 >> and >> > Duo >> > > > did >> > > > > his own on https://github.com/apache/hbase/pull/3910. >> > > > > >> > > > > [ERROR] Tests run: 17, Failures: 0, Errors: 1, Skipped: 2, Time >> > > elapsed: >> > > > >> 2.29 s <<< FAILURE! - in >> org.apache.hadoop.hbase.http.TestHttpServer >> > > > >> [ERROR] org.apache.hadoop.hbase.http.TestHttpServer.testJersey >> Time >> > > > >> elapsed: 0.123 s <<< ERROR! >> > > > >> java.io.FileNotFoundException: >> > > http://localhost:55106/jersey/foo?op=bar >> > > > >> at >> > > > >> >> > > > >> >> > > > >> > > >> > >> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1898) >> > > > >> at >> > > > >> >> > > > >> >> > > > >> > > >> > >> sun.net.www.protocol.http.HttpURLConnection.access$200(HttpURLConnection.java:92) >> > > > >> at >> > > > >> >> > > > >> >> > > > >> > > >> > >> sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1492) >> > > > >> at >> > > > >> >> > > > >> >> > > > >> > > >> > >> sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1490) >> > > > >> at java.security.AccessController.doPrivileged(Native Method) >> > > > >> at >> > > > >> >> > > > >> >> > > > >> > > >> > >> java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:784) >> > > > >> at >> > > > >> >> > > > >> >> > > > >> > > >> > >> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1489) >> > > > >> at >> > > > >> >> > > > >> >> > > > >> > > >> > >> org.apache.hadoop.hbase.http.HttpServerFunctionalTest.readOutput(HttpServerFunctionalTest.java:248) >> > > > >> at >> >
Re: [VOTE] The second release condidate for hbase-thirdparty 4.0.0 is available for download
Here is my +1, I've been testing it by opening a PR against hbase master branch and it works. See https://github.com/apache/hbase/pull/3910 Yu Li 于2021年12月10日周五 09:37写道： > +1 > > Checked the diff between 3.5.1 and 4.0.0-rc1: OK ( > https://github.com/apache/hbase-thirdparty/compare/rel/3.5.1...4.0.0RC1) > Checked release note and changes: OK > Checked sums and signatures: OK > Maven clean install from source (1.8.0_121): OK > - Minor: I tried to build a tarball from source following README but failed > with "No assembly descriptors found" error > Checked the jars in the staging repo: OK > > btw, I haven't followed up for a while and could anyone kindly let me know > where to find this fancy hbase-vote.sh script, so next time I could also > try it out? Thanks :-) > > Best Regards, > Yu > > > On Fri, 10 Dec 2021 at 05:46, Nick Dimiduk wrote: > > > +1 > > > > I've used the hbase-vote.sh script to evaluate this artifact and there's > a > > problem in the final `run_tests` , executed after `build_from_source`. > > > > * Signature: ok > > * Checksum : ok > > * Rat check (11.0.11): ok > > - mvn clean apache-rat:check > > * Built from source (11.0.11): ok > > - mvn clean install -DskipTests > > * Unit tests pass (11.0.11): failed > > - mvn package -P runAllTests > -Dsurefire.rerunFailingTestsCount=3 > > > > [WARNING] The requested profile "runAllTests" could not be activated > > because it does not exist. > > [ERROR] Failed to execute goal > > org.apache.maven.plugins:maven-shade-plugin:3.2.4:shade (default) on > > project hbase-shaded-jackson-jaxrs-json-provider: Error creating shaded > > jar: duplicate entry: > > META-INF/services/org.apache.hbase.thirdparty.javax.ws.rs > > .ext.MessageBodyWriter > > > > Manually running `mvn clean package` within the source tarball, we > succeed. > > > > I have also triggered a PR build of HBASE-25864 / PR#3243 that uses this > > RC. The tests are still running. > > > > > https://ci-hadoop.apache.org/blue/organizations/jenkins/HBase%2FHBase-PreCommit-GitHub-PR/detail/PR-3243/5/pipeline/ > > > > On Tue, Dec 7, 2021 at 6:02 PM 张铎(Duo Zhang) > > wrote: > > > > > Ah, Thanks Nick for explaining and thanks Andrew for testing. > > > > > > We still need one more +1 to close this vote. > > > > > > Andrew Purtell 于2021年12月7日周二 05:50写道： > > > > > > > Ok, change my vote to +1 (binding). The hbase-thirdparty build and > > > > artifacts are good. > > > > > > > > > On Dec 6, 2021, at 1:18 PM, Nick Dimiduk > > wrote: > > > > > > > > > > On Mon, Dec 6, 2021 at 11:49 AM Andrew Purtell < > apurt...@apache.org > > > > > > > wrote: > > > > > > > > > >> -1 (binding) > > > > >> > > > > >> Checked sums and signature, ok > > > > >> RAT check passed, ok > > > > >> Built from source, ok > > > > >> Built HEAD of master (d9315fa043) with > > > -Dhbase-thirdparty.version=4.0.0, > > > > >> hbase-http module tests fail > > > > >> > > > > > > > > > > Adoption of this dependency will require changes to master. I had > > > posted > > > > > necessary changes on https://github.com/apache/hbase/pull/3243 and > > Duo > > > > did > > > > > his own on https://github.com/apache/hbase/pull/3910. > > > > > > > > > > [ERROR] Tests run: 17, Failures: 0, Errors: 1, Skipped: 2, Time > > > elapsed: > > > > >> 2.29 s <<< FAILURE! - in > org.apache.hadoop.hbase.http.TestHttpServer > > > > >> [ERROR] org.apache.hadoop.hbase.http.TestHttpServer.testJersey > Time > > > > >> elapsed: 0.123 s <<< ERROR! > > > > >> java.io.FileNotFoundException: > > > http://localhost:55106/jersey/foo?op=bar > > > > >> at > > > > >> > > > > >> > > > > > > > > > > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1898) > > > > >> at > > > > >> > > > > >> > > > > > > > > > > sun.net.www.protocol.http.HttpURLConnection.access$200(HttpURLConnection.java:92) > > > > >> at > > > > >> > > > > >> > > > > > > > > > > sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1492) > > > > >> at > > > > >> > > > > >> > > > > > > > > > > sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1490) > > > > >> at java.security.AccessController.doPrivileged(Native Method) > > > > >> at > > > > >> > > > > >> > > > > > > > > > > java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:784) > > > > >> at > > > > >> > > > > >> > > > > > > > > > > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1489) > > > > >> at > > > > >> > > > > >> > > > > > > > > > > org.apache.hadoop.hbase.http.HttpServerFunctionalTest.readOutput(HttpServerFunctionalTest.java:248) > > > > >> at > > > > >> > > > > >> > > > > > > > > > > org.apache.hadoop.hbase.http.TestHttpServer.testJersey(TestHttpServer.java:519) > > > > >> > > > > >> > > > > >>> On Fri, Dec 3, 2021 at 6:23 AM 张铎(Duo Zhang) < > > palomino...@gmail.com> > > > > >>> wrote: > > > > >>> > > > > >>> Please vote on this Apache hbase
[jira] [Created] (HBASE-26557) log4j2 has a critical RCE vulnerability
Yutong Xiao created HBASE-26557: --- Summary: log4j2 has a critical RCE vulnerability Key: HBASE-26557 URL: https://issues.apache.org/jira/browse/HBASE-26557 Project: HBase Issue Type: Bug Reporter: Yutong Xiao Assignee: Yutong Xiao -- This message was sent by Atlassian Jira (v8.20.1#820001)
[NOTICE] Apache log4j2 security vulnerability
See this PR https://github.com/apache/logging-log4j2/pull/608 Although the final 2.15.0 release for log4j2 has not been published yet, at least on the Chinese internet the details and how to make use of this vulnerability has already been public[1]. HBase 3.0.0-alpha-1 is affected, so once 2.15.0 is out, we will push a 3.0.0-alpha-2 release out soon. And for those who already use HBase 3.0.0-alpha-1, please consider using the following ways to disable JNDI Add '-Dlog4j2.formatMsgNoLookups=true' when starting JVM Add 'log4j2.formatMsgNoLookups=True' to config file 'export FORMAT_MESSAGES_PATTERN_DISABLE_LOOKUPS=true' before starting JVM Thanks. 1. https://nosec.org/home/detail/4917.html
Re: [VOTE] The second release condidate for hbase-thirdparty 4.0.0 is available for download
+1 Checked the diff between 3.5.1 and 4.0.0-rc1: OK ( https://github.com/apache/hbase-thirdparty/compare/rel/3.5.1...4.0.0RC1) Checked release note and changes: OK Checked sums and signatures: OK Maven clean install from source (1.8.0_121): OK - Minor: I tried to build a tarball from source following README but failed with "No assembly descriptors found" error Checked the jars in the staging repo: OK btw, I haven't followed up for a while and could anyone kindly let me know where to find this fancy hbase-vote.sh script, so next time I could also try it out? Thanks :-) Best Regards, Yu On Fri, 10 Dec 2021 at 05:46, Nick Dimiduk wrote: > +1 > > I've used the hbase-vote.sh script to evaluate this artifact and there's a > problem in the final `run_tests` , executed after `build_from_source`. > > * Signature: ok > * Checksum : ok > * Rat check (11.0.11): ok > - mvn clean apache-rat:check > * Built from source (11.0.11): ok > - mvn clean install -DskipTests > * Unit tests pass (11.0.11): failed > - mvn package -P runAllTests -Dsurefire.rerunFailingTestsCount=3 > > [WARNING] The requested profile "runAllTests" could not be activated > because it does not exist. > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-shade-plugin:3.2.4:shade (default) on > project hbase-shaded-jackson-jaxrs-json-provider: Error creating shaded > jar: duplicate entry: > META-INF/services/org.apache.hbase.thirdparty.javax.ws.rs > .ext.MessageBodyWriter > > Manually running `mvn clean package` within the source tarball, we succeed. > > I have also triggered a PR build of HBASE-25864 / PR#3243 that uses this > RC. The tests are still running. > > https://ci-hadoop.apache.org/blue/organizations/jenkins/HBase%2FHBase-PreCommit-GitHub-PR/detail/PR-3243/5/pipeline/ > > On Tue, Dec 7, 2021 at 6:02 PM 张铎(Duo Zhang) > wrote: > > > Ah, Thanks Nick for explaining and thanks Andrew for testing. > > > > We still need one more +1 to close this vote. > > > > Andrew Purtell 于2021年12月7日周二 05:50写道： > > > > > Ok, change my vote to +1 (binding). The hbase-thirdparty build and > > > artifacts are good. > > > > > > > On Dec 6, 2021, at 1:18 PM, Nick Dimiduk > wrote: > > > > > > > > On Mon, Dec 6, 2021 at 11:49 AM Andrew Purtell > > > > wrote: > > > > > > > >> -1 (binding) > > > >> > > > >> Checked sums and signature, ok > > > >> RAT check passed, ok > > > >> Built from source, ok > > > >> Built HEAD of master (d9315fa043) with > > -Dhbase-thirdparty.version=4.0.0, > > > >> hbase-http module tests fail > > > >> > > > > > > > > Adoption of this dependency will require changes to master. I had > > posted > > > > necessary changes on https://github.com/apache/hbase/pull/3243 and > Duo > > > did > > > > his own on https://github.com/apache/hbase/pull/3910. > > > > > > > > [ERROR] Tests run: 17, Failures: 0, Errors: 1, Skipped: 2, Time > > elapsed: > > > >> 2.29 s <<< FAILURE! - in org.apache.hadoop.hbase.http.TestHttpServer > > > >> [ERROR] org.apache.hadoop.hbase.http.TestHttpServer.testJersey Time > > > >> elapsed: 0.123 s <<< ERROR! > > > >> java.io.FileNotFoundException: > > http://localhost:55106/jersey/foo?op=bar > > > >> at > > > >> > > > >> > > > > > > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1898) > > > >> at > > > >> > > > >> > > > > > > sun.net.www.protocol.http.HttpURLConnection.access$200(HttpURLConnection.java:92) > > > >> at > > > >> > > > >> > > > > > > sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1492) > > > >> at > > > >> > > > >> > > > > > > sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1490) > > > >> at java.security.AccessController.doPrivileged(Native Method) > > > >> at > > > >> > > > >> > > > > > > java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:784) > > > >> at > > > >> > > > >> > > > > > > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1489) > > > >> at > > > >> > > > >> > > > > > > org.apache.hadoop.hbase.http.HttpServerFunctionalTest.readOutput(HttpServerFunctionalTest.java:248) > > > >> at > > > >> > > > >> > > > > > > org.apache.hadoop.hbase.http.TestHttpServer.testJersey(TestHttpServer.java:519) > > > >> > > > >> > > > >>> On Fri, Dec 3, 2021 at 6:23 AM 张铎(Duo Zhang) < > palomino...@gmail.com> > > > >>> wrote: > > > >>> > > > >>> Please vote on this Apache hbase thirdparty release candidate, > > > >>> hbase-thirdparty-4.0.0RC1 > > > >>> > > > >>> The VOTE will remain open for at least 72 hours. > > > >>> > > > >>> [ ] +1 Release this package as Apache hbase thirdparty 4.0.0 > > > >>> [ ] -1 Do not release this package because ... > > > >>> > > > >>> The tag to be voted on is 4.0.0RC1: > > > >>> > > > >>> https://github.com/apache/hbase-thirdparty/tree/4.0.0RC1 > > > >>> > > > >>> This tag currently points to git reference > > > >>> > > > >>>
[jira] [Created] (HBASE-26556) IT and Chaos Monkey improvements
Wellington Chevreuil created HBASE-26556: Summary: IT and Chaos Monkey improvements Key: HBASE-26556 URL: https://issues.apache.org/jira/browse/HBASE-26556 Project: HBase Issue Type: Improvement Components: integration tests Reporter: Wellington Chevreuil Assignee: Wellington Chevreuil While running the IntegrationTestIngest with slowDeterministic chaos monkey in our environment, we faced some issues with some of the chaos actions that required run some linux shell commands, due to the way we set hbase environment variables and start/stop scripts. This patch allows for defining extensions of HBaseShellCommandProvider in the configuration, so that custom commands can be used. It also extends SlowDeterministicMonkeyFactory to provide the ability to configure the destructive actions to be executed. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [DISCUSS] In flight work to complete before 2.5.0RC0
The OpenTelemetry improvements are tracked as subtasks of HBASE-26419 [0]. More eyes on the open PRs would be very helpful, and if anyone is interested in joining the discussion in the CNCF community, you can join their Slack [1]. One change often depends on the previous, so I have been working through the list and pushing patches onto a feature branch [2]. Thanks, Nick [0]: https://issues.apache.org/jira/browse/HBASE-26419 [1]: https://cloud-native.slack.com/archives/C0150QF88FL [2]: https://github.com/ndimiduk/hbase/tree/26419-otel-semantic-conventions On Wed, Dec 8, 2021 at 6:09 PM 张铎(Duo Zhang) wrote: > OpenTracing -> OpenTelemetry :) > > For me, I think the OpenTelemetry part is a blocker, we must finish it > before cutting an RC since the current implementation is already landed on > branch-2.5 and it breaks some Otel best practises, so we should not release > it out. > > Now it is only Nick doing the work and Tak Lon Wu and I reviewing the PRs. > And I also joined the CNCF slack channel and saw Nick is working hard in > communication with the Otel community on how to better implement tracing in > HBase, for example, how to trace big scans. > I would encourage more people in our community to involve so we can make > progress faster. > > Thanks. > > Sean Busbey 于2021年12月9日周四 10:02写道： > > > If we don't want to wait for HBASE-26543 (fix arg parsing for shell) > > then we should revert HBASE-24772 from branch-2.5 prior to an RC. > > > > > > On Wed, Dec 8, 2021 at 7:34 PM Andrew Purtell > wrote: > > > > > > As your branch-2.5 RM I am assembling a list of work items that should > be > > > completed before a 2.5.0RC0 candidate is submitted for the PMC's > > > consideration. > > > > > > I have so far: > > > > > > - OpenTracing span naming convention and coverage improvements. > > > > > > - Shell exit code fixes/improvements. > > > > > > - The "encryption improvements umbrella". Arguable, but let's include > it > > > for now. Can all be resolved as Later if need be. > > > > > > Let's discuss what else, if anything, should be on this list, or if one > > or > > > more of the above items does not constitute a release blocker. I > consider > > > incomplete work-in-progress a blocker. Obviously all of the work in > > > progress should land before release. For WIP, let's also agree on a > > > definition of done. > > > > > > -- > > > Best regards, > > > Andrew > > > > > > Words like orphans lost among the crosstalk, meaning torn from truth's > > > decrepit hands > > >- A23, Crosstalk > > >
A tweak to our checkstyle configuration
Heya,
I have posted a small change to our checkstyle configuration on
HBASE-26536. This change will relax the whitespace rules regarding the
left-curly-bracket ('{') character. Specifically, I intend this change to
allow short expressions that include a nested scope that fits entirely on
one line. The example I provide is:
if (foo == null) { return null; }
This whitespace style is already present (though I think not in popular
usage) within the codebase. Please take a look and let me know if you have
any concerns about making this change.
Thanks,
Nick
https://issues.apache.org/jira/browse/HBASE-26536
https://github.com/apache/hbase/pull/3913
[jira] [Created] (HBASE-26555) Fix findbugs/spotbugs findings
Andrew Kyle Purtell created HBASE-26555: --- Summary: Fix findbugs/spotbugs findings Key: HBASE-26555 URL: https://issues.apache.org/jira/browse/HBASE-26555 Project: HBase Issue Type: Task Reporter: Andrew Kyle Purtell Fix For: 2.5.0, 3.0.0-alpha-2, 2.3.8, 2.4.9, 2.6.0 Clean up findbugs/spotbugs findings on branch HEADs (master, branch-2, branch-2.5, branch-2.4, branch-2.3) For example, from recent precommits: Possible null pointer dereference in org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.opMeterName(Object) due to return value of called method Dereferenced at MetaTableMetrics.java:org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.opMeterName(Object) due to return value of called method Dereferenced at MetaTableMetrics.java:[line 230] Possible null pointer dereference in org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.opWithClientMeterName(Object) due to return value of called method Dereferenced at MetaTableMetrics.java:org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.opWithClientMeterName(Object) due to return value of called method Dereferenced at MetaTableMetrics.java:[line 210] Possible null pointer dereference in org.apache.hadoop.hbase.io.hfile.LruCachedBlockQueue.add(LruCachedBlock) due to return value of called method Method invoked at LruCachedBlockQueue.java:org.apache.hadoop.hbase.io.hfile.LruCachedBlockQueue.add(LruCachedBlock) due to return value of called method Method invoked at LruCachedBlockQueue.java:[line 73] Possible null pointer dereference in org.apache.hadoop.hbase.io.hfile.bucket.CachedEntryQueue.add(Map$Entry) due to return value of called method Dereferenced at CachedEntryQueue.java:org.apache.hadoop.hbase.io.hfile.bucket.CachedEntryQueue.add(Map$Entry) due to return value of called method Dereferenced at CachedEntryQueue.java:[line 79] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26122) Limit max result size of individual Gets
[
https://issues.apache.org/jira/browse/HBASE-26122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Beaudreault resolved HBASE-26122.
---
Fix Version/s: (was: 3.0.0-alpha-2)
(was: 2.6.0)
Release Note: (was: Can now call Get.setMaxResultSize(). When set to a
positive value, the server will return the results when that threshold is met.
This may result in partial results for large rows, so the caller is expected to
handle the case where Result#mayHaveMoreCellsInRow() is true when
setMaxResultSize is used. Possible options include paginating using PageFilter,
reducing the returned data set using other filters, converting the Get to a
Scan (which can take advantage of partial response stitching), or throwing a
non-retryable exception if using this as a guardrail. See below for example
usage in shell:
Create table
hbase:005:0> create 't1', 'f1'
Created table t1
Took 1.1306 seconds
Insert test data
hbase:012:0> put 't1', 'r1', 'f1:c1', 'a'
Took 0.0416 seconds
hbase:014:0> put 't1', 'r1', 'f1:c2', 'b'
Took 0.0059 seconds
hbase:015:0> put 't1', 'r1', 'f1:c3', 'c'
Took 0.0097 seconds
Get without setMaxResultSize, returns full row and mayHaveMoreCellsInRow = false
hbase:037:0> g = Get.new('r1'.to_s.to_java_bytes)
=> #
hbase:038:0> result = @hbase.table('t1',
@shell).instance_variable_get(:@table).get(g)
=> #
hbase:039:0> result.mayHaveMoreCellsInRow
=> false
hbase:040:0> result.toString
=> "keyvalues={r1/f1:c1/1627498270850/Put/vlen=1/seqid=0,
r1/f1:c2/1627498276326/Put/vlen=1/seqid=0,
r1/f1:c3/1627498280413/Put/vlen=1/seqid=0}"
Get with setMaxResultSize, returns first two columns and mayHaveMoreCellsInRow
= true
hbase:059:0> g = Get.new('r1'.to_s.to_java_bytes).setMaxResultSize(100)
=> #
hbase:060:0> result = @hbase.table('t1',
@shell).instance_variable_get(:@table).get(g)
=> #
hbase:061:0> result.mayHaveMoreCellsInRow
=> true
hbase:062:0> result.toString
=> "keyvalues={r1/f1:c1/1627498270850/Put/vlen=1/seqid=0,
r1/f1:c2/1627498276326/Put/vlen=1/seqid=0}")
Resolution: Won't Fix
> Limit max result size of individual Gets
>
>
> Key: HBASE-26122
> URL: https://issues.apache.org/jira/browse/HBASE-26122
> Project: HBase
> Issue Type: New Feature
> Components: Client, regionserver
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
>
> Scans have the ability to have a configured max result size, which causes
> them to return a partial result once the limit has been reached. MultiGets
> also can throw MultiActionResultTooLarge if the response size is over a
> configured quota. Neither of these really accounts for a single Get of a
> too-large row. Such too-large Gets can cause substantial GC pressure or worse
> if sent at volume.
> Currently one can work around this by converting their Get to a single row
> Scan, but this requires a developer to proactively know about and prepare for
> the issue by using a Scan upfront or wait for the RegionServer to choke on a
> large request and only then rewrite the Get for future requests.
> We should implement the same response size limits for for Get as for Scan,
> whereby the server returns a partial result to the client for handling.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
Re: [VOTE] The second release condidate for hbase-thirdparty 4.0.0 is available for download
+1 I've used the hbase-vote.sh script to evaluate this artifact and there's a problem in the final `run_tests` , executed after `build_from_source`. * Signature: ok * Checksum : ok * Rat check (11.0.11): ok - mvn clean apache-rat:check * Built from source (11.0.11): ok - mvn clean install -DskipTests * Unit tests pass (11.0.11): failed - mvn package -P runAllTests -Dsurefire.rerunFailingTestsCount=3 [WARNING] The requested profile "runAllTests" could not be activated because it does not exist. [ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.4:shade (default) on project hbase-shaded-jackson-jaxrs-json-provider: Error creating shaded jar: duplicate entry: META-INF/services/org.apache.hbase.thirdparty.javax.ws.rs.ext.MessageBodyWriter Manually running `mvn clean package` within the source tarball, we succeed. I have also triggered a PR build of HBASE-25864 / PR#3243 that uses this RC. The tests are still running. https://ci-hadoop.apache.org/blue/organizations/jenkins/HBase%2FHBase-PreCommit-GitHub-PR/detail/PR-3243/5/pipeline/ On Tue, Dec 7, 2021 at 6:02 PM 张铎(Duo Zhang) wrote: > Ah, Thanks Nick for explaining and thanks Andrew for testing. > > We still need one more +1 to close this vote. > > Andrew Purtell 于2021年12月7日周二 05:50写道： > > > Ok, change my vote to +1 (binding). The hbase-thirdparty build and > > artifacts are good. > > > > > On Dec 6, 2021, at 1:18 PM, Nick Dimiduk wrote: > > > > > > On Mon, Dec 6, 2021 at 11:49 AM Andrew Purtell > > wrote: > > > > > >> -1 (binding) > > >> > > >> Checked sums and signature, ok > > >> RAT check passed, ok > > >> Built from source, ok > > >> Built HEAD of master (d9315fa043) with > -Dhbase-thirdparty.version=4.0.0, > > >> hbase-http module tests fail > > >> > > > > > > Adoption of this dependency will require changes to master. I had > posted > > > necessary changes on https://github.com/apache/hbase/pull/3243 and Duo > > did > > > his own on https://github.com/apache/hbase/pull/3910. > > > > > > [ERROR] Tests run: 17, Failures: 0, Errors: 1, Skipped: 2, Time > elapsed: > > >> 2.29 s <<< FAILURE! - in org.apache.hadoop.hbase.http.TestHttpServer > > >> [ERROR] org.apache.hadoop.hbase.http.TestHttpServer.testJersey Time > > >> elapsed: 0.123 s <<< ERROR! > > >> java.io.FileNotFoundException: > http://localhost:55106/jersey/foo?op=bar > > >> at > > >> > > >> > > > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1898) > > >> at > > >> > > >> > > > sun.net.www.protocol.http.HttpURLConnection.access$200(HttpURLConnection.java:92) > > >> at > > >> > > >> > > > sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1492) > > >> at > > >> > > >> > > > sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1490) > > >> at java.security.AccessController.doPrivileged(Native Method) > > >> at > > >> > > >> > > > java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:784) > > >> at > > >> > > >> > > > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1489) > > >> at > > >> > > >> > > > org.apache.hadoop.hbase.http.HttpServerFunctionalTest.readOutput(HttpServerFunctionalTest.java:248) > > >> at > > >> > > >> > > > org.apache.hadoop.hbase.http.TestHttpServer.testJersey(TestHttpServer.java:519) > > >> > > >> > > >>> On Fri, Dec 3, 2021 at 6:23 AM 张铎(Duo Zhang) > > >>> wrote: > > >>> > > >>> Please vote on this Apache hbase thirdparty release candidate, > > >>> hbase-thirdparty-4.0.0RC1 > > >>> > > >>> The VOTE will remain open for at least 72 hours. > > >>> > > >>> [ ] +1 Release this package as Apache hbase thirdparty 4.0.0 > > >>> [ ] -1 Do not release this package because ... > > >>> > > >>> The tag to be voted on is 4.0.0RC1: > > >>> > > >>> https://github.com/apache/hbase-thirdparty/tree/4.0.0RC1 > > >>> > > >>> This tag currently points to git reference > > >>> > > >>> c81cf5c3d10e85f2f5168b09b51ce14468deddbd > > >>> > > >>> The release files, including signatures, digests, as well as > CHANGES.md > > >>> and RELEASENOTES.md included in this RC can be found at: > > >>> > > >>> > > >> > https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty-4.0.0RC1/ > > >>> > > >>> Maven artifacts are available in a staging repository at: > > >>> > > >>> > > >> > https://repository.apache.org/content/repositories/orgapachehbase-1471/ > > >>> > > >>> Artifacts were signed with the 9AD2AE49 key which can be found in: > > >>> > > >>> https://downloads.apache.org/hbase/KEYS > > >>> > > >>> To learn more about Apache hbase thirdparty, please see > > >>> > > >>> http://hbase.apache.org/ > > >>> > > >>> Thanks, > > >>> Your HBase Release Manager > > >>> > > >> > > >> > > >> -- > > >> Best regards, > > >> Andrew > > >> > > >> Words like orphans lost among the crosstalk, meaning torn from truth's > > >> decrepit hands > > >> - A23, Crosstalk > > >> > >
Re: [DISCUSS] releasing hbase-connector 1.1.0 ?
1. to Sean's point of "convenience binaries compiled against Spark 3 as well", sorry that I didn't get your words at the beginning, I will make sure the 1.1.0 release will have compiled binaries for spark 2 and spark 3 (if anything fails spark3, I will file JIRAs for them) 2. (cont.) update on this thread, HBASE-16247 has been sidelined (thanks Sean), HBASE-24356 and HBASE-22338 should be the blockers, and let's see how the communication goes (I may pick up them if needed). In addition, I'm proposing to have a 1.1.0 instead of 1.0.1 release, and move all the removed JIRAs with a fixed version of 1.0.1 to 1.1.0. I should be actively working on them next week before doing the actual release. (sorry that I used a different email in my last reply) -Stephen On Tue, Dec 7, 2021 at 10:29 AM Stephen Wu wrote: > Thanks Sean and Duo for your inputs, I have few questions before we > move forward. > > 1. For Spark 3.0 support, HBASE-25326 seems to already allow building > it with spark-3.0, or doesn't it work? > 2. For the upcoming release version, after reviewing the comments on > HBASE-26491, I found two pending releases, 1.1.0 (3 overall item, 0 > pending issues) and 1.0.1 (46 issues, 3 pending issues, HBASE-22338, > HBASE-25326, HBASE-16247). should we have a 1.0.1 release? > > Thanks, > Stephen > > > On Tue, Dec 7, 2021 at 12:26 AM 张铎(Duo Zhang) > wrote: > > > > Thank you Tak Lon Wu. This is really a great news. > > > > https://issues.apache.org/jira/browse/HBASE-26491 > > > > In this issue, we noticed that the problem has already been fixed but not > > released yet... > > > > It will be good if we could have a 1.1.0 release soon. > > > > Sean Busbey 于2021年12月7日周二 07:45写道： > > > > > That'd be great! > > > > > > I can help with release process. > > > > > > I'd like us to ensure this go around we have convenience binaries > compiled > > > against Spark 3 as well. That's been a long standing need. > > > > > > On Mon, Dec 6, 2021, 16:43 Tak Lon (Stephen) Wu > wrote: > > > > > > > Hi guys, > > > > > > > > While I'm learning how to release a minor version of hbase via > > > > https://hbase.apache.org/book.html#releasing and an example of 2.3.7 > > > > via HBASE-26373, I found hbase-connector was last released on 2019 > > > > April. > > > > > > > > So, I'm wondering if I could help to release hbase-connector e.g. > > > > 1.1.0 and get familiar with the release process, at the same time, I > > > > may need a mentor/adviser and discuss the steps/tasks about it (my > > > > plan is to create JIRAs like HBASE-26373). > > > > > > > > any comments? > > > > > > > > Thanks, > > > > Stephen > > > > > > > >
[jira] [Created] (HBASE-26554) Introduce a new parameter in jmx servlet to exclude the specific mbean
ruanhui created HBASE-26554: --- Summary: Introduce a new parameter in jmx servlet to exclude the specific mbean Key: HBASE-26554 URL: https://issues.apache.org/jira/browse/HBASE-26554 Project: HBase Issue Type: Improvement Components: metrics Affects Versions: 2.4.8 Reporter: ruanhui Assignee: ruanhui Fix For: 3.0.0-alpha-2 There are many regionservers serving over a thousand regions, and the metric load is pretty big. I tried to exclude some huge mbean like 'Hadoop:service=HBase,name=RegionServer,sub=Regions' with regex, but didn't succeed. So I want to propose a new parameter 'excl' in jmx servlet to exclude the splecific bean or beans. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: New dedicated Jenkins Conrtoller
Thanks you Gavin. You can do it in your convenience, just let me know so I could change the label in the jenkinsfile of our jenkins jobs, at least our pre commit job still uses the Hadoop label. And another thing is about the hbase_generate_website job, it needs to be executed on the nodes under the git-websites label. I'm not sure if the label is still available after the migration. Gavin McDonald 于2021年12月8日周三 17:28写道： > Hi HBase devs. > > This is to let you know that Infra has created a dedicated Jenkins > Controller at https://ci-hbase.apache.org . > > You have new incoming donated agents which will get connected to this new > controller by Infra. > > You also have 10 or so agents connected to https://ci-hadoop.apache.org - > I > would like to move these agents - and all of your jobs - over to the new > controller as soon as possible. > > You may or may not need to reconfigure your jobs and or JenkinsFile or > other. > > Can I get the go ahead to move these at my convenience or suggest please a > timeline for Infra to move these agents. > > Thanks! > > -- > > *Gavin McDonald* > Systems Administrator > ASF Infrastructure Team >
[jira] [Created] (HBASE-26553) OAuth Bearer authentication mech plugin for SASL
Andor Molnar created HBASE-26553: Summary: OAuth Bearer authentication mech plugin for SASL Key: HBASE-26553 URL: https://issues.apache.org/jira/browse/HBASE-26553 Project: HBase Issue Type: New Feature Components: security Reporter: Andor Molnar Assignee: Andor Molnar Implementation of a new SASL plugin to add support for OAuth Bearer token authentication for HBase client RPC. * The plugin supports secured (cryptographically signed) JSON Web Token authentication as defined in [RFC-7628|https://datatracker.ietf.org/doc/html/rfc7628] and the JWT format in [RFC-7519|https://datatracker.ietf.org/doc/html/rfc7519] . * The implementation is inspired by [Apache Kafka's OAuth Bearer token|https://docs.confluent.io/platform/current/kafka/authentication_sasl/authentication_sasl_oauth.html] support with the important difference that HBase version is intended for production usage. The two main differences are that Kafka supports unsecured tokens only and it issues the tokens for itself which breaks the principle of OAuth token authentication. * We use the [Nimbus JOSE + JWT|https://bitbucket.org/connect2id/nimbus-jose-jwt/src/master/] Java library for signature verification and token processing and we add it as a new dependency to HBase. * We add secure JWT support and verification of digital signatures with multiple algorithms as supported by Nimbus. Json-formatted JWK set is required for the signature verification as defined in [RFC-7517|https://datatracker.ietf.org/doc/html/rfc7517]. * The impl is verified with Apache Knox issued tokens, because that's the primary use case of this new feature. * New client example is added to the hbase-examples project to showcase the feature. * It's important that this Jira does not cover the solution for obtaining a token from Knox. The assumption is that the client already has a valid token in base64 encoded string and we only provide a helper method for adding it to user's credentials. * Renewing expired tokens is also the responsibility of the client. We don't provide a mechanism for that in this Jira, but it's planned to be covered in a follow-up ticket. The following new parameters are introduced in hbase-site.xml: * hbase.security.oauth.jwt.jwks.file - Path of a local file for JWK set. (required if URL not specified) * hbase.security.oauth.jwt.jwks.url - URL to download the JWK set. (required if File not specified) * hbase.security.oauth.jwt.requiredaudience - Required audience of the JWT. (optional) -- This message was sent by Atlassian Jira (v8.20.1#820001)
