[GitHub] [hbase] Apache-HBase commented on pull request #3421: HBASE-26026 HBase Write may be stuck forever when using CompactingMem…
Apache-HBase commented on pull request #3421: URL: https://github.com/apache/hbase/pull/3421#issuecomment-867352744 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 1s | master passed | | +1 :green_heart: | compile | 3m 11s | master passed | | +1 :green_heart: | checkstyle | 1m 6s | master passed | | +1 :green_heart: | spotbugs | 2m 6s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 38s | the patch passed | | +1 :green_heart: | compile | 3m 7s | the patch passed | | +1 :green_heart: | javac | 3m 7s | the patch passed | | -0 :warning: | checkstyle | 1m 4s | hbase-server: The patch generated 2 new + 74 unchanged - 0 fixed = 76 total (was 74) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 8s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 46m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3421/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3421 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux c363038dd07b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / dcd0fb8b1d | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3421/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 95 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3421/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r657546722
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -319,38 +318,33 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
return true;
}
+sumMultiplier = 0.0f;
double total = 0.0;
-float sumMultiplier = 0.0f;
for (CostFunction c : costFunctions) {
float multiplier = c.getMultiplier();
- if (multiplier <= 0) {
-LOG.trace("{} not needed because multiplier is <= 0",
c.getClass().getSimpleName());
-continue;
- }
+ double cost = c.cost();
if (!c.isNeeded()) {
LOG.trace("{} not needed", c.getClass().getSimpleName());
continue;
}
+ total += cost * multiplier;
sumMultiplier += multiplier;
- total += c.cost() * multiplier;
}
-boolean balanced = total <= 0 || sumMultiplier <= 0 ||
-(sumMultiplier > 0 && (total / sumMultiplier) < minCostNeedBalance);
+boolean balanced = (total / sumMultiplier < minCostNeedBalance);
+
if (balanced) {
final double calculatedTotal = total;
- final double calculatedMultiplier = sumMultiplier;
- sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal,
calculatedMultiplier),
-costFunctions);
-}
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} {}; total cost={}, sum multiplier={}; cost/multiplier to
need a balance is {}",
- balanced ? "Skipping load balancing because balanced" : "We need to
load balance",
- isByTable ? String.format("table (%s)", tableName) : "cluster",
- total, sumMultiplier, minCostNeedBalance);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Balance decision detailed function costs={}",
functionCost());
- }
+ sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal),
costFunctions);
+ LOG.info("{} - skipping load balancing because weighted average
imbalance={} > threshold({})."
+ + "functionCost={}."
+ + "If you want more aggressive balancing, either lower
minCostNeedbalance {}"
+ + "or increase the relative weight(s) of the specific cost
function(s).",
+isByTable ? "Table specific ("+tableName+")" : "Cluster wide",
functionCost(),
+total / sumMultiplier, minCostNeedBalance);
Review comment:
Updated log from test run: Cluster wide - skipping load balancing
because weighted average imbalance=0.02843086355657742 <= threshold(0.05). If
you want more aggressive balancing, either lower
hbase.master.balancer.stochastic.minCostNeedBalance from 0.05 or increase the
relative multiplier(s) of the specific cost function(s).
functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0,
balanced); PrimaryRegionCountSkewCostFunction : (not needed); MoveCostFunction
: (multiplier=7.0, imbalance=0.0, balanced); RackLocalityCostFunction :
(multiplier=15.0, imbalance=0.0, balanced); TableSkewCostFunction :
(multiplier=35.0, imbalance=0.4727646454265159); RegionReplicaHostCostFunction
: (not needed); RegionReplicaRackCostFunction : (not needed);
ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0, balanced);
CPRequestCostFunction : (multiplier=5.0, imbalance=0.0, balanced);
WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0, balanced);
MemStoreSizeCostFun
ction : (multiplier=5.0, imbalance=0.0, balanced); StoreFileCostFunction :
(multiplier=5.0, imbalance=0.0, balanced);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] comnetwork commented on pull request #3421: HBASE-26026 HBase Write may be stuck forever when using CompactingMem…
comnetwork commented on pull request #3421: URL: https://github.com/apache/hbase/pull/3421#issuecomment-867330387 Seems the failed test org.apache.hadoop.hbase.client.TestFromClientSide5 for jdk8 uncorrelated to this PR, jdk11 is ok and TestFromClientSide5 is passed in my local machine with jdk8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-26027) The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException
[
https://issues.apache.org/jira/browse/HBASE-26027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368612#comment-17368612
]
Zheng Wang edited comment on HBASE-26027 at 6/24/21, 4:30 AM:
--
Occur at 2.2.2, and i think 2.x all exists(checked 2.3.3 yet). [~reidchan]
was (Author: filtertip):
Occur at 2.2.2, and i think 2.x all exists(checked 2.3.3 yet).
> The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone
> caused by ArrayStoreException
> -
>
> Key: HBASE-26027
> URL: https://issues.apache.org/jira/browse/HBASE-26027
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
>
> The batch api of HTable contains a param named results to store result or
> exception, its type is Object[].
> If user pass an array with other type, eg:
> org.apache.hadoop.hbase.client.Result, then the ArrayStoreException will
> occur in AsyncRequestFutureImpl.updateResult, then the
> AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
> AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
> actionsInProgress again and again, forever.
> It is better to add an cutoff calculated by operationTimeout, instead of only
> depend on the value of actionsInProgress.
> BTW, this issue only for 2.x, since 3.x the implement has refactored.
> {code:java}
> [ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 -
> id=1 error for test processing localhost,16020,1624343786295
> java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10 actions to
> finish on table:
> [INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10 actions to
> finish on table: test
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26027) The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException
[
https://issues.apache.org/jira/browse/HBASE-26027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368612#comment-17368612
]
Zheng Wang commented on HBASE-26027:
Occur at 2.2.2, and i think 2.x all exists(checked 2.3.3 yet).
> The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone
> caused by ArrayStoreException
> -
>
> Key: HBASE-26027
> URL: https://issues.apache.org/jira/browse/HBASE-26027
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
>
> The batch api of HTable contains a param named results to store result or
> exception, its type is Object[].
> If user pass an array with other type, eg:
> org.apache.hadoop.hbase.client.Result, then the ArrayStoreException will
> occur in AsyncRequestFutureImpl.updateResult, then the
> AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
> AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
> actionsInProgress again and again, forever.
> It is better to add an cutoff calculated by operationTimeout, instead of only
> depend on the value of actionsInProgress.
> BTW, this issue only for 2.x, since 3.x the implement has refactored.
> {code:java}
> [ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 -
> id=1 error for test processing localhost,16020,1624343786295
> java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10 actions to
> finish on table:
> [INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10 actions to
> finish on table: test
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26027) The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException
[
https://issues.apache.org/jira/browse/HBASE-26027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368607#comment-17368607
]
Reid Chan commented on HBASE-26027:
---
which version?
> The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone
> caused by ArrayStoreException
> -
>
> Key: HBASE-26027
> URL: https://issues.apache.org/jira/browse/HBASE-26027
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
>
> The batch api of HTable contains a param named results to store result or
> exception, its type is Object[].
> If user pass an array with other type, eg:
> org.apache.hadoop.hbase.client.Result, then the ArrayStoreException will
> occur in AsyncRequestFutureImpl.updateResult, then the
> AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
> AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
> actionsInProgress again and again, forever.
> It is better to add an cutoff calculated by operationTimeout, instead of only
> depend on the value of actionsInProgress.
> BTW, this issue only for 2.x, since 3.x the implement has refactored.
> {code:java}
> [ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 -
> id=1 error for test processing localhost,16020,1624343786295
> java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10 actions to
> finish on table:
> [INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10 actions to
> finish on table: test
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-26025) Add a flag to mark if the IOError can be solved by retry in thrift IOError
[ https://issues.apache.org/jira/browse/HBASE-26025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-26025: -- Component/s: Thrift > Add a flag to mark if the IOError can be solved by retry in thrift IOError > -- > > Key: HBASE-26025 > URL: https://issues.apache.org/jira/browse/HBASE-26025 > Project: HBase > Issue Type: Improvement > Components: Thrift >Reporter: Yutong Xiao >Assignee: Yutong Xiao >Priority: Major > > Currently, if an HBaseIOException occurs, the thrift client can only get the > error message. This is inconvenient for the client constructing a retry > mechanism to handle the exception. So I added a canRetry mark in IOError to > make the client side exception handling smarter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26022) DNS jitter causes hbase client to get stuck
[
https://issues.apache.org/jira/browse/HBASE-26022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368577#comment-17368577
]
zhuobin zheng commented on HBASE-26022:
---
In *master branch*, it seem like RpcClient will dynamic generate server
principal before create saslClient everyTime. So, it's not a problem.
But it seems to be a problem too in branch-1. I will try to fix it latter.
> DNS jitter causes hbase client to get stuck
> ---
>
> Key: HBASE-26022
> URL: https://issues.apache.org/jira/browse/HBASE-26022
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: zhuobin zheng
>Assignee: zhuobin zheng
>Priority: Major
>
> In our product hbase cluster, we occasionally encounter below errors, and
> stuck hbase a long time. Then hbase requests to this machine will fail
> forever.
> {code:java}
> WARN org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:${user@realm} (auth:KERBEROS)
> cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Server not
> found in Kerberos database (7) - LOOKING_UP_SERVER)]
> WARN org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:${user@realm} (auth:KERBEROS)
> cause:java.io.IOException: Couldn't setup connection for ${user@realm} to
> hbase/${ip}@realm
> {code}
> The main problem is the trully server principal we generated in KDC is
> hbase/*${hostname}*@realm, so we must can't find hbase/*${ip}*@realm in KDC.
> When RpcClientImpl#Connection construct, the field serverPrincial which never
> changed generated by method InetAddress.getCanonicalHostName() which will
> return IP when failed to get hostname.
> Therefor, once DNS jitter when RpcClientImpl#Connection, this connection will
> never setup sasl env. And I'm not see connection abandon logic in sasl failed
> code path.
> I think of two solutions to this problem:
> # Abandon connection when sasl failed. So next request will reconstruct a
> connection, and will regenerate a new server principal.
> # Refresh serverPrincial field when sasl failed. So next retry will use new
> server principal.
> HBase Version: 1.2.0-cdh5.14.4
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] busbey commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
busbey commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r657586765
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -285,14 +286,12 @@ private boolean
areSomeRegionReplicasColocated(BalancerClusterState c) {
return false;
}
- private String getBalanceReason(double total, double sumMultiplier) {
+ private String getBalanceReason(double total) {
if (total <= 0) {
- return "(cost1*multiplier1)+(cost2*multiplier2)+...+(costn*multipliern)
= " + total + " <= 0";
-} else if (sumMultiplier <= 0) {
- return "sumMultiplier = " + sumMultiplier + " <= 0";
+ return "(weighted sum of imbalance = " + total + " <= 0";
} else if ((total / sumMultiplier) < minCostNeedBalance) {
- return
"[(cost1*multiplier1)+(cost2*multiplier2)+...+(costn*multipliern)]/sumMultiplier
= " +
-(total / sumMultiplier) + " <= minCostNeedBalance(" +
minCostNeedBalance + ")";
+ return "(weighted average imbalance = " +
+(total / sumMultiplier) + " < threshold (" + minCostNeedBalance + ")";
Review comment:
If they’re all 0 won’t sumMultiplier be 0 here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26020) Split TestWALEntryStream.testDifferentCounts out
[
https://issues.apache.org/jira/browse/HBASE-26020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368560#comment-17368560
]
Hudson commented on HBASE-26020:
Results for branch branch-2.4
[build #149 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/149/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/149/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/149/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/149/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/149/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Split TestWALEntryStream.testDifferentCounts out
>
>
> Key: HBASE-26020
> URL: https://issues.apache.org/jira/browse/HBASE-26020
> Project: HBase
> Issue Type: Improvement
> Components: Replication, test
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5
>
>
> It consumes too much time and may cause the whole UT to timeout.
> And in fact, it should be implemented as parameterized.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25934) Add username for RegionScannerHolder
[
https://issues.apache.org/jira/browse/HBASE-25934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368559#comment-17368559
]
Hudson commented on HBASE-25934:
Results for branch branch-2.4
[build #149 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/149/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/149/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/149/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/149/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/149/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add username for RegionScannerHolder
>
>
> Key: HBASE-25934
> URL: https://issues.apache.org/jira/browse/HBASE-25934
> Project: HBase
> Issue Type: Wish
>Reporter: tomscut
>Assignee: tomscut
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5
>
>
> This JIRA[HBASE-25542|https://issues.apache.org/jira/browse/HBASE-25542] has
> added part of the client information before, we can also add username for
> RegionScannerHolder.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] clarax commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r657551561
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -423,8 +417,9 @@ private long calculateMaxSteps(BalancerClusterState
cluster) {
maxSteps);
}
}
-LOG.info("start StochasticLoadBalancer.balancer, initCost=" + currentCost
+ ", functionCost="
-+ functionCost() + " computedMaxSteps: " + computedMaxSteps);
+LOG.info("Start StochasticLoadBalancer.balancer, initial weighted average
imbalance ="
Review comment:
Start StochasticLoadBalancer.balancer, initial weighted average
imbalance=0.06013745704467354, functionCost=RegionCountSkewCostFunction :
(multiplier=500.0, imbalance=0.0, balanced); PrimaryRegionCountSkewCostFunction
: (not needed); MoveCostFunction : (multiplier=7.0, imbalance=0.0, balanced);
RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0, balanced);
TableSkewCostFunction : (multiplier=35.0, imbalance=1.0);
RegionReplicaHostCostFunction : (not needed); RegionReplicaRackCostFunction :
(not needed); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0,
balanced); CPRequestCostFunction : (multiplier=5.0, imbalance=0.0, balanced);
WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0, balanced);
MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0, balanced);
StoreFileCostFunction : (multiplier=5.0, imbalance=0.0, balanced);
computedMaxSteps=3200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r657551561
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -423,8 +417,9 @@ private long calculateMaxSteps(BalancerClusterState
cluster) {
maxSteps);
}
}
-LOG.info("start StochasticLoadBalancer.balancer, initCost=" + currentCost
+ ", functionCost="
-+ functionCost() + " computedMaxSteps: " + computedMaxSteps);
+LOG.info("Start StochasticLoadBalancer.balancer, initial weighted average
imbalance ="
Review comment:
Start StochasticLoadBalancer.balancer, initial weighted average
imbalance=0.06013745704467354, functionCost=RegionCountSkewCostFunction :
(multiplier=500.0, imbalance=0.0, balanced); PrimaryRegionCountSkewCostFunction
: (not needed); MoveCostFunction : (multiplier=7.0, imbalance=0.0, balanced);
RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0, balanced);
TableSkewCostFunction : (multiplier=35.0, imbalance=1.0);
RegionReplicaHostCostFunction : (not needed); RegionReplicaRackCostFunction :
(not needed); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0,
balanced); CPRequestCostFunction : (multiplier=5.0, imbalance=0.0, balanced);
WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0, balanced);
MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0, balanced);
StoreFileCostFunction : (multiplier=5.0, imbalance=0.0, balanced); ,
computedMaxSteps=3200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r657550971
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -319,38 +318,33 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
return true;
}
+sumMultiplier = 0.0f;
double total = 0.0;
-float sumMultiplier = 0.0f;
for (CostFunction c : costFunctions) {
float multiplier = c.getMultiplier();
- if (multiplier <= 0) {
-LOG.trace("{} not needed because multiplier is <= 0",
c.getClass().getSimpleName());
-continue;
- }
+ double cost = c.cost();
if (!c.isNeeded()) {
LOG.trace("{} not needed", c.getClass().getSimpleName());
continue;
}
+ total += cost * multiplier;
sumMultiplier += multiplier;
- total += c.cost() * multiplier;
}
-boolean balanced = total <= 0 || sumMultiplier <= 0 ||
-(sumMultiplier > 0 && (total / sumMultiplier) < minCostNeedBalance);
+boolean balanced = (total / sumMultiplier < minCostNeedBalance);
+
if (balanced) {
final double calculatedTotal = total;
- final double calculatedMultiplier = sumMultiplier;
- sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal,
calculatedMultiplier),
-costFunctions);
-}
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} {}; total cost={}, sum multiplier={}; cost/multiplier to
need a balance is {}",
- balanced ? "Skipping load balancing because balanced" : "We need to
load balance",
- isByTable ? String.format("table (%s)", tableName) : "cluster",
- total, sumMultiplier, minCostNeedBalance);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Balance decision detailed function costs={}",
functionCost());
- }
+ sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal),
costFunctions);
+ LOG.info("{} - skipping load balancing because weighted average
imbalance={} > threshold({})."
+ + "functionCost={}."
+ + "If you want more aggressive balancing, either lower
minCostNeedbalance {}"
+ + "or increase the relative weight(s) of the specific cost
function(s).",
+isByTable ? "Table specific ("+tableName+")" : "Cluster wide",
functionCost(),
+total / sumMultiplier, minCostNeedBalance);
+} else {
+ LOG.info("{} - Calculating plan. may take up to {}ms to complete.",
+isByTable ? "Table specific ("+tableName+")" : "Cluster wide",
maxRunningTime);
Review comment:
From test run: Cluster wide - Calculating plan. may take up to 9ms
to complete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r657546722
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -319,38 +318,33 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
return true;
}
+sumMultiplier = 0.0f;
double total = 0.0;
-float sumMultiplier = 0.0f;
for (CostFunction c : costFunctions) {
float multiplier = c.getMultiplier();
- if (multiplier <= 0) {
-LOG.trace("{} not needed because multiplier is <= 0",
c.getClass().getSimpleName());
-continue;
- }
+ double cost = c.cost();
if (!c.isNeeded()) {
LOG.trace("{} not needed", c.getClass().getSimpleName());
continue;
}
+ total += cost * multiplier;
sumMultiplier += multiplier;
- total += c.cost() * multiplier;
}
-boolean balanced = total <= 0 || sumMultiplier <= 0 ||
-(sumMultiplier > 0 && (total / sumMultiplier) < minCostNeedBalance);
+boolean balanced = (total / sumMultiplier < minCostNeedBalance);
+
if (balanced) {
final double calculatedTotal = total;
- final double calculatedMultiplier = sumMultiplier;
- sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal,
calculatedMultiplier),
-costFunctions);
-}
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} {}; total cost={}, sum multiplier={}; cost/multiplier to
need a balance is {}",
- balanced ? "Skipping load balancing because balanced" : "We need to
load balance",
- isByTable ? String.format("table (%s)", tableName) : "cluster",
- total, sumMultiplier, minCostNeedBalance);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Balance decision detailed function costs={}",
functionCost());
- }
+ sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal),
costFunctions);
+ LOG.info("{} - skipping load balancing because weighted average
imbalance={} > threshold({})."
+ + "functionCost={}."
+ + "If you want more aggressive balancing, either lower
minCostNeedbalance {}"
+ + "or increase the relative weight(s) of the specific cost
function(s).",
+isByTable ? "Table specific ("+tableName+")" : "Cluster wide",
functionCost(),
+total / sumMultiplier, minCostNeedBalance);
Review comment:
Updated log from test run: Table specific (table1) - skipping load
balancing because weighted average imbalance=RegionCountSkewCostFunction :
(multiplier=500.0, imbalance=0.0, balanced); PrimaryRegionCountSkewCostFunction
: (not needed); MoveCostFunction : (multiplier=7.0, imbalance=0.0, balanced);
RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0, balanced);
TableSkewCostFunction : (multiplier=35.0, imbalance=1.0);
RegionReplicaHostCostFunction : (not needed); RegionReplicaRackCostFunction :
(not needed); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0,
balanced); CPRequestCostFunction : (multiplier=5.0, imbalance=0.0, balanced);
WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0, balanced);
MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0, balanced);
StoreFileCostFunction : (multiplier=5.0, imbalance=0.0, balanced); >
threshold(0.06013745704467354).functionCost=1.0. If you want more aggressive
balancing, either lower hbase.master.balancer.
stochastic.minCostNeedBalance from {}or increase the relative multiplier(s) of
the specific cost function(s).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3389: HBASE-25392 Direct insert compacted HFiles into data directory.
Apache-HBase commented on pull request #3389: URL: https://github.com/apache/hbase/pull/3389#issuecomment-867249369 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-24749 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 57s | HBASE-24749 passed | | +1 :green_heart: | compile | 1m 0s | HBASE-24749 passed | | +1 :green_heart: | shadedjars | 8m 10s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | HBASE-24749 passed | | -0 :warning: | patch | 9m 0s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 42s | the patch passed | | +1 :green_heart: | compile | 1m 1s | the patch passed | | +1 :green_heart: | javac | 1m 1s | the patch passed | | +1 :green_heart: | shadedjars | 8m 7s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 149m 15s | hbase-server in the patch failed. | | | | 179m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3389/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3389 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f1b4a4dc0a43 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24749 / 49b68b0e00 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3389/5/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3389/5/testReport/ | | Max. process+thread count | 4717 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3389/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3389: HBASE-25392 Direct insert compacted HFiles into data directory.
Apache-HBase commented on pull request #3389: URL: https://github.com/apache/hbase/pull/3389#issuecomment-867247248 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-24749 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 28s | HBASE-24749 passed | | +1 :green_heart: | compile | 1m 11s | HBASE-24749 passed | | +1 :green_heart: | shadedjars | 8m 10s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | HBASE-24749 passed | | -0 :warning: | patch | 9m 5s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 12s | the patch passed | | +1 :green_heart: | compile | 1m 11s | the patch passed | | +1 :green_heart: | javac | 1m 11s | the patch passed | | +1 :green_heart: | shadedjars | 8m 5s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 142m 2s | hbase-server in the patch failed. | | | | 173m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3389/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3389 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d0d900233c03 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24749 / 49b68b0e00 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3389/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3389/5/testReport/ | | Max. process+thread count | 4730 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3389/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r657546722
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -319,38 +318,33 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
return true;
}
+sumMultiplier = 0.0f;
double total = 0.0;
-float sumMultiplier = 0.0f;
for (CostFunction c : costFunctions) {
float multiplier = c.getMultiplier();
- if (multiplier <= 0) {
-LOG.trace("{} not needed because multiplier is <= 0",
c.getClass().getSimpleName());
-continue;
- }
+ double cost = c.cost();
if (!c.isNeeded()) {
LOG.trace("{} not needed", c.getClass().getSimpleName());
continue;
}
+ total += cost * multiplier;
sumMultiplier += multiplier;
- total += c.cost() * multiplier;
}
-boolean balanced = total <= 0 || sumMultiplier <= 0 ||
-(sumMultiplier > 0 && (total / sumMultiplier) < minCostNeedBalance);
+boolean balanced = (total / sumMultiplier < minCostNeedBalance);
+
if (balanced) {
final double calculatedTotal = total;
- final double calculatedMultiplier = sumMultiplier;
- sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal,
calculatedMultiplier),
-costFunctions);
-}
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} {}; total cost={}, sum multiplier={}; cost/multiplier to
need a balance is {}",
- balanced ? "Skipping load balancing because balanced" : "We need to
load balance",
- isByTable ? String.format("table (%s)", tableName) : "cluster",
- total, sumMultiplier, minCostNeedBalance);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Balance decision detailed function costs={}",
functionCost());
- }
+ sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal),
costFunctions);
+ LOG.info("{} - skipping load balancing because weighted average
imbalance={} > threshold({})."
+ + "functionCost={}."
+ + "If you want more aggressive balancing, either lower
minCostNeedbalance {}"
+ + "or increase the relative weight(s) of the specific cost
function(s).",
+isByTable ? "Table specific ("+tableName+")" : "Cluster wide",
functionCost(),
+total / sumMultiplier, minCostNeedBalance);
Review comment:
Updated log from test run: skipping load balancing because weighted
average imbalance=RegionCountSkewCostFunction : (multiplier=500.0,
imbalance=0.0, balanced); PrimaryRegionCountSkewCostFunction : (not needed);
MoveCostFunction : (multiplier=7.0, imbalance=0.0, balanced);
RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0, balanced);
TableSkewCostFunction : (multiplier=35.0, imbalance=1.0);
RegionReplicaHostCostFunction : (not needed); RegionReplicaRackCostFunction :
(not needed); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0,
balanced); CPRequestCostFunction : (multiplier=5.0, imbalance=0.0, balanced);
WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0, balanced);
MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0, balanced);
StoreFileCostFunction : (multiplier=5.0, imbalance=0.0, balanced); >
threshold(0.06013745704467354).functionCost=1.0. If you want more aggressive
balancing, either lower hbase.master.balancer.stochastic.minCostNeedBala
nce from {}or increase the relative multiplier(s) of the specific cost
function(s).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r657546722
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -319,38 +318,33 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
return true;
}
+sumMultiplier = 0.0f;
double total = 0.0;
-float sumMultiplier = 0.0f;
for (CostFunction c : costFunctions) {
float multiplier = c.getMultiplier();
- if (multiplier <= 0) {
-LOG.trace("{} not needed because multiplier is <= 0",
c.getClass().getSimpleName());
-continue;
- }
+ double cost = c.cost();
if (!c.isNeeded()) {
LOG.trace("{} not needed", c.getClass().getSimpleName());
continue;
}
+ total += cost * multiplier;
sumMultiplier += multiplier;
- total += c.cost() * multiplier;
}
-boolean balanced = total <= 0 || sumMultiplier <= 0 ||
-(sumMultiplier > 0 && (total / sumMultiplier) < minCostNeedBalance);
+boolean balanced = (total / sumMultiplier < minCostNeedBalance);
+
if (balanced) {
final double calculatedTotal = total;
- final double calculatedMultiplier = sumMultiplier;
- sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal,
calculatedMultiplier),
-costFunctions);
-}
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} {}; total cost={}, sum multiplier={}; cost/multiplier to
need a balance is {}",
- balanced ? "Skipping load balancing because balanced" : "We need to
load balance",
- isByTable ? String.format("table (%s)", tableName) : "cluster",
- total, sumMultiplier, minCostNeedBalance);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Balance decision detailed function costs={}",
functionCost());
- }
+ sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal),
costFunctions);
+ LOG.info("{} - skipping load balancing because weighted average
imbalance={} > threshold({})."
+ + "functionCost={}."
+ + "If you want more aggressive balancing, either lower
minCostNeedbalance {}"
+ + "or increase the relative weight(s) of the specific cost
function(s).",
+isByTable ? "Table specific ("+tableName+")" : "Cluster wide",
functionCost(),
+total / sumMultiplier, minCostNeedBalance);
Review comment:
Updated log from test run: skipping load balancing because weighted
average imbalance=RegionCountSkewCostFunction : (multiplier=500.0,
imbalance=0.0, balanced); PrimaryRegionCountSkewCostFunction : (not needed);
MoveCostFunction : (multiplier=7.0, imbalance=0.0, balanced);
RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0, balanced);
TableSkewCostFunction : (multiplier=35.0, imbalance=1.0);
RegionReplicaHostCostFunction : (not needed); RegionReplicaRackCostFunction :
(not needed); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0,
balanced); CPRequestCostFunction : (multiplier=5.0, imbalance=0.0, balanced);
WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0, balanced);
MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0, balanced);
StoreFileCostFunction : (multiplier=5.0, imbalance=0.0, balanced); >
threshold(0.06013745704467354).functionCost=1.0. If you want more aggressive
balancing, either lower hbase.master.balancer.stochastic.minCostNeedBala
nce from {}or increase the relative multiplier(s) of the specific cost
function(s)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26021) HBase 1.7 to 2.4 upgrade issue
[
https://issues.apache.org/jira/browse/HBASE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368532#comment-17368532
]
Andrew Kyle Purtell commented on HBASE-26021:
-
We can't undo, because 1.7.0 has been released with this change included.
Unless it actually prevents an upgrade from 1.6, which moots the release and we
will have to withdraw it.
The patch replaces HTableDescriptor with TableDescriptor in FSTableDescriptors.
Diffing the serialization details of those two classes should do it.
> HBase 1.7 to 2.4 upgrade issue
> --
>
> Key: HBASE-26021
> URL: https://issues.apache.org/jira/browse/HBASE-26021
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.7.0, 2.4.4
>Reporter: Viraj Jasani
>Priority: Major
> Attachments: Screenshot 2021-06-22 at 12.54.21 PM.png, Screenshot
> 2021-06-22 at 12.54.30 PM.png
>
>
> As of today, if we bring up HBase cluster using branch-1 and upgrade to
> branch-2.4, we are facing issue while parsing namespace from HDFS fileinfo.
> Instead of "*hbase:meta*" and "*hbase:namespace*", parsing using ProtobufUtil
> seems to be producing "*\n hbase:\n meta*" and "*\n hbase:\n namespace*"
> {code:java}
> 2021-06-22 00:05:56,611 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> regionserver.RSRpcServices: Open hbase:meta,,1.1588230740
> 2021-06-22 00:05:56,648 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> regionserver.RSRpcServices: Open
> hbase:namespace,,1624297762817.396cb6cc00cd4334cb1ea3a792d7529a.
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R namespace
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
> at org.apache.hadoop.hbase.TableName.(TableName.java:348)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2292)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableDescriptor(ProtobufUtil.java:2937)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.parseFrom(TableDescriptorBuilder.java:1625)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.access$200(TableDescriptorBuilder.java:597)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder.parseFrom(TableDescriptorBuilder.java:320)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:511)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:496)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:482)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:210)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:2112)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:35218)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R^Dmeta
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
> at org.apache.hadoop.hbase.TableName.(TableName.java:348)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2292)
> at
>
[GitHub] [hbase] clarax commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r657532632
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -423,8 +417,9 @@ private long calculateMaxSteps(BalancerClusterState
cluster) {
maxSteps);
}
}
-LOG.info("start StochasticLoadBalancer.balancer, initCost=" + currentCost
+ ", functionCost="
-+ functionCost() + " computedMaxSteps: " + computedMaxSteps);
+LOG.info("Start StochasticLoadBalancer.balancer, initial weighted average
imbalance ="
+ + currentCost / sumMultiplier + ", functionCost=" + functionCost() + "
computedMaxSteps: "
+ + computedMaxSteps);
Review comment:
No, it is the weighted sum.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r657532327
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -319,38 +318,33 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
return true;
}
+sumMultiplier = 0.0f;
double total = 0.0;
-float sumMultiplier = 0.0f;
for (CostFunction c : costFunctions) {
float multiplier = c.getMultiplier();
- if (multiplier <= 0) {
-LOG.trace("{} not needed because multiplier is <= 0",
c.getClass().getSimpleName());
-continue;
- }
+ double cost = c.cost();
if (!c.isNeeded()) {
LOG.trace("{} not needed", c.getClass().getSimpleName());
continue;
}
+ total += cost * multiplier;
sumMultiplier += multiplier;
- total += c.cost() * multiplier;
}
-boolean balanced = total <= 0 || sumMultiplier <= 0 ||
-(sumMultiplier > 0 && (total / sumMultiplier) < minCostNeedBalance);
+boolean balanced = (total / sumMultiplier < minCostNeedBalance);
+
Review comment:
see above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r657532279
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -285,14 +286,12 @@ private boolean
areSomeRegionReplicasColocated(BalancerClusterState c) {
return false;
}
- private String getBalanceReason(double total, double sumMultiplier) {
+ private String getBalanceReason(double total) {
if (total <= 0) {
- return "(cost1*multiplier1)+(cost2*multiplier2)+...+(costn*multipliern)
= " + total + " <= 0";
-} else if (sumMultiplier <= 0) {
- return "sumMultiplier = " + sumMultiplier + " <= 0";
+ return "(weighted sum of imbalance = " + total + " <= 0";
} else if ((total / sumMultiplier) < minCostNeedBalance) {
- return
"[(cost1*multiplier1)+(cost2*multiplier2)+...+(costn*multipliern)]/sumMultiplier
= " +
-(total / sumMultiplier) + " <= minCostNeedBalance(" +
minCostNeedBalance + ")";
+ return "(weighted average imbalance = " +
+(total / sumMultiplier) + " < threshold (" + minCostNeedBalance + ")";
Review comment:
If it is 0, the cost function is excluded at constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] clarax commented on pull request #3415: HBASE-25739 TableSkewCostFunction need to use aggregated deviation
clarax commented on pull request #3415: URL: https://github.com/apache/hbase/pull/3415#issuecomment-867227046 Enabling runMaxStep and increasing max run time for TestStochasticBalancerLargeCluster takes care of the flaky tests. I removed the increase for max run time for TestStochasticBalancerBalanceCluster since it is not really needed and increase total rest run time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3415: HBASE-25739 TableSkewCostFunction need to use aggregated deviation
Apache-HBase commented on pull request #3415: URL: https://github.com/apache/hbase/pull/3415#issuecomment-867203377 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 3s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 58s | master passed | | +1 :green_heart: | compile | 0m 35s | master passed | | +1 :green_heart: | checkstyle | 0m 17s | master passed | | +1 :green_heart: | spotbugs | 0m 36s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 36s | the patch passed | | +1 :green_heart: | compile | 0m 32s | the patch passed | | -0 :warning: | javac | 0m 32s | hbase-balancer generated 1 new + 17 unchanged - 0 fixed = 18 total (was 17) | | -0 :warning: | checkstyle | 0m 15s | hbase-balancer: The patch generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 11s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 0m 44s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 37m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3415 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 7227782187c7 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / dcd0fb8b1d | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/5/artifact/yetus-general-check/output/diff-compile-javac-hbase-balancer.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-balancer.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-balancer U: hbase-balancer | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/5/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3389: HBASE-25392 Direct insert compacted HFiles into data directory.
Apache-HBase commented on pull request #3389: URL: https://github.com/apache/hbase/pull/3389#issuecomment-867203325 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-24749 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 44s | HBASE-24749 passed | | +1 :green_heart: | compile | 3m 14s | HBASE-24749 passed | | +1 :green_heart: | checkstyle | 1m 8s | HBASE-24749 passed | | +1 :green_heart: | spotbugs | 2m 2s | HBASE-24749 passed | | -0 :warning: | patch | 2m 11s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 36s | the patch passed | | +1 :green_heart: | compile | 3m 14s | the patch passed | | -0 :warning: | javac | 3m 14s | hbase-server generated 1 new + 192 unchanged - 1 fixed = 193 total (was 193) | | -0 :warning: | checkstyle | 1m 6s | hbase-server: The patch generated 3 new + 204 unchanged - 2 fixed = 207 total (was 206) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 4s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 46m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3389/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3389 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 6c044b10ec31 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24749 / 49b68b0e00 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3389/5/artifact/yetus-general-check/output/diff-compile-javac-hbase-server.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3389/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3389/5/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3415: HBASE-25739 TableSkewCostFunction need to use aggregated deviation
Apache-HBase commented on pull request #3415: URL: https://github.com/apache/hbase/pull/3415#issuecomment-867202693 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 30s | master passed | | +1 :green_heart: | compile | 0m 21s | master passed | | +1 :green_heart: | shadedjars | 8m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 21s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 19s | the patch passed | | +1 :green_heart: | compile | 0m 21s | the patch passed | | +1 :green_heart: | javac | 0m 21s | the patch passed | | +1 :green_heart: | shadedjars | 8m 8s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 18s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 8m 14s | hbase-balancer in the patch passed. | | | | 36m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3415 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d4c3c1ad5fdf 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / dcd0fb8b1d | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/5/testReport/ | | Max. process+thread count | 320 (vs. ulimit of 3) | | modules | C: hbase-balancer U: hbase-balancer | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3415: HBASE-25739 TableSkewCostFunction need to use aggregated deviation
Apache-HBase commented on pull request #3415: URL: https://github.com/apache/hbase/pull/3415#issuecomment-867201761 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 35s | master passed | | +1 :green_heart: | compile | 0m 20s | master passed | | +1 :green_heart: | shadedjars | 8m 10s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 16s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 40s | the patch passed | | +1 :green_heart: | compile | 0m 23s | the patch passed | | +1 :green_heart: | javac | 0m 23s | the patch passed | | +1 :green_heart: | shadedjars | 8m 15s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 7m 24s | hbase-balancer in the patch passed. | | | | 34m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3415 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 541f14ce6119 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / dcd0fb8b1d | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/5/testReport/ | | Max. process+thread count | 355 (vs. ulimit of 3) | | modules | C: hbase-balancer U: hbase-balancer | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #3389: HBASE-25392 Direct insert compacted HFiles into data directory.
wchevreuil commented on a change in pull request #3389:
URL: https://github.com/apache/hbase/pull/3389#discussion_r657484035
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DirectInStoreCompactor.java
##
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver.compactions;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.compress.Compression;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.regionserver.HStore;
+import org.apache.hadoop.hbase.regionserver.HStoreFile;
+import org.apache.hadoop.hbase.regionserver.StoreFileWriter;
+import org.apache.yetus.audience.InterfaceAudience;
+
+@InterfaceAudience.Private
+public class DirectInStoreCompactor extends DefaultCompactor {
+ public DirectInStoreCompactor(Configuration conf, HStore store) {
+super(conf, store);
+ }
+
+ @Override
+ protected StoreFileWriter initWriter(FileDetails fd, boolean
shouldDropBehind, boolean major)
+throws IOException {
+// When all MVCC readpoints are 0, don't write them.
+// See HBASE-8166, HBASE-12600, and HBASE-13389.
+return createWriterInFamilyDir(fd.maxKeyCount,
+ major ? majorCompactionCompression : minorCompactionCompression,
+ fd.maxMVCCReadpoint > 0, fd.maxTagsLength > 0, shouldDropBehind);
+ }
+
+ private StoreFileWriter createWriterInFamilyDir(long maxKeyCount,
+ Compression.Algorithm compression, boolean includeMVCCReadpoint, boolean
includesTag,
+boolean shouldDropBehind) throws IOException {
+final CacheConfig writerCacheConf;
+// Don't cache data on write on compactions.
+writerCacheConf = new CacheConfig(store.getCacheConfig());
+writerCacheConf.setCacheDataOnWrite(false);
+
+InetSocketAddress[] favoredNodes = null;
+if (store.getHRegion().getRegionServerServices() != null) {
+ favoredNodes =
store.getHRegion().getRegionServerServices().getFavoredNodesForRegion(
+store.getHRegion().getRegionInfo().getEncodedName());
Review comment:
Wasn't finding `store.getStoreContext()` before, since it was package
private. Made it public on HStore and am calling it there now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #3389: HBASE-25392 Direct insert compacted HFiles into data directory.
wchevreuil commented on a change in pull request #3389:
URL: https://github.com/apache/hbase/pull/3389#discussion_r657483340
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DirectStoreCompactor.java
##
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver.compactions;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.compress.Compression;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.regionserver.HStore;
+import org.apache.hadoop.hbase.regionserver.HStoreFile;
+import org.apache.hadoop.hbase.regionserver.StoreFileWriter;
+import org.apache.yetus.audience.InterfaceAudience;
+
+@InterfaceAudience.Private
+public class DirectStoreCompactor extends DefaultCompactor {
+ public DirectStoreCompactor(Configuration conf, HStore store) {
+super(conf, store);
+ }
+
+ @Override
+ protected StoreFileWriter initWriter(FileDetails fd, boolean
shouldDropBehind, boolean major)
+throws IOException {
+// When all MVCC readpoints are 0, don't write them.
+// See HBASE-8166, HBASE-12600, and HBASE-13389.
+return createWriterInFamilyDir(fd.maxKeyCount,
+ major ? majorCompactionCompression : minorCompactionCompression,
+ fd.maxMVCCReadpoint > 0, fd.maxTagsLength > 0,
+ shouldDropBehind, fd.totalCompactedFilesSize);
+ }
+
+ private StoreFileWriter createWriterInFamilyDir(long maxKeyCount,
+ Compression.Algorithm compression, boolean includeMVCCReadpoint, boolean
includesTag,
+boolean shouldDropBehind, long totalCompactedFilesSize) throws
IOException {
+final CacheConfig writerCacheConf;
+// Don't cache data on write on compactions.
Review comment:
Removed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #3389: HBASE-25392 Direct insert compacted HFiles into data directory.
wchevreuil commented on a change in pull request #3389:
URL: https://github.com/apache/hbase/pull/3389#discussion_r657483258
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
##
@@ -297,6 +297,30 @@ public void enableCacheOnWrite() {
this.cacheBloomsOnWrite = true;
}
+ /**
+ * If hbase.rs.cachecompactedblocksonwrite configuration is set to true and
+ * 'totalCompactedFilesSize' is lower than
'cacheCompactedDataOnWriteThreshold',
+ * enables cache on write for below properties:
+ * - cacheDataOnWrite
+ * - cacheIndexesOnWrite
+ * - cacheBloomsOnWrite
+ *
+ * Otherwise, sets 'cacheDataOnWrite' only to false.
+ *
+ * @param totalCompactedFilesSize the total size of compacted files.
+ * @return true if the checks mentioned above pass and the cache is enabled,
false otherwise.
+ */
+ public boolean enableCacheOnWrite(long totalCompactedFilesSize) {
Review comment:
Renamed it on last commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3415: HBASE-25739 TableSkewCostFunction need to use aggregated deviation
Apache-HBase commented on pull request #3415: URL: https://github.com/apache/hbase/pull/3415#issuecomment-867180485 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 59s | master passed | | +1 :green_heart: | compile | 0m 21s | master passed | | +1 :green_heart: | shadedjars | 8m 11s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 18s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 41s | the patch passed | | +1 :green_heart: | compile | 0m 21s | the patch passed | | +1 :green_heart: | javac | 0m 21s | the patch passed | | +1 :green_heart: | shadedjars | 8m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 16s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 16m 51s | hbase-balancer in the patch failed. | | | | 43m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3415 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f8373defdd0a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / dcd0fb8b1d | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-balancer.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/4/testReport/ | | Max. process+thread count | 355 (vs. ulimit of 3) | | modules | C: hbase-balancer U: hbase-balancer | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3415: HBASE-25739 TableSkewCostFunction need to use aggregated deviation
Apache-HBase commented on pull request #3415: URL: https://github.com/apache/hbase/pull/3415#issuecomment-867179837 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 18s | master passed | | +1 :green_heart: | compile | 0m 22s | master passed | | +1 :green_heart: | shadedjars | 8m 11s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 18s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 16s | the patch passed | | +1 :green_heart: | compile | 0m 21s | the patch passed | | +1 :green_heart: | javac | 0m 21s | the patch passed | | +1 :green_heart: | shadedjars | 8m 15s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 18s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 14m 16s | hbase-balancer in the patch passed. | | | | 42m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3415 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 14d22bf0b811 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / dcd0fb8b1d | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/4/testReport/ | | Max. process+thread count | 323 (vs. ulimit of 3) | | modules | C: hbase-balancer U: hbase-balancer | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3415: HBASE-25739 TableSkewCostFunction need to use aggregated deviation
Apache-HBase commented on pull request #3415: URL: https://github.com/apache/hbase/pull/3415#issuecomment-867179131 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 4s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 19s | master passed | | +1 :green_heart: | compile | 0m 32s | master passed | | +1 :green_heart: | checkstyle | 0m 14s | master passed | | +1 :green_heart: | spotbugs | 0m 35s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 0s | the patch passed | | +1 :green_heart: | compile | 0m 29s | the patch passed | | -0 :warning: | javac | 0m 29s | hbase-balancer generated 1 new + 17 unchanged - 0 fixed = 18 total (was 17) | | -0 :warning: | checkstyle | 0m 12s | hbase-balancer: The patch generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 57s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 0m 43s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 40m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3415 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux e450ef69683e 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / dcd0fb8b1d | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/4/artifact/yetus-general-check/output/diff-compile-javac-hbase-balancer.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-balancer.txt | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-balancer U: hbase-balancer | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26021) HBase 1.7 to 2.4 upgrade issue
[
https://issues.apache.org/jira/browse/HBASE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368469#comment-17368469
]
Bharath Vissapragada commented on HBASE-26021:
--
[~vjasani] Thanks for digging. I suspected that commit could cause some edge
cases like this. The commit in it's exact form is not needed for the work IIRC
(it diverged the code quite a bit, so had to backport it), so we can undo
certain parts of the serialization that is incompatible with branch-1. You
happen to dig into what exactly is off in the serialization piece?
> HBase 1.7 to 2.4 upgrade issue
> --
>
> Key: HBASE-26021
> URL: https://issues.apache.org/jira/browse/HBASE-26021
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.7.0, 2.4.4
>Reporter: Viraj Jasani
>Priority: Major
> Attachments: Screenshot 2021-06-22 at 12.54.21 PM.png, Screenshot
> 2021-06-22 at 12.54.30 PM.png
>
>
> As of today, if we bring up HBase cluster using branch-1 and upgrade to
> branch-2.4, we are facing issue while parsing namespace from HDFS fileinfo.
> Instead of "*hbase:meta*" and "*hbase:namespace*", parsing using ProtobufUtil
> seems to be producing "*\n hbase:\n meta*" and "*\n hbase:\n namespace*"
> {code:java}
> 2021-06-22 00:05:56,611 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> regionserver.RSRpcServices: Open hbase:meta,,1.1588230740
> 2021-06-22 00:05:56,648 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> regionserver.RSRpcServices: Open
> hbase:namespace,,1624297762817.396cb6cc00cd4334cb1ea3a792d7529a.
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R namespace
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
> at org.apache.hadoop.hbase.TableName.(TableName.java:348)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2292)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableDescriptor(ProtobufUtil.java:2937)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.parseFrom(TableDescriptorBuilder.java:1625)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.access$200(TableDescriptorBuilder.java:597)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder.parseFrom(TableDescriptorBuilder.java:320)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:511)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:496)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:482)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:210)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:2112)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:35218)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R^Dmeta
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
> at org.apache.hadoop.hbase.TableName.(TableName.java:348)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2292)
>
[GitHub] [hbase] busbey commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
busbey commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r657441767
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -319,38 +318,33 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
return true;
}
+sumMultiplier = 0.0f;
double total = 0.0;
-float sumMultiplier = 0.0f;
for (CostFunction c : costFunctions) {
float multiplier = c.getMultiplier();
- if (multiplier <= 0) {
-LOG.trace("{} not needed because multiplier is <= 0",
c.getClass().getSimpleName());
-continue;
- }
+ double cost = c.cost();
if (!c.isNeeded()) {
LOG.trace("{} not needed", c.getClass().getSimpleName());
continue;
}
+ total += cost * multiplier;
sumMultiplier += multiplier;
- total += c.cost() * multiplier;
}
-boolean balanced = total <= 0 || sumMultiplier <= 0 ||
-(sumMultiplier > 0 && (total / sumMultiplier) < minCostNeedBalance);
+boolean balanced = (total / sumMultiplier < minCostNeedBalance);
+
Review comment:
Do we have a guard somewhere against all the multipliers being 0?
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -285,14 +286,12 @@ private boolean
areSomeRegionReplicasColocated(BalancerClusterState c) {
return false;
}
- private String getBalanceReason(double total, double sumMultiplier) {
+ private String getBalanceReason(double total) {
if (total <= 0) {
- return "(cost1*multiplier1)+(cost2*multiplier2)+...+(costn*multipliern)
= " + total + " <= 0";
-} else if (sumMultiplier <= 0) {
- return "sumMultiplier = " + sumMultiplier + " <= 0";
+ return "(weighted sum of imbalance = " + total + " <= 0";
} else if ((total / sumMultiplier) < minCostNeedBalance) {
- return
"[(cost1*multiplier1)+(cost2*multiplier2)+...+(costn*multipliern)]/sumMultiplier
= " +
-(total / sumMultiplier) + " <= minCostNeedBalance(" +
minCostNeedBalance + ")";
+ return "(weighted average imbalance = " +
+(total / sumMultiplier) + " < threshold (" + minCostNeedBalance + ")";
Review comment:
Do we have a guard somewhere against all the multipliers being 0?
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -423,8 +417,9 @@ private long calculateMaxSteps(BalancerClusterState
cluster) {
maxSteps);
}
}
-LOG.info("start StochasticLoadBalancer.balancer, initCost=" + currentCost
+ ", functionCost="
-+ functionCost() + " computedMaxSteps: " + computedMaxSteps);
+LOG.info("Start StochasticLoadBalancer.balancer, initial weighted average
imbalance ="
+ + currentCost / sumMultiplier + ", functionCost=" + functionCost() + "
computedMaxSteps: "
+ + computedMaxSteps);
Review comment:
why is `currentCost` divided by `sumMultiplier` here? isn't
`currentCost` by itself already the weighted average measure of imbalance?
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -319,38 +318,33 @@ boolean needsBalance(TableName tableName,
BalancerClusterState cluster) {
return true;
}
+sumMultiplier = 0.0f;
double total = 0.0;
-float sumMultiplier = 0.0f;
for (CostFunction c : costFunctions) {
float multiplier = c.getMultiplier();
- if (multiplier <= 0) {
-LOG.trace("{} not needed because multiplier is <= 0",
c.getClass().getSimpleName());
-continue;
- }
+ double cost = c.cost();
if (!c.isNeeded()) {
LOG.trace("{} not needed", c.getClass().getSimpleName());
continue;
}
+ total += cost * multiplier;
sumMultiplier += multiplier;
- total += c.cost() * multiplier;
}
-boolean balanced = total <= 0 || sumMultiplier <= 0 ||
-(sumMultiplier > 0 && (total / sumMultiplier) < minCostNeedBalance);
+boolean balanced = (total / sumMultiplier < minCostNeedBalance);
+
if (balanced) {
final double calculatedTotal = total;
- final double calculatedMultiplier = sumMultiplier;
- sendRejectionReasonToRingBuffer(() -> getBalanceReason(calculatedTotal,
calculatedMultiplier),
-costFunctions);
-}
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} {}; total cost={}, sum multiplier={}; cost/multiplier to
need a balance is {}",
- balanced ? "Skipping load balancing because balanced" : "We need to
load balance",
- isByTable ? String.format("table (%s)", tableName) : "cluster",
- total, sumMultiplier, minCostNeedBalance);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Balance decision
[GitHub] [hbase] Apache-HBase commented on pull request #3421: HBASE-26026 HBase Write may be stuck forever when using CompactingMem…
Apache-HBase commented on pull request #3421: URL: https://github.com/apache/hbase/pull/3421#issuecomment-867071115 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 41s | master passed | | +1 :green_heart: | compile | 1m 51s | master passed | | +1 :green_heart: | shadedjars | 9m 42s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 15s | the patch passed | | +1 :green_heart: | compile | 1m 32s | the patch passed | | +1 :green_heart: | javac | 1m 32s | the patch passed | | +1 :green_heart: | shadedjars | 10m 4s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 49s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 199m 54s | hbase-server in the patch passed. | | | | 238m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3421/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3421 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e113842c053d 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / cb247f9464 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3421/1/testReport/ | | Max. process+thread count | 2355 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3421/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26007) java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI
[ https://issues.apache.org/jira/browse/HBASE-26007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368401#comment-17368401 ] Josh Elser commented on HBASE-26007: Decreased priority as there's no ability to currently debug this farther. If you have more information you can provide, Venkat, perhaps someone can help you look into it. > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > --- > > Key: HBASE-26007 > URL: https://issues.apache.org/jira/browse/HBASE-26007 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.3.5 >Reporter: Venkat A >Priority: Major > > Hi All, > We have Hadoop 3.2.2 and HBase 2.3.5 Versions installed. (java version > "1.8.0_291") > > While bringing up HBase master, I'm seeing following error messages in HBase > master log. > > Other HDFS. clients like Spark,MapReduce,Solr etc are able to write HDFS but > HBase is unable to write its meta files in HDFS with following exceptions. > > > Summary of Error logs from hbase master > 2021-06-15 03:57:45,968 INFO [Thread-7] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:45,939 WARN [Thread-7] hdfs.DataStreamer: Abandoning > BP-1583998547-10.10.10.3-1622148262434:blk_1073743393_2570 > 2021-06-15 03:57:45,946 WARN [Thread-7] hdfs.DataStreamer: Excluding > datanode > DatanodeInfoWithStorage[10.10.10.3:50010,DS-281c3377-2bc1-47ea-8302-43108ee69430,DISK] > 2021-06-15 03:57:45,994 WARN [Thread-7] hdfs.DataStreamer: DataStreamer > Exception > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /hbase/data/data/hbase/meta/.tmp/.tableinfo.01 could only be written > to 0 of the 1 minReplication nodes. There are 3 datanode(s) running and 3 > node(s) are excluded in this operation. > 2021-06-15 03:57:46,023 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > 2021-06-15 03:57:46,035 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:46,508 ERROR [main] regionserver.HRegionServer: Failed > construction RegionServer > java.io.IOException: Failed update hbase:meta table descriptor > 2021-06-15 03:57:46,509 ERROR [main] master.HMasterCommandLine: Master > exiting > java.lang.RuntimeException: Failed construction of Master: class > org.apache.hadoop.hbase.master.HMaster. > Caused by: java.io.IOException: Failed update hbase:meta table descriptor > > Not sure what is the root cause behind this. Any comments/suggestions on this > is much appreciated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26007) java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI
[ https://issues.apache.org/jira/browse/HBASE-26007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-26007: --- Priority: Major (was: Blocker) > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > --- > > Key: HBASE-26007 > URL: https://issues.apache.org/jira/browse/HBASE-26007 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.3.5 >Reporter: Venkat A >Priority: Major > > Hi All, > We have Hadoop 3.2.2 and HBase 2.3.5 Versions installed. (java version > "1.8.0_291") > > While bringing up HBase master, I'm seeing following error messages in HBase > master log. > > Other HDFS. clients like Spark,MapReduce,Solr etc are able to write HDFS but > HBase is unable to write its meta files in HDFS with following exceptions. > > > Summary of Error logs from hbase master > 2021-06-15 03:57:45,968 INFO [Thread-7] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:45,939 WARN [Thread-7] hdfs.DataStreamer: Abandoning > BP-1583998547-10.10.10.3-1622148262434:blk_1073743393_2570 > 2021-06-15 03:57:45,946 WARN [Thread-7] hdfs.DataStreamer: Excluding > datanode > DatanodeInfoWithStorage[10.10.10.3:50010,DS-281c3377-2bc1-47ea-8302-43108ee69430,DISK] > 2021-06-15 03:57:45,994 WARN [Thread-7] hdfs.DataStreamer: DataStreamer > Exception > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /hbase/data/data/hbase/meta/.tmp/.tableinfo.01 could only be written > to 0 of the 1 minReplication nodes. There are 3 datanode(s) running and 3 > node(s) are excluded in this operation. > 2021-06-15 03:57:46,023 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > 2021-06-15 03:57:46,035 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:46,508 ERROR [main] regionserver.HRegionServer: Failed > construction RegionServer > java.io.IOException: Failed update hbase:meta table descriptor > 2021-06-15 03:57:46,509 ERROR [main] master.HMasterCommandLine: Master > exiting > java.lang.RuntimeException: Failed construction of Master: class > org.apache.hadoop.hbase.master.HMaster. > Caused by: java.io.IOException: Failed update hbase:meta table descriptor > > Not sure what is the root cause behind this. Any comments/suggestions on this > is much appreciated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25997) NettyRpcFrameDecoder decode request header wrong when handleTooBigRequest
[
https://issues.apache.org/jira/browse/HBASE-25997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368398#comment-17368398
]
Josh Elser commented on HBASE-25997:
bq. 2.3.x also need it，Push to branch-2.3 also.
I've added fixVersion=2.3.6 here :)
> NettyRpcFrameDecoder decode request header wrong when handleTooBigRequest
> --
>
> Key: HBASE-25997
> URL: https://issues.apache.org/jira/browse/HBASE-25997
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Affects Versions: 2.4.4
>Reporter: Lijin Bin
>Assignee: Lijin Bin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.6, 2.4.5
>
>
> Client write a big request to server, server decode request wrong, so client
> do not get a RequestTooBigException as expected.
> {code}
> 2021-06-11 18:57:27,340 INFO [RS-EventLoopGroup-1-20] ipc.NettyRpcServer:
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException$InvalidWireTypeException:
> Protocol message tag had invalid wire type.
> at
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException.invalidWireType(InvalidProtocolBufferException.java:111)
> at
> org.apache.hbase.thirdparty.com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:519)
> at
> org.apache.hbase.thirdparty.com.google.protobuf.GeneratedMessageV3.parseUnknownField(GeneratedMessageV3.java:298)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RPCProtos$RequestHeader.(RPCProtos.java:5958)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RPCProtos$RequestHeader.(RPCProtos.java:5916)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RPCProtos$RequestHeader$1.parsePartialFrom(RPCProtos.java:7249)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RPCProtos$RequestHeader$1.parsePartialFrom(RPCProtos.java:7244)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RPCProtos$RequestHeader$Builder.mergeFrom(RPCProtos.java:6679)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RPCProtos$RequestHeader$Builder.mergeFrom(RPCProtos.java:6482)
> at
> org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:420)
> at
> org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:317)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.mergeFrom(ProtobufUtil.java:2716)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcFrameDecoder.getHeader(NettyRpcFrameDecoder.java:174)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcFrameDecoder.handleTooBigRequest(NettyRpcFrameDecoder.java:126)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcFrameDecoder.decode(NettyRpcFrameDecoder.java:65)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:405)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:372)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:355)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:242)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:228)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:221)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1403)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:242)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:228)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:912)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:827)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
> at
>
[jira] [Updated] (HBASE-25997) NettyRpcFrameDecoder decode request header wrong when handleTooBigRequest
[
https://issues.apache.org/jira/browse/HBASE-25997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-25997:
---
Fix Version/s: 2.3.6
> NettyRpcFrameDecoder decode request header wrong when handleTooBigRequest
> --
>
> Key: HBASE-25997
> URL: https://issues.apache.org/jira/browse/HBASE-25997
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Affects Versions: 2.4.4
>Reporter: Lijin Bin
>Assignee: Lijin Bin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.6, 2.4.5
>
>
> Client write a big request to server, server decode request wrong, so client
> do not get a RequestTooBigException as expected.
> {code}
> 2021-06-11 18:57:27,340 INFO [RS-EventLoopGroup-1-20] ipc.NettyRpcServer:
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException$InvalidWireTypeException:
> Protocol message tag had invalid wire type.
> at
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException.invalidWireType(InvalidProtocolBufferException.java:111)
> at
> org.apache.hbase.thirdparty.com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:519)
> at
> org.apache.hbase.thirdparty.com.google.protobuf.GeneratedMessageV3.parseUnknownField(GeneratedMessageV3.java:298)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RPCProtos$RequestHeader.(RPCProtos.java:5958)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RPCProtos$RequestHeader.(RPCProtos.java:5916)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RPCProtos$RequestHeader$1.parsePartialFrom(RPCProtos.java:7249)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RPCProtos$RequestHeader$1.parsePartialFrom(RPCProtos.java:7244)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RPCProtos$RequestHeader$Builder.mergeFrom(RPCProtos.java:6679)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RPCProtos$RequestHeader$Builder.mergeFrom(RPCProtos.java:6482)
> at
> org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:420)
> at
> org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:317)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.mergeFrom(ProtobufUtil.java:2716)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcFrameDecoder.getHeader(NettyRpcFrameDecoder.java:174)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcFrameDecoder.handleTooBigRequest(NettyRpcFrameDecoder.java:126)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcFrameDecoder.decode(NettyRpcFrameDecoder.java:65)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:405)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:372)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:355)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:242)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:228)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:221)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1403)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:242)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:228)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:912)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:827)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
> at
>
[jira] [Commented] (HBASE-26024) Region server JVM Crash - A fatal error has been detected by the Java Runtime Environment
[ https://issues.apache.org/jira/browse/HBASE-26024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368397#comment-17368397 ] Josh Elser commented on HBASE-26024: Any chance you have access to an hbase 2.2 or 2.3 cluster in which you can see if this same problem happens there? There have been some fixes I recall seeing around Netty and direct bytebuffers in the recent past. > Region server JVM Crash - A fatal error has been detected by the Java Runtime > Environment > - > > Key: HBASE-26024 > URL: https://issues.apache.org/jira/browse/HBASE-26024 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.9 >Reporter: Mohamed Mohideen Meeran >Priority: Major > Attachments: error.log > > > Our production Region servers JVM crashed with the following error logs. > > Register to memory mapping: > RAX=0x2eea8f42 is an unknown value > RBX=0x7f1f8c7900d6 is an unknown value > RCX=0x0021 is an unknown value > RDX=0x is an unknown value > RSP=0x7f1fe3092200 is pointing into the stack for thread: > 0x7f29775fb000 > RBP=0x7f1fe3092200 is pointing into the stack for thread: > 0x7f29775fb000 > RSI=0x7f1f8c7900cc is an unknown value > RDI=0x2eea8f38 is an unknown value > R8 =0x7f28e14a3a38 is an oop > java.nio.DirectByteBuffer > - klass: 'java/nio/DirectByteBuffer' > R9 =0x7f1f8c790094 is an unknown value > R10=0x7f2965053400 is at begin+0 in a stub > StubRoutines::unsafe_arraycopy [0x7f2965053400, 0x7f296505343b[ (59 > bytes) > R11=0x7f28e14a3a38 is an oop > java.nio.DirectByteBuffer > - klass: 'java/nio/DirectByteBuffer' > R12= > [error occurred during error reporting (printing register info), id 0xb] > > Stack: [0x7f1fe2f93000,0x7f1fe3094000], sp=0x7f1fe3092200, free > space=1020k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > v ~StubRoutines::jshort_disjoint_arraycopy > J 18388 C2 > org.apache.hadoop.hbase.io.ByteBufferListOutputStream.write(Ljava/nio/ByteBuffer;II)V > (53 bytes) @ 0x7f2967fa0ea2 [0x7f2967fa0d40+0x162] > J 11722 C2 > org.apache.hadoop.hbase.util.ByteBufferUtils.copyBufferToStream(Ljava/io/OutputStream;Ljava/nio/ByteBuffer;II)V > (75 bytes) @ 0x7f29670aa0fc [0x7f29670a9fa0+0x15c] > J 13251 C2 > org.apache.hadoop.hbase.ByteBufferKeyValue.write(Ljava/io/OutputStream;Z)I > (21 bytes) @ 0x7f2965cbe87c [0x7f2965cbe820+0x5c] > J 8703 C2 > org.apache.hadoop.hbase.KeyValueUtil.oswrite(Lorg/apache/hadoop/hbase/Cell;Ljava/io/OutputStream;Z)I > (259 bytes) @ 0x7f296684a2d4 [0x7f296684a140+0x194] > J 15474 C2 > org.apache.hadoop.hbase.ipc.CellBlockBuilder.buildCellBlockStream(Lorg/apache/hadoop/hbase/codec/Codec;Lorg/apache/hadoop/io/compress/CompressionCodec;Lorg/apache/hadoop/hbase/CellScanner;Lorg/apache/hadoop/hbase/io/ByteBufferPool;)Lorg/apache/hadoop/hbase/io/ByteBufferListOutputStream; > (75 bytes) @ 0x7f29675f9dc8 [0x7f29675f7c80+0x2148] > J 14260 C2 > org.apache.hadoop.hbase.ipc.ServerCall.setResponse(Lorg/apache/hbase/thirdparty/com/google/protobuf/Message;Lorg/apache/hadoop/hbase/CellScanner;Ljava/lang/Throwable;Ljava/lang/String;)V > (408 bytes) @ 0x7f29678ad11c [0x7f29678acec0+0x25c] > J 14732 C2 org.apache.hadoop.hbase.ipc.CallRunner.run()V (1376 bytes) @ > 0x7f296797f690 [0x7f296797e6a0+0xff0] > J 14293 C2 > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(Lorg/apache/hadoop/hbase/ipc/CallRunner;)V > (268 bytes) @ 0x7f29667b7464 [0x7f29667b72e0+0x184] > J 17796% C1 org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run()V (72 bytes) > @ 0x7f2967c9cbe4 [0x7f2967c9caa0+0x144] > v ~StubRoutines::call_stub > V [libjvm.so+0x65ebbb] JavaCalls::call_helper(JavaValue*, methodHandle*, > JavaCallArguments*, Thread*)+0x108b > V [libjvm.so+0x65ffd7] JavaCalls::call_virtual(JavaValue*, KlassHandle, > Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x2f7 > V [libjvm.so+0x660497] JavaCalls::call_virtual(JavaValue*, Handle, > KlassHandle, Symbol*, Symbol*, Thread*)+0x47 > V [libjvm.so+0x6ada71] thread_entry(JavaThread*, Thread*)+0x91 > V [libjvm.so+0x9f24f1] JavaThread::thread_main_inner()+0xf1 > V [libjvm.so+0x9f26d8] JavaThread::run()+0x1b8 > V [libjvm.so+0x8af502] java_start(Thread*)+0x122 > C [libpthread.so.0+0x7dc5] start_thread+0xc5 > > we used the workaround by switching *hbase.rpc.server.impl* back to > SimpleRpcServer as mentioned in the following JIRA > https://issues.apache.org/jira/browse/HBASE-22539?focusedCommentId=16855688=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16855688 > Also, attached the error logs during JVM crash. Any help? -- This message was
[jira] [Commented] (HBASE-25937) Clarify UnknownRegionException
[
https://issues.apache.org/jira/browse/HBASE-25937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368391#comment-17368391
]
Hudson commented on HBASE-25937:
Results for branch branch-2
[build #284 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/284/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/284/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/284/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/284/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/284/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/284//console].
> Clarify UnknownRegionException
> --
>
> Key: HBASE-25937
> URL: https://issues.apache.org/jira/browse/HBASE-25937
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.5
>
>
> UnknownRegionException seems to accept a "region name" but it's actually a
> normal Exception message (and is used that way). Fix this to be "message"
> and add a "cause" capability as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25934) Add username for RegionScannerHolder
[
https://issues.apache.org/jira/browse/HBASE-25934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368393#comment-17368393
]
Hudson commented on HBASE-25934:
Results for branch branch-2
[build #284 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/284/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/284/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/284/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/284/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/284/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/284//console].
> Add username for RegionScannerHolder
>
>
> Key: HBASE-25934
> URL: https://issues.apache.org/jira/browse/HBASE-25934
> Project: HBase
> Issue Type: Wish
>Reporter: tomscut
>Assignee: tomscut
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5
>
>
> This JIRA[HBASE-25542|https://issues.apache.org/jira/browse/HBASE-25542] has
> added part of the client information before, we can also add username for
> RegionScannerHolder.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25877) Add access check for compactionSwitch
[
https://issues.apache.org/jira/browse/HBASE-25877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368392#comment-17368392
]
Hudson commented on HBASE-25877:
Results for branch branch-2
[build #284 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/284/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/284/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/284/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/284/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/284/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/284//console].
> Add access check for compactionSwitch
> --
>
> Key: HBASE-25877
> URL: https://issues.apache.org/jira/browse/HBASE-25877
> Project: HBase
> Issue Type: Bug
> Components: security
>Reporter: lujie
>Assignee: lujie
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.5
>
>
> Should we add access check for
> org.apache.hadoop.hbase.regionserver.RSRpcServices#compactionSwitch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25937) Clarify UnknownRegionException
[
https://issues.apache.org/jira/browse/HBASE-25937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368367#comment-17368367
]
Hudson commented on HBASE-25937:
Results for branch branch-2.3
[build #242 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/242/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/242/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/242/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/242/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/242/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Clarify UnknownRegionException
> --
>
> Key: HBASE-25937
> URL: https://issues.apache.org/jira/browse/HBASE-25937
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.5
>
>
> UnknownRegionException seems to accept a "region name" but it's actually a
> normal Exception message (and is used that way). Fix this to be "message"
> and add a "cause" capability as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25877) Add access check for compactionSwitch
[
https://issues.apache.org/jira/browse/HBASE-25877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368368#comment-17368368
]
Hudson commented on HBASE-25877:
Results for branch branch-2.3
[build #242 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/242/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/242/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/242/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/242/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/242/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add access check for compactionSwitch
> --
>
> Key: HBASE-25877
> URL: https://issues.apache.org/jira/browse/HBASE-25877
> Project: HBase
> Issue Type: Bug
> Components: security
>Reporter: lujie
>Assignee: lujie
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.5
>
>
> Should we add access check for
> org.apache.hadoop.hbase.regionserver.RSRpcServices#compactionSwitch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3421: HBASE-26026 HBase Write may be stuck forever when using CompactingMem…
Apache-HBase commented on pull request #3421: URL: https://github.com/apache/hbase/pull/3421#issuecomment-867021511 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 27s | master passed | | +1 :green_heart: | compile | 1m 4s | master passed | | +1 :green_heart: | shadedjars | 8m 17s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 44s | the patch passed | | +1 :green_heart: | compile | 1m 2s | the patch passed | | +1 :green_heart: | javac | 1m 2s | the patch passed | | +1 :green_heart: | shadedjars | 8m 17s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 131m 17s | hbase-server in the patch failed. | | | | 162m 13s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3421/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3421 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 67ac84ac31c4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / cb247f9464 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3421/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3421/1/testReport/ | | Max. process+thread count | 3677 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3421/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3420: HBASE-26028 The view as json page shows exception when using TinyLfuB…
Apache-HBase commented on pull request #3420: URL: https://github.com/apache/hbase/pull/3420#issuecomment-867019313 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 56s | master passed | | +1 :green_heart: | compile | 1m 0s | master passed | | +1 :green_heart: | shadedjars | 8m 9s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 44s | the patch passed | | +1 :green_heart: | compile | 1m 0s | the patch passed | | +1 :green_heart: | javac | 1m 0s | the patch passed | | +1 :green_heart: | shadedjars | 8m 18s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 129m 58s | hbase-server in the patch passed. | | | | 160m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3420/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3420 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux efac83a029fe 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / cb247f9464 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3420/1/testReport/ | | Max. process+thread count | 4083 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3420/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3420: HBASE-26028 The view as json page shows exception when using TinyLfuB…
Apache-HBase commented on pull request #3420: URL: https://github.com/apache/hbase/pull/3420#issuecomment-867016942 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 50s | master passed | | +1 :green_heart: | compile | 1m 16s | master passed | | +1 :green_heart: | shadedjars | 8m 41s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 14s | the patch passed | | +1 :green_heart: | compile | 1m 12s | the patch passed | | +1 :green_heart: | javac | 1m 12s | the patch passed | | +1 :green_heart: | shadedjars | 8m 18s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 123m 40s | hbase-server in the patch passed. | | | | 156m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3420/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3420 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux daa202a06c0d 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / cb247f9464 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3420/1/testReport/ | | Max. process+thread count | 3997 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3420/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #3372: HBASE-25986 set default value of normalization enabled from hbase site
ndimiduk commented on a change in pull request #3372: URL: https://github.com/apache/hbase/pull/3372#discussion_r657295798 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptor.java ## @@ -275,11 +275,11 @@ /** * Check if normalization enable flag of the table is true. If flag is false - * then no region normalizer won't attempt to normalize this table. + * then region normalizer won't attempt to normalize this table. * - * @return true if region normalization is enabled for this table + * @return value of NORMALIZATION_ENABLED key for this table if present else return defaultValue */ - boolean isNormalizationEnabled(); + boolean isNormalizationEnabled(boolean defaulValue); Review comment: I agree. Do not mix concerns of what's in the TableDescriptor with what's in hbase-site. Instead, have the code that needs to use the value look in both places and make a determination. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-26021) HBase 1.7 to 2.4 upgrade issue
[
https://issues.apache.org/jira/browse/HBASE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368267#comment-17368267
]
Viraj Jasani edited comment on HBASE-26021 at 6/23/21, 4:51 PM:
This is most likely caused by branch-1 backport of HBASE-7767, which I believe
was necessary as part of backporting MasterRegistry work on branch-1 i.e
HBASE-18095 ([~bharathv] can confirm better).
I have verified that branch-1 cluster (in absence of MasterRegistry commits and
HBASE-7767), TableDescriptor data can be deserialized by HBase 2 RS without any
issues.
was (Author: vjasani):
This is most likely caused by branch-1 backport of HBASE-7767, which I believe
was necessary as part of backporting MasterRegistry work on branch-1 i.e
HBASE-18095 ([~bharathv] can confirm better).
I have confirmed on branch-1 without MasterRegistry commits that HBase 2
deserialization of TableDescriptor from Hdfs works fine.
> HBase 1.7 to 2.4 upgrade issue
> --
>
> Key: HBASE-26021
> URL: https://issues.apache.org/jira/browse/HBASE-26021
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.7.0, 2.4.4
>Reporter: Viraj Jasani
>Priority: Major
> Attachments: Screenshot 2021-06-22 at 12.54.21 PM.png, Screenshot
> 2021-06-22 at 12.54.30 PM.png
>
>
> As of today, if we bring up HBase cluster using branch-1 and upgrade to
> branch-2.4, we are facing issue while parsing namespace from HDFS fileinfo.
> Instead of "*hbase:meta*" and "*hbase:namespace*", parsing using ProtobufUtil
> seems to be producing "*\n hbase:\n meta*" and "*\n hbase:\n namespace*"
> {code:java}
> 2021-06-22 00:05:56,611 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> regionserver.RSRpcServices: Open hbase:meta,,1.1588230740
> 2021-06-22 00:05:56,648 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> regionserver.RSRpcServices: Open
> hbase:namespace,,1624297762817.396cb6cc00cd4334cb1ea3a792d7529a.
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R namespace
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
> at org.apache.hadoop.hbase.TableName.(TableName.java:348)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2292)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableDescriptor(ProtobufUtil.java:2937)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.parseFrom(TableDescriptorBuilder.java:1625)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.access$200(TableDescriptorBuilder.java:597)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder.parseFrom(TableDescriptorBuilder.java:320)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:511)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:496)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:482)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:210)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:2112)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:35218)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R^Dmeta
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
>
[jira] [Updated] (HBASE-25966) Fix typo in NOTICE.vm
[
https://issues.apache.org/jira/browse/HBASE-25966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-25966:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Fix typo in NOTICE.vm
> -
>
> Key: HBASE-25966
> URL: https://issues.apache.org/jira/browse/HBASE-25966
> Project: HBase
> Issue Type: Bug
> Components: build, community
>Affects Versions: 3.0.0-alpha-1, 1.4.0, 2.3.0, 1.7.0, 2.4.0, 2.5.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> I noticed there's a typo in
> {{hbase-resource-bundle/src/main/resources/META-INF/NOTICE.vm}}:
> "bundled-boostrap" instead of "bundled-bootstrap".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk merged pull request #3422: Backport "HBASE-25966 Fix typo in NOTICE.vm" to branch-2
ndimiduk merged pull request #3422: URL: https://github.com/apache/hbase/pull/3422 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk merged pull request #3349: HBASE-25966 Fix typo in NOTICE.vm
ndimiduk merged pull request #3349: URL: https://github.com/apache/hbase/pull/3349 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3422: Backport "HBASE-25966 Fix typo in NOTICE.vm" to branch-2
Apache-HBase commented on pull request #3422: URL: https://github.com/apache/hbase/pull/3422#issuecomment-866999165 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 40s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 34s | branch-2 passed | | +1 :green_heart: | javadoc | 0m 14s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 5s | the patch passed | | +1 :green_heart: | javadoc | 0m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 14s | hbase-resource-bundle in the patch passed. | | | | 14m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3422/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3422 | | Optional Tests | javac javadoc unit | | uname | Linux 28fdd5505e79 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9fe1781037 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3422/1/testReport/ | | Max. process+thread count | 76 (vs. ulimit of 12500) | | modules | C: hbase-resource-bundle U: hbase-resource-bundle | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3422/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25966) Fix typo in NOTICE.vm
[
https://issues.apache.org/jira/browse/HBASE-25966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368342#comment-17368342
]
Nick Dimiduk commented on HBASE-25966:
--
There are some nice comments on #3349 left by [~busbey], showing what to look
for in the output jars and how we might implement some automated verification
tests over the build artifacts in the future.
> Fix typo in NOTICE.vm
> -
>
> Key: HBASE-25966
> URL: https://issues.apache.org/jira/browse/HBASE-25966
> Project: HBase
> Issue Type: Bug
> Components: build, community
>Affects Versions: 3.0.0-alpha-1, 1.4.0, 2.3.0, 1.7.0, 2.4.0, 2.5.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> I noticed there's a typo in
> {{hbase-resource-bundle/src/main/resources/META-INF/NOTICE.vm}}:
> "bundled-boostrap" instead of "bundled-bootstrap".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25966) Fix typo in NOTICE.vm
[
https://issues.apache.org/jira/browse/HBASE-25966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-25966:
-
Status: Patch Available (was: Open)
> Fix typo in NOTICE.vm
> -
>
> Key: HBASE-25966
> URL: https://issues.apache.org/jira/browse/HBASE-25966
> Project: HBase
> Issue Type: Bug
> Components: build, community
>Affects Versions: 2.4.0, 1.7.0, 2.3.0, 1.4.0, 3.0.0-alpha-1, 2.5.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> I noticed there's a typo in
> {{hbase-resource-bundle/src/main/resources/META-INF/NOTICE.vm}}:
> "bundled-boostrap" instead of "bundled-bootstrap".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25966) Fix typo in NOTICE.vm
[
https://issues.apache.org/jira/browse/HBASE-25966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-25966:
-
Fix Version/s: 2.5.0
3.0.0-alpha-1
> Fix typo in NOTICE.vm
> -
>
> Key: HBASE-25966
> URL: https://issues.apache.org/jira/browse/HBASE-25966
> Project: HBase
> Issue Type: Bug
> Components: build, community
>Affects Versions: 3.0.0-alpha-1, 1.4.0, 2.3.0, 1.7.0, 2.4.0, 2.5.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> I noticed there's a typo in
> {{hbase-resource-bundle/src/main/resources/META-INF/NOTICE.vm}}:
> "bundled-boostrap" instead of "bundled-bootstrap".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-25966) Fix typo in NOTICE.vm
[
https://issues.apache.org/jira/browse/HBASE-25966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk reassigned HBASE-25966:
Assignee: Nick Dimiduk
> Fix typo in NOTICE.vm
> -
>
> Key: HBASE-25966
> URL: https://issues.apache.org/jira/browse/HBASE-25966
> Project: HBase
> Issue Type: Bug
> Components: build, community
>Affects Versions: 3.0.0-alpha-1, 1.4.0, 2.3.0, 1.7.0, 2.4.0, 2.5.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
>
> I noticed there's a typo in
> {{hbase-resource-bundle/src/main/resources/META-INF/NOTICE.vm}}:
> "bundled-boostrap" instead of "bundled-bootstrap".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3422: Backport "HBASE-25966 Fix typo in NOTICE.vm" to branch-2
Apache-HBase commented on pull request #3422: URL: https://github.com/apache/hbase/pull/3422#issuecomment-866996726 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 14s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 6s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 38s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 10m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3422/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3422 | | Optional Tests | dupname asflicense javac | | uname | Linux 75cb0558da1c 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9fe1781037 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 66 (vs. ulimit of 12500) | | modules | C: hbase-resource-bundle U: hbase-resource-bundle | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3422/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3422: Backport "HBASE-25966 Fix typo in NOTICE.vm" to branch-2
Apache-HBase commented on pull request #3422: URL: https://github.com/apache/hbase/pull/3422#issuecomment-866995966 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 8s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | branch-2 passed | | +1 :green_heart: | javadoc | 0m 15s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | javadoc | 0m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 13s | hbase-resource-bundle in the patch passed. | | | | 9m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3422/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3422 | | Optional Tests | javac javadoc unit | | uname | Linux d6aff58bd1ef 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 9fe1781037 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3422/1/testReport/ | | Max. process+thread count | 62 (vs. ulimit of 12500) | | modules | C: hbase-resource-bundle U: hbase-resource-bundle | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3422/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3349: HBASE-25966 Fix typo in NOTICE.vm
Apache-HBase commented on pull request #3349: URL: https://github.com/apache/hbase/pull/3349#issuecomment-866993828 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 43s | master passed | | +1 :green_heart: | javadoc | 0m 14s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 23s | the patch passed | | +1 :green_heart: | javadoc | 0m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 13s | hbase-resource-bundle in the patch passed. | | | | 12m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3349/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3349 | | Optional Tests | javac javadoc unit | | uname | Linux 568453cd93da 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 39d143f290 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3349/2/testReport/ | | Max. process+thread count | 93 (vs. ulimit of 3) | | modules | C: hbase-resource-bundle U: hbase-resource-bundle | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3349/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3349: HBASE-25966 Fix typo in NOTICE.vm
Apache-HBase commented on pull request #3349: URL: https://github.com/apache/hbase/pull/3349#issuecomment-866993454 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 4s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 47s | master passed | | +1 :green_heart: | javadoc | 0m 13s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 6s | the patch passed | | +1 :green_heart: | javadoc | 0m 11s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 12s | hbase-resource-bundle in the patch passed. | | | | 11m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3349/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3349 | | Optional Tests | javac javadoc unit | | uname | Linux 0d48e69dbca5 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 39d143f290 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3349/2/testReport/ | | Max. process+thread count | 65 (vs. ulimit of 3) | | modules | C: hbase-resource-bundle U: hbase-resource-bundle | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3349/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3349: HBASE-25966 Fix typo in NOTICE.vm
Apache-HBase commented on pull request #3349: URL: https://github.com/apache/hbase/pull/3349#issuecomment-866992891 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 0s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 10m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3349/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3349 | | Optional Tests | dupname asflicense javac | | uname | Linux 32c36c9d880f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 39d143f290 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 78 (vs. ulimit of 3) | | modules | C: hbase-resource-bundle U: hbase-resource-bundle | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3349/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #3422: Backport "HBASE-25966 Fix typo in NOTICE.vm" to branch-2
ndimiduk commented on pull request #3422: URL: https://github.com/apache/hbase/pull/3422#issuecomment-866990542 @busbey @Apache9 @apurtell I think we should only backport this as far as branch-2 because we shouldn't mess with license and notice files on existing release lines. Do any of you strongly disagree? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk opened a new pull request #3422: Backport "HBASE-25966 Fix typo in NOTICE.vm" to branch-2
ndimiduk opened a new pull request #3422: URL: https://github.com/apache/hbase/pull/3422 Signed-off-by: Sean Busbey -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #3349: HBASE-25966 Fix typo in NOTICE.vm
ndimiduk commented on pull request #3349: URL: https://github.com/apache/hbase/pull/3349#issuecomment-866978311 > Let's get this in? @ndimiduk Yes, I suppose this doesn't make things worse. It sounds like we need further fixes in this area, and some kind of test that the notice files are generated as we intend. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #2596: Backport "HBASE-24419 Normalizer merge plans should consider more than 2 region…" to branch-2
ndimiduk commented on a change in pull request #2596:
URL: https://github.com/apache/hbase/pull/2596#discussion_r657261818
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -315,35 +316,60 @@ private boolean skipForMerge(final RegionStates
regionStates, final RegionInfo r
* towards target average or target region count.
*/
private List computeMergeNormalizationPlans(final
NormalizeContext ctx) {
-if (ctx.getTableRegions().size() < minRegionCount) {
+if (isEmpty(ctx.getTableRegions()) || ctx.getTableRegions().size() <
minRegionCount) {
LOG.debug("Table {} has {} regions, required min number of regions for
normalizer to run"
+ " is {}, not computing merge plans.", ctx.getTableName(),
ctx.getTableRegions().size(),
minRegionCount);
return Collections.emptyList();
}
-final double avgRegionSizeMb = ctx.getAverageRegionSizeMb();
+final long avgRegionSizeMb = (long) ctx.getAverageRegionSizeMb();
+if (avgRegionSizeMb < mergeMinRegionSizeMb) {
Review comment:
I'd have to look back through the original JIRA and PR, but I believe
there was a concern that normalizer would merge away the splits made on table
creation, between the time when the table was created and when the operator got
around to loading it with data. The "minimum table size" configuration was
designed to prevent this. This behavior pre-dated HBASE-24419 ; the
functionality was preserved when this optimization was implemented.
I personally am not a fan, and prefer the "minimum table age" configuration
for handling of this concern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-26018) Perf improvement in L1 cache
[ https://issues.apache.org/jira/browse/HBASE-26018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani updated HBASE-26018: - Description: After HBASE-25698 is in, all L1 caching strategies perform buffer.retain() in order to maintain refCount atomically while retrieving cached blocks (CHM#computeIfPresent). Retaining refCount is coming up bit expensive in terms of performance. Using computeIfPresent API, CHM uses coarse grained segment locking and even if our computation is not so complex (we just call block retain API), it will block other update APIs for the keys within bucket that is locked. computeIfPresent keeps showing up on flame graphs as well (attached one of them). Specifically when we see aggressive cache hits for meta blocks (with major blocks in cache), we would want to get away from coarse grained locking. One of the suggestions came up while reviewing PR#3215 is to treat cache read API as optimistic read and deal with block retain based refCount issues by catching the respective Exception and let it treat as cache miss. This should allow us to go ahead with lockless get API. was: After HBASE-25698 is in, all L1 caching strategies perform buffer.retain() in order to maintain refCount atomically while retrieving cached blocks (CHM#computeIfPresent). Retaining refCount is coming up bit expensive in terms of performance. Using computeIfPresent API, CHM uses coarse grained segment locking and even if our computation is not so complex (we just call block retain API), it will block other update APIs for the key. computeIfPresent keeps showing up on flame graphs as well (attached one of them). Specifically when we see aggressive cache hits for meta blocks (with major blocks in cache), we would want to get away from coarse grained locking. One of the suggestions came up while reviewing PR#3215 is to treat cache read API as optimistic read and deal with block retain based refCount issues by catching the respective Exception and let it treat as cache miss. This should allow us to go ahead with lockless get API. > Perf improvement in L1 cache > > > Key: HBASE-26018 > URL: https://issues.apache.org/jira/browse/HBASE-26018 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-1, 2.3.5, 2.4.4 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.5 > > Attachments: computeIfPresent.png > > > After HBASE-25698 is in, all L1 caching strategies perform buffer.retain() in > order to maintain refCount atomically while retrieving cached blocks > (CHM#computeIfPresent). Retaining refCount is coming up bit expensive in > terms of performance. Using computeIfPresent API, CHM uses coarse grained > segment locking and even if our computation is not so complex (we just call > block retain API), it will block other update APIs for the keys within bucket > that is locked. computeIfPresent keeps showing up on flame graphs as well > (attached one of them). Specifically when we see aggressive cache hits for > meta blocks (with major blocks in cache), we would want to get away from > coarse grained locking. > One of the suggestions came up while reviewing PR#3215 is to treat cache read > API as optimistic read and deal with block retain based refCount issues by > catching the respective Exception and let it treat as cache miss. This should > allow us to go ahead with lockless get API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26029) It is not reliable to use nodeDeleted event to track region server's death
[ https://issues.apache.org/jira/browse/HBASE-26029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26029: -- Description: When implementing HBASE-26011, [~sunxin] pointed out an interesting scenario, where a region server up and down between two sync requests, then we can not know the death of the region server. https://github.com/apache/hbase/pull/3405#discussion_r656720923 This is a valid point, and when thinking of a solution, I noticed that, the current zk iplementation has the same problem. Notice that, a watcher on zk can only be triggered once, so after zk triggers the watcher, and before you set a new watcher, it is possible that a region server is up and down, and you will miss the nodeDeleted event for this region server. I think, the general approach here, which could works for both master based and zk based replication tracker is that, we should not rely on the tracker to tell you which region server is dead. Instead, we just provide the list of live regionservers, and the upper layer should compare this list with the expected list(for replication, the list should be gotten by listing replicators), to detect the dead region servers. was: When implementing HBASE-26011, [~sunxin] pointed out an interesting scenario, where a region server up and down between two sync requests, then we can not know the death of the region server. This is a valid point, and when thinking of a solution, I noticed that, the current zk iplementation has the same problem. Notice that, a watcher on zk can only be triggered once, so after zk triggers the watcher, and before you set a new watcher, it is possible that a region server is up and down, and you will miss the nodeDeleted event for this region server. I think, the general approach here, which could works for both master based and zk based replication tracker is that, we should not rely on the tracker to tell you which region server is dead. Instead, we just provide the list of live regionservers, and the upper layer should compare this list with the expected list(for replication, the list should be gotten by listing replicators), to detect the dead region servers. > It is not reliable to use nodeDeleted event to track region server's death > -- > > Key: HBASE-26029 > URL: https://issues.apache.org/jira/browse/HBASE-26029 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > > When implementing HBASE-26011, [~sunxin] pointed out an interesting scenario, > where a region server up and down between two sync requests, then we can not > know the death of the region server. > https://github.com/apache/hbase/pull/3405#discussion_r656720923 > This is a valid point, and when thinking of a solution, I noticed that, the > current zk iplementation has the same problem. Notice that, a watcher on zk > can only be triggered once, so after zk triggers the watcher, and before you > set a new watcher, it is possible that a region server is up and down, and > you will miss the nodeDeleted event for this region server. > I think, the general approach here, which could works for both master based > and zk based replication tracker is that, we should not rely on the tracker > to tell you which region server is dead. Instead, we just provide the list of > live regionservers, and the upper layer should compare this list with the > expected list(for replication, the list should be gotten by listing > replicators), to detect the dead region servers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26029) It is not reliable to use nodeDeleted event to track region server's death
Duo Zhang created HBASE-26029: - Summary: It is not reliable to use nodeDeleted event to track region server's death Key: HBASE-26029 URL: https://issues.apache.org/jira/browse/HBASE-26029 Project: HBase Issue Type: Bug Reporter: Duo Zhang Assignee: Duo Zhang When implementing HBASE-26011, [~sunxin] pointed out an interesting scenario, where a region server up and down between two sync requests, then we can not know the death of the region server. This is a valid point, and when thinking of a solution, I noticed that, the current zk iplementation has the same problem. Notice that, a watcher on zk can only be triggered once, so after zk triggers the watcher, and before you set a new watcher, it is possible that a region server is up and down, and you will miss the nodeDeleted event for this region server. I think, the general approach here, which could works for both master based and zk based replication tracker is that, we should not rely on the tracker to tell you which region server is dead. Instead, we just provide the list of live regionservers, and the upper layer should compare this list with the expected list(for replication, the list should be gotten by listing replicators), to detect the dead region servers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26020) Split TestWALEntryStream.testDifferentCounts out
[ https://issues.apache.org/jira/browse/HBASE-26020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26020. --- Fix Version/s: 2.4.5 2.5.0 3.0.0-alpha-1 Hadoop Flags: Reviewed Resolution: Fixed Pushed to branch-2.4+. Thank [~haxiaolin] for reviewing. The code on branch-2.3 is different so not easy to cherry pick, give up for now. > Split TestWALEntryStream.testDifferentCounts out > > > Key: HBASE-26020 > URL: https://issues.apache.org/jira/browse/HBASE-26020 > Project: HBase > Issue Type: Improvement > Components: Replication, test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5 > > > It consumes too much time and may cause the whole UT to timeout. > And in fact, it should be implemented as parameterized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3421: HBASE-26026 HBase Write may be stuck forever when using CompactingMem…
Apache-HBase commented on pull request #3421: URL: https://github.com/apache/hbase/pull/3421#issuecomment-866936432 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 58s | master passed | | +1 :green_heart: | compile | 3m 11s | master passed | | +1 :green_heart: | checkstyle | 1m 7s | master passed | | +1 :green_heart: | spotbugs | 2m 6s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 37s | the patch passed | | +1 :green_heart: | compile | 3m 6s | the patch passed | | +1 :green_heart: | javac | 3m 6s | the patch passed | | -0 :warning: | checkstyle | 1m 5s | hbase-server: The patch generated 2 new + 74 unchanged - 0 fixed = 76 total (was 74) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 5s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 11s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 47m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3421/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3421 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux f724aa5492ed 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / cb247f9464 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3421/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 95 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3421/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3420: HBASE-26028 The view as json page shows exception when using TinyLfuB…
Apache-HBase commented on pull request #3420: URL: https://github.com/apache/hbase/pull/3420#issuecomment-866936057 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 59s | master passed | | +1 :green_heart: | compile | 3m 15s | master passed | | +1 :green_heart: | checkstyle | 1m 3s | master passed | | +1 :green_heart: | spotbugs | 2m 3s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 35s | the patch passed | | +1 :green_heart: | compile | 3m 13s | the patch passed | | +1 :green_heart: | javac | 3m 13s | the patch passed | | +1 :green_heart: | checkstyle | 1m 2s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 13s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 11s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 47m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3420/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3420 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 93fda3034f55 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / cb247f9464 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3420/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26021) HBase 1.7 to 2.4 upgrade issue
[
https://issues.apache.org/jira/browse/HBASE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368267#comment-17368267
]
Viraj Jasani commented on HBASE-26021:
--
This is most likely caused by branch-1 backport of HBASE-7767, which I believe
was necessary as part of backporting MasterRegistry work on branch-1 i.e
HBASE-18095 ([~bharathv] can confirm better).
I have confirmed on branch-1 without MasterRegistry commits that HBase 2
deserialization of TableDescriptor from Hdfs works fine.
> HBase 1.7 to 2.4 upgrade issue
> --
>
> Key: HBASE-26021
> URL: https://issues.apache.org/jira/browse/HBASE-26021
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.7.0, 2.4.4
>Reporter: Viraj Jasani
>Priority: Major
> Attachments: Screenshot 2021-06-22 at 12.54.21 PM.png, Screenshot
> 2021-06-22 at 12.54.30 PM.png
>
>
> As of today, if we bring up HBase cluster using branch-1 and upgrade to
> branch-2.4, we are facing issue while parsing namespace from HDFS fileinfo.
> Instead of "*hbase:meta*" and "*hbase:namespace*", parsing using ProtobufUtil
> seems to be producing "*\n hbase:\n meta*" and "*\n hbase:\n namespace*"
> {code:java}
> 2021-06-22 00:05:56,611 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> regionserver.RSRpcServices: Open hbase:meta,,1.1588230740
> 2021-06-22 00:05:56,648 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> regionserver.RSRpcServices: Open
> hbase:namespace,,1624297762817.396cb6cc00cd4334cb1ea3a792d7529a.
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R namespace
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
> at org.apache.hadoop.hbase.TableName.(TableName.java:348)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2292)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableDescriptor(ProtobufUtil.java:2937)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.parseFrom(TableDescriptorBuilder.java:1625)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.access$200(TableDescriptorBuilder.java:597)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder.parseFrom(TableDescriptorBuilder.java:320)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:511)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:496)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:482)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:210)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:2112)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:35218)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R^Dmeta
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
> at org.apache.hadoop.hbase.TableName.(TableName.java:348)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2292)
> at
>
[GitHub] [hbase] Apache9 merged pull request #3409: HBASE-26020 Split TestWALEntryStream.testDifferentCounts out
Apache9 merged pull request #3409: URL: https://github.com/apache/hbase/pull/3409 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3409: HBASE-26020 Split TestWALEntryStream.testDifferentCounts out
Apache-HBase commented on pull request #3409: URL: https://github.com/apache/hbase/pull/3409#issuecomment-866896841 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 28s | master passed | | +1 :green_heart: | compile | 1m 56s | master passed | | +1 :green_heart: | shadedjars | 11m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 27s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 0s | the patch passed | | +1 :green_heart: | compile | 1m 41s | the patch passed | | +1 :green_heart: | javac | 1m 41s | the patch passed | | +1 :green_heart: | shadedjars | 10m 15s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 12s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 37s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 208m 27s | hbase-server in the patch passed. | | | | 252m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3409/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3409 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ce08a8a7537d 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / fa2d127b7f | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3409/4/testReport/ | | Max. process+thread count | 2559 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3409/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26026) HBase Write may be stuck forever when using CompactingMemStore
[
https://issues.apache.org/jira/browse/HBASE-26026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenglei updated HBASE-26026:
-
Assignee: chenglei
Status: Patch Available (was: Open)
> HBase Write may be stuck forever when using CompactingMemStore
> --
>
> Key: HBASE-26026
> URL: https://issues.apache.org/jira/browse/HBASE-26026
> Project: HBase
> Issue Type: Bug
> Components: in-memory-compaction
>Affects Versions: 2.4.0, 2.3.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
>
> Sometimes I observed that HBase Write might be stuck in my hbase cluster
> which enabling {{CompactingMemStore}}. I have simulated the problem by unit
> test in my PR.
> The problem is caused by {{CompactingMemStore.checkAndAddToActiveSize}} :
> {code:java}
> 425 private boolean checkAndAddToActiveSize(MutableSegment currActive, Cell
> cellToAdd,
> 426 MemStoreSizing memstoreSizing) {
> 427if (shouldFlushInMemory(currActive, cellToAdd, memstoreSizing)) {
> 428 if (currActive.setInMemoryFlushed()) {
> 429flushInMemory(currActive);
> 430if (setInMemoryCompactionFlag()) {
> 431 // The thread is dispatched to do in-memory compaction in the
> background
> ..
> }
> {code}
> In line 427, if {{currActive.getDataSize}} adding the size of {{cellToAdd}}
> exceeds {{CompactingMemStore.inmemoryFlushSize}}, then {{currActive}} should
> be flushed, {{MutableSegment.setInMemoryFlushed()}} is invoked in above line
> 428 ：
> {code:java}
> public boolean setInMemoryFlushed() {
> return flushed.compareAndSet(false, true);
> }
> {code}
> After set {{currActive.flushed}} to true, in above line 429
> {{flushInMemory(currActive)}} invokes
> {{CompactingMemStore.pushActiveToPipeline}} :
> {code:java}
> protected void pushActiveToPipeline(MutableSegment currActive) {
> if (!currActive.isEmpty()) {
> pipeline.pushHead(currActive);
> resetActive();
> }
> }
> {code}
> In above {{CompactingMemStore.pushActiveToPipeline}} method , if the
> {{currActive.cellSet}} is empty, then nothing is done. Due to concurrent
> writes and because we first add cell size to {{currActive.getDataSize}} and
> then actually add cell to {{currActive.cellSet}}, it is possible that
> {{currActive.getDataSize}} could not accommodate {{cellToAdd}} but
> {{currActive.cellSet}} is still empty if pending writes which not yet add
> cells to {{currActive.cellSet}}.
> So if the {{currActive.cellSet}} is empty now, then no {{ActiveSegment}} is
> created, and new writes still continue target to {{currActive}}, but
> {{currActive.flushed}} is true, {{currActive}} could not enter
> {{flushInMemory(currActive)}} again，and new {{ActiveSegment}} could not be
> created forever ! In the end all writes would be stuck.
> In my opinion , once {{currActive.flushed}} is set true, it could not
> continue use as {{ActiveSegment}} , and because of concurrent pending writes,
> only after {{currActive.updatesLock.writeLock()}} is acquired(i.e.
> {{currActive.waitForUpdates}} is called) in
> {{CompactingMemStore.inMemoryCompaction}} ,we can safely say {{currActive}}
> is empty or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] comnetwork opened a new pull request #3421: HBASE-26026 HBase Write may be stuck forever when using CompactingMem…
comnetwork opened a new pull request #3421: URL: https://github.com/apache/hbase/pull/3421 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3405: HBASE-26011 Introduce a new API to sync the live region server list m…
Apache9 commented on a change in pull request #3405:
URL: https://github.com/apache/hbase/pull/3405#discussion_r657168459
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
##
@@ -163,6 +169,44 @@
private final ConcurrentNavigableMap
onlineServers =
new ConcurrentSkipListMap<>();
+ /**
+ * Store the snapshot of the current region server list, for improving read
performance.
+ *
+ * The hashCode is used to determine whether there are changes to the region
servers.
+ */
+ private static final class OnlineServerListSnapshot {
+
+private static final HashFunction HASH = Hashing.murmur3_128();
+
+final List servers;
+
+final long hashCode;
+
+public OnlineServerListSnapshot(List servers) {
+ this.servers = Collections.unmodifiableList(servers);
+ Hasher hasher = HASH.newHasher();
+ for (ServerName server : servers) {
+hasher.putString(server.getServerName(), StandardCharsets.UTF_8);
+ }
+ this.hashCode = hasher.hash().asLong();
Review comment:
And this could also be a problem for the old zk based implementation. We
just watch znode and process the znode deleted event, but it is also possible
that, after a trigger of the watcher, before set a new watcher, a region server
is up and then down, then you can not get the deleted event either.
So the solution should be general, maybe we need to get all the replicators
from the replication queue storage, and compare it with the current live region
server set, to find out the dead region server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bsglz opened a new pull request #3420: HBASE-26028 The view as json page shows exception when using TinyLfuB…
bsglz opened a new pull request #3420: URL: https://github.com/apache/hbase/pull/3420 …lockCache -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26028) The view as json page shows exception when using TinyLfuBlockCache
[ https://issues.apache.org/jira/browse/HBASE-26028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Wang updated HBASE-26028: --- Summary: The view as json page shows exception when using TinyLfuBlockCache (was: The viewJson page shows exception when using TinyLfuBlockCache) > The view as json page shows exception when using TinyLfuBlockCache > -- > > Key: HBASE-26028 > URL: https://issues.apache.org/jira/browse/HBASE-26028 > Project: HBase > Issue Type: Bug > Components: UI >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > > Some variable in TinyLfuBlockCache should be marked as transient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on a change in pull request #3405: HBASE-26011 Introduce a new API to sync the live region server list m…
Apache9 commented on a change in pull request #3405:
URL: https://github.com/apache/hbase/pull/3405#discussion_r657159026
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
##
@@ -163,6 +169,44 @@
private final ConcurrentNavigableMap
onlineServers =
new ConcurrentSkipListMap<>();
+ /**
+ * Store the snapshot of the current region server list, for improving read
performance.
+ *
+ * The hashCode is used to determine whether there are changes to the region
servers.
+ */
+ private static final class OnlineServerListSnapshot {
+
+private static final HashFunction HASH = Hashing.murmur3_128();
+
+final List servers;
+
+final long hashCode;
+
+public OnlineServerListSnapshot(List servers) {
+ this.servers = Collections.unmodifiableList(servers);
+ Hasher hasher = HASH.newHasher();
+ for (ServerName server : servers) {
+hasher.putString(server.getServerName(), StandardCharsets.UTF_8);
+ }
+ this.hashCode = hasher.hash().asLong();
Review comment:
Ah, this is a problem. In real world, for the replication scenario,
usually it is not a problem as it is not likely that a regionserver which can
only live for less than 5 seconds could hold any regions. But consider general
usage, maybe we should let users know there is a dead server.
Let me think how to deal with this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-26028) The viewJson page shows exception when using TinyLfuBlockCache
[ https://issues.apache.org/jira/browse/HBASE-26028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Wang updated HBASE-26028: --- Description: Some variable in TinyLfuBlockCache should be marked as transient. (was: Some variable in TinyLfuBlockCache should be marked as transient.) > The viewJson page shows exception when using TinyLfuBlockCache > -- > > Key: HBASE-26028 > URL: https://issues.apache.org/jira/browse/HBASE-26028 > Project: HBase > Issue Type: Bug > Components: UI >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > > Some variable in TinyLfuBlockCache should be marked as transient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3409: HBASE-26020 Split TestWALEntryStream.testDifferentCounts out
Apache-HBase commented on pull request #3409: URL: https://github.com/apache/hbase/pull/3409#issuecomment-866823381 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 1s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 28s | master passed | | +1 :green_heart: | compile | 1m 39s | master passed | | +1 :green_heart: | shadedjars | 8m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 6s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 11s | the patch passed | | +1 :green_heart: | compile | 1m 35s | the patch passed | | +1 :green_heart: | javac | 1m 35s | the patch passed | | +1 :green_heart: | shadedjars | 8m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 4s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 53s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 128m 53s | hbase-server in the patch passed. | | | | 165m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3409/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3409 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d89f5a875434 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / fa2d127b7f | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3409/4/testReport/ | | Max. process+thread count | 3844 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3409/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3418: fix pretty tool print header missing last indexblock and bloomchunk
Apache-HBase commented on pull request #3418: URL: https://github.com/apache/hbase/pull/3418#issuecomment-866822522 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 25s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 55s | master passed | | +1 :green_heart: | compile | 1m 42s | master passed | | +1 :green_heart: | shadedjars | 10m 28s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 52s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 28s | the patch passed | | +1 :green_heart: | compile | 1m 36s | the patch passed | | +1 :green_heart: | javac | 1m 36s | the patch passed | | +1 :green_heart: | shadedjars | 10m 53s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 55s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 202m 36s | hbase-server in the patch passed. | | | | 243m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3418/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3418 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 68cba94aa531 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / fa2d127b7f | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3418/1/testReport/ | | Max. process+thread count | 2285 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3418/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3419: HBASE-26027 The calling of HTable.batch blocked at AsyncRequestFuture…
Apache-HBase commented on pull request #3419: URL: https://github.com/apache/hbase/pull/3419#issuecomment-866819172 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 40s | branch-2 passed | | +1 :green_heart: | compile | 1m 6s | branch-2 passed | | +1 :green_heart: | checkstyle | 0m 34s | branch-2 passed | | +1 :green_heart: | spotbugs | 1m 12s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 20s | the patch passed | | +1 :green_heart: | compile | 1m 3s | the patch passed | | +1 :green_heart: | javac | 1m 3s | the patch passed | | +1 :green_heart: | checkstyle | 0m 34s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 46s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 1m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 33m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3419/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3419 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 2f349c1e3848 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / e3eb760864 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 12500) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3419/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26028) The viewJson page shows exception when using TinyLfuBlockCache
[ https://issues.apache.org/jira/browse/HBASE-26028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Wang updated HBASE-26028: --- Summary: The viewJson page shows exception when using TinyLfuBlockCache (was: The viewJson page show exception when using TinyLfuBlockCache) > The viewJson page shows exception when using TinyLfuBlockCache > -- > > Key: HBASE-26028 > URL: https://issues.apache.org/jira/browse/HBASE-26028 > Project: HBase > Issue Type: Bug > Components: UI >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > > Some variable in TinyLfuBlockCache should be marked as > transient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3417: HBASE-25902 Add missing CFs in meta during HBase 1 to 2 Upgrade
Apache-HBase commented on pull request #3417: URL: https://github.com/apache/hbase/pull/3417#issuecomment-866817411 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 0s | branch-2.4 passed | | +1 :green_heart: | compile | 1m 0s | branch-2.4 passed | | +1 :green_heart: | shadedjars | 6m 34s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 39s | the patch passed | | +1 :green_heart: | compile | 1m 0s | the patch passed | | +1 :green_heart: | javac | 1m 0s | the patch passed | | +1 :green_heart: | shadedjars | 6m 31s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 210m 51s | hbase-server in the patch passed. | | | | 238m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3417/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3417 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 65a7f156e8a5 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / 8d1473d321 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3417/1/testReport/ | | Max. process+thread count | 2467 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3417/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-26028) The viewJson page show exception when using TinyLfuBlockCache
Zheng Wang created HBASE-26028: -- Summary: The viewJson page show exception when using TinyLfuBlockCache Key: HBASE-26028 URL: https://issues.apache.org/jira/browse/HBASE-26028 Project: HBase Issue Type: Bug Components: UI Reporter: Zheng Wang Assignee: Zheng Wang Some variable in TinyLfuBlockCache should be marked as transient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25877) Add access check for compactionSwitch
[
https://issues.apache.org/jira/browse/HBASE-25877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368105#comment-17368105
]
Hudson commented on HBASE-25877:
Results for branch master
[build #330 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/330/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/330/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/330/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/330/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add access check for compactionSwitch
> --
>
> Key: HBASE-25877
> URL: https://issues.apache.org/jira/browse/HBASE-25877
> Project: HBase
> Issue Type: Bug
> Components: security
>Reporter: lujie
>Assignee: lujie
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.5
>
>
> Should we add access check for
> org.apache.hadoop.hbase.regionserver.RSRpcServices#compactionSwitch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25937) Clarify UnknownRegionException
[
https://issues.apache.org/jira/browse/HBASE-25937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368104#comment-17368104
]
Hudson commented on HBASE-25937:
Results for branch master
[build #330 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/330/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/330/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/330/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/330/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Clarify UnknownRegionException
> --
>
> Key: HBASE-25937
> URL: https://issues.apache.org/jira/browse/HBASE-25937
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.5
>
>
> UnknownRegionException seems to accept a "region name" but it's actually a
> normal Exception message (and is used that way). Fix this to be "message"
> and add a "cause" capability as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3419: HBASE-26027 The calling of HTable.batch blocked at AsyncRequestFuture…
Apache-HBase commented on pull request #3419: URL: https://github.com/apache/hbase/pull/3419#issuecomment-866815088 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | -1 :x: | mvninstall | 4m 9s | root in branch-2 failed. | | +1 :green_heart: | compile | 0m 32s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 51s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 30s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 58s | the patch passed | | +1 :green_heart: | compile | 0m 31s | the patch passed | | +1 :green_heart: | javac | 0m 31s | the patch passed | | +1 :green_heart: | shadedjars | 6m 50s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 28s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 22s | hbase-client in the patch passed. | | | | 28m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3419/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3419 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 785986e68faa 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / e3eb760864 | | Default Java | AdoptOpenJDK-11.0.10+9 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3419/1/artifact/yetus-jdk11-hadoop3-check/output/branch-mvninstall-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3419/1/testReport/ | | Max. process+thread count | 282 (vs. ulimit of 12500) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3419/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3419: HBASE-26027 The calling of HTable.batch blocked at AsyncRequestFuture…
Apache-HBase commented on pull request #3419: URL: https://github.com/apache/hbase/pull/3419#issuecomment-866813082 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 40s | branch-2 passed | | +1 :green_heart: | compile | 0m 28s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 2s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 19s | the patch passed | | +1 :green_heart: | compile | 0m 26s | the patch passed | | +1 :green_heart: | javac | 0m 26s | the patch passed | | +1 :green_heart: | shadedjars | 6m 6s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 46s | hbase-client in the patch passed. | | | | 25m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3419/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3419 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d736878b56c2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / e3eb760864 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3419/1/testReport/ | | Max. process+thread count | 343 (vs. ulimit of 12500) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3419/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26027) The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException
[
https://issues.apache.org/jira/browse/HBASE-26027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Wang updated HBASE-26027:
---
Description:
The batch api of HTable contains a param named results to store result or
exception, its type is Object[].
If user pass an array with other type, eg:
org.apache.hadoop.hbase.client.Result, then the ArrayStoreException will occur
in AsyncRequestFutureImpl.updateResult, then the
AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
actionsInProgress again and again, forever.
It is better to add an cutoff calculated by operationTimeout, instead of only
depend on the value of actionsInProgress.
BTW, this issue only for 2.x, since 3.x the implement has refactored.
{code:java}
[ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 - id=1
error for test processing localhost,16020,1624343786295
java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10 actions to finish
on table:
[INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10 actions to finish
on table: test
{code}
was:
[for 2.x]
The batch api of HTable contains a param named results to store result or
exception, its type is Object[].
If user pass an array with other type, eg:
org.apache.hadoop.hbase.client.Result, then the ArrayStoreException will occur
in AsyncRequestFutureImpl.updateResult, then the
AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
actionsInProgress again and again, forever.
It is better to add an cutoff calculated by operationTimeout, instead of only
depend on the value of actionsInProgress.
{code:java}
[ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 - id=1
error for test processing localhost,16020,1624343786295
java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at
[jira] [Updated] (HBASE-26027) The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException
[
https://issues.apache.org/jira/browse/HBASE-26027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Wang updated HBASE-26027:
---
Description:
[for 2.x]
The batch api of HTable contains a param named results to store result or
exception, its type is Object[].
If user pass an array with other type, eg:
org.apache.hadoop.hbase.client.Result, then the ArrayStoreException will occur
in AsyncRequestFutureImpl.updateResult, then the
AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
actionsInProgress again and again, forever.
It is better to add an cutoff calculated by operationTimeout, instead of only
depend on the value of actionsInProgress.
{code:java}
[ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 - id=1
error for test processing localhost,16020,1624343786295
java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10 actions to finish
on table:
[INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10 actions to finish
on table: test
{code}
was:
The batch api of HTable contains a param named results to store result or
exception, its type is Object[].
If user pass an array with other type, eg:
org.apache.hadoop.hbase.client.Result, then the ArrayStoreException will occur
in AsyncRequestFutureImpl.updateResult, then the
AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
actionsInProgress again and again, forever.
It is better to add an cutoff calculated by operationTimeout, instead of only
depend on the value of actionsInProgress.
{code:java}
[ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 - id=1
error for test processing localhost,16020,1624343786295
java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at
[GitHub] [hbase] bsglz opened a new pull request #3419: HBASE-26027 The calling of HTable.batch blocked at AsyncRequestFuture…
bsglz opened a new pull request #3419: URL: https://github.com/apache/hbase/pull/3419 …Impl.waitUntilDone caused by ArrayStoreException -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26027) The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException
[
https://issues.apache.org/jira/browse/HBASE-26027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Wang updated HBASE-26027:
---
Description:
The batch api of HTable contains a param named results to store result or
exception, its type is Object[].
If user pass an array with other type, eg:
org.apache.hadoop.hbase.client.Result, then the ArrayStoreException will occur
in AsyncRequestFutureImpl.updateResult, then the
AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
actionsInProgress again and again, forever.
It is better to add an cutoff calculated by operationTimeout, instead of only
depend on the value of actionsInProgress.
{code:java}
[ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 - id=1
error for test processing localhost,16020,1624343786295
java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10 actions to finish
on table:
[INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10 actions to finish
on table: test
{code}
was:
The batch api of HTable contains a param named results to store result or
exception, its type is Object[].
If user pass an array with other type, eg:
org.apache.hadoop.hbase.client.Result, then the ArrayStoreException will occur
in AsyncRequestFutureImpl.updateResult, then the
AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
actionsInProgress again and again, can not back.
It is better to add an cutoff calculated by operationTimeout, instead of only
depend on the value of actionsInProgress.
{code:java}
[ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 - id=1
error for test processing localhost,16020,1624343786295
java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at
[jira] [Created] (HBASE-26027) The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException
Zheng Wang created HBASE-26027:
--
Summary: The calling of HTable.batch blocked at
AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException
Key: HBASE-26027
URL: https://issues.apache.org/jira/browse/HBASE-26027
Project: HBase
Issue Type: Bug
Components: Client
Reporter: Zheng Wang
Assignee: Zheng Wang
The batch api of HTable contains a param named results to store result or
exception, its type is Object[].
If user pass an array with other type, eg:
org.apache.hadoop.hbase.client.Result, then the ArrayStoreException will occur
in AsyncRequestFutureImpl.updateResult, then the
AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
actionsInProgress again and again, can not back.
It is better to add an cutoff calculated by operationTimeout, instead of only
depend on the value of actionsInProgress.
{code:java}
[ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 - id=1
error for test processing localhost,16020,1624343786295
java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10 actions to finish
on table:
[INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10 actions to finish
on table: test
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
