[jira] [Updated] (HBASE-26552) Introduce retry to logroller when encounters IOException
[
https://issues.apache.org/jira/browse/HBASE-26552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaolin Ha updated HBASE-26552:
---
Description:
When calling RollController#rollWal in AbstractWALRoller, the regionserver may
abort when encounters exception,
{code:java}
...
} catch (FailedLogCloseException | ConnectException e) {
abort("Failed log close in log roller", e);
} catch (IOException ex) {
// Abort if we get here. We probably won't recover an IOE. HBASE-1132
abort("IOE in log roller",
ex instanceof RemoteException ? ((RemoteException)
ex).unwrapRemoteException() : ex);
} catch (Exception ex) {
LOG.error("Log rolling failed", ex);
abort("Log rolling failed", ex);
} {code}
I think we should support retry of rollWal here to avoid recovering the service
by killing regionserver. The restart of regionserver is costly and very not
friendly to the availability.
I find that when creating new writer for the WAL in
FanOutOneBlockAsyncDFSOutputHelper#createOutput, it supports retry to addBlock
by setting this config "hbase.fs.async.create.retries". The idea of retry to
roll WAL is similar to it, they both try best to make roll WAL succeed.
But the initialization of new WAL writer also includes flushing the write
buffer flush and waiting until it is completed by
AsyncProtobufLogWriter#writeMagicAndWALHeader, which can also fail by some
hardware reasons. The regionserver connected to the datanodes after addBlock,
but that not means the magic and header can be flushed successfully.
{code:java}
protected long writeMagicAndWALHeader(byte[] magic, WALHeader header) throws
IOException {
return write(future -> {
output.write(magic);
try {
header.writeDelimitedTo(asyncOutputWrapper);
} catch (IOException e) {
// should not happen
throw new AssertionError(e);
}
addListener(output.flush(false), (len, error) -> {
if (error != null) {
future.completeExceptionally(error);
} else {
future.complete(len);
}
});
});
}{code}
We have found that in our production clusters, there exists aborting of
regionservers that caused by "IOE in log roller". And the practice in our
clusters is that just one more retry of rollWal can make the WAL roll complete
and continue serving.
was:
When calling RollController#rollWal in AbstractWALRoller, the regionserver may
abort when encounters exception,
{code:java}
...
} catch (FailedLogCloseException | ConnectException e) {
abort("Failed log close in log roller", e);
} catch (IOException ex) {
// Abort if we get here. We probably won't recover an IOE. HBASE-1132
abort("IOE in log roller",
ex instanceof RemoteException ? ((RemoteException)
ex).unwrapRemoteException() : ex);
} catch (Exception ex) {
LOG.error("Log rolling failed", ex);
abort("Log rolling failed", ex);
} {code}
I think we should support retry of rollWal here to avoid recovering the service
by killing regionserver. The restart of regionserver is costly and very not
friendly to the availability.
I find that when creating new writer for the WAL in
FanOutOneBlockAsyncDFSOutputHelper#createOutput, it supports retry to addBlock
by setting this config "hbase.fs.async.create.retries".
But the initialization of new WAL writer also includes flushing the write
buffer flush and waiting until it is completed by
AsyncProtobufLogWriter#writeMagicAndWALHeader, which can also fail by some
hardware reasons. The regionserver connected to the datanodes after addBlock,
but that not means the magic and header can be flushed successfully.
{code:java}
protected long writeMagicAndWALHeader(byte[] magic, WALHeader header) throws
IOException {
return write(future -> {
output.write(magic);
try {
header.writeDelimitedTo(asyncOutputWrapper);
} catch (IOException e) {
// should not happen
throw new AssertionError(e);
}
addListener(output.flush(false), (len, error) -> {
if (error != null) {
future.completeExceptionally(error);
} else {
future.complete(len);
}
});
});
}{code}
We have found that in our production clusters, there exists aborting of
regionservers that caused by "IOE in log roller". And the practice in our
clusters is that just one more retry of rollWal can make the WAL roll complete
and continue serving.
> Introduce retry to logroller when encounters IOException
>
>
> Key: HBASE-26552
> URL: https://issues.apache.org/jira/browse/HBASE-26552
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 3.0.0-alpha-1, 2.0.0
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
>
> When calling RollController#rollWal in AbstractWALRoller, the regionserver
> may abort when
[jira] [Updated] (HBASE-26551) Add FastPath feature to HBase RWQueueRpcExecutor
[ https://issues.apache.org/jira/browse/HBASE-26551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yutong Xiao updated HBASE-26551: Description: In ticket [HBASE-17808|https://issues.apache.org/jira/browse/HBASE-17808], the author introduced a fastpath implementation for RWQueueRpcExecutor. It aggregated 3 different independent RpcExecutor to implement the mechanism. This redundancy costed more memory and from its own performance test, it cannot outperform the original implementation. This time, I directly extended RWQueueRpcExecutor to implement the fast path mechanism. From my test result, it has a better queue time performance than before. YCSB Test: Constant Configurations: hbase.regionserver.handler.count: 1000 hbase.ipc.server.callqueue.read.ratio: 0.5 hbase.ipc.server.callqueue.handler.factor: 0.2 Test Workload: YCSB: 50% Write, 25% Get, 25% Scan. Max Scan length: 1000. Client Threads: 100 ||FastPathRWQueueRpcExecutor||RWQueueRpcExecutor|| |[OVERALL], RunTime(ms), 909365 [OVERALL], Throughput(ops/sec), 5498.3422498116815 [TOTAL_GCS_PS_Scavenge], Count, 1208 [TOTAL_GC_TIME_PS_Scavenge], Time(ms), 8006 [TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.8803945610398465 [TOTAL_GCS_PS_MarkSweep], Count, 2 [TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 96 [TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.010556817119638429 [TOTAL_GCs], Count, 1210 [TOTAL_GC_TIME], Time(ms), 8102 [TOTAL_GC_TIME_%], Time(%), 0.8909513781594849 [READ], Operations, 1248885 [READ], AverageLatency(us), 14080.154160711354 [READ], MinLatency(us), 269 [READ], MaxLatency(us), 180735 [READ], 95thPercentileLatency(us), 29775 [READ], 99thPercentileLatency(us), 39391 [READ], Return=OK, 1248885 [CLEANUP], Operations, 200 [CLEANUP], AverageLatency(us), 311.78 [CLEANUP], MinLatency(us), 1 [CLEANUP], MaxLatency(us), 59647 [CLEANUP], 95thPercentileLatency(us), 26 [CLEANUP], 99thPercentileLatency(us), 173 [INSERT], Operations, 1251067 [INSERT], AverageLatency(us), 14235.898240461942 [INSERT], MinLatency(us), 393 [INSERT], MaxLatency(us), 204159 [INSERT], 95thPercentileLatency(us), 29919 [INSERT], 99thPercentileLatency(us), 39647 [INSERT], Return=OK, 1251067 [UPDATE], Operations, 1249582 [UPDATE], AverageLatency(us), 14166.923049467741 [UPDATE], MinLatency(us), 321 [UPDATE], MaxLatency(us), 203647 [UPDATE], 95thPercentileLatency(us), 29855 [UPDATE], 99thPercentileLatency(us), 39551 [UPDATE], Return=OK, 1249582 [SCAN], Operations, 1250466 [SCAN], AverageLatency(us), 30056.68854251135 [SCAN], MinLatency(us), 787 [SCAN], MaxLatency(us), 509183 [SCAN], 95thPercentileLatency(us), 57823 [SCAN], 99thPercentileLatency(us), 74751 [SCAN], Return=OK, 1250466|[OVERALL], RunTime(ms), 958763 [OVERALL], Throughput(ops/sec), 5215.053146606617 [TOTAL_GCS_PS_Scavenge], Count, 1264 [TOTAL_GC_TIME_PS_Scavenge], Time(ms), 8680 [TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.9053332262509086 [TOTAL_GCS_PS_MarkSweep], Count, 1 [TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 38 [TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.00396344039142103 [TOTAL_GCs], Count, 1265 [TOTAL_GC_TIME], Time(ms), 8718 [TOTAL_GC_TIME_%], Time(%), 0.909296423298 [READ], Operations, 1250961 [READ], AverageLatency(us), 14663.084518222391 [READ], MinLatency(us), 320 [READ], MaxLatency(us), 204415 [READ], 95thPercentileLatency(us), 30815 [READ], 99thPercentileLatency(us), 43071 [READ], Return=OK, 1250961 [CLEANUP], Operations, 200 [CLEANUP], AverageLatency(us), 366.845 [CLEANUP], MinLatency(us), 1 [CLEANUP], MaxLatency(us), 70719 [CLEANUP], 95thPercentileLatency(us), 36 [CLEANUP], 99thPercentileLatency(us), 80 [INSERT], Operations, 1248183 [INSERT], AverageLatency(us), 14334.938754974231 [INSERT], MinLatency(us), 390 [INSERT], MaxLatency(us), 2828287 [INSERT], 95thPercentileLatency(us), 30271 [INSERT], 99thPercentileLatency(us), 41919 [INSERT], Return=OK, 1248183 [UPDATE], Operations, 1250212 [UPDATE], AverageLatency(us), 14283.836318960304 [UPDATE], MinLatency(us), 337 [UPDATE], MaxLatency(us), 2828287 [UPDATE], 95thPercentileLatency(us), 30255 [UPDATE], 99thPercentileLatency(us), 41855 [UPDATE], Return=OK, 1250212 [SCAN], Operations, 1250644 [SCAN], AverageLatency(us), 33153.01709839091 [SCAN], MinLatency(us), 742 [SCAN], MaxLatency(us), 645119 [SCAN], 95thPercentileLatency(us), 62879 [SCAN], 99thPercentileLatency(us), 80447 [SCAN], Return=OK, 1250644| In the metrics screenshot, the first interval is the performance of RWQueueRpcExecutor and the second interval is FastPathRWQueueRpcExecutor. was: In ticket [HBASE-17808|https://issues.apache.org/jira/browse/HBASE-17808], the author introduced a fastpath implementation for RWQueueRpcExecutor. It aggregated 3 different independent RpcExecutor to implement the mechanism. This redundancy costed more memory and from its own performance test, it cannot outperform the original implementation. This time, I directly extended RWQueueRpcExecutor to implement the fast path mechanism. From my test
[jira] [Updated] (HBASE-26551) Add FastPath feature to HBase RWQueueRpcExecutor
[ https://issues.apache.org/jira/browse/HBASE-26551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yutong Xiao updated HBASE-26551: Description: In ticket [HBASE-17808|https://issues.apache.org/jira/browse/HBASE-17808], the author introduced a fastpath implementation for RWQueueRpcExecutor. It aggregated 3 different independent RpcExecutor to implement the mechanism. This redundancy costed more memory and from its own performance test, it cannot outperform the original implementation. This time, I directly extended RWQueueRpcExecutor to implement the fast path mechanism. From my test result, it has a better queue time performance than before. YCSB Test: Constant Configurations: hbase.regionserver.handler.count: 1000 hbase.ipc.server.callqueue.read.ratio: 0.5 hbase.ipc.server.callqueue.handler.factor: 0.2 Test Workload: YCSB: 50% Write, 25% Get, 25% Scan. Max Scan length: 1000. Client Threads: 100 ||FastPathRWQueueRpcExecutor||RWQueueRpcExecutor|| |[OVERALL], RunTime(ms), 909365 [OVERALL], Throughput(ops/sec), 5498.3422498116815 [TOTAL_GCS_PS_Scavenge], Count, 1208 [TOTAL_GC_TIME_PS_Scavenge], Time(ms), 8006 [TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.8803945610398465 [TOTAL_GCS_PS_MarkSweep], Count, 2 [TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 96 [TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.010556817119638429 [TOTAL_GCs], Count, 1210 [TOTAL_GC_TIME], Time(ms), 8102 [TOTAL_GC_TIME_%], Time(%), 0.8909513781594849 [READ], Operations, 1248885 [READ], AverageLatency(us), 14080.154160711354 [READ], MinLatency(us), 269 [READ], MaxLatency(us), 180735 [READ], 95thPercentileLatency(us), 29775 [READ], 99thPercentileLatency(us), 39391 [READ], Return=OK, 1248885 [CLEANUP], Operations, 200 [CLEANUP], AverageLatency(us), 311.78 [CLEANUP], MinLatency(us), 1 [CLEANUP], MaxLatency(us), 59647 [CLEANUP], 95thPercentileLatency(us), 26 [CLEANUP], 99thPercentileLatency(us), 173 [INSERT], Operations, 1251067 [INSERT], AverageLatency(us), 14235.898240461942 [INSERT], MinLatency(us), 393 [INSERT], MaxLatency(us), 204159 [INSERT], 95thPercentileLatency(us), 29919 [INSERT], 99thPercentileLatency(us), 39647 [INSERT], Return=OK, 1251067 [UPDATE], Operations, 1249582 [UPDATE], AverageLatency(us), 14166.923049467741 [UPDATE], MinLatency(us), 321 [UPDATE], MaxLatency(us), 203647 [UPDATE], 95thPercentileLatency(us), 29855 [UPDATE], 99thPercentileLatency(us), 39551 [UPDATE], Return=OK, 1249582 [SCAN], Operations, 1250466 [SCAN], AverageLatency(us), 30056.68854251135 [SCAN], MinLatency(us), 787 [SCAN], MaxLatency(us), 509183 [SCAN], 95thPercentileLatency(us), 57823 [SCAN], 99thPercentileLatency(us), 74751 [SCAN], Return=OK, 1250466|[OVERALL], RunTime(ms), 958763 [OVERALL], Throughput(ops/sec), 5215.053146606617 [TOTAL_GCS_PS_Scavenge], Count, 1264 [TOTAL_GC_TIME_PS_Scavenge], Time(ms), 8680 [TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.9053332262509086 [TOTAL_GCS_PS_MarkSweep], Count, 1 [TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 38 [TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.00396344039142103 [TOTAL_GCs], Count, 1265 [TOTAL_GC_TIME], Time(ms), 8718 [TOTAL_GC_TIME_%], Time(%), 0.909296423298 [READ], Operations, 1250961 [READ], AverageLatency(us), 14663.084518222391 [READ], MinLatency(us), 320 [READ], MaxLatency(us), 204415 [READ], 95thPercentileLatency(us), 30815 [READ], 99thPercentileLatency(us), 43071 [READ], Return=OK, 1250961 [CLEANUP], Operations, 200 [CLEANUP], AverageLatency(us), 366.845 [CLEANUP], MinLatency(us), 1 [CLEANUP], MaxLatency(us), 70719 [CLEANUP], 95thPercentileLatency(us), 36 [CLEANUP], 99thPercentileLatency(us), 80 [INSERT], Operations, 1248183 [INSERT], AverageLatency(us), 14334.938754974231 [INSERT], MinLatency(us), 390 [INSERT], MaxLatency(us), 2828287 [INSERT], 95thPercentileLatency(us), 30271 [INSERT], 99thPercentileLatency(us), 41919 [INSERT], Return=OK, 1248183 [UPDATE], Operations, 1250212 [UPDATE], AverageLatency(us), 14283.836318960304 [UPDATE], MinLatency(us), 337 [UPDATE], MaxLatency(us), 2828287 [UPDATE], 95thPercentileLatency(us), 30255 [UPDATE], 99thPercentileLatency(us), 41855 [UPDATE], Return=OK, 1250212 [SCAN], Operations, 1250644 [SCAN], AverageLatency(us), 33153.01709839091 [SCAN], MinLatency(us), 742 [SCAN], MaxLatency(us), 645119 [SCAN], 95thPercentileLatency(us), 62879 [SCAN], 99thPercentileLatency(us), 80447 [SCAN], Return=OK, 1250644| was:In ticket [HBASE-17808|https://issues.apache.org/jira/browse/HBASE-17808], the author introduced a fastpath implementation for RWQueueRpcExecutor. It aggregated 3 different independent RpcExecutor to implement the mechanism. This redundancy costed more memory and from its own performance test, it cannot outperform the original implementation. This time, I directly extended RWQueueRpcExecutor to implement the fast path mechanism. From my test result, it has a better queue time performance than before. > Add FastPath feature to HBase RWQueueRpcExecutor >
[jira] [Created] (HBASE-26552) Introduce retry to logroller when encounters IOException
Xiaolin Ha created HBASE-26552:
--
Summary: Introduce retry to logroller when encounters IOException
Key: HBASE-26552
URL: https://issues.apache.org/jira/browse/HBASE-26552
Project: HBase
Issue Type: Improvement
Components: wal
Affects Versions: 2.0.0, 3.0.0-alpha-1
Reporter: Xiaolin Ha
Assignee: Xiaolin Ha
When calling RollController#rollWal in AbstractWALRoller, the regionserver may
abort when encounters exception,
{code:java}
...
} catch (FailedLogCloseException | ConnectException e) {
abort("Failed log close in log roller", e);
} catch (IOException ex) {
// Abort if we get here. We probably won't recover an IOE. HBASE-1132
abort("IOE in log roller",
ex instanceof RemoteException ? ((RemoteException)
ex).unwrapRemoteException() : ex);
} catch (Exception ex) {
LOG.error("Log rolling failed", ex);
abort("Log rolling failed", ex);
} {code}
I think we should support retry of rollWal here to avoid recovering the service
by killing regionserver. The restart of regionserver is costly and very not
friendly to the availability.
I find that when creating new writer for the WAL in
FanOutOneBlockAsyncDFSOutputHelper#createOutput, it supports retry to addBlock
by setting this config "hbase.fs.async.create.retries".
But the initialization of new WAL writer also includes flushing the write
buffer flush and waiting until it is completed by
AsyncProtobufLogWriter#writeMagicAndWALHeader, which can also fail by some
hardware reasons. The regionserver connected to the datanodes after addBlock,
but that not means the magic and header can be flushed successfully.
{code:java}
protected long writeMagicAndWALHeader(byte[] magic, WALHeader header) throws
IOException {
return write(future -> {
output.write(magic);
try {
header.writeDelimitedTo(asyncOutputWrapper);
} catch (IOException e) {
// should not happen
throw new AssertionError(e);
}
addListener(output.flush(false), (len, error) -> {
if (error != null) {
future.completeExceptionally(error);
} else {
future.complete(len);
}
});
});
}{code}
We have found that in our production clusters, there exists aborting of
regionservers that caused by "IOE in log roller". And the practice in our
clusters is that just one more retry of rollWal can make the WAL roll complete
and continue serving.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26551) Add FastPath feature to HBase RWQueueRpcExecutor
[ https://issues.apache.org/jira/browse/HBASE-26551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yutong Xiao updated HBASE-26551: Attachment: QueueTimeComparisonWithMax.png > Add FastPath feature to HBase RWQueueRpcExecutor > > > Key: HBASE-26551 > URL: https://issues.apache.org/jira/browse/HBASE-26551 > Project: HBase > Issue Type: Task > Components: rpc >Reporter: Yutong Xiao >Assignee: Yutong Xiao >Priority: Major > Attachments: QueueTimeComparison.png, QueueTimeComparisonWithMax.png > > > In ticket [HBASE-17808|https://issues.apache.org/jira/browse/HBASE-17808], > the author introduced a fastpath implementation for RWQueueRpcExecutor. It > aggregated 3 different independent RpcExecutor to implement the mechanism. > This redundancy costed more memory and from its own performance test, it > cannot outperform the original implementation. This time, I directly extended > RWQueueRpcExecutor to implement the fast path mechanism. From my test result, > it has a better queue time performance than before. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26551) Add FastPath feature to HBase RWQueueRpcExecutor
[ https://issues.apache.org/jira/browse/HBASE-26551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yutong Xiao updated HBASE-26551: Attachment: QueueTimeComparison.png > Add FastPath feature to HBase RWQueueRpcExecutor > > > Key: HBASE-26551 > URL: https://issues.apache.org/jira/browse/HBASE-26551 > Project: HBase > Issue Type: Task > Components: rpc >Reporter: Yutong Xiao >Assignee: Yutong Xiao >Priority: Major > Attachments: QueueTimeComparison.png > > > In ticket [HBASE-17808|https://issues.apache.org/jira/browse/HBASE-17808], > the author introduced a fastpath implementation for RWQueueRpcExecutor. It > aggregated 3 different independent RpcExecutor to implement the mechanism. > This redundancy costed more memory and from its own performance test, it > cannot outperform the original implementation. This time, I directly extended > RWQueueRpcExecutor to implement the fast path mechanism. From my test result, > it has a better queue time performance than before. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] YutSean commented on pull request #3929: HBASE-26551 Add FastPath feature to HBase RWQueueRpcExecutor
YutSean commented on pull request #3929: URL: https://github.com/apache/hbase/pull/3929#issuecomment-989577939 I have two considerations. 1. Wether we should make the fastpath as a new config that user can turn on or off alternatively (Because it seems that when the number of handlers is really small, fastpath does not has such good performance). 2. I copy the fastpath handler from FastPathBalancedRpcExecutor. This looks a little bit ugly. Should we extract the fastpath mechanism in a new file and just import it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean opened a new pull request #3929: HBASE-26551 Add FastPath feature to HBase RWQueueRpcExecutor
YutSean opened a new pull request #3929: URL: https://github.com/apache/hbase/pull/3929 https://issues.apache.org/jira/browse/HBASE-26551 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26551) Add FastPath feature to HBase RWQueueRpcExecutor
[ https://issues.apache.org/jira/browse/HBASE-26551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yutong Xiao updated HBASE-26551: Description: In ticket [HBASE-17808|https://issues.apache.org/jira/browse/HBASE-17808], the author introduced a fastpath implementation for RWQueueRpcExecutor. It aggregated 3 different independent RpcExecutor to implement the mechanism. This redundancy costed more memory and from its own performance test, it cannot outperform the original implementation. This time, I directly extended RWQueueRpcExecutor to implement the fast path mechanism. From my test result, it has a better queue time performance than before. > Add FastPath feature to HBase RWQueueRpcExecutor > > > Key: HBASE-26551 > URL: https://issues.apache.org/jira/browse/HBASE-26551 > Project: HBase > Issue Type: Task > Components: rpc >Reporter: Yutong Xiao >Assignee: Yutong Xiao >Priority: Major > > In ticket [HBASE-17808|https://issues.apache.org/jira/browse/HBASE-17808], > the author introduced a fastpath implementation for RWQueueRpcExecutor. It > aggregated 3 different independent RpcExecutor to implement the mechanism. > This redundancy costed more memory and from its own performance test, it > cannot outperform the original implementation. This time, I directly extended > RWQueueRpcExecutor to implement the fast path mechanism. From my test result, > it has a better queue time performance than before. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26551) Add FastPath feature to HBase RWQueueRpcExecutor
Yutong Xiao created HBASE-26551: --- Summary: Add FastPath feature to HBase RWQueueRpcExecutor Key: HBASE-26551 URL: https://issues.apache.org/jira/browse/HBASE-26551 Project: HBase Issue Type: Task Components: rpc Reporter: Yutong Xiao Assignee: Yutong Xiao -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #3908: HBASE-26530 Backport HBASE-26524 Support remove coprocessor by class …
Apache-HBase commented on pull request #3908: URL: https://github.com/apache/hbase/pull/3908#issuecomment-989566231 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 44s | branch-2 passed | | +1 :green_heart: | compile | 2m 30s | branch-2 passed | | +1 :green_heart: | shadedjars | 8m 47s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 45s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 14s | the patch passed | | +1 :green_heart: | compile | 2m 26s | the patch passed | | +1 :green_heart: | javac | 2m 26s | the patch passed | | +1 :green_heart: | shadedjars | 8m 40s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 23s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 257m 26s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 7m 29s | hbase-shell in the patch passed. | | | | 310m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3908/4/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3908 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ca1dbbefd313 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / b1bc5f3a5c | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3908/4/testReport/ | | Max. process+thread count | 2612 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3908/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3908: HBASE-26530 Backport HBASE-26524 Support remove coprocessor by class …
Apache-HBase commented on pull request #3908: URL: https://github.com/apache/hbase/pull/3908#issuecomment-989537069 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 1s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 4s | branch-2 passed | | +1 :green_heart: | compile | 2m 10s | branch-2 passed | | +1 :green_heart: | shadedjars | 8m 2s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 26s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 48s | the patch passed | | +1 :green_heart: | compile | 2m 9s | the patch passed | | +1 :green_heart: | javac | 2m 9s | the patch passed | | +1 :green_heart: | shadedjars | 7m 58s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 51s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 203m 48s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 7m 4s | hbase-shell in the patch passed. | | | | 250m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3908/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3908 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4d82c954d5e0 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / b1bc5f3a5c | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3908/4/testReport/ | | Max. process+thread count | 2910 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3908/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3928: HBASE-26550 Make sure the master is running normally before accepting…
Apache-HBase commented on pull request #3928: URL: https://github.com/apache/hbase/pull/3928#issuecomment-989512493 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 32s | master passed | | +1 :green_heart: | compile | 1m 4s | master passed | | +1 :green_heart: | shadedjars | 9m 6s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 13s | the patch passed | | +1 :green_heart: | compile | 1m 5s | the patch passed | | +1 :green_heart: | javac | 1m 5s | the patch passed | | +1 :green_heart: | shadedjars | 9m 5s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 213m 13s | hbase-server in the patch passed. | | | | 246m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3928/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3928 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 38b380b0da92 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / ca3ba494cb | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3928/1/testReport/ | | Max. process+thread count | 2853 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3928/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26027) The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException
[
https://issues.apache.org/jira/browse/HBASE-26027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Wang updated HBASE-26027:
---
Fix Version/s: 2.6.0
> The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone
> caused by ArrayStoreException
> -
>
> Key: HBASE-26027
> URL: https://issues.apache.org/jira/browse/HBASE-26027
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.2.7, 2.5.0, 2.3.5, 2.4.4
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
> Fix For: 2.6.0
>
>
> The batch api of HTable contains a param named results to store result or
> exception, its type is Object[].
> If user pass an array with other type, eg:
> org.apache.hadoop.hbase.client.Result, and if we need to put an exception
> into it by some reason, then the ArrayStoreException will occur in
> AsyncRequestFutureImpl.updateResult, then the
> AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
> AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
> actionsInProgress again and again, forever.
> It is better to add an cutoff calculated by operationTimeout, instead of only
> depend on the value of actionsInProgress.
> BTW, this issue only for 2.x, since 3.x the implement has refactored.
> How to reproduce:
> 1: add sleep in RSRpcServices.multi to mock slow response
> {code:java}
> try {
> Thread.sleep(2000);
> } catch (InterruptedException e) {
> e.printStackTrace();
> }
> {code}
> 2: set time out in config
> {code:java}
> conf.set("hbase.rpc.timeout","2000");
> conf.set("hbase.client.operation.timeout","6000");
> {code}
> 3: call batch api
> {code:java}
> Table table = HbaseUtil.getTable("test");
> byte[] cf = Bytes.toBytes("f");
> byte[] c = Bytes.toBytes("c1");
> List gets = new ArrayList<>();
> for (int i = 0; i < 10; i++) {
> byte[] rk = Bytes.toBytes("rk-" + i);
> Get get = new Get(rk);
> get.addColumn(cf, c);
> gets.add(get);
> }
> Result[] results = new Result[gets.size()];
> table.batch(gets, results);
> {code}
> The log will looks like below:
> {code:java}
> [ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 -
> id=1 error for test processing localhost,16020,1624343786295
> java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10 actions to
> finish on table:
> [INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10 actions to
> finish on table: test
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (HBASE-26027) The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException
[
https://issues.apache.org/jira/browse/HBASE-26027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456137#comment-17456137
]
Zheng Wang edited comment on HBASE-26027 at 12/9/21, 4:33 AM:
--
Pushed a new PR #3925, just for branch-2.
[~apurtell]
was (Author: filtertip):
Pushed a new PR #3925, just for branch-2.
And not plan to push 2.3.x and 2.4.x since they are already stable.
[~apurtell]
> The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone
> caused by ArrayStoreException
> -
>
> Key: HBASE-26027
> URL: https://issues.apache.org/jira/browse/HBASE-26027
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.2.7, 2.5.0, 2.3.5, 2.4.4
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
>
> The batch api of HTable contains a param named results to store result or
> exception, its type is Object[].
> If user pass an array with other type, eg:
> org.apache.hadoop.hbase.client.Result, and if we need to put an exception
> into it by some reason, then the ArrayStoreException will occur in
> AsyncRequestFutureImpl.updateResult, then the
> AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
> AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
> actionsInProgress again and again, forever.
> It is better to add an cutoff calculated by operationTimeout, instead of only
> depend on the value of actionsInProgress.
> BTW, this issue only for 2.x, since 3.x the implement has refactored.
> How to reproduce:
> 1: add sleep in RSRpcServices.multi to mock slow response
> {code:java}
> try {
> Thread.sleep(2000);
> } catch (InterruptedException e) {
> e.printStackTrace();
> }
> {code}
> 2: set time out in config
> {code:java}
> conf.set("hbase.rpc.timeout","2000");
> conf.set("hbase.client.operation.timeout","6000");
> {code}
> 3: call batch api
> {code:java}
> Table table = HbaseUtil.getTable("test");
> byte[] cf = Bytes.toBytes("f");
> byte[] c = Bytes.toBytes("c1");
> List gets = new ArrayList<>();
> for (int i = 0; i < 10; i++) {
> byte[] rk = Bytes.toBytes("rk-" + i);
> Get get = new Get(rk);
> get.addColumn(cf, c);
> gets.add(get);
> }
> Result[] results = new Result[gets.size()];
> table.batch(gets, results);
> {code}
> The log will looks like below:
> {code:java}
> [ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 -
> id=1 error for test processing localhost,16020,1624343786295
> java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10 actions to
> finish on table:
> [INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10 actions to
> finish on table: test
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (HBASE-26027) The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException
[
https://issues.apache.org/jira/browse/HBASE-26027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456137#comment-17456137
]
Zheng Wang edited comment on HBASE-26027 at 12/9/21, 4:29 AM:
--
Pushed a new PR #3925, just for branch-2.
And not plan to push 2.3.x and 2.4.x since they are already stable.
[~apurtell]
was (Author: filtertip):
Pushed a new PR #3925, just for branch-2.
And not plan to push 2.3.x and 2.4.x since they are already stable.
> The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone
> caused by ArrayStoreException
> -
>
> Key: HBASE-26027
> URL: https://issues.apache.org/jira/browse/HBASE-26027
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.2.7, 2.5.0, 2.3.5, 2.4.4
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
>
> The batch api of HTable contains a param named results to store result or
> exception, its type is Object[].
> If user pass an array with other type, eg:
> org.apache.hadoop.hbase.client.Result, and if we need to put an exception
> into it by some reason, then the ArrayStoreException will occur in
> AsyncRequestFutureImpl.updateResult, then the
> AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
> AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
> actionsInProgress again and again, forever.
> It is better to add an cutoff calculated by operationTimeout, instead of only
> depend on the value of actionsInProgress.
> BTW, this issue only for 2.x, since 3.x the implement has refactored.
> How to reproduce:
> 1: add sleep in RSRpcServices.multi to mock slow response
> {code:java}
> try {
> Thread.sleep(2000);
> } catch (InterruptedException e) {
> e.printStackTrace();
> }
> {code}
> 2: set time out in config
> {code:java}
> conf.set("hbase.rpc.timeout","2000");
> conf.set("hbase.client.operation.timeout","6000");
> {code}
> 3: call batch api
> {code:java}
> Table table = HbaseUtil.getTable("test");
> byte[] cf = Bytes.toBytes("f");
> byte[] c = Bytes.toBytes("c1");
> List gets = new ArrayList<>();
> for (int i = 0; i < 10; i++) {
> byte[] rk = Bytes.toBytes("rk-" + i);
> Get get = new Get(rk);
> get.addColumn(cf, c);
> gets.add(get);
> }
> Result[] results = new Result[gets.size()];
> table.batch(gets, results);
> {code}
> The log will looks like below:
> {code:java}
> [ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 -
> id=1 error for test processing localhost,16020,1624343786295
> java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10 actions to
> finish on table:
> [INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10 actions to
> finish on table: test
> {code}
--
This message
[jira] [Commented] (HBASE-26027) The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException
[
https://issues.apache.org/jira/browse/HBASE-26027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456137#comment-17456137
]
Zheng Wang commented on HBASE-26027:
Pushed a new PR #3925, just for branch-2.
And not plan to push 2.3.x and 2.4.x since they are already stable.
> The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone
> caused by ArrayStoreException
> -
>
> Key: HBASE-26027
> URL: https://issues.apache.org/jira/browse/HBASE-26027
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.2.7, 2.5.0, 2.3.5, 2.4.4
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
>
> The batch api of HTable contains a param named results to store result or
> exception, its type is Object[].
> If user pass an array with other type, eg:
> org.apache.hadoop.hbase.client.Result, and if we need to put an exception
> into it by some reason, then the ArrayStoreException will occur in
> AsyncRequestFutureImpl.updateResult, then the
> AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
> AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
> actionsInProgress again and again, forever.
> It is better to add an cutoff calculated by operationTimeout, instead of only
> depend on the value of actionsInProgress.
> BTW, this issue only for 2.x, since 3.x the implement has refactored.
> How to reproduce:
> 1: add sleep in RSRpcServices.multi to mock slow response
> {code:java}
> try {
> Thread.sleep(2000);
> } catch (InterruptedException e) {
> e.printStackTrace();
> }
> {code}
> 2: set time out in config
> {code:java}
> conf.set("hbase.rpc.timeout","2000");
> conf.set("hbase.client.operation.timeout","6000");
> {code}
> 3: call batch api
> {code:java}
> Table table = HbaseUtil.getTable("test");
> byte[] cf = Bytes.toBytes("f");
> byte[] c = Bytes.toBytes("c1");
> List gets = new ArrayList<>();
> for (int i = 0; i < 10; i++) {
> byte[] rk = Bytes.toBytes("rk-" + i);
> Get get = new Get(rk);
> get.addColumn(cf, c);
> gets.add(get);
> }
> Result[] results = new Result[gets.size()];
> table.batch(gets, results);
> {code}
> The log will looks like below:
> {code:java}
> [ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 -
> id=1 error for test processing localhost,16020,1624343786295
> java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10 actions to
> finish on table:
> [INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10 actions to
> finish on table: test
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #3928: HBASE-26550 Make sure the master is running normally before accepting…
Apache-HBase commented on pull request #3928: URL: https://github.com/apache/hbase/pull/3928#issuecomment-989480554 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 43s | master passed | | +1 :green_heart: | compile | 1m 13s | master passed | | +1 :green_heart: | shadedjars | 8m 18s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 28s | the patch passed | | +1 :green_heart: | compile | 1m 13s | the patch passed | | +1 :green_heart: | javac | 1m 13s | the patch passed | | +1 :green_heart: | shadedjars | 8m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 138m 22s | hbase-server in the patch passed. | | | | 170m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3928/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3928 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 86383b889ce1 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / ca3ba494cb | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3928/1/testReport/ | | Max. process+thread count | 4521 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3928/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3908: HBASE-26530 Backport HBASE-26524 Support remove coprocessor by class …
Apache-HBase commented on pull request #3908: URL: https://github.com/apache/hbase/pull/3908#issuecomment-989448873 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 44s | branch-2 passed | | +1 :green_heart: | compile | 4m 49s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 48s | branch-2 passed | | +1 :green_heart: | spotbugs | 3m 16s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 29s | the patch passed | | +1 :green_heart: | compile | 4m 53s | the patch passed | | +1 :green_heart: | javac | 4m 53s | the patch passed | | +1 :green_heart: | checkstyle | 2m 1s | the patch passed | | -0 :warning: | rubocop | 0m 21s | The patch generated 23 new + 620 unchanged - 5 fixed = 643 total (was 625) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 31s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 3m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 39s | The patch does not generate ASF License warnings. | | | | 51m 11s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3908/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3908 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile rubocop | | uname | Linux 180edecc5b38 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / b1bc5f3a5c | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | rubocop | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3908/4/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 96 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3908/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 rubocop=0.80.0 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #3922: HBASE-26541 hbase-protocol-shaded not buildable on M1 MacOSX
Apache9 commented on pull request #3922: URL: https://github.com/apache/hbase/pull/3922#issuecomment-989443873 Ah, sorry to be late here. I'm +1 on this PR. Thanks Andrew for fixing this on M1. I think there will be more and more developers using M1 MBP to it is important to make sure hbase can work on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #3924: HBASE-26542 Apply a `package` to test protobuf files
Apache9 commented on pull request #3924: URL: https://github.com/apache/hbase/pull/3924#issuecomment-989439991 So which PR is effected? In the past we have a lot of PRs which changes the protobuf files but I haven't seen any problems on the pre commit run... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26533) KeyValueScanner might not be properly closed when using InternalScan.checkOnlyMemStore()
[
https://issues.apache.org/jira/browse/HBASE-26533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26533:
Fix Version/s: 2.5.0
3.0.0-alpha-2
2.4.9
2.6.0
> KeyValueScanner might not be properly closed when using
> InternalScan.checkOnlyMemStore()
>
>
> Key: HBASE-26533
> URL: https://issues.apache.org/jira/browse/HBASE-26533
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.6
>Reporter: Raman Chodźka
>Assignee: Raman Chodźka
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.9, 2.6.0
>
> Attachments: HBASE-26533.patch
>
>
> While writing a custom RegionObserver and using
> InternalScan.checkOnlyMemStore() I stumbled upon an issue. The number of
> files opened by region servers would grow steadily and eventually region
> servers would crash with
> {code:java}
> 2021-11-15 00:54:34,290 ERROR [MemStoreFlusher.1] regionserver.HStore: Failed
> to commit store file
> hdfs://<...removed...>:8020/hbase/data/default/<...removed...>/743071139057c819d7e6f7b59f065152/.tmp/f/394ba71102ec401d8779aa5f45819f84
> java.io.IOException: Failed on local exception: java.io.IOException: Too many
> open files; Host Details : local host is: "<...removed...>/<...removed...>";
> destination host is: "<...removed...>":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:805)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1544)
> at org.apache.hadoop.ipc.Client.call(Client.java:1486)
> at org.apache.hadoop.ipc.Client.call(Client.java:1385)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy27.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:800)
> {code}
> Another symptom is the following messages in region server logs:
> {code:java}
> 2021-11-18 13:41:29,539 INFO
> [RS_COMPACTED_FILES_DISCHARGER-regionserver/<...removed...>:16020-7]
> regionserver.HStore: Can't archive compacted file
> hdfs://<...removed...>:8020/hbase/data/default/<...removed...>/ce6f08fdd82967df94d1c83e289d3142/f/9b01b8bdea324c22a92f1ba6b386e050.6261a24c6a689d5406ae1ea87dc9bb9f
> because of either isCompactedAway=true or file has reference,
> isReferencedInReads=true, refCount=7, skipping for now.
> {code}
> The culprit is KeyValueScanner not being closed in
> StoreScanner.selectScannersFrom() before `continue`
> [https://github.com/apache/hbase/blame/f000b775320330eb2f426f6b2a3b5e27a794a707/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java#L467]
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #3928: HBASE-26550 Make sure the master is running normally before accepting…
Apache-HBase commented on pull request #3928: URL: https://github.com/apache/hbase/pull/3928#issuecomment-989420895 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 17s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 36s | master passed | | +1 :green_heart: | compile | 3m 34s | master passed | | +1 :green_heart: | checkstyle | 1m 17s | master passed | | +1 :green_heart: | spotbugs | 2m 19s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 16s | the patch passed | | +1 :green_heart: | compile | 3m 25s | the patch passed | | +1 :green_heart: | javac | 3m 25s | the patch passed | | +1 :green_heart: | checkstyle | 1m 13s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 21m 55s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotbugs | 2m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 55m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3928/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3928 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 57ad05278715 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / ca3ba494cb | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3928/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3908: HBASE-26530 Backport HBASE-26524 Support remove coprocessor by class …
Apache-HBase commented on pull request #3908: URL: https://github.com/apache/hbase/pull/3908#issuecomment-989408831 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 3s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 33s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 59s | branch-2 passed | | +1 :green_heart: | compile | 1m 49s | branch-2 passed | | +1 :green_heart: | shadedjars | 7m 9s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 14s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 54s | the patch passed | | +1 :green_heart: | compile | 1m 52s | the patch passed | | +1 :green_heart: | javac | 1m 52s | the patch passed | | +1 :green_heart: | shadedjars | 7m 5s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 12s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 216m 2s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 9m 32s | hbase-shell in the patch passed. | | | | 260m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3908/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3908 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ea14f5b8c00f 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 140b5d8d26 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3908/3/testReport/ | | Max. process+thread count | 2821 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3908/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3908: HBASE-26530 Backport HBASE-26524 Support remove coprocessor by class …
Apache-HBase commented on pull request #3908: URL: https://github.com/apache/hbase/pull/3908#issuecomment-989399578 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 3s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 2s | branch-2 passed | | +1 :green_heart: | compile | 2m 8s | branch-2 passed | | +1 :green_heart: | shadedjars | 8m 5s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 28s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 48s | the patch passed | | +1 :green_heart: | compile | 2m 9s | the patch passed | | +1 :green_heart: | javac | 2m 9s | the patch passed | | +1 :green_heart: | shadedjars | 7m 56s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 56s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 203m 38s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 7m 13s | hbase-shell in the patch passed. | | | | 255m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3908/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3908 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 010c05ce9c6f 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 140b5d8d26 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3908/3/testReport/ | | Max. process+thread count | 2710 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3908/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-13503) Encryption improvements umbrella
[ https://issues.apache.org/jira/browse/HBASE-13503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-13503: Summary: Encryption improvements umbrella (was: Encryption improvements for 2.5) > Encryption improvements umbrella > > > Key: HBASE-13503 > URL: https://issues.apache.org/jira/browse/HBASE-13503 > Project: HBase > Issue Type: Umbrella > Components: encryption, security >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2 > > > Repurposing this JIRA for a review of encryption for a potential release > 2.5.0. > Umbrella issue for a collection of related encryption improvements: > - Additional ciphers > - Better data key derivation > - Support master key rotation without process restarts (where applicable) > - Performance or functional improvements -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-13503) Encryption improvements umbrella
[ https://issues.apache.org/jira/browse/HBASE-13503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-13503: Fix Version/s: 2.6.0 > Encryption improvements umbrella > > > Key: HBASE-13503 > URL: https://issues.apache.org/jira/browse/HBASE-13503 > Project: HBase > Issue Type: Umbrella > Components: encryption, security >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2, 2.6.0 > > > Repurposing this JIRA for a review of encryption for a potential release > 2.5.0. > Umbrella issue for a collection of related encryption improvements: > - Additional ciphers > - Better data key derivation > - Support master key rotation without process restarts (where applicable) > - Performance or functional improvements -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] ndimiduk commented on pull request #3924: HBASE-26542 Apply a `package` to test protobuf files
ndimiduk commented on pull request #3924: URL: https://github.com/apache/hbase/pull/3924#issuecomment-989389410 This is better. It didn't run any tests impacted by the changed protoc files, but it didn't fail either. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-13503) Encryption improvements for 2.5
[ https://issues.apache.org/jira/browse/HBASE-13503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell reassigned HBASE-13503: --- Assignee: Andrew Kyle Purtell > Encryption improvements for 2.5 > --- > > Key: HBASE-13503 > URL: https://issues.apache.org/jira/browse/HBASE-13503 > Project: HBase > Issue Type: Umbrella > Components: encryption, security >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2 > > > Repurposing this JIRA for a review of encryption for a potential release > 2.5.0. > Umbrella issue for a collection of related encryption improvements: > - Additional ciphers > - Better data key derivation > - Support master key rotation without process restarts (where applicable) > - Performance or functional improvements -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-13503) Encryption improvements for 2.5
[ https://issues.apache.org/jira/browse/HBASE-13503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456084#comment-17456084 ] Andrew Kyle Purtell commented on HBASE-13503: - Based on the number of watchers on this issue looks like I wrote a check I should cash. > Encryption improvements for 2.5 > --- > > Key: HBASE-13503 > URL: https://issues.apache.org/jira/browse/HBASE-13503 > Project: HBase > Issue Type: Umbrella > Components: encryption, security >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2 > > > Repurposing this JIRA for a review of encryption for a potential release > 2.5.0. > Umbrella issue for a collection of related encryption improvements: > - Additional ciphers > - Better data key derivation > - Support master key rotation without process restarts (where applicable) > - Performance or functional improvements -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-20897) Port HBASE-20866 "HBase 1.x scan performance degradation compared to 0.98 version" to branch-2 and up
[ https://issues.apache.org/jira/browse/HBASE-20897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-20897: Fix Version/s: (was: 2.5.0) > Port HBASE-20866 "HBase 1.x scan performance degradation compared to 0.98 > version" to branch-2 and up > - > > Key: HBASE-20897 > URL: https://issues.apache.org/jira/browse/HBASE-20897 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-2 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-22671) ByteBufferUtils#findCommonPrefix() can be safely changed to ArraysSupport#mismatch()
[
https://issues.apache.org/jira/browse/HBASE-22671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-22671:
Affects Version/s: 2.6.0
> ByteBufferUtils#findCommonPrefix() can be safely changed to
> ArraysSupport#mismatch()
>
>
> Key: HBASE-22671
> URL: https://issues.apache.org/jira/browse/HBASE-22671
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.4.0, 2.5.0, 2.6.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
>
> Microbenchmarks reveal that finding the common prefix for encoders can safely
> be replaced with ArraysSupport#mismatch().
> the microbenchmark just compares Cells that are backed with array and BB.
> For a 27 bit common row prefix the existing BBUtils#findCommonPrefix
> {code}
> BenchmarkMode CntScoreError Units
> PrefixComparator.arrayBBCompare avgt 10 869.897 ± 9.429 ns/op
> PrefixComparator.arrayCompareavgt 10 302.074 ± 13.448 ns/op
> PrefixComparator.bbArrayCompare avgt 10 869.369 ± 5.368 ns/op
> PrefixComparator.bbCompare avgt 10 409.479 ± 1.587 ns/op
> {code}
> the same with ArraysSupport#mismatch() change gives this
> {code}
> BenchmarkMode CntScore Error Units
> PrefixComparator.arrayBBCompare avgt 10 311.946 ± 1.902 ns/op
> PrefixComparator.arrayCompareavgt 10 157.010 ± 4.482 ns/op
> PrefixComparator.bbArrayCompare avgt 10 311.568 ± 1.348 ns/op
> PrefixComparator.bbCompare avgt 10 92.540 ± 0.501 ns/op
> {code}
> How ever note that this comes in flushes/compaction and not during the read
> path.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-23602) TTL Before Which No Data is Purged
[ https://issues.apache.org/jira/browse/HBASE-23602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456083#comment-17456083 ] Andrew Kyle Purtell commented on HBASE-23602: - This issue appears abandoned. Unassigned. Unscheduling. > TTL Before Which No Data is Purged > -- > > Key: HBASE-23602 > URL: https://issues.apache.org/jira/browse/HBASE-23602 > Project: HBase > Issue Type: New Feature >Reporter: Geoffrey Jacoby >Priority: Major > > HBase currently offers operators a choice. They can set > KEEP_DELETED_CELLS=true and VERSIONS to max value, plus no TTL, and they will > always have a complete history of all changes (but high storage costs and > penalties to read performance). Or they can have KEEP_DELETED_CELLS=false and > VERSIONS/TTL set to some reasonable values, but that means that major > compactions can destroy the ability to do a consistent snapshot read of any > prior time. (This limits the usefulness and correctness of, for example, > Phoenix's SCN lookback feature.) > I propose having a new TTL property to give a minimum age that an expired or > deleted Cell would have to achieve before it could be purged. (I see that > HBASE-10118 already does something similar for the delete markers > themselves.) > This would allow operators to have a consistent history for some finite > amount of recent time while still purging out the "long tail" of obsolete / > deleted versions. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HBASE-23639) Moving classes out of hbase-it /test for direct API use of chaos.
[ https://issues.apache.org/jira/browse/HBASE-23639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell reassigned HBASE-23639: --- Assignee: (was: Lokesh Khurana) > Moving classes out of hbase-it /test for direct API use of chaos. > - > > Key: HBASE-23639 > URL: https://issues.apache.org/jira/browse/HBASE-23639 > Project: HBase > Issue Type: Improvement > Components: API, integration tests, test >Reporter: Mihir Monani >Priority: Minor > Attachments: HBASE-23639-master.patch > > > In hbase-it package, Class like RESTApiClusterManager and HBaseClusterManager > which has some part of implementation code for Chaos Action. > > If any user wants to create private implementation which resembles > HBaseClusterManager, then it has to be merged with hbase-it package as > ClusterManager Interface and it's extended implementations are private. > > We should make them public so anyone can have their own implementation and > need to be merged with hbase-it. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-23602) TTL Before Which No Data is Purged
[ https://issues.apache.org/jira/browse/HBASE-23602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23602: Fix Version/s: (was: 2.5.0) (was: 1.8.0) (was: 3.0.0-alpha-2) > TTL Before Which No Data is Purged > -- > > Key: HBASE-23602 > URL: https://issues.apache.org/jira/browse/HBASE-23602 > Project: HBase > Issue Type: New Feature >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > > HBase currently offers operators a choice. They can set > KEEP_DELETED_CELLS=true and VERSIONS to max value, plus no TTL, and they will > always have a complete history of all changes (but high storage costs and > penalties to read performance). Or they can have KEEP_DELETED_CELLS=false and > VERSIONS/TTL set to some reasonable values, but that means that major > compactions can destroy the ability to do a consistent snapshot read of any > prior time. (This limits the usefulness and correctness of, for example, > Phoenix's SCN lookback feature.) > I propose having a new TTL property to give a minimum age that an expired or > deleted Cell would have to achieve before it could be purged. (I see that > HBASE-10118 already does something similar for the delete markers > themselves.) > This would allow operators to have a consistent history for some finite > amount of recent time while still purging out the "long tail" of obsolete / > deleted versions. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-23639) Moving classes out of hbase-it /test for direct API use of chaos.
[ https://issues.apache.org/jira/browse/HBASE-23639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23639: Fix Version/s: (was: 2.5.0) (was: 1.8.0) (was: 3.0.0-alpha-2) > Moving classes out of hbase-it /test for direct API use of chaos. > - > > Key: HBASE-23639 > URL: https://issues.apache.org/jira/browse/HBASE-23639 > Project: HBase > Issue Type: Improvement > Components: API, integration tests, test >Reporter: Mihir Monani >Assignee: Lokesh Khurana >Priority: Minor > Attachments: HBASE-23639-master.patch > > > In hbase-it package, Class like RESTApiClusterManager and HBaseClusterManager > which has some part of implementation code for Chaos Action. > > If any user wants to create private implementation which resembles > HBaseClusterManager, then it has to be merged with hbase-it package as > ClusterManager Interface and it's extended implementations are private. > > We should make them public so anyone can have their own implementation and > need to be merged with hbase-it. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HBASE-23778) Region History Redo
[
https://issues.apache.org/jira/browse/HBASE-23778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell reassigned HBASE-23778:
---
Assignee: (was: Swaroopa Kadam)
> Region History Redo
> ---
>
> Key: HBASE-23778
> URL: https://issues.apache.org/jira/browse/HBASE-23778
> Project: HBase
> Issue Type: Improvement
>Reporter: Swaroopa Kadam
>Priority: Major
>
> My initial thought is mainly to extend the HBase shell(gradually extend to
> CLI and UI) to allow the use of
> {code:java}
> where {code}
> to get the necessary information and allow passing history as an additional
> parameter to get the history. We can configure how many transitions we want
> to store so that anything (ZK or small data structure in the table region
> itself or maybe something else) that is used for state management is not
> exploded.
> As pointed by
> [~andrew.purt...@gmail.com] need to be watchful of mistakes done in the past:
> HBASE-533
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HBASE-23602) TTL Before Which No Data is Purged
[ https://issues.apache.org/jira/browse/HBASE-23602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell reassigned HBASE-23602: --- Assignee: (was: Geoffrey Jacoby) > TTL Before Which No Data is Purged > -- > > Key: HBASE-23602 > URL: https://issues.apache.org/jira/browse/HBASE-23602 > Project: HBase > Issue Type: New Feature >Reporter: Geoffrey Jacoby >Priority: Major > > HBase currently offers operators a choice. They can set > KEEP_DELETED_CELLS=true and VERSIONS to max value, plus no TTL, and they will > always have a complete history of all changes (but high storage costs and > penalties to read performance). Or they can have KEEP_DELETED_CELLS=false and > VERSIONS/TTL set to some reasonable values, but that means that major > compactions can destroy the ability to do a consistent snapshot read of any > prior time. (This limits the usefulness and correctness of, for example, > Phoenix's SCN lookback feature.) > I propose having a new TTL property to give a minimum age that an expired or > deleted Cell would have to achieve before it could be purged. (I see that > HBASE-10118 already does something similar for the delete markers > themselves.) > This would allow operators to have a consistent history for some finite > amount of recent time while still purging out the "long tail" of obsolete / > deleted versions. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-23778) Region History Redo
[
https://issues.apache.org/jira/browse/HBASE-23778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-23778:
Fix Version/s: (was: 2.5.0)
(was: 3.0.0-alpha-2)
> Region History Redo
> ---
>
> Key: HBASE-23778
> URL: https://issues.apache.org/jira/browse/HBASE-23778
> Project: HBase
> Issue Type: Improvement
>Reporter: Swaroopa Kadam
>Assignee: Swaroopa Kadam
>Priority: Major
>
> My initial thought is mainly to extend the HBase shell(gradually extend to
> CLI and UI) to allow the use of
> {code:java}
> where {code}
> to get the necessary information and allow passing history as an additional
> parameter to get the history. We can configure how many transitions we want
> to store so that anything (ZK or small data structure in the table region
> itself or maybe something else) that is used for state management is not
> exploded.
> As pointed by
> [~andrew.purt...@gmail.com] need to be watchful of mistakes done in the past:
> HBASE-533
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-24509) Make PE to have both read and write ability to go in parallel as in YCSB
[ https://issues.apache.org/jira/browse/HBASE-24509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24509: Fix Version/s: (was: 2.5.0) (was: 3.0.0-alpha-2) > Make PE to have both read and write ability to go in parallel as in YCSB > > > Key: HBASE-24509 > URL: https://issues.apache.org/jira/browse/HBASE-24509 > Project: HBase > Issue Type: Improvement > Components: Performance, test >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > > Currently we can either run pure writes or pure reads in case of PE. The case > where few threads can do write and few can do reads is not available. We can > improve the PE to atleast support writes and either Scans or Gets in > parallel. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-24436) The store file open and close thread pool should be shared at the region level
[ https://issues.apache.org/jira/browse/HBASE-24436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24436: Fix Version/s: 2.6.0 (was: 2.5.0) > The store file open and close thread pool should be shared at the region level > -- > > Key: HBASE-24436 > URL: https://issues.apache.org/jira/browse/HBASE-24436 > Project: HBase > Issue Type: Improvement >Reporter: Junhong Xu >Assignee: Junhong Xu >Priority: Minor > Fix For: 3.0.0-alpha-2, 2.6.0 > > > For now, we provide threads per column family evenly in general, but there > are some cases that some column families have much more store files than > others( maybe that's the life, right? ). So in that case, some Stores have > beed done quickly while others are struggling.We should share the thread pool > at the region level in case of data skew. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-24509) Make PE to have both read and write ability to go in parallel as in YCSB
[ https://issues.apache.org/jira/browse/HBASE-24509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24509: Affects Version/s: 2.5.0 3.0.0-alpha-2 2.6.0 > Make PE to have both read and write ability to go in parallel as in YCSB > > > Key: HBASE-24509 > URL: https://issues.apache.org/jira/browse/HBASE-24509 > Project: HBase > Issue Type: Improvement > Components: Performance, test >Affects Versions: 2.5.0, 3.0.0-alpha-2, 2.6.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > > Currently we can either run pure writes or pure reads in case of PE. The case > where few threads can do write and few can do reads is not available. We can > improve the PE to atleast support writes and either Scans or Gets in > parallel. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-24754) Bulk load performance is degraded in HBase 2
[ https://issues.apache.org/jira/browse/HBASE-24754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24754: Fix Version/s: (was: 2.5.0) (was: 3.0.0-alpha-2) > Bulk load performance is degraded in HBase 2 > - > > Key: HBASE-24754 > URL: https://issues.apache.org/jira/browse/HBASE-24754 > Project: HBase > Issue Type: Bug > Components: Performance >Affects Versions: 2.2.3 >Reporter: Ajeet Rai >Assignee: ramkrishna.s.vasudevan >Priority: Major > Attachments: Branc2_withComparator_atKeyValue.patch, > Branch1.3_putSortReducer_sampleCode.patch, > Branch2_putSortReducer_sampleCode.patch, flamegraph_branch-1_new.svg, > flamegraph_branch-2.svg, flamegraph_branch-2_afterpatch.svg > > > in our Test,It is observed that Bulk load performance is degraded in HBase 2 . > Test Input: > 1: Table with 500 region(300 column family) > 2: data =2 TB > Data Sample > 186000120150205100068110,1860001,20150205,5,404,735412,2938,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,1 > 3: Cluster: 7 node(2 master+5 Region Server) > 4: No of Container Launched are same in both case > HBase 2 took 10% more time then HBase 1.3 where test input is same for both > cluster > > |Feature|HBase 2.2.3 > Time(Sec)|HBase 1.3.1 > Time(Sec)|Diff%|Snappy lib: > | > |BulkLoad|21837|19686.16|-10.93|Snappy lib: > HBase 2.2.3: 1.4 > HBase 1.3.1: 1.4| -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-24352) Skip HDFSBlockDistribution calc when FS is not HDFS
[ https://issues.apache.org/jira/browse/HBASE-24352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24352: Affects Version/s: 2.5.0 3.0.0-alpha-2 2.6.0 > Skip HDFSBlockDistribution calc when FS is not HDFS > --- > > Key: HBASE-24352 > URL: https://issues.apache.org/jira/browse/HBASE-24352 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.5.0, 3.0.0-alpha-2, 2.6.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > > HDFSBlockDistribution is used in different places like Balancer, > DateTieredCompaction, CompactionTool etc. In Balancer area there is a config > 'hbase.master.balancer.uselocality' using which we can skip this. But > irrespective of this config, if we are on non HDFS FS, we should skip this. > The block distribution issue many file status commands to underlying FS which > wont be that cheap in a cloud FS. This jira aims at correcting all these > places. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-24352) Skip HDFSBlockDistribution calc when FS is not HDFS
[ https://issues.apache.org/jira/browse/HBASE-24352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24352: Fix Version/s: (was: 2.5.0) (was: 3.0.0-alpha-2) > Skip HDFSBlockDistribution calc when FS is not HDFS > --- > > Key: HBASE-24352 > URL: https://issues.apache.org/jira/browse/HBASE-24352 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > > HDFSBlockDistribution is used in different places like Balancer, > DateTieredCompaction, CompactionTool etc. In Balancer area there is a config > 'hbase.master.balancer.uselocality' using which we can skip this. But > irrespective of this config, if we are on non HDFS FS, we should skip this. > The block distribution issue many file status commands to underlying FS which > wont be that cheap in a cloud FS. This jira aims at correcting all these > places. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-24754) Bulk load performance is degraded in HBase 2
[ https://issues.apache.org/jira/browse/HBASE-24754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24754: Affects Version/s: 2.5.0 3.0.0-alpha-2 2.6.0 > Bulk load performance is degraded in HBase 2 > - > > Key: HBASE-24754 > URL: https://issues.apache.org/jira/browse/HBASE-24754 > Project: HBase > Issue Type: Bug > Components: Performance >Affects Versions: 2.2.3, 2.5.0, 3.0.0-alpha-2, 2.6.0 >Reporter: Ajeet Rai >Assignee: ramkrishna.s.vasudevan >Priority: Major > Attachments: Branc2_withComparator_atKeyValue.patch, > Branch1.3_putSortReducer_sampleCode.patch, > Branch2_putSortReducer_sampleCode.patch, flamegraph_branch-1_new.svg, > flamegraph_branch-2.svg, flamegraph_branch-2_afterpatch.svg > > > in our Test,It is observed that Bulk load performance is degraded in HBase 2 . > Test Input: > 1: Table with 500 region(300 column family) > 2: data =2 TB > Data Sample > 186000120150205100068110,1860001,20150205,5,404,735412,2938,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,1 > 3: Cluster: 7 node(2 master+5 Region Server) > 4: No of Container Launched are same in both case > HBase 2 took 10% more time then HBase 1.3 where test input is same for both > cluster > > |Feature|HBase 2.2.3 > Time(Sec)|HBase 1.3.1 > Time(Sec)|Diff%|Snappy lib: > | > |BulkLoad|21837|19686.16|-10.93|Snappy lib: > HBase 2.2.3: 1.4 > HBase 1.3.1: 1.4| -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-24804) Follow up work to add client side scan metrics for read replica
[ https://issues.apache.org/jira/browse/HBASE-24804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24804: Affects Version/s: 2.5.0 3.0.0-alpha-2 2.6.0 > Follow up work to add client side scan metrics for read replica > --- > > Key: HBASE-24804 > URL: https://issues.apache.org/jira/browse/HBASE-24804 > Project: HBase > Issue Type: New Feature > Components: read replicas >Affects Versions: 2.5.0, 3.0.0-alpha-2, 2.6.0 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Minor > > This is a followup work for HBASE-18436, which adds client metrics for read > replica get. Will add metrics for scan as well. This metrics will be used in > PE and any interested applications. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-24804) Follow up work to add client side scan metrics for read replica
[ https://issues.apache.org/jira/browse/HBASE-24804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24804: Fix Version/s: (was: 2.5.0) (was: 3.0.0-alpha-2) > Follow up work to add client side scan metrics for read replica > --- > > Key: HBASE-24804 > URL: https://issues.apache.org/jira/browse/HBASE-24804 > Project: HBase > Issue Type: New Feature > Components: read replicas >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Minor > > This is a followup work for HBASE-18436, which adds client metrics for read > replica get. Will add metrics for scan as well. This metrics will be used in > PE and any interested applications. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-25019) check if we're impacted by MASSEMBLY-941 and mitigate if needed
[ https://issues.apache.org/jira/browse/HBASE-25019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25019: Affects Version/s: 2.5.0 1.8.0 3.0.0-alpha-2 2.6.0 (was: 3.0.0-alpha-1) (was: 2.3.0) (was: 2.3.1) (was: 1.6.0) (was: 1.7.0) > check if we're impacted by MASSEMBLY-941 and mitigate if needed > --- > > Key: HBASE-25019 > URL: https://issues.apache.org/jira/browse/HBASE-25019 > Project: HBase > Issue Type: Task > Components: scripts >Affects Versions: 2.5.0, 1.8.0, 3.0.0-alpha-2, 2.6.0 >Reporter: Sean Busbey >Priority: Minor > > MASSEMBLY-941 notes a bug starting in version 3.2.0 of the assembly plugin > where scripts lose their executable bit. > We've had this version since updating to the apache parent pom version 22 in > HBASE-23675. We should check our release artifacts to see if we are impacted > and if so downgrade the assembly plugin in our poms to 3.1.1. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-25019) check if we're impacted by MASSEMBLY-941 and mitigate if needed
[ https://issues.apache.org/jira/browse/HBASE-25019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25019: Fix Version/s: (was: 2.5.0) (was: 1.8.0) (was: 3.0.0-alpha-2) > check if we're impacted by MASSEMBLY-941 and mitigate if needed > --- > > Key: HBASE-25019 > URL: https://issues.apache.org/jira/browse/HBASE-25019 > Project: HBase > Issue Type: Task > Components: scripts >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.3.1, 1.6.0, 1.7.0 >Reporter: Sean Busbey >Priority: Minor > > MASSEMBLY-941 notes a bug starting in version 3.2.0 of the assembly plugin > where scripts lose their executable bit. > We've had this version since updating to the apache parent pom version 22 in > HBASE-23675. We should check our release artifacts to see if we are impacted > and if so downgrade the assembly plugin in our poms to 3.1.1. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-25343) Avoid the failed meta replica region temporarily in Load Balance mode
[ https://issues.apache.org/jira/browse/HBASE-25343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25343: Fix Version/s: (was: 2.5.0) (was: 3.0.0-alpha-2) > Avoid the failed meta replica region temporarily in Load Balance mode > - > > Key: HBASE-25343 > URL: https://issues.apache.org/jira/browse/HBASE-25343 > Project: HBase > Issue Type: Sub-task > Components: meta replicas >Affects Versions: 2.4.0 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > > This is a follow-up enhancement with Stack, Duo. With the newly introduced > meta replica LoadBalance mode, if there is something wrong with one of meta > replica regions, the current logic is that it keeps trying until the meta > replica region is onlined again or it reports error, i.e, there is no HA at > LoadBalance mode. HA can be implemented if it reports timeout with one meta > replica region and tries another meta replica region. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-25034) Table regions details on master GUI display slowly.
[ https://issues.apache.org/jira/browse/HBASE-25034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25034: Fix Version/s: (was: 2.5.0) > Table regions details on master GUI display slowly. > --- > > Key: HBASE-25034 > URL: https://issues.apache.org/jira/browse/HBASE-25034 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.3.2 >Reporter: DingSheng Li >Assignee: LiDingSheng >Priority: Major > Labels: newbie > Fix For: 3.0.0-alpha-2 > > Attachments: The table display after pagination.html > > > When a table has a large number of regions (e.g.,a single table contains more > than 100,000 regions), it takes about 20 to 30 minutes to display the table > regions on the master GUI, which is unacceptable to users. After testing, we > find that web page rendering takes up the most time,and this can be solved by > pagination query. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-25034) Table regions details on master GUI display slowly.
[ https://issues.apache.org/jira/browse/HBASE-25034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25034: Status: Patch Available (was: Open) > Table regions details on master GUI display slowly. > --- > > Key: HBASE-25034 > URL: https://issues.apache.org/jira/browse/HBASE-25034 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.3.2 >Reporter: DingSheng Li >Assignee: LiDingSheng >Priority: Major > Labels: newbie > Fix For: 3.0.0-alpha-2 > > Attachments: The table display after pagination.html > > > When a table has a large number of regions (e.g.,a single table contains more > than 100,000 regions), it takes about 20 to 30 minutes to display the table > regions on the master GUI, which is unacceptable to users. After testing, we > find that web page rendering takes up the most time,and this can be solved by > pagination query. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-25433) There is no limit on the table name length when creating a table
[ https://issues.apache.org/jira/browse/HBASE-25433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25433: Fix Version/s: (was: 2.5.0) > There is no limit on the table name length when creating a table > > > Key: HBASE-25433 > URL: https://issues.apache.org/jira/browse/HBASE-25433 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.2.3 >Reporter: ZouWeiMing >Priority: Major > Fix For: 3.0.0-alpha-2 > > Attachments: attachment_tableNameLengthLimit.html > > > It occurs > Exeception(ERROR：java.util.concurrent.ExecutionExceptionjava.io.IOExpcetionException > in createDir) because of 8000 bytes length limits on HDFS directory name if > the length of HBase table name misses validation when creating. > So,we decide to verify the length of table name when creating a > table.IllegalArgumentExcetion will be thrown when length is larger than 8000 > bytes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-25343) Avoid the failed meta replica region temporarily in Load Balance mode
[ https://issues.apache.org/jira/browse/HBASE-25343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25343: Affects Version/s: 2.5.0 3.0.0-alpha-2 2.6.0 (was: 2.4.0) > Avoid the failed meta replica region temporarily in Load Balance mode > - > > Key: HBASE-25343 > URL: https://issues.apache.org/jira/browse/HBASE-25343 > Project: HBase > Issue Type: Sub-task > Components: meta replicas >Affects Versions: 2.5.0, 3.0.0-alpha-2, 2.6.0 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > > This is a follow-up enhancement with Stack, Duo. With the newly introduced > meta replica LoadBalance mode, if there is something wrong with one of meta > replica regions, the current logic is that it keeps trying until the meta > replica region is onlined again or it reports error, i.e, there is no HA at > LoadBalance mode. HA can be implemented if it reports timeout with one meta > replica region and tries another meta replica region. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-25433) There is no limit on the table name length when creating a table
[ https://issues.apache.org/jira/browse/HBASE-25433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25433: Status: Patch Available (was: Open) > There is no limit on the table name length when creating a table > > > Key: HBASE-25433 > URL: https://issues.apache.org/jira/browse/HBASE-25433 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.2.3 >Reporter: ZouWeiMing >Priority: Major > Fix For: 3.0.0-alpha-2 > > Attachments: attachment_tableNameLengthLimit.html > > > It occurs > Exeception(ERROR：java.util.concurrent.ExecutionExceptionjava.io.IOExpcetionException > in createDir) because of 8000 bytes length limits on HDFS directory name if > the length of HBase table name misses validation when creating. > So,we decide to verify the length of table name when creating a > table.IllegalArgumentExcetion will be thrown when length is larger than 8000 > bytes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-25547) Thread pools should release unused resources
[ https://issues.apache.org/jira/browse/HBASE-25547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25547: Resolution: Fixed Status: Resolved (was: Patch Available) The PRs were merged. > Thread pools should release unused resources > > > Key: HBASE-25547 > URL: https://issues.apache.org/jira/browse/HBASE-25547 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.5.0 >Reporter: Bharath Vissapragada >Assignee: Bharath Vissapragada >Priority: Minor > Fix For: 2.5.0, 3.0.0-alpha-2 > > > On this [PR|https://github.com/apache/hbase/pull/2909] [~stack] rightly > pointed out that the thread pool just spins up threads that remain idle for > 99.999% of time. Java's > [ThreadPoolExecutor|https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/ThreadPoolExecutor.html] > lets us tune some configurations like the following for releasing the unused > resources > * corePoolSize > * keepAliveTimeMs > * allowCoreThreadTimeOut > Current code makes it difficult to pass along these configurations to the > executor service. Refactor and fix the defaults for low-priority thread pools > to release resources quicker. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-25510) Optimize TableName.valueOf from O(n) to O(1). We can get benefits when the number of tables in the cluster is greater than dozens
[
https://issues.apache.org/jira/browse/HBASE-25510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25510:
Fix Version/s: 2.6.0
(was: 2.5.0)
> Optimize TableName.valueOf from O(n) to O(1). We can get benefits when the
> number of tables in the cluster is greater than dozens
> --
>
> Key: HBASE-25510
> URL: https://issues.apache.org/jira/browse/HBASE-25510
> Project: HBase
> Issue Type: Improvement
> Components: master, Replication
>Affects Versions: 1.2.12, 1.4.13, 2.4.1
>Reporter: zhuobin zheng
>Assignee: zhuobin zheng
>Priority: Major
> Fix For: 1.8.0, 3.0.0-alpha-2, 2.6.0
>
> Attachments: TestTableNameJMH.java, optimiz_benchmark,
> optimiz_benchmark_fix, origin_benchmark, stucks-profile-info
>
>
> Now, TableName.valueOf will try to find TableName Object in cache
> linearly(code show as below). So it is too slow when we has thousands of
> tables on cluster.
> {code:java}
> // code placeholder
> for (TableName tn : tableCache) {
> if (Bytes.equals(tn.getQualifier(), qns) && Bytes.equals(tn.getNamespace(),
> bns)) {
> return tn;
> }
> }{code}
> I try to store the object in the hash table, so it can look up more quickly.
> code like this
> {code:java}
> // code placeholder
> TableName oldTable = tableCache.get(nameAsStr);{code}
>
> In our cluster which has tens thousands of tables. (Most of that is KYLIN
> table).
> We found that in the following two cases, the TableName.valueOf method will
> severely restrict our performance.
>
> Common premise: tens of thousands table in cluster
> cause: TableName.valueOf with low performance. (because we need to traverse
> all caches linearly)
>
> Case1. Replication
> premise1: one of table write with high qps, small value, Non-batch request.
> cause too much wal entry
> premise2: deserialize WAL Entry includes calling the TableName.valueOf method.
> Cause: Replicat Stuck. A lot of WAL files pile up.
>
> Case2. Active Master Start up
> NamespaceStateManager init should init all RegionInfo, and regioninfo init
> will call TableName.valueOf. It will cost some time if TableName.valueOf is
> slow.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-25510) Optimize TableName.valueOf from O(n) to O(1). We can get benefits when the number of tables in the cluster is greater than dozens
[
https://issues.apache.org/jira/browse/HBASE-25510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25510:
Status: Patch Available (was: Open)
> Optimize TableName.valueOf from O(n) to O(1). We can get benefits when the
> number of tables in the cluster is greater than dozens
> --
>
> Key: HBASE-25510
> URL: https://issues.apache.org/jira/browse/HBASE-25510
> Project: HBase
> Issue Type: Improvement
> Components: master, Replication
>Affects Versions: 2.4.1, 1.4.13, 1.2.12
>Reporter: zhuobin zheng
>Assignee: zhuobin zheng
>Priority: Major
> Fix For: 1.8.0, 3.0.0-alpha-2, 2.6.0
>
> Attachments: TestTableNameJMH.java, optimiz_benchmark,
> optimiz_benchmark_fix, origin_benchmark, stucks-profile-info
>
>
> Now, TableName.valueOf will try to find TableName Object in cache
> linearly(code show as below). So it is too slow when we has thousands of
> tables on cluster.
> {code:java}
> // code placeholder
> for (TableName tn : tableCache) {
> if (Bytes.equals(tn.getQualifier(), qns) && Bytes.equals(tn.getNamespace(),
> bns)) {
> return tn;
> }
> }{code}
> I try to store the object in the hash table, so it can look up more quickly.
> code like this
> {code:java}
> // code placeholder
> TableName oldTable = tableCache.get(nameAsStr);{code}
>
> In our cluster which has tens thousands of tables. (Most of that is KYLIN
> table).
> We found that in the following two cases, the TableName.valueOf method will
> severely restrict our performance.
>
> Common premise: tens of thousands table in cluster
> cause: TableName.valueOf with low performance. (because we need to traverse
> all caches linearly)
>
> Case1. Replication
> premise1: one of table write with high qps, small value, Non-batch request.
> cause too much wal entry
> premise2: deserialize WAL Entry includes calling the TableName.valueOf method.
> Cause: Replicat Stuck. A lot of WAL files pile up.
>
> Case2. Active Master Start up
> NamespaceStateManager init should init all RegionInfo, and regioninfo init
> will call TableName.valueOf. It will cost some time if TableName.valueOf is
> slow.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-25613) [Branch-2 and Master]Handle the NoNode exception in remove log replication in a better way then just log WARN
[ https://issues.apache.org/jira/browse/HBASE-25613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25613: Affects Version/s: 2.5.0 3.0.0-alpha-2 2.6.0 > [Branch-2 and Master]Handle the NoNode exception in remove log replication in > a better way then just log WARN > - > > Key: HBASE-25613 > URL: https://issues.apache.org/jira/browse/HBASE-25613 > Project: HBase > Issue Type: Bug >Affects Versions: 2.5.0, 3.0.0-alpha-2, 2.6.0 >Reporter: Sandeep Pal >Assignee: Sandeep Pal >Priority: Major > > Currently, when we see the NoNodeException in the replication source while > removing a log from ZK, we swallow that exception and log WARN. > > In certain cases, we might have the peer removed and corresponding logs > removing as well but the replication source continuous to run because of an > RPC failure or anything. > In stead of just log WARN we should check if the peer is removed, if it is > the case, we should terminate the source or try to execute the removePeer > workflow again. > > This would prevent the orphaned source execution infinitely. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-25799) add clusterReadRequests and clusterWriteRequests jmx
[ https://issues.apache.org/jira/browse/HBASE-25799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25799: Fix Version/s: (was: 2.5.0) > add clusterReadRequests and clusterWriteRequests jmx > - > > Key: HBASE-25799 > URL: https://issues.apache.org/jira/browse/HBASE-25799 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: xijiawen >Assignee: xijiawen >Priority: Major > Fix For: 3.0.0-alpha-2 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-25694) Improve the shell's 'status replication' command output
[ https://issues.apache.org/jira/browse/HBASE-25694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25694: Fix Version/s: 2.6.0 (was: 2.5.0) > Improve the shell's 'status replication' command output > --- > > Key: HBASE-25694 > URL: https://issues.apache.org/jira/browse/HBASE-25694 > Project: HBase > Issue Type: Task > Components: Operability, Replication, shell >Affects Versions: 2.4.2 >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-2, 2.6.0 > > > The output of the shell's 'status "replication"' command could use some > cleaning up. > The basic format is: > version VERSION > N live servers > SERVER_1: >SOURCE: PeerID=PEERID, AgeOfLastShippedOp=160347, SizeOfLogQueue=49, > TimeStampsOfLastShippedOp=Thu Mar 25 01:55:21 UTC 2021, Replication Lag=161013 >SINK : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Thu Mar 25 > 01:49:53 UTC 2021 > ... > Issues: > - Per line format is KEY=VALUE, ... . Most KEYs are CamelCase but some have > spaces. Should all be camel case for consistency. > - Age is printed as long and timestamp is string (local TZ) format. I think > the string formatting of the timestamp makes it difficult to read. Lets just > print timestamps as longs. Or provide a non-default option for string > formatted timestamps. > - There is no sense of effective rate or bandwidth. Each source and sink line > emitted should provide some sense of current workload: rpcs/sec, bytes/sec, > cells/sec... something. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-25799) add clusterReadRequests and clusterWriteRequests jmx
[ https://issues.apache.org/jira/browse/HBASE-25799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-25799. - Resolution: Fixed > add clusterReadRequests and clusterWriteRequests jmx > - > > Key: HBASE-25799 > URL: https://issues.apache.org/jira/browse/HBASE-25799 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: xijiawen >Assignee: xijiawen >Priority: Major > Fix For: 3.0.0-alpha-2 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-25613) [Branch-2 and Master]Handle the NoNode exception in remove log replication in a better way then just log WARN
[ https://issues.apache.org/jira/browse/HBASE-25613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25613: Fix Version/s: (was: 2.5.0) (was: 3.0.0-alpha-2) > [Branch-2 and Master]Handle the NoNode exception in remove log replication in > a better way then just log WARN > - > > Key: HBASE-25613 > URL: https://issues.apache.org/jira/browse/HBASE-25613 > Project: HBase > Issue Type: Bug >Reporter: Sandeep Pal >Assignee: Sandeep Pal >Priority: Major > > Currently, when we see the NoNodeException in the replication source while > removing a log from ZK, we swallow that exception and log WARN. > > In certain cases, we might have the peer removed and corresponding logs > removing as well but the replication source continuous to run because of an > RPC failure or anything. > In stead of just log WARN we should check if the peer is removed, if it is > the case, we should terminate the source or try to execute the removePeer > workflow again. > > This would prevent the orphaned source execution infinitely. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-25624) Bound LoadBalancer's RegionLocationFinder cache
[ https://issues.apache.org/jira/browse/HBASE-25624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25624: Fix Version/s: 2.6.0 (was: 2.5.0) > Bound LoadBalancer's RegionLocationFinder cache > --- > > Key: HBASE-25624 > URL: https://issues.apache.org/jira/browse/HBASE-25624 > Project: HBase > Issue Type: Bug > Components: Balancer, master, Operability >Affects Versions: 1.6.0, 2.4.1 >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 1.8.0, 3.0.0-alpha-2, 2.6.0 > > > We have a large table in production that causes the balancer's > RegionLocationFinder cache to consume 4 GB of heap, which, among other > factors, triggered OOMEs, and made us aware of this problem. > RegionLocationFinder embeds a cache backed by Guava's CacheLoader. The > RegionLocationFinder cache comes to consume heap for RegionInfos for all > table regions and all HDFS block locations of all store files for all regions > of all tables. > The only limit we pass to the CacheBuilder is an expiration time of 1440 > milliseconds for individual cache entries. That's 4 hours. That's much too > long; however, the cache also periodically refreshes itself, where the need > for a refresh is checked whenever BaseLoadBalancer calls > RegionLocationFinder's setClusterMetrics() method, which defeats the > expiration based limit anyway. > We should be bounding this cache with effective resource controls. Time based > expiry is fine but the periodic refresh logic must be removed to make it > effective. Implement size based limits too. CacheBuilder#maximumSize will > limit by number cache entries. This might be fine but > CacheBuilder#maximumWeight would be better, where weight is something > determined by the API user. In this case it can be an estimate of the heap > size of the hash map entries kept in the cache. > Default should remain unbounded. Specific bounds should be supported by new > site configuration options. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HBASE-25624) Bound LoadBalancer's RegionLocationFinder cache
[ https://issues.apache.org/jira/browse/HBASE-25624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell reassigned HBASE-25624: --- Assignee: (was: Prathyusha) > Bound LoadBalancer's RegionLocationFinder cache > --- > > Key: HBASE-25624 > URL: https://issues.apache.org/jira/browse/HBASE-25624 > Project: HBase > Issue Type: Bug > Components: Balancer, master, Operability >Affects Versions: 1.6.0, 2.4.1 >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 2.5.0, 1.8.0, 3.0.0-alpha-2 > > > We have a large table in production that causes the balancer's > RegionLocationFinder cache to consume 4 GB of heap, which, among other > factors, triggered OOMEs, and made us aware of this problem. > RegionLocationFinder embeds a cache backed by Guava's CacheLoader. The > RegionLocationFinder cache comes to consume heap for RegionInfos for all > table regions and all HDFS block locations of all store files for all regions > of all tables. > The only limit we pass to the CacheBuilder is an expiration time of 1440 > milliseconds for individual cache entries. That's 4 hours. That's much too > long; however, the cache also periodically refreshes itself, where the need > for a refresh is checked whenever BaseLoadBalancer calls > RegionLocationFinder's setClusterMetrics() method, which defeats the > expiration based limit anyway. > We should be bounding this cache with effective resource controls. Time based > expiry is fine but the periodic refresh logic must be removed to make it > effective. Implement size based limits too. CacheBuilder#maximumSize will > limit by number cache entries. This might be fine but > CacheBuilder#maximumWeight would be better, where weight is something > determined by the API user. In this case it can be an estimate of the heap > size of the hash map entries kept in the cache. > Default should remain unbounded. Specific bounds should be supported by new > site configuration options. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HBASE-25694) Improve the shell's 'status replication' command output
[ https://issues.apache.org/jira/browse/HBASE-25694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell reassigned HBASE-25694: --- Assignee: (was: Divyesh Chandra) > Improve the shell's 'status replication' command output > --- > > Key: HBASE-25694 > URL: https://issues.apache.org/jira/browse/HBASE-25694 > Project: HBase > Issue Type: Task > Components: Operability, Replication, shell >Affects Versions: 2.4.2 >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2 > > > The output of the shell's 'status "replication"' command could use some > cleaning up. > The basic format is: > version VERSION > N live servers > SERVER_1: >SOURCE: PeerID=PEERID, AgeOfLastShippedOp=160347, SizeOfLogQueue=49, > TimeStampsOfLastShippedOp=Thu Mar 25 01:55:21 UTC 2021, Replication Lag=161013 >SINK : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Thu Mar 25 > 01:49:53 UTC 2021 > ... > Issues: > - Per line format is KEY=VALUE, ... . Most KEYs are CamelCase but some have > spaces. Should all be camel case for consistency. > - Age is printed as long and timestamp is string (local TZ) format. I think > the string formatting of the timestamp makes it difficult to read. Lets just > print timestamps as longs. Or provide a non-default option for string > formatted timestamps. > - There is no sense of effective rate or bandwidth. Each source and sink line > emitted should provide some sense of current workload: rpcs/sec, bytes/sec, > cells/sec... something. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-25823) TestSlowLogAccessor.testHigherSlowLogs repeatable failure
[
https://issues.apache.org/jira/browse/HBASE-25823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25823:
Fix Version/s: 2.6.0
> TestSlowLogAccessor.testHigherSlowLogs repeatable failure
> -
>
> Key: HBASE-25823
> URL: https://issues.apache.org/jira/browse/HBASE-25823
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.9, 2.6.0
>
>
> {noformat}
> [ERROR] TestSlowLogAccessor.testHigherSlowLogs:211 Waiting timed out after
> [7,000] msec{noformat}
> Repeatable failure.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-25844) Fix Jersey for hbase-server processes
[
https://issues.apache.org/jira/browse/HBASE-25844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25844:
Fix Version/s: 2.6.0
> Fix Jersey for hbase-server processes
> -
>
> Key: HBASE-25844
> URL: https://issues.apache.org/jira/browse/HBASE-25844
> Project: HBase
> Issue Type: Task
> Components: master, regionserver, thirdparty
>Affects Versions: 3.0.0-alpha-1, 2.5.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.6.0
>
>
> I spent some time trying to use Jersey from within the Master and it's not
> working. To summarize, we have unshaded resources from both
> jersey-server-1.19 and jersey-server-2.32 on the hbase-server classpath.
> Jersey's initialization uses ServiceLoader to look up concrete implementation
> classes of {{javax.ws.rs}} classes at runtime. Because we do not shade
> {{javax.ws.rs}} in hbase-thirdparty-jersey, an attempt to use shaded
> jersey-2.x still results in loading unshaded jersey-1.x jars, leading to an
> error like this
> {noformat}
> java.lang.AbstractMethodError:
> javax.ws.rs.core.UriBuilder.uri(Ljava/lang/String;)Ljavax/ws/rs/core/UriBuilder;
> at javax.ws.rs.core.UriBuilder.fromUri(UriBuilder.java:96)
> at
> org.apache.hbase.thirdparty.org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:275)
> at
> org.apache.hbase.thirdparty.org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
> at
> org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
> at
> org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
> at
> org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:112)
> {noformat}
> We cannot override what version of these classes are loaded at runtime via
> Java property because Jersey's load order implementation checks system
> properties as a last resort, not first thing as is claimed by javadoc.
> So I can think of two solutions.
> # One is to shade {{javax.ws.rs}} in hbase-thirdparty-jersey. This would
> shade both the interfaces and the resource files that are referenced at
> runtime, allowing for an entirely isolated jersey container instantiate.
> # Another idea is to add a custom {{ClassLoader}} that is inserted before
> jersey is initialized. This would filter out resources that are "banned",
> allowing our desired implementation through.
> Between these, I think (1) is better, but I don't know what else might break.
> I've made an effort of both, but with neither approach can I get a jersey
> environment to response from my resource class... either because the solution
> is incomplete, or because I don't have the jersey environment configured
> properly.
> See also some discussion that happened over on Slack,
> https://apache-hbase.slack.com/archives/C13K8NVAM/p1618857521051700
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-25865) Visualize current state of region assignment
[
https://issues.apache.org/jira/browse/HBASE-25865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25865:
Fix Version/s: 2.6.0
> Visualize current state of region assignment
>
>
> Key: HBASE-25865
> URL: https://issues.apache.org/jira/browse/HBASE-25865
> Project: HBase
> Issue Type: New Feature
> Components: master, Operability, Usability
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.6.0
>
> Attachments: image1.png, image2.png
>
>
> After several months of debugging and tuning the balancer and normalizer on a
> large production cluster, we found that working from visualizations of the
> current region state was very useful for understanding behaviors and
> quantifying improvements we made along the way. Specifically, we found that a
> chart of total assigned region count and total assigned region store files
> size per table per host was immensely useful for tuning the balancer.
> Histograms of store file size made understanding normalizer activity much
> more intuitive.
> Our scripts would parse the output of the shell's {{status 'detailed'}}
> command, extract the desired metric, and produce charts. I'd like to build
> into the master UI the equivalent functionality, with data coming directly
> from the {{ClusterMetrics object}}, and data rendered into an interactive
> chart rendered in the browser.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-25828) CompactionProgress WARNS: "totalCompactingKVs=N less than currentCompactedKVs=M"
[
https://issues.apache.org/jira/browse/HBASE-25828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25828:
Fix Version/s: 2.6.0
> CompactionProgress WARNS: "totalCompactingKVs=N less than
> currentCompactedKVs=M"
>
>
> Key: HBASE-25828
> URL: https://issues.apache.org/jira/browse/HBASE-25828
> Project: HBase
> Issue Type: Bug
> Components: Compaction, Operability, regionserver
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.6.0
>
>
> Similar to HBASE-25642, but compaction progress warnings.
> Lots of warnings like:
> {noformat}
> 2021-04-30 00:55:15,436 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=15589 less than
> currentCompactedKVs=21411
> {noformat}
> {noformat}
> 2021-04-30 00:55:15,437 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=21916 less than
> currentCompactedKVs=33328
> {noformat}
> {noformat}
> 2021-04-30 00:55:15,438 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=89731 less than
> currentCompactedKVs=92808
> {noformat}
> A couple of issues:
> - Is there a way to make CompactionProgress more reliable? I seem to recall
> this is the second or third time around the block on this.
> - This is annoying because compaction progress isn't always accurate, but
> this information is not used to determine anything significant, so WARN level
> is a bit much. (Or, if this is a real correctness problem, see point above.)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-25865) Visualize current state of region assignment
[
https://issues.apache.org/jira/browse/HBASE-25865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456079#comment-17456079
]
Andrew Kyle Purtell commented on HBASE-25865:
-
Has this issue been abandoned?
> Visualize current state of region assignment
>
>
> Key: HBASE-25865
> URL: https://issues.apache.org/jira/browse/HBASE-25865
> Project: HBase
> Issue Type: New Feature
> Components: master, Operability, Usability
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2
>
> Attachments: image1.png, image2.png
>
>
> After several months of debugging and tuning the balancer and normalizer on a
> large production cluster, we found that working from visualizations of the
> current region state was very useful for understanding behaviors and
> quantifying improvements we made along the way. Specifically, we found that a
> chart of total assigned region count and total assigned region store files
> size per table per host was immensely useful for tuning the balancer.
> Histograms of store file size made understanding normalizer activity much
> more intuitive.
> Our scripts would parse the output of the shell's {{status 'detailed'}}
> command, extract the desired metric, and produce charts. I'd like to build
> into the master UI the equivalent functionality, with data coming directly
> from the {{ClusterMetrics object}}, and data rendered into an interactive
> chart rendered in the browser.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-25955) Setting NAMESPACES when adding a replication peer doesn't have any effect
[ https://issues.apache.org/jira/browse/HBASE-25955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25955: Affects Version/s: 2.6.0 > Setting NAMESPACES when adding a replication peer doesn't have any effect > -- > > Key: HBASE-25955 > URL: https://issues.apache.org/jira/browse/HBASE-25955 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 3.0.0-alpha-1, 2.5.0, 2.3.5, 2.4.3, 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > > A common ask from our customers is for an easy way to enable replication for > all tables within a namespace, but we had noticed that setting *NAMESPACES* > option when adding a peer has no effect on enabling replication for the > defined namespaces tables, it still replicates only entries where the *CF > REPLICATION_SCOPE* is set to '1'. > The problem is due to the chaining of *ScopeWALEntryFilter* and > *NamespaceTableCfWALEntryFilter* together. This is currently done on > *BaseReplicationEndpoint* class, but these two filters are not compatible, as > *ScopeWALEntryFilter.filterCells* logic filters out cells from CFs where > REPLICATION_SCOPE property is set to 0, before > *NamespaceTableCfWALEntryFilter.filterCells* can have a chance to apply it's > own logic, based on namespace definition. > This PR changes the logic described above, making *ScopeWALEntryFilter* and > *NamespaceTableCfWALEntryFilter* mutual exclusive, allowing for entries from > all tables within the defined namespaces to be replicated without needing to > have *REPLICATION_SCOPE* set to true. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-25915) ‘mobdir’ overlaps in HFileLink.mobPath
[ https://issues.apache.org/jira/browse/HBASE-25915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25915: Fix Version/s: 2.6.0 Status: Patch Available (was: Open) > ‘mobdir’ overlaps in HFileLink.mobPath > -- > > Key: HBASE-25915 > URL: https://issues.apache.org/jira/browse/HBASE-25915 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 2.4.3, 1.4.13, 3.0.0-alpha-1 >Reporter: wenfeiyi666 >Assignee: wenfeiyi666 >Priority: Minor > Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.9, 2.6.0 > > > ‘mobdir’ overlaps in HFileLink.mobPath, like > '/root/mobdir/mobdir/data/table/...' -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26027) The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException
[
https://issues.apache.org/jira/browse/HBASE-26027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456077#comment-17456077
]
Andrew Kyle Purtell commented on HBASE-26027:
-
Has this issue been abandoned?
> The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone
> caused by ArrayStoreException
> -
>
> Key: HBASE-26027
> URL: https://issues.apache.org/jira/browse/HBASE-26027
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.2.7, 2.3.5, 2.4.4
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
>
> The batch api of HTable contains a param named results to store result or
> exception, its type is Object[].
> If user pass an array with other type, eg:
> org.apache.hadoop.hbase.client.Result, and if we need to put an exception
> into it by some reason, then the ArrayStoreException will occur in
> AsyncRequestFutureImpl.updateResult, then the
> AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
> AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
> actionsInProgress again and again, forever.
> It is better to add an cutoff calculated by operationTimeout, instead of only
> depend on the value of actionsInProgress.
> BTW, this issue only for 2.x, since 3.x the implement has refactored.
> How to reproduce:
> 1: add sleep in RSRpcServices.multi to mock slow response
> {code:java}
> try {
> Thread.sleep(2000);
> } catch (InterruptedException e) {
> e.printStackTrace();
> }
> {code}
> 2: set time out in config
> {code:java}
> conf.set("hbase.rpc.timeout","2000");
> conf.set("hbase.client.operation.timeout","6000");
> {code}
> 3: call batch api
> {code:java}
> Table table = HbaseUtil.getTable("test");
> byte[] cf = Bytes.toBytes("f");
> byte[] c = Bytes.toBytes("c1");
> List gets = new ArrayList<>();
> for (int i = 0; i < 10; i++) {
> byte[] rk = Bytes.toBytes("rk-" + i);
> Get get = new Get(rk);
> get.addColumn(cf, c);
> gets.add(get);
> }
> Result[] results = new Result[gets.size()];
> table.batch(gets, results);
> {code}
> The log will looks like below:
> {code:java}
> [ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 -
> id=1 error for test processing localhost,16020,1624343786295
> java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10 actions to
> finish on table:
> [INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10 actions to
> finish on table: test
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26027) The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException
[
https://issues.apache.org/jira/browse/HBASE-26027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26027:
Affects Version/s: 2.5.0
> The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone
> caused by ArrayStoreException
> -
>
> Key: HBASE-26027
> URL: https://issues.apache.org/jira/browse/HBASE-26027
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.2.7, 2.5.0, 2.3.5, 2.4.4
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
>
> The batch api of HTable contains a param named results to store result or
> exception, its type is Object[].
> If user pass an array with other type, eg:
> org.apache.hadoop.hbase.client.Result, and if we need to put an exception
> into it by some reason, then the ArrayStoreException will occur in
> AsyncRequestFutureImpl.updateResult, then the
> AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
> AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
> actionsInProgress again and again, forever.
> It is better to add an cutoff calculated by operationTimeout, instead of only
> depend on the value of actionsInProgress.
> BTW, this issue only for 2.x, since 3.x the implement has refactored.
> How to reproduce:
> 1: add sleep in RSRpcServices.multi to mock slow response
> {code:java}
> try {
> Thread.sleep(2000);
> } catch (InterruptedException e) {
> e.printStackTrace();
> }
> {code}
> 2: set time out in config
> {code:java}
> conf.set("hbase.rpc.timeout","2000");
> conf.set("hbase.client.operation.timeout","6000");
> {code}
> 3: call batch api
> {code:java}
> Table table = HbaseUtil.getTable("test");
> byte[] cf = Bytes.toBytes("f");
> byte[] c = Bytes.toBytes("c1");
> List gets = new ArrayList<>();
> for (int i = 0; i < 10; i++) {
> byte[] rk = Bytes.toBytes("rk-" + i);
> Get get = new Get(rk);
> get.addColumn(cf, c);
> gets.add(get);
> }
> Result[] results = new Result[gets.size()];
> table.batch(gets, results);
> {code}
> The log will looks like below:
> {code:java}
> [ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 -
> id=1 error for test processing localhost,16020,1624343786295
> java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10 actions to
> finish on table:
> [INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10 actions to
> finish on table: test
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26027) The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException
[
https://issues.apache.org/jira/browse/HBASE-26027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26027:
Fix Version/s: (was: 2.5.0)
(was: 2.3.8)
(was: 2.4.9)
> The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone
> caused by ArrayStoreException
> -
>
> Key: HBASE-26027
> URL: https://issues.apache.org/jira/browse/HBASE-26027
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.2.7, 2.3.5, 2.4.4
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
>
> The batch api of HTable contains a param named results to store result or
> exception, its type is Object[].
> If user pass an array with other type, eg:
> org.apache.hadoop.hbase.client.Result, and if we need to put an exception
> into it by some reason, then the ArrayStoreException will occur in
> AsyncRequestFutureImpl.updateResult, then the
> AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
> AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
> actionsInProgress again and again, forever.
> It is better to add an cutoff calculated by operationTimeout, instead of only
> depend on the value of actionsInProgress.
> BTW, this issue only for 2.x, since 3.x the implement has refactored.
> How to reproduce:
> 1: add sleep in RSRpcServices.multi to mock slow response
> {code:java}
> try {
> Thread.sleep(2000);
> } catch (InterruptedException e) {
> e.printStackTrace();
> }
> {code}
> 2: set time out in config
> {code:java}
> conf.set("hbase.rpc.timeout","2000");
> conf.set("hbase.client.operation.timeout","6000");
> {code}
> 3: call batch api
> {code:java}
> Table table = HbaseUtil.getTable("test");
> byte[] cf = Bytes.toBytes("f");
> byte[] c = Bytes.toBytes("c1");
> List gets = new ArrayList<>();
> for (int i = 0; i < 10; i++) {
> byte[] rk = Bytes.toBytes("rk-" + i);
> Get get = new Get(rk);
> get.addColumn(cf, c);
> gets.add(get);
> }
> Result[] results = new Result[gets.size()];
> table.batch(gets, results);
> {code}
> The log will looks like below:
> {code:java}
> [ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 -
> id=1 error for test processing localhost,16020,1624343786295
> java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10 actions to
> finish on table:
> [INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10 actions to
> finish on table: test
> [INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10 actions to
> finish on table: test
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26122) Limit max result size of individual Gets
[ https://issues.apache.org/jira/browse/HBASE-26122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456076#comment-17456076 ] Andrew Kyle Purtell commented on HBASE-26122: - What is the status of this change? Has it been reverted everywhere? > Limit max result size of individual Gets > > > Key: HBASE-26122 > URL: https://issues.apache.org/jira/browse/HBASE-26122 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2 > > > Scans have the ability to have a configured max result size, which causes > them to return a partial result once the limit has been reached. MultiGets > also can throw MultiActionResultTooLarge if the response size is over a > configured quota. Neither of these really accounts for a single Get of a > too-large row. Such too-large Gets can cause substantial GC pressure or worse > if sent at volume. > Currently one can work around this by converting their Get to a single row > Scan, but this requires a developer to proactively know about and prepare for > the issue by using a Scan upfront or wait for the RegionServer to choke on a > large request and only then rewrite the Get for future requests. > We should implement the same response size limits for for Get as for Scan, > whereby the server returns a partial result to the client for handling. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26122) Limit max result size of individual Gets
[ https://issues.apache.org/jira/browse/HBASE-26122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-26122: Fix Version/s: 2.6.0 (was: 2.5.0) > Limit max result size of individual Gets > > > Key: HBASE-26122 > URL: https://issues.apache.org/jira/browse/HBASE-26122 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > Fix For: 3.0.0-alpha-2, 2.6.0 > > > Scans have the ability to have a configured max result size, which causes > them to return a partial result once the limit has been reached. MultiGets > also can throw MultiActionResultTooLarge if the response size is over a > configured quota. Neither of these really accounts for a single Get of a > too-large row. Such too-large Gets can cause substantial GC pressure or worse > if sent at volume. > Currently one can work around this by converting their Get to a single row > Scan, but this requires a developer to proactively know about and prepare for > the issue by using a Scan upfront or wait for the RegionServer to choke on a > large request and only then rewrite the Get for future requests. > We should implement the same response size limits for for Get as for Scan, > whereby the server returns a partial result to the client for handling. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26192) hbck.jsp should provide a JSON formatted output option
[
https://issues.apache.org/jira/browse/HBASE-26192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456075#comment-17456075
]
Andrew Kyle Purtell commented on HBASE-26192:
-
Let me take this for 2.5.0+
> hbck.jsp should provide a JSON formatted output option
> --
>
> Key: HBASE-26192
> URL: https://issues.apache.org/jira/browse/HBASE-26192
> Project: HBase
> Issue Type: New Feature
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.6.0
>
>
> It used to be possible to get hbck's verdict of cluster status from the
> command line, especially useful for headless deployments, i.e. without
> requiring a browser with sufficient connectivity to load a UI, or scrape
> information out of raw HTML, or write regex to comb over log4j output. The
> hbck tool's output wasn't particularly convenient to parse but it was
> straightforward to extract the desired information with a handful of regular
> expressions.
> HBCK2 has a different design philosophy than the old hbck, which is to serve
> as a collection of small and discrete recovery and repair functions, rather
> than attempt to be a universal repair tool. This makes a lot of sense and
> isn't the issue at hand. Unfortunately the old hbck's utility for reporting
> the current cluster health assessment has not been replaced either in whole
> or in part. Instead:
> {quote}
> HBCK2 is for fixes. For listings of inconsistencies or blockages in the
> running cluster, you go elsewhere, to the logs and UI of the running cluster
> Master. Once an issue has been identified, you use the HBCK2 tool to ask the
> Master to effect fixes or to skip-over bad state. Asking the Master to make
> the fixes rather than try and effect the repair locally in a fix-it tool's
> context is another important difference between HBCK2 and hbck1.
> {quote}
> Developing custom tooling to mine logs and scrape UI simply to gain a top
> level assessment of system health is unsatisfying. There should be a
> convenient means for querying the system if issues that rise to the level of
> _inconsistency_, in the hbck parlance, are believed to be present. It would
> be relatively simple to bring back the experience of invoking a command line
> tool to deliver a verdict. This could be added to the hbck2 tool itself but
> given that hbase-operator-tools is a separate project an intrinsic solution
> is desirable.
> An option that immediately comes to mind is modification of the Master's
> hbck.jsp page to provide a JSON formatted output option if the HTTP Accept
> header asks for text/json. However, looking at the source of hbck.jsp, it
> makes more sense to leave it as is and implement a convenient machine
> parseable output format elsewhere. This can be trivially accomplished with a
> new servlet. Like hbck.jsp the servlet implementation would get a reference
> to HbckChore and present the information this class makes available via its
> various getters.
> The machine parseable output is sufficient to enable headless hbck status
> checking but it still would be nice if we could provide operators a command
> line tool that formats the information for convenient viewing in a terminal.
> That part could be implemented in the hbck2 tool after this proposal is
> implemented.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26192) hbck.jsp should provide a JSON formatted output option
[
https://issues.apache.org/jira/browse/HBASE-26192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26192:
Priority: Major (was: Minor)
> hbck.jsp should provide a JSON formatted output option
> --
>
> Key: HBASE-26192
> URL: https://issues.apache.org/jira/browse/HBASE-26192
> Project: HBase
> Issue Type: New Feature
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.6.0
>
>
> It used to be possible to get hbck's verdict of cluster status from the
> command line, especially useful for headless deployments, i.e. without
> requiring a browser with sufficient connectivity to load a UI, or scrape
> information out of raw HTML, or write regex to comb over log4j output. The
> hbck tool's output wasn't particularly convenient to parse but it was
> straightforward to extract the desired information with a handful of regular
> expressions.
> HBCK2 has a different design philosophy than the old hbck, which is to serve
> as a collection of small and discrete recovery and repair functions, rather
> than attempt to be a universal repair tool. This makes a lot of sense and
> isn't the issue at hand. Unfortunately the old hbck's utility for reporting
> the current cluster health assessment has not been replaced either in whole
> or in part. Instead:
> {quote}
> HBCK2 is for fixes. For listings of inconsistencies or blockages in the
> running cluster, you go elsewhere, to the logs and UI of the running cluster
> Master. Once an issue has been identified, you use the HBCK2 tool to ask the
> Master to effect fixes or to skip-over bad state. Asking the Master to make
> the fixes rather than try and effect the repair locally in a fix-it tool's
> context is another important difference between HBCK2 and hbck1.
> {quote}
> Developing custom tooling to mine logs and scrape UI simply to gain a top
> level assessment of system health is unsatisfying. There should be a
> convenient means for querying the system if issues that rise to the level of
> _inconsistency_, in the hbck parlance, are believed to be present. It would
> be relatively simple to bring back the experience of invoking a command line
> tool to deliver a verdict. This could be added to the hbck2 tool itself but
> given that hbase-operator-tools is a separate project an intrinsic solution
> is desirable.
> An option that immediately comes to mind is modification of the Master's
> hbck.jsp page to provide a JSON formatted output option if the HTTP Accept
> header asks for text/json. However, looking at the source of hbck.jsp, it
> makes more sense to leave it as is and implement a convenient machine
> parseable output format elsewhere. This can be trivially accomplished with a
> new servlet. Like hbck.jsp the servlet implementation would get a reference
> to HbckChore and present the information this class makes available via its
> various getters.
> The machine parseable output is sufficient to enable headless hbck status
> checking but it still would be nice if we could provide operators a command
> line tool that formats the information for convenient viewing in a terminal.
> That part could be implemented in the hbck2 tool after this proposal is
> implemented.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HBASE-26192) hbck.jsp should provide a JSON formatted output option
[
https://issues.apache.org/jira/browse/HBASE-26192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell reassigned HBASE-26192:
---
Assignee: Andrew Kyle Purtell
> hbck.jsp should provide a JSON formatted output option
> --
>
> Key: HBASE-26192
> URL: https://issues.apache.org/jira/browse/HBASE-26192
> Project: HBase
> Issue Type: New Feature
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.6.0
>
>
> It used to be possible to get hbck's verdict of cluster status from the
> command line, especially useful for headless deployments, i.e. without
> requiring a browser with sufficient connectivity to load a UI, or scrape
> information out of raw HTML, or write regex to comb over log4j output. The
> hbck tool's output wasn't particularly convenient to parse but it was
> straightforward to extract the desired information with a handful of regular
> expressions.
> HBCK2 has a different design philosophy than the old hbck, which is to serve
> as a collection of small and discrete recovery and repair functions, rather
> than attempt to be a universal repair tool. This makes a lot of sense and
> isn't the issue at hand. Unfortunately the old hbck's utility for reporting
> the current cluster health assessment has not been replaced either in whole
> or in part. Instead:
> {quote}
> HBCK2 is for fixes. For listings of inconsistencies or blockages in the
> running cluster, you go elsewhere, to the logs and UI of the running cluster
> Master. Once an issue has been identified, you use the HBCK2 tool to ask the
> Master to effect fixes or to skip-over bad state. Asking the Master to make
> the fixes rather than try and effect the repair locally in a fix-it tool's
> context is another important difference between HBCK2 and hbck1.
> {quote}
> Developing custom tooling to mine logs and scrape UI simply to gain a top
> level assessment of system health is unsatisfying. There should be a
> convenient means for querying the system if issues that rise to the level of
> _inconsistency_, in the hbck parlance, are believed to be present. It would
> be relatively simple to bring back the experience of invoking a command line
> tool to deliver a verdict. This could be added to the hbck2 tool itself but
> given that hbase-operator-tools is a separate project an intrinsic solution
> is desirable.
> An option that immediately comes to mind is modification of the Master's
> hbck.jsp page to provide a JSON formatted output option if the HTTP Accept
> header asks for text/json. However, looking at the source of hbck.jsp, it
> makes more sense to leave it as is and implement a convenient machine
> parseable output format elsewhere. This can be trivially accomplished with a
> new servlet. Like hbck.jsp the servlet implementation would get a reference
> to HbckChore and present the information this class makes available via its
> various getters.
> The machine parseable output is sufficient to enable headless hbck status
> checking but it still would be nice if we could provide operators a command
> line tool that formats the information for convenient viewing in a terminal.
> That part could be implemented in the hbck2 tool after this proposal is
> implemented.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26192) hbck.jsp should provide a JSON formatted output option
[
https://issues.apache.org/jira/browse/HBASE-26192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26192:
Fix Version/s: 2.6.0
> hbck.jsp should provide a JSON formatted output option
> --
>
> Key: HBASE-26192
> URL: https://issues.apache.org/jira/browse/HBASE-26192
> Project: HBase
> Issue Type: New Feature
>Reporter: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.6.0
>
>
> It used to be possible to get hbck's verdict of cluster status from the
> command line, especially useful for headless deployments, i.e. without
> requiring a browser with sufficient connectivity to load a UI, or scrape
> information out of raw HTML, or write regex to comb over log4j output. The
> hbck tool's output wasn't particularly convenient to parse but it was
> straightforward to extract the desired information with a handful of regular
> expressions.
> HBCK2 has a different design philosophy than the old hbck, which is to serve
> as a collection of small and discrete recovery and repair functions, rather
> than attempt to be a universal repair tool. This makes a lot of sense and
> isn't the issue at hand. Unfortunately the old hbck's utility for reporting
> the current cluster health assessment has not been replaced either in whole
> or in part. Instead:
> {quote}
> HBCK2 is for fixes. For listings of inconsistencies or blockages in the
> running cluster, you go elsewhere, to the logs and UI of the running cluster
> Master. Once an issue has been identified, you use the HBCK2 tool to ask the
> Master to effect fixes or to skip-over bad state. Asking the Master to make
> the fixes rather than try and effect the repair locally in a fix-it tool's
> context is another important difference between HBCK2 and hbck1.
> {quote}
> Developing custom tooling to mine logs and scrape UI simply to gain a top
> level assessment of system health is unsatisfying. There should be a
> convenient means for querying the system if issues that rise to the level of
> _inconsistency_, in the hbck parlance, are believed to be present. It would
> be relatively simple to bring back the experience of invoking a command line
> tool to deliver a verdict. This could be added to the hbck2 tool itself but
> given that hbase-operator-tools is a separate project an intrinsic solution
> is desirable.
> An option that immediately comes to mind is modification of the Master's
> hbck.jsp page to provide a JSON formatted output option if the HTTP Accept
> header asks for text/json. However, looking at the source of hbck.jsp, it
> makes more sense to leave it as is and implement a convenient machine
> parseable output format elsewhere. This can be trivially accomplished with a
> new servlet. Like hbck.jsp the servlet implementation would get a reference
> to HbckChore and present the information this class makes available via its
> various getters.
> The machine parseable output is sufficient to enable headless hbck status
> checking but it still would be nice if we could provide operators a command
> line tool that formats the information for convenient viewing in a terminal.
> That part could be implemented in the hbck2 tool after this proposal is
> implemented.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26330) Document new provided compression codecs
[ https://issues.apache.org/jira/browse/HBASE-26330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-26330: Priority: Blocker (was: Minor) Documentation blocker for release. > Document new provided compression codecs > > > Key: HBASE-26330 > URL: https://issues.apache.org/jira/browse/HBASE-26330 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Blocker > Fix For: 2.5.0, 3.0.0-alpha-2, 2.6.0 > > > Document the new compression codecs: > - The configuration keys used for setting the codec implementations for the > various algorithms > - The provided compression codecs > - Default schema recommendations (LZ4) > - Default WAL value compression recommendations (LZ4) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26330) Document new provided compression codecs
[ https://issues.apache.org/jira/browse/HBASE-26330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-26330: Fix Version/s: 2.6.0 > Document new provided compression codecs > > > Key: HBASE-26330 > URL: https://issues.apache.org/jira/browse/HBASE-26330 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Minor > Fix For: 2.5.0, 3.0.0-alpha-2, 2.6.0 > > > Document the new compression codecs: > - The configuration keys used for setting the codec implementations for the > various algorithms > - The provided compression codecs > - Default schema recommendations (LZ4) > - Default WAL value compression recommendations (LZ4) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26332) Update next set of minor releases for Hadoop 3.1 EOM
[ https://issues.apache.org/jira/browse/HBASE-26332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-26332: Fix Version/s: 2.6.0 > Update next set of minor releases for Hadoop 3.1 EOM > - > > Key: HBASE-26332 > URL: https://issues.apache.org/jira/browse/HBASE-26332 > Project: HBase > Issue Type: Task > Components: hadoop3 >Affects Versions: 2.5.0, 3.0.0-alpha-2 >Reporter: Sean Busbey >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2, 2.6.0 > > > Looks like Hadoop 3.1 went EOL back in June. I can't find a formal > announcement, but "[\[VOTE\] Hadoop 3.1.x > EOL|https://s.apache.org/hadoop-vote-3.1-eol]; passed and HADOOP-17759 > removed 3.1 from their active downloads page and the asf active distribution > mirroring. > We should update our minimum testing version to 3.2 and update the hadoop > dependency matrix to reflect this change. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26366) ZK interaction during Master startup produces loads of root-less spans
[
https://issues.apache.org/jira/browse/HBASE-26366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26366:
Fix Version/s: 2.5.0
3.0.0-alpha-2
2.6.0
> ZK interaction during Master startup produces loads of root-less spans
> --
>
> Key: HBASE-26366
> URL: https://issues.apache.org/jira/browse/HBASE-26366
> Project: HBase
> Issue Type: Bug
> Components: tracing
>Affects Versions: 2.5.0, 3.0.0-alpha-2
>Reporter: Nick Dimiduk
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.6.0
>
> Attachments: image.png,
> regroup-spans-for-recoverablezookeeper.create.png
>
>
> Enable tracing with {{-Dotel.traces.sampler=always_on}} and launch the
> standalone master, check the trace store. We see lots of single-span traces,
> each a call to different {{RecoverableZooKeeper}} methods, like {{create}}
> and {{exists}}. Whatever master startup thread this is should define its own
> root span.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (HBASE-26399) branch-2 modules dependencies are confused
[
https://issues.apache.org/jira/browse/HBASE-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456073#comment-17456073
]
Andrew Kyle Purtell edited comment on HBASE-26399 at 12/9/21, 1:13 AM:
---
[~mdrob] Are you planning to submit a patch for this report?
As is otherwise I plan to resolve as Cannot Reproduce.
was (Author: apurtell):
[~mdrob] Are you planning to submit a patch for this report?
> branch-2 modules dependencies are confused
> --
>
> Key: HBASE-26399
> URL: https://issues.apache.org/jira/browse/HBASE-26399
> Project: HBase
> Issue Type: Task
> Components: build
>Affects Versions: 2.5.0
> Environment: branch-2
> Java version: 11.0.12
> Apache Maven 3.8.3
>Reporter: Mike Drob
>Priority: Major
> Fix For: 2.5.0, 2.6.0
>
>
> When trying to build branch-2, I get several dependency resolution issues.
> Different issues based on what approach I try.
>
> {{mvn -e -B --no-transfer-progress -Dhadoop.profile=3.0
> -Dhadoop-three.version=3.0.0-cdh6.3.2 clean install -Dmaven.test.skip=true}}
> will fail with:
> {noformat}
> 231.4 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M3:enforce
> (hadoop-profile-min-maven-min-java-banned-xerces) on project
> hbase-build-configuration: Execution
> hadoop-profile-min-maven-min-java-banned-xerces of goal
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M3:enforce failed:
> org.apache.maven.shared.dependency.graph.DependencyGraphBuilderException:
> Could not resolve following dependencies:
> [org.apache.hbase:hbase-annotations:jar:tests:2.5.0-SNAPSHOT (test)]: Could
> not resolve dependencies for project
> org.apache.hbase:hbase-build-configuration:pom:2.5.0-SNAPSHOT: Could not find
> artifact org.apache.hbase:hbase-annotations:jar:tests:2.5.0-SNAPSHOT in
> snapshots{noformat}
> Note that I'm running this behind a corporate firewall, so if there is a
> snapshot available upstream in apache snapshots repo, then I'm not able to
> access it. I would have thought that it can use the locally built and
> installed one instead, since build-configuration does correctly come after
> annotations.
>
> If I change to {{-DskipTests}} then this target succeeds, but later {{mvn
> site}} fails with
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-site-plugin:3.7.1:site (default-site) on
> project hbase: Error generating maven-javadoc-plugin:3.2.0:aggregate-no-fork
> report:
> [ERROR] Exit code: 1 -
> /home/jenkins-agent/hbase/hbase-testing-util/src/main/java/org/apache/hadoop/hbase/testing/TestingHBaseClusterOption.java:24:
> error: cannot find symbol
> [ERROR] import org.apache.hadoop.hbase.StartMiniClusterOption;
> [ERROR] ^
> [ERROR] symbol: class StartMiniClusterOption
> [ERROR] location: package org.apache.hadoop.hbase {noformat}
> Again, I'm missing testing packages, this time for javadocs.
>
> I believe I was getting similar errors on main branch, although I don't have
> logs for that anymore.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26399) branch-2 modules dependencies are confused
[
https://issues.apache.org/jira/browse/HBASE-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26399:
Fix Version/s: 2.6.0
> branch-2 modules dependencies are confused
> --
>
> Key: HBASE-26399
> URL: https://issues.apache.org/jira/browse/HBASE-26399
> Project: HBase
> Issue Type: Task
> Components: build
>Affects Versions: 2.5.0
> Environment: branch-2
> Java version: 11.0.12
> Apache Maven 3.8.3
>Reporter: Mike Drob
>Priority: Major
> Fix For: 2.5.0, 2.6.0
>
>
> When trying to build branch-2, I get several dependency resolution issues.
> Different issues based on what approach I try.
>
> {{mvn -e -B --no-transfer-progress -Dhadoop.profile=3.0
> -Dhadoop-three.version=3.0.0-cdh6.3.2 clean install -Dmaven.test.skip=true}}
> will fail with:
> {noformat}
> 231.4 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M3:enforce
> (hadoop-profile-min-maven-min-java-banned-xerces) on project
> hbase-build-configuration: Execution
> hadoop-profile-min-maven-min-java-banned-xerces of goal
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M3:enforce failed:
> org.apache.maven.shared.dependency.graph.DependencyGraphBuilderException:
> Could not resolve following dependencies:
> [org.apache.hbase:hbase-annotations:jar:tests:2.5.0-SNAPSHOT (test)]: Could
> not resolve dependencies for project
> org.apache.hbase:hbase-build-configuration:pom:2.5.0-SNAPSHOT: Could not find
> artifact org.apache.hbase:hbase-annotations:jar:tests:2.5.0-SNAPSHOT in
> snapshots{noformat}
> Note that I'm running this behind a corporate firewall, so if there is a
> snapshot available upstream in apache snapshots repo, then I'm not able to
> access it. I would have thought that it can use the locally built and
> installed one instead, since build-configuration does correctly come after
> annotations.
>
> If I change to {{-DskipTests}} then this target succeeds, but later {{mvn
> site}} fails with
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-site-plugin:3.7.1:site (default-site) on
> project hbase: Error generating maven-javadoc-plugin:3.2.0:aggregate-no-fork
> report:
> [ERROR] Exit code: 1 -
> /home/jenkins-agent/hbase/hbase-testing-util/src/main/java/org/apache/hadoop/hbase/testing/TestingHBaseClusterOption.java:24:
> error: cannot find symbol
> [ERROR] import org.apache.hadoop.hbase.StartMiniClusterOption;
> [ERROR] ^
> [ERROR] symbol: class StartMiniClusterOption
> [ERROR] location: package org.apache.hadoop.hbase {noformat}
> Again, I'm missing testing packages, this time for javadocs.
>
> I believe I was getting similar errors on main branch, although I don't have
> logs for that anymore.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26405) IntegrationTestLoadSmallValues
[ https://issues.apache.org/jira/browse/HBASE-26405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-26405: Fix Version/s: 2.6.0 (was: 2.5.0) > IntegrationTestLoadSmallValues > -- > > Key: HBASE-26405 > URL: https://issues.apache.org/jira/browse/HBASE-26405 > Project: HBase > Issue Type: Test > Components: integration tests, test >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Minor > Fix For: 3.0.0-alpha-2, 2.6.0 > > > This integration test emulates a use case that stores a lot of small values > into a table that would likely be heavily indexed (ROW_INDEX_V1, small > blocks, etc.), an application that crowdsources weather (temperature) > observation data. This IT can be used to test and optimize compression > settings for such cases. It comes with a companion utility, > HFileBlockExtracter, which extracts block data from HFiles into a set of > individual files for each block's data, for use in training external > compression dictionaries, perhaps with ZStandard's 'zstd' utility. > See javadoc on class IntegrationTestLoadSmallValues for usage. > This was used to test the changes on HBASE-26353. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26399) branch-2 modules dependencies are confused
[
https://issues.apache.org/jira/browse/HBASE-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456073#comment-17456073
]
Andrew Kyle Purtell commented on HBASE-26399:
-
[~mdrob] Are you planning to submit a patch for this report?
> branch-2 modules dependencies are confused
> --
>
> Key: HBASE-26399
> URL: https://issues.apache.org/jira/browse/HBASE-26399
> Project: HBase
> Issue Type: Task
> Components: build
>Affects Versions: 2.5.0
> Environment: branch-2
> Java version: 11.0.12
> Apache Maven 3.8.3
>Reporter: Mike Drob
>Priority: Major
> Fix For: 2.5.0, 2.6.0
>
>
> When trying to build branch-2, I get several dependency resolution issues.
> Different issues based on what approach I try.
>
> {{mvn -e -B --no-transfer-progress -Dhadoop.profile=3.0
> -Dhadoop-three.version=3.0.0-cdh6.3.2 clean install -Dmaven.test.skip=true}}
> will fail with:
> {noformat}
> 231.4 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M3:enforce
> (hadoop-profile-min-maven-min-java-banned-xerces) on project
> hbase-build-configuration: Execution
> hadoop-profile-min-maven-min-java-banned-xerces of goal
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M3:enforce failed:
> org.apache.maven.shared.dependency.graph.DependencyGraphBuilderException:
> Could not resolve following dependencies:
> [org.apache.hbase:hbase-annotations:jar:tests:2.5.0-SNAPSHOT (test)]: Could
> not resolve dependencies for project
> org.apache.hbase:hbase-build-configuration:pom:2.5.0-SNAPSHOT: Could not find
> artifact org.apache.hbase:hbase-annotations:jar:tests:2.5.0-SNAPSHOT in
> snapshots{noformat}
> Note that I'm running this behind a corporate firewall, so if there is a
> snapshot available upstream in apache snapshots repo, then I'm not able to
> access it. I would have thought that it can use the locally built and
> installed one instead, since build-configuration does correctly come after
> annotations.
>
> If I change to {{-DskipTests}} then this target succeeds, but later {{mvn
> site}} fails with
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-site-plugin:3.7.1:site (default-site) on
> project hbase: Error generating maven-javadoc-plugin:3.2.0:aggregate-no-fork
> report:
> [ERROR] Exit code: 1 -
> /home/jenkins-agent/hbase/hbase-testing-util/src/main/java/org/apache/hadoop/hbase/testing/TestingHBaseClusterOption.java:24:
> error: cannot find symbol
> [ERROR] import org.apache.hadoop.hbase.StartMiniClusterOption;
> [ERROR] ^
> [ERROR] symbol: class StartMiniClusterOption
> [ERROR] location: package org.apache.hadoop.hbase {noformat}
> Again, I'm missing testing packages, this time for javadocs.
>
> I believe I was getting similar errors on main branch, although I don't have
> logs for that anymore.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] apurtell commented on pull request #3802: HBASE-26405 IntegrationTestLoadSmallValues
apurtell commented on pull request #3802: URL: https://github.com/apache/hbase/pull/3802#issuecomment-989369008 This was used to simulate a use case with small integer values in order to prove a compression improvement. The value here, as such, is the simulation of a use case with a potentially very large set of rows comprised of small integer values / small cells. We don't need to accept this into the suite if there isn't enough value there. I have no strong opinion either way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26419) Tracing Spans of RPCs should follow otel semantic conventions
[
https://issues.apache.org/jira/browse/HBASE-26419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26419:
Fix Version/s: 2.6.0
> Tracing Spans of RPCs should follow otel semantic conventions
> -
>
> Key: HBASE-26419
> URL: https://issues.apache.org/jira/browse/HBASE-26419
> Project: HBase
> Issue Type: Improvement
> Components: tracing
>Affects Versions: 3.0.0-alpha-1, 2.5.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Blocker
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.6.0
>
>
> The OpenTelemetry project has a [conventions
> document|https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/rpc.md]
> for RPCs that is pretty prescriptive. One thing I noticed is that it
> requires there be spans named as {{$package.$service/$method}} ; we don't do
> this. We should address this and any other gaps.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26445) Procedure state pretty-printing should use printable character friendly encoding
[
https://issues.apache.org/jira/browse/HBASE-26445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-26445.
-
Resolution: Later
> Procedure state pretty-printing should use printable character friendly
> encoding
>
>
> Key: HBASE-26445
> URL: https://issues.apache.org/jira/browse/HBASE-26445
> Project: HBase
> Issue Type: Task
> Components: Operability
>Affects Versions: 2.4.8
>Reporter: Andrew Kyle Purtell
>Priority: Minor
>
> The shell 'list_procedures' command produces output like:
>
> 889 org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure
> SUCCESS 2021-11-10 22:20:34 UTC 2021-11-10 22:20:35 UTC [{"state"=>[1, 2, 3,
> 11, 4, 5, 6, 7, 8, 9, 10, 2147483648]}, {"regionId"=>"1636579678894",
> "tableName"=>{"namespace"=>"ZGVmYXVsdA==",
> "qualifier"=>"SW50ZWdyYXRpb25UZXN0TG9hZENvbW1vbkNyYXds"},
> "startKey"=>"dWsuY28uZ3Jhbml0ZXRyYW5zZm9ybWF0aW9ucy53d3d8L2dhbGxlcnkvdA==",
> "endKey"=>"dXMuYmFuZHwvYmFuZC81OA==", "offline"=>false, "split"=>false,
> "replicaId"=>0}, {"userInfo"=>{"effectiveUser"=>"apurtell"},
> "parentRegionInfo"=>{"regionId"=>"1636579678894",
> "tableName"=>{"namespace"=>"ZGVmYXVsdA==",
> "qualifier"=>"SW50ZWdyYXRpb25UZXN0TG9hZENvbW1vbkNyYXds"},
> "startKey"=>"dWsuY28uZ3Jhbml0ZXRyYW5zZm9ybWF0aW9ucy53d3d8L2dhbGxlcnkvdA==",
> "endKey"=>"dXMuYmFuZHwvYmFuZC81OA==", "offline"=>false, "split"=>false,
> "replicaId"=>0}, "childRegionInfo"=>[{"regionId"=>"1636582834759",
> "tableName"=>{"namespace"=>"ZGVmYXVsdA==",
> "qualifier"=>"SW50ZWdyYXRpb25UZXN0TG9hZENvbW1vbkNyYXds"},
> "startKey"=>"dWsuY28uZ3Jhbml0ZXRyYW5zZm9ybWF0aW9ucy53d3d8L2dhbGxlcnkvdA==",
> "endKey"=>"dWsuY28uc2ltb25hbmRzY2h1c3Rlci53d3d8L2Jvb2tzL1RoZS1P",
> "offline"=>false, "split"=>false, "replicaId"=>0},
> {"regionId"=>"1636582834759", "tableName"=>{"namespace"=>"ZGVmYXVsdA==",
> "qualifier"=>"SW50ZWdyYXRpb25UZXN0TG9hZENvbW1vbkNyYXds"},
> "startKey"=>"dWsuY28uc2ltb25hbmRzY2h1c3Rlci53d3d8L2Jvb2tzL1RoZS1P",
> "endKey"=>"dXMuYmFuZHwvYmFuZC81OA==", "offline"=>false, "split"=>false,
> "replicaId"=>0}]}]
>
> The base64 encoding of byte[] values offers poor usability.
> Generally, table names etc are printable characters encoded in byte[]. Base64
> encoding them totally obfuscates information that is important to see at a
> glance. Even start keys and end keys might be printable characters.
> It would be better to use Bytes.toStringBinary or something else that shows
> printable characters as-is while escaping others that would be invalid in
> JSON formatting.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26445) Procedure state pretty-printing should use printable character friendly encoding
[
https://issues.apache.org/jira/browse/HBASE-26445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26445:
Fix Version/s: (was: 2.5.0)
(was: 3.0.0-alpha-2)
(was: 2.4.9)
> Procedure state pretty-printing should use printable character friendly
> encoding
>
>
> Key: HBASE-26445
> URL: https://issues.apache.org/jira/browse/HBASE-26445
> Project: HBase
> Issue Type: Task
> Components: Operability
>Affects Versions: 2.4.8
>Reporter: Andrew Kyle Purtell
>Priority: Minor
>
> The shell 'list_procedures' command produces output like:
>
> 889 org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure
> SUCCESS 2021-11-10 22:20:34 UTC 2021-11-10 22:20:35 UTC [{"state"=>[1, 2, 3,
> 11, 4, 5, 6, 7, 8, 9, 10, 2147483648]}, {"regionId"=>"1636579678894",
> "tableName"=>{"namespace"=>"ZGVmYXVsdA==",
> "qualifier"=>"SW50ZWdyYXRpb25UZXN0TG9hZENvbW1vbkNyYXds"},
> "startKey"=>"dWsuY28uZ3Jhbml0ZXRyYW5zZm9ybWF0aW9ucy53d3d8L2dhbGxlcnkvdA==",
> "endKey"=>"dXMuYmFuZHwvYmFuZC81OA==", "offline"=>false, "split"=>false,
> "replicaId"=>0}, {"userInfo"=>{"effectiveUser"=>"apurtell"},
> "parentRegionInfo"=>{"regionId"=>"1636579678894",
> "tableName"=>{"namespace"=>"ZGVmYXVsdA==",
> "qualifier"=>"SW50ZWdyYXRpb25UZXN0TG9hZENvbW1vbkNyYXds"},
> "startKey"=>"dWsuY28uZ3Jhbml0ZXRyYW5zZm9ybWF0aW9ucy53d3d8L2dhbGxlcnkvdA==",
> "endKey"=>"dXMuYmFuZHwvYmFuZC81OA==", "offline"=>false, "split"=>false,
> "replicaId"=>0}, "childRegionInfo"=>[{"regionId"=>"1636582834759",
> "tableName"=>{"namespace"=>"ZGVmYXVsdA==",
> "qualifier"=>"SW50ZWdyYXRpb25UZXN0TG9hZENvbW1vbkNyYXds"},
> "startKey"=>"dWsuY28uZ3Jhbml0ZXRyYW5zZm9ybWF0aW9ucy53d3d8L2dhbGxlcnkvdA==",
> "endKey"=>"dWsuY28uc2ltb25hbmRzY2h1c3Rlci53d3d8L2Jvb2tzL1RoZS1P",
> "offline"=>false, "split"=>false, "replicaId"=>0},
> {"regionId"=>"1636582834759", "tableName"=>{"namespace"=>"ZGVmYXVsdA==",
> "qualifier"=>"SW50ZWdyYXRpb25UZXN0TG9hZENvbW1vbkNyYXds"},
> "startKey"=>"dWsuY28uc2ltb25hbmRzY2h1c3Rlci53d3d8L2Jvb2tzL1RoZS1P",
> "endKey"=>"dXMuYmFuZHwvYmFuZC81OA==", "offline"=>false, "split"=>false,
> "replicaId"=>0}]}]
>
> The base64 encoding of byte[] values offers poor usability.
> Generally, table names etc are printable characters encoded in byte[]. Base64
> encoding them totally obfuscates information that is important to see at a
> glance. Even start keys and end keys might be printable characters.
> It would be better to use Bytes.toStringBinary or something else that shows
> printable characters as-is while escaping others that would be invalid in
> JSON formatting.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26440) Region opens sometimes fail when bucket cache is in use
[ https://issues.apache.org/jira/browse/HBASE-26440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-26440: Fix Version/s: 2.6.0 > Region opens sometimes fail when bucket cache is in use > --- > > Key: HBASE-26440 > URL: https://issues.apache.org/jira/browse/HBASE-26440 > Project: HBase > Issue Type: Bug > Components: BlockCache, BucketCache, regionserver >Affects Versions: 2.4.8 >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.9, 2.6.0 > > > After a split, I think, the region is reopened, and: > 2021-11-10 16:47:48,929 ERROR > [RS_OPEN_REGION-regionserver/ip-172-31-63-83:8120-1] regionserver.HRegion: > Could not initialize all stores for the > region=IntegrationTestLoadCommonCrawl,,1636562865609.e2df2061bcdc037070041de734a187ff. > Caused by: java.io.IOException: java.lang.RuntimeException: Cached block > contents differ, which should not have happened. > cacheKey:1e395fa6f0584ebc9fd831f1451dc4ba.cbb4e3a74e703805898a2a3bf0af94e7_48543272 > at > org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:577) > at > org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:534) > at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:298) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6546) > Caused by: java.lang.RuntimeException: Cached block contents differ, which > should not have happened. > cacheKey:1e395fa6f0584ebc9fd831f1451dc4ba.cbb4e3a74e703805898a2a3bf0af94e7_48543272 > at > org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.validateBlockAddition(BlockCacheUtil.java:205) > at > org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.shouldReplaceExistingCacheBlock(BlockCacheUtil.java:237) > at > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.shouldReplaceExistingCacheBlock(BucketCache.java:446) > at > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlockWithWait(BucketCache.java:431) > at > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlock(BucketCache.java:417) > at > org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.cacheBlock(CombinedBlockCache.java:64) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.lambda$readBlock$2(HFileReaderImpl.java:1346) > The open fails but another attempt somewhere else succeeds. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26469) HBase shell has changed exit behavior
[
https://issues.apache.org/jira/browse/HBASE-26469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26469:
Fix Version/s: 2.5.0
2.4.9
2.6.0
> HBase shell has changed exit behavior
> -
>
> Key: HBASE-26469
> URL: https://issues.apache.org/jira/browse/HBASE-26469
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.5.0, 3.0.0-alpha-2, 2.4.8
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 2.5.0, 2.4.9, 2.6.0
>
> Attachments: hbase-1.4.14-exit-behavior.log,
> hbase-1.7.1-exit-behavior.log, hbase-2.0.6-exit-behavior.log,
> hbase-2.1.9-exit-behavior.log, hbase-2.2.7-exit-behavior.log,
> hbase-2.3.7-exit-behavior.log, hbase-2.4.8-exit-behavior.log,
> hbase-3.0.0-alpha-2-exit-behavior.log
>
>
> The HBase shell has changed behavior in a way that breaks being able to exit
> properly.
> Two example scripts to act as stand ins for hbase shell scripts to "do
> something simple then exit":
> {code}
> tmp % echo "list\nexit" > clean_exit.rb
> tmp % echo "list\nexit 1" > error_exit.rb
> {code}
> Giving these two scripts is possible:
> * passed as a cli argument
> * via redirected stdin
> Additionally the shell invocation can be:
> * in the default compatibility mode
> * with the "non interactive" flag that gives an exit code that reflects
> runtime errors
> I'll post logs of the details as attachments but here are some tables of the
> exit codes.
> The {{clean_exit.rb}} invocations ought to exit with success, exit code 0.
> || || 1.4.14 || 1.7.1 || 2.0.6 || 2.1.9 || 2.2.7 || 2.3.7 || 2.4.8
> || master ||
> | cli, default |0 |0 |0 |0 |0 |0 |1 |
>1* |
> | cli, -n | 0 |0 |0 |0 |0 |0 |1 |
> hang |
> | stdin, default | 0 |0 |0 |0 |0 |0 |0 |
>0|
> | stdin, -n | 1 |1 |1 |1 |1 |1 |1* |
>1* |
> The {{error_exit.rb}} invocation should return a non-zero exit code, unless
> we're specifically trying to match a normal hbase shell session.
> || || 1.4.14 || 1.7.1 || 2.0.6 || 2.1.9 || 2.2.7 || 2.3.7 || 2.4.8 ||
> master ||
> | cli, default | 1 |1 |1 |1 |1 |1 |1* |
> 1* |
> | cli, -n |1 |1 |1 |1 |1 |1 |1* |
> hang |
> | stdin, default | 0 |0 |0 |0 |0 |0 |0 |
> 0|
> | stdin, -n | 1 |1 |1 |1 |1 |1 |1* |
> 1* |
> In cases marked with * the error details are different.
> The biggest concern are the new-to-2.4 non-zero exit code when we should have
> a success and the hanging.
> The former looks like this:
> {code}
> ERROR NoMethodError: private method `exit' called for nil:NilClass
> {code}
> The change in error details for the error exit script also shows this same
> detail.
> This behavior appears to be a side effect of HBASE-11686. As far as I can
> tell, the IRB handling of 'exit' calls fail because we implement our own
> handling of sessoins rather than rely on the intended session interface. We
> never set a current session, and IRB's exit implementation presumes there
> will be one.
> Running in debug shows this in a stacktrace:
> {code}
> Took 0.4563 seconds
> ERROR NoMethodError: private method `exit' called for nil:NilClass
> NoMethodError: private method `exit' called for nil:NilClass
> irb_exit at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/extend-command.rb:30
> evaluate at stdin:2
> eval at org/jruby/RubyKernel.java:1048
> evaluate at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/workspace.rb:85
> eval_io at uri:classloader:/shell.rb:327
> each_top_level_statement at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/ruby-lex.rb:246
> loop at org/jruby/RubyKernel.java:1442
> each_top_level_statement at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/ruby-lex.rb:232
> catch at org/jruby/RubyKernel.java:1189
> each_top_level_statement at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/ruby-lex.rb:231
> eval_io at uri:classloader:/shell.rb:326
> classpath:/jar-bootstrap.rb at classpath:/jar-bootstrap.rb:194
> exception_handler at uri:classloader:/shell.rb:339
> at classpath:/jar-bootstrap.rb:194
> {code}
> And in our version of IRB (0.9.6) [line 30 for
>
[jira] [Updated] (HBASE-26492) [branch-2, hbase-thirdparty, Java 8] TestUnloadAccessController and other unit tests fail to start due to ByteBuffer link error
[
https://issues.apache.org/jira/browse/HBASE-26492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26492:
Fix Version/s: 2.5.0
2.4.9
2.6.0
> [branch-2, hbase-thirdparty, Java 8] TestUnloadAccessController and other
> unit tests fail to start due to ByteBuffer link error
> ---
>
> Key: HBASE-26492
> URL: https://issues.apache.org/jira/browse/HBASE-26492
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.4.8, 2.4.9
> Environment: Java 1.8.0_312, vendor: Azul Systems, Inc., runtime:
> /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre
> OS name: "mac os x", version: "12.0.1", arch: "aarch64", family: "mac"
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 2.5.0, 2.4.9, 2.6.0
>
>
> org.apache.hadoop.hbase.security.access.TestUnloadAccessController
> Hang in setUpBeforeClass. Master will not initialize. Root cause is a
> NoSuchMethodError.
> {noformat}
> 2021-11-26 19:09:56,465 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=62950]
> ipc.RpcExecutor$Handler(370):
> Handler errors java.lang.NoSuchMethodError:
> java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
> at
> org.apache.hbase.thirdparty.com.google.protobuf.CodedOutputStream$HeapNioEncoder.flush(CodedOutputStream.java:1546)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.writeToCOS(ServerCall.java:378)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.createHeaderAndMessageBytes(ServerCall.java:385)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.createHeaderAndMessageBytes(ServerCall.java:363)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.setResponse(ServerCall.java:267)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:168)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:354)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:334)
> {noformat}
> This is a known issue with ByteBuffer in JDK 8 vs ByteBuffer in later
> versions. When code is compiled with Java 9 or later using a specific subset
> of ByteBuffer APIs, the resulting bytecode will not link with Java 8's
> runtime. It works fine the other way. When compiled with Java 8, the bytecode
> will link with later Java runtimes just fine.
> protobuf included into hbase-thirdparty was likely compiled with Java 9 or
> later. We shade that bytecode as is into hbase-thirdparty. Tests were
> attempted with Java 8, so this failure case manifested.
> Apache Maven 3.8.3 (ff8e977a158738155dc465c6a97ffaf31982d739)
> Java version: 1.8.0_312, vendor: Azul Systems, Inc., runtime:
> /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre
> OS name: "mac os x", version: "12.0.1", arch: "aarch64", family: "mac"
> We should be able to fix this problem by compiling protobuf with Java 8 and
> then shading the result when building hbase-thirdparty.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26492) [branch-2, hbase-thirdparty, Java 8] TestUnloadAccessController and other unit tests fail to start due to ByteBuffer link error
[
https://issues.apache.org/jira/browse/HBASE-26492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26492:
Fix Version/s: (was: 2.5.0)
(was: 2.4.9)
> [branch-2, hbase-thirdparty, Java 8] TestUnloadAccessController and other
> unit tests fail to start due to ByteBuffer link error
> ---
>
> Key: HBASE-26492
> URL: https://issues.apache.org/jira/browse/HBASE-26492
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.4.8, 2.4.9
> Environment: Java 1.8.0_312, vendor: Azul Systems, Inc., runtime:
> /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre
> OS name: "mac os x", version: "12.0.1", arch: "aarch64", family: "mac"
>Reporter: Andrew Kyle Purtell
>Priority: Major
>
> org.apache.hadoop.hbase.security.access.TestUnloadAccessController
> Hang in setUpBeforeClass. Master will not initialize. Root cause is a
> NoSuchMethodError.
> {noformat}
> 2021-11-26 19:09:56,465 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=62950]
> ipc.RpcExecutor$Handler(370):
> Handler errors java.lang.NoSuchMethodError:
> java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
> at
> org.apache.hbase.thirdparty.com.google.protobuf.CodedOutputStream$HeapNioEncoder.flush(CodedOutputStream.java:1546)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.writeToCOS(ServerCall.java:378)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.createHeaderAndMessageBytes(ServerCall.java:385)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.createHeaderAndMessageBytes(ServerCall.java:363)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.setResponse(ServerCall.java:267)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:168)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:354)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:334)
> {noformat}
> This is a known issue with ByteBuffer in JDK 8 vs ByteBuffer in later
> versions. When code is compiled with Java 9 or later using a specific subset
> of ByteBuffer APIs, the resulting bytecode will not link with Java 8's
> runtime. It works fine the other way. When compiled with Java 8, the bytecode
> will link with later Java runtimes just fine.
> protobuf included into hbase-thirdparty was likely compiled with Java 9 or
> later. We shade that bytecode as is into hbase-thirdparty. Tests were
> attempted with Java 8, so this failure case manifested.
> Apache Maven 3.8.3 (ff8e977a158738155dc465c6a97ffaf31982d739)
> Java version: 1.8.0_312, vendor: Azul Systems, Inc., runtime:
> /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre
> OS name: "mac os x", version: "12.0.1", arch: "aarch64", family: "mac"
> We should be able to fix this problem by compiling protobuf with Java 8 and
> then shading the result when building hbase-thirdparty.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26493) [branch-2] TestLruBlockCache is flaky
[ https://issues.apache.org/jira/browse/HBASE-26493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-26493: Fix Version/s: 2.6.0 > [branch-2] TestLruBlockCache is flaky > - > > Key: HBASE-26493 > URL: https://issues.apache.org/jira/browse/HBASE-26493 > Project: HBase > Issue Type: Bug >Affects Versions: 2.4.8, 2.4.9 > Environment: Java version: 11.0.13, vendor: Azul Systems, Inc., > runtime: /Library/Java/JavaVirtualMachines/zulu-11.jdk/Contents/Home > OS name: "mac os x", version: "12.0.1", arch: "aarch64", family: "mac" >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 2.5.0, 2.4.9, 2.6.0 > > > [WARNING] Flakes: > [WARNING] > org.apache.hadoop.hbase.io.hfile.TestLruBlockCache.testBackgroundEvictionThread > [ERROR] Run 1: TestLruBlockCache.testBackgroundEvictionThread:147 Eviction > never happened. > [INFO] Run 2: PASS -- This message was sent by Atlassian Jira (v8.20.1#820001)
