[GitHub] [hbase] virajjasani commented on pull request #3303: HBASE-25906 UI of master-status to show recent history of balancer de…
virajjasani commented on pull request #3303: URL: https://github.com/apache/hbase/pull/3303#issuecomment-847555722 @GeorryHuang There is no header.jsp change in this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani merged pull request #3296: HBASE-25906 UI of master-status to show recent history of balancer de…
virajjasani merged pull request #3296: URL: https://github.com/apache/hbase/pull/3296 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3302: HBASE-25911 Replace calls to System.currentTimeMillis with EnvironmentEdgeManager.currentTime
Apache-HBase commented on pull request #3302: URL: https://github.com/apache/hbase/pull/3302#issuecomment-847548849 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 21s | master passed | | +1 :green_heart: | compile | 7m 14s | master passed | | +1 :green_heart: | shadedjars | 9m 36s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 19s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 5s | the patch passed | | +1 :green_heart: | compile | 7m 21s | the patch passed | | +1 :green_heart: | javac | 7m 21s | the patch passed | | +1 :green_heart: | shadedjars | 9m 32s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 50s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 14s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 45s | hbase-zookeeper in the patch passed. | | +1 :green_heart: | unit | 0m 28s | hbase-balancer in the patch passed. | | +1 :green_heart: | unit | 0m 49s | hbase-http in the patch passed. | | +1 :green_heart: | unit | 1m 52s | hbase-procedure in the patch passed. | | -1 :x: | unit | 152m 56s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 10m 37s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | unit | 7m 40s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 3m 5s | hbase-endpoint in the patch passed. | | -1 :x: | unit | 17m 2s | hbase-backup in the patch failed. | | +1 :green_heart: | unit | 1m 17s | hbase-it in the patch passed. | | +1 :green_heart: | unit | 4m 12s | hbase-rest in the patch passed. | | +1 :green_heart: | unit | 2m 11s | hbase-examples in the patch passed. | | +1 :green_heart: | unit | 0m 52s | hbase-hbtop in the patch passed. | | | | 266m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3302 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0e3f8bb6bc40 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21aa553bc1 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-backup.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/1/testReport/ | | Max. process+thread count | 5096 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-zookeeper hbase-balancer hbase-http hbase-procedure hbase-server hbase-mapreduce hbase-thrift hbase-endpoint hbase-backup hbase-it hbase-rest hbase-examples hbase-hbtop U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3302: HBASE-25911 Replace calls to System.currentTimeMillis with EnvironmentEdgeManager.currentTime
Apache-HBase commented on pull request #3302: URL: https://github.com/apache/hbase/pull/3302#issuecomment-847547207 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 5s | master passed | | +1 :green_heart: | compile | 7m 55s | master passed | | +1 :green_heart: | shadedjars | 9m 32s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 6m 6s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 30s | the patch passed | | +1 :green_heart: | compile | 7m 39s | the patch passed | | +1 :green_heart: | javac | 7m 39s | the patch passed | | +1 :green_heart: | shadedjars | 8m 23s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 51s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 51s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 17s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 46s | hbase-zookeeper in the patch passed. | | +1 :green_heart: | unit | 0m 30s | hbase-balancer in the patch passed. | | +1 :green_heart: | unit | 0m 47s | hbase-http in the patch passed. | | +1 :green_heart: | unit | 1m 43s | hbase-procedure in the patch passed. | | -1 :x: | unit | 149m 55s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 10m 33s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | unit | 7m 26s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 2m 59s | hbase-endpoint in the patch passed. | | -1 :x: | unit | 14m 39s | hbase-backup in the patch failed. | | +1 :green_heart: | unit | 1m 16s | hbase-it in the patch passed. | | +1 :green_heart: | unit | 3m 11s | hbase-rest in the patch passed. | | +1 :green_heart: | unit | 1m 58s | hbase-examples in the patch passed. | | +1 :green_heart: | unit | 0m 56s | hbase-hbtop in the patch passed. | | | | 263m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3302 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 463cc188fecf 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21aa553bc1 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-backup.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/1/testReport/ | | Max. process+thread count | 4986 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-zookeeper hbase-balancer hbase-http hbase-procedure hbase-server hbase-mapreduce hbase-thrift hbase-endpoint hbase-backup hbase-it hbase-rest hbase-examples hbase-hbtop U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3304: Backport "HBASE-25534 Honor TableDescriptor settings earlier in normalization (#2917)" to branch-2
Apache-HBase commented on pull request #3304: URL: https://github.com/apache/hbase/pull/3304#issuecomment-847533758 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 7s | branch-2 passed | | +1 :green_heart: | compile | 3m 22s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 16s | branch-2 passed | | +1 :green_heart: | spotbugs | 2m 12s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 40s | the patch passed | | +1 :green_heart: | compile | 3m 21s | the patch passed | | +1 :green_heart: | javac | 3m 21s | the patch passed | | +1 :green_heart: | checkstyle | 1m 13s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 58s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 44m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3304/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3304 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux e956efec7448 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 073f23e2bd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 85 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3304/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25534) Honor TableDescriptor settings earlier in normalization
[
https://issues.apache.org/jira/browse/HBASE-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350784#comment-17350784
]
Baiqiang Zhao commented on HBASE-25534:
---
Put up a PR for branch-2.
> Honor TableDescriptor settings earlier in normalization
> ---
>
> Key: HBASE-25534
> URL: https://issues.apache.org/jira/browse/HBASE-25534
> Project: HBase
> Issue Type: Improvement
> Components: Normalizer
>Affects Versions: 3.0.0-alpha-1, 2.4.2
>Reporter: Baiqiang Zhao
>Assignee: Baiqiang Zhao
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.2
>
>
> -Table can be enabled and disabled merge/split in TableDescriptor, we should
> judge before calculating the plan.-
> The normalizer configuration can be set by table level. For example,
> hbase.normalizer.min.region.count can be set by "alter ‘table’,
> CONFIGURATION=>\{'hbase.normalizer.min.region.count' => '5'}". If the table
> is not set, then use the global configuration which is set in hbase-site.xml.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ZhaoBQ opened a new pull request #3304: Backport "HBASE-25534 Honor TableDescriptor settings earlier in normalization (#2917)" to branch-2
ZhaoBQ opened a new pull request #3304: URL: https://github.com/apache/hbase/pull/3304 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ZhaoBQ commented on pull request #3139: HBASE-25745 Deprecate/Rename config `hbase.normalizer.min.region.coun…
ZhaoBQ commented on pull request #3139: URL: https://github.com/apache/hbase/pull/3139#issuecomment-847517956 Thanks @ndimiduk, let me do the backport. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] sunhelly commented on a change in pull request #3297: HBASE-25905 Limit the shutdown time of WAL
sunhelly commented on a change in pull request #3297:
URL: https://github.com/apache/hbase/pull/3297#discussion_r638439826
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
##
@@ -986,14 +996,43 @@ public void shutdown() throws IOException {
i.logCloseRequested();
}
}
-rollWriterLock.lock();
+
+ExecutorService shutdownExecutor = Executors.newFixedThreadPool(1,
Review comment:
Thanks. The WAL shutdown task should have priority to be executed, it
may be delayed if put to the same single thread pool as archive WAL. What do
you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-25898) RS getting aborted due to NPE in Replication WALEntryStream
[
https://issues.apache.org/jira/browse/HBASE-25898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350778#comment-17350778
]
Hudson commented on HBASE-25898:
Results for branch branch-2
[build #259 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/259/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/259/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/259//console].
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/259/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/259/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/259//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> RS getting aborted due to NPE in Replication WALEntryStream
> ---
>
> Key: HBASE-25898
> URL: https://issues.apache.org/jira/browse/HBASE-25898
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4
>
>
> Below sequence of events happened in a customer cluster
> An empty WAL file got roll req.
> The close of file failed at HDFS side but as there file had all edits
> synced, we continue.
> New WAL file is created and old rolled.
> This old WAL file got archived to oldWAL
> {code}
> 2021-05-13 13:38:46.000 Riding over failed WAL close of
> hdfs://xxx/WALs/xxx,16020,1620828102351/xxx%2C16020%2C1620828102351.1620910673678,
> cause="Unexpected EOF while trying to read response from server", errors=1;
> THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK
> 2021-05-13 13:38:46.000 Rolled WAL
> /xx/WALs/xxx,16020,1620828102351/xxx%2C16020%2C1620828102351.1620910673678
> with entries=0, filesize=90 B; new WAL
> /xx/WALs/xxx,16020,1620828102351/xxx%2C16020%2C1620828102351.1620913126549
> 2021-05-13 13:38:46.000Archiving
> hdfs://xxx/WALs/xxx,16020,1620828102351/xxx%2C16020%2C1620828102351.1620910673678
> to hdfs://xxx/oldWALs/xxxt%2C16020%2C1620828102351.1620910673678
> 2021-05-13 13:38:46.000 Log
> hdfs://xxx/WALs/xxx,16020,1620828102351/xxx%2C16020%2C1620828102351.1620910673678
> was moved to hdfs://xxx/oldWALs/xxx%2C16020%2C1620828102351.1620910673678
> {code}
> As there was move of file, the WALEntryStream got IOE and we will recreate
> the stream .
> {code}
> ReplicationSourceWALReader#run
> while (isReaderRunning()) {
> try {
> entryStream =
> new WALEntryStream(logQueue, conf, currentPosition,
> source.getWALFileLengthProvider(),
> source.getServerWALsBelongTo(), source.getSourceMetrics(),
> walGroupId);
> while (isReaderRunning()) {
> ...
> ...
> } catch (IOException e) { // stream related
> if (handleEofException(e, batch)) {
> sleepMultiplier = 1;
> } else {
> LOG.warn("Failed to read stream of replication entries", e);
> if (sleepMultiplier < maxRetriesMultiplier) {
> sleepMultiplier++;
> }
> Threads.sleep(sleepForRetries * sleepMultiplier);
> }
> }
> {code}
> eofAutoRecovery is turned off anyways. So it will go to outer while loop and
> create new WALEntryStream object
> Then we do readWALEntries
> {code}
> protected WALEntryBatch readWALEntries(WALEntryStream entryStream,
> WALEntryBatch batch) throws IOException, InterruptedException {
> Path currentPath = entryStream.getCurrentPath();
> if (!entryStream.hasNext()) {
> {code}
> Here the currentPath will be still null.
> WALEntryStream#hasNext -> tryAdvanceEntry -> checkReader -> openNextLog
> {code}
> private boolean openNextLog() throws IOException {
> PriorityBlockingQueue queue = logQueue.getQueue(walGroupId);
> Path nextPath = queue.peek();
> if
[jira] [Commented] (HBASE-25899) Improve efficiency of SnapshotHFileCleaner
[
https://issues.apache.org/jira/browse/HBASE-25899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350777#comment-17350777
]
Hudson commented on HBASE-25899:
Results for branch branch-2
[build #259 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/259/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/259/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/259//console].
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/259/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/259/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/259//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Improve efficiency of SnapshotHFileCleaner
> --
>
> Key: HBASE-25899
> URL: https://issues.apache.org/jira/browse/HBASE-25899
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 3.0.0-alpha-1, 2.0.0
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
> Attachments: 78631.jstack, cleaner-result.png
>
>
> We have met same problems of thousands threads in HBASE-22867, but after this
> issue, the cleaner becomes more inefficient.
> From the jstack we can see that most dir-scan threads are blocked at
> SnapshotHFileCleaner#getDeletableFiles,
> {code:java}
> "dir-scan-pool-19" #694 daemon prio=5 os_prio=0 tid=0x02ab1800
> nid=0x26a7e waiting for monitor entry [0x7fb0a9913000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner.getDeletableFiles(SnapshotHFileCleaner.java:74)
> - waiting to lock <0x7fb148737048> (a
> org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:498)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.lambda$traverseAndDelete$1(CleanerChore.java:246)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore$$Lambda$41/1187372779.act(Unknown
> Source)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.deleteAction(CleanerChore.java:358)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.traverseAndDelete(CleanerChore.java:246)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.lambda$null$2(CleanerChore.java:255)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore$$Lambda$38/2003131501.run(Unknown
> Source)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){code}
> and all the HFileCleaner threads are waiting at the delete tasks queue,
> {code:java}
> "gha-data-hbase0002:16000.activeMasterManager-HFileCleaner.large.2-1621210982419"
> #358 daemon prio=5 os_prio=0 tid=0x7fb967fc nid=0x266f2 waiting on
> condition [0x7fb0c57d6000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7fb1486db9f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:106)
> at
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:264)
> at
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner$1.run(HFileCleaner.java:233)
> {code}
> So it's need to increase the speed of
[jira] [Commented] (HBASE-25873) Refactor and cleanup the code for CostFunction
[
https://issues.apache.org/jira/browse/HBASE-25873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350776#comment-17350776
]
Hudson commented on HBASE-25873:
Results for branch branch-2
[build #259 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/259/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/259/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/259//console].
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/259/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/259/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/259//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Refactor and cleanup the code for CostFunction
> --
>
> Key: HBASE-25873
> URL: https://issues.apache.org/jira/browse/HBASE-25873
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, Performance
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
> Attachments: profile_balance_cluster.html
>
>
> For later performance improvements.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] sunhelly commented on a change in pull request #3297: HBASE-25905 Limit the shutdown time of WAL
sunhelly commented on a change in pull request #3297: URL: https://github.com/apache/hbase/pull/3297#discussion_r638437219 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java ## @@ -145,6 +148,10 @@ public static final String RING_BUFFER_SLOT_COUNT = "hbase.regionserver.wal.disruptor.event.count"; + public static final String WAL_SHUTDOWN_WAIT_TIMEOUT_MS = +"hbase.wal.shutdown.wait.timeout.ms"; + public static final int DEFAULT_WAL_SHUTDOWN_WAIT_TIMEOUT_MS = 15000; Review comment: Thanks, I'll change it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3303: HBASE-25906 UI of master-status to show recent history of balancer de…
Apache-HBase commented on pull request #3303: URL: https://github.com/apache/hbase/pull/3303#issuecomment-847509014 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 1s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 15s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 10m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3303/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3303 | | Optional Tests | dupname asflicense javac | | uname | Linux 31845c875fd6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 073f23e2bd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 79 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3303/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] GeorryHuang opened a new pull request #3303: HBASE-25906 UI of master-status to show recent history of balancer de…
GeorryHuang opened a new pull request #3303: URL: https://github.com/apache/hbase/pull/3303 …sicion -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3206: HBASE-25818 Move StochasticLoadBalancer to hbase-balancer module
Apache-HBase commented on pull request #3206: URL: https://github.com/apache/hbase/pull/3206#issuecomment-847503489 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 2s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 9s | master passed | | +1 :green_heart: | compile | 3m 49s | master passed | | +1 :green_heart: | checkstyle | 1m 22s | master passed | | +1 :green_heart: | spotbugs | 2m 40s | master passed | | -0 :warning: | patch | 2m 23s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 59s | the patch passed | | +1 :green_heart: | compile | 3m 46s | the patch passed | | -0 :warning: | javac | 3m 18s | hbase-server generated 3 new + 190 unchanged - 3 fixed = 193 total (was 193) | | +1 :green_heart: | checkstyle | 0m 12s | The patch passed checkstyle in hbase-balancer | | +1 :green_heart: | checkstyle | 1m 8s | hbase-server: The patch generated 0 new + 2 unchanged - 4 fixed = 2 total (was 6) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 55s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 7s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | | | | 54m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3206 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 2831cf6b07ae 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21aa553bc1 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/6/artifact/yetus-general-check/output/diff-compile-javac-hbase-server.txt | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/6/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25914) Provide slow/large logs on RegionServer UI
Zhuoyue Huang created HBASE-25914: - Summary: Provide slow/large logs on RegionServer UI Key: HBASE-25914 URL: https://issues.apache.org/jira/browse/HBASE-25914 Project: HBase Issue Type: Improvement Components: regionserver, UI Affects Versions: 3.0.0-alpha-1, 2.5.0 Reporter: Zhuoyue Huang Assignee: Zhuoyue Huang Pulling slow/large log from in-memory queues on RegionServer then display details info in RegionServer status UI -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[ https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350761#comment-17350761 ] Andrew Kyle Purtell commented on HBASE-24440: - One early A/B test worth doing is A=spin wait, B=thread yield(). > Prevent temporal misordering on timescales smaller than one clock tick > -- > > Key: HBASE-24440 > URL: https://issues.apache.org/jira/browse/HBASE-24440 > Project: HBase > Issue Type: Brainstorming >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > When mutations are sent to the servers without a timestamp explicitly > assigned by the client the server will substitute the current wall clock > time. There are edge cases where it is at least theoretically possible for > more than one mutation to be committed to a given row within the same clock > tick. When this happens we have to track and preserve the ordering of these > mutations in some other way besides the timestamp component of the key. Let > me bypass most discussion here by noting that whether we do this or not, we > do not pass such ordering information in the cross cluster replication > protocol. We also have interesting edge cases regarding key type precedence > when mutations arrive "simultaneously": we sort deletes ahead of puts. This, > especially in the presence of replication, can lead to visible anomalies for > clients able to interact with both source and sink. > There is a simple solution that removes the possibility that these edge cases > can occur: > We can detect, when we are about to commit a mutation to a row, if we have > already committed a mutation to this same row in the current clock tick. > Occurrences of this condition will be rare. We are already tracking current > time. We have to know this in order to assign the timestamp. Where this > becomes interesting is how we might track the last commit time per row. > Making the detection of this case efficient for the normal code path is the > bulk of the challenge. One option is to keep track of the last locked time > for row locks. (Todo: How would we track and garbage collect this efficiently > and correctly. Not the ideal option.) We might also do this tracking somehow > via the memstore. (At least in this case the lifetime and distribution of in > memory row state, including the proposed timestamps, would align.) Assuming > we can efficiently know if we are about to commit twice to the same row > within a single clock tick, we would simply sleep/yield the current thread > until the clock ticks over, and then proceed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[ https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350759#comment-17350759 ] Andrew Kyle Purtell commented on HBASE-24440: - That worst case could be possible but it’s more likely there will be an interleaving of handlers as they yield. Yielding may not be necessary at all if the clock has already ticked over, which should be pretty likely on a typical Linux server. I say we start simple and then consider clever if and only if there’s a real issue revealed in the performance data. > Prevent temporal misordering on timescales smaller than one clock tick > -- > > Key: HBASE-24440 > URL: https://issues.apache.org/jira/browse/HBASE-24440 > Project: HBase > Issue Type: Brainstorming >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > When mutations are sent to the servers without a timestamp explicitly > assigned by the client the server will substitute the current wall clock > time. There are edge cases where it is at least theoretically possible for > more than one mutation to be committed to a given row within the same clock > tick. When this happens we have to track and preserve the ordering of these > mutations in some other way besides the timestamp component of the key. Let > me bypass most discussion here by noting that whether we do this or not, we > do not pass such ordering information in the cross cluster replication > protocol. We also have interesting edge cases regarding key type precedence > when mutations arrive "simultaneously": we sort deletes ahead of puts. This, > especially in the presence of replication, can lead to visible anomalies for > clients able to interact with both source and sink. > There is a simple solution that removes the possibility that these edge cases > can occur: > We can detect, when we are about to commit a mutation to a row, if we have > already committed a mutation to this same row in the current clock tick. > Occurrences of this condition will be rare. We are already tracking current > time. We have to know this in order to assign the timestamp. Where this > becomes interesting is how we might track the last commit time per row. > Making the detection of this case efficient for the normal code path is the > bulk of the challenge. One option is to keep track of the last locked time > for row locks. (Todo: How would we track and garbage collect this efficiently > and correctly. Not the ideal option.) We might also do this tracking somehow > via the memstore. (At least in this case the lifetime and distribution of in > memory row state, including the proposed timestamps, would align.) Assuming > we can efficiently know if we are about to commit twice to the same row > within a single clock tick, we would simply sleep/yield the current thread > until the clock ticks over, and then proceed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25906) UI of master-status to show recent history of balancer desicion
[ https://issues.apache.org/jira/browse/HBASE-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350756#comment-17350756 ] Bharath Vissapragada commented on HBASE-25906: -- Thanks for the patch, useful addition. > Would you also like to work on providing slow/large logs on RegionServer UI Agree this will be helpful, we plan to add more RS side debugging information to this buffer, so making it accessible via RS web UI is nice too. > UI of master-status to show recent history of balancer desicion > --- > > Key: HBASE-25906 > URL: https://issues.apache.org/jira/browse/HBASE-25906 > Project: HBase > Issue Type: Improvement > Components: Balancer, master, UI >Affects Versions: 3.0.0-alpha-1, 2.5.0 >Reporter: Zhuoyue Huang >Assignee: Zhuoyue Huang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4 > > Attachments: screenshot-1.png > > > HBASE-24528 provide ‘Balancer Decision’ to display the history that includes > decision factor details and weights and costs while running balancer. > This issue implement 'Balancer Decision' UI web page -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3302: HBASE-25911 Replace calls to System.currentTimeMillis with EnvironmentEdgeManager.currentTime
Apache-HBase commented on pull request #3302: URL: https://github.com/apache/hbase/pull/3302#issuecomment-847491257 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 5s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 52s | master passed | | +1 :green_heart: | compile | 12m 19s | master passed | | +1 :green_heart: | checkstyle | 6m 12s | master passed | | +1 :green_heart: | spotbugs | 11m 30s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 34s | the patch passed | | +1 :green_heart: | compile | 12m 23s | the patch passed | | -0 :warning: | javac | 0m 48s | hbase-mapreduce generated 1 new + 197 unchanged - 1 fixed = 198 total (was 198) | | -0 :warning: | checkstyle | 0m 28s | hbase-client: The patch generated 2 new + 12 unchanged - 0 fixed = 14 total (was 12) | | -0 :warning: | checkstyle | 1m 37s | hbase-server: The patch generated 19 new + 1257 unchanged - 5 fixed = 1276 total (was 1262) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 5s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 14m 25s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 3m 4s | The patch does not generate ASF License warnings. | | | | 106m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3302 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 4ebfc877c1be 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 21aa553bc1 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/1/artifact/yetus-general-check/output/diff-compile-javac-hbase-mapreduce.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-client.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-zookeeper hbase-balancer hbase-http hbase-procedure hbase-server hbase-mapreduce hbase-thrift hbase-endpoint hbase-backup hbase-it hbase-rest hbase-examples hbase-hbtop U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3302/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25906) UI of master-status to show recent history of balancer desicion
[ https://issues.apache.org/jira/browse/HBASE-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350750#comment-17350750 ] Zhuoyue Huang commented on HBASE-25906: --- No problem, I would like to provide UI on RegionServer > UI of master-status to show recent history of balancer desicion > --- > > Key: HBASE-25906 > URL: https://issues.apache.org/jira/browse/HBASE-25906 > Project: HBase > Issue Type: Improvement > Components: Balancer, master, UI >Affects Versions: 3.0.0-alpha-1, 2.5.0 >Reporter: Zhuoyue Huang >Assignee: Zhuoyue Huang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4 > > Attachments: screenshot-1.png > > > HBASE-24528 provide ‘Balancer Decision’ to display the history that includes > decision factor details and weights and costs while running balancer. > This issue implement 'Balancer Decision' UI web page -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25906) UI of master-status to show recent history of balancer desicion
[ https://issues.apache.org/jira/browse/HBASE-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350751#comment-17350751 ] Zhuoyue Huang commented on HBASE-25906: --- [~zhangduo] [~vjasani] Thanks for reviewing. > UI of master-status to show recent history of balancer desicion > --- > > Key: HBASE-25906 > URL: https://issues.apache.org/jira/browse/HBASE-25906 > Project: HBase > Issue Type: Improvement > Components: Balancer, master, UI >Affects Versions: 3.0.0-alpha-1, 2.5.0 >Reporter: Zhuoyue Huang >Assignee: Zhuoyue Huang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4 > > Attachments: screenshot-1.png > > > HBASE-24528 provide ‘Balancer Decision’ to display the history that includes > decision factor details and weights and costs while running balancer. > This issue implement 'Balancer Decision' UI web page -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25906) UI of master-status to show recent history of balancer desicion
[ https://issues.apache.org/jira/browse/HBASE-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350749#comment-17350749 ] Zhuoyue Huang commented on HBASE-25906: --- [~vjasani] Thanks! > UI of master-status to show recent history of balancer desicion > --- > > Key: HBASE-25906 > URL: https://issues.apache.org/jira/browse/HBASE-25906 > Project: HBase > Issue Type: Improvement > Components: Balancer, master, UI >Affects Versions: 3.0.0-alpha-1, 2.5.0 >Reporter: Zhuoyue Huang >Assignee: Zhuoyue Huang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4 > > Attachments: screenshot-1.png > > > HBASE-24528 provide ‘Balancer Decision’ to display the history that includes > decision factor details and weights and costs while running balancer. > This issue implement 'Balancer Decision' UI web page -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25666) Explain why balancer is skipping runs
[ https://issues.apache.org/jira/browse/HBASE-25666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350747#comment-17350747 ] Zhuoyue Huang commented on HBASE-25666: --- If there are no other suggestions, this issue could be resolved. > Explain why balancer is skipping runs > - > > Key: HBASE-25666 > URL: https://issues.apache.org/jira/browse/HBASE-25666 > Project: HBase > Issue Type: Improvement > Components: Balancer, master, UI >Reporter: Clara Xiong >Assignee: Zhuoyue Huang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4 > > > Balancer needs to find the balance of keeping cluster balanced and stable. > There is a configuration minCostNeedBalance as the threshold of imbalance to > try to balance. Since we use a single score to combine all the factors we > consider, it is hard for operation to understand why balancer is "stuck". We > should add to master-status UI to show balancer is skipping runs and explain > all factors considered, such as the weight and cost of all cost functions. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25666) Explain why balancer is skipping runs
[ https://issues.apache.org/jira/browse/HBASE-25666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhuoyue Huang updated HBASE-25666: -- Fix Version/s: 2.4.4 2.5.0 3.0.0-alpha-1 > Explain why balancer is skipping runs > - > > Key: HBASE-25666 > URL: https://issues.apache.org/jira/browse/HBASE-25666 > Project: HBase > Issue Type: Improvement > Components: Balancer, master, UI >Reporter: Clara Xiong >Assignee: Zhuoyue Huang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4 > > > Balancer needs to find the balance of keeping cluster balanced and stable. > There is a configuration minCostNeedBalance as the threshold of imbalance to > try to balance. Since we use a single score to combine all the factors we > consider, it is hard for operation to understand why balancer is "stuck". We > should add to master-status UI to show balancer is skipping runs and explain > all factors considered, such as the weight and cost of all cost functions. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[ https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350742#comment-17350742 ] Bharath Vissapragada commented on HBASE-24440: -- Ok, looking forward to the performance analysis. The reason I thought it could be a concern is because if 16 RPC threads all call Region#append() within the same tick, in the worst case the last append can wait for 16 ticks (or I misunderstood something). Anyway agree this can wait until we have actual data and the patch. > Prevent temporal misordering on timescales smaller than one clock tick > -- > > Key: HBASE-24440 > URL: https://issues.apache.org/jira/browse/HBASE-24440 > Project: HBase > Issue Type: Brainstorming >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > When mutations are sent to the servers without a timestamp explicitly > assigned by the client the server will substitute the current wall clock > time. There are edge cases where it is at least theoretically possible for > more than one mutation to be committed to a given row within the same clock > tick. When this happens we have to track and preserve the ordering of these > mutations in some other way besides the timestamp component of the key. Let > me bypass most discussion here by noting that whether we do this or not, we > do not pass such ordering information in the cross cluster replication > protocol. We also have interesting edge cases regarding key type precedence > when mutations arrive "simultaneously": we sort deletes ahead of puts. This, > especially in the presence of replication, can lead to visible anomalies for > clients able to interact with both source and sink. > There is a simple solution that removes the possibility that these edge cases > can occur: > We can detect, when we are about to commit a mutation to a row, if we have > already committed a mutation to this same row in the current clock tick. > Occurrences of this condition will be rare. We are already tracking current > time. We have to know this in order to assign the timestamp. Where this > becomes interesting is how we might track the last commit time per row. > Making the detection of this case efficient for the normal code path is the > bulk of the challenge. One option is to keep track of the last locked time > for row locks. (Todo: How would we track and garbage collect this efficiently > and correctly. Not the ideal option.) We might also do this tracking somehow > via the memstore. (At least in this case the lifetime and distribution of in > memory row state, including the proposed timestamps, would align.) Assuming > we can efficiently know if we are about to commit twice to the same row > within a single clock tick, we would simply sleep/yield the current thread > until the clock ticks over, and then proceed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[ https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350736#comment-17350736 ] Andrew Kyle Purtell commented on HBASE-24440: - Also as I said for batches the advancing time is taken only once. If your metrics ingesting application is fine with approximate time stamps then it is obviously fine with submitting mutations in batch where all the cells in the batch will get the same stamp (this is covered in the issue description) and there will be exactly one call to getTimeAdvancing for the entire batch so I wonder if the difference can be reliably measured it should be so small. > Prevent temporal misordering on timescales smaller than one clock tick > -- > > Key: HBASE-24440 > URL: https://issues.apache.org/jira/browse/HBASE-24440 > Project: HBase > Issue Type: Brainstorming >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > When mutations are sent to the servers without a timestamp explicitly > assigned by the client the server will substitute the current wall clock > time. There are edge cases where it is at least theoretically possible for > more than one mutation to be committed to a given row within the same clock > tick. When this happens we have to track and preserve the ordering of these > mutations in some other way besides the timestamp component of the key. Let > me bypass most discussion here by noting that whether we do this or not, we > do not pass such ordering information in the cross cluster replication > protocol. We also have interesting edge cases regarding key type precedence > when mutations arrive "simultaneously": we sort deletes ahead of puts. This, > especially in the presence of replication, can lead to visible anomalies for > clients able to interact with both source and sink. > There is a simple solution that removes the possibility that these edge cases > can occur: > We can detect, when we are about to commit a mutation to a row, if we have > already committed a mutation to this same row in the current clock tick. > Occurrences of this condition will be rare. We are already tracking current > time. We have to know this in order to assign the timestamp. Where this > becomes interesting is how we might track the last commit time per row. > Making the detection of this case efficient for the normal code path is the > bulk of the challenge. One option is to keep track of the last locked time > for row locks. (Todo: How would we track and garbage collect this efficiently > and correctly. Not the ideal option.) We might also do this tracking somehow > via the memstore. (At least in this case the lifetime and distribution of in > memory row state, including the proposed timestamps, would align.) Assuming > we can efficiently know if we are about to commit twice to the same row > within a single clock tick, we would simply sleep/yield the current thread > until the clock ticks over, and then proceed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[ https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350729#comment-17350729 ] Andrew Kyle Purtell commented on HBASE-24440: - No this will not be pluggable. It is a new invariant, as proposed. We have a ton of configuration settings as is and especially for something fundamental like this the confusion factor will be amplified. Because everyone will be using EnvironmentEdge to get current time it is trivial to track last retrieved time. We will spin if and only if getTimeAdvancing is called. That will be called only one time per mutation request handling. The overhead is expected to be hard to measure. (But we will measure it.) If the overhead turns out to be larger than expected then we can have a discussion about optimization. Otherwise that discussion is premature. > Prevent temporal misordering on timescales smaller than one clock tick > -- > > Key: HBASE-24440 > URL: https://issues.apache.org/jira/browse/HBASE-24440 > Project: HBase > Issue Type: Brainstorming >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > When mutations are sent to the servers without a timestamp explicitly > assigned by the client the server will substitute the current wall clock > time. There are edge cases where it is at least theoretically possible for > more than one mutation to be committed to a given row within the same clock > tick. When this happens we have to track and preserve the ordering of these > mutations in some other way besides the timestamp component of the key. Let > me bypass most discussion here by noting that whether we do this or not, we > do not pass such ordering information in the cross cluster replication > protocol. We also have interesting edge cases regarding key type precedence > when mutations arrive "simultaneously": we sort deletes ahead of puts. This, > especially in the presence of replication, can lead to visible anomalies for > clients able to interact with both source and sink. > There is a simple solution that removes the possibility that these edge cases > can occur: > We can detect, when we are about to commit a mutation to a row, if we have > already committed a mutation to this same row in the current clock tick. > Occurrences of this condition will be rare. We are already tracking current > time. We have to know this in order to assign the timestamp. Where this > becomes interesting is how we might track the last commit time per row. > Making the detection of this case efficient for the normal code path is the > bulk of the challenge. One option is to keep track of the last locked time > for row locks. (Todo: How would we track and garbage collect this efficiently > and correctly. Not the ideal option.) We might also do this tracking somehow > via the memstore. (At least in this case the lifetime and distribution of in > memory row state, including the proposed timestamps, would align.) Assuming > we can efficiently know if we are about to commit twice to the same row > within a single clock tick, we would simply sleep/yield the current thread > until the clock ticks over, and then proceed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25894) Improve the performance for region load and region count related cost functions
[ https://issues.apache.org/jira/browse/HBASE-25894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-25894: -- Release Note: In CostFromRegionLoadFunction, now we will only recompute the cost for a given region server in regionMoved function, instead of computing all the costs every time. Introduced a DoubleArrayCost for better abstraction, and also try to only compute the final cost on demand as the computation is also a bit expensive. > Improve the performance for region load and region count related cost > functions > --- > > Key: HBASE-25894 > URL: https://issues.apache.org/jira/browse/HBASE-25894 > Project: HBase > Issue Type: Sub-task > Components: Balancer, Performance >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > For a large cluster, we have a lot of regions, so computing the whole cost > will be expensive. We should try to remove the unnecessary calculation as > much as possible. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[ https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350726#comment-17350726 ] Bharath Vissapragada commented on HBASE-24440: -- > new EnvironmentEdge#currentTimeAdvancing which ensures that when the current > time is returned, it is the current time in a different clock tick from the > last time the EnvironmentEdge was used to get the current time. Curious how you plan to achieve this, is the plan to sleep for a clock tick between two consecutive invocations (if current_ts == last_read_ts) or something more fancy? Is the plan to make this pluggable at a table/namespace scope or for the service? Reason I ask this because the performance penalty with delayed clock ticks may not be acceptable for throughput favoring applications like metrics ingestion that are fine with approximate timestamps. Also per your design, it appears the scope of uniqueness is RS so we have a total order of all mutations for a given RS (across regions). In most cases we are ok with a partial order (within a region) since that is where the order of mutations matters, so we can have optimizations like per region clock instances that guarantee this partial order and also achieves the pluggability part (thinking out loud). WDYT. > Prevent temporal misordering on timescales smaller than one clock tick > -- > > Key: HBASE-24440 > URL: https://issues.apache.org/jira/browse/HBASE-24440 > Project: HBase > Issue Type: Brainstorming >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > When mutations are sent to the servers without a timestamp explicitly > assigned by the client the server will substitute the current wall clock > time. There are edge cases where it is at least theoretically possible for > more than one mutation to be committed to a given row within the same clock > tick. When this happens we have to track and preserve the ordering of these > mutations in some other way besides the timestamp component of the key. Let > me bypass most discussion here by noting that whether we do this or not, we > do not pass such ordering information in the cross cluster replication > protocol. We also have interesting edge cases regarding key type precedence > when mutations arrive "simultaneously": we sort deletes ahead of puts. This, > especially in the presence of replication, can lead to visible anomalies for > clients able to interact with both source and sink. > There is a simple solution that removes the possibility that these edge cases > can occur: > We can detect, when we are about to commit a mutation to a row, if we have > already committed a mutation to this same row in the current clock tick. > Occurrences of this condition will be rare. We are already tracking current > time. We have to know this in order to assign the timestamp. Where this > becomes interesting is how we might track the last commit time per row. > Making the detection of this case efficient for the normal code path is the > bulk of the challenge. One option is to keep track of the last locked time > for row locks. (Todo: How would we track and garbage collect this efficiently > and correctly. Not the ideal option.) We might also do this tracking somehow > via the memstore. (At least in this case the lifetime and distribution of in > memory row state, including the proposed timestamps, would align.) Assuming > we can efficiently know if we are about to commit twice to the same row > within a single clock tick, we would simply sleep/yield the current thread > until the clock ticks over, and then proceed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25913) Introduce EnvironmentEdge.currentTimeAdvancing
[
https://issues.apache.org/jira/browse/HBASE-25913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350718#comment-17350718
]
Andrew Kyle Purtell edited comment on HBASE-25913 at 5/25/21, 1:46 AM:
---
{quote}Clients must simply not submit mutations that must be committed with
guaranteed distinct timestamps in the same batch. Easy to understand, easy to
document, and it aligns with our design philosophy of the client knows best.
{quote}
[~gjacoby] [~kadir] [~larsh] I assume this will be fine for Phoenix but let me
know if there is a complication that will make this difficult.
The rule will be: Every mutation request submitting a call with a wildcard
timestamp (Long.MAX) will get a guaranteed distinct timestamp on the server
side. Clients must submit cells with wildcard timestamps, where guaranteed
distinct timestamps are required, in separate requests to ensure this
invariant. (A batch mutation will be considered one request, therefore each
cell within the batch with a wildcard timestamp will have the same value
substituted for same.)
was (Author: apurtell):
{quote}Clients must simply not submit mutations that must be committed with
guaranteed distinct timestamps in the same batch. Easy to understand, easy to
document, and it aligns with our design philosophy of the client knows best.
{quote}
[~gjacoby] [~kadir] [~larsh] I assume this will be fine for Phoenix but let me
know if there is a complication that will make this difficult.
The rule will be: Every mutation request submitting a call with a wildcard
timestamp (Long.MAX) will get a guaranteed distinct timestamp on the server
side. Clients must submit values that require guaranteed distinct timestamps in
separate requests to ensure this invariant. (A batch mutation will be
considered one request, therefore each cell within the batch with a wildcard
timestamp will have the same value substituted for same.)
> Introduce EnvironmentEdge.currentTimeAdvancing
> --
>
> Key: HBASE-25913
> URL: https://issues.apache.org/jira/browse/HBASE-25913
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that
> when the current time is returned, it is the current time in a different
> clock tick from the last time the {{EnvironmentEdge}} was used to get the
> current time.
> When processing mutations we substitute the {{Long.MAX_VALUE}} timestamp
> placeholder with a real placeholder just before committing the mutation. The
> current code gets the current time for timestamp substitution while under row
> lock and mvcc. We will simply use {{EnvironmentEdge#currentTimeAdvancing}}
> instead of {{EnvironmentEdge#currentTime}} at this point in the code to
> ensure we have seen the clock tick over. When processing a batch of mutations
> (doMiniBatchMutation etc) we will call {{currentTimeAdvancing}} only once.
> This means the client cannot bundle cells with wildcard timestamps into a
> batch where those cells must be committed with different timestamps. Clients
> must simply not submit mutations that must be committed with guaranteed
> distinct timestamps in the same batch. Easy to understand, easy to document,
> and it aligns with our design philosophy of the client knows best.
> It is not required to handle batches as proposed. We could guarantee a
> distinct timestamp for every mutation in a batch. Count the number of
> mutations, call this M. Acquire all row locks and get the current time. Then,
> wait for at least M milliseconds. Then, set the first mutation timestamp with
> this value and increment by 1 for all remaining. Then, do the rest of
> mutation processing as normal. I don't think this extra waiting to reserve
> the range of timestamps is necessary. See reasoning in above paragraph.
> Mentioned here for sake of discussion.
> It will be fine to continue to use {{EnvironmentEdge#currentTime}} everywhere
> else. In this way we will only potentially spin wait where it matters, and
> won't suffer serious overheads during batch processing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25913) Introduce EnvironmentEdge.currentTimeAdvancing
[
https://issues.apache.org/jira/browse/HBASE-25913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350718#comment-17350718
]
Andrew Kyle Purtell edited comment on HBASE-25913 at 5/25/21, 1:45 AM:
---
{quote}Clients must simply not submit mutations that must be committed with
guaranteed distinct timestamps in the same batch. Easy to understand, easy to
document, and it aligns with our design philosophy of the client knows best.
{quote}
[~gjacoby] [~kadir] [~larsh] I assume this will be fine for Phoenix but let me
know if there is a complication that will make this difficult.
The rule will be: Every mutation request submitting a call with a wildcard
timestamp (Long.MAX) will get a guaranteed distinct timestamp on the server
side. Clients must submit values that require guaranteed distinct timestamps in
separate requests to ensure this invariant. (A batch mutation will be
considered one request, therefore each cell within the batch with a wildcard
timestamp will have the same value substituted for same.)
was (Author: apurtell):
{quote}Clients must simply not submit mutations that must be committed with
guaranteed distinct timestamps in the same batch. Easy to understand, easy to
document, and it aligns with our design philosophy of the client knows best.
{quote}
[~gjacoby] [~kadir] [~larsh] I assume this will be fine for Phoenix but let me
know if there is a complication that will make this difficult.
> Introduce EnvironmentEdge.currentTimeAdvancing
> --
>
> Key: HBASE-25913
> URL: https://issues.apache.org/jira/browse/HBASE-25913
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that
> when the current time is returned, it is the current time in a different
> clock tick from the last time the {{EnvironmentEdge}} was used to get the
> current time.
> When processing mutations we substitute the {{Long.MAX_VALUE}} timestamp
> placeholder with a real placeholder just before committing the mutation. The
> current code gets the current time for timestamp substitution while under row
> lock and mvcc. We will simply use {{EnvironmentEdge#currentTimeAdvancing}}
> instead of {{EnvironmentEdge#currentTime}} at this point in the code to
> ensure we have seen the clock tick over. When processing a batch of mutations
> (doMiniBatchMutation etc) we will call {{currentTimeAdvancing}} only once.
> This means the client cannot bundle cells with wildcard timestamps into a
> batch where those cells must be committed with different timestamps. Clients
> must simply not submit mutations that must be committed with guaranteed
> distinct timestamps in the same batch. Easy to understand, easy to document,
> and it aligns with our design philosophy of the client knows best.
> It is not required to handle batches as proposed. We could guarantee a
> distinct timestamp for every mutation in a batch. Count the number of
> mutations, call this M. Acquire all row locks and get the current time. Then,
> wait for at least M milliseconds. Then, set the first mutation timestamp with
> this value and increment by 1 for all remaining. Then, do the rest of
> mutation processing as normal. I don't think this extra waiting to reserve
> the range of timestamps is necessary. See reasoning in above paragraph.
> Mentioned here for sake of discussion.
> It will be fine to continue to use {{EnvironmentEdge#currentTime}} everywhere
> else. In this way we will only potentially spin wait where it matters, and
> won't suffer serious overheads during batch processing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25913) Introduce EnvironmentEdge.currentTimeAdvancing
[
https://issues.apache.org/jira/browse/HBASE-25913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350718#comment-17350718
]
Andrew Kyle Purtell commented on HBASE-25913:
-
{quote}Clients must simply not submit mutations that must be committed with
guaranteed distinct timestamps in the same batch. Easy to understand, easy to
document, and it aligns with our design philosophy of the client knows best.
{quote}
[~gjacoby] [~kadir] [~larsh] I assume this will be fine for Phoenix but let me
know if there is a complication that will make this difficult.
> Introduce EnvironmentEdge.currentTimeAdvancing
> --
>
> Key: HBASE-25913
> URL: https://issues.apache.org/jira/browse/HBASE-25913
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that
> when the current time is returned, it is the current time in a different
> clock tick from the last time the {{EnvironmentEdge}} was used to get the
> current time.
> When processing mutations we substitute the {{Long.MAX_VALUE}} timestamp
> placeholder with a real placeholder just before committing the mutation. The
> current code gets the current time for timestamp substitution while under row
> lock and mvcc. We will simply use {{EnvironmentEdge#currentTimeAdvancing}}
> instead of {{EnvironmentEdge#currentTime}} at this point in the code to
> ensure we have seen the clock tick over. When processing a batch of mutations
> (doMiniBatchMutation etc) we will call {{currentTimeAdvancing}} only once.
> This means the client cannot bundle cells with wildcard timestamps into a
> batch where those cells must be committed with different timestamps. Clients
> must simply not submit mutations that must be committed with guaranteed
> distinct timestamps in the same batch. Easy to understand, easy to document,
> and it aligns with our design philosophy of the client knows best.
> It is not required to handle batches as proposed. We could guarantee a
> distinct timestamp for every mutation in a batch. Count the number of
> mutations, call this M. Acquire all row locks and get the current time. Then,
> wait for at least M milliseconds. Then, set the first mutation timestamp with
> this value and increment by 1 for all remaining. Then, do the rest of
> mutation processing as normal. I don't think this extra waiting to reserve
> the range of timestamps is necessary. See reasoning in above paragraph.
> Mentioned here for sake of discussion.
> It will be fine to continue to use {{EnvironmentEdge#currentTime}} everywhere
> else. In this way we will only potentially spin wait where it matters, and
> won't suffer serious overheads during batch processing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[
https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350711#comment-17350711
]
Andrew Kyle Purtell edited comment on HBASE-24440 at 5/25/21, 1:39 AM:
---
I'm considering something simple:
* Ensure our discipline in using {{EnvironmentEdge#currentTime}} instead of
{{System#currentTimeMillis}} (HBASE-25912)
* Fix current issues (HBASE-25911)
* Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that
when the current time is returned, it is the current time in a different clock
tick from the last time the {{EnvironmentEdge}} was used to get the current
time. Use {{EnvironmentEdge#currentTimeAdvancing}} wherever we go to substitute
a {{Long.MAX_VALUE}} timestamp placeholder with a real timestamp when
processing mutations. See HBASE-25913 for more details.
was (Author: apurtell):
I'm considering something simple:
* Ensure our discipline in using {{EnvironmentEdge#currentTime}} instead of
{{System#currentTimeMillis}} (HBASE-25912)
* Fix current issues (HBASE-25911)
* Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that
when the current time is returned, it is the current time in a different clock
tick from the last time the {{EnvironmentEdge}} was used to get the current
time. Use {{EnvironmentEdge#currentTimeAdvancing}} wherever we go to substitute
a {{Long.MAX_VALUE}} timestamp placeholder with a real placeholder. See
HBASE-25913 for more details.
> Prevent temporal misordering on timescales smaller than one clock tick
> --
>
> Key: HBASE-24440
> URL: https://issues.apache.org/jira/browse/HBASE-24440
> Project: HBase
> Issue Type: Brainstorming
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When mutations are sent to the servers without a timestamp explicitly
> assigned by the client the server will substitute the current wall clock
> time. There are edge cases where it is at least theoretically possible for
> more than one mutation to be committed to a given row within the same clock
> tick. When this happens we have to track and preserve the ordering of these
> mutations in some other way besides the timestamp component of the key. Let
> me bypass most discussion here by noting that whether we do this or not, we
> do not pass such ordering information in the cross cluster replication
> protocol. We also have interesting edge cases regarding key type precedence
> when mutations arrive "simultaneously": we sort deletes ahead of puts. This,
> especially in the presence of replication, can lead to visible anomalies for
> clients able to interact with both source and sink.
> There is a simple solution that removes the possibility that these edge cases
> can occur:
> We can detect, when we are about to commit a mutation to a row, if we have
> already committed a mutation to this same row in the current clock tick.
> Occurrences of this condition will be rare. We are already tracking current
> time. We have to know this in order to assign the timestamp. Where this
> becomes interesting is how we might track the last commit time per row.
> Making the detection of this case efficient for the normal code path is the
> bulk of the challenge. One option is to keep track of the last locked time
> for row locks. (Todo: How would we track and garbage collect this efficiently
> and correctly. Not the ideal option.) We might also do this tracking somehow
> via the memstore. (At least in this case the lifetime and distribution of in
> memory row state, including the proposed timestamps, would align.) Assuming
> we can efficiently know if we are about to commit twice to the same row
> within a single clock tick, we would simply sleep/yield the current thread
> until the clock ticks over, and then proceed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25913) Introduce EnvironmentEdge.currentTimeAdvancing
[
https://issues.apache.org/jira/browse/HBASE-25913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25913:
Description:
Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that when
the current time is returned, it is the current time in a different clock tick
from the last time the {{EnvironmentEdge}} was used to get the current time.
When processing mutations we substitute the {{Long.MAX_VALUE}} timestamp
placeholder with a real placeholder just before committing the mutation. The
current code gets the current time for timestamp substitution while under row
lock and mvcc. We will simply use {{EnvironmentEdge#currentTimeAdvancing}}
instead of {{EnvironmentEdge#currentTime}} at this point in the code to ensure
we have seen the clock tick over. When processing a batch of mutations
(doMiniBatchMutation etc) we will call {{currentTimeAdvancing}} only once. This
means the client cannot bundle cells with wildcard timestamps into a batch
where those cells must be committed with different timestamps. Clients must
simply not submit mutations that must be committed with guaranteed distinct
timestamps in the same batch. Easy to understand, easy to document, and it
aligns with our design philosophy of the client knows best.
It is not required to handle batches as proposed. We could guarantee a distinct
timestamp for every mutation in a batch. Count the number of mutations, call
this M. Acquire all row locks and get the current time. Then, wait for at least
M milliseconds. Then, set the first mutation timestamp with this value and
increment by 1 for all remaining. Then, do the rest of mutation processing as
normal. I don't think this extra waiting to reserve the range of timestamps is
necessary. See reasoning in above paragraph. Mentioned here for sake of
discussion.
It will be fine to continue to use {{EnvironmentEdge#currentTime}} everywhere
else. In this way we will only potentially spin wait where it matters, and
won't suffer serious overheads during batch processing.
was:
Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that when
the current time is returned, it is the current time in a different clock tick
from the last time the {{EnvironmentEdge}} was used to get the current time.
When processing mutations we substitute the {{Long.MAX_VALUE}} timestamp
placeholder with a real placeholder just before committing the mutation. The
current code gets the current time for timestamp substitution while under row
lock and mvcc. We will simply use {{EnvironmentEdge#currentTimeAdvancing}}
instead of {{EnvironmentEdge#currentTime}} at this point in the code to ensure
we have seen the clock tick over. When processing a batch of mutations
(doMiniBatchMutation etc) we will call {{currentTimeAdvancing}} only once. This
means the client cannot bundle cells with wildcard timestamps into a batch
where those cells must be committed with different timestamps. Clients must
simply not submit mutations that must be committed with guaranteed distinct
timestamps in the same batch. Easy to understand, easy to document, and it
aligns with our design philosophy of the client knows best.
It is not required to handle batches as proposed. We could guarantee a distinct
timestamp for every mutation in a batch. Count the number of mutations, call
this M. Acquire all row locks and get the current time. Then, wait for at least
M milliseconds. Then, set the first mutation timestamp with this value and
increment by 1 for all remaining. Then, release the locks. I don't think this
is necessary. See reasoning in above paragraph. Mentioned here for sake of
discussion.
It will be fine to continue to use {{EnvironmentEdge#currentTime}} everywhere
else. In this way we will only potentially spin wait where it matters, and
won't suffer serious overheads during batch processing.
> Introduce EnvironmentEdge.currentTimeAdvancing
> --
>
> Key: HBASE-25913
> URL: https://issues.apache.org/jira/browse/HBASE-25913
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that
> when the current time is returned, it is the current time in a different
> clock tick from the last time the {{EnvironmentEdge}} was used to get the
> current time.
> When processing mutations we substitute the {{Long.MAX_VALUE}} timestamp
> placeholder with a real placeholder just before committing the mutation. The
> current code gets the current time for timestamp substitution while under row
> lock and mvcc. We will simply use {{EnvironmentEdge#currentTimeAdvancing}}
>
[jira] [Updated] (HBASE-25913) Introduce EnvironmentEdge.currentTimeAdvancing
[
https://issues.apache.org/jira/browse/HBASE-25913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25913:
Description:
Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that when
the current time is returned, it is the current time in a different clock tick
from the last time the {{EnvironmentEdge}} was used to get the current time.
When processing mutations we substitute the {{Long.MAX_VALUE}} timestamp
placeholder with a real placeholder just before committing the mutation. The
current code gets the current time for timestamp substitution while under row
lock and mvcc. We will simply use {{EnvironmentEdge#currentTimeAdvancing}}
instead of {{EnvironmentEdge#currentTime}} at this point in the code to ensure
we have seen the clock tick over. When processing a batch of mutations
(doMiniBatchMutation etc) we will call {{currentTimeAdvancing}} only once. This
means the client cannot bundle cells with wildcard timestamps into a batch
where those cells must be committed with different timestamps. Clients must
simply not submit mutations that must be committed with guaranteed distinct
timestamps in the same batch. Easy to understand, easy to document, and it
aligns with our design philosophy of the client knows best.
It is not required to handle batches as proposed. We could guarantee a distinct
timestamp for every mutation in a batch. Count the number of mutations, call
this M. Acquire all row locks and get the current time. Then, wait for at least
M milliseconds. Then, set the first mutation timestamp with this value and
increment by 1 for all remaining. Then, release the locks. I don't think this
is necessary. See reasoning in above paragraph. Mentioned here for sake of
discussion.
It will be fine to continue to use {{EnvironmentEdge#currentTime}} everywhere
else. In this way we will only potentially spin wait where it matters, and
won't suffer serious overheads during batch processing.
was:
Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that when
the current time is returned, it is the current time in a different clock tick
from the last time the {{EnvironmentEdge}} was used to get the current time.
Use {{EnvironmentEdge#currentTimeAdvancing}} wherever we go to substitute a
{{Long.MAX_VALUE}} timestamp placeholder with a real placeholder just before
committing the mutation. When processing a batch of mutations
(doMiniBatchMutation etc) we will call {{currentTimeAdvancing}} only once. This
means the client cannot bundle cells with wildcard timestamps into a batch
where those cells must be committed with different timestamps. Clients must
simply not submit mutations that must be committed with guaranteed distinct
timestamps in the same batch. Easy to understand, easy to document, and it
aligns with our design philosophy of the client knows best.
It is not required to handle batches as proposed. We could guarantee a distinct
timestamp for every mutation in a batch. Count the number of mutations, call
this M. Get the current time. Then, wait for at least M milliseconds. Then, set
the first mutation timestamp with this value and increment by 1 for all
remaining. I don't think this is necessary. See reasoning in above paragraph.
Mentioned here for sake of discussion.
It will be fine to continue to use {{EnvironmentEdge#currentTime}} everywhere
else. In this way we will only potentially spin wait where it matters, and
won't suffer serious overheads during batch processing.
> Introduce EnvironmentEdge.currentTimeAdvancing
> --
>
> Key: HBASE-25913
> URL: https://issues.apache.org/jira/browse/HBASE-25913
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that
> when the current time is returned, it is the current time in a different
> clock tick from the last time the {{EnvironmentEdge}} was used to get the
> current time.
> When processing mutations we substitute the {{Long.MAX_VALUE}} timestamp
> placeholder with a real placeholder just before committing the mutation. The
> current code gets the current time for timestamp substitution while under row
> lock and mvcc. We will simply use {{EnvironmentEdge#currentTimeAdvancing}}
> instead of {{EnvironmentEdge#currentTime}} at this point in the code to
> ensure we have seen the clock tick over. When processing a batch of mutations
> (doMiniBatchMutation etc) we will call {{currentTimeAdvancing}} only once.
> This means the client cannot bundle cells with wildcard timestamps into a
> batch where those cells must be committed with different
[jira] [Commented] (HBASE-25913) Introduce EnvironmentEdge.currentTimeAdvancing
[
https://issues.apache.org/jira/browse/HBASE-25913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350713#comment-17350713
]
Duo Zhang commented on HBASE-25913:
---
+1
> Introduce EnvironmentEdge.currentTimeAdvancing
> --
>
> Key: HBASE-25913
> URL: https://issues.apache.org/jira/browse/HBASE-25913
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that
> when the current time is returned, it is the current time in a different
> clock tick from the last time the {{EnvironmentEdge}} was used to get the
> current time.
> Use {{EnvironmentEdge#currentTimeAdvancing}} wherever we go to substitute a
> {{Long.MAX_VALUE}} timestamp placeholder with a real placeholder just before
> committing the mutation. When processing a batch of mutations
> (doMiniBatchMutation etc) we will call {{currentTimeAdvancing}} only once.
> This means the client cannot bundle cells with wildcard timestamps into a
> batch where those cells must be committed with different timestamps. Clients
> must simply not submit mutations that must be committed with guaranteed
> distinct timestamps in the same batch. Easy to understand, easy to document,
> and it aligns with our design philosophy of the client knows best.
> It is not required to handle batches as proposed. We could guarantee a
> distinct timestamp for every mutation in a batch. Count the number of
> mutations, call this M. Get the current time. Then, wait for at least M
> milliseconds. Then, set the first mutation timestamp with this value and
> increment by 1 for all remaining. I don't think this is necessary. See
> reasoning in above paragraph. Mentioned here for sake of discussion.
> It will be fine to continue to use {{EnvironmentEdge#currentTime}} everywhere
> else. In this way we will only potentially spin wait where it matters, and
> won't suffer serious overheads during batch processing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25913) Introduce EnvironmentEdge.currentTimeAdvancing
[
https://issues.apache.org/jira/browse/HBASE-25913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25913:
Description:
Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that when
the current time is returned, it is the current time in a different clock tick
from the last time the {{EnvironmentEdge}} was used to get the current time.
Use {{EnvironmentEdge#currentTimeAdvancing}} wherever we go to substitute a
{{Long.MAX_VALUE}} timestamp placeholder with a real placeholder just before
committing the mutation. When processing a batch of mutations
(doMiniBatchMutation etc) we will call {{currentTimeAdvancing}} only once. This
means the client cannot bundle cells with wildcard timestamps into a batch
where those cells must be committed with different timestamps. Clients must
simply not submit mutations that must be committed with guaranteed distinct
timestamps in the same batch. Easy to understand, easy to document, and it
aligns with our design philosophy of the client knows best.
It is not required to handle batches as proposed. We could guarantee a distinct
timestamp for every mutation in a batch. Count the number of mutations, call
this M. Get the current time. Then, wait for at least M milliseconds. Then, set
the first mutation timestamp with this value and increment by 1 for all
remaining. I don't think this is necessary. See reasoning in above paragraph.
Mentioned here for sake of discussion.
It will be fine to continue to use {{EnvironmentEdge#currentTime}} everywhere
else. In this way we will only potentially spin wait where it matters, and
won't suffer serious overheads during batch processing.
was:
Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that when
the current time is returned, it is the current time in a different clock tick
from the last time the {{EnvironmentEdge}} was used to get the current time.
Use {{EnvironmentEdge#currentTimeAdvancing}} wherever we go to substitute a
{{Long.MAX_VALUE}} timestamp placeholder with a real placeholder just before
committing the mutation. When processing a batch of mutations
(doMiniBatchMutation etc) we will call {{currentTimeAdvancing}} only once. This
means the client cannot bundle cells with wildcard timestamps into a batch
where those cells must be committed with different timestamps. Clients must
simply not submit mutations that must be committed with guaranteed distinct
timestamps in the same batch. Easy to understand, easy to document, and it
aligns with our design philosophy of the client knows best.
It is not required to handle batches as proposed; we could guarantee a distinct
timestamp for every mutation in the batch. Count the number of mutations, call
this M. Get the current time. Set the first mutation timestamp with this value
and increment by 1 for all remaining. Then, wait for at least M milliseconds. I
don't think this is necessary. See reasoning in above paragraph. Mentioned here
for sake of discussion.
It will be fine to continue to use {{EnvironmentEdge#currentTime}} everywhere
else. In this way we will only potentially spin wait where it matters, and
won't suffer serious overheads during batch processing.
> Introduce EnvironmentEdge.currentTimeAdvancing
> --
>
> Key: HBASE-25913
> URL: https://issues.apache.org/jira/browse/HBASE-25913
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that
> when the current time is returned, it is the current time in a different
> clock tick from the last time the {{EnvironmentEdge}} was used to get the
> current time.
> Use {{EnvironmentEdge#currentTimeAdvancing}} wherever we go to substitute a
> {{Long.MAX_VALUE}} timestamp placeholder with a real placeholder just before
> committing the mutation. When processing a batch of mutations
> (doMiniBatchMutation etc) we will call {{currentTimeAdvancing}} only once.
> This means the client cannot bundle cells with wildcard timestamps into a
> batch where those cells must be committed with different timestamps. Clients
> must simply not submit mutations that must be committed with guaranteed
> distinct timestamps in the same batch. Easy to understand, easy to document,
> and it aligns with our design philosophy of the client knows best.
> It is not required to handle batches as proposed. We could guarantee a
> distinct timestamp for every mutation in a batch. Count the number of
> mutations, call this M. Get the current time. Then, wait for at least M
> milliseconds. Then, set the first mutation timestamp with this value and
>
[GitHub] [hbase] Apache9 commented on pull request #3302: HBASE-25911 Replace calls to System.currentTimeMillis with EnvironmentEdgeManager.currentTime
Apache9 commented on pull request #3302: URL: https://github.com/apache/hbase/pull/3302#issuecomment-847463261 What about System.nanoTime? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[
https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350711#comment-17350711
]
Andrew Kyle Purtell edited comment on HBASE-24440 at 5/25/21, 1:28 AM:
---
I'm considering something simple:
* Ensure our discipline in using {{EnvironmentEdge#currentTime}} instead of
{{System#currentTimeMillis}} (HBASE-25912)
* Fix current issues (HBASE-25911)
* Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that
when the current time is returned, it is the current time in a different clock
tick from the last time the {{EnvironmentEdge}} was used to get the current
time. Use {{EnvironmentEdge#currentTimeAdvancing}} wherever we go to substitute
a {{Long.MAX_VALUE}} timestamp placeholder with a real placeholder. See
HBASE-25913 for more details.
was (Author: apurtell):
I'm considering something simple:
* Ensure our discipline in using {{EnvironmentEdge#currentTime}} instead of
{{System#currentTimeMillis}} (HBASE-25912)
* Fix current issues (HBASE-25911)
* Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that
when the current time is returned, it is the current time in a different clock
tick from the last time the {{EnvironmentEdge}} was used to get the current
time.
* Use {{EnvironmentEdge#currentTimeAdvancing}} wherever we go to substitute a
{{Long.MAX_VALUE}} timestamp placeholder with a real placeholder. When
processing a batch mutation we will call {{currentTimeAdvancing}} only once.
This means the client cannot bundle cells with wildcard timestamps into a batch
where those cells must be committed with different timestamps. Clients must
simply not submit mutations that must be committed with guaranteed distinct
timestamps in the same batch. Easy to understand, easy to document, and it
aligns with our design philosophy of the client knows best. It will be fine to
continue to use {{EnvironmentEdge#currentTime}} everywhere else. In this way we
will only potentially spin wait where it matters, and won't suffer serious
overheads during batch processing.
> Prevent temporal misordering on timescales smaller than one clock tick
> --
>
> Key: HBASE-24440
> URL: https://issues.apache.org/jira/browse/HBASE-24440
> Project: HBase
> Issue Type: Brainstorming
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When mutations are sent to the servers without a timestamp explicitly
> assigned by the client the server will substitute the current wall clock
> time. There are edge cases where it is at least theoretically possible for
> more than one mutation to be committed to a given row within the same clock
> tick. When this happens we have to track and preserve the ordering of these
> mutations in some other way besides the timestamp component of the key. Let
> me bypass most discussion here by noting that whether we do this or not, we
> do not pass such ordering information in the cross cluster replication
> protocol. We also have interesting edge cases regarding key type precedence
> when mutations arrive "simultaneously": we sort deletes ahead of puts. This,
> especially in the presence of replication, can lead to visible anomalies for
> clients able to interact with both source and sink.
> There is a simple solution that removes the possibility that these edge cases
> can occur:
> We can detect, when we are about to commit a mutation to a row, if we have
> already committed a mutation to this same row in the current clock tick.
> Occurrences of this condition will be rare. We are already tracking current
> time. We have to know this in order to assign the timestamp. Where this
> becomes interesting is how we might track the last commit time per row.
> Making the detection of this case efficient for the normal code path is the
> bulk of the challenge. One option is to keep track of the last locked time
> for row locks. (Todo: How would we track and garbage collect this efficiently
> and correctly. Not the ideal option.) We might also do this tracking somehow
> via the memstore. (At least in this case the lifetime and distribution of in
> memory row state, including the proposed timestamps, would align.) Assuming
> we can efficiently know if we are about to commit twice to the same row
> within a single clock tick, we would simply sleep/yield the current thread
> until the clock ticks over, and then proceed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-25913) Introduce EnvironmentEdge.currentTimeAdvancing
[
https://issues.apache.org/jira/browse/HBASE-25913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell reassigned HBASE-25913:
---
Assignee: Andrew Kyle Purtell
> Introduce EnvironmentEdge.currentTimeAdvancing
> --
>
> Key: HBASE-25913
> URL: https://issues.apache.org/jira/browse/HBASE-25913
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that
> when the current time is returned, it is the current time in a different
> clock tick from the last time the {{EnvironmentEdge}} was used to get the
> current time.
> Use {{EnvironmentEdge#currentTimeAdvancing}} wherever we go to substitute a
> {{Long.MAX_VALUE}} timestamp placeholder with a real placeholder just before
> committing the mutation. When processing a batch of mutations
> (doMiniBatchMutation etc) we will call {{currentTimeAdvancing}} only once.
> This means the client cannot bundle cells with wildcard timestamps into a
> batch where those cells must be committed with different timestamps. Clients
> must simply not submit mutations that must be committed with guaranteed
> distinct timestamps in the same batch. Easy to understand, easy to document,
> and it aligns with our design philosophy of the client knows best.
> It is not required to handle batches as proposed; we could guarantee a
> distinct timestamp for every mutation in the batch. Count the number of
> mutations, call this M. Get the current time. Set the first mutation
> timestamp with this value and increment by 1 for all remaining. Then, wait
> for at least M milliseconds. I don't think this is necessary. See reasoning
> in above paragraph. Mentioned here for sake of discussion.
> It will be fine to continue to use {{EnvironmentEdge#currentTime}} everywhere
> else. In this way we will only potentially spin wait where it matters, and
> won't suffer serious overheads during batch processing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25913) Introduce EnvironmentEdge.currentTimeAdvancing
[
https://issues.apache.org/jira/browse/HBASE-25913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25913:
Description:
Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that when
the current time is returned, it is the current time in a different clock tick
from the last time the {{EnvironmentEdge}} was used to get the current time.
Use {{EnvironmentEdge#currentTimeAdvancing}} wherever we go to substitute a
{{Long.MAX_VALUE}} timestamp placeholder with a real placeholder just before
committing the mutation. When processing a batch of mutations
(doMiniBatchMutation etc) we will call {{currentTimeAdvancing}} only once. This
means the client cannot bundle cells with wildcard timestamps into a batch
where those cells must be committed with different timestamps. Clients must
simply not submit mutations that must be committed with guaranteed distinct
timestamps in the same batch. Easy to understand, easy to document, and it
aligns with our design philosophy of the client knows best.
It is not required to handle batches as proposed; we could guarantee a distinct
timestamp for every mutation in the batch. Count the number of mutations, call
this M. Get the current time. Set the first mutation timestamp with this value
and increment by 1 for all remaining. Then, wait for at least M milliseconds. I
don't think this is necessary. See reasoning in above paragraph. Mentioned here
for sake of discussion.
It will be fine to continue to use {{EnvironmentEdge#currentTime}} everywhere
else. In this way we will only potentially spin wait where it matters, and
won't suffer serious overheads during batch processing.
was:Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that
when the current time is returned, it is the current time in a different clock
tick from the last time the {{EnvironmentEdge}} was used to get the current time
> Introduce EnvironmentEdge.currentTimeAdvancing
> --
>
> Key: HBASE-25913
> URL: https://issues.apache.org/jira/browse/HBASE-25913
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that
> when the current time is returned, it is the current time in a different
> clock tick from the last time the {{EnvironmentEdge}} was used to get the
> current time.
> Use {{EnvironmentEdge#currentTimeAdvancing}} wherever we go to substitute a
> {{Long.MAX_VALUE}} timestamp placeholder with a real placeholder just before
> committing the mutation. When processing a batch of mutations
> (doMiniBatchMutation etc) we will call {{currentTimeAdvancing}} only once.
> This means the client cannot bundle cells with wildcard timestamps into a
> batch where those cells must be committed with different timestamps. Clients
> must simply not submit mutations that must be committed with guaranteed
> distinct timestamps in the same batch. Easy to understand, easy to document,
> and it aligns with our design philosophy of the client knows best.
> It is not required to handle batches as proposed; we could guarantee a
> distinct timestamp for every mutation in the batch. Count the number of
> mutations, call this M. Get the current time. Set the first mutation
> timestamp with this value and increment by 1 for all remaining. Then, wait
> for at least M milliseconds. I don't think this is necessary. See reasoning
> in above paragraph. Mentioned here for sake of discussion.
> It will be fine to continue to use {{EnvironmentEdge#currentTime}} everywhere
> else. In this way we will only potentially spin wait where it matters, and
> won't suffer serious overheads during batch processing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25913) Introduce EnvironmentEdge.currentTimeAdvancing
Andrew Kyle Purtell created HBASE-25913:
---
Summary: Introduce EnvironmentEdge.currentTimeAdvancing
Key: HBASE-25913
URL: https://issues.apache.org/jira/browse/HBASE-25913
Project: HBase
Issue Type: Sub-task
Reporter: Andrew Kyle Purtell
Fix For: 3.0.0-alpha-1, 2.5.0
Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that when
the current time is returned, it is the current time in a different clock tick
from the last time the {{EnvironmentEdge}} was used to get the current time
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[ https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-24440 started by Andrew Kyle Purtell. --- > Prevent temporal misordering on timescales smaller than one clock tick > -- > > Key: HBASE-24440 > URL: https://issues.apache.org/jira/browse/HBASE-24440 > Project: HBase > Issue Type: Brainstorming >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > > When mutations are sent to the servers without a timestamp explicitly > assigned by the client the server will substitute the current wall clock > time. There are edge cases where it is at least theoretically possible for > more than one mutation to be committed to a given row within the same clock > tick. When this happens we have to track and preserve the ordering of these > mutations in some other way besides the timestamp component of the key. Let > me bypass most discussion here by noting that whether we do this or not, we > do not pass such ordering information in the cross cluster replication > protocol. We also have interesting edge cases regarding key type precedence > when mutations arrive "simultaneously": we sort deletes ahead of puts. This, > especially in the presence of replication, can lead to visible anomalies for > clients able to interact with both source and sink. > There is a simple solution that removes the possibility that these edge cases > can occur: > We can detect, when we are about to commit a mutation to a row, if we have > already committed a mutation to this same row in the current clock tick. > Occurrences of this condition will be rare. We are already tracking current > time. We have to know this in order to assign the timestamp. Where this > becomes interesting is how we might track the last commit time per row. > Making the detection of this case efficient for the normal code path is the > bulk of the challenge. One option is to keep track of the last locked time > for row locks. (Todo: How would we track and garbage collect this efficiently > and correctly. Not the ideal option.) We might also do this tracking somehow > via the memstore. (At least in this case the lifetime and distribution of in > memory row state, including the proposed timestamps, would align.) Assuming > we can efficiently know if we are about to commit twice to the same row > within a single clock tick, we would simply sleep/yield the current thread > until the clock ticks over, and then proceed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[ https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24440: Fix Version/s: 2.5.0 3.0.0-alpha-1 > Prevent temporal misordering on timescales smaller than one clock tick > -- > > Key: HBASE-24440 > URL: https://issues.apache.org/jira/browse/HBASE-24440 > Project: HBase > Issue Type: Brainstorming >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > When mutations are sent to the servers without a timestamp explicitly > assigned by the client the server will substitute the current wall clock > time. There are edge cases where it is at least theoretically possible for > more than one mutation to be committed to a given row within the same clock > tick. When this happens we have to track and preserve the ordering of these > mutations in some other way besides the timestamp component of the key. Let > me bypass most discussion here by noting that whether we do this or not, we > do not pass such ordering information in the cross cluster replication > protocol. We also have interesting edge cases regarding key type precedence > when mutations arrive "simultaneously": we sort deletes ahead of puts. This, > especially in the presence of replication, can lead to visible anomalies for > clients able to interact with both source and sink. > There is a simple solution that removes the possibility that these edge cases > can occur: > We can detect, when we are about to commit a mutation to a row, if we have > already committed a mutation to this same row in the current clock tick. > Occurrences of this condition will be rare. We are already tracking current > time. We have to know this in order to assign the timestamp. Where this > becomes interesting is how we might track the last commit time per row. > Making the detection of this case efficient for the normal code path is the > bulk of the challenge. One option is to keep track of the last locked time > for row locks. (Todo: How would we track and garbage collect this efficiently > and correctly. Not the ideal option.) We might also do this tracking somehow > via the memstore. (At least in this case the lifetime and distribution of in > memory row state, including the proposed timestamps, would align.) Assuming > we can efficiently know if we are about to commit twice to the same row > within a single clock tick, we would simply sleep/yield the current thread > until the clock ticks over, and then proceed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[
https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350711#comment-17350711
]
Andrew Kyle Purtell commented on HBASE-24440:
-
I'm considering something simple:
* Ensure our discipline in using {{EnvironmentEdge#currentTime}} instead of
{{System#currentTimeMillis}} (HBASE-25912)
* Fix current issues (HBASE-25911)
* Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that
when the current time is returned, it is the current time in a different clock
tick from the last time the {{EnvironmentEdge}} was used to get the current
time.
* Use {{EnvironmentEdge#currentTimeAdvancing}} wherever we go to substitute a
{{Long.MAX_VALUE}} timestamp placeholder with a real placeholder. When
processing a batch mutation we will call {{currentTimeAdvancing}} only once.
This means the client cannot bundle cells with wildcard timestamps into a batch
where those cells must be committed with different timestamps. Clients must
simply not submit mutations that must be committed with guaranteed distinct
timestamps in the same batch. Easy to understand, easy to document, and it
aligns with our design philosophy of the client knows best. It will be fine to
continue to use {{EnvironmentEdge#currentTime}} everywhere else. In this way we
will only potentially spin wait where it matters, and won't suffer serious
overheads during batch processing.
> Prevent temporal misordering on timescales smaller than one clock tick
> --
>
> Key: HBASE-24440
> URL: https://issues.apache.org/jira/browse/HBASE-24440
> Project: HBase
> Issue Type: Brainstorming
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
>
> When mutations are sent to the servers without a timestamp explicitly
> assigned by the client the server will substitute the current wall clock
> time. There are edge cases where it is at least theoretically possible for
> more than one mutation to be committed to a given row within the same clock
> tick. When this happens we have to track and preserve the ordering of these
> mutations in some other way besides the timestamp component of the key. Let
> me bypass most discussion here by noting that whether we do this or not, we
> do not pass such ordering information in the cross cluster replication
> protocol. We also have interesting edge cases regarding key type precedence
> when mutations arrive "simultaneously": we sort deletes ahead of puts. This,
> especially in the presence of replication, can lead to visible anomalies for
> clients able to interact with both source and sink.
> There is a simple solution that removes the possibility that these edge cases
> can occur:
> We can detect, when we are about to commit a mutation to a row, if we have
> already committed a mutation to this same row in the current clock tick.
> Occurrences of this condition will be rare. We are already tracking current
> time. We have to know this in order to assign the timestamp. Where this
> becomes interesting is how we might track the last commit time per row.
> Making the detection of this case efficient for the normal code path is the
> bulk of the challenge. One option is to keep track of the last locked time
> for row locks. (Todo: How would we track and garbage collect this efficiently
> and correctly. Not the ideal option.) We might also do this tracking somehow
> via the memstore. (At least in this case the lifetime and distribution of in
> memory row state, including the proposed timestamps, would align.) Assuming
> we can efficiently know if we are about to commit twice to the same row
> within a single clock tick, we would simply sleep/yield the current thread
> until the clock ticks over, and then proceed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25911) Fix uses of System.currentTimeMillis (should be EnvironmentEdgeManager.currentTime)
[ https://issues.apache.org/jira/browse/HBASE-25911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25911: Status: Patch Available (was: Open) > Fix uses of System.currentTimeMillis (should be > EnvironmentEdgeManager.currentTime) > --- > > Key: HBASE-25911 > URL: https://issues.apache.org/jira/browse/HBASE-25911 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.5.0 > > > We introduced EnvironmentEdgeManager a long time ago as a way to inject > alternate clocks (gettimeofday() aka System.currentTimeMillis()) for unit > tests. In order for this to be effective, all callers that would otherwise > use System.currentTimeMillis() must call EnvironmentEdgeManager.currentTime() > instead, except obviously the implementors of EnvironmentEdge. > It's common for contributors to be unaware of this practice and reviewers > might not catch it. > It will be much more important to have EnvironmentEdgeManager in use where > expected once we have EnvironmentEdge also providing a monotonic clock > source. (See parent.) > On another subtask I will introduce a build enforcer that bans > System.currentTimeMillis() except where annotated to allow it. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] apurtell opened a new pull request #3302: HBASE-25911 Replace calls to System.currentTimeMillis with EnvironmentEdgeManager.currentTime
apurtell opened a new pull request #3302: URL: https://github.com/apache/hbase/pull/3302 We introduced EnvironmentEdgeManager a long time ago as a way to inject alternate clocks (gettimeofday() aka System.currentTimeMillis()) for unit tests. In order for this to be effective, all callers that would otherwise use System.currentTimeMillis() must call EnvironmentEdgeManager.currentTime() instead, except obviously the implementors of EnvironmentEdge. It's common for contributors to be unaware of this practice and reviewers might not catch it. On another subtask I will introduce a build enforcer that bans System.currentTimeMillis() except where we annotate to allow it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25513) When the table is turned on normalize, the first region may not be merged even the size is 0
[ https://issues.apache.org/jira/browse/HBASE-25513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-25513. -- Resolution: Fixed > When the table is turned on normalize, the first region may not be merged > even the size is 0 > > > Key: HBASE-25513 > URL: https://issues.apache.org/jira/browse/HBASE-25513 > Project: HBase > Issue Type: Bug > Components: Normalizer >Affects Versions: 3.0.0-alpha-1, 2.4.1 >Reporter: Baiqiang Zhao >Assignee: Baiqiang Zhao >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.2 > > > Suppose a table has 8 regions, the sizes are [0, 10, 1, 0, 9, 0, 12, 0], the > average region size is 4, and split is disabled. > The current Normalizer can only get three merge plans (use size to represent > region): > [1, 0], [9, 0],[12, 0] > It can not merge the first region, even it's size is 0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25513) When the table is turned on normalize, the first region may not be merged even the size is 0
[ https://issues.apache.org/jira/browse/HBASE-25513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-25513: - Fix Version/s: 2.5.0 > When the table is turned on normalize, the first region may not be merged > even the size is 0 > > > Key: HBASE-25513 > URL: https://issues.apache.org/jira/browse/HBASE-25513 > Project: HBase > Issue Type: Bug > Components: Normalizer >Affects Versions: 3.0.0-alpha-1, 2.4.1 >Reporter: Baiqiang Zhao >Assignee: Baiqiang Zhao >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.2 > > > Suppose a table has 8 regions, the sizes are [0, 10, 1, 0, 9, 0, 12, 0], the > average region size is 4, and split is disabled. > The current Normalizer can only get three merge plans (use size to represent > region): > [1, 0], [9, 0],[12, 0] > It can not merge the first region, even it's size is 0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk merged pull request #3301: Backport "HBASE-25513 When the table is turned on normalize, the first region may not be merged even the size is 0" to branch-2
ndimiduk merged pull request #3301: URL: https://github.com/apache/hbase/pull/3301 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25912) Ban System.currentTimeMillis with a build enforcer
Andrew Kyle Purtell created HBASE-25912: --- Summary: Ban System.currentTimeMillis with a build enforcer Key: HBASE-25912 URL: https://issues.apache.org/jira/browse/HBASE-25912 Project: HBase Issue Type: Sub-task Components: build Reporter: Andrew Kyle Purtell Assignee: Andrew Kyle Purtell We introduced EnvironmentEdgeManager a long time ago as a way to inject alternate clocks (gettimeofday() aka System.currentTimeMillis()) for unit tests. In order for this to be effective, all callers that would otherwise use System.currentTimeMillis() must call EnvironmentEdgeManager.currentTime() instead, except obviously the implementors of EnvironmentEdge. It's common for contributors to be unaware of this practice and reviewers might not catch it. It will be much more important to have EnvironmentEdgeManager in use where expected once we have EnvironmentEdge also providing a monotonic clock source. (See parent.) Introduce a build enforcer that bans System.currentTimeMillis() except where annotated to allow it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25911) Fix uses of System.currentTimeMillis (should be EnvironmentEdgeManager.currentTime)
[ https://issues.apache.org/jira/browse/HBASE-25911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25911: Summary: Fix uses of System.currentTimeMillis (should be EnvironmentEdgeManager.currentTime) (was: Replace uses of System.currentTimeMillis with EnvironmentEdgeManager.currentTime) > Fix uses of System.currentTimeMillis (should be > EnvironmentEdgeManager.currentTime) > --- > > Key: HBASE-25911 > URL: https://issues.apache.org/jira/browse/HBASE-25911 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.5.0 > > > We introduced EnvironmentEdgeManager a long time ago as a way to inject > alternate clocks (gettimeofday() aka System.currentTimeMillis()) for unit > tests. In order for this to be effective, all callers that would otherwise > use System.currentTimeMillis() must call EnvironmentEdgeManager.currentTime() > instead, except obviously the implementors of EnvironmentEdge. > It's common for contributors to be unaware of this practice and reviewers > might not catch it. > It will be much more important to have EnvironmentEdgeManager in use where > expected once we have EnvironmentEdge also providing a monotonic clock > source. (See parent.) > On another subtask I will introduce a build enforcer that bans > System.currentTimeMillis() except where annotated to allow it. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25911) Replace uses of System.currentTimeMillis with EnvironmentEdgeManager.currentTime
Andrew Kyle Purtell created HBASE-25911: --- Summary: Replace uses of System.currentTimeMillis with EnvironmentEdgeManager.currentTime Key: HBASE-25911 URL: https://issues.apache.org/jira/browse/HBASE-25911 Project: HBase Issue Type: Sub-task Reporter: Andrew Kyle Purtell Assignee: Andrew Kyle Purtell Fix For: 3.0.0-alpha-1, 2.5.0 We introduced EnvironmentEdgeManager a long time ago as a way to inject alternate clocks (gettimeofday() aka System.currentTimeMillis()) for unit tests. In order for this to be effective, all callers that would otherwise use System.currentTimeMillis() must call EnvironmentEdgeManager.currentTime() instead, except obviously the implementors of EnvironmentEdge. It's common for contributors to be unaware of this practice and reviewers might not catch it. It will be much more important to have EnvironmentEdgeManager in use where expected once we have EnvironmentEdge also providing a monotonic clock source. (See parent.) On another subtask I will introduce a build enforcer that bans System.currentTimeMillis() except where annotated to allow it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3301: Backport "HBASE-25513 When the table is turned on normalize, the first region may not be merged even the size is 0" to branch-2
Apache-HBase commented on pull request #3301: URL: https://github.com/apache/hbase/pull/3301#issuecomment-847430846 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 17s | branch-2 passed | | +1 :green_heart: | compile | 1m 7s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 55s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 0s | the patch passed | | +1 :green_heart: | compile | 1m 8s | the patch passed | | +1 :green_heart: | javac | 1m 8s | the patch passed | | +1 :green_heart: | shadedjars | 6m 46s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 140m 52s | hbase-server in the patch passed. | | | | 170m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3301/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3301 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b31b083d5737 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 1649013b99 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3301/1/testReport/ | | Max. process+thread count | 3757 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3301/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-24440) Prevent temporal misordering on timescales smaller than one clock tick
[ https://issues.apache.org/jira/browse/HBASE-24440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell reassigned HBASE-24440: --- Assignee: Andrew Kyle Purtell > Prevent temporal misordering on timescales smaller than one clock tick > -- > > Key: HBASE-24440 > URL: https://issues.apache.org/jira/browse/HBASE-24440 > Project: HBase > Issue Type: Brainstorming >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > > When mutations are sent to the servers without a timestamp explicitly > assigned by the client the server will substitute the current wall clock > time. There are edge cases where it is at least theoretically possible for > more than one mutation to be committed to a given row within the same clock > tick. When this happens we have to track and preserve the ordering of these > mutations in some other way besides the timestamp component of the key. Let > me bypass most discussion here by noting that whether we do this or not, we > do not pass such ordering information in the cross cluster replication > protocol. We also have interesting edge cases regarding key type precedence > when mutations arrive "simultaneously": we sort deletes ahead of puts. This, > especially in the presence of replication, can lead to visible anomalies for > clients able to interact with both source and sink. > There is a simple solution that removes the possibility that these edge cases > can occur: > We can detect, when we are about to commit a mutation to a row, if we have > already committed a mutation to this same row in the current clock tick. > Occurrences of this condition will be rare. We are already tracking current > time. We have to know this in order to assign the timestamp. Where this > becomes interesting is how we might track the last commit time per row. > Making the detection of this case efficient for the normal code path is the > bulk of the challenge. One option is to keep track of the last locked time > for row locks. (Todo: How would we track and garbage collect this efficiently > and correctly. Not the ideal option.) We might also do this tracking somehow > via the memstore. (At least in this case the lifetime and distribution of in > memory row state, including the proposed timestamps, would align.) Assuming > we can efficiently know if we are about to commit twice to the same row > within a single clock tick, we would simply sleep/yield the current thread > until the clock ticks over, and then proceed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3301: Backport "HBASE-25513 When the table is turned on normalize, the first region may not be merged even the size is 0" to branch-2
Apache-HBase commented on pull request #3301: URL: https://github.com/apache/hbase/pull/3301#issuecomment-847430262 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 41s | branch-2 passed | | +1 :green_heart: | compile | 0m 59s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 57s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 25s | the patch passed | | +1 :green_heart: | compile | 0m 59s | the patch passed | | +1 :green_heart: | javac | 0m 59s | the patch passed | | +1 :green_heart: | shadedjars | 6m 4s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 143m 18s | hbase-server in the patch passed. | | | | 168m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3301/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3301 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e3af6e3a5fcb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 1649013b99 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3301/1/testReport/ | | Max. process+thread count | 4129 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3301/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25910) Fix TestClusterPortAssignment.testClusterPortAssignment test and re-enable it.
Victor Li created HBASE-25910: - Summary: Fix TestClusterPortAssignment.testClusterPortAssignment test and re-enable it. Key: HBASE-25910 URL: https://issues.apache.org/jira/browse/HBASE-25910 Project: HBase Issue Type: Test Components: flakies, test Reporter: Victor Li Fix For: 3.0.0-alpha-1 This test assigned with random unused ports for master and region server ports and start a cluster, however it can fail as the previously unsigned port might be taken by other application at the moment when cluster start to bind the ports. Server has an option to allow auto assign an port if the port is unavailable. This option needs to be disabled in this test in order to test explicit port assignment. The exception for the failure of starting server needs to change to IOException and there is a need to set a retry count instead of infinite retry. Related ticket: https://issues.apache.org/jira/browse/HBASE-24342 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3300: HBASE-25898 RS getting aborted due to NPE in Replication WALEntryStream
Apache-HBase commented on pull request #3300: URL: https://github.com/apache/hbase/pull/3300#issuecomment-847387579 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 15m 12s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-1 Compile Tests _ | | -1 :x: | mvninstall | 10m 17s | root in branch-1 failed. | | +1 :green_heart: | compile | 0m 51s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | compile | 0m 57s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | checkstyle | 2m 6s | branch-1 passed | | +1 :green_heart: | shadedjars | 3m 27s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 0m 42s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +0 :ok: | spotbugs | 3m 18s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 14s | branch-1 passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 1m 58s | root in the patch failed. | | +1 :green_heart: | compile | 0m 43s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javac | 0m 43s | the patch passed | | +1 :green_heart: | compile | 0m 48s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | javac | 0m 48s | the patch passed | | +1 :green_heart: | checkstyle | 1m 45s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 3m 15s | patch has no errors when building our shaded downstream artifacts. | | -1 :x: | hadoopcheck | 1m 47s | The patch causes 10 errors with Hadoop v2.8.5. | | -1 :x: | hadoopcheck | 4m 16s | The patch causes 10 errors with Hadoop v2.9.2. | | +1 :green_heart: | javadoc | 0m 34s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 0m 41s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | findbugs | 3m 10s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 168m 46s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. | | | | 226m 3s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.mapreduce.TestLoadIncrementalHFiles | | | hadoop.hbase.replication.TestReplicationKillSlaveRS | | | hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint | | | hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles | | | hadoop.hbase.client.TestAdmin1 | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3300/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3300 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 1a0d463042e0 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-3300/out/precommit/personality/provided.sh | | git revision | branch-1 / 3cff107 | | Default Java | Azul Systems, Inc.-1.7.0_272-b10 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3300/1/artifact/out/branch-mvninstall-root.txt | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3300/1/artifact/out/patch-mvninstall-root.txt | | hadoopcheck |
[GitHub] [hbase] Apache-HBase commented on pull request #3301: Backport "HBASE-25513 When the table is turned on normalize, the first region may not be merged even the size is 0" to branch-2
Apache-HBase commented on pull request #3301: URL: https://github.com/apache/hbase/pull/3301#issuecomment-847368526 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 23s | branch-2 passed | | +1 :green_heart: | compile | 3m 45s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 21s | branch-2 passed | | +1 :green_heart: | spotbugs | 2m 24s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | the patch passed | | +1 :green_heart: | compile | 3m 23s | the patch passed | | +1 :green_heart: | javac | 3m 23s | the patch passed | | +1 :green_heart: | checkstyle | 1m 16s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 13m 53s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 46s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 47m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3301/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3301 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 7a876eebe7bf 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 1649013b99 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 86 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3301/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Reopened] (HBASE-25534) Honor TableDescriptor settings earlier in normalization
[
https://issues.apache.org/jira/browse/HBASE-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk reopened HBASE-25534:
--
Reopen for backport to branch-2.
> Honor TableDescriptor settings earlier in normalization
> ---
>
> Key: HBASE-25534
> URL: https://issues.apache.org/jira/browse/HBASE-25534
> Project: HBase
> Issue Type: Improvement
> Components: Normalizer
>Affects Versions: 3.0.0-alpha-1, 2.4.2
>Reporter: Baiqiang Zhao
>Assignee: Baiqiang Zhao
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.2
>
>
> -Table can be enabled and disabled merge/split in TableDescriptor, we should
> judge before calculating the plan.-
> The normalizer configuration can be set by table level. For example,
> hbase.normalizer.min.region.count can be set by "alter ‘table’,
> CONFIGURATION=>\{'hbase.normalizer.min.region.count' => '5'}". If the table
> is not set, then use the global configuration which is set in hbase-site.xml.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk opened a new pull request #3301: Backport "HBASE-25513 When the table is turned on normalize, the first region may not be merged even the size is 0" to branch-2
ndimiduk opened a new pull request #3301: URL: https://github.com/apache/hbase/pull/3301 Signed-off-by: Nick Dimiduk -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Reopened] (HBASE-25513) When the table is turned on normalize, the first region may not be merged even the size is 0
[ https://issues.apache.org/jira/browse/HBASE-25513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk reopened HBASE-25513: -- Missed backport to branch-2. > When the table is turned on normalize, the first region may not be merged > even the size is 0 > > > Key: HBASE-25513 > URL: https://issues.apache.org/jira/browse/HBASE-25513 > Project: HBase > Issue Type: Bug > Components: Normalizer >Affects Versions: 3.0.0-alpha-1, 2.4.1 >Reporter: Baiqiang Zhao >Assignee: Baiqiang Zhao >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.2 > > > Suppose a table has 8 regions, the sizes are [0, 10, 1, 0, 9, 0, 12, 0], the > average region size is 4, and split is disabled. > The current Normalizer can only get three merge plans (use size to represent > region): > [1, 0], [9, 0],[12, 0] > It can not merge the first region, even it's size is 0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25888) Backup tests are categorically flakey
[
https://issues.apache.org/jira/browse/HBASE-25888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350608#comment-17350608
]
Nick Dimiduk commented on HBASE-25888:
--
Thank you [~rda3mon] for the contribution, and [~zhangduo] for finishing the
review!
> Backup tests are categorically flakey
> -
>
> Key: HBASE-25888
> URL: https://issues.apache.org/jira/browse/HBASE-25888
> Project: HBase
> Issue Type: Bug
> Components: backuprestore, test
>Reporter: Nick Dimiduk
>Assignee: Mallikarjun
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
> Attachments:
> TEST-org.apache.hadoop.hbase.backup.TestBackupDeleteRestore.xml.gz,
> TEST-org.apache.hadoop.hbase.backup.TestBackupMerge.xml.gz,
> TEST-org.apache.hadoop.hbase.backup.TestFullBackupSet.xml.gz,
> TEST-org.apache.hadoop.hbase.backup.TestIncrementalBackupMergeWithFailures.xml.gz,
>
> TEST-org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.xml.gz,
> TEST-org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests.xml.gz,
> org.apache.hadoop.hbase.backup.TestBackupDeleteRestore.txt,
> org.apache.hadoop.hbase.backup.TestBackupDeleteRestore.txt.gz,
> org.apache.hadoop.hbase.backup.TestBackupMerge-output.txt.gz,
> org.apache.hadoop.hbase.backup.TestBackupMerge.txt.gz,
> org.apache.hadoop.hbase.backup.TestFullBackupSet-output.txt.gz,
> org.apache.hadoop.hbase.backup.TestFullBackupSet.txt.gz,
> org.apache.hadoop.hbase.backup.TestIncrementalBackupMergeWithFailures-output.txt.gz,
>
> org.apache.hadoop.hbase.backup.TestIncrementalBackupMergeWithFailures.txt.gz,
> org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad-output.txt.gz,
> org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.txt.gz,
> org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests-output.txt.gz,
> org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests.txt.gz
>
>
> Here's some logs from a PR build vs. master that suffered a significant
> number of failures in the backup tests. I suspect that a single improvement
> could fix all of these tests to be more robust.
> {noformat}
> Test Name
> Duration
> Age
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestBackupDeleteRestore.testBackupDeleteRestore
> 6 min 23 sec1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestBackupDeleteRestore.(?)1 min 6 sec
> 1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestBackupMerge.TestIncBackupMergeRestore
> 5 min 3 sec 1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestBackupMerge.(?)1 min 6 sec 1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestFullBackupSet.testFullBackupSetExist
> 6 min 16 sec1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestFullBackupSet.(?) 1 min 6 sec 1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestIncrementalBackupMergeWithFailures.TestIncBackupMergeRestore
>5 min 55 sec1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestIncrementalBackupMergeWithFailures.(?)
> 1 min 6 sec 1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.TestIncBackupDeleteTable
> 5 min 56 sec1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.(?) 1 min 6
> sec 1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests.testFullRestoreSingleEmpty
> 6 min 5 sec 1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests.testFullRestoreMultipleEmpty
> 0.17 sec1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests.(?)
> {noformat}
> https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3249/4/testReport/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3206: HBASE-25818 Move StochasticLoadBalancer to hbase-balancer module
Apache-HBase commented on pull request #3206: URL: https://github.com/apache/hbase/pull/3206#issuecomment-847314388 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 7s | master passed | | +1 :green_heart: | compile | 1m 18s | master passed | | +1 :green_heart: | shadedjars | 8m 56s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 50s | master passed | | -0 :warning: | patch | 10m 6s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 1s | the patch passed | | +1 :green_heart: | compile | 1m 21s | the patch passed | | +1 :green_heart: | javac | 1m 21s | the patch passed | | +1 :green_heart: | shadedjars | 8m 59s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 9m 5s | hbase-balancer in the patch failed. | | +1 :green_heart: | unit | 212m 17s | hbase-server in the patch passed. | | | | 255m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3206 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ca3848f9ed33 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b02c8102b7 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/5/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-balancer.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/5/testReport/ | | Max. process+thread count | 2592 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] joshelser commented on a change in pull request #3298: HBASE-25907 Move StoreFlushContext out of HStore and make it pluggable
joshelser commented on a change in pull request #3298:
URL: https://github.com/apache/hbase/pull/3298#discussion_r638244458
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFlushContext.java
##
@@ -0,0 +1,183 @@
+/*
+ *
Review comment:
nit unnecessary line
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlushContext.java
##
@@ -29,15 +29,25 @@
* A store flush context carries the state required to prepare/flush/commit
the store's cache.
*/
@InterfaceAudience.Private
-interface StoreFlushContext {
+public abstract class StoreFlushContext {
+
+ protected HStore store;
+ protected long cacheFlushSeqNum;
+ protected FlushLifeCycleTracker tracker;
+
+ public StoreFlushContext(HStore store, Long cacheFlushSeqNum,
FlushLifeCycleTracker tracker){
Review comment:
It looks like it would take some work to do it, but can we make this be
a `Store` instead of `HStore`?
It looks like most of the methods called by DefaultStoreFileContext are
actually defined on HStore right now.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -2360,149 +2364,15 @@ public void upsert(Iterable cells, long
readpoint, MemStoreSizing memstore
}
}
- public StoreFlushContext createFlushContext(long cacheFlushId,
FlushLifeCycleTracker tracker) {
-return new StoreFlusherImpl(cacheFlushId, tracker);
- }
-
- private final class StoreFlusherImpl implements StoreFlushContext {
-
-private final FlushLifeCycleTracker tracker;
-private final long cacheFlushSeqNum;
-private MemStoreSnapshot snapshot;
-private List tempFiles;
-private List committedFiles;
-private long cacheFlushCount;
-private long cacheFlushSize;
-private long outputFileSize;
-
-private StoreFlusherImpl(long cacheFlushSeqNum, FlushLifeCycleTracker
tracker) {
- this.cacheFlushSeqNum = cacheFlushSeqNum;
- this.tracker = tracker;
-}
-
-/**
- * This is not thread safe. The caller should have a lock on the region or
the store.
- * If necessary, the lock can be added with the patch provided in
HBASE-10087
- */
-@Override
-public MemStoreSize prepare() {
- // passing the current sequence number of the wal - to allow bookkeeping
in the memstore
- this.snapshot = memstore.snapshot();
- this.cacheFlushCount = snapshot.getCellsCount();
- this.cacheFlushSize = snapshot.getDataSize();
- committedFiles = new ArrayList<>(1);
- return snapshot.getMemStoreSize();
-}
-
-@Override
-public void flushCache(MonitoredTask status) throws IOException {
- RegionServerServices rsService = region.getRegionServerServices();
- ThroughputController throughputController =
- rsService == null ? null : rsService.getFlushThroughputController();
- tempFiles =
- HStore.this.flushCache(cacheFlushSeqNum, snapshot, status,
throughputController, tracker);
-}
-
-@Override
-public boolean commit(MonitoredTask status) throws IOException {
- if (CollectionUtils.isEmpty(this.tempFiles)) {
-return false;
- }
- List storeFiles = new ArrayList<>(this.tempFiles.size());
- for (Path storeFilePath : tempFiles) {
-try {
- HStoreFile sf = HStore.this.commitFile(storeFilePath,
cacheFlushSeqNum, status);
- outputFileSize += sf.getReader().length();
- storeFiles.add(sf);
-} catch (IOException ex) {
- LOG.error("Failed to commit store file {}", storeFilePath, ex);
- // Try to delete the files we have committed before.
- for (HStoreFile sf : storeFiles) {
-Path pathToDelete = sf.getPath();
-try {
- sf.deleteStoreFile();
-} catch (IOException deleteEx) {
- LOG.error(HBaseMarkers.FATAL, "Failed to delete store file we
committed, "
- + "halting {}", pathToDelete, ex);
- Runtime.getRuntime().halt(1);
-}
- }
- throw new IOException("Failed to commit the flush", ex);
-}
- }
-
- for (HStoreFile sf : storeFiles) {
-if (HStore.this.getCoprocessorHost() != null) {
- HStore.this.getCoprocessorHost().postFlush(HStore.this, sf, tracker);
-}
-committedFiles.add(sf.getPath());
- }
-
- HStore.this.flushedCellsCount.addAndGet(cacheFlushCount);
- HStore.this.flushedCellsSize.addAndGet(cacheFlushSize);
- HStore.this.flushedOutputFileSize.addAndGet(outputFileSize);
-
- // Add new file to store files. Clear snapshot too while we have the
Store write lock.
- return HStore.this.updateStorefiles(storeFiles, snapshot.getId());
-}
-
-@Override
-public long getOutputFileSize() {
- return outputFileSize;
-}
-
-@Override
-public List
[GitHub] [hbase] Apache-HBase commented on pull request #3206: HBASE-25818 Move StochasticLoadBalancer to hbase-balancer module
Apache-HBase commented on pull request #3206: URL: https://github.com/apache/hbase/pull/3206#issuecomment-847310645 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 17s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 2s | master passed | | +1 :green_heart: | compile | 1m 48s | master passed | | +1 :green_heart: | shadedjars | 9m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 0s | master passed | | -0 :warning: | patch | 10m 35s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 54s | the patch passed | | +1 :green_heart: | compile | 1m 48s | the patch passed | | +1 :green_heart: | javac | 1m 48s | the patch passed | | +1 :green_heart: | shadedjars | 9m 21s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 3s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 7m 11s | hbase-balancer in the patch failed. | | +1 :green_heart: | unit | 203m 15s | hbase-server in the patch passed. | | | | 248m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3206 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ae122bca08cf 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b02c8102b7 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-balancer.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/5/testReport/ | | Max. process+thread count | 2680 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25909) Improve logging in "packaging and integration" stage
Nick Dimiduk created HBASE-25909:
Summary: Improve logging in "packaging and integration" stage
Key: HBASE-25909
URL: https://issues.apache.org/jira/browse/HBASE-25909
Project: HBase
Issue Type: Task
Components: build
Affects Versions: 3.0.0-alpha-1
Reporter: Nick Dimiduk
We had a nightly build of master [fail
inscrutably|https://ci-hadoop.apache.org/blue/organizations/jenkins/HBase%2FHBase%20Nightly/detail/master/298/pipeline/103].
Looks like the build went down some dark hole and didn't make a peep for
almost 16h. It would be nice to get some logging from whatever process that was.
{noformat}
[2021-05-20T20:13:45.733Z] Attempting to use run an instance on top of Hadoop 3.
[2021-05-21T11:58:19.827Z] Sending interrupt signal to process
[2021-05-21T11:58:20.683Z] Sending interrupt signal to process
[2021-05-21T11:58:32.771Z] script returned exit code 2
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25745) Deprecate/Rename config `hbase.normalizer.min.region.count` to `hbase.normalizer.merge.min.region.count`
[
https://issues.apache.org/jira/browse/HBASE-25745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-25745:
-
Status: Patch Available (was: Open)
> Deprecate/Rename config `hbase.normalizer.min.region.count` to
> `hbase.normalizer.merge.min.region.count`
>
>
> Key: HBASE-25745
> URL: https://issues.apache.org/jira/browse/HBASE-25745
> Project: HBase
> Issue Type: Improvement
> Components: master, Normalizer
>Affects Versions: 3.0.0-alpha-1, 2.5.0
>Reporter: Nick Dimiduk
>Assignee: Baiqiang Zhao
>Priority: Minor
> Fix For: 3.0.0-alpha-1
>
>
> After HBASE-24416, {{hbase.normalizer.min.region.count}} only applies to
> merge plans. Let's deprecate/rename the configuration key so that it is clear
> in what context it applies, and so that it matches the configuration
> structure of related keys.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on pull request #3139: HBASE-25745 Deprecate/Rename config `hbase.normalizer.min.region.coun…
ndimiduk commented on pull request #3139: URL: https://github.com/apache/hbase/pull/3139#issuecomment-847307159 @ZhaoBQ the master patch has conflicts when I backport it to branch-2. I'm guessing there's a patch on master that's missing on branch-2, because branch-2 doesn't have the `NormalizeContext#getOrDefault` method. If I don't get to it later today, maybe you can take a look? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25745) Deprecate/Rename config `hbase.normalizer.min.region.count` to `hbase.normalizer.merge.min.region.count`
[
https://issues.apache.org/jira/browse/HBASE-25745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-25745:
-
Fix Version/s: 3.0.0-alpha-1
> Deprecate/Rename config `hbase.normalizer.min.region.count` to
> `hbase.normalizer.merge.min.region.count`
>
>
> Key: HBASE-25745
> URL: https://issues.apache.org/jira/browse/HBASE-25745
> Project: HBase
> Issue Type: Improvement
> Components: master, Normalizer
>Affects Versions: 3.0.0-alpha-1, 2.5.0
>Reporter: Nick Dimiduk
>Assignee: Baiqiang Zhao
>Priority: Minor
> Fix For: 3.0.0-alpha-1
>
>
> After HBASE-24416, {{hbase.normalizer.min.region.count}} only applies to
> merge plans. Let's deprecate/rename the configuration key so that it is clear
> in what context it applies, and so that it matches the configuration
> structure of related keys.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ben-manes commented on a change in pull request #3215: HBASE-25698 Fixing IllegalReferenceCountException when using TinyLfuBlockCache
ben-manes commented on a change in pull request #3215:
URL: https://github.com/apache/hbase/pull/3215#discussion_r638239220
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/TinyLfuBlockCache.java
##
@@ -158,7 +158,13 @@ public boolean containsBlock(BlockCacheKey cacheKey) {
@Override
public Cacheable getBlock(BlockCacheKey cacheKey,
boolean caching, boolean repeat, boolean updateCacheMetrics) {
-Cacheable value = cache.getIfPresent(cacheKey);
+Cacheable value = cache.asMap().computeIfPresent(cacheKey, (blockCacheKey,
cacheable) -> {
+ // It will be referenced by RPC path, so increase here. NOTICE: Must do
the retain inside
+ // this block. because if retain outside the map#computeIfPresent, the
evictBlock may remove
+ // the block and release, then we're retaining a block with refCnt=0
which is disallowed.
+ cacheable.retain();
+ return cacheable;
+});
Review comment:
Thanks @saintstack. This was from my analysis when contributing the
original patch.
Zipfian is wonderful for a perf benchmark by stressing locks, etc. to find
bottlenecks, but isn't realistic for actual production performance. I'm not
sure if there is a great approach other than network record/replay or
canarying.
If you have a workload trace we can try to simulate that, where the hit
rates should be better (e.g. [database
trace](https://github.com/ben-manes/caffeine/wiki/Efficiency#database). That
wouldn't show actual system behavior, just the cache's expected hit rates in
isolation. HBase's LRU is similar-ish to SLRU, so then ARC might be a good
upper bound of expectations.
Between zipfian benchmark and trace simulations, we can get a roughish idea
of if there is a benefit. Otherwise canarying is the best that I've seen so
far, which is a bit heavy handed but simple.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk merged pull request #3139: HBASE-25745 Deprecate/Rename config `hbase.normalizer.min.region.coun…
ndimiduk merged pull request #3139: URL: https://github.com/apache/hbase/pull/3139 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25888) Backup tests are categorically flakey
[
https://issues.apache.org/jira/browse/HBASE-25888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350596#comment-17350596
]
Hudson commented on HBASE-25888:
Results for branch master
[build #302 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/302/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/302/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/302/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/302/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 3. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/302//artifact/output-integration/hadoop-3.log].
(note that this means we didn't check the Hadoop 3 shaded client)
> Backup tests are categorically flakey
> -
>
> Key: HBASE-25888
> URL: https://issues.apache.org/jira/browse/HBASE-25888
> Project: HBase
> Issue Type: Bug
> Components: backuprestore, test
>Reporter: Nick Dimiduk
>Assignee: Mallikarjun
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
> Attachments:
> TEST-org.apache.hadoop.hbase.backup.TestBackupDeleteRestore.xml.gz,
> TEST-org.apache.hadoop.hbase.backup.TestBackupMerge.xml.gz,
> TEST-org.apache.hadoop.hbase.backup.TestFullBackupSet.xml.gz,
> TEST-org.apache.hadoop.hbase.backup.TestIncrementalBackupMergeWithFailures.xml.gz,
>
> TEST-org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.xml.gz,
> TEST-org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests.xml.gz,
> org.apache.hadoop.hbase.backup.TestBackupDeleteRestore.txt,
> org.apache.hadoop.hbase.backup.TestBackupDeleteRestore.txt.gz,
> org.apache.hadoop.hbase.backup.TestBackupMerge-output.txt.gz,
> org.apache.hadoop.hbase.backup.TestBackupMerge.txt.gz,
> org.apache.hadoop.hbase.backup.TestFullBackupSet-output.txt.gz,
> org.apache.hadoop.hbase.backup.TestFullBackupSet.txt.gz,
> org.apache.hadoop.hbase.backup.TestIncrementalBackupMergeWithFailures-output.txt.gz,
>
> org.apache.hadoop.hbase.backup.TestIncrementalBackupMergeWithFailures.txt.gz,
> org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad-output.txt.gz,
> org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.txt.gz,
> org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests-output.txt.gz,
> org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests.txt.gz
>
>
> Here's some logs from a PR build vs. master that suffered a significant
> number of failures in the backup tests. I suspect that a single improvement
> could fix all of these tests to be more robust.
> {noformat}
> Test Name
> Duration
> Age
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestBackupDeleteRestore.testBackupDeleteRestore
> 6 min 23 sec1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestBackupDeleteRestore.(?)1 min 6 sec
> 1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestBackupMerge.TestIncBackupMergeRestore
> 5 min 3 sec 1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestBackupMerge.(?)1 min 6 sec 1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestFullBackupSet.testFullBackupSetExist
> 6 min 16 sec1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestFullBackupSet.(?) 1 min 6 sec 1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestIncrementalBackupMergeWithFailures.TestIncBackupMergeRestore
>5 min 55 sec1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestIncrementalBackupMergeWithFailures.(?)
> 1 min 6 sec 1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.TestIncBackupDeleteTable
> 5 min 56 sec1
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.(?) 1 min 6
> sec 1
> precommit checks / yetus jdk8
[GitHub] [hbase] Apache-HBase commented on pull request #3298: HBASE-25907 Move StoreFlushContext out of HStore and make it pluggable
Apache-HBase commented on pull request #3298: URL: https://github.com/apache/hbase/pull/3298#issuecomment-847296624 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-24749 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 18s | HBASE-24749 passed | | +1 :green_heart: | compile | 1m 1s | HBASE-24749 passed | | +1 :green_heart: | shadedjars | 9m 0s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | HBASE-24749 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 6s | the patch passed | | +1 :green_heart: | compile | 1m 3s | the patch passed | | +1 :green_heart: | javac | 1m 3s | the patch passed | | +1 :green_heart: | shadedjars | 9m 1s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 228m 58s | hbase-server in the patch passed. | | | | 262m 16s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3298/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3298 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e376b8dbdc40 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24749 / 77a272609d | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3298/1/testReport/ | | Max. process+thread count | 3720 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3298/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #3139: HBASE-25745 Deprecate/Rename config `hbase.normalizer.min.region.coun…
ndimiduk commented on a change in pull request #3139:
URL: https://github.com/apache/hbase/pull/3139#discussion_r638234965
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -540,24 +546,34 @@ public boolean isMergeEnabled() {
return mergeEnabled;
}
-public int getMinRegionCount() {
- return minRegionCount;
+public int getMergeMinRegionCount() {
+ return mergeMinRegionCount;
}
-public int getMinRegionCount(NormalizeContext context) {
- int minRegionCount = context.getOrDefault(MIN_REGION_COUNT_KEY,
Integer::parseInt, 0);
- if (minRegionCount <= 0) {
-minRegionCount = getMinRegionCount();
+public int getMergeMinRegionCount(NormalizeContext context) {
+ String stringValue = context.getOrDefault(MERGE_MIN_REGION_COUNT_KEY,
+Function.identity(), null);
+ if (stringValue == null) {
+stringValue = context.getOrDefault(MIN_REGION_COUNT_KEY,
Function.identity(), null);
+if (stringValue != null) {
+ LOG.debug("The config key {} in table descriptor is deprecated.
Instead please use {}. "
Review comment:
huh. maybe `TableDescriptor` should handle deprecations as well. For
another patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #3215: HBASE-25698 Fixing IllegalReferenceCountException when using TinyLfuBlockCache
saintstack commented on a change in pull request #3215:
URL: https://github.com/apache/hbase/pull/3215#discussion_r638224781
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/TinyLfuBlockCache.java
##
@@ -158,7 +158,13 @@ public boolean containsBlock(BlockCacheKey cacheKey) {
@Override
public Cacheable getBlock(BlockCacheKey cacheKey,
boolean caching, boolean repeat, boolean updateCacheMetrics) {
-Cacheable value = cache.getIfPresent(cacheKey);
+Cacheable value = cache.asMap().computeIfPresent(cacheKey, (blockCacheKey,
cacheable) -> {
+ // It will be referenced by RPC path, so increase here. NOTICE: Must do
the retain inside
+ // this block. because if retain outside the map#computeIfPresent, the
evictBlock may remove
+ // the block and release, then we're retaining a block with refCnt=0
which is disallowed.
+ cacheable.retain();
+ return cacheable;
+});
Review comment:
Thanks for showing up @ben-manes Let me look at your suggestion...
Its promising.
On tinylfu being 'slower', I'm trying to repro a production workload... What
I have is messy at the moment still in need of work. My test compares were
coarse. I didn't want to say too much more until I had solidified the test
rig (as noted above).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ben-manes commented on a change in pull request #3215: HBASE-25698 Fixing IllegalReferenceCountException when using TinyLfuBlockCache
ben-manes commented on a change in pull request #3215:
URL: https://github.com/apache/hbase/pull/3215#discussion_r638213661
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/TinyLfuBlockCache.java
##
@@ -158,7 +158,13 @@ public boolean containsBlock(BlockCacheKey cacheKey) {
@Override
public Cacheable getBlock(BlockCacheKey cacheKey,
boolean caching, boolean repeat, boolean updateCacheMetrics) {
-Cacheable value = cache.getIfPresent(cacheKey);
+Cacheable value = cache.asMap().computeIfPresent(cacheKey, (blockCacheKey,
cacheable) -> {
+ // It will be referenced by RPC path, so increase here. NOTICE: Must do
the retain inside
+ // this block. because if retain outside the map#computeIfPresent, the
evictBlock may remove
+ // the block and release, then we're retaining a block with refCnt=0
which is disallowed.
+ cacheable.retain();
+ return cacheable;
+});
Review comment:
I suspect that you might be able to switch to an optimistic read, where
you validate that the `cacheable.retain()` was successful. My naive inspection
of the code is that it will throw an `IllegalReferenceCountException`, which
could be caught and returned as a cache miss. Because it is decremented after
the entry was removed from the mapping (in `evictBlock`, though I don't see a
release in the removal listener), the stale read should be fine as a subsequent
read/write would not observe that discarded entry.
In YCSB benchmarks, `TinyLfuBlockCache` is a little bit slower because
Caffeine adds a small overhead on reads to have O(1) evictions with a higher
hit rate. In comparison, the `LruBlockCache` does no policy work on read and
performs an O(n lg n) operation on eviction. Due to using an SLRU policy, the
Zipfian workload has the same optimal hit rate, causing only the small policy
cost to on reads to be exposed.
In a realistic workload, however, one will expect that the TinyLfu policy
should have an improved hit rate which will reduces latencies (more requests
served from cache, reducing I/O and cpu load). The O(1) eviction should be
helpful for large caches when the system in under high load, as it avoids a
spike of work and amortizes the cost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3276: HBASE-25894 Improve the performance for region load and region count related cost functions
Apache-HBase commented on pull request #3276: URL: https://github.com/apache/hbase/pull/3276#issuecomment-847255393 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 21s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 19s | master passed | | +1 :green_heart: | compile | 1m 23s | master passed | | +1 :green_heart: | shadedjars | 9m 37s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 45s | the patch passed | | +1 :green_heart: | compile | 1m 19s | the patch passed | | +1 :green_heart: | javac | 1m 19s | the patch passed | | +1 :green_heart: | shadedjars | 9m 0s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 218m 41s | hbase-server in the patch passed. | | | | 254m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3276 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 099027cdce98 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b02c8102b7 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/testReport/ | | Max. process+thread count | 2569 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] anoopsjohn opened a new pull request #3300: HBASE-25898 RS getting aborted due to NPE in Replication WALEntryStream
anoopsjohn opened a new pull request #3300: URL: https://github.com/apache/hbase/pull/3300 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3298: HBASE-25907 Move StoreFlushContext out of HStore and make it pluggable
Apache-HBase commented on pull request #3298: URL: https://github.com/apache/hbase/pull/3298#issuecomment-847243958 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-24749 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 30s | HBASE-24749 passed | | +1 :green_heart: | compile | 1m 15s | HBASE-24749 passed | | +1 :green_heart: | shadedjars | 8m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | HBASE-24749 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 13s | the patch passed | | +1 :green_heart: | compile | 1m 10s | the patch passed | | +1 :green_heart: | javac | 1m 10s | the patch passed | | +1 :green_heart: | shadedjars | 8m 13s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 140m 43s | hbase-server in the patch passed. | | | | 172m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3298/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3298 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 657a40b7b90c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24749 / 77a272609d | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3298/1/testReport/ | | Max. process+thread count | 4474 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3298/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] pankaj72981 commented on a change in pull request #3297: HBASE-25905 Limit the shutdown time of WAL
pankaj72981 commented on a change in pull request #3297:
URL: https://github.com/apache/hbase/pull/3297#discussion_r638171667
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
##
@@ -145,6 +148,10 @@
public static final String RING_BUFFER_SLOT_COUNT =
"hbase.regionserver.wal.disruptor.event.count";
+ public static final String WAL_SHUTDOWN_WAIT_TIMEOUT_MS =
+"hbase.wal.shutdown.wait.timeout.ms";
+ public static final int DEFAULT_WAL_SHUTDOWN_WAIT_TIMEOUT_MS = 15000;
Review comment:
For readability let's change it to 15 * 1000.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
##
@@ -986,14 +996,43 @@ public void shutdown() throws IOException {
i.logCloseRequested();
}
}
-rollWriterLock.lock();
+
+ExecutorService shutdownExecutor = Executors.newFixedThreadPool(1,
Review comment:
This will be the second single thread ExecutorService in AbstractFSWAL,
can we do it optimized way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] anoopsjohn merged pull request #3292: HBASE-25898 RS getting aborted due to NPE in Replication WALEntryStream
anoopsjohn merged pull request #3292: URL: https://github.com/apache/hbase/pull/3292 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3295: HBASE-25890 [Addendum] create profiles for different jdk version
Apache-HBase commented on pull request #3295: URL: https://github.com/apache/hbase/pull/3295#issuecomment-847230392 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-1 Compile Tests _ | | +0 :ok: | mvndep | 2m 47s | Maven dependency ordering for branch | | -1 :x: | mvninstall | 7m 57s | root in branch-1 failed. | | +1 :green_heart: | compile | 1m 48s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | -1 :x: | compile | 1m 32s | root in branch-1 failed with JDK Azul Systems, Inc.-1.7.0_272-b10. | | -1 :x: | shadedjars | 0m 19s | branch has 7 errors when building our shaded downstream artifacts. | | -1 :x: | javadoc | 2m 19s | root in branch-1 failed with JDK Azul Systems, Inc.-1.8.0_262-b19. | | -1 :x: | javadoc | 0m 8s | hbase-assembly in branch-1 failed with JDK Azul Systems, Inc.-1.8.0_262-b19. | | -1 :x: | javadoc | 2m 40s | root in branch-1 failed with JDK Azul Systems, Inc.-1.7.0_272-b10. | | -1 :x: | javadoc | 0m 11s | hbase-assembly in branch-1 failed with JDK Azul Systems, Inc.-1.7.0_272-b10. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 5s | the patch passed | | +1 :green_heart: | compile | 2m 4s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javac | 2m 4s | the patch passed | | +1 :green_heart: | compile | 2m 10s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | javac | 2m 10s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | -1 :x: | xml | 0m 2s | The patch has 6 ill-formed XML file(s). | | -1 :x: | shadedjars | 0m 13s | patch has 7 errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 5m 29s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 2m 54s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 3m 21s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | ||| _ Other Tests _ | | -1 :x: | unit | 188m 11s | root in the patch failed. | | -1 :x: | asflicense | 0m 52s | The patch generated 9 ASF License warnings. | | | | 232m 21s | | | Reason | Tests | |---:|:--| | XML | Parsing Error(s): | | | hbase-assembly/pom.xml | | | hbase-assembly/src/main/assembly/components-jdk8.xml | | | hbase-assembly/src/main/assembly/components.xml | | | hbase-assembly/src/main/assembly/hadoop-two-compat-jdk8.xml | | | hbase-assembly/src/main/assembly/hadoop-two-compat.xml | | | pom.xml | | Failed junit tests | hadoop.hbase.mapreduce.TestLoadIncrementalHFiles | | | hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint | | | hadoop.hbase.client.TestAdmin1 | | | hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3295/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3295 | | Optional Tests | dupname asflicense javac javadoc unit shadedjars hadoopcheck xml compile | | uname | Linux 28a2fbd9b537 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-3295/out/precommit/personality/provided.sh | | git revision | branch-1 / 3cff107 | | Default Java | Azul Systems, Inc.-1.7.0_272-b10 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3295/2/artifact/out/branch-mvninstall-root.txt | | compile |
[GitHub] [hbase] anoopsjohn commented on pull request #3215: HBASE-25698 Fixing IllegalReferenceCountException when using TinyLfuBlockCache
anoopsjohn commented on pull request #3215: URL: https://github.com/apache/hbase/pull/3215#issuecomment-847220592 Ref count is used in all cases now. This has become a part of the ByteBuff which holds the block data. Ya this was added so as to unify the code. We needed refCount at offheap BC. Later we added a way to read into our pooled DBBs when we read from HDFS. So there also ref counting. But then we suffer from perf! Lets take a step back and see what we can do. As such for this issue, the 3 on heap LRU cache having same code in many parts. (mostly copy paste). Only few parts of change like how/what evicted and not aggressive caching. May be its time to refactor it and have an base class. Else we will repeat code everywhere and might miss changes (like this issue) in future. But may be not easy too.. This change, the latest PR I will see tomorrow . tks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on pull request #3215: HBASE-25698 Fixing IllegalReferenceCountException when using TinyLfuBlockCache
virajjasani commented on pull request #3215: URL: https://github.com/apache/hbase/pull/3215#issuecomment-847217261 Okk looks like refCount was introduced with ByteBuff only, checking HBASE-22005 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on pull request #3215: HBASE-25698 Fixing IllegalReferenceCountException when using TinyLfuBlockCache
virajjasani commented on pull request #3215: URL: https://github.com/apache/hbase/pull/3215#issuecomment-847215393 > Will open an issue to address once an idea on how to address perf (this is secondary to fixing refcounting exceptions). refCount is not used for just L1 cache right? refCount is used only if we use Offheap BucketCache? And if this is the case, perhaps we could achieve CHM#get for L1 onheap cache and CHM#computeIfPresent (as is) for L1+L2 cache? WDYT @anoopsjohn @saintstack ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3276: HBASE-25894 Improve the performance for region load and region count related cost functions
Apache-HBase commented on pull request #3276: URL: https://github.com/apache/hbase/pull/3276#issuecomment-847214523 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 54s | master passed | | +1 :green_heart: | compile | 1m 2s | master passed | | +1 :green_heart: | shadedjars | 8m 39s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 48s | the patch passed | | +1 :green_heart: | compile | 1m 1s | the patch passed | | +1 :green_heart: | javac | 1m 1s | the patch passed | | +1 :green_heart: | shadedjars | 8m 27s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 150m 57s | hbase-server in the patch passed. | | | | 181m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3276 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6022185d17d4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b02c8102b7 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/testReport/ | | Max. process+thread count | 3359 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3276/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3297: HBASE-25905 Limit the shutdown time of WAL
Apache-HBase commented on pull request #3297: URL: https://github.com/apache/hbase/pull/3297#issuecomment-847213651 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 23s | master passed | | +1 :green_heart: | compile | 1m 12s | master passed | | +1 :green_heart: | shadedjars | 9m 33s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 40s | the patch passed | | +1 :green_heart: | compile | 0m 59s | the patch passed | | +1 :green_heart: | javac | 0m 59s | the patch passed | | +1 :green_heart: | shadedjars | 8m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 152m 39s | hbase-server in the patch passed. | | | | 184m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3297/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3297 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1ec8367f5783 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b02c8102b7 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3297/1/testReport/ | | Max. process+thread count | 3887 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3297/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3297: HBASE-25905 Limit the shutdown time of WAL
Apache-HBase commented on pull request #3297: URL: https://github.com/apache/hbase/pull/3297#issuecomment-847207839 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 34s | master passed | | +1 :green_heart: | compile | 1m 13s | master passed | | +1 :green_heart: | shadedjars | 8m 12s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 17s | the patch passed | | +1 :green_heart: | compile | 1m 15s | the patch passed | | +1 :green_heart: | javac | 1m 15s | the patch passed | | +1 :green_heart: | shadedjars | 8m 7s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 143m 9s | hbase-server in the patch passed. | | | | 174m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3297/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3297 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 969de54c2eee 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b02c8102b7 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3297/1/testReport/ | | Max. process+thread count | 3718 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3297/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3206: HBASE-25818 Move StochasticLoadBalancer to hbase-balancer module
Apache-HBase commented on pull request #3206: URL: https://github.com/apache/hbase/pull/3206#issuecomment-847207147 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 53s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 38s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 9s | master passed | | +1 :green_heart: | compile | 4m 12s | master passed | | +1 :green_heart: | checkstyle | 1m 31s | master passed | | +1 :green_heart: | spotbugs | 3m 3s | master passed | | -0 :warning: | patch | 2m 41s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 41s | the patch passed | | +1 :green_heart: | compile | 4m 15s | the patch passed | | -0 :warning: | javac | 0m 29s | hbase-balancer generated 1 new + 30 unchanged - 0 fixed = 31 total (was 30) | | -0 :warning: | checkstyle | 0m 15s | hbase-balancer: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | -0 :warning: | checkstyle | 1m 25s | hbase-server: The patch generated 4 new + 2 unchanged - 4 fixed = 6 total (was 6) | | -0 :warning: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | hadoopcheck | 24m 25s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 46s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | | | | 66m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3206 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux f1f3af2af690 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b02c8102b7 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/5/artifact/yetus-general-check/output/diff-compile-javac-hbase-balancer.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-balancer.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | whitespace | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/5/artifact/yetus-general-check/output/whitespace-eol.txt | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/5/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25893) Corruption in recovered WAL in WALSplitter
[
https://issues.apache.org/jira/browse/HBASE-25893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350548#comment-17350548
]
Rushabh Shah commented on HBASE-25893:
--
> Applicable only for branch-1?
[~anoop.hbase] Yes. could you please review the PR ?
https://github.com/apache/hbase/pull/3277
Thank you !
> Corruption in recovered WAL in WALSplitter
> --
>
> Key: HBASE-25893
> URL: https://issues.apache.org/jira/browse/HBASE-25893
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, wal
>Affects Versions: 1.6.0
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Critical
>
> Recently we encountered RS aborts due to NPE while replaying edits from split
> logs during region open.
> {noformat}
> 2021-05-13 19:34:28,871 ERROR [:60020-17] handler.OpenRegionHandler
> - Failed open of
> region=,1619036437822.0556ab96be88000b6f5f3fad47938ccd., starting
> to roll back the global memstore size.
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:411)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:4682)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsForPaths(HRegion.java:4557)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:4470)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:949)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:908)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7253)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7214)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7185)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7141)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7092)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:364)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:131)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> Tracing back how the corrupt wal was generated.
> {noformat}
> 2021-05-12 05:21:23,333 FATAL [:60020-0-Writer-1] wal.WALSplitter
> - 556ab96be88000b6f5f3fad47938ccd/5039807= to log
> java.nio.channels.ClosedChannelException
> at
> org.apache.hadoop.hdfs.DataStreamer$LastExceptionInStreamer.throwException4Close(DataStreamer.java:331)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:151)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:105)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at org.apache.hadoop.hbase.KeyValue.write(KeyValue.java:2543)
> at
> org.apache.phoenix.hbase.index.wal.KeyValueCodec.write(KeyValueCodec.java:104)
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$IndexKeyValueEncoder.write(IndexedWALEditCodec.java:218)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.append(ProtobufLogWriter.java:128)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.appendBuffer(WALSplitter.java:1742)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.append(WALSplitter.java:1714)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.writeBuffer(WALSplitter.java:1179)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.doRun(WALSplitter.java:1171)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.run(WALSplitter.java:1141)
> 2021-05-12 05:21:23,333 ERROR [:60020-0-Writer-1] wal.WALSplitter -
> Exiting thread
> java.nio.channels.ClosedChannelException
> at
> org.apache.hadoop.hdfs.DataStreamer$LastExceptionInStreamer.throwException4Close(DataStreamer.java:331)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:151)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:105)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
>
[GitHub] [hbase] saintstack commented on a change in pull request #3215: HBASE-25698 Fixing IllegalReferenceCountException when using TinyLfuBlockCache
saintstack commented on a change in pull request #3215:
URL: https://github.com/apache/hbase/pull/3215#discussion_r638134220
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/TinyLfuBlockCache.java
##
@@ -158,7 +158,13 @@ public boolean containsBlock(BlockCacheKey cacheKey) {
@Override
public Cacheable getBlock(BlockCacheKey cacheKey,
boolean caching, boolean repeat, boolean updateCacheMetrics) {
-Cacheable value = cache.getIfPresent(cacheKey);
+Cacheable value = cache.asMap().computeIfPresent(cacheKey, (blockCacheKey,
cacheable) -> {
+ // It will be referenced by RPC path, so increase here. NOTICE: Must do
the retain inside
+ // this block. because if retain outside the map#computeIfPresent, the
evictBlock may remove
+ // the block and release, then we're retaining a block with refCnt=0
which is disallowed.
+ cacheable.retain();
+ return cacheable;
+});
Review comment:
Thanks @virajjasani ... I just read over the HBASE-22422. Nice work by
@openinx . Agree need locking but currently the lock spans more than the
buffer... covering CHM bucket. We might be able to scope the lock to the
buffer... but I'm not sure and a little afraid to touch the code here.
On the sawtooth, I've not looked.
Not using offheap. All onheap but in async-profiler, the
CHM#computeIfPresent is the main blocking point by far (Trying to drive up the
throughput when inmemory meta lookups).
On this patch generally, +1. I'd noticed messing w/ this stuff that tinylfu
was missing this bit of code... I'd also noticed that the locking profile with
tinylfu in place looked much better I'd failed to put the two bits of
info together. My sense is that once this goes in, that tinylfu will be the
main blocker... just like CHM is now.
Oddly, for me, tinylfu seemed to run slower in my compares which I didn't
expect given the nice batching it does and its WAL scheme. Its probably
variance in my test rig. Working on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3299: HBASE-25908 Exclude jakarta.activation-api
Apache-HBase commented on pull request #3299: URL: https://github.com/apache/hbase/pull/3299#issuecomment-847204515 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 7s | master passed | | +1 :green_heart: | compile | 0m 52s | master passed | | +1 :green_heart: | shadedjars | 11m 33s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 33s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 4s | the patch passed | | +1 :green_heart: | compile | 0m 50s | the patch passed | | +1 :green_heart: | javac | 0m 50s | the patch passed | | +1 :green_heart: | shadedjars | 10m 55s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 23s | hbase-shaded in the patch passed. | | +1 :green_heart: | unit | 0m 18s | hbase-shaded-client in the patch passed. | | | | 40m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3299/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3299 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux aa11204d3f8a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b02c8102b7 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3299/1/testReport/ | | Max. process+thread count | 489 (vs. ulimit of 3) | | modules | C: hbase-shaded hbase-shaded/hbase-shaded-client U: hbase-shaded | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3299/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3299: HBASE-25908 Exclude jakarta.activation-api
Apache-HBase commented on pull request #3299: URL: https://github.com/apache/hbase/pull/3299#issuecomment-847202415 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 52s | master passed | | +1 :green_heart: | compile | 0m 51s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 40s | the patch passed | | +1 :green_heart: | compile | 0m 49s | the patch passed | | +1 :green_heart: | javac | 0m 49s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 3s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 18m 9s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | | | | 36m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3299/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3299 | | Optional Tests | dupname asflicense javac hadoopcheck xml compile | | uname | Linux d033fb25041a 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b02c8102b7 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-shaded hbase-shaded/hbase-shaded-client U: hbase-shaded | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3299/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25901) Replication may lose data during regionserver restart when multiwal is in use
[
https://issues.apache.org/jira/browse/HBASE-25901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350536#comment-17350536
]
Hudson commented on HBASE-25901:
Results for branch branch-2.3
[build #223 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/223/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/223/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/223/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/223/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/223/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Replication may lose data during regionserver restart when multiwal is in use
> -
>
> Key: HBASE-25901
> URL: https://issues.apache.org/jira/browse/HBASE-25901
> Project: HBase
> Issue Type: Bug
> Components: multiwal, Replication
>Affects Versions: 2.1.4
> Environment: hbase version: 2.1.4
> hbase.wal.provider: multiwal
>
>Reporter: youhailang
>Assignee: youhailang
>Priority: Major
> Fix For: 2.3.6
>
> Attachments: HBASE-25901-v1.patch
>
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> When multiwal is used, restarting the region server may result in the loss of
> replication data.This may be due to a non thread safe update of the queues in
> the ReplicationSource#
> enqueueLog.
> .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3299: HBASE-25908 Exclude jakarta.activation-api
Apache-HBase commented on pull request #3299: URL: https://github.com/apache/hbase/pull/3299#issuecomment-847199527 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 26s | master passed | | +1 :green_heart: | compile | 0m 48s | master passed | | +1 :green_heart: | shadedjars | 8m 16s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 32s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 17s | the patch passed | | +1 :green_heart: | compile | 0m 47s | the patch passed | | +1 :green_heart: | javac | 0m 47s | the patch passed | | +1 :green_heart: | shadedjars | 8m 18s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 30s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 57s | hbase-shaded in the patch passed. | | +1 :green_heart: | unit | 0m 17s | hbase-shaded-client in the patch passed. | | | | 31m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3299/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3299 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f5fbe526ce20 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b02c8102b7 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3299/1/testReport/ | | Max. process+thread count | 493 (vs. ulimit of 3) | | modules | C: hbase-shaded hbase-shaded/hbase-shaded-client U: hbase-shaded | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3299/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on a change in pull request #3215: HBASE-25698 Fixing IllegalReferenceCountException when using TinyLfuBlockCache
virajjasani commented on a change in pull request #3215:
URL: https://github.com/apache/hbase/pull/3215#discussion_r638125789
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/TinyLfuBlockCache.java
##
@@ -158,7 +158,13 @@ public boolean containsBlock(BlockCacheKey cacheKey) {
@Override
public Cacheable getBlock(BlockCacheKey cacheKey,
boolean caching, boolean repeat, boolean updateCacheMetrics) {
-Cacheable value = cache.getIfPresent(cacheKey);
+Cacheable value = cache.asMap().computeIfPresent(cacheKey, (blockCacheKey,
cacheable) -> {
+ // It will be referenced by RPC path, so increase here. NOTICE: Must do
the retain inside
+ // this block. because if retain outside the map#computeIfPresent, the
evictBlock may remove
+ // the block and release, then we're retaining a block with refCnt=0
which is disallowed.
+ cacheable.retain();
+ return cacheable;
+});
Review comment:
Hmm, yeah locks would slow us down. On the other hand, based on
discussion on HBASE-22422 , it seems computeIfPresent (locking) is necessary to
prevent concurrency issues with #retain and #release.
Based on @openinx's comment
[here](https://issues.apache.org/jira/browse/HBASE-22422?focusedCommentId=16848024=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16848024),
wondering if the sawtooth graph of QPS is similar concurrency issue and not
resolved yet.
@saintstack Any suggestions? Have you been using Offheap read path with LRU
recently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] jojochuang opened a new pull request #3299: HBASE-25908 Exclude jakarta.activation-api
jojochuang opened a new pull request #3299: URL: https://github.com/apache/hbase/pull/3299 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
