[jira] [Updated] (HBASE-25694) Improve the shell's 'status replication' command output
[ https://issues.apache.org/jira/browse/HBASE-25694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xi chaomin updated HBASE-25694: --- Attachment: HBASE-25694-2.4.patch > Improve the shell's 'status replication' command output > --- > > Key: HBASE-25694 > URL: https://issues.apache.org/jira/browse/HBASE-25694 > Project: HBase > Issue Type: Task > Components: Operability, Replication, shell >Affects Versions: 2.4.2 >Reporter: Andrew Kyle Purtell >Assignee: xi chaomin >Priority: Major > Fix For: 2.6.0, 3.0.0-alpha-3 > > Attachments: HBASE-25694-2.4.patch > > > The output of the shell's 'status "replication"' command could use some > cleaning up. > The basic format is: > version VERSION > N live servers > SERVER_1: >SOURCE: PeerID=PEERID, AgeOfLastShippedOp=160347, SizeOfLogQueue=49, > TimeStampsOfLastShippedOp=Thu Mar 25 01:55:21 UTC 2021, Replication Lag=161013 >SINK : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Thu Mar 25 > 01:49:53 UTC 2021 > ... > Issues: > - Per line format is KEY=VALUE, ... . Most KEYs are CamelCase but some have > spaces. Should all be camel case for consistency. > - Age is printed as long and timestamp is string (local TZ) format. I think > the string formatting of the timestamp makes it difficult to read. Lets just > print timestamps as longs. Or provide a non-default option for string > formatted timestamps. > - There is no sense of effective rate or bandwidth. Each source and sink line > emitted should provide some sense of current workload: rpcs/sec, bytes/sec, > cells/sec... something. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] Apache-HBase commented on pull request #4403: HBASE-26342 Support custom paths of independent configuration and pool for hfile cleaner
Apache-HBase commented on PR #4403: URL: https://github.com/apache/hbase/pull/4403#issuecomment-1118149174 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 37s | master passed | | +1 :green_heart: | compile | 2m 12s | master passed | | +1 :green_heart: | checkstyle | 0m 29s | master passed | | +1 :green_heart: | spotless | 0m 41s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 16s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 7s | the patch passed | | +1 :green_heart: | compile | 2m 15s | the patch passed | | +1 :green_heart: | javac | 2m 15s | the patch passed | | -0 :warning: | checkstyle | 0m 27s | hbase-server: The patch generated 2 new + 8 unchanged - 0 fixed = 10 total (was 8) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 8s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | -1 :x: | spotless | 0m 33s | patch has 65 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 20s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 32m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4403/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4403 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux cbb029e52a52 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 64a6ba3647 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4403/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4403/1/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 61 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4403/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26862) Fix CME when closing region
[
https://issues.apache.org/jira/browse/HBASE-26862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaolin Ha updated HBASE-26862:
---
Resolution: Won't Fix
Status: Resolved (was: Patch Available)
This error is introduced by branch-1, and I find there are some differences
between the upper layer scanner close codes of branch-1 and branch-2+. There is
no need for branch-2+ to fix this issue. Just close it now.
> Fix CME when closing region
> ---
>
> Key: HBASE-26862
> URL: https://issues.apache.org/jira/browse/HBASE-26862
> Project: HBase
> Issue Type: Bug
> Components: regionserver, Scanners
>Affects Versions: 2.4.11
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Minor
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.13
>
>
> KeyValueHeap.scannersForDelayedClose is not thread safe, CME might occurs
> when closing a scanning region, who is adding scanners to the delayed close
> list.
> {code:java}
> 2022-03-18 17:24:17,435 INFO [regionserver/hostname/ip:16020-splits-0]
> regionserver.SplitRequest: Running rollback/cleanup of failed split of
> hmart-waimaiups_poi_bitmap_theme_pool,,1645769651132.56363df04da85752aa67669989de19fd.;
> java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1767)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1593)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:394)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:276)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:567)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:156)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
> at java.util.ArrayList$Itr.next(ArrayList.java:851)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:221)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:548)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:520)
> at
> org.apache.hadoop.hbase.regionserver.HStore.closeChangedReadersObservers(HStore.java:1296)
> at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:1001)
> at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:131)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1745)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1741)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [hbase] sunhelly closed pull request #4247: HBASE-26862 Fix CME when closing region
sunhelly closed pull request #4247: HBASE-26862 Fix CME when closing region URL: https://github.com/apache/hbase/pull/4247 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26342) Support custom paths of independent configuration and pool for hfile cleaner
[ https://issues.apache.org/jira/browse/HBASE-26342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaolin Ha updated HBASE-26342: --- Summary: Support custom paths of independent configuration and pool for hfile cleaner (was: Supports custom paths of independent configuration and pool for hfile cleaner) > Support custom paths of independent configuration and pool for hfile cleaner > > > Key: HBASE-26342 > URL: https://issues.apache.org/jira/browse/HBASE-26342 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 1.7.1, 3.0.0-alpha-2, 2.4.10 >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3 > > Attachments: result.png > > > With this, we can clean some paths more quickly. > We found in our cluster, when the very huge table with thousands of regions > and high write throughputs and many snapshots tables on the same cluster, the > speed of delete files in archive path will lower than the speed of moved in > files by compaction. Then archive may remains PB level data. > The bottleneck is in cleaner but not in the thread pool size or queue size. > It is because there is synchronized lock in SnapshotFileCache, and a batch of > files need once SnapshotFileCache#refreshCache(), which look through all the > snapshot dirs. > The speed of clear a path without the SnapshotHFileCleaner is thirty times > faster. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] sunhelly closed pull request #4402: HBASE-26342 cleaner supports custom paths of independent configuration and pool for hfile cleaner
sunhelly closed pull request #4402: HBASE-26342 cleaner supports custom paths of independent configuration and pool for hfile cleaner URL: https://github.com/apache/hbase/pull/4402 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4401: Porting HBASE-25694 to 2.4
Apache-HBase commented on PR #4401: URL: https://github.com/apache/hbase/pull/4401#issuecomment-1118129907 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 47s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 28s | branch-2.4 passed | | +1 :green_heart: | javadoc | 0m 11s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 15s | the patch passed | | +1 :green_heart: | javadoc | 0m 8s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 7m 24s | hbase-shell in the patch passed. | | | | 15m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4401/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4401 | | Optional Tests | javac javadoc unit | | uname | Linux 27a3db520cd7 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / 58aecd4f94 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4401/1/testReport/ | | Max. process+thread count | 1745 (vs. ulimit of 12500) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4401/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4401: Porting HBASE-25694 to 2.4
Apache-HBase commented on PR #4401: URL: https://github.com/apache/hbase/pull/4401#issuecomment-1118129334 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 5s | branch-2.4 passed | | +1 :green_heart: | javadoc | 0m 12s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 54s | the patch passed | | +1 :green_heart: | javadoc | 0m 10s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 7m 26s | hbase-shell in the patch passed. | | | | 14m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4401/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4401 | | Optional Tests | javac javadoc unit | | uname | Linux 352b7651b218 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / 58aecd4f94 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4401/1/testReport/ | | Max. process+thread count | 1654 (vs. ulimit of 12500) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4401/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4401: Porting HBASE-25694 to 2.4
Apache-HBase commented on PR #4401: URL: https://github.com/apache/hbase/pull/4401#issuecomment-1118126511 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 54s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 19s | branch-2.4 passed | | +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 46s | the patch passed | | -0 :warning: | rubocop | 0m 12s | The patch generated 16 new + 492 unchanged - 17 fixed = 508 total (was 509) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | spotless | 0m 37s | patch has no errors when running spotless:check. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 9s | The patch does not generate ASF License warnings. | | | | 8m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4401/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4401 | | Optional Tests | dupname asflicense javac spotless rubocop | | uname | Linux 522e47157e7d 5.4.0-1068-aws #72~18.04.1-Ubuntu SMP Thu Mar 3 08:49:49 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / 58aecd4f94 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | rubocop | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4401/1/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 64 (vs. ulimit of 12500) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4401/1/console | | versions | git=2.17.1 maven=3.6.3 rubocop=0.80.0 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a diff in pull request #4390: HBASE-26987 The length of compact queue grows too big when the compac…
Apache9 commented on code in PR #4390: URL: https://github.com/apache/hbase/pull/4390#discussion_r865528662 ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java: ## @@ -347,7 +347,6 @@ public class HRegion implements HeapSize, PropagatingConfigurationObserver, Regi final LongAdder compactionsFailed = new LongAdder(); final LongAdder compactionNumFilesCompacted = new LongAdder(); final LongAdder compactionNumBytesCompacted = new LongAdder(); - final LongAdder compactionsQueued = new LongAdder(); Review Comment: It seems a bit strange that we only move this field to HStore but still keep other compaction related metrics in HRegion... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26342) Supports custom paths of independent configuration and pool for hfile cleaner
[ https://issues.apache.org/jira/browse/HBASE-26342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaolin Ha updated HBASE-26342: --- Summary: Supports custom paths of independent configuration and pool for hfile cleaner (was: cleaner supports custom paths of independent configuration and pool) > Supports custom paths of independent configuration and pool for hfile cleaner > - > > Key: HBASE-26342 > URL: https://issues.apache.org/jira/browse/HBASE-26342 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 1.7.1, 3.0.0-alpha-2, 2.4.10 >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3 > > Attachments: result.png > > > With this, we can clean some paths more quickly. > We found in our cluster, when the very huge table with thousands of regions > and high write throughputs and many snapshots tables on the same cluster, the > speed of delete files in archive path will lower than the speed of moved in > files by compaction. Then archive may remains PB level data. > The bottleneck is in cleaner but not in the thread pool size or queue size. > It is because there is synchronized lock in SnapshotFileCache, and a batch of > files need once SnapshotFileCache#refreshCache(), which look through all the > snapshot dirs. > The speed of clear a path without the SnapshotHFileCleaner is thirty times > faster. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] sunhelly closed pull request #3739: HBASE-26342 cleaner supports custom paths of independent configuratio…
sunhelly closed pull request #3739: HBASE-26342 cleaner supports custom paths of independent configuratio… URL: https://github.com/apache/hbase/pull/3739 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26987) The length of compact queue grows too big when the compacting is slow
[ https://issues.apache.org/jira/browse/HBASE-26987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Wang updated HBASE-26987: --- External issue URL: (was: https://issues.apache.org/jira/browse/HBASE-8665) > The length of compact queue grows too big when the compacting is slow > - > > Key: HBASE-26987 > URL: https://issues.apache.org/jira/browse/HBASE-26987 > Project: HBase > Issue Type: Improvement >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Attachments: image-2022-04-29-10-26-09-351.png, > image-2022-04-29-10-26-18-323.png, image-2022-04-29-10-26-24-087.png > > > For some system compaction, we set the selectNow to false, so the file > selecting will not be done until the compaction running, it brings side > effect, if another compacting is slow, we may put lots of compaction to > queue, because the filesCompacting of Hstore is empty in the meantime. > An example shows at attachments, there are 154 regions and about 2000 hfiles, > but the length of compact queue grows to 1391, it cause confusion and may > trigger unexpected alarm. > My approach is limit the compaction queue count, by compute the > filesNotCompating and hbase.hstore.compaction.max. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HBASE-26987) The length of compact queue grows too big when the compacting is slow
[ https://issues.apache.org/jira/browse/HBASE-26987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Wang updated HBASE-26987: --- External issue URL: https://issues.apache.org/jira/browse/HBASE-8665 > The length of compact queue grows too big when the compacting is slow > - > > Key: HBASE-26987 > URL: https://issues.apache.org/jira/browse/HBASE-26987 > Project: HBase > Issue Type: Improvement >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Attachments: image-2022-04-29-10-26-09-351.png, > image-2022-04-29-10-26-18-323.png, image-2022-04-29-10-26-24-087.png > > > For some system compaction, we set the selectNow to false, so the file > selecting will not be done until the compaction running, it brings side > effect, if another compacting is slow, we may put lots of compaction to > queue, because the filesCompacting of Hstore is empty in the meantime. > An example shows at attachments, there are 154 regions and about 2000 hfiles, > but the length of compact queue grows to 1391, it cause confusion and may > trigger unexpected alarm. > My approach is limit the compaction queue count, by compute the > filesNotCompating and hbase.hstore.compaction.max. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26987) The length of compact queue grows too big when the compacting is slow
[ https://issues.apache.org/jira/browse/HBASE-26987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531999#comment-17531999 ] Zheng Wang commented on HBASE-26987: BTW, this issue seems introduced by HBASE-8665. > The length of compact queue grows too big when the compacting is slow > - > > Key: HBASE-26987 > URL: https://issues.apache.org/jira/browse/HBASE-26987 > Project: HBase > Issue Type: Improvement >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Attachments: image-2022-04-29-10-26-09-351.png, > image-2022-04-29-10-26-18-323.png, image-2022-04-29-10-26-24-087.png > > > For some system compaction, we set the selectNow to false, so the file > selecting will not be done until the compaction running, it brings side > effect, if another compacting is slow, we may put lots of compaction to > queue, because the filesCompacting of Hstore is empty in the meantime. > An example shows at attachments, there are 154 regions and about 2000 hfiles, > but the length of compact queue grows to 1391, it cause confusion and may > trigger unexpected alarm. > My approach is limit the compaction queue count, by compute the > filesNotCompating and hbase.hstore.compaction.max. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] (HBASE-26987) The length of compact queue grows too big when the compacting is slow
[ https://issues.apache.org/jira/browse/HBASE-26987 ] Zheng Wang deleted comment on HBASE-26987: was (Author: filtertip): Before this issue resolved, i think we should have a config to disable this feature. > The length of compact queue grows too big when the compacting is slow > - > > Key: HBASE-26987 > URL: https://issues.apache.org/jira/browse/HBASE-26987 > Project: HBase > Issue Type: Improvement >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Attachments: image-2022-04-29-10-26-09-351.png, > image-2022-04-29-10-26-18-323.png, image-2022-04-29-10-26-24-087.png > > > For some system compaction, we set the selectNow to false, so the file > selecting will not be done until the compaction running, it brings side > effect, if another compacting is slow, we may put lots of compaction to > queue, because the filesCompacting of Hstore is empty in the meantime. > An example shows at attachments, there are 154 regions and about 2000 hfiles, > but the length of compact queue grows to 1391, it cause confusion and may > trigger unexpected alarm. > My approach is limit the compaction queue count, by compute the > filesNotCompating and hbase.hstore.compaction.max. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] Apache9 commented on pull request #4361: HBASE-26963 ReplicationSource#removePeer hangs if we try to remove bad peer.
Apache9 commented on PR #4361: URL: https://github.com/apache/hbase/pull/4361#issuecomment-1118071350 I need to review HBASE-24877 first to see what is the purpose of this piece of code... Give me some time. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] shahrs87 commented on pull request #4361: HBASE-26963 ReplicationSource#removePeer hangs if we try to remove bad peer.
shahrs87 commented on PR #4361: URL: https://github.com/apache/hbase/pull/4361#issuecomment-1118019495 Looks like Wellington is busy/unavailable. @Apache9 Do you have any suggestions on this patch ? @apurtell @virajjasani Can you please help review this patch ? Thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-26997) Auto renew scanner lease in TableRecordReader
[ https://issues.apache.org/jira/browse/HBASE-26997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault resolved HBASE-26997. --- Resolution: Not A Problem I'm going to close this actually. I realized that the retries of UnknownScannerException are actually scoped to the RPC, not the overall scanner itself. So it should theoretically work to exceed the scanner lease timeout for every single next() call and the job should still continue. This may not be the most efficient thing, but you're already not doing the most efficient thing by this point and calling renewLease probably won't improve that. > Auto renew scanner lease in TableRecordReader > - > > Key: HBASE-26997 > URL: https://issues.apache.org/jira/browse/HBASE-26997 > Project: HBase > Issue Type: New Feature >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > Labels: patch-available > > A common problem with hadoop jobs is when the mapper takes too long to > process individual inputs. This is especially problematic with > TableInputFormat because if you don't process a scanner.next() batch within > the scanner timeout period your job will fail with UnknownScannerException. > The fix here is usually to reduce Scan.setCaching, so that fewer rows are > returned within each batch. This isn't always a great solution because maybe > not all batches are uniform in their processing time, or maybe even > processing a single row (the smallest caching size) might take a while. > We can improve this for users by providing a configurable period at which the > TableRecordReader will automatically call scanner.renewLease() unless next() > was recently called. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HBASE-26997) Auto renew scanner lease in TableRecordReader
[ https://issues.apache.org/jira/browse/HBASE-26997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26997: -- Status: Open (was: Patch Available) > Auto renew scanner lease in TableRecordReader > - > > Key: HBASE-26997 > URL: https://issues.apache.org/jira/browse/HBASE-26997 > Project: HBase > Issue Type: New Feature >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > Labels: patch-available > > A common problem with hadoop jobs is when the mapper takes too long to > process individual inputs. This is especially problematic with > TableInputFormat because if you don't process a scanner.next() batch within > the scanner timeout period your job will fail with UnknownScannerException. > The fix here is usually to reduce Scan.setCaching, so that fewer rows are > returned within each batch. This isn't always a great solution because maybe > not all batches are uniform in their processing time, or maybe even > processing a single row (the smallest caching size) might take a while. > We can improve this for users by providing a configurable period at which the > TableRecordReader will automatically call scanner.renewLease() unless next() > was recently called. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] bbeaudreault commented on pull request #4400: HBASE-26997 Auto renew scanner lease in TableRecordReader
bbeaudreault commented on PR #4400: URL: https://github.com/apache/hbase/pull/4400#issuecomment-1117803954 I'm going to close this actually. I realized that the retries of UnknownScannerException are actually scoped to the RPC, not the overall scanner itself. So it should theoretically work to exceed the scanner lease timeout for every single next() call and the job should still continue. This may not be the most efficient thing, but you're already not doing the most efficient thing by this point and calling renewLease probably won't improve that. Going to copy this into the JIRA as well and resolve as Not A Problem -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bbeaudreault closed pull request #4400: HBASE-26997 Auto renew scanner lease in TableRecordReader
bbeaudreault closed pull request #4400: HBASE-26997 Auto renew scanner lease in TableRecordReader URL: https://github.com/apache/hbase/pull/4400 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4400: HBASE-26997 Auto renew scanner lease in TableRecordReader
Apache-HBase commented on PR #4400: URL: https://github.com/apache/hbase/pull/4400#issuecomment-1117764706 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 44s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 15s | branch-2 passed | | +1 :green_heart: | compile | 0m 17s | branch-2 passed | | +1 :green_heart: | shadedjars | 4m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 15s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 33s | the patch passed | | +1 :green_heart: | compile | 0m 17s | the patch passed | | +1 :green_heart: | javac | 0m 17s | the patch passed | | +1 :green_heart: | shadedjars | 4m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 12s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 12m 50s | hbase-mapreduce in the patch failed. | | | | 30m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4400/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4400 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux abef5db1cb7e 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / a0258441ab | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4400/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-mapreduce.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4400/1/testReport/ | | Max. process+thread count | 2955 (vs. ulimit of 12500) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4400/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4400: HBASE-26997 Auto renew scanner lease in TableRecordReader
Apache-HBase commented on PR #4400: URL: https://github.com/apache/hbase/pull/4400#issuecomment-1117758766 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 54s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 7s | branch-2 passed | | +1 :green_heart: | compile | 0m 15s | branch-2 passed | | +1 :green_heart: | shadedjars | 3m 50s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 13s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 6s | the patch passed | | +1 :green_heart: | compile | 0m 15s | the patch passed | | +1 :green_heart: | javac | 0m 15s | the patch passed | | +1 :green_heart: | shadedjars | 3m 50s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 11s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 12m 40s | hbase-mapreduce in the patch failed. | | | | 29m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4400/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4400 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9983c0f56e03 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / a0258441ab | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4400/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-mapreduce.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4400/1/testReport/ | | Max. process+thread count | 2952 (vs. ulimit of 12500) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4400/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4400: HBASE-26997 Auto renew scanner lease in TableRecordReader
Apache-HBase commented on PR #4400: URL: https://github.com/apache/hbase/pull/4400#issuecomment-1117741962 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 26s | branch-2 passed | | +1 :green_heart: | compile | 0m 33s | branch-2 passed | | +1 :green_heart: | checkstyle | 0m 13s | branch-2 passed | | +1 :green_heart: | spotless | 0m 45s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 0m 31s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 7s | the patch passed | | +1 :green_heart: | compile | 0m 32s | the patch passed | | -0 :warning: | javac | 0m 32s | hbase-mapreduce generated 1 new + 197 unchanged - 1 fixed = 198 total (was 198) | | +1 :green_heart: | checkstyle | 0m 12s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 7m 53s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotless | 0m 42s | patch has no errors when running spotless:check. | | -1 :x: | spotbugs | 0m 36s | hbase-mapreduce generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 23m 14s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-mapreduce | | | Inconsistent synchronization of org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.htable; locked 50% of time Unsynchronized access at TableRecordReaderImpl.java:50% of time Unsynchronized access at TableRecordReaderImpl.java:[line 187] | | | Inconsistent synchronization of org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.renewLeasePeriod; locked 80% of time Unsynchronized access at TableRecordReaderImpl.java:80% of time Unsynchronized access at TableRecordReaderImpl.java:[line 186] | | | Inconsistent synchronization of org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.scanner; locked 88% of time Unsynchronized access at TableRecordReaderImpl.java:88% of time Unsynchronized access at TableRecordReaderImpl.java:[line 337] | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4400/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4400 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 19fb9ca2d391 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / a0258441ab | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4400/1/artifact/yetus-general-check/output/diff-compile-javac-hbase-mapreduce.txt | | spotbugs | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4400/1/artifact/yetus-general-check/output/new-spotbugs-hbase-mapreduce.html | | Max. process+thread count | 65 (vs. ulimit of 12500) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4400/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bbeaudreault commented on a diff in pull request #4400: HBASE-26997 Auto renew scanner lease in TableRecordReader
bbeaudreault commented on code in PR #4400:
URL: https://github.com/apache/hbase/pull/4400#discussion_r865162848
##
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java:
##
@@ -66,12 +76,18 @@ public class TableRecordReaderImpl {
private boolean logScannerActivity = false;
private int logPerRowCount = 100;
+ private Duration renewLeasePeriod = null;
+
+ private Instant lastScannerRpcAt = null;
+
+ private ScheduledExecutorService renewTaskExecutor = null;
+
/**
* Restart from survivable exceptions by creating a new scanner.
* @param firstRow The first row to start at.
* @throws IOException When restarting fails.
*/
- public void restart(byte[] firstRow) throws IOException {
+ public synchronized void restart(byte[] firstRow) throws IOException {
Review Comment:
these 3 methods are synchronized because it probably would not be good if
renewLease were allowed to execute while creating the scanner or calling
`scanner.next`. the contention should be low since it's only 2 threads, one of
which only executes on the configured interval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bbeaudreault commented on a diff in pull request #4400: HBASE-26997 Auto renew scanner lease in TableRecordReader
bbeaudreault commented on code in PR #4400:
URL: https://github.com/apache/hbase/pull/4400#discussion_r865162848
##
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java:
##
@@ -66,12 +76,18 @@ public class TableRecordReaderImpl {
private boolean logScannerActivity = false;
private int logPerRowCount = 100;
+ private Duration renewLeasePeriod = null;
+
+ private Instant lastScannerRpcAt = null;
+
+ private ScheduledExecutorService renewTaskExecutor = null;
+
/**
* Restart from survivable exceptions by creating a new scanner.
* @param firstRow The first row to start at.
* @throws IOException When restarting fails.
*/
- public void restart(byte[] firstRow) throws IOException {
+ public synchronized void restart(byte[] firstRow) throws IOException {
Review Comment:
these 3 methods are synchronized because it probably would not be good if
renewLease were allowed to execute while creating the scanner or calling
`scanner.next`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-26997) Auto renew scanner lease in TableRecordReader
[ https://issues.apache.org/jira/browse/HBASE-26997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26997: -- Labels: patch-available (was: ) Status: Patch Available (was: Open) > Auto renew scanner lease in TableRecordReader > - > > Key: HBASE-26997 > URL: https://issues.apache.org/jira/browse/HBASE-26997 > Project: HBase > Issue Type: New Feature >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > Labels: patch-available > > A common problem with hadoop jobs is when the mapper takes too long to > process individual inputs. This is especially problematic with > TableInputFormat because if you don't process a scanner.next() batch within > the scanner timeout period your job will fail with UnknownScannerException. > The fix here is usually to reduce Scan.setCaching, so that fewer rows are > returned within each batch. This isn't always a great solution because maybe > not all batches are uniform in their processing time, or maybe even > processing a single row (the smallest caching size) might take a while. > We can improve this for users by providing a configurable period at which the > TableRecordReader will automatically call scanner.renewLease() unless next() > was recently called. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] bbeaudreault opened a new pull request, #4400: HBASE-26997 Auto renew scanner lease in TableRecordReader
bbeaudreault opened a new pull request, #4400: URL: https://github.com/apache/hbase/pull/4400 This feature only really applies to hbase 2.x, since it seems impossible to cause the async scanner in 3.x to hit UnknownScannerException. If spurious long mapper calls occur, one can mitigate UnknownScannerException by upping the hbase.client.retries.number. If mappers are consistently slow enough to trigger the exception, this new feature can help avoid them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24408) Introduce a general 'local region' to store data on master
[ https://issues.apache.org/jira/browse/HBASE-24408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531866#comment-17531866 ] ruanhui commented on HBASE-24408: - hi all Since we have MasterLocalRegion now, I think we can put some data that was previously on zookeeper in the MasterLocalRegion, like the balancer switch, mergeOrsplit switch, region normalizer switch. This is also a attempt to make hbase less dependent on zk. Do you think this is worthwhile work? > Introduce a general 'local region' to store data on master > -- > > Key: HBASE-24408 > URL: https://issues.apache.org/jira/browse/HBASE-24408 > Project: HBase > Issue Type: Task > Components: master >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1, 2.3.0 > > > We already have a local region to store the procedure data and when > implementing HBASE-11288, splittable meta, we are thinking of also storing > the data for root table in a local region. > Now in the patch for HBASE-24388, we introduced another local region to store > the data for root table, but maybe it is better to store the procedure data > and root table together in a single region(with different families). > And this should be done before 2.3.0, to prevent shipping the procedure store > region out in a release. Set it a blocker for 2.3.0. > Patch will be available soon. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] bharathv commented on pull request #4382: HBASE-26925 Create WAL event tracker table to track all the WAL events.
bharathv commented on PR #4382: URL: https://github.com/apache/hbase/pull/4382#issuecomment-1117607666 @shahrs87 Thanks for putting this together. Lets do the work in a feature branch to make it easier for the community to review all the patches in one-go? I created this [branch](https://github.com/apache/hbase/tree/HBASE-26913-replication-observability-framework) for this work. I can periodically rebase it to keep it up to date with master HEAD. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26990) Add default implementation for BufferedMutator interface setters
[
https://issues.apache.org/jira/browse/HBASE-26990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531792#comment-17531792
]
Hudson commented on HBASE-26990:
Results for branch master
[build #579 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/579/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/579/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/579/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/579/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add default implementation for BufferedMutator interface setters
>
>
> Key: HBASE-26990
> URL: https://issues.apache.org/jira/browse/HBASE-26990
> Project: HBase
> Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
> Labels: patch-available
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.13
>
>
> HBASE-20812 added default implementations to a bunch of the "optional"
> methods of Table interface. This is very useful for downstream users who
> might create delegate table implementations. Some of the BufferedMutator
> interface methods also have default implementations, but setRpcTimeout,
> setOperationTimeout, getWriteBufferSize do not.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26648) Improve fidelity of RegionLocator spans
[
https://issues.apache.org/jira/browse/HBASE-26648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531790#comment-17531790
]
Hudson commented on HBASE-26648:
Results for branch master
[build #579 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/579/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/579/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/579/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/579/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Improve fidelity of RegionLocator spans
> ---
>
> Key: HBASE-26648
> URL: https://issues.apache.org/jira/browse/HBASE-26648
> Project: HBase
> Issue Type: Sub-task
> Components: tracing
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> While backporting HBASE-26474 to branch-2, I realized that the nuance of
> tracing of region locator calls is lacking. In the current implementation,
> all calls to the region locator produce {{SpanKind.CLIENT}} spans. This is
> correct only for the code path where the client reaches out to a remote host
> to populate cache locations. This doesn't happen every time.
> Instead, the top-level span should be {{SpanKind.INTERNAL}} and only when a
> region location must be requested remotely should we create a child span with
> {{SpanKind.CLIENT}}.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26892) Add spotless:check in our pre commit general check
[
https://issues.apache.org/jira/browse/HBASE-26892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531791#comment-17531791
]
Hudson commented on HBASE-26892:
Results for branch master
[build #579 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/579/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/579/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/579/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/579/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add spotless:check in our pre commit general check
> --
>
> Key: HBASE-26892
> URL: https://issues.apache.org/jira/browse/HBASE-26892
> Project: HBase
> Issue Type: Task
> Components: jenkins
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.13
>
>
> To force the style.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [hbase] Apache-HBase commented on pull request #4399: HBASE-26995 Remove ref guide check in pre commit and nightly for bran…
Apache-HBase commented on PR #4399: URL: https://github.com/apache/hbase/pull/4399#issuecomment-1117464031 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 14s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | spotless | 0m 40s | branch has no errors when running spotless:check. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 4s | Maven dependency ordering for patch | | +1 :green_heart: | shellcheck | 0m 1s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | spotless | 0m 37s | patch has no errors when running spotless:check. | ||| _ Other Tests _ | | +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? | | | | 3m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4399/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4399 | | Optional Tests | dupname asflicense spotless shellcheck shelldocs | | uname | Linux 973a41f82a88 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / a0258441ab | | Max. process+thread count | 33 (vs. ulimit of 12500) | | modules | C: U: | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4399/1/console | | versions | git=2.17.1 maven=3.6.3 shellcheck=0.4.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4399: HBASE-26995 Remove ref guide check in pre commit and nightly for bran…
Apache-HBase commented on PR #4399: URL: https://github.com/apache/hbase/pull/4399#issuecomment-1117461591 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 20s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 5s | Maven dependency ordering for patch | ||| _ Other Tests _ | | | | 2m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4399/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4399 | | Optional Tests | | | uname | Linux ec1de91c7637 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / a0258441ab | | Max. process+thread count | 39 (vs. ulimit of 12500) | | modules | C: U: | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4399/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4399: HBASE-26995 Remove ref guide check in pre commit and nightly for bran…
Apache-HBase commented on PR #4399: URL: https://github.com/apache/hbase/pull/4399#issuecomment-1117460930 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 46s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 45s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 3s | Maven dependency ordering for patch | ||| _ Other Tests _ | | | | 2m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4399/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4399 | | Optional Tests | | | uname | Linux 8bb363fdd9c2 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / a0258441ab | | Max. process+thread count | 33 (vs. ulimit of 12500) | | modules | C: U: | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4399/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4399: HBASE-26995 Remove ref guide check in pre commit and nightly for bran…
Apache-HBase commented on PR #4399: URL: https://github.com/apache/hbase/pull/4399#issuecomment-1117457618 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4399/1/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4399: HBASE-26995 Remove ref guide check in pre commit and nightly for bran…
Apache-HBase commented on PR #4399: URL: https://github.com/apache/hbase/pull/4399#issuecomment-1117456867 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4399/1/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4399: HBASE-26995 Remove ref guide check in pre commit and nightly for bran…
Apache-HBase commented on PR #4399: URL: https://github.com/apache/hbase/pull/4399#issuecomment-1117456572 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4399/1/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #4399: HBASE-26995 Remove ref guide check in pre commit and nightly for bran…
Apache9 commented on PR #4399: URL: https://github.com/apache/hbase/pull/4399#issuecomment-1117436035 Please see here: https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-26995/5/General_20Nightly_20Build_20Report/ This is the result for this change. And for nightly, we need to apply the patch master since we will always use the personality script on master for all branches. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26995) Remove ref guide check in pre commit and nightly for branches other than master
[ https://issues.apache.org/jira/browse/HBASE-26995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26995: -- Summary: Remove ref guide check in pre commit and nightly for branches other than master (was: Remove ref guide check in pre commit for branches other than master) > Remove ref guide check in pre commit and nightly for branches other than > master > --- > > Key: HBASE-26995 > URL: https://issues.apache.org/jira/browse/HBASE-26995 > Project: HBase > Issue Type: Sub-task > Components: build, scripts >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Our nightly builds are failing after HBASE-26932. Let's fix it. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] Apache9 opened a new pull request, #4399: HBASE-26995 Remove ref guide check in pre commit and nightly for bran…
Apache9 opened a new pull request, #4399: URL: https://github.com/apache/hbase/pull/4399 …ches other than master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26892) Add spotless:check in our pre commit general check
[
https://issues.apache.org/jira/browse/HBASE-26892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531756#comment-17531756
]
Hudson commented on HBASE-26892:
Results for branch branch-2.5
[build #111 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/111/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/111/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/111/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/111/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/111/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add spotless:check in our pre commit general check
> --
>
> Key: HBASE-26892
> URL: https://issues.apache.org/jira/browse/HBASE-26892
> Project: HBase
> Issue Type: Task
> Components: jenkins
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.13
>
>
> To force the style.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26990) Add default implementation for BufferedMutator interface setters
[
https://issues.apache.org/jira/browse/HBASE-26990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531757#comment-17531757
]
Hudson commented on HBASE-26990:
Results for branch branch-2.5
[build #111 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/111/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/111/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/111/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/111/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/111/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add default implementation for BufferedMutator interface setters
>
>
> Key: HBASE-26990
> URL: https://issues.apache.org/jira/browse/HBASE-26990
> Project: HBase
> Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
> Labels: patch-available
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.13
>
>
> HBASE-20812 added default implementations to a bunch of the "optional"
> methods of Table interface. This is very useful for downstream users who
> might create delegate table implementations. Some of the BufferedMutator
> interface methods also have default implementations, but setRpcTimeout,
> setOperationTimeout, getWriteBufferSize do not.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26648) Improve fidelity of RegionLocator spans
[
https://issues.apache.org/jira/browse/HBASE-26648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531755#comment-17531755
]
Hudson commented on HBASE-26648:
Results for branch branch-2.5
[build #111 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/111/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/111/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/111/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/111/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/111/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Improve fidelity of RegionLocator spans
> ---
>
> Key: HBASE-26648
> URL: https://issues.apache.org/jira/browse/HBASE-26648
> Project: HBase
> Issue Type: Sub-task
> Components: tracing
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> While backporting HBASE-26474 to branch-2, I realized that the nuance of
> tracing of region locator calls is lacking. In the current implementation,
> all calls to the region locator produce {{SpanKind.CLIENT}} spans. This is
> correct only for the code path where the client reaches out to a remote host
> to populate cache locations. This doesn't happen every time.
> Instead, the top-level span should be {{SpanKind.INTERNAL}} and only when a
> region location must be requested remotely should we create a child span with
> {{SpanKind.CLIENT}}.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26997) Auto renew scanner lease in TableRecordReader
[ https://issues.apache.org/jira/browse/HBASE-26997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531746#comment-17531746 ] Bryan Beaudreault commented on HBASE-26997: --- The impetus for this issue was to find an hbase 2+ solution to a problem we often see in hbase 1.2. In our 1.2-based fork we have a patch to automatically re-open scanners when UnknownScannerException is seen in MR jobs, and I'd like to remove that. I wrote up an initial solution here for master branch and in testing realized that it's virtually impossible to get an UnknownScannerException in master branch. The AsyncClientScanner's prefetching routines will automatically call renewLease when prefetching is paused due to cache full. In master branch, setCaching just determines the number of rows returned per RPC and setMaxResultSize determines the size of the local cache. I tried numerous configurations to try to get an UnknownScannerException and never could. This is great! Next I tried applying my patch to branch-2 and was surprised to see that my test also passed there without even enabling the feature. Digging in it's because error handling has improved a lot since 1.2, so UnknownScannerException is automatically retried up until the max number of retries. So it seems an additional mitigation for this issue in hbase2 would be to bump max retries up a bunch. This could still be problematic though if your use-case really can't avoid long periods between next calls. Over the course of a long mapper where you might hit UnknownScannerException multiple times, it'd still be possible to exceed retries. UnknownScannerExceptions also just add noise to both exception metrics and logs, which can cause you to think there are underlying issues when there actually aren't. So I think it makes sense to add this feature to branch-2x only > Auto renew scanner lease in TableRecordReader > - > > Key: HBASE-26997 > URL: https://issues.apache.org/jira/browse/HBASE-26997 > Project: HBase > Issue Type: New Feature >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > > A common problem with hadoop jobs is when the mapper takes too long to > process individual inputs. This is especially problematic with > TableInputFormat because if you don't process a scanner.next() batch within > the scanner timeout period your job will fail with UnknownScannerException. > The fix here is usually to reduce Scan.setCaching, so that fewer rows are > returned within each batch. This isn't always a great solution because maybe > not all batches are uniform in their processing time, or maybe even > processing a single row (the smallest caching size) might take a while. > We can improve this for users by providing a configurable period at which the > TableRecordReader will automatically call scanner.renewLease() unless next() > was recently called. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] ndimiduk commented on pull request #4312: HBASE-26899 Run spotless:apply
ndimiduk commented on PR #4312: URL: https://github.com/apache/hbase/pull/4312#issuecomment-1117176949 Thank you @Apache9 for landing this effort! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26366) ZK interaction during Master startup produces loads of root-less spans
[
https://issues.apache.org/jira/browse/HBASE-26366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-26366:
-
Fix Version/s: (was: 2.6.0)
(was: 3.0.0-alpha-3)
> ZK interaction during Master startup produces loads of root-less spans
> --
>
> Key: HBASE-26366
> URL: https://issues.apache.org/jira/browse/HBASE-26366
> Project: HBase
> Issue Type: Improvement
> Components: tracing
>Affects Versions: 2.5.0, 3.0.0-alpha-2
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Attachments: image.png,
> regroup-spans-for-recoverablezookeeper.create.png
>
>
> Enable tracing with {{-Dotel.traces.sampler=always_on}} and launch the
> standalone master, check the trace store. We see lots of single-span traces,
> each a call to different {{RecoverableZooKeeper}} methods, like {{create}}
> and {{exists}}. Whatever master startup thread this is should define its own
> root span.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [hbase] Apache-HBase commented on pull request #4377: HBASE-26977 HMaster's ShutdownHook does not take effect, if tablesOnMaster is false
Apache-HBase commented on PR #4377: URL: https://github.com/apache/hbase/pull/4377#issuecomment-1116960436 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 51s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ branch-1 Compile Tests _ | | +1 :green_heart: | mvninstall | 9m 32s | branch-1 passed | | +1 :green_heart: | compile | 0m 24s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | compile | 0m 29s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | checkstyle | 1m 5s | branch-1 passed | | +1 :green_heart: | shadedjars | 2m 7s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 0m 25s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +0 :ok: | spotbugs | 2m 4s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 2m 1s | branch-1 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 17s | the patch passed | | +1 :green_heart: | compile | 0m 24s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javac | 0m 24s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | javac | 0m 28s | the patch passed | | +1 :green_heart: | checkstyle | 0m 55s | the patch passed | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 1m 53s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 3m 30s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 0m 18s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | findbugs | 1m 50s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 118m 59s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 150m 23s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.regionserver.TestRegionProcessRowsWithLocks | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4377/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4377 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 2a2b19a13a90 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4377/out/precommit/personality/provided.sh | | git revision | branch-1 / 5c15ac63de | | Default Java | Azul Systems, Inc.-1.7.0_272-b10 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4377/7/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4377/7/testReport/ | | Max. process+thread count | 3760 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4377/7/console | | versions | git=2.17.1 maven=3.6.0 findbugs=3.0.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries
