[jira] [Updated] (HBASE-26026) HBase Write may be stuck forever when using CompactingMemStore
[
https://issues.apache.org/jira/browse/HBASE-26026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuobin zheng updated HBASE-26026:
--
Description:
Sometimes I observed that HBase Write might be stuck in my hbase cluster which
enabling {{{}CompactingMemStore{}}}. I have simulated the problem by unit test
in my PR.
The problem is caused by {{CompactingMemStore.checkAndAddToActiveSize}} :
{code:java}
425 private boolean checkAndAddToActiveSize(MutableSegment currActive, Cell
cellToAdd,
426 MemStoreSizing memstoreSizing) {
427if (shouldFlushInMemory(currActive, cellToAdd, memstoreSizing)) {
428 if (currActive.setInMemoryFlushed()) {
429flushInMemory(currActive);
430if (setInMemoryCompactionFlag()) {
431 // The thread is dispatched to do in-memory compaction in the
background
..
}
{code}
In line 427, {{shouldFlushInMemory}} checking if {{currActive.getDataSize}}
adding the size of {{cellToAdd}} exceeds
{{{}CompactingMemStore.inmemoryFlushSize{}}},if true, then {{currActive}}
should be flushed, {{currActive.setInMemoryFlushed()}} is invoked in line 428 ：
{code:java}
public boolean setInMemoryFlushed() {
return flushed.compareAndSet(false, true);
}
{code}
After sucessfully set {{currActive.flushed}} to true, in above line 429
{{flushInMemory(currActive)}} invokes
{{CompactingMemStore.pushActiveToPipeline}} :
{code:java}
protected void pushActiveToPipeline(MutableSegment currActive) {
if (!currActive.isEmpty()) {
pipeline.pushHead(currActive);
resetActive();
}
}
{code}
In above {{CompactingMemStore.pushActiveToPipeline}} method , if the
{{currActive.cellSet}} is empty, then nothing is done. Due to concurrent writes
and because we first add cell size to {{currActive.getDataSize}} and then
actually add cell to {{{}currActive.cellSet{}}}, it is possible that
{{currActive.getDataSize}} could not accommodate {{cellToAdd}} but
{{currActive.cellSet}} is still empty if pending writes which not yet add cells
to {{{}currActive.cellSet{}}}.
So if the {{currActive.cellSet}} is empty now, then no {{ActiveSegment}} is
created, and new writes still continue target to {{{}currActive{}}}, but
{{currActive.flushed}} is true, {{currActive}} could not enter
{{flushInMemory(currActive)}} again，and new {{ActiveSegment}} could not be
created forever ! In the end all writes would be stuck.
In my opinion , once {{currActive.flushed}} is set true, it could not continue
use as {{ActiveSegment}} , and because of concurrent pending writes, only after
{{currActive.updatesLock.writeLock()}} is acquired(i.e.
{{currActive.waitForUpdates}} is called) in
{{CompactingMemStore.inMemoryCompaction}} ,we can safely say {{currActive}} is
empty or not.
My fix is remove the {{if (!currActive.isEmpty())}} check here and left the
check to background {{InMemoryCompactionRunnable}} after
{{currActive.waitForUpdates}} is called. An alternative fix is we use
synchronization mechanism in {{checkAndAddToActiveSize}} method to prevent all
writes , wait for all pending write completed(i.e. currActive.waitForUpdates is
called) and if {{currActive}} is still empty ,then we set
{{currActive.flushed}} back to false,but I am not inclined to use so heavy
synchronization in write path, and I think we would better maintain lockless
implementation for {{CompactingMemStore.add}} method just as now and
{{currActive.waitForUpdates}} would better be left in background
{{{}InMemoryCompactionRunnable{}}}.
was:
Sometimes I observed that HBase Write might be stuck in my hbase cluster which
enabling {{CompactingMemStore}}. I have simulated the problem by unit test in
my PR.
The problem is caused by {{CompactingMemStore.checkAndAddToActiveSize}} :
{code:java}
425 private boolean checkAndAddToActiveSize(MutableSegment currActive, Cell
cellToAdd,
426 MemStoreSizing memstoreSizing) {
427if (shouldFlushInMemory(currActive, cellToAdd, memstoreSizing)) {
428 if (currActive.setInMemoryFlushed()) {
429flushInMemory(currActive);
430if (setInMemoryCompactionFlag()) {
431 // The thread is dispatched to do in-memory compaction in the
background
..
}
{code}
In line 427, {{shouldFlushInMemory}} checking if {{currActive.getDataSize}}
adding the size of {{cellToAdd}} exceeds
{{CompactingMemStore.inmemoryFlushSize}},if true, then {{currActive}} should
be flushed, {{currActive.setInMemoryFlushed()}} is invoked in line 428 ：
{code:java}
public boolean setInMemoryFlushed() {
return flushed.compareAndSet(false, true);
}
{code}
After sucessfully set {{currActive.flushed}} to true, in above line 429
{{flushInMemory(currActive)}} invokes
{{CompactingMemStore.pushActiveToPipeline}} :
{code:java}
protected void pushActiveToPipeline(MutableSegment currActive) {
if (!currActive.isEmpty()) {
pipeline.pushHead(currActive);
[jira] [Commented] (HBASE-25720) Sync WAL stuck when prepare flush cache will prevent flush cache and cause OOM
[ https://issues.apache.org/jira/browse/HBASE-25720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17617378#comment-17617378 ] March Wang commented on HBASE-25720: Hi [~Xiaolin Ha] , Sorry late response, I will do it. Thank you so much! > Sync WAL stuck when prepare flush cache will prevent flush cache and cause OOM > -- > > Key: HBASE-25720 > URL: https://issues.apache.org/jira/browse/HBASE-25720 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.4.13 >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Attachments: prepare-flush-cache-stuck.png > > > We call HRegion#doSyncOfUnflushedWALChanges when preparing to flush cache. > But this WAL sync may stuck, and abort the flush of cache. > !prepare-flush-cache-stuck.png|width=519,height=246! > If we cannot aware of this problem in time, RS will OOM kill. > I think we should force abort RS when sync stuck in preparing, like in > committing snapshots. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-25166) MobFileCompactionChore is closing the master's shared cluster connection
[
https://issues.apache.org/jira/browse/HBASE-25166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17617336#comment-17617336
]
Hudson commented on HBASE-25166:
Results for branch branch-2
[build #663 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/663/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/663/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/663/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/663/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/663/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> MobFileCompactionChore is closing the master's shared cluster connection
>
>
> Key: HBASE-25166
> URL: https://issues.apache.org/jira/browse/HBASE-25166
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 3.0.0-alpha-1
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.6.0, 2.5.1
>
>
> Code which is doing so in MobFileCompactionChore
> {code:java}
> try (Connection conn = master.getConnection();
> Admin admin = conn.getAdmin();) { {code}
> As master uses this connection to read the meta or other system tables, so
> none of the meta operations through master will work.
> Symptoms in master logs:-
> {code:java}
> s, events=841, succcessCount=123, totalEvents=12824192,
> totalSuccessCount=1891300
> 2020-10-05 16:34:25,062 INFO
> org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor: Unable
> to get remote Address
> 2020-10-05 16:34:25,062 ERROR
> org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore: Failed to
> normalize regions.
> java.io.IOException: connection is closed
> at
> org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:241)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:797)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:768)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:727)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.fullScanTables(MetaTableAccessor.java:215)
> at
> org.apache.hadoop.hbase.master.TableStateManager.getTablesInStates(TableStateManager.java:189)
> at
> org.apache.hadoop.hbase.master.HMaster.normalizeRegions(HMaster.java:1821)
> at
> org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore.chore(RegionNormalizerChore.java:48)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:188)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) {code}
> Symptoms at the client:-
> {code:java}
> RpcRetryingCaller{globalStartTime=1602099132430, pause=100, maxAttempts=11},
> java.io.IOException: java.io.IOException: connection is closed
> at
> org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:241)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1116)
> at
> org.apache.hadoop.hbase.master.TableStateManager.readMetaState(TableStateManager.java:258)
> at
> org.apache.hadoop.hbase.master.TableStateManager.isTablePresent(TableStateManager.java:175)
>
[GitHub] [hbase] Apache-HBase commented on pull request #4829: HBASE-27426 - Fix ZKWatcher shutdown seqence to avoid InterruptExcept…
Apache-HBase commented on PR #4829: URL: https://github.com/apache/hbase/pull/4829#issuecomment-1278129499 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 3s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 45s | master passed | | +1 :green_heart: | compile | 1m 1s | master passed | | +1 :green_heart: | shadedjars | 3m 46s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 37s | the patch passed | | +1 :green_heart: | compile | 1m 1s | the patch passed | | +1 :green_heart: | javac | 1m 1s | the patch passed | | +1 :green_heart: | shadedjars | 3m 47s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 38s | hbase-zookeeper in the patch passed. | | -1 :x: | unit | 219m 40s | hbase-server in the patch failed. | | | | 239m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4829/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4829 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0cd6791c4512 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 0d260f5b22 | | Default Java | Eclipse Adoptium-11.0.16.1+1 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4829/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4829/2/testReport/ | | Max. process+thread count | 2458 (vs. ulimit of 3) | | modules | C: hbase-zookeeper hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4829/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-27421) Bump spotless plugin to 2.27.2 and reimplement the 'Remove unhelpful javadoc stubs' rule
[
https://issues.apache.org/jira/browse/HBASE-27421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17617286#comment-17617286
]
Hudson commented on HBASE-27421:
Results for branch branch-2.5
[build #224 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/224/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/224/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/224/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/224/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/224/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Bump spotless plugin to 2.27.2 and reimplement the 'Remove unhelpful javadoc
> stubs' rule
>
>
> Key: HBASE-27421
> URL: https://issues.apache.org/jira/browse/HBASE-27421
> Project: HBase
> Issue Type: Sub-task
> Components: documentation, pom
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.6.0, 2.5.1, 3.0.0-alpha-4, 2.4.15
>
>
> spotless maven 2.27.2 is released and it contains
> https://github.com/diffplug/spotless/issues/1359, which allows the
> replacement to be null, so we can add back the 'Remove unhelpful javadoc
> stubs' rule now.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [hbase] Apache-HBase commented on pull request #4829: HBASE-27426 - Fix ZKWatcher shutdown seqence to avoid InterruptExcept…
Apache-HBase commented on PR #4829: URL: https://github.com/apache/hbase/pull/4829#issuecomment-1278115722 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 19s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 20s | master passed | | +1 :green_heart: | compile | 0m 44s | master passed | | +1 :green_heart: | shadedjars | 4m 2s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 32s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 4s | the patch passed | | +1 :green_heart: | compile | 0m 44s | the patch passed | | +1 :green_heart: | javac | 0m 44s | the patch passed | | +1 :green_heart: | shadedjars | 3m 59s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 31s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 32s | hbase-zookeeper in the patch passed. | | +1 :green_heart: | unit | 204m 23s | hbase-server in the patch passed. | | | | 224m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4829/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4829 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 80fd6c4bd212 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 0d260f5b22 | | Default Java | Temurin-1.8.0_345-b01 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4829/2/testReport/ | | Max. process+thread count | 2589 (vs. ulimit of 3) | | modules | C: hbase-zookeeper hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4829/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-27421) Bump spotless plugin to 2.27.2 and reimplement the 'Remove unhelpful javadoc stubs' rule
[
https://issues.apache.org/jira/browse/HBASE-27421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17617280#comment-17617280
]
Hudson commented on HBASE-27421:
Results for branch master
[build #701 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/701/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/701/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/701/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/701/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Bump spotless plugin to 2.27.2 and reimplement the 'Remove unhelpful javadoc
> stubs' rule
>
>
> Key: HBASE-27421
> URL: https://issues.apache.org/jira/browse/HBASE-27421
> Project: HBase
> Issue Type: Sub-task
> Components: documentation, pom
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.6.0, 2.5.1, 3.0.0-alpha-4, 2.4.15
>
>
> spotless maven 2.27.2 is released and it contains
> https://github.com/diffplug/spotless/issues/1359, which allows the
> replacement to be null, so we can add back the 'Remove unhelpful javadoc
> stubs' rule now.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HBASE-27383) Add dead region server to SplitLogManager#deadWorkers set as the first step.
[ https://issues.apache.org/jira/browse/HBASE-27383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh Shah reassigned HBASE-27383: Assignee: (was: Rushabh Shah) > Add dead region server to SplitLogManager#deadWorkers set as the first step. > > > Key: HBASE-27383 > URL: https://issues.apache.org/jira/browse/HBASE-27383 > Project: HBase > Issue Type: Bug >Affects Versions: 2.5.0, 1.7.2, 2.4.14 >Reporter: Rushabh Shah >Priority: Major > > Currently we add a dead region server to +SplitLogManager#deadWorkers+ set in > SERVER_CRASH_SPLIT_LOGS state. > Consider a case where a region server is handling split log task for > hbase:meta table and SplitLogManager has exhausted all the retries and won't > try any more region server. > The region server which is handling split log task has died. > We have a check in SplitLogManager where if a region server is declared dead > and if that region server is responsible for split log task then we > forcefully resubmit split log task. See the code > [here|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java#L721-L726]. > But we add a region server to SplitLogManager#deadWorkers set in > [SERVER_CRASH_SPLIT_LOGS|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java#L252] > state. > Before that it runs > [SERVER_CRASH_GET_REGIONS|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java#L214] > state and checks if hbase:meta table is up. In this case, hbase:meta table > was not online and that prevented SplitLogManager to add this RS to > deadWorkers list. This created a deadlock and hbase cluster was completely > down for an extended period of time until we failed over active hmaster. See > HBASE-27382 for more details. > Improvements: > 1. We should a dead region server to +SplitLogManager#deadWorkers+ list as > the first step. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [hbase] Apache-HBase commented on pull request #4829: HBASE-27426 - Fix ZKWatcher shutdown seqence to avoid InterruptExcept…
Apache-HBase commented on PR #4829: URL: https://github.com/apache/hbase/pull/4829#issuecomment-1277901409 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 23s | master passed | | +1 :green_heart: | compile | 2m 36s | master passed | | +1 :green_heart: | checkstyle | 0m 36s | master passed | | +1 :green_heart: | spotless | 0m 40s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 35s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 9s | the patch passed | | +1 :green_heart: | compile | 2m 33s | the patch passed | | +1 :green_heart: | javac | 2m 33s | the patch passed | | +1 :green_heart: | checkstyle | 0m 33s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 8m 5s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. | | +1 :green_heart: | spotless | 0m 38s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 45s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 30m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4829/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4829 | | Optional Tests | dupname asflicense javac hadoopcheck spotless xml compile spotbugs hbaseanti checkstyle | | uname | Linux 3a3250b4a32d 5.4.0-1085-aws #92~18.04.1-Ubuntu SMP Wed Aug 31 17:21:08 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 0d260f5b22 | | Default Java | Temurin-1.8.0_345-b01 | | Max. process+thread count | 64 (vs. ulimit of 3) | | modules | C: hbase-zookeeper hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4829/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.7.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25166) MobFileCompactionChore is closing the master's shared cluster connection
[
https://issues.apache.org/jira/browse/HBASE-25166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lijin Bin updated HBASE-25166:
--
Fix Version/s: 2.6.0
2.5.1
> MobFileCompactionChore is closing the master's shared cluster connection
>
>
> Key: HBASE-25166
> URL: https://issues.apache.org/jira/browse/HBASE-25166
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 3.0.0-alpha-1
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.6.0, 2.5.1
>
>
> Code which is doing so in MobFileCompactionChore
> {code:java}
> try (Connection conn = master.getConnection();
> Admin admin = conn.getAdmin();) { {code}
> As master uses this connection to read the meta or other system tables, so
> none of the meta operations through master will work.
> Symptoms in master logs:-
> {code:java}
> s, events=841, succcessCount=123, totalEvents=12824192,
> totalSuccessCount=1891300
> 2020-10-05 16:34:25,062 INFO
> org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor: Unable
> to get remote Address
> 2020-10-05 16:34:25,062 ERROR
> org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore: Failed to
> normalize regions.
> java.io.IOException: connection is closed
> at
> org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:241)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:797)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:768)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:727)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.fullScanTables(MetaTableAccessor.java:215)
> at
> org.apache.hadoop.hbase.master.TableStateManager.getTablesInStates(TableStateManager.java:189)
> at
> org.apache.hadoop.hbase.master.HMaster.normalizeRegions(HMaster.java:1821)
> at
> org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore.chore(RegionNormalizerChore.java:48)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:188)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) {code}
> Symptoms at the client:-
> {code:java}
> RpcRetryingCaller{globalStartTime=1602099132430, pause=100, maxAttempts=11},
> java.io.IOException: java.io.IOException: connection is closed
> at
> org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:241)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1116)
> at
> org.apache.hadoop.hbase.master.TableStateManager.readMetaState(TableStateManager.java:258)
> at
> org.apache.hadoop.hbase.master.TableStateManager.isTablePresent(TableStateManager.java:175)
> at
> org.apache.hadoop.hbase.master.HMaster.getTableDescriptors(HMaster.java:3277)
> at
> org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:3221)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1064)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:418)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [hbase] binlijin merged pull request #4826: HBASE-25166 MobFileCompactionChore is closing the master's shared clu…
binlijin merged PR #4826: URL: https://github.com/apache/hbase/pull/4826 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] binlijin merged pull request #4825: HBASE-25166 MobFileCompactionChore is closing the master's shared clu…
binlijin merged PR #4825: URL: https://github.com/apache/hbase/pull/4825 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-27109) Move replication queue storage from zookeeper to a separated HBase table
[
https://issues.apache.org/jira/browse/HBASE-27109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17616895#comment-17616895
]
Hudson commented on HBASE-27109:
Results for branch HBASE-27109/table_based_rqs
[build #28 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-27109%252Ftable_based_rqs/28/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-27109%252Ftable_based_rqs/28/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-27109%252Ftable_based_rqs/28/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-27109%252Ftable_based_rqs/28/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Move replication queue storage from zookeeper to a separated HBase table
>
>
> Key: HBASE-27109
> URL: https://issues.apache.org/jira/browse/HBASE-27109
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> This is a more specific issue based on the works which are already done in
> HBASE-15867.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [hbase] Apache-HBase commented on pull request #4827: HBASE-27422: Support replication for hbase:acl
Apache-HBase commented on PR #4827: URL: https://github.com/apache/hbase/pull/4827#issuecomment-1277111439 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 24s | master passed | | +1 :green_heart: | compile | 0m 39s | master passed | | +1 :green_heart: | shadedjars | 3m 47s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 13s | the patch passed | | +1 :green_heart: | compile | 0m 39s | the patch passed | | +1 :green_heart: | javac | 0m 39s | the patch passed | | +1 :green_heart: | shadedjars | 3m 47s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 212m 38s | hbase-server in the patch failed. | | | | 229m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4827/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4827 | | JIRA Issue | HBASE-27422 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2ff04f22936e 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 0d260f5b22 | | Default Java | Temurin-1.8.0_345-b01 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4827/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4827/3/testReport/ | | Max. process+thread count | 2503 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4827/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4827: HBASE-27422: Support replication for hbase:acl
Apache-HBase commented on PR #4827: URL: https://github.com/apache/hbase/pull/4827#issuecomment-1277109971 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 10s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 37s | master passed | | +1 :green_heart: | compile | 0m 48s | master passed | | +1 :green_heart: | shadedjars | 3m 45s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 38s | the patch passed | | +1 :green_heart: | compile | 0m 47s | the patch passed | | +1 :green_heart: | javac | 0m 47s | the patch passed | | +1 :green_heart: | shadedjars | 3m 46s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 208m 31s | hbase-server in the patch passed. | | | | 227m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4827/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4827 | | JIRA Issue | HBASE-27422 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 27706069b6a5 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 0d260f5b22 | | Default Java | Eclipse Adoptium-11.0.16.1+1 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4827/3/testReport/ | | Max. process+thread count | 2654 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4827/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
