[
https://issues.apache.org/jira/browse/HBASE-30493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prathyusha updated HBASE-30493:
-------------------------------
Description:
SplitWALProcedure creates millions of SplitWALRemoteProcedures in a loop when
WAL archival fails without any retry strategy.
On our new cluster due to a directory file limit issue at HDFS, WAL archival
failed and we saw
* ~11 Million SplitWALRemoteProcedures created in a loop
* Storming the Procedure store
* blocked all SCPs, bringing the cluster down
* MasterData/WALs reached around ~22.5K, which on crash resulted in 6-7 hours
of startup time
* HMaster memory had to be increased to accommodate the loading of procedure
store at startup (this is cause all the SplitWALRemoteProcedures are unfinished
childs of SCPs).
Proposed fix -
Instead of creating SplitWALRemoteProcedures in a loop, we attach a retry
strategy [here
|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/SplitWALProcedure.java#L96]
was:
SplitWALProcedure creates millions of SplitWALRemoteProcedures in a loop when
WAL archival fails without any retry strategy.
On our new cluster due to a directory file limit issue at HDFS, WAL archival
failed and we saw
~11 Million SplitWALRemoteProcedures created in a loop
Storming the Procedure store
blocked all SCPs, bringing the cluster down
MasterData/WALs reached around ~22.5K, which on crash resulted in 6-7 hours of
startup time
HMaster memory had to be increased to accommodate the loading of procedure
store at startup (this is cause all the SplitWALRemoteProcedures are unfinished
childs of SCPs).
Proposed fix
Instead of creating SplitWALRemoteProcedures in a loop, we attach a retry
strategy [here
|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/SplitWALProcedure.java#L96]
> SplitWALProcedure creates SplitWALRemoteProcedures in a loop when WAL
> archival fails without any retry strategy
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-30493
> URL: https://issues.apache.org/jira/browse/HBASE-30493
> Project: HBase
> Issue Type: Bug
> Reporter: Prathyusha
> Assignee: Prathyusha
> Priority: Major
>
> SplitWALProcedure creates millions of SplitWALRemoteProcedures in a loop when
> WAL archival fails without any retry strategy.
> On our new cluster due to a directory file limit issue at HDFS, WAL archival
> failed and we saw
> * ~11 Million SplitWALRemoteProcedures created in a loop
> * Storming the Procedure store
> * blocked all SCPs, bringing the cluster down
> * MasterData/WALs reached around ~22.5K, which on crash resulted in 6-7
> hours of startup time
> * HMaster memory had to be increased to accommodate the loading of procedure
> store at startup (this is cause all the SplitWALRemoteProcedures are
> unfinished childs of SCPs).
> Proposed fix -
> Instead of creating SplitWALRemoteProcedures in a loop, we attach a retry
> strategy [here
> |https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/SplitWALProcedure.java#L96]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
