[
https://issues.apache.org/jira/browse/HBASE-30493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prathyusha reassigned HBASE-30493:
----------------------------------
Assignee: Prathyusha
> SplitWALProcedure creates SplitWALRemoteProcedures in a loop when WAL
> archival fails without any retry strategy
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-30493
> URL: https://issues.apache.org/jira/browse/HBASE-30493
> Project: HBase
> Issue Type: Bug
> Reporter: Prathyusha
> Assignee: Prathyusha
> Priority: Major
>
> SplitWALProcedure creates millions of SplitWALRemoteProcedures in a loop when
> WAL archival fails without any retry strategy.
> On our new cluster due to a directory file limit issue at HDFS, WAL archival
> failed and we saw
> ~11 Million SplitWALRemoteProcedures created in a loop
> Storming the Procedure store
> blocked all SCPs, bringing the cluster down
> MasterData/WALs reached around ~22.5K, which on crash resulted in 6-7 hours
> of startup time
> HMaster memory had to be increased to accommodate the loading of procedure
> store at startup (this is cause all the SplitWALRemoteProcedures are
> unfinished childs of SCPs).
> Proposed fix
> Instead of creating SplitWALRemoteProcedures in a loop, we attach a retry
> strategy [here
> |https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/SplitWALProcedure.java#L96]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
