[
https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16612638#comment-16612638
]
stack commented on HBASE-21191:
-------------------------------
One note on how the 'holding pattern' was implemented. It was suggested that we
just do longer timesout on the RPCs when we are trying to read meta. I tried
it. What I found was that the Retrying Callable we use to make the RPC takes
the # of retries from default rather than those of the current Configuration so
extending RPC retries and operation timeout has a bug or is need of a bit of
work (HBASE-21193). The other awkward thing about using a long RPC
timeout/retry rather than what we have here is that it is easier flagging
operator they need to intercede if we have the explicit hold as we have in this
patch.
> Add a holding-pattern if no assign for meta or namespace (Can happen if
> masterprocwals have been cleared).
> ----------------------------------------------------------------------------------------------------------
>
> Key: HBASE-21191
> URL: https://issues.apache.org/jira/browse/HBASE-21191
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
> Reporter: stack
> Assignee: stack
> Priority: Major
> Fix For: 2.1.1
>
> Attachments: HBASE-21191.branch-2.1.001.patch
>
>
> If the masterprocwals have been removed -- operator error, hdfs dataloss, or
> because we have gotten ourselves into a pathological state where we have
> hundreds of masterprocwals too process and it is taking too long so we just
> want to startover -- then master startup will have a dilemma. Master startup
> needs hbase:meta to be online. If the masterprocwals have been removed, there
> may be no outstanding assign or a servercrashprocedure with coverage for
> hbase:meta (I ran into this issue repeatedly in internal testing purging
> masterprocwals on a large test cluster). Worse, when master startup cannot
> find an online hbase:meta, it exits after exhausting the RPC retries.
> So, we need a holding-pattern for master startup if hbase:meta is not online
> if only so an operator can schedule an assign for meta or so they can assign
> fixup procedures (HBASE-20786 has discussion on why we cannot just
> auto-schedule an assign of meta).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
