[
https://issues.apache.org/jira/browse/KUDU-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732043#comment-17732043
]
Zoltan Martonka commented on KUDU-3458:
---------------------------------------
To my best understanding without the tablet metadata we don't know what blocks
belong to it.
The minimal solution is to just ignore the tablet in the TSTabletManager::Init.
And delete the (corrupted) tablet meta data and consensus data when we receive
a delete table request (or we can delete them right at TSTabletManager::Init
when we detect that the meta data is corrupted).
This works and the tablet will be replicated properly. It can even restore the
replica when RF = 3 and there are only 3 tservers.
However with this method the problematic tserver will leave the data blocks in
the data file, and they will stay unused, but eating up disk space.
Is it acceptable for the unassigned blocks to remain in the data, or should it
be addressed somehow?
> Continue loading other tablets even if metadata for some tablets failed to
> load
> -------------------------------------------------------------------------------
>
> Key: KUDU-3458
> URL: https://issues.apache.org/jira/browse/KUDU-3458
> Project: Kudu
> Issue Type: Improvement
> Components: tserver
> Reporter: Alexey Serbin
> Assignee: Zoltan Martonka
> Priority: Major
> Labels: scalability, supportability, troubleshooting
>
> kudu-tserver stops tablet bootstrapping if a single tablet's metadata failed
> to load (the kudu-tserver process exits on such an event, but with caveat of
> KUDU-3419).
> This current behavior requires manual intervention. In most cases, the
> reason behind the failure to load tablet metadata is corrupted metadata file.
> The suspect behind such a corruption is a power failure, kernel panic, etc.
> where opened file isn't synced.
> In case of a cluster with many tablet servers, where RF=3, if majority of
> tablet replicas is present, such a situation with corrupted file could be
> addressed automatically if the tablet server would continue bootstrapping of
> other tablet replicas and eventually registered with Kudu masters. The
> system catalog would detect that the tablet is under-replicated because one
> replica isn't running, and would re-replicate it elsewhere, sending
> DELETE_TABLET for the tablet replica that has the corrupted metadata file.
> That'd be similar to what happens if a consensus metadata for a tablet
> replica were corrupted.
> It's necessary to update the code in {{TSTabletManager}} and allow
> {{TSTabletManager::Init()}} to complete successfully in such case, marking
> corresponding tablet replicas as failed to load (similar to what's done in
> case of replica's consensus metadata).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
