[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534360#comment-17534360
]
Szabolcs Bukros commented on HBASE-26969:
-----------------------------------------
I would like to start by stating that this issue grow bigger than just removing
the renames and exposed multiple issues in the MOB-SFT interaction.
I have uploaded a draft PR containing my changes. I intend to use it as a
reference to show the issues when it comes to using MOB on FileBased SFT.
My main problem was that while MOB files were already tracked in the hfile
metadata, the "single source of truth" is widely distributed and not easily
available.
Both the WriterCreationTracker and the StoreFileTracker are RS based data and
the MOB cleaner needs it to work reliably when FileBased SFT is used. Exposing
this data and allowing the Master to request this from RSes, collect it and run
the cleaner based on this, while technically possible, looked less than
optimal. It would result in a single cluster wide spike that we should try to
avoid and considering the delay that certain RSes could have (uneven load, GC
pauses, etc) the data can be already outdated by the time the collection is
done. So instead I tried to move the cleaner to the RSes. This solution also
had it's drawbacks.
MOB file names contain the encoded name of the region that created them so the
RS hosting that specific region can check it's hfiles for references and can
clean it up if it does not find anything. The problem comes with merge/split
parent regions. When the parent region is archived the new region's hfiles will
still hold references to the old MOB files but now the only way to make sure if
the old MOB file is referenced or not is to check every single hfile in every
store belonging to the same columnfamily, because we can not tell based on it's
name where it could be referenced from. Like the old cleaner did. So while I
moved the MOB cleaner to the RS level and reduced it's scope to only clean up
MOB files belonging to regions hosted by that RS I had to leave a "global" MOB
cleaner running on Master to deal with MOB files created by archived regions
but potentially still being referenced. And I think this is very ugly.
This whole process could have been significantly simpler if we would have
tracker files in MOB stores but then we would have TWO competing sources of
truth. The tracker files and the hfile metadata.
HBASE-27017 is a related issue where the snapshot code tries to get the active
MOB files based on the configured SFT, but since MOB stores do not have tracker
files it returns an empty list. If the store had tracker files it would work.
Without a tracker file we either include every MOB files in the dir (garbage
included) or scan every single hfile metadata for MOB references.
What I'm trying to say is that while I think my solution would work and solve
the immediate issues I would much prefer if there would be a centralized,
easily available active MOB list and create a solution based on that.
[~apurtell] ,[~zhangduo],[~elserj] ,[~wchevreuil] What do you think?
> Eliminate MOB renames when SFT is enabled
> -----------------------------------------
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Task
> Components: mob
> Affects Versions: 2.5.0, 3.0.0-alpha-3
> Reporter: Szabolcs Bukros
> Assignee: Szabolcs Bukros
> Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#000000}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#000000}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#000000}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#000000}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
