[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17537787#comment-17537787
]
Szabolcs Bukros commented on HBASE-26969:
-----------------------------------------
{{{quote}}}
I guess if we want to compact the mob files, we always need to compact the
normal files which references the mob files so we can update the references in
the metadata?
{quote}
Yes, that is how it works.
{{{quote}}}
The mob files should have a different name prefix or under a different
directory?
{{{}{quote}{}}}{{{}{}}}
They have a different directory structure.
"/mobdir/data/default/table_name/a0209c070c85d1e4d500af8ba33c3c02/cf"
They are stored fully separated. Please note these regions only contain mob
files and are fully independent from the referencing regions. A single mob
region could theoretically contain every MOB file in hbase regardless of where
it is referenced from. Also their naming convention is different. For us the
only important thing is that it ends with "_<encoded name of the region that
referenced it when it was created>", so something like this:
"0cc175b9c0f1b6a831c399e2697726612022050314ecf20b51674cd6bd647bfb2d88b1ff_b593e96e821ba62333311d8a4b101a88"
{quote}
So at least for loading, there will be no problem
{quote}
That's true. Reads are very straightforward.
{{{quote}}}
I think the only problem here is how do we clean up the half written mob files,
I think the logic is mainly the same with what we have now, get all the mob
refs from all the normal storefiles, to construct the base list, and then get
all the mob files which are currently being written, all MOB files besides them
are the ones should be deleted.
{quote}{{{}{}}}
{{That is part of the problme, yes. To have access to the half written mob file
list the cleaner have to run on the RS. But each RS only has access to it's own
}}half written mob file list so each can only clean a subset of the existing
mob files. To be precise if a mob file name ends with a region's name that is
hosted on the current RS then the cleaner can decide if it can be archived or
not. Unfortunately with merges and splits regions get archived so after a point
there will be mob files containing names of regions not hosted on any RS and
none of the cleaners running on RSes could clean these up. So we need one more
cleaner specifically for these (I put it on master to replace the original
cleaner), that have to read every available hfile to make sure we have every
active mob reference and are able to decide if a mob file created by a since
archived region can be archived or not.
> Eliminate MOB renames when SFT is enabled
> -----------------------------------------
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
> Affects Versions: 2.5.0, 3.0.0-alpha-3
> Reporter: Szabolcs Bukros
> Assignee: Szabolcs Bukros
> Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#000000}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#000000}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#000000}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#000000}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
