[
https://issues.apache.org/jira/browse/HBASE-29185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17935542#comment-17935542
]
Duo Zhang commented on HBASE-29185:
-----------------------------------
The design in HBASE-2231 is to write a compaction event to WAL and replay it
while opening the region to finish the compaction.
But since renaming the store files and writing wal marker are still not atomic,
in HBASE-20724, we decided to add the compacted files list to storefile's
metadata. When opening a store, after loading all the store files, we will also
collect all the compacted store files, remove them from the store file list and
archive them.
You can see the code in StoreEngine.openStoreFiles about this logic.
https://github.com/apache/hbase/blob/c2af2b3c0de938f2c2d094860f33e31177d197ea/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreEngine.java#L313
Thanks.
> How to ensure the atomicity of data files during compaction
> -----------------------------------------------------------
>
> Key: HBASE-29185
> URL: https://issues.apache.org/jira/browse/HBASE-29185
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
> Affects Versions: 2.6.0
> Reporter: yuting sun
> Priority: Major
>
> {code:java}
> //代码占位符
> {code}
> protected List<HStoreFile> doCompaction(CompactionRequestImpl cr,
> Collection<HStoreFile> filesToCompact, User user, long compactionStartTime,
> List<Path> newFiles) throws IOException \{ // Do the steps necessary to
> complete the compaction. setStoragePolicyFromFileName(newFiles);
> List<HStoreFile> sfs = storeEngine.commitStoreFiles(newFiles, true); if
> (this.getCoprocessorHost() != null) { for (HStoreFile sf : sfs) {
> getCoprocessorHost().postCompact(this, sf, cr.getTracker(), cr, user); } }
> replaceStoreFiles(filesToCompact, sfs, true); long outputBytes =
> getTotalSize(sfs); // At this point the store will use new files for all new
> scanners. refreshStoreSizeAndTotalBytes(); // update store size. long now =
> EnvironmentEdgeManager.currentTime(); if ( region.getRegionServerServices()
> != null && region.getRegionServerServices().getMetrics() != null ) \{
> region.getRegionServerServices().getMetrics().updateCompaction(
> region.getTableDescriptor().getTableName().getNameAsString(), cr.isMajor(),
> now - compactionStartTime, cr.getFiles().size(), newFiles.size(),
> cr.getSize(), outputBytes); } logCompactionEndMessage(cr, sfs, now,
> compactionStartTime); return sfs; }
> {code:java}
> //代码占位符{code}
> In the commitStoreFiles logic and replaceStoreFiles logic, after the sfs file
> is moved to the data directory and before it is loaded by storeFile, if
> compaction abort occurs, how should the sfs collection be handled? It seems
> that it is retained in the data directory and waits for the next loading.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
