[
https://issues.apache.org/jira/browse/HBASE-11368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14163162#comment-14163162
]
Qiang Tian commented on HBASE-11368:
------------------------------------
ideas for lowering down the lock granularity(based on 0.98.5 code base)
1)read/scan
is it the primary goal for atomic multi-CF bulkload in HBASE-4552?
After DefaultStoreFileManager#storefiles is updated in HStore#bulkLoadHFile,
notifyChangedReadersObservers is called to reset the StoreScanner#heap, so
checkReseek->resetScannerStack will be triggered in next scan/read to recreate
store scanners based on new storefiles.
so we could introduce a new region level rwlock multiCFLock,
HRegion#bulkLoadHFiles acquires the writelock before multi-CF
HStore.bulkLoadHFile call. and StoreScanner#resetScannerStack acquires the
readlock. this way the scanners are recreated after all CFs' store files are
populated.
2)split region.
the region will be closed in SplitTransaction#stepsBeforePONR, which falls into
the HRegion#lock protection area. bulk load still still need to acquire its
readlock at start.
3) memstore flush.
we flush to a new file which is not related to the loaded files.
4)compaction.
the compaction is performed store by store. if bulkload inserts new files to
{{storefiles}} during the selectCompaction process, the file list to be
compacted might be impacted. e.g., the compaction for some CF do not include
new loaded files, while others might include. but this does not impact the data
integrity and read behavior?
at the end of compaction, {{storefiles}} access is still protected by
HStore#lock if there is bulk load change to the same CF.
comments?
thanks
> Multi-column family BulkLoad fails if compactions go on too long
> ----------------------------------------------------------------
>
> Key: HBASE-11368
> URL: https://issues.apache.org/jira/browse/HBASE-11368
> Project: HBase
> Issue Type: Bug
> Reporter: stack
>
> Compactions take a read lock. If a multi-column family region, before bulk
> loading, we want to take a write lock on the region. If the compaction takes
> too long, the bulk load fails.
> Various recipes include:
> + Making smaller regions (lame)
> + [~victorunique] suggests major compacting just before bulk loading over in
> HBASE-10882 as a work around.
> Does the compaction need a read lock for that long? Does the bulk load need
> a full write lock when multiple column families? Can we fail more gracefully
> at least?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
