[
https://issues.apache.org/jira/browse/HBASE-15871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-15871:
-------------------------------------------
Attachment: HBASE-15871.patch
Patch for trunk. This unnecessary backward seek can also be seen in trunk. But
in lower branches this could be more significant because we hold a lock too. In
trunk the lock is removed.
How ever I think [~Jeongdae Kim]'s first patch is good as I am not sure if a
snapshot fails we do retry and even if that fails we throw
DroppedSnapshotException. Do we try to reuse that snapshot? Also during the
retry process the seqId to be generated by this flush is fixed and I think for
the current store there cannot be concurrent flushes till the snapshot is
cleared.
This patch also adds a test case. Ping for reviews/feedback. It seems to be a
critical fix for any user using 'reverse' scan. should be a perf fix too.
> Memstore flush doesn't finish because of backwardseek() in memstore scanner.
> ----------------------------------------------------------------------------
>
> Key: HBASE-15871
> URL: https://issues.apache.org/jira/browse/HBASE-15871
> Project: HBase
> Issue Type: Bug
> Components: Scanners
> Affects Versions: 1.1.2
> Reporter: Jeongdae Kim
> Fix For: 1.1.2
>
> Attachments: HBASE-15871.branch-1.1.001.patch,
> HBASE-15871.branch-1.1.002.patch, HBASE-15871.branch-1.1.003.patch,
> HBASE-15871.patch, memstore_backwardSeek().PNG
>
>
> Sometimes in our production hbase cluster, it takes a long time to finish
> memstore flush.( for about more than 30 minutes)
> the reason is that a memstore flusher thread calls
> StoreScanner.updateReaders(), waits for acquiring a lock that store scanner
> holds in StoreScanner.next() and backwardseek() in memstore scanner runs for
> a long time.
> I think that this condition could occur in reverse scan by the following
> process.
> 1) create a reversed store scanner by requesting a reverse scan.
> 2) flush a memstore in the same HStore.
> 3) puts a lot of cells in memstore and memstore is almost full.
> 4) call the reverse scanner.next() and re-create all scanners in this store
> because all scanners was already closed by 2)'s flush() and backwardseek()
> with store's lastTop for all new scanners.
> 5) in this status, memstore is almost full by 2) and all cells in memstore
> have sequenceID greater than this scanner's readPoint because of 2)'s
> flush(). this condition causes searching all cells in memstore, and
> seekToPreviousRow() repeatly seach cells that are already searched if a row
> has one column. (described this in more detail in a attached file.)
> 6) flush a memstore again in the same HStore, and wait until 4-5) process
> finished, to update store files in the same HStore after flusing.
> I searched HBase jira. and found a similar issue. (HBASE-14497) but,
> HBASE-14497's fix can't solve this issue because that fix just changed
> recursive call to loop.(and already applied to our HBase version)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
