[
https://issues.apache.org/jira/browse/HBASE-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837343#action_12837343
]
stack commented on HBASE-2248:
------------------------------
.bq Can anyone shed light on why HBASE-2037 introduced this clone in the first
place? Seems like a totally braindead thing for performance.
Mea Culpa. I should have caught this in review, the non-scalable, expensive
full-copy. Dumb.
I also should have run PE to catch degradation in performance before release
though in this case, according to Dan, as PE is now, we'd not have caught the
slowed-down memstore since we flush after each PE run and since the short-scan
test is new with no history (Long time ago I wrote up a how-to-release:
http://wiki.apache.org/hadoop/Hbase/HowToRelease. It says PE required but I
think I've not followed this receipe in a good while now).
.bq The 0.20.2 Memstore was using the ConcurrentSkipListMap#tailMap for every
row. tailMap incurs an O(log) overhead when called on a ConcurrentSkipListMap
so the total overhead of scanning the whole memstore in some cases, may be very
close to the overhead of a complete sort of the KVs in memstore.
In the old implementation, we used to also make a copy of a row, everytime we
called a next, to protect against the case where snapshot was removed out from
under us.
.bq The scanner scans incorrectly when a snapshot exists
Why was this again?
.bq ... increased GC overhead on multiple concurrent scans
Dave, can you enable GC logging? Even if this is the case, it needs to be
addressed.
.bq Is it possible to avoid both 'partial puts' and cloning by 'timestamping'
memstore records? e.g. each new KV in memstore gets a 'memstore timestamp' and
when a scanner is created it grabs the current timestamp so that it knows to
ignore KVs which entered the store after its creation? Should probably use a
counter and not currentTimeMillis to ensure a clear-cut.
How would we snapshot such a thing?
We could add another ts/counter to KV. We could do an AND on the type setting
a bit if extra ts is present. We then write out the KV as old style, dropping
extra ts when we flush to hfile, or we just dump it all out. System would need
to be able to work with old-style KVs. Comparator would be adjusted to
accomodate new KV. We'd do a tailset each time we made a scanner? This would
be a big change. We should probably bump rpc version and require a restart of
hbase cluster on upgrade.
> New MemStoreScanner copies memstore for each scan, makes short scans slow
> -------------------------------------------------------------------------
>
> Key: HBASE-2248
> URL: https://issues.apache.org/jira/browse/HBASE-2248
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.3
> Reporter: Dave Latham
> Fix For: 0.20.4
>
> Attachments: threads.txt
>
>
> HBASE-2037 introduced a new MemStoreScanner which triggers a
> ConcurrentSkipListMap.buildFromSorted clone of the memstore and snapshot when
> starting a scan.
> After upgrading to 0.20.3, we noticed a big slowdown in our use of short
> scans. Some of our data repesent a time series. The data is stored in time
> series order, MR jobs often insert/update new data at the end of the series,
> and queries usually have to pick up some or all of the series. These are
> often scans of 0-100 rows at a time. To load one page, we'll observe about
> 20 such scans being triggered concurrently, and they take 2 seconds to
> complete. Doing a thread dump of a region server shows many threads in
> ConcurrentSkipListMap.biuldFromSorted which traverses the entire map of key
> values to copy it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
