[
https://issues.apache.org/jira/browse/HADOOP-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12563012#action_12563012
]
Billy Pearson commented on HADOOP-2636:
---------------------------------------
works ok until the first flush then they flush back to back.
I got my flush size set to 64MB and you see below the flush is on the columns
are in the low KB's and they are flushed several times a min
{code}
2008-01-27 13:23:12,428 DEBUG org.apache.hadoop.hbase.HStore: Added
673880687/anchor/3002468228224142295 with 2198 entries, sequence id 2169440,
and size 250.1k for 673880687/anchor
2008-01-27 13:23:12,428 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for store 673880687/anchor in 503ms, sequenceid=2169440
2008-01-27 13:23:12,429 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region webdata,,1201461077996 store 673880687/in_rank
2008-01-27 13:23:13,009 DEBUG org.apache.hadoop.hbase.HStore: Added
673880687/in_rank/8574408391914465288 with 1428 entries, sequence id 2171549,
and size 158.0k for 673880687/in_rank
2008-01-27 13:23:13,010 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for store 673880687/in_rank in 581ms, sequenceid=2171549
2008-01-27 13:23:13,011 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region webdata,,1201461077996 store 673880687/last_seen
2008-01-27 13:23:13,603 DEBUG org.apache.hadoop.hbase.HStore: Added
673880687/last_seen/1870135185038371518 with 2736 entries, sequence id 2173487,
and size 295.4k for 673880687/last_seen
2008-01-27 13:23:13,603 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for store 673880687/last_seen in 592ms, sequenceid=2173487
2008-01-27 13:23:13,609 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region webdata,,1201461077996 store 673880687/anchor
2008-01-27 13:23:14,179 DEBUG org.apache.hadoop.hbase.HStore: Added
673880687/anchor/580140819100995513 with 1989 entries, sequence id 2175620, and
size 216.4k for 673880687/anchor
2008-01-27 13:23:14,180 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for store 673880687/anchor in 572ms, sequenceid=2175620
2008-01-27 13:23:14,181 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region webdata,,1201461077996 store 673880687/in_rank
2008-01-27 13:23:14,693 DEBUG org.apache.hadoop.hbase.HStore: Added
673880687/in_rank/5202798596377303202 with 1896 entries, sequence id 2177448,
and size 213.7k for 673880687/in_rank
2008-01-27 13:23:14,693 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for store 673880687/in_rank in 512ms, sequenceid=2177448
2008-01-27 13:23:14,696 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region webdata,,1201461077996 store 673880687/last_seen
2008-01-27 13:23:15,260 DEBUG org.apache.hadoop.hbase.HStore: Added
673880687/last_seen/8482135742832457522 with 1940 entries, sequence id 2179528,
and size 206.2k for 673880687/last_seen
2008-01-27 13:23:15,260 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for store 673880687/last_seen in 564ms, sequenceid=2179528
2008-01-27 13:23:15,276 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region webdata,,1201461077996 store 673880687/anchor
2008-01-27 13:23:15,839 DEBUG org.apache.hadoop.hbase.HStore: Added
673880687/anchor/7980708509699803761 with 1991 entries, sequence id 2181788,
and size 204.6k for 673880687/anchor
2008-01-27 13:23:15,839 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for store 673880687/anchor in 564ms, sequenceid=2181788
2008-01-27 13:23:15,840 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region webdata,,1201461077996 store 673880687/in_rank
2008-01-27 13:23:16,423 DEBUG org.apache.hadoop.hbase.HStore: Added
673880687/in_rank/1305487102801331533 with 2150 entries, sequence id 2184079,
and size 221.0k for 673880687/in_rank
2008-01-27 13:23:16,423 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for store 673880687/in_rank in 583ms, sequenceid=2184079
2008-01-27 13:23:16,425 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region webdata,,1201461077996 store 673880687/last_seen
2008-01-27 13:23:17,138 DEBUG org.apache.hadoop.hbase.HStore: Added
673880687/last_seen/1135861004256783910 with 2258 entries, sequence id 2186492,
and size 221.2k for 673880687/last_seen
2008-01-27 13:23:17,138 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for store 673880687/last_seen in 713ms, sequenceid=2186492
2008-01-27 13:23:17,139 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region webdata,,1201461077996 store 673880687/anchor
2008-01-27 13:23:17,366 INFO org.apache.hadoop.hbase.HRegionServer: Rolling
hlog. Number of entries: 30003
2008-01-27 13:23:17,738 DEBUG org.apache.hadoop.hbase.HStore: Added
673880687/anchor/5010915044698800730 with 2232 entries, sequence id 2188716,
and size 236.5k for 673880687/anchor
2008-01-27 13:23:17,738 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for store 673880687/anchor in 599ms, sequenceid=2188716
{code}
> [hbase] Make cache flush triggering less simplistic
> ---------------------------------------------------
>
> Key: HADOOP-2636
> URL: https://issues.apache.org/jira/browse/HADOOP-2636
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Reporter: stack
> Assignee: Jim Kellerman
> Fix For: 0.17.0
>
> Attachments: patch.txt, patch.txt, patch.txt
>
>
> When flusher runs -- its triggered when the sum of all Stores in a Region > a
> configurable max size -- we flush all Stores though a Store memcache might
> have but a few bytes.
> I would think Stores should only dump their memcache disk if they have some
> substance.
> The problem becomes more acute, the more families you have in a Region.
> Possible behaviors would be to dump the biggest Store only, or only those
> Stores > 50% of max memcache size. Behavior would vary dependent on the
> prompt that provoked the flush. Would also log why the flush is running:
> optional or > max size.
> This issue comes out of HADOOP-2621.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
