[
https://issues.apache.org/jira/browse/HADOOP-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564568#action_12564568
]
Billy Pearson commented on HADOOP-2636:
---------------------------------------
latest patch #5
I get a ton of these per sec after the first flush starts
{code}
2008-01-31 18:01:09,961 DEBUG org.apache.hadoop.hbase.HRegionServer: not
queuing flush request because one is already in progress
2008-01-31 18:01:09,961 DEBUG org.apache.hadoop.hbase.HRegionServer: not
queuing compaction request because one is already in progress
2008-01-31 18:01:09,961 DEBUG org.apache.hadoop.hbase.HRegionServer: not
queuing flush request because one is already in queue
2008-01-31 18:01:09,961 DEBUG org.apache.hadoop.hbase.HRegionServer: not
queuing compaction because one is already in the queue
2008-01-31 18:01:09,961 DEBUG org.apache.hadoop.hbase.HRegionServer: not
queuing flush request because one is already in queue
2008-01-31 18:01:09,962 DEBUG org.apache.hadoop.hbase.HRegionServer: not
queuing compaction because one is already in the queue
{code}
but still flushing lots of small flushes back top back
flush size set to 16MB (16777216)
I comment out the above logs from the code so I could get this part
{code}
2008-01-31 18:26:09,314 DEBUG org.apache.hadoop.hbase.HStore: Added
51984898/in_rank/7445623251870888179 with 151286 entries, sequence id 468191,
and size 16.0m for 51984898/in_rank
2008-01-31 18:26:09,315 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for store 51984898/in_rank in 5918ms, sequenceid=468191
2008-01-31 18:26:09,368 DEBUG org.apache.hadoop.hbase.HLog: Closing current log
writer
hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/log_10.0.0.2_1201825376852_60020/hlog.dat.015
to get a new one
2008-01-31 18:26:09,374 INFO org.apache.hadoop.hbase.HLog: new log writer
created at
hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/log_10.0.0.2_1201825376852_60020/hlog.dat.016
2008-01-31 18:26:09,374 DEBUG org.apache.hadoop.hbase.HLog: Found 0 logs to
remove using oldest outstanding seqnum of 0 from HStore 51984898/size
2008-01-31 18:26:09,408 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region webdata,,1201825394888 store 51984898/anchor
2008-01-31 18:26:16,574 DEBUG org.apache.hadoop.hbase.HStore: Added
51984898/anchor/5033243542266094285 with 159021 entries, sequence id 492132,
and size 16.4m for 51984898/anchor
2008-01-31 18:26:16,574 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for store 51984898/anchor in 7166ms, sequenceid=492132
2008-01-31 18:26:16,611 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region webdata,,1201825394888 store 51984898/last_seen
2008-01-31 18:26:17,223 INFO org.apache.hadoop.hbase.HRegionServer: Rolling
hlog. Number of entries: 30018
2008-01-31 18:26:22,469 DEBUG org.apache.hadoop.hbase.HStore: Added
51984898/last_seen/789487054095285345 with 167871 entries, sequence id 519368,
and size 17.2m for 51984898/last_seen
2008-01-31 18:26:22,470 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for store 51984898/last_seen in 5859ms, sequenceid=519368
2008-01-31 18:26:22,520 DEBUG org.apache.hadoop.hbase.HLog: Closing current log
writer
hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/log_10.0.0.2_1201825376852_60020/hlog.dat.016
to get a new one
2008-01-31 18:26:22,526 INFO org.apache.hadoop.hbase.HLog: new log writer
created at
hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/log_10.0.0.2_1201825376852_60020/hlog.dat.017
2008-01-31 18:26:22,526 DEBUG org.apache.hadoop.hbase.HLog: Found 0 logs to
remove using oldest outstanding seqnum of 0 from HStore 51984898/size
2008-01-31 18:26:22,532 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region webdata,,1201825394888 store 51984898/in_rank
2008-01-31 18:26:23,643 DEBUG org.apache.hadoop.hbase.HStore: Added
51984898/in_rank/6607743928392488812 with 24913 entries, sequence id 545013,
and size 2.7m for 51984898/in_rank
2008-01-31 18:26:23,643 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for store 51984898/in_rank in 1111ms, sequenceid=545013
2008-01-31 18:26:23,648 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region webdata,,1201825394888 store 51984898/anchor
2008-01-31 18:26:24,707 DEBUG org.apache.hadoop.hbase.HStore: Added
51984898/anchor/3932738492736374231 with 18919 entries, sequence id 550311, and
size 1.9m for 51984898/anchor
2008-01-31 18:26:24,707 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for store 51984898/anchor in 1059ms, sequenceid=550311
2008-01-31 18:26:24,710 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region webdata,,1201825394888 store 51984898/last_seen
2008-01-31 18:26:25,352 DEBUG org.apache.hadoop.hbase.HStore: Added
51984898/last_seen/5324440789419095782 with 11724 entries, sequence id 555402,
and size 1.1m for 51984898/last_seen
2008-01-31 18:26:25,352 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for store 51984898/last_seen in 642ms, sequenceid=555402
2008-01-31 18:26:25,355 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region webdata,,1201825394888 store 51984898/in_rank
2008-01-31 18:26:25,869 DEBUG org.apache.hadoop.hbase.HStore: Added
51984898/in_rank/2378178173153292440 with 4425 entries, sequence id 558549, and
size 491.2k for 51984898/in_rank
2008-01-31 18:26:25,870 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for store 51984898/in_rank in 515ms, sequenceid=558549
{code}
Flushes are still way smaller after the first flush then the set size.
I thank we should be looking at the process that calls the flush its seams the
be the part thats going crazy after the first flush maybe some loop we are not
getting out of or something along those lines.
> [hbase] Make cache flush triggering less simplistic
> ---------------------------------------------------
>
> Key: HADOOP-2636
> URL: https://issues.apache.org/jira/browse/HADOOP-2636
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/hbase
> Affects Versions: 0.16.0
> Reporter: stack
> Assignee: Jim Kellerman
> Fix For: 0.17.0
>
> Attachments: patch.txt, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> When flusher runs -- its triggered when the sum of all Stores in a Region > a
> configurable max size -- we flush all Stores though a Store memcache might
> have but a few bytes.
> I would think Stores should only dump their memcache disk if they have some
> substance.
> The problem becomes more acute, the more families you have in a Region.
> Possible behaviors would be to dump the biggest Store only, or only those
> Stores > 50% of max memcache size. Behavior would vary dependent on the
> prompt that provoked the flush. Would also log why the flush is running:
> optional or > max size.
> This issue comes out of HADOOP-2621.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
