[
https://issues.apache.org/jira/browse/HADOOP-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561247#action_12561247
]
viper799 edited comment on HADOOP-2673 at 1/21/08 11:54 PM:
-----------------------------------------------------------------
just adding some stats about this problem
{code}
2008-01-22 01:07:31,944 DEBUG org.apache.hadoop.hbase.HStore: started
compaction of 432 files using
hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/compaction.dir for
540402979/in_rank
2008-01-22 01:23:17,384 DEBUG org.apache.hadoop.hbase.HStore: moving
540402979/in_rank/8016207671941404930 in
hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/compaction.dir to
540402979/in_rank/8587356094792327071 in
hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata for 540402979/in_rank
2008-01-22 01:23:22,605 DEBUG org.apache.hadoop.hbase.HStore: started
compaction of 328 files using
hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/compaction.dir for
540402979/rank_visual
2008-01-22 01:23:31,248 DEBUG org.apache.hadoop.hbase.HStore: moving
540402979/rank_visual/4003180598508023113 in
hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/compaction.dir to
540402979/rank_visual/5001502002807562425 in
hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata for
540402979/rank_visual
2008-01-22 01:23:34,438 INFO org.apache.hadoop.hbase.HRegion: compaction
completed on region webdata,,1200970130265. Took 57mins, 10sec
2008-01-22 01:23:34,439 DEBUG org.apache.hadoop.hbase.HStore: compaction for
HStore 540402979/size needed.
2008-01-22 01:23:34,439 DEBUG org.apache.hadoop.hbase.HRegion: 540402979/size
needs compaction
2008-01-22 01:23:34,439 INFO org.apache.hadoop.hbase.HRegion: starting
compaction on region webdata,,1200970130265
2008-01-22 01:23:34,439 DEBUG org.apache.hadoop.hbase.HStore: started
compaction of 323 files using
hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/compaction.dir for
540402979/size
{code}
after finished compacting 432 files on the last column the compaction took
57min and 10 sec in that time there was 323 more flushes to the first column.
the job stop writeing to the table at 01:23:34
was (Author: viper799):
just adding some stats about this problem
{code}
2008-01-22 01:07:31,944 DEBUG org.apache.hadoop.hbase.HStore: started
compaction of 432 files using
hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/compaction.dir for
540402979/in_rank
2008-01-22 01:23:17,384 DEBUG org.apache.hadoop.hbase.HStore: moving
540402979/in_rank/8016207671941404930 in
hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/compaction.dir to
540402979/in_rank/8587356094792327071 in
hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata for 540402979/in_rank
2008-01-22 01:23:22,605 DEBUG org.apache.hadoop.hbase.HStore: started
compaction of 328 files using
hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/compaction.dir for
540402979/rank_visual
2008-01-22 01:23:31,248 DEBUG org.apache.hadoop.hbase.HStore: moving
540402979/rank_visual/4003180598508023113 in
hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/compaction.dir to
540402979/rank_visual/5001502002807562425 in
hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata for
540402979/rank_visual
2008-01-22 01:23:34,438 INFO org.apache.hadoop.hbase.HRegion: compaction
completed on region webdata,,1200970130265. Took 57mins, 10sec
2008-01-22 01:23:34,439 DEBUG org.apache.hadoop.hbase.HStore: compaction for
HStore 540402979/size needed.
2008-01-22 01:23:34,439 DEBUG org.apache.hadoop.hbase.HRegion: 540402979/size
needs compaction
2008-01-22 01:23:34,439 INFO org.apache.hadoop.hbase.HRegion: starting
compaction on region webdata,,1200970130265
2008-01-22 01:23:34,439 DEBUG org.apache.hadoop.hbase.HStore: started
compaction of 323 files using
hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/compaction.dir for
540402979/size
{code}
after finished compacting 432 files on the last column the compaction took
57min and 10 sec in that time there was 323 more flushes to the first column.
> Flusher flushing back to back when it should not be
> ---------------------------------------------------
>
> Key: HADOOP-2673
> URL: https://issues.apache.org/jira/browse/HADOOP-2673
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Environment: hadoop svn 613782
> Reporter: Billy Pearson
> Fix For: 0.16.0
>
> Attachments: log.log
>
>
> settings:
> hbase.regionserver.optionalcacheflushinterval = 300000 (5 mins)
> hbase.hregion.memcache.flush.size = 67108864 (64MB)
> I am attaching a log for this everything works fine until the first flush of
> the memcache. but once it flushes the first time correctly it flushes back to
> back many times even when it does not exceed one of the two limits above.
> I have not found but one spot where we update the variable LastFlushTime
> except when the server starts so this could be the problem but I may be
> missing it in the code somewhere. If it is getting updated then we need to
> look if something could be blocking it from updating after a memcache flush.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
