[
https://issues.apache.org/jira/browse/HBASE-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654178#action_12654178
]
stack commented on HBASE-1048:
------------------------------
Testing I see logs like the following:
{code}
2008-12-07 06:45:05,736 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] INFO
org.apache.hadoop.hbase.regionserver.HLog: Too many logs: logs=41, maxlogs=40;
forcing flush of region with oldest edits: -ROOT-,,0
2008-12-07 06:45:29,346 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] INFO
org.apache.hadoop.hbase.regionserver.HLog: Too many logs: logs=42, maxlogs=40;
forcing flush of region with oldest edits: -ROOT-,,0
2008-12-07 06:45:46,294 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] INFO
org.apache.hadoop.hbase.regionserver.HLog: Too many logs: logs=43, maxlogs=40;
forcing flush of region with oldest edits: .META.,,1
{code}
If I just extract the HLog edits, then I see:
{code}
2008-12-07 06:44:39,945 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] DEBUG
org.apache.hadoop.hbase.regionserver.HLog: Found 0 logs to remove out of total
38; oldest outstanding seqnum is 2 from region -ROOT-,,0
2008-12-07 06:44:45,892 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] INFO
org.apache.hadoop.hbase.regionserver.HLog: Closed
hdfs://aa0-000-12.u.powerset.com:9000/hbasetrunk2/log_208.76.44.140_1228631862878_60020/hlog.dat.1228632271713,
entries=100008. New log writer:
/hbasetrunk2/log_208.76.44.140_1228631862878_60020/hlog.dat.1228632285857
2008-12-07 06:44:45,892 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] DEBUG
org.apache.hadoop.hbase.regionserver.HLog: Found 0 logs to remove out of total
39; oldest outstanding seqnum is 2 from region -ROOT-,,0
2008-12-07 06:44:52,674 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] INFO
org.apache.hadoop.hbase.regionserver.HLog: Closed
hdfs://aa0-000-12.u.powerset.com:9000/hbasetrunk2/log_208.76.44.140_1228631862878_60020/hlog.dat.1228632279929,
entries=112307. New log writer:
/hbasetrunk2/log_208.76.44.140_1228631862878_60020/hlog.dat.1228632292660
2008-12-07 06:44:52,674 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] DEBUG
org.apache.hadoop.hbase.regionserver.HLog: Found 0 logs to remove out of total
40; oldest outstanding seqnum is 2 from region -ROOT-,,0
2008-12-07 06:45:05,736 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] INFO
org.apache.hadoop.hbase.regionserver.HLog: Closed
hdfs://aa0-000-12.u.powerset.com:9000/hbasetrunk2/log_208.76.44.140_1228631862878_60020/hlog.dat.1228632285857,
entries=180201. New log writer:
/hbasetrunk2/log_208.76.44.140_1228631862878_60020/hlog.dat.1228632305721
2008-12-07 06:45:05,736 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] DEBUG
org.apache.hadoop.hbase.regionserver.HLog: Found 0 logs to remove out of total
41; oldest outstanding seqnum is 2 from region -ROOT-,,0
2008-12-07 06:45:05,736 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] INFO
org.apache.hadoop.hbase.regionserver.HLog: Too many logs: logs=41, maxlogs=40;
forcing flush of region with oldest edits: -ROOT-,,0
2008-12-07 06:45:29,346 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] INFO
org.apache.hadoop.hbase.regionserver.HLog: Closed
hdfs://aa0-000-12.u.powerset.com:9000/hbasetrunk2/log_208.76.44.140_1228631862878_60020/hlog.dat.1228632292660,
entries=249390. New log writer:
/hbasetrunk2/log_208.76.44.140_1228631862878_60020/hlog.dat.1228632329343
2008-12-07 06:45:29,346 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] DEBUG
org.apache.hadoop.hbase.regionserver.HLog: Found 0 logs to remove out of total
42; oldest outstanding seqnum is 2 from region -ROOT-,,0
2008-12-07 06:45:29,346 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] INFO
org.apache.hadoop.hbase.regionserver.HLog: Too many logs: logs=42, maxlogs=40;
forcing flush of region with oldest edits: -ROOT-,,0
2008-12-07 06:45:46,294 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] INFO
org.apache.hadoop.hbase.regionserver.HLog: Closed
hdfs://aa0-000-12.u.powerset.com:9000/hbasetrunk2/log_208.76.44.140_1228631862878_60020/hlog.dat.1228632305721,
entries=132451. New log writer:
/hbasetrunk2/log_208.76.44.140_1228631862878_60020/hlog.dat.1228632346266
2008-12-07 06:45:46,294 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] DEBUG
org.apache.hadoop.hbase.regionserver.HLog: Found 0 logs to remove out of total
43; oldest outstanding seqnum is 4 from region .META.,,1
2008-12-07 06:45:46,294 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] INFO
org.apache.hadoop.hbase.regionserver.HLog: Too many logs: logs=43, maxlogs=40;
forcing flush of region with oldest edits: .META.,,1
2008-12-07 06:45:55,785 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] INFO
org.apache.hadoop.hbase.regionserver.HLog: Closed
hdfs://aa0-000-12.u.powerset.com:9000/hbasetrunk2/log_208.76.44.140_1228631862878_60020/hlog.dat.1228632329343,
entries=155862. New log writer:
/hbasetrunk2/log_208.76.44.140_1228631862878_60020/hlog.dat.1228632355780
2008-12-07 06:45:55,785 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] DEBUG
org.apache.hadoop.hbase.regionserver.HLog: Found 42 logs to remove out of
total 44; oldest outstanding seqnum is 6335280 from region
TestTable,0006486018,1228632049567
2008-12-07 06:45:55,785 [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] INFO
org.apache.hadoop.hbase.regionserver.HLog: removing old log file
/hbasetrunk2/log_208.76.44.140_1228631862878_60020/hlog.dat.0 whose highest
sequence/edit id is 114842
{code}
Notice how the flushing of the root and meta allowed us clean up 42 of 44 logs.
Committing because it seems to work. Will make default 100 though instead of
40 logs so we flush meta and root less often.
> HLog: Found 0 logs to remove out of total 1450; oldest outstanding seqnum is
> 162297053 fr om region -ROOT-,,0
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1048
> URL: https://issues.apache.org/jira/browse/HBASE-1048
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.19.0
>
> Attachments: 1048.patch
>
>
> Above was in tim sell log just before it OOME'd. 1450 log files are going to
> mess us up. Need to clear them. Catalog tables should probably keep up the
> old optional flush or, if N commit logs, schedule a flush on the region with
> the oldest sequent number.
> Bringing into 0.19.0. Gets in the way of our scaling.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
