[
https://issues.apache.org/jira/browse/HBASE-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683995#action_12683995
]
Jean-Daniel Cryans commented on HBASE-1008:
-------------------------------------------
A multi-threaded version that I run at openplaces was able to process 33 logs
in a "record" time and our job didn't even failed like it usually does :
{quote}
2009-03-20 15:06:24,047 INFO org.apache.hadoop.hbase.regionserver.HLog:
Splitting 33 log(s) in
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020
2009-03-20 15:06:24,047 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 1 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237560687378
2009-03-20 15:06:24,106 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Adding
queue for entities,,1236805004423
2009-03-20 15:06:25,443 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
100006 entries
2009-03-20 15:06:25,459 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 2 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237560790320
2009-03-20 15:06:25,879 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Adding
queue for hbase_types,426,1225564254435
2009-03-20 15:06:27,101 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
100867 entries
2009-03-20 15:06:27,103 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 3 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561649939
2009-03-20 15:06:28,694 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
101754 entries
2009-03-20 15:06:28,696 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 4 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561658514
2009-03-20 15:06:33,324 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
332220 entries
2009-03-20 15:06:33,327 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 5 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561669181
2009-03-20 15:06:38,707 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
349439 entries
2009-03-20 15:06:38,711 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 6 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561688463
2009-03-20 15:06:40,922 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
207909 entries
2009-03-20 15:06:40,925 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 7 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561698495
2009-03-20 15:06:42,048 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
102829 entries
2009-03-20 15:06:42,050 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 8 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561703659
2009-03-20 15:06:44,199 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
204528 entries
2009-03-20 15:06:44,201 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 9 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561715064
2009-03-20 15:06:46,875 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
225964 entries
2009-03-20 15:06:46,878 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 10 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561726289
2009-03-20 15:06:47,885 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
100645 entries
2009-03-20 15:06:47,887 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 11 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561793535
2009-03-20 15:06:49,198 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
105605 entries
2009-03-20 15:06:49,222 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 12 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561854543
2009-03-20 15:06:50,227 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
100363 entries
2009-03-20 15:06:50,229 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 13 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561941125
2009-03-20 15:06:51,305 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
100648 entries
2009-03-20 15:06:51,307 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 14 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561953452
2009-03-20 15:06:53,111 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
159954 entries
2009-03-20 15:06:53,113 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 15 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561986701
2009-03-20 15:06:54,450 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
101025 entries
2009-03-20 15:06:54,452 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 16 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562003837
2009-03-20 15:06:55,717 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
100449 entries
2009-03-20 15:06:55,719 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 17 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562016248
2009-03-20 15:06:56,682 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
101244 entries
2009-03-20 15:06:56,699 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 18 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562049500
2009-03-20 15:06:57,749 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
100274 entries
2009-03-20 15:06:57,751 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 19 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562060231
2009-03-20 15:06:59,012 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
111015 entries
2009-03-20 15:06:59,014 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 20 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562127374
2009-03-20 15:06:59,999 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
100373 entries
2009-03-20 15:07:00,001 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 21 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562177943
2009-03-20 15:07:01,001 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
100213 entries
2009-03-20 15:07:01,003 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 22 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562277537
2009-03-20 15:07:04,116 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
192782 entries
2009-03-20 15:07:04,119 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 23 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562304890
2009-03-20 15:07:05,774 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
101842 entries
2009-03-20 15:07:05,776 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 24 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562315409
2009-03-20 15:07:06,843 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
104371 entries
2009-03-20 15:07:06,845 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 25 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562321140
2009-03-20 15:07:08,213 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
102252 entries
2009-03-20 15:07:08,215 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 26 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562342084
2009-03-20 15:07:09,371 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
100330 entries
2009-03-20 15:07:09,373 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 27 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562347414
2009-03-20 15:07:12,583 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
258947 entries
2009-03-20 15:07:12,585 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 28 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562678929
2009-03-20 15:07:13,926 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
100082 entries
2009-03-20 15:07:13,928 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 29 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562683228
2009-03-20 15:07:15,216 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
100115 entries
2009-03-20 15:07:15,218 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 30 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562708153
2009-03-20 15:07:16,418 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
100030 entries
2009-03-20 15:07:16,420 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 31 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562734111
2009-03-20 15:07:17,783 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
100054 entries
2009-03-20 15:07:17,785 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 32 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562748277
2009-03-20 15:07:19,902 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed
100116 entries
2009-03-20 15:07:19,904 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Splitting 33 of 33:
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562763336
2009-03-20 15:07:36,115 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Thread
got 49699 to process
2009-03-20 15:07:36,114 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Thread
got 4357600 to process
2009-03-20 15:07:36,168 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Creating new log file writer for path
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/entities/1200514131/oldlogfile.log
and region entities,,1236805004423
2009-03-20 15:07:36,199 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Creating new log file writer for path
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/entities/862114639/oldlogfile.log
and region hbase_types,426,1225564254435
2009-03-20 15:07:36,599 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Applied 49699 total edits to hbase_types,426,1225564254435 in 484ms
2009-03-20 15:07:44,734 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit for
row <> in tableName .META.: location server 192.168.1.109:62020, location
region name .META.,,1
2009-03-20 15:07:51,633 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Applied 4357600 total edits to entities,,1236805004423 in 15517ms
2009-03-20 15:07:51,633 INFO org.apache.hadoop.hbase.regionserver.HLog: Took
87586ms
2009-03-20 15:07:51,650 INFO org.apache.hadoop.hbase.regionserver.HLog: log
file splitting completed for
hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020
{quote}
> [performance] The replay of logs on server crash takes way too long
> -------------------------------------------------------------------
>
> Key: HBASE-1008
> URL: https://issues.apache.org/jira/browse/HBASE-1008
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Priority: Blocker
> Fix For: 0.20.0
>
>
> Watching recovery from a crash on streamy.com where there were 1048 logs and
> repay is running at rate of about 20 seconds each. Meantime these regions
> are not online. This is way too long to wait on recovery for a live site.
> Marking critical. Performance related so priority and in 0.20.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
