[
https://issues.apache.org/jira/browse/HBASE-812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12621608#action_12621608
]
Billy Pearson commented on HBASE-812:
-------------------------------------
I am running the 0.2.0 release and no its not compacting a reference. after a
split we do not do the skip algo when we are compacting a reference its a force
compaction of all the files.
I have more examples thats just one of the columns but we are doing this a lot
when I am running a import job.
But the skip algo is working great over all I see compaction's of about 1 min
vs 5-7 mins per compaction while under insert load.
There is some in the below log I just found a group of compaction's and cut
this out of the log I thank its after a startup but still you can see the skip
algo picks 2 on
some of them and about half it only has 1 left we just need to skip the columns
that only have 1 mapfile after the algo does the selection of the files. then
on the next
compaction for the region it will have 2 mapfiles and do something when its
compacting.
{code}
2008-08-10 23:56:13,563 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
moving
/hbase/webdata/compaction.dir/200761489/anchor/mapfiles/4695022772964499808 to
/hbase/webdata/200761489/anchor/mapfiles/6542744344240857591
2008-08-10 23:56:13,631 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 200761489/anchor store size is 72.8m
2008-08-10 23:56:13,637 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Compaction size of 200761489/stime: 5.1m, skipped 1, 4191303
2008-08-10 23:56:13,706 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
started compaction of 2 files into
/hbase/webdata/compaction.dir/200761489/stime/mapfiles/6863349056798671266
2008-08-10 23:56:15,559 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
moving
/hbase/webdata/compaction.dir/200761489/stime/mapfiles/6863349056798671266 to
/hbase/webdata/200761489/stime/mapfiles/3093620455129104149
2008-08-10 23:56:15,639 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 200761489/stime store size is 5.1m
2008-08-10 23:56:15,647 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Compaction size of 200761489/in_rank: 62.5m, skipped 1, 55688666
2008-08-10 23:56:15,724 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
started compaction of 2 files into
/hbase/webdata/compaction.dir/200761489/in_rank/mapfiles/1559371687332050999
2008-08-10 23:56:32,318 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
moving
/hbase/webdata/compaction.dir/200761489/in_rank/mapfiles/1559371687332050999 to
/hbase/webdata/200761489/in_rank/mapfiles/5872775000657698199
2008-08-10 23:56:32,400 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 200761489/in_rank store size is 61.1m
2008-08-10 23:56:32,407 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Compaction size of 200761489/size: 5.5m, skipped 1, 4465439
2008-08-10 23:56:32,472 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
started compaction of 2 files into
/hbase/webdata/compaction.dir/200761489/size/mapfiles/3296022424629716093
2008-08-10 23:56:34,710 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
moving
/hbase/webdata/compaction.dir/200761489/size/mapfiles/3296022424629716093 to
/hbase/webdata/200761489/size/mapfiles/8583717577394542471
2008-08-10 23:56:34,782 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 200761489/size store size is 5.4m
2008-08-10 23:56:34,789 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Compaction size of 200761489/last_seen: 61.5m, skipped 1, 54824835
2008-08-10 23:56:34,872 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
started compaction of 2 files into
/hbase/webdata/compaction.dir/200761489/last_seen/mapfiles/3943134197356605963
2008-08-10 23:56:50,981 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
moving
/hbase/webdata/compaction.dir/200761489/last_seen/mapfiles/3943134197356605963
to /hbase/webdata/200761489/last_seen/mapfiles/307647539041731711
2008-08-10 23:56:51,219 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 200761489/last_seen store size is 60.1m
2008-08-10 23:56:51,242 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webdata,de.popstars-new-angels.www%2Fpopstars-biographien.html%3Ahttp,1218422328154
in 52sec
2008-08-10 23:56:51,245 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region
webdata,fr.asso.agirpourlenfant.www%2Findex.html%3Ahttp,1218422328154
2008-08-10 23:56:51,252 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Compaction size of 338914710/anchor: 81.6m, skipped 2, 85047362
2008-08-10 23:56:51,268 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
started compaction of 1 files into
/hbase/webdata/compaction.dir/338914710/anchor/mapfiles/2546621217286575105
2008-08-10 23:56:52,143 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
moving
/hbase/webdata/compaction.dir/338914710/anchor/mapfiles/2546621217286575105 to
/hbase/webdata/338914710/anchor/mapfiles/3839219944106123077
2008-08-10 23:56:52,188 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 338914710/anchor store size is 80.4m
2008-08-10 23:56:52,195 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Compaction size of 338914710/stime: 5.6m, skipped 2, 5784755
2008-08-10 23:56:52,210 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
started compaction of 1 files into
/hbase/webdata/compaction.dir/338914710/stime/mapfiles/1286630914618726188
2008-08-10 23:56:52,414 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
moving
/hbase/webdata/compaction.dir/338914710/stime/mapfiles/1286630914618726188 to
/hbase/webdata/338914710/stime/mapfiles/5121005992036000876
2008-08-10 23:56:52,456 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 338914710/stime store size is 5.5m
2008-08-10 23:56:52,462 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Compaction size of 338914710/in_rank: 71.3m, skipped 2, 74283017
2008-08-10 23:56:52,486 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
started compaction of 1 files into
/hbase/webdata/compaction.dir/338914710/in_rank/mapfiles/1957109104676001133
2008-08-10 23:56:53,524 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
moving
/hbase/webdata/compaction.dir/338914710/in_rank/mapfiles/1957109104676001133 to
/hbase/webdata/338914710/in_rank/mapfiles/9145853810894409994
2008-08-10 23:56:53,569 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 338914710/in_rank store size is 69.8m
2008-08-10 23:56:53,576 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Compaction size of 338914710/size: 5.9m, skipped 2, 6167045
2008-08-10 23:56:53,589 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
started compaction of 1 files into
/hbase/webdata/compaction.dir/338914710/size/mapfiles/5911079568309110370
2008-08-10 23:56:53,809 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
moving
/hbase/webdata/compaction.dir/338914710/size/mapfiles/5911079568309110370 to
/hbase/webdata/338914710/size/mapfiles/5518032354875458699
2008-08-10 23:56:53,853 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 338914710/size store size is 5.9m
2008-08-10 23:56:53,860 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Compaction size of 338914710/last_seen: 70.6m, skipped 2, 73521266
2008-08-10 23:56:53,902 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
started compaction of 1 files into
/hbase/webdata/compaction.dir/338914710/last_seen/mapfiles/4617217208970844256
2008-08-10 23:56:54,898 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
moving
/hbase/webdata/compaction.dir/338914710/last_seen/mapfiles/4617217208970844256
to /hbase/webdata/338914710/last_seen/mapfiles/6045607152317204195
2008-08-10 23:56:54,943 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 338914710/last_seen store size is 69.1m
2008-08-10 23:56:54,948 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webdata,fr.asso.agirpourlenfant.www%2Findex.html%3Ahttp,1218422328154 in 3sec
2008-08-10 23:56:54,951 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region
webdata,org.jessejacksonjr.www%2Fissues%2Fi0809001569.html%3Ahttp,1218423851154
2008-08-10 23:56:54,961 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Compaction size of 376160388/anchor: 70.0m, skipped 2, 72330348
2008-08-10 23:56:55,258 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
started compaction of 2 files into
/hbase/webdata/compaction.dir/376160388/anchor/mapfiles/6390472557291621209
2008-08-10 23:56:56,953 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
moving
/hbase/webdata/compaction.dir/376160388/anchor/mapfiles/6390472557291621209 to
/hbase/webdata/376160388/anchor/mapfiles/6570235857466899342
2008-08-10 23:56:57,001 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 376160388/anchor store size is 69.0m
2008-08-10 23:56:57,008 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Compaction size of 376160388/stime: 4.5m, skipped 1, 4373520
2008-08-10 23:56:57,041 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
started compaction of 2 files into
/hbase/webdata/compaction.dir/376160388/stime/mapfiles/8058281260768773414
2008-08-10 23:56:57,655 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
moving
/hbase/webdata/compaction.dir/376160388/stime/mapfiles/8058281260768773414 to
/hbase/webdata/376160388/stime/mapfiles/6655752511205630946
2008-08-10 23:56:57,710 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 376160388/stime store size is 4.4m
2008-08-10 23:56:57,719 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Compaction size of 376160388/in_rank: 60.7m, skipped 2, 62605727
2008-08-10 23:56:58,020 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
started compaction of 2 files into
/hbase/webdata/compaction.dir/376160388/in_rank/mapfiles/6388173210597572693
2008-08-10 23:56:59,448 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
moving
/hbase/webdata/compaction.dir/376160388/in_rank/mapfiles/6388173210597572693 to
/hbase/webdata/376160388/in_rank/mapfiles/7217625771579221464
2008-08-10 23:56:59,498 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 376160388/in_rank store size is 59.4m
2008-08-10 23:56:59,504 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Compaction size of 376160388/size: 4.8m, skipped 1, 4651030
2008-08-10 23:56:59,912 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
started compaction of 2 files into
/hbase/webdata/compaction.dir/376160388/size/mapfiles/6505866612377469653
2008-08-10 23:57:00,566 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
moving
/hbase/webdata/compaction.dir/376160388/size/mapfiles/6505866612377469653 to
/hbase/webdata/376160388/size/mapfiles/3951148926146160516
2008-08-10 23:57:00,612 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 376160388/size store size is 4.7m
2008-08-10 23:57:00,621 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Compaction size of 376160388/last_seen: 59.8m, skipped 2, 61665980
2008-08-10 23:57:00,814 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
started compaction of 2 files into
/hbase/webdata/compaction.dir/376160388/last_seen/mapfiles/6278945227615596465
2008-08-10 23:57:02,202 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
moving
/hbase/webdata/compaction.dir/376160388/last_seen/mapfiles/6278945227615596465
to /hbase/webdata/376160388/last_seen/mapfiles/4551300457309362976
2008-08-10 23:57:02,249 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 376160388/last_seen store size is 58.5m
2008-08-10 23:57:02,253 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webdata,org.jessejacksonjr.www%2Fissues%2Fi0809001569.html%3Ahttp,1218423851154
in 7sec
{code}
> Compaction needs little better skip algo
> ----------------------------------------
>
> Key: HBASE-812
> URL: https://issues.apache.org/jira/browse/HBASE-812
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.2.0, 0.3.0
> Reporter: Billy Pearson
> Priority: Critical
> Fix For: 0.2.1, 0.3.0
>
>
> Looking at this section of one of my compaction's we have 3 files to compact
> the new algo is working great in my test but I see this below often we are
> skipping 2 out of the 3 files and compacting 1 file. 1 file is kind of a wast
> might as well just copy the file my suggestion is if there is only 1 file
> left after the new algo skips then just go on to the next column and skip the
> last file also. This will help improve compaction times a little more.
> {code}
> 2008-08-10 10:00:45,310 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
> Compaction size of 1339600874/size: 4.6m, skipped 2, 4851776
> 2008-08-10 10:00:45,438 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
> started compaction of 1 files into
> /hbase/webdata/compaction.dir/1339600874/size/mapfiles/8653208152776334891
> 2008-08-10 10:00:46,838 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
> moving
> /hbase/webdata/compaction.dir/1339600874/size/mapfiles/8653208152776334891 to
> /hbase/webdata/1339600874/size/mapfiles/7539342470259528578
> 2008-08-10 10:00:47,166 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
> Completed compaction of 1339600874/size store size is 4.6m
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
