That could be the case. Though there was no log statement in isBetterSelection(). So we don't know for sure if mightBeStuck was false.
On Thu, May 28, 2015 at 9:30 AM, Vladimir Rodionov <[email protected]> wrote: > Its possible that logic of ExploringCompactionPolicy (default compaction > policy) is broken. I am looking into this code (master): > > private boolean isBetterSelection(List<StoreFile> bestSelection, > long bestSize, List<StoreFile> selection, long size, boolean > mightBeStuck) { > if (mightBeStuck && bestSize > 0 && size > 0) { > // Keep the selection that removes most files for least size. That > penaltizes adding > // large files to compaction, but not small files, so we don't become > totally inefficient > // (might want to tweak that in future). Also, given the current > order of looking at > // permutations, prefer earlier files and smaller selection if the > difference is small. > final double REPLACE_IF_BETTER_BY = 1.05; > double thresholdQuality = ((double)bestSelection.size() / bestSize) * > REPLACE_IF_BETTER_BY; > return thresholdQuality < ((double)selection.size() / size); > } > // Keep if this gets rid of more files. Or the same number of files > for less io. > return selection.size() > bestSelection.size() > || (selection.size() == bestSelection.size() && size < bestSize); > } > > which compares two selections and what I see here is when mightBeStuck = > false selection with more files will > always be preferred. > > Correct if I am wrong. > > -Vlad > > On Thu, May 28, 2015 at 8:00 AM, Akmal Abbasov <[email protected]> > wrote: > > > Hi Ted, > > Thank you for reply. > > Yes, it was promoted to a major compaction because all files were > > eligible, but the thing I don’t understand, is why all of them were > > eligible? > > afaik, the compaction algorithm should select the best match for > > compaction, and it should include files with similar sizes. > > But as you can see from the logs the files selected have: 4.7K, 5.1K, > 3.8K > > and 10.8M. > > Why it is including 10.8M file? > > Which setting should be tuned to avoid this? > > Thank you. > > > > Kind regards, > > Akmal Abbasov > > > > > On 28 May 2015, at 16:54, Ted Yu <[email protected]> wrote: > > > > > > bq. Completed major compaction of 4 file(s) in s of metrics, > > > V\xA36\x56\x5E\xC5}\xA1\x43\x00\x32\x00T\x1BU\xE0, > > > 3547f43afae5ac3f4e8a162d43a892b4.1417707276446. > > > > > > The compaction involved all the files of store 's' for the region. Thus > > it > > > was considered major compaction. > > > > > > Cheers > > > > > > On Thu, May 28, 2015 at 2:16 AM, Akmal Abbasov < > [email protected] > > > > > > wrote: > > > > > >> Hi Ted, > > >> Sorry for a late reply. > > >> Here is a snippet from log file > > >> 2015-05-28 00:54:39,754 DEBUG > > >> [regionserver60020-smallCompactions-1432714643311] > > >> regionserver.CompactSplitThread: CompactSplitThread Status: > > >> compaction_queue=(0:27), split_queue=0, merge_queue=0 > > >> 2015-05-28 00:54:39,754 DEBUG > > >> [regionserver60020-smallCompactions-1432714643311] > > >> compactions.RatioBasedCompactionPolicy: Selecting compaction from 4 > > store > > >> files, 0 compacting, 4 eligible, 10 blocking > > >> 2015-05-28 00:54:39,755 DEBUG > > >> [regionserver60020-smallCompactions-1432714643311] > > >> compactions.ExploringCompactionPolicy: Exploring compaction algorithm > > has > > >> selected 4 files of size 11304175 starting at candidate #0 after > > >> considering 3 permutations with 3 in ratio > > >> 2015-05-28 00:54:39,755 DEBUG > > >> [regionserver60020-smallCompactions-1432714643311] > regionserver.HStore: > > >> 3547f43afae5ac3f4e8a162d43a892b4 - s: Initiating major compaction > > >> 2015-05-28 00:54:39,755 INFO > > >> [regionserver60020-smallCompactions-1432714643311] > regionserver.HRegion: > > >> Starting compaction on s in region > > >> > > > metrics,V\xA36\x56\x5E\xC5}\xA1\x43\x00\x32\x00T\x1BU\xE0,1417707276446.3547f43afae5ac3f4e8a162d43a892b4. > > >> 2015-05-28 00:54:39,755 INFO > > >> [regionserver60020-smallCompactions-1432714643311] > regionserver.HStore: > > >> Starting compaction of 4 file(s) in s of > > >> > > > metrics,V\xA36\x56\x5E\xC5}\xA1\x43\x00\x32\x00T\x1BU\xE0,1417707276446.3547f43afae5ac3f4e8a162d43a892b4. > > >> into > > >> > > > tmpdir=hdfs://prod1/hbase/data/default/metrics/3547f43afae5ac3f4e8a162d43a892b4/.tmp, > > >> totalSize=10.8 M > > >> 2015-05-28 00:54:39,756 DEBUG > > >> [regionserver60020-smallCompactions-1432714643311] > > compactions.Compactor: > > >> Compacting > > >> > > > hdfs://prod1/hbase/data/default/metrics/3547f43afae5ac3f4e8a162d43a892b4/s/dab3e768593e44a39097451038c5ebd0, > > >> keycount=3203, bloomtype=ROW, size=10.8 M, encoding=NONE, > > seqNum=172299974, > > >> earliestPutTs=1407941317178 > > >> 2015-05-28 00:54:39,756 DEBUG > > >> [regionserver60020-smallCompactions-1432714643311] > > compactions.Compactor: > > >> Compacting > > >> > > > hdfs://prod1/hbase/data/default/metrics/3547f43afae5ac3f4e8a162d43a892b4/s/2d6472ef99a5478689f7ba822bc407a7, > > >> keycount=4, bloomtype=ROW, size=4.7 K, encoding=NONE, > seqNum=172299976, > > >> earliestPutTs=1432761158066 > > >> 2015-05-28 00:54:39,756 DEBUG > > >> [regionserver60020-smallCompactions-1432714643311] > > compactions.Compactor: > > >> Compacting > > >> > > > hdfs://prod1/hbase/data/default/metrics/3547f43afae5ac3f4e8a162d43a892b4/s/bdbc806d045740e69ab34e3ea2e113c4, > > >> keycount=6, bloomtype=ROW, size=5.1 K, encoding=NONE, > seqNum=172299977, > > >> earliestPutTs=1432764757438 > > >> 2015-05-28 00:54:39,756 DEBUG > > >> [regionserver60020-smallCompactions-1432714643311] > > compactions.Compactor: > > >> Compacting > > >> > > > hdfs://prod1/hbase/data/default/metrics/3547f43afae5ac3f4e8a162d43a892b4/s/561f93db484b4b9fb6446152c3eef5b8, > > >> keycount=2, bloomtype=ROW, size=3.8 K, encoding=NONE, > seqNum=172299978, > > >> earliestPutTs=1432768358747 > > >> 2015-05-28 00:54:41,881 DEBUG > > >> [regionserver60020-smallCompactions-1432714643311] > > >> regionserver.HRegionFileSystem: Committing store file > > >> > > > hdfs://prod1/hbase/data/default/metrics/3547f43afae5ac3f4e8a162d43a892b4/.tmp/144f05a9546f446984a5b8fa173dd13e > > >> as > > >> > > > hdfs://prod1/hbase/data/default/metrics/3547f43afae5ac3f4e8a162d43a892b4/s/144f05a9546f446984a5b8fa173dd13e > > >> 2015-05-28 00:54:41,918 DEBUG > > >> [regionserver60020-smallCompactions-1432714643311] > regionserver.HStore: > > >> Removing store files after compaction... > > >> 2015-05-28 00:54:41,959 DEBUG > > >> [regionserver60020-smallCompactions-1432714643311] > backup.HFileArchiver: > > >> Finished archiving from class > > >> org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, > > >> > > > file:hdfs://prod1/hbase/data/default/metrics/3547f43afae5ac3f4e8a162d43a892b4/s/dab3e768593e44a39097451038c5ebd0, > > >> to > > >> > > > hdfs://prod1/hbase/archive/data/default/metrics/3547f43afae5ac3f4e8a162d43a892b4/s/dab3e768593e44a39097451038c5ebd0 > > >> 2015-05-28 00:54:42,030 DEBUG > > >> [regionserver60020-smallCompactions-1432714643311] > backup.HFileArchiver: > > >> Finished archiving from class > > >> org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, > > >> > > > file:hdfs://prod1/hbase/data/default/metrics/3547f43afae5ac3f4e8a162d43a892b4/s/2d6472ef99a5478689f7ba822bc407a7, > > >> to > > >> > > > hdfs://prod1/hbase/archive/data/default/metrics/3547f43afae5ac3f4e8a162d43a892b4/s/2d6472ef99a5478689f7ba822bc407a7 > > >> 2015-05-28 00:54:42,051 DEBUG > > >> [regionserver60020-smallCompactions-1432714643311] > backup.HFileArchiver: > > >> Finished archiving from class > > >> org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, > > >> > > > file:hdfs://prod1/hbase/data/default/metrics/3547f43afae5ac3f4e8a162d43a892b4/s/bdbc806d045740e69ab34e3ea2e113c4, > > >> to > > >> > > > hdfs://prod1/hbase/archive/data/default/metrics/3547f43afae5ac3f4e8a162d43a892b4/s/bdbc806d045740e69ab34e3ea2e113c4 > > >> 2015-05-28 00:54:42,071 DEBUG > > >> [regionserver60020-smallCompactions-1432714643311] > backup.HFileArchiver: > > >> Finished archiving from class > > >> org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, > > >> > > > file:hdfs://prod1/hbase/data/default/metrics/3547f43afae5ac3f4e8a162d43a892b4/s/561f93db484b4b9fb6446152c3eef5b8, > > >> to > > >> > > > hdfs://prod1/hbase/archive/data/default/metrics/3547f43afae5ac3f4e8a162d43a892b4/s/561f93db484b4b9fb6446152c3eef5b8 > > >> 2015-05-28 00:54:42,072 INFO > > >> [regionserver60020-smallCompactions-1432714643311] > regionserver.HStore: > > >> Completed major compaction of 4 file(s) in s of > > >> > > > metrics,V\xA36\x56\x5E\xC5}\xA1\x43\x00\x32\x00T\x1BU\xE0,1417707276446.3547f43afae5ac3f4e8a162d43a892b4. > > >> into 144f05a9546f446984a5b8fa173dd13e(size=10.8 M), total size for > > store is > > >> 10.8 M. This selection was in queue for 0sec, and took 2sec to > execute. > > >> 2015-05-28 00:54:42,072 INFO > > >> [regionserver60020-smallCompactions-1432714643311] > > >> regionserver.CompactSplitThread: Completed compaction: Request = > > >> > > > regionName=metrics,V\xA36\x56\x5E\xC5}\xA1\x43\x00\x32\x00T\x1BU\xE0,1417707276446.3547f43afae5ac3f4e8a162d43a892b4., > > >> storeName=s, fileCount=4, fileSize=10.8 M, priority=6, > > >> time=1368019430741233; duration=2sec > > >> > > >> My question is, why the major compaction was executed instead of minor > > >> compaction. > > >> I have this messages all over the log file. > > >> Thank you! > > >> > > >>> On 12 May 2015, at 23:53, Ted Yu <[email protected]> wrote: > > >>> > > >>> Can you pastebin major compaction related log snippets ? > > >>> See the following for example of such logs: > > >>> > > >>> 2015-05-09 10:57:58,961 INFO > > >>> [PriorityRpcServer.handler=13,queue=1,port=16020] > > >>> regionserver.RSRpcServices: Compacting > > >>> > > >> > > > IntegrationTestBigLinkedList,\x91\x11\x11\x11\x11\x11\x11\x08,1431193978741.700b34f5d2a3aa10804eff35906fd6d8. > > >>> 2015-05-09 10:57:58,962 DEBUG > > >>> [PriorityRpcServer.handler=13,queue=1,port=16020] > regionserver.HStore: > > >>> Skipping expired store file removal due to min version being 1 > > >>> 2015-05-09 10:57:58,962 DEBUG > > >>> [PriorityRpcServer.handler=13,queue=1,port=16020] > > >>> compactions.RatioBasedCompactionPolicy: Selecting compaction from 5 > > store > > >>> files, 0 compacting, 5 eligible, 10 blocking > > >>> 2015-05-09 10:57:58,963 DEBUG > > >>> [PriorityRpcServer.handler=13,queue=1,port=16020] > regionserver.HStore: > > >>> 700b34f5d2a3aa10804eff35906fd6d8 - meta: Initiating major compaction > > (all > > >>> files) > > >>> > > >>> > > >>> Cheers > > >>> > > >>> On Tue, May 12, 2015 at 2:06 PM, Akmal Abbasov < > > [email protected] > > >>> > > >>> wrote: > > >>> > > >>>> Hi Ted, > > >>>> Thank you for reply. > > >>>> I am running with the default settings. > > >>>> > > >>>> Sent from my iPhone > > >>>> > > >>>>> On 12 May 2015, at 22:02, Ted Yu <[email protected]> wrote: > > >>>>> > > >>>>> Can you show us compaction related parameters you use ? > > >>>>> > > >>>>> e.g. hbase.hregion.majorcompaction , > > >>>> hbase.hregion.majorcompaction.jitter , > > >>>>> etc > > >>>>> > > >>>>> On Tue, May 12, 2015 at 9:52 AM, Akmal Abbasov < > > >> [email protected] > > >>>>> > > >>>>> wrote: > > >>>>> > > >>>>>> HI, > > >>>>>> I am using HBase 0.98.7. > > >>>>>> I am using HBase snapshots to backup data. I create snapshot of > > tables > > >>>>>> each our. > > >>>>>> Each create snapshot process will cause the flush of the memstore, > > and > > >>>>>> creation of hfiles. > > >>>>>> When the number of hfiles will reach 3 the MINOR compaction > process > > >> will > > >>>>>> start for each CF. > > >>>>>> Ok, I was expecting that the compaction will process only small > > >> hfiles, > > >>>>>> and I won’t have problems with moving all data to archive folder > > >>>>>> each time after compaction process ends. > > >>>>>> But most of the times, the minor compaction is promoted to > > major(more > > >>>> than > > >>>>>> 100 in 24 hours without loads). > > >>>>>> As far as I know, the only possibility for this is that all hfiles > > are > > >>>>>> eligible for compaction. > > >>>>>> But when I tested the archive folder for a CF I see the strange > > >>>> situation > > >>>>>> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 06:04 > > >>>>>> > > >>>> > > >> > > > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/36dc06f4c34242daadc343d857a35734 > > >>>>>> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 06:04 > > >>>>>> > > >>>> > > >> > > > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/7e8b993f97b84f4594542144f15b0a1e > > >>>>>> -rw-r--r-- 3 akmal supergroup 1.1 K 2015-05-10 06:04 > > >>>>>> > > >>>> > > >> > > > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/b9afc64792ba4bf99a08f34033cc46ac > > >>>>>> -rw-r--r-- 3 akmal supergroup 638.4 K 2015-05-10 06:04 > > >>>>>> > > >>>> > > >> > > > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/dff846ae4fc24d418289a95322b35d46 > > >>>>>> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 08:50 > > >>>>>> > > >>>> > > >> > > > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/228eee22c32e458e8eb7f5d031f64b58 > > >>>>>> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 08:50 > > >>>>>> > > >>>> > > >> > > > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/529257432308466f971e41db49ecffdf > > >>>>>> -rw-r--r-- 3 akmal supergroup 638.5 K 2015-05-10 08:50 > > >>>>>> > > >>>> > > >> > > > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/839d3a6fc523435d8b44f63315fd11b8 > > >>>>>> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 08:50 > > >>>>>> > > >>>> > > >> > > > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/8c245e8661b140439e719f69a535d57f > > >>>>>> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 11:37 > > >>>>>> > > >>>> > > >> > > > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/23497a31d3e721fe9b63c58fbe0224d5 > > >>>>>> -rw-r--r-- 3 akmal supergroup 638.7 K 2015-05-10 11:37 > > >>>>>> > > >>>> > > >> > > > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/8c9af0357d164221ad46b336cd660b30 > > >>>>>> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 11:37 > > >>>>>> > > >>>> > > >> > > > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/8eb55b43c22d434954e2e0bfda656018 > > >>>>>> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 11:37 > > >>>>>> > > >>>> > > >> > > > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/b8b6210d9e6d4ec2344238c6e9c17ddf > > >>>>>> > > >>>>>> As I understood this files were copied to archive folder after > > >>>> compaction. > > >>>>>> The part I didn’t understand is, why the file with 638 K was also > > >>>> selected > > >>>>>> for compaction? > > >>>>>> Any ideas? > > >>>>>> Thank you. > > >>>>>> > > >>>>>> Kind regards, > > >>>>>> Akmal Abbasov > > >>>>>> > > >>>>>> > > >>>>>> > > >>>> > > >> > > >> > > > > >
