Can you pastebin major compaction related log snippets ? See the following for example of such logs:
2015-05-09 10:57:58,961 INFO [PriorityRpcServer.handler=13,queue=1,port=16020] regionserver.RSRpcServices: Compacting IntegrationTestBigLinkedList,\x91\x11\x11\x11\x11\x11\x11\x08,1431193978741.700b34f5d2a3aa10804eff35906fd6d8. 2015-05-09 10:57:58,962 DEBUG [PriorityRpcServer.handler=13,queue=1,port=16020] regionserver.HStore: Skipping expired store file removal due to min version being 1 2015-05-09 10:57:58,962 DEBUG [PriorityRpcServer.handler=13,queue=1,port=16020] compactions.RatioBasedCompactionPolicy: Selecting compaction from 5 store files, 0 compacting, 5 eligible, 10 blocking 2015-05-09 10:57:58,963 DEBUG [PriorityRpcServer.handler=13,queue=1,port=16020] regionserver.HStore: 700b34f5d2a3aa10804eff35906fd6d8 - meta: Initiating major compaction (all files) Cheers On Tue, May 12, 2015 at 2:06 PM, Akmal Abbasov <[email protected]> wrote: > Hi Ted, > Thank you for reply. > I am running with the default settings. > > Sent from my iPhone > > > On 12 May 2015, at 22:02, Ted Yu <[email protected]> wrote: > > > > Can you show us compaction related parameters you use ? > > > > e.g. hbase.hregion.majorcompaction , > hbase.hregion.majorcompaction.jitter , > > etc > > > > On Tue, May 12, 2015 at 9:52 AM, Akmal Abbasov <[email protected] > > > > wrote: > > > >> HI, > >> I am using HBase 0.98.7. > >> I am using HBase snapshots to backup data. I create snapshot of tables > >> each our. > >> Each create snapshot process will cause the flush of the memstore, and > >> creation of hfiles. > >> When the number of hfiles will reach 3 the MINOR compaction process will > >> start for each CF. > >> Ok, I was expecting that the compaction will process only small hfiles, > >> and I won’t have problems with moving all data to archive folder > >> each time after compaction process ends. > >> But most of the times, the minor compaction is promoted to major(more > than > >> 100 in 24 hours without loads). > >> As far as I know, the only possibility for this is that all hfiles are > >> eligible for compaction. > >> But when I tested the archive folder for a CF I see the strange > situation > >> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 06:04 > >> > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/36dc06f4c34242daadc343d857a35734 > >> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 06:04 > >> > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/7e8b993f97b84f4594542144f15b0a1e > >> -rw-r--r-- 3 akmal supergroup 1.1 K 2015-05-10 06:04 > >> > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/b9afc64792ba4bf99a08f34033cc46ac > >> -rw-r--r-- 3 akmal supergroup 638.4 K 2015-05-10 06:04 > >> > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/dff846ae4fc24d418289a95322b35d46 > >> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 08:50 > >> > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/228eee22c32e458e8eb7f5d031f64b58 > >> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 08:50 > >> > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/529257432308466f971e41db49ecffdf > >> -rw-r--r-- 3 akmal supergroup 638.5 K 2015-05-10 08:50 > >> > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/839d3a6fc523435d8b44f63315fd11b8 > >> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 08:50 > >> > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/8c245e8661b140439e719f69a535d57f > >> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 11:37 > >> > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/23497a31d3e721fe9b63c58fbe0224d5 > >> -rw-r--r-- 3 akmal supergroup 638.7 K 2015-05-10 11:37 > >> > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/8c9af0357d164221ad46b336cd660b30 > >> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 11:37 > >> > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/8eb55b43c22d434954e2e0bfda656018 > >> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 11:37 > >> > /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/b8b6210d9e6d4ec2344238c6e9c17ddf > >> > >> As I understood this files were copied to archive folder after > compaction. > >> The part I didn’t understand is, why the file with 638 K was also > selected > >> for compaction? > >> Any ideas? > >> Thank you. > >> > >> Kind regards, > >> Akmal Abbasov > >> > >> > >> >
