I triggered several time major_compaction between 22:30 and the time i wrote the original email 00:58 >From the log of one of the severs between those time no major compaction was running a major_compaction ended at 22:23 on the same server but this is because the number of file was high (after a load phase) and not manually triggered. On the system major_compaction are disabled.
>From the logs: 1. 22:23:42 --> 22:23:48 - single major compaction runs 2. 22:23:48 --> 23:44:27 - minor compaction runs 3. 23:44:27 --> 23:44:32 - single major compaction runs 5. 23:44:32 --> 01:13:49 - minor compaction are run 5. 01:13:49 the cycle of major compaction began to run (heavily working only from 01:40:). ------------------------------- *below a grep of compaction (ends, with major and minor to illustrate the above)* 2012-02-23 22:23:48,272 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 1 file(s) in PI of TC,00000000000012921999,1329399349811.8dcc952ff57e87c6d82840e84eb870a6. into 7400a662b5924d3baf3ac520b1eb7ad9, size=49.0m; total size for store is 49.0m 2012-02-23 22:25:38,477 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 3 file(s) in EVENT of TC,00000000000010451999,1329399349799.38739d7acb5f10198540bcfa0d4b5aef. into 77c43127e38846f78e7ada302d399732, size=540.8m; total size for store is 2.1g 2012-02-23 22:27:41,930 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 3 file(s) in EVENT of TC,00000000000003951999,1329399349764.f55cfda6067ad1425ba68d20d82c945b. into f9c31ac78b3d4feb848b1e8d78975e34, size=532.0m; total size for store is 2.1g 2012-02-23 22:29:43,260 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 3 file(s) in EVENT of TC,00000000000010971999,1329399349802.15720106584fe32f58287b6174970f31. into af2472f30a9f4c558323a3268f98183c, size=517.9m; total size for store is 2.6g 2012-02-23 22:30:19,476 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 3 file(s) in EVENT of TC,00000000000010061999,1329399349797.27a2f37d5d6ac48c860cca5251be5731. into 3ac255c439ab4c518f4a938d542c9220, size=335.4m; total size for store is 2.1g 2012-02-23 22:31:15,051 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 3 file(s) in EVENT of TC,00000000000008501999,1329399349789.fa0ac64d8ac9050f4acf2b0deb6e2dc7. into b8c733a27d02468a82f8a2c4d84db4ff, size=523.0m; total size for store is 2.1g 2012-02-23 22:32:33,430 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 3 file(s) in EVENT of TC,00000000000004991999,1329399349770.46a2a16d484faacfcda36bc52bb218ba. into 5344911081424de69299128412d41573, size=538.0m; total size for store is 2.2g 2012-02-23 22:33:28,882 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 3 file(s) in EVENT of TC,00000000000004081999,1329399349765.a5ca0a6a0a5561af5b6964e458615493. into d15c1e5b089d46beb2c4d01c796367a7, size=520.7m; total size for store is 2.1g 2012-02-23 22:34:46,714 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 3 file(s) in EVENT of TC,00000000000007591999,1329399349784.711b032969e1ae6cce8e16bb9b ... .. .. 2012-02-23 23:44:32,215 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 1 file(s) in PI of TC,00000000000000051999,1329399349734.e7ba4a35ee8524fbf4e6f9f9ba845206. into 2c701bd613314198889d8b3d79b82992, size=40.8m; total size for store is 40.8m 2012-02-23 23:45:00,838 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 4 file(s) in EVENT of TC,00000000000005251999,1329399349771.50df7d847b5e36be9982b47eb3234ad4. into 701d49ca010a41169ca495917b4ce3d4, size=1.3g; total size for store is 2.1g 2012-02-23 23:47:12,191 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 4 file(s) in EVENT of TC,00000000000002261999,1329399349751.ffb8f2f3d9d5a23aee280359793557ae. into d2b8336a36b1451d9a7d38c6bf3e79ee, size=1.2g; total size for store is 2.1g 2012-02-23 23:49:28,440 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 4 file(s) in EVENT of TC,00000000000011881999,1329399349806.81f2bb0de10974b617521b62b9b85f09. into 19208d50fe8543a682e3351c60801eb9, size=1.2g; total size for store is 2.1g 2012-02-23 23:52:12,523 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 4 file(s) in EVENT of TC,00000000000000051999,1329399349734.e7ba4a35ee8524fbf4e6f9f9ba845206. into 639ffb3d8c434363aac0d007b6f78ce9, size=1.5g; total size for store is 2.1g 2012-02-23 23:56:14,058 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 4 file(s) in EVENT of TC,00000000000005901999,1329399349775.48aa14a1feea4793ec68bfe076570257. into 3e2f293a33f14612b8fa1ad570799feb, size=1.6g; total size for store is 2.1g 2012-02-23 23:59:08,211 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 4 file(s) in EVENT of TC,00000000000010451999,1329399349799.38739d7acb5f10198540bcfa0d4b5aef. into dd7f92dd786d4e3ea9cb733cb3a6d2f8, size=1.6g; total size for store is 2.1g ... ... ... ... 2012-02-24 01:17:15,611 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 6 file(s) in EVENT of TC,00000000000012401999,1329399349808.03a3d34ac2c0c7d09619 8032a151120f. into eff7b5bda7bf4653a9f229c1b40e0e4f, size=1.3g; total size for store is 2.1g 2012-02-24 01:17:51,057 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 4 file(s) in EVENT of TC,00000000000000441999,1329399349737.02354040d2650db1d59de0e01b 0edd1d. into ac1d5e5f1ab746229bad223f56534054, size=329.8m; total size for store is 2.2g 2012-02-24 01:20:40,454 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 5 file(s) in EVENT of TC,00000000000012921999,1329399349811.8dcc952ff57e87c6d828 40e84eb870a6. into ae82b7832eed4bafb9e64ed45574831c, size=978.9m; total size for store is 2.5g 2012-02-24 01:21:13,635 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 4 file(s) in EVENT of TC,00000000000007981999,1329399349786.7e26a6440421112a0a15ed78ac 33b168. into 94aa6f0328834b0896a05cda6a6a9a0f, size=309.1m; total size for store is 2.1g 2012-02-24 01:22:50,578 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 4 file(s) in EVENT of TC,00000000000002391999,1329399349752.05a0d235efa8135618628e5124 dc9507. into 0ca4a3b8221c42f992649431e7182a6d, size=329.6m; total size for store is 2.2g 2012-02-24 01:25:03,055 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 5 file(s) in EVENT of TC,00000000000001221999,1329399349743.56d16361d17dce747fb0 23ab15ded16b. into 47e9f68efb8a475dbce7db2c08b46f9c, size=1.0g; total size for store is 2.5g 2012-02-24 01:25:38,876 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 4 file(s) in EVENT of TC,00000000000001091999,1329399349742.5907e62062b4edc83033b4fda0 fc105f. into d5148c6b4dbc4d3c8339f151b987e51a, size=326.4m; total size for store is 2.2g 2012-02-24 01:26:10,242 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 4 file(s) in EVENT of TC,00000000000005381999,1329399349772.15d2f2fb1ad338ab3b5e57077d 874601. into bacf547a2b4f4627b630a18b88ec8ec6, size=306.5m; total size for store is 2.1g 2012-02-24 01:27:47,690 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 4 file(s) in EVENT of TC,00000000000010581999,1329399349800.8144106ed55b1571a720e46569 af8ea6. into 827d75a8b3144ee3a7e53dae5fdecf3a, size=296.5m; total size for store is 2.1g 2012-02-24 01:28:23,907 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 4 file(s) in EVENT of TC,00000000000006681999,1329399349779.9e0dc98e734715e66702419dc2 1e23ed. into 4a93423a056645d7a6f869f66c76fd0e, size=341.4m; total size for store is 2.1g 2012-02-24 01:29:11,192 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 4 file(s) in EVENT of TC,00000000000012661999,1329399349810.76715703e8bf5f3f960564834f d9417d. into 3ef0ae62a13a4ed98cd772eeb1e55fd0, size=438.1m; total size for store is 2.1g 2012-02-24 01:29:49,930 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 4 file(s) in EVENT of TC,00000000000011101999,1329399349802.a63a43695fdcc0be243cfe3ea6 263981. into 2a542d20b2ed49b5964777c97f5f2a5b, size=355.2m; total size for store is 2.1g 2012-02-24 01:33:08,747 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 5 file(s) in EVENT of TC,00000000000008761999,1329399349791.7c0b5c9e5c98e9e39ae7 a3afcfaad4d9. into 93fb6ab4de7a4672aceed0e23cc9effc, size=1.2g; total size for store is 2.1g 2012-02-24 01:35:21,194 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 5 file(s) in EVENT of TC,00000000000006811999,1329399349780.6051d2a9230460194d41 f6249d087f6f. into 3adba15b5ae04bd1a18a18afeb7cf107, size=1.2g; total size for store is 2.1g 2012-02-24 01:37:08,338 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 4 file(s) in EVENT of TC,00000000000011621999,1329399349805.4d2bb3d00b0fa089163d6a0491dedadb. into 26ac704d826a495080e93f07b3c6e24d, size=395.6m; total size for store is 2.6g 2012-02-24 01:37:51,454 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 4 file(s) in EVENT of TC,00000000000012011999,1329399349807.99d15c956a3897276d0fdeaa33611417. into 28aac65938bf493187f84e7019ca126d, size=393.8m; total size for store is 2.1g 2012-02-24 01:38:33,930 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 4 file(s) in EVENT of TC,00000000000004861999,1329399349769.dce64ca59ca897a34aa6416de4f32dea. into 5dda42912eb64701b20639065a7612cb, size=393.6m; total size for store is 2.1g 2012-02-24 01:39:23,057 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 4 file(s) in EVENT of TC,00000000000010841999,1329399349801.c7f801818067718327858c10b0f3ded2. into 39ce298723bc4f66a5bfe1f0729143d4, size=468.0m; total size for store is 2.2g 2012-02-24 01:40:16,369 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 4 file(s) in EVENT of TC,00000000000001611999,1329399349746.eb6f8b45363b6d3f254d4b6d3dab3617. into 7821cb65510840afaff12c1e008b28c4, size=491.1m; total size for store is 2.1g 2012-02-24 01:42:18,657 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 4 file(s) in EVENT of TC,00000000000002651999,1329399349754.5c75673b41370540e3c758e937181651. into a7a12651ff3d414bb722945157686c11, size=505.1m; total size for store is 2.1g 2012-02-24 01:43:10,404 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 4 file(s) in EVENT of TC,00000000000004211999,1329399349766.041d7f3c86c4d5657c6eeded7584fad8. into 7b8d8075a4d04873a50e517431935e03, size=482.2m; total size for store is 2.1g 2012-02-24 01:44:03,388 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 4 file(s) in EVENT of TC,00000000000011751999,1329399349805.21ac7408f58fa1b022b6a38b2f197915. into f2906f916c16442a89161fdf9551bc0c, size=507.9m; total size for store is 2.1g 2012-02-24 01:44:59,108 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 4 file(s) in EVENT of TC,00000000000010061999,1329399349797.27a2f37d5d6ac48c860cca5251be5731. into 60defcf94fce43488fffc619c509a9d3, size=524.2m; total size for store is 2.1g 2012-02-24 01:46:17,683 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 4 file(s) in EVENT of TC,00000000000011491999,1329399349804.c613c26eff42730acd1f .. .. .. .. Mikael.S On Fri, Feb 24, 2012 at 2:05 AM, Jean-Daniel Cryans <[email protected]>wrote: > Can you show us some logs that exhibit this behavior? > > Thx, > > J-D > > On Thu, Feb 23, 2012 at 2:58 PM, Mikael Sitruk <[email protected]> > wrote: > > Hi > > > > Before filling a jira, I would like to check this with you. > > It seems that the major compaction behavior changed between 0.90.x and > 0.92 > > (not taking about all the nice stuff you added there...) > > > > Effectively when i trigger a major compaction (via hbase shell or ui) > when > > other compactions (minor) are running (because i had a lot of storefiles > > and a big compactionQueueSize), the major compaction is not executed. > While > > in 0.90.X major_compact command was executed immediately after any > current > > compaction. > > > > > > Mikael.S > -- Mikael.S
