The following is the snippets of nodetool compactions history. It shows many records relates to keyspace system.
Compaction History: id keyspace_name columnfamily_name compacted_at bytes_in bytes_out rows_merged 8e4f8830-b04f-11e5-a211-45b7aa88107c system sstable_activity 1451629144115 3342 915 {4:23} 96a6fcb0-b04b-11e5-a211-45b7aa88107c system hints 1451627440123 18970740 18970740 {1:1} 7c42c940-adac-11e5-8bd4-45b7aa88107c system hints 1451339203540 56969835 56782732 {2:3} 585b97a0-ad98-11e5-8bd4-45b7aa88107c system sstable_activity 1451330553370 3700 956 {4:24} aefc3f10-b1b2-11e5-a211-45b7aa88107c system sstable_activity 1451781670273 3201 906 {4:23} 1e76f1b0-b180-11e5-a211-45b7aa88107c system sstable_activity 1451759952971 3303 700 {4:23} e7b75b70-aec2-11e5-8bd4-45b7aa88107c system hints 1451458783911 57690316 57497847 {2:3} ad102280-af6d-11e5-b1dc-45b7aa88107c webtrn_study_log_formallySCORM_STU_COURSE 1451532129448 45671877 41137664 {1:11, 3:1, 4:8} 60906970-aec7-11e5-8bd4-45b7aa88107c system sstable_activity 1451460704647 3751 974 {4:25} 88aed310-ad91-11e5-8bd4-45b7aa88107c system hints 1451327627969 56984347 56765328 {2:3} 3ad14f00-af6d-11e5-b1dc-45b7aa88107c webtrn_study_log_formallySCORM_STU_COURSE 1451531937776 46696097 38827028 {1:8, 3:2, 4:9} 84df8fb0-b00f-11e5-a211-45b7aa88107c system hints 1451601640491 18970740 18970740 {1:1} 657482e0-ad33-11e5-8bd4-45b7aa88107c system sstable_activity 1451287196174 3701 931 {4:24} 9cc8af70-b24a-11e5-a211-45b7aa88107c system sstable_activity 1451846923239 3134 773 {4:23} dcbe5e30-afd0-11e5-a211-45b7aa88107c system sstable_activity 1451574729619 3357 790 {4:23} b285ced0-afa0-11e5-84e3-45b7aa88107c system hints 1451554042941 43310718 42137761 {1:1, 2:2} 119770e0-ad4e-11e5-8bd4-45b7aa88107c system hints 1451298651886 57397441 57190519 {2:3} f1bb37a0-b204-11e5-a211-45b7aa88107c system hints 1451817000986 17713746 17566349 {1:1} dcd0c720-b0b4-11e5-a211-45b7aa88107c system sstable_activity 1451672654993 3189 797 {4:23} cadb2320-ae94-11e5-8bd4-45b7aa88107c system sstable_activity 1451438978642 3642 959 {4:25} 321a84d0-af63-11e5-8bd4-45b7aa88107c system hints 1451527628189 57407115 57179555 {1:3} 5d760ca0-af20-11e5-8bd4-45b7aa88107c system hints 1451498924650 57405307 57234936 {1:3} 2e59a0b0-af6d-11e5-b1dc-45b7aa88107c system compactions_in_progress 1451531916859 464 190 {1:1, 2:1} 8723a290-adba-11e5-8bd4-45b7aa88107c system hints 1451345234745 57065795 56814910 {2:3} add0a8d0-af6c-11e5-b1dc-45b7aa88107c system compactions_in_progress 1451531701213 464 42 {2:2} 04c0d2f0-af6d-11e5-b1dc-45b7aa88107c system compactions_in_progress 1451531847071 446 0 {2:2} 315dd170-af98-11e5-b1dc-45b7aa88107c system schema_columns 1451550390279 129142 45841 {2:1, 3:1, 4:9} a17d8f80-b24b-11e5-a211-45b7aa88107c system hints 1451847360632 16746317 16600978 {1:1} e22e3b90-ad39-11e5-8bd4-45b7aa88107c system hints 1451289982409 57240038 56984895 {2:3} 38591c10-b22b-11e5-a211-45b7aa88107c system hints 1451833440337 17351684 17002773 {1:1} fc274930-af6c-11e5-b1dc-45b7aa88107c system hints 1451531832643 63540861 59716559 {2:3} aba057f0-b1b4-11e5-a211-45b7aa88107c system hints 1451782523631 18743329 18577220 {1:1} 81d5e980-af5c-11e5-8bd4-45b7aa88107c system hints 1451524755480 57603966 57407115 {2:3} a1801510-af6d-11e5-b1dc-45b7aa88107c webtrn_study_log_formallySCORM_STU_COURSE 1451532110049 46735894 40839394 {1:11, 4:9} f7e02540-b1bf-11e5-a211-45b7aa88107c system hints 1451787376020 18577220 18403648 {1:1} 3e9a72b0-af72-11e5-b1dc-45b7aa88107c system hints 1451534091611 60455761 43314155 {2:1, 3:2} 30c103c0-af6d-11e5-b1dc-45b7aa88107c system peers 1451531920892 1396 567 {2:3, 3:1} 65f47770-aed7-11e5-8bd4-45b7aa88107c system hints 1451467585639 57780382 57445632 {2:3} ecabc460-afa6-11e5-a211-45b7aa88107c system hints 1451556717478 29776619 29776619 {1:2} 5da4df30-afac-11e5-a211-45b7aa88107c system hints 1451559054499 20309429 18970740 {1:1, 2:1} f5ce2d30-afa6-11e5-a211-45b7aa88107c system hints 1451556732803 29776619 29776619 {1:2} 4ada6610-aeec-11e5-8bd4-45b7aa88107c system hints 1451476559601 57459788 57221330 {1:3} 74ffd7f0-ae32-11e5-8bd4-45b7aa88107c system hints 1451396743919 57210071 57053060 {2:3} 621f05e0-b0ce-11e5-a211-45b7aa88107c system sstable_activity 1451683616062 3060 719 {4:23} 5ec0e5b0-af2c-11e5-8bd4-45b7aa88107c system sstable_activity 1451504080779 3756 971 {4:25} 92707090-b09b-11e5-a211-45b7aa88107c system sstable_activity 1451661792793 3149 689 {4:23} c35b51f0-ae40-11e5-8bd4-45b7aa88107c system hints 1451402888335 57190905 57037697 {2:3} 2a813e10-ade4-11e5-8bd4-45b7aa88107c system sstable_activity 1451363118193 3772 960 {4:24} c99593a0-af6c-11e5-b1dc-45b7aa88107c system hints 1451531747802 57645358 57613272 {3:3} 50002a00-aee5-11e5-8bd4-45b7aa88107c system hints 1451473561760 57670199 57459788 {2:3} c2cf2570-ae6a-11e5-8bd4-45b7aa88107c system hints 1451420926279 57375776 57227330 {2:3} 1b5e7bc0-ad1a-11e5-8bd4-45b7aa88107c system sstable_activity 1451276334460 3606 861 {4:24} bf2f8d90-adc7-11e5-8bd4-45b7aa88107c system hints 1451350912233 56901257 56731829 {2:3} 55e055e0-b263-11e5-a211-45b7aa88107c system hints 1451857541694 16600978 16364535 {1:1} 1a60f830-afa7-11e5-a211-45b7aa88107c system hints 1451556794163 29776619 29776619 {1:2} 7b193990-adfd-11e5-8bd4-45b7aa88107c system sstable_activity 1451373990825 3679 939 {4:24} 6bf27720-aec9-11e5-8bd4-45b7aa88107c system hints 1451461582737 57571883 57372615 {2:3} d68f67b0-af13-11e5-8bd4-45b7aa88107c system hints 1451493544363 57518024 57335312 {2:3} 7946c2a0-b162-11e5-a211-45b7aa88107c system hints 1451747220426 18970740 18970740 {1:1} 1b686990-ad98-11e5-8bd4-45b7aa88107c system hints 1451330451113 56988235 56854912 {2:3} 30883420-afa7-11e5-a211-45b7aa88107c system hints 1451556831330 29776619 29776619 {1:2} e5651290-b129-11e5-a211-45b7aa88107c system hints 1451722920505 18970740 18970740 {1:1} feedabd0-ad61-11e5-8bd4-45b7aa88107c system hints 1451307210509 57251764 57133776 {2:3} d75a36b0-afa0-11e5-84e3-45b7aa88107c webtrn_study_log_formallySCORM_STU_COURSE 1451554104731 47893133 41298592 {1:10, 2:1, 4:9} b2226820-b05b-11e5-a211-45b7aa88107c system hints 1451634358178 18970740 18970740 {1:1} 7c7e0d80-af6d-11e5-b1dc-45b7aa88107c webtrn_study_log_formallySCORM_STU_COURSE 1451532047960 47584630 40254518 {1:10, 2:1, 4:9} 7d39a5d0-ad7f-11e5-8bd4-45b7aa88107c system sstable_activity 1451319877805 3637 882 {4:24} c05d07c0-b1fe-11e5-a211-45b7aa88107c system sstable_activity 1451814341180 3329 852 {4:23} a227fdb0-add4-11e5-8bd4-45b7aa88107c system hints 1451356446987 57070333 56869637 {2:3} 26a79a60-aeb5-11e5-8bd4-45b7aa88107c system hints 1451452876550 57637242 57408618 {2:3} def14230-af6c-11e5-b1dc-45b7aa88107c system hints 1451531783635 57613272 57622817 {1:3} ce8c20b0-afa6-11e5-a211-45b7aa88107c system local 1451556666939 1167 696 {4:1} 75ace380-b1e5-11e5-a211-45b7aa88107c system sstable_activity 1451803478456 3171 820 {4:23} 88d05020-af86-11e5-b1dc-45b7aa88107c system hints 1451542806050 43921794 43937401 {2:3} 0a59ef20-b1f8-11e5-a211-45b7aa88107c system hints 1451811458834 17972574 17713746 {1:1} 7ace0730-afa7-11e5-a211-45b7aa88107c system hints 1451556955939 29776619 29776619 {1:2} 9e1ee8b0-af6d-11e5-b1dc-45b7aa88107c webtrn_study_log_formallySCORM_STU_COURSE 1451532104379 5935114 2171959 {4:9} 0580cea0-ae5c-11e5-8bd4-45b7aa88107c system hints 1451414595722 57388662 57179650 {2:3} 378110c0-afa8-11e5-a211-45b7aa88107c system hints 1451557272524 28568485 24640663 {1:1, 2:1} ef231ab0-af5e-11e5-8bd4-45b7aa88107c system sstable_activity 1451525797851 3739 973 {4:25} 301fc6c0-ae25-11e5-8bd4-45b7aa88107c system hints 1451391044908 56996239 56821882 {2:3} 5ceccc10-afa7-11e5-a211-45b7aa88107c system hints 1451556905809 29776619 29776619 {1:2} 44713fe0-afa7-11e5-a211-45b7aa88107c system hints 1451556864734 29776619 29776619 {1:2} 0b0786b0-afa7-11e5-a211-45b7aa88107c system hints 1451556768411 29776619 29776619 {1:2} 5a667200-ae78-11e5-8bd4-45b7aa88107c system hints 1451426764064 57497152 57334697 {2:3} d4306ba0-af6c-11e5-b1dc-45b7aa88107c webtrn_study_log_formallySCORM_STU_COURSE 1451531765594 43985647 32914710 {1:7, 3:1, 4:9} 7596f1a0-ad9f-11e5-8bd4-45b7aa88107c system hints 1451333608890 56854912 56608995 {1:3} e31fc540-adee-11e5-8bd4-45b7aa88107c system hints 1451367722900 57057877 56867011 {2:3} 52458b10-b1e3-11e5-a211-45b7aa88107c system hints 1451802560065 18087954 17972574 {1:1} 278bf030-af13-11e5-8bd4-45b7aa88107c system sstable_activity 1451493250739 3720 922 {4:25} 76443290-af55-11e5-8bd4-45b7aa88107c system hints 1451521729593 57500226 57294613 {2:3} 8065b450-af33-11e5-8bd4-45b7aa88107c system hints 1451507143701 57557563 57359219 {2:3} ae29e180-b0ce-11e5-a211-45b7aa88107c system hints 1451683743640 18970740 18970740 {1:1} d942a360-af6c-11e5-b1dc-45b7aa88107c webtrn_study_log_formallySCORM_STU_COURSE 1451531774102 8615536 4447310 {1:1, 3:1, 4:9} 6b039a10-b289-11e5-a211-45b7aa88107c system hints 1451873898033 15967755 15796743 {1:1} 3afa2ba0-b14d-11e5-a211-45b7aa88107c system sstable_activity 1451738096474 3196 778 {4:23} 3d6e34a0-afa7-11e5-a211-45b7aa88107c system hints 1451556852970 29776619 29776619 {1:2} b27e7970-af6c-11e5-b1dc-45b7aa88107c system local 1451531709063 1215 668 {4:1} 78715730-ae62-11e5-8bd4-45b7aa88107c system sstable_activity 1451417365539 3749 950 {4:25} 46bd9b90-aed0-11e5-8bd4-45b7aa88107c system hints 1451464526793 57589209 57452596 {2:3} d9030250-af6c-11e5-b1dc-45b7aa88107c webtrn_study_log_formallySCORM_STU_SCO 1451531773685 61554578 59278964 {1:76826, 2:11126, 3:604, 4:9} c70ceb50-afcc-11e5-a211-45b7aa88107c system hints 1451572975237 18970740 18970740 {1:1} 69f8a090-b106-11e5-a211-45b7aa88107c system hints 1451707681048 18970740 18970740 {1:1} 7dd087b0-af79-11e5-b1dc-45b7aa88107c system hints 1451537204139 43314155 43324175 {1:3} 94404a50-b166-11e5-a211-45b7aa88107c system sstable_activity 1451748983669 3364 926 {4:23} d99598d0-ad10-11e5-8bd4-45b7aa88107c system hints 1451272358621 57162919 56981333 {2:3} 7fe03f20-b23d-11e5-a211-45b7aa88107c system hints 1451841291282 17002773 16746317 {1:1} e664d240-b1cc-11e5-a211-45b7aa88107c system hints 1451792930148 18403648 18252171 {1:1} 04178e40-afa7-11e5-a211-45b7aa88107c system hints 1451556756772 29776619 29776619 {1:2} 1a414650-ada6-11e5-8bd4-45b7aa88107c system hints 1451336462133 56826857 56710683 {2:3} ad9ef5f0-b026-11e5-a211-45b7aa88107c system hints 1451611587279 18970740 18970740 {1:1} 60a4e320-ad17-11e5-8bd4-45b7aa88107c system hints 1451275162194 57236552 57076664 {2:3} 917e6370-aeae-11e5-8bd4-45b7aa88107c system hints 1451450049319 57539934 57384675 {2:3} eb1440d0-af6c-11e5-b1dc-45b7aa88107c webtrn_study_log_formallySCORM_STU_COURSE 1451531803997 43867391 36299816 {1:8, 4:10} a20bfc70-aee0-11e5-8bd4-45b7aa88107c system sstable_activity 1451471551927 3833 945 {4:25} 47cbe960-aef3-11e5-8bd4-45b7aa88107c system hints 1451479560950 57443969 57255886 {2:3} e2a84f00-af25-11e5-8bd4-45b7aa88107c system hints 1451501295600 57449985 57298217 {2:3} af1628a0-ade1-11e5-8bd4-45b7aa88107c system hints 1451362052138 57001606 56839416 {2:3} b0fea4a0-ae16-11e5-8bd4-45b7aa88107c system sstable_activity 1451384818666 3645 872 {3:1, 4:24} 15473980-aeae-11e5-8bd4-45b7aa88107c system sstable_activity 1451449840920 3789 975 {4:25} e5bdc7c0-afa6-11e5-a211-45b7aa88107c system hints 1451556705852 29776619 29776619 {1:2} d7808360-afa0-11e5-84e3-45b7aa88107c system compactions_in_progress 1451554104982 461 42 {2:2} 36a22720-ae63-11e5-8bd4-45b7aa88107c system hints 1451417684626 57282895 57148294 {2:3} 99222cc0-ad54-11e5-8bd4-45b7aa88107c system hints 1451301456268 57258179 57086571 {2:3} a3480190-adb1-11e5-8bd4-45b7aa88107c system sstable_activity 1451341416489 3683 925 {4:24} 1198f0c0-af6d-11e5-b1dc-45b7aa88107c webtrn_study_log_formallySCORM_STU_COURSE 1451531868620 8937482 3569558 {2:1, 3:1, 4:8} 422d8470-b036-11e5-a211-45b7aa88107c system sstable_activity 1451618278967 3243 825 {4:23} 12aee840-b218-11e5-a211-45b7aa88107c system sstable_activity 1451825216708 3176 819 {4:23} 85af2f30-adb3-11e5-8bd4-45b7aa88107c system hints 1451342225827 57028608 56811880 {2:3} 8b98e730-afa0-11e5-84e3-45b7aa88107c system hints 1451553977634 88625803 44359037 {4:2, 5:1} 0c2b9aa0-ae02-11e5-8bd4-45b7aa88107c system hints 1451375952202 56891360 56548629 {2:3} ec9afff0-af00-11e5-8bd4-45b7aa88107c system hints 1451485420911 57594351 57326414 {2:3} 98a23cb0-b199-11e5-a211-45b7aa88107c system hints 1451770895355 18970740 18906200 {1:1} 8f90dde0-afb7-11e5-a211-45b7aa88107c system sstable_activity 1451563862718 3508 827 {1:2, 2:8, 3:2, 4:19} 822b7da0-b14b-11e5-a211-45b7aa88107c system hints 1451737356922 18970740 18970740 {1:1} f5dc9cb0-ae54-11e5-8bd4-45b7aa88107c system hints 1451411563003 57341080 57162518 {2:3} 83efd290-af8d-11e5-b1dc-45b7aa88107c system hints 1451545804345 43996474 44000895 {2:3} ba28f830-af80-11e5-b1dc-45b7aa88107c system hints 1451540311859 43744401 43725750 {1:1, 2:2} decb8560-afa6-11e5-a211-45b7aa88107c system hints 1451556694198 29776619 29776619 {1:2} 8fb80860-ad5b-11e5-8bd4-45b7aa88107c system hints 1451304446950 57324867 56998191 {2:3} 969514e0-afa7-11e5-a211-45b7aa88107c system hints 1451557002542 29776619 29776619 {1:2} bf7d00d0-afb5-11e5-a211-45b7aa88107c system hints 1451563084125 18970740 18970740 {1:1} 113e2660-af48-11e5-8bd4-45b7aa88107c system hints 1451515976646 57413867 57219902 {2:3} 32ee9f00-adf5-11e5-8bd4-45b7aa88107c system hints 1451370433776 57034296 56618944 {2:3} 3c583550-afea-11e5-a211-45b7aa88107c system sstable_activity 1451585627429 3073 720 {4:23} 58437cd0-af3a-11e5-8bd4-45b7aa88107c system hints 1451510082845 57369449 57188682 {2:3} 871b8c30-b0ef-11e5-a211-45b7aa88107c system hints 1451697851507 18970740 18970740 {1:1} 00522050-ae7f-11e5-8bd4-45b7aa88107c system hints 1451429619413 57450075 57258800 {2:3} 14e3fd10-af6d-11e5-b1dc-45b7aa88107c system compactions_in_progress 1451531874145 472 42 {2:2} b3e23af0-b0e7-11e5-a211-45b7aa88107c system sstable_activity 1451694490655 3013 726 {4:23} aa5986d0-ad24-11e5-8bd4-45b7aa88107c system hints 1451280869309 57181046 56983904 {2:3} 29697550-aebc-11e5-8bd4-45b7aa88107c system hints 1451455887653 57617579 57451553 {2:3} 8201d200-ae8c-11e5-8bd4-45b7aa88107c system hints 1451435420448 57558970 57354959 {2:3} 45bc2770-addb-11e5-8bd4-45b7aa88107c system hints 1451359298407 56977263 56817508 {2:3} ebd29a70-affe-11e5-a211-45b7aa88107c system hints 1451594511767 18970740 18970740 {1:1} 9530ef00-ad4c-11e5-8bd4-45b7aa88107c system sstable_activity 1451298013680 3580 885 {4:24} caa48640-afa6-11e5-a211-45b7aa88107c system local 1451556660388 1013 677 {4:1} acee31c0-aea0-11e5-8bd4-45b7aa88107c system hints 1451444082396 57659639 57460800 {2:3} aa2ad760-ae71-11e5-8bd4-45b7aa88107c system hints 1451423891414 57468574 57242109 {2:3} 9e29cbd0-b036-11e5-a211-45b7aa88107c system hints 1451618433293 18970740 18970740 {1:1} df7b80d0-af6c-11e5-b1dc-45b7aa88107c system compactions_in_progress 1451531784541 542 172 {1:2, 2:2} 48b11430-ad7d-11e5-8bd4-45b7aa88107c system hints 1451318930675 56911864 56738503 {2:3} 6afb39b0-b199-11e5-a211-45b7aa88107c system sstable_activity 1451770818763 3039 788 {4:23} f77313f0-af40-11e5-8bd4-45b7aa88107c system hints 1451512926895 57410690 57197897 {2:3} 38c86a20-adc1-11e5-8bd4-45b7aa88107c system hints 1451348109762 56814910 56689825 {1:3} c2406070-af0e-11e5-8bd4-45b7aa88107c system hints 1451491362807 57391505 57316604 {2:3} 610527f0-b26f-11e5-a211-45b7aa88107c system hints 1451862714351 16364535 16082982 {1:1} 23d18f10-ae49-11e5-8bd4-45b7aa88107c system sstable_activity 1451406486145 3587 941 {4:25} 76303090-ad40-11e5-8bd4-45b7aa88107c system hints 1451292807705 57224902 57033897 {2:3} 787be5c0-ad76-11e5-8bd4-45b7aa88107c system hints 1451316004380 56853811 56717047 {2:3} f446f990-aef9-11e5-8bd4-45b7aa88107c system hints 1451482427305 57466666 57373339 {2:3} 3ba17b30-ad47-11e5-8bd4-45b7aa88107c system hints 1451295715939 57320671 57178557 {2:3} 85f76e20-afa8-11e5-a211-45b7aa88107c system hints 1451557404162 24767833 20275337 {1:1, 2:1} 464b1550-af6d-11e5-b1dc-45b7aa88107c system compactions_in_progress 1451531957029 464 42 {2:2} d39fbc30-af9f-11e5-b1dc-45b7aa88107c system sstable_activity 1451553668979 3895 835 {4:26} acf90bb0-afa7-11e5-a211-45b7aa88107c system hints 1451557040107 29214113 28457489 {1:1, 2:1} d09de8e0-ad65-11e5-8bd4-45b7aa88107c system sstable_activity 1451308850798 3685 899 {4:24} 6e371d30-b27d-11e5-a211-45b7aa88107c system hints 1451868749443 16082982 15967755 {1:1} fca1eea0-ad83-11e5-8bd4-45b7aa88107c system hints 1451321809546 56925900 56697130 {2:3} 31372130-b0f9-11e5-a211-45b7aa88107c system hints 1451702002371 18970740 18970740 {1:1} a0dd4c90-af45-11e5-8bd4-45b7aa88107c system sstable_activity 1451514929113 3785 882 {4:25} a180b150-af6d-11e5-b1dc-45b7aa88107c system compactions_in_progress 1451532110053 446 0 {2:2} 46239020-af6d-11e5-b1dc-45b7aa88107c webtrn_study_log_formallySCORM_STU_COURSE 1451531956770 45648060 39245044 {1:8, 2:1, 3:1, 4:9} 76cf6680-afa0-11e5-84e3-45b7aa88107c system local 1451553942760 1265 674 {4:1} e4c92910-b11b-11e5-a211-45b7aa88107c system hints 1451716906529 18970740 18970740 {1:1} 16e97320-b264-11e5-a211-45b7aa88107c system sstable_activity 1451857865554 3114 726 {4:23} fddd18f0-b100-11e5-a211-45b7aa88107c system sstable_activity 1451705352191 2959 716 {4:23} ea955a00-b173-11e5-a211-45b7aa88107c system hints 1451754711968 18970740 18970740 {1:1} 52156430-adce-11e5-8bd4-45b7aa88107c system hints 1451353735667 56989435 56824386 {2:3} b9d73cf0-ae47-11e5-8bd4-45b7aa88107c system hints 1451405878847 57223358 57041982 {2:3} 2ab91b60-afaf-11e5-a211-45b7aa88107c system schema_triggers 1451560257558 134 60 {1:1, 3:1} 9712f8c0-ae39-11e5-8bd4-45b7aa88107c system hints 1451399807564 57261044 57081560 {2:3} 86ec5560-adfb-11e5-8bd4-45b7aa88107c system hints 1451373151670 56802348 56679018 {2:3} 4b9ffb20-b27d-11e5-a211-45b7aa88107c system sstable_activity 1451868691410 3037 791 {4:23} e0ed0e70-aef9-11e5-8bd4-45b7aa88107c system sstable_activity 1451482394839 3749 941 {4:25} 28e78950-afa7-11e5-a211-45b7aa88107c system hints 1451556818533 29776619 29776619 {1:2} 47589f00-b231-11e5-a211-45b7aa88107c system sstable_activity 1451836042480 3212 785 {4:23} 499c5d30-afaf-11e5-a211-45b7aa88107c system peers 1451560309379 1810 556 {3:4} 2d335080-af6a-11e5-8bd4-45b7aa88107c system hints 1451530626440 57406002 57261820 {2:3} d46ed430-af6c-11e5-b1dc-45b7aa88107c system compactions_in_progress 1451531766003 608 268 {1:2, 2:1} 31cbd580-af98-11e5-b1dc-45b7aa88107c system local 1451550391000 905 741 {4:1} 95dec160-afa0-11e5-84e3-45b7aa88107c system hints 1451553994870 44359037 44359037 {1:3} 4726d640-afe4-11e5-a211-45b7aa88107c system hints 1451583068580 18970740 18970740 {1:1} 9ed33380-ae7b-11e5-8bd4-45b7aa88107c system sstable_activity 1451428167352 3638 906 {4:25} 50fa9650-b1a8-11e5-a211-45b7aa88107c system hints 1451777217589 18906200 18743329 {1:1} b66e4420-b183-11e5-a211-45b7aa88107c system hints 1451761496418 18970740 18970740 {1:1} 5a003890-ade8-11e5-8bd4-45b7aa88107c system hints 1451364915865 57066041 56952696 {2:3} 451f4710-ad1e-11e5-8bd4-45b7aa88107c system hints 1451278122497 57133634 56933373 {2:3} b1e973b0-ad6f-11e5-8bd4-45b7aa88107c system hints 1451313094251 56935662 56616319 {1:3} 0ce9d500-ae17-11e5-8bd4-45b7aa88107c system hints 1451384972880 56788738 56649325 {2:3} 0fc24530-ae1e-11e5-8bd4-45b7aa88107c system hints 1451387984131 56905778 56758301 {2:3} 31d54e10-b0da-11e5-a211-45b7aa88107c system hints 1451688689009 18970740 18970740 {1:1} cb322de0-b133-11e5-a211-45b7aa88107c system sstable_activity 1451727171518 3177 862 {4:23} 4f987470-af1a-11e5-8bd4-45b7aa88107c system hints 1451496324407 57567325 57405307 {2:3} bbad5970-af2c-11e5-8bd4-45b7aa88107c system hints 1451504236679 57519774 57367465 {2:3} 5e31ef40-aff4-11e5-a211-45b7aa88107c system hints 1451589979188 18970740 On Mon, Jan 4, 2016 at 9:54 AM, Shuo Chen <chenatu2...@gmail.com> wrote: > The attachment is the current system.log and config of the cassandra > cluster. Over 8000 full gc happens in 3 days with 121 young gc. > > There are client operation in these days. Besides most columnfamily in the > cluster are supercolumnfamily created by cassandra-cli. Most rows have > average 30 sub-rows and each sub-row has 20 columns. > > You said that there are many compaction tasks. So how to check the source > of this compactions? Thank you! > > On Fri, Jan 1, 2016 at 10:09 AM, Graham Sanderson <gra...@vast.com> wrote: > >> If you are lucky that might mask the real issue, but I doubt it… that is >> an insane number of compaction tasks and indicative of another problem. I >> would check release notes of 2.0.6+, if I recall that was not a stable >> version and may have had leaks. >> >> Aside from that, just FYI, if you use native_objects for memtables you >> can run CMS fine even up to largish (20-30G) heap sizes without long GC >> pauses (at least over months) . That said look for promotion failures of >> 131074 DWORDS. Not sure what your GC logging options are, but what you >> posted doesn’t say why a full GC happened which would be totally tunable, >> but as I say I don’t think GC is actually your problem >> >> >> On Dec 31, 2015, at 10:16 AM, Ipremyadav <ipremya...@gmail.com> wrote: >> >> Simplest option is to use java 8 with G1 gc. >> >> On 31 Dec 2015, at 10:23 a.m., Shuo Chen <chenatu2...@gmail.com> wrote: >> >> I have a cassandra 2.0.6 cluster with four nodes as backup database. The >> only operation is posting data into db. Recently, the full gc of the nodes >> increases apparently and blocks cluster operation. >> >> The load of each node is 10G. The heap is 8G each with default jvm memory >> settings. The cpu is 24 cores. The settings of cassandra.yaml are almost >> default. >> >> For debug, all the clients are disconnected, however the gc is still high. >> >> The GC is too often that it appears in every minute and blocks for nealy >> 30s. >> >> How to debug this situation >> >> The following is the snippets of gc log: >> >> before full gc histogram >> >> >> Total Free Space: 0 >> Max Chunk Size: 0 >> Number of Blocks: 0 >> Tree Height: 0 >> 1620.992: [GC 7377342K(8178944K), 0.1380260 secs] >> Before GC: >> Statistics for BinaryTreeDictionary: >> ------------------------------------ >> Total Free Space: 0 >> Max Chunk Size: 0 >> Number of Blocks: 0 >> Tree Height: 0 >> Before GC: >> Statistics for BinaryTreeDictionary: >> ------------------------------------ >> Total Free Space: 0 >> Max Chunk Size: 0 >> Number of Blocks: 0 >> Tree Height: 0 >> ---------------------------------------------- >> 1: 61030245 1952967840 java.util.concurrent.FutureTask >> 2: 61031398 1464753552 >> java.util.concurrent.Executors$RunnableAdapter >> 3: 61030312 1464727488 >> java.util.concurrent.LinkedBlockingQueue$Node >> 4: 61030244 1464725856 >> >> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask >> 5: 214181 13386992 [B >> 6: 202890 9738720 java.nio.HeapByteBuffer >> 7: 37622 5994808 [C >> 8: 42163 5722096 <constMethodKlass> >> 9: 42163 5407152 <methodKlass> >> 10: 4170 4648240 <constantPoolKlass> >> 11: 100060 4002400 >> org.apache.cassandra.io.sstable.IndexHelper$IndexInfo >> 12: 4170 2860816 <instanceKlassKlass> >> 13: 3664 2702720 <constantPoolCacheKlass> >> 14: 4817 2701576 [J >> 15: 2575 1056600 <methodDataKlass> >> 16: 37431 898344 java.lang.String >> 17: 2585 627352 [Ljava.lang.Object; >> 18: 17728 567296 >> java.util.concurrent.ConcurrentHashMap$HashEntry >> 19: 22111 530664 javax.management.ObjectName$Property >> 20: 7595 463824 [S >> 21: 4534 433560 java.lang.Class >> 22: 11581 370592 java.util.HashMap$Entry >> >> ... >> Total 289942930 8399196504 >> 1658.022: [Full GC 8178943K->6241829K(8178944K), 27.8562520 secs] >> CMS: Large block 0x00000007f7da9588 >> >> After GC: >> Statistics for BinaryTreeDictionary: >> ------------------------------------ >> Total Free Space: 6335823 >> Max Chunk Size: 6335823 >> Number of Blocks: 1 >> Av. Block Size: 6335823 >> Tree Height: 1 >> After GC: >> Statistics for BinaryTreeDictionary: >> ------------------------------------ >> Total Free Space: 0 >> Max Chunk Size: 0 >> Number of Blocks: 0 >> Tree Height: 0 >> >> >> It seems related to objects of Futuretask. >> >> -- >> *陈硕* *Shuo Chen* >> chenatu2...@gmail.com >> chens...@whaty.com >> >> >> > > > -- > *陈硕* *Shuo Chen* > chenatu2...@gmail.com > chens...@whaty.com > -- *陈硕* *Shuo Chen* chenatu2...@gmail.com chens...@whaty.com