I was having trouble with snapshots failing while trying to repair that table (http://www.mail-archive.com/user@cassandra.apache.org/msg40686.html). I have a repair running on it now, and it seems to be going successfully this time. I am going to wait for that to finish, then try a manual nodetool compact. If that goes successfully, then would it be safe to chalk the lack of compaction on this table in the past up to 2.1.2 problems?
~ Paul Nickerson On Tue, Feb 10, 2015 at 3:34 PM, Chris Lohfink <clohfin...@gmail.com> wrote: > Your cluster is probably having issues with compactions (with STCS you > should never have this many). I would probably punt with > OpsCenter/rollups60. Turn the node off and move all of the sstables off to > a different directory for backup (or just rm if you really don't care about > 1 minute metrics), than turn the server back on. > > Once you get your cluster running again go back and investigate why > compactions stopped, my guess is you hit an exception in past that killed > your CompactionExecutor and things just built up slowly until you got to > this point. > > Chris > > On Tue, Feb 10, 2015 at 2:15 PM, Paul Nickerson <pgn...@gmail.com> wrote: > >> Thank you Rob. I tried a 12 GiB heap size, and still crashed out. There >> are 1,617,289 files under OpsCenter/rollups60. >> >> Once I downgraded Cassandra to 2.1.1 (apt-get install cassandra=2.1.1), I >> was able to start up Cassandra OK with the default heap size formula. >> >> Now my cluster is running multiple versions of Cassandra. I think I will >> downgrade the rest to 2.1.1. >> >> ~ Paul Nickerson >> >> On Tue, Feb 10, 2015 at 2:05 PM, Robert Coli <rc...@eventbrite.com> >> wrote: >> >>> On Tue, Feb 10, 2015 at 11:02 AM, Paul Nickerson <pgn...@gmail.com> >>> wrote: >>> >>>> I am getting an out of memory error why I try to start Cassandra on one >>>> of my nodes. Cassandra will run for a minute, and then exit without >>>> outputting any error in the log file. It is happening while SSTableReader >>>> is opening a couple hundred thousand things. >>>> >>> ... >>> >>>> Does anyone know how I might get Cassandra on this node running again? >>>> I'm not very familiar with correctly tuning Java memory parameters, and I'm >>>> not sure if that's the right solution in this case anyway. >>>> >>> >>> Try running 2.1.1, and/or increasing heap size beyond 8gb. >>> >>> Are there actually that many SSTables on disk? >>> >>> =Rob >>> >>> >> >> >
