large number of pending compactions, sstables steadily increasing

2016-08-17 Thread Ezra Stuetzel
I have one node in my cluster 2.2.7 (just upgraded from 2.2.6 hoping to fix issue) which seems to be stuck in a weird state -- with a large number of pending compactions and sstables. The node is compacting about 500gb/day, number of pending compactions is going up at about 50/day. It is at about 2

Re: large number of pending compactions, sstables steadily increasing

2016-08-17 Thread Ezra Stuetzel
11:49 AM, Jeff Jirsa wrote: > What compaction strategy? Looks like leveled – is that what you expect? > > > > Any exceptions in the logs? > > > > Are you throttling compaction? > > > > SSD or spinning disks? > > > > How many cores? > > >

large system hint partition

2016-09-16 Thread Ezra Stuetzel
What would be the likely causes of large system hint partitions? Normally large partition warnings are for user defined tables which they are writing large partitions to. In this case, it appears C* is writing large partitions to the system.hints table. Gossip is not backed up. version: C* 2.2.7

Re: large system hint partition

2016-09-17 Thread Ezra Stuetzel
gmail.com> wrote: > Hi Erza, > > Have you a dead node in your cluster? > Because the coordinator stores a hint about dead replicas in the local > system.hints when a node is dead or didn't respond to a write request. > > -- > Nicolas > > > > Le sam. 17 sept

Re: large system hint partition

2016-09-18 Thread Ezra Stuetzel
os Alonso | Software Engineer | @calonso <https://twitter.com/calonso> > > On 18 September 2016 at 04:35, Ezra Stuetzel > wrote: > >> Hey Nicolas, >> >> There are no dead nodes. 'nodetool status' and 'nodetool describecluster' >> both