Hi Romain, Thanks again. My response are inline.
kant On Tue, Feb 28, 2017 at 10:04 AM, Romain Hardouin <romainh...@yahoo.fr> wrote: > > we are currently using 3.0.9. should we use 3.8 or 3.10 > > No, don't use 3.X in production unless you really need a major feature. > I would advise to stick to 3.0.X (i.e. 3.0.11 now). > You can backport CASSANDRA-11966 easily but of course you have to deploy > from source as a prerequisite. > * By backporting you mean I should cherry pick CASSANDRA-11966 commit and compile from source?* > > > I haven't done any tuning yet. > > So it's a good news because maybe there is room for improvement > > > Can I change this on a running instance? If so, how? or does it require > a downtime? > > You can throttle compaction at runtime with "nodetool > setcompactionthroughput". Be sure to read all nodetool commmands, some of > them are really useful for a day to day tuning/management. > > If GC is fine, then check other things -> "[...] different pool sizes for > NTR, concurrent reads and writes, compaction executors, etc. Also check if > you can improve network latency (e.g. VF or ENA on AWS)." > > Regarding thread pools, some of them can be resized at runtime via JMX. > > > 5000 is the target. > > Right now you reached 1500. Is it per node or for the cluster? > We don't know your setup so it's hard to say it's doable. Can you provide > more details? VM, physical nodes, #nodes, etc. > Generally speaking LWT should be seldom used. AFAIK you won't achieve > 10,000 writes/s per node. > > Maybe someone on the list already made some tuning for heavy LWT workload? > * 1500 total cluster. * * I have a 8 node cassandra cluster. Each node is AWS m4.xlarge instance (so 4 vCPU, 16GB, 1Gbit network=125MB/s)* * I have 1 node (m4.xlarge) for my application which just inserts a bunch of data and each insert is an LWT I tested the network throughput of the node. I can get up 98 MB/s.* * Now, when I start my application. I see that Cassandra nodes Receive rate/ throughput is about 4MB/s (yes it is in Mega Bytes. I checked this by running sudo iftop -B). The Disk I/O is also same and the Cassandra process CPU usage is about 360% (the max is 400% since it is a 4 core machine). The application node transmission throughput is about 6MB/s. so even with 4MB/s receive throughput at Cassandra node the CPU is almost maxed out. I am not sure what this says about Cassandra? But, what I can tell is that Network is way underutilized and that 8 nodes are unnecessary so we plan to bring it down to 4 nodes except each node this time will have 8 cores. All said, I am still not sure how to scale up from 1500 writes/sec? * > > Best, > > Romain > >
