Jeff, We have both commitlog and data on a 4TB EBS with 10k IOPS.
Mike On Wed, Feb 10, 2016 at 5:28 PM, Jeff Jirsa <jeff.ji...@crowdstrike.com> wrote: > What disk size are you using? > > > > From: Mike Heffner > Reply-To: "user@cassandra.apache.org" > Date: Wednesday, February 10, 2016 at 2:24 PM > To: "user@cassandra.apache.org" > Cc: Peter Norton > Subject: Re: Debugging write timeouts on Cassandra 2.2.5 > > Paulo, > > Thanks for the suggestion, we ran some tests against CMS and saw the same > timeouts. On that note though, we are going to try doubling the instance > sizes and testing with double the heap (even though current usage is low). > > Mike > > On Wed, Feb 10, 2016 at 3:40 PM, Paulo Motta <pauloricard...@gmail.com> > wrote: > >> Are you using the same GC settings as the staging 2.0 cluster? If not, >> could you try using the default GC settings (CMS) and see if that changes >> anything? This is just a wild guess, but there were reports before of >> G1-caused instabilities with small heap sizes (< 16GB - see CASSANDRA-10403 >> for more context). Please ignore if you already tried reverting back to CMS. >> >> 2016-02-10 16:51 GMT-03:00 Mike Heffner <m...@librato.com>: >> >>> Hi all, >>> >>> We've recently embarked on a project to update our Cassandra >>> infrastructure running on EC2. We are long time users of 2.0.x and are >>> testing out a move to version 2.2.5 running on VPC with EBS. Our test setup >>> is a 3 node, RF=3 cluster supporting a small write load (mirror of our >>> staging load). >>> >>> We are writing at QUORUM and while p95's look good compared to our >>> staging 2.0.x cluster, we are seeing frequent write operations that time >>> out at the max write_request_timeout_in_ms (10 seconds). CPU across the >>> cluster is < 10% and EBS write load is < 100 IOPS. Cassandra is running >>> with the Oracle JDK 8u60 and we're using G1GC and any GC pauses are less >>> than 500ms. >>> >>> We run on c4.2xl instances with GP2 EBS attached storage for data and >>> commitlog directories. The nodes are using EC2 enhanced networking and have >>> the latest Intel network driver module. We are running on HVM instances >>> using Ubuntu 14.04.2. >>> >>> Our schema is 5 tables, all with COMPACT STORAGE. Each table is similar >>> to the definition here: >>> https://gist.github.com/mheffner/4d80f6b53ccaa24cc20a >>> >>> This is our cassandra.yaml: >>> https://gist.github.com/mheffner/fea80e6e939dd483f94f#file-cassandra-yaml >>> >>> Like I mentioned we use 8u60 with G1GC and have used many of the GC >>> settings in Al Tobey's tuning guide. This is our upstart config with JVM >>> and other CPU settings: >>> https://gist.github.com/mheffner/dc44613620b25c4fa46d >>> >>> We've used several of the sysctl settings from Al's guide as well: >>> https://gist.github.com/mheffner/ea40d58f58a517028152 >>> >>> Our client application is able to write using either Thrift batches >>> using Asytanax driver or CQL async INSERT's using the Datastax Java driver. >>> >>> For testing against Thrift (our legacy infra uses this) we write batches >>> of anywhere from 6 to 1500 rows at a time. Our p99 for batch execution is >>> around 45ms but our maximum (p100) sits less than 150ms except when it >>> periodically spikes to the full 10seconds. >>> >>> Testing the same write path using CQL writes instead demonstrates >>> similar behavior. Low p99s except for periodic full timeouts. We enabled >>> tracing for several operations but were unable to get a trace that >>> completed successfully -- Cassandra started logging many messages as: >>> >>> INFO [ScheduledTasks:1] - MessagingService.java:946 - _TRACE messages >>> were dropped in last 5000 ms: 52499 for internal timeout and 0 for cross >>> node timeout >>> >>> And all the traces contained rows with a "null" source_elapsed row: >>> https://gist.githubusercontent.com/mheffner/1d68a70449bd6688a010/raw/0327d7d3d94c3a93af02b64212e3b7e7d8f2911b/trace.out >>> >>> >>> We've exhausted as many configuration option permutations that we can >>> think of. This cluster does not appear to be under any significant load and >>> latencies seem to largely fall in two bands: low normal or max timeout. >>> This seems to imply that something is getting stuck and timing out at the >>> max write timeout. >>> >>> Any suggestions on what to look for? We had debug enabled for awhile but >>> we didn't see any msg that pointed to something obvious. Happy to provide >>> any more information that may help. >>> >>> We are pretty much at the point of sprinkling debug around the code to >>> track down what could be blocking. >>> >>> >>> Thanks, >>> >>> Mike >>> >>> -- >>> >>> Mike Heffner <m...@librato.com> >>> Librato, Inc. >>> >>> >> > > > -- > > Mike Heffner <m...@librato.com> > Librato, Inc. > > -- Mike Heffner <m...@librato.com> Librato, Inc.
