Just to clarify, Cassandra only uses UDP for gossip -- passing data between nodes is done over TCP.
-Jonathan On Tue, Oct 6, 2009 at 5:31 PM, Jonathan Mischo <[email protected]> wrote: > Back in the day, I was involved in performance testing JVMs, Solaris on > Intel, and SWS at SunSoft, and our lab actually ran not just our own > numbers, but competitors' numbers as well, for comparison. One thing we > discovered, when analyzing some performance issues we'd seen with Windows > clients, was that Windows' network stacks don't behave as you'd expect. > They very regularly transmitted packets out of sequence. > I'm not sure if this is still the case, as this was the late 90's, but we > discovered this when we were working on SPECweb numbers (we were a SPEC lab) > for SWS and, in looking at the JVM and other system settings to understand > why we were seeing unexpectedly large TCP buffers. When we started sniffing > packets directly from the Windows clients, we discovered the packets were > being emitted out of sequence, which was causing the server to require > larger per-connection buffers and was pushing TCP window size boundaries. > Now, Cassandra uses UDP primarily, but we never tested to find out whether > this was a TCP stack issue, IP stack issue, or Ethernet stack issue, so it > may be a similar case. Since reassembly still has to happen, if the packets > are being transmitted out of order, it makes sense that your write latency > would be significantly higher. > I'd be interested to see the results if you dig deeper into this. > -Jon > > On Oct 6, 2009, at 11:14 AM, Igor Katkov wrote: > > I think I finally found what. It's implementation of Java NIOon Windows (JVM > 1.6.0.16, 64b on Windows 2003) > The very same code, same network but CentOS linux gives almost 4x > performance. (in cassan...@linux -> cassan...@windows setup) > I don't have another linux box to test (cassan...@linux -> cassan...@linux) > performance, but expect it to be even better. > > A lesson learnt: don't use windows. > > P.S. > Here at Viigo we also learnt the hard way that async IO is also broken in > .NET (C#). Now I start to wonder if there is some fundamental flaw in async > IO on windows... > > On Mon, Oct 5, 2009 at 3:23 PM, Jonathan Ellis <[email protected]> wrote: >> >> On Mon, Oct 5, 2009 at 2:17 PM, Igor Katkov <[email protected]> wrote: >> > measured via JMX console i.e. does not include client-cassandra-client >> > latency >> > >> > 20 client threads 176975b value StorageProxy.WriteLatency ~660ms >> > 10 client threads 176975b value StorageProxy.WriteLatency ~350ms >> > 05 client threads 176975b value StorageProxy.WriteLatency ~156ms >> >> this is going up basically linearly with amount of (data x clients), >> so clearly something is getting saturated. > > >
