Re: Flush / Snapshot Triggering Full GCs, Leaving Ring

2011-04-10 Thread Janne Jalkanen
On Apr 7, 2011, at 23:43 , Jonathan Ellis wrote: The history is that, way back in the early days, we used to max it out the other way (MTT=128) but observed behavior is that objects that survive 1 new gen collection are very likely to survive forever. Just a quick note: my own tests seem to

Re: Flush / Snapshot Triggering Full GCs, Leaving Ring

2011-04-07 Thread ruslan usifov
2011/4/7 Jonathan Ellis jbel...@gmail.com Hypothesis: it's probably the flush causing the CMS, not the snapshot linking. Confirmation possibility #1: Add a logger.warn to CLibrary.createHardLinkWithExec -- with JNA enabled it shouldn't be called, but let's rule it out. Confirmation

Re: Flush / Snapshot Triggering Full GCs, Leaving Ring

2011-04-07 Thread Jonathan Ellis
No, 2252 is not suitable for backporting to 0.7. On Thu, Apr 7, 2011 at 7:33 AM, ruslan usifov ruslan.usi...@gmail.com wrote: 2011/4/7 Jonathan Ellis jbel...@gmail.com Hypothesis: it's probably the flush causing the CMS, not the snapshot linking. Confirmation possibility #1: Add a

Re: Flush / Snapshot Triggering Full GCs, Leaving Ring

2011-04-07 Thread Erik Onnen
I'll capture what I we're seeing here for anyone else who may look into this in more detail later. Our standard heap growth is ~300K in between collections with regular ParNew collections happening on average about every 4 seconds. All very healthy. The memtable flush (where we see almost all

Re: Flush / Snapshot Triggering Full GCs, Leaving Ring

2011-04-07 Thread Jonathan Ellis
On Thu, Apr 7, 2011 at 2:27 PM, Erik Onnen eon...@gmail.com wrote: 1) Does this seem like a sane amount of garbage (512MB) to generate when flushing a 64MB table to disk? Sort of -- that's just about exactly the amount of space you'd expect 64MB of serialized data to take, in memory. (Not very

Flush / Snapshot Triggering Full GCs, Leaving Ring

2011-04-06 Thread C. Scott Andreas
Hello, We're running a six-node 0.7.4 ring in EC2 on m1.xlarge instances with 4GB heap (15GB total memory, 4 cores, dataset fits in RAM, storage on ephemeral disk). We've noticed a brief flurry of query failures during the night corresponding with our backup schedule. More specifically, our

Re: Flush / Snapshot Triggering Full GCs, Leaving Ring

2011-04-06 Thread Jonathan Ellis
Hypothesis: it's probably the flush causing the CMS, not the snapshot linking. Confirmation possibility #1: Add a logger.warn to CLibrary.createHardLinkWithExec -- with JNA enabled it shouldn't be called, but let's rule it out. Confirmation possibility #2: Force some flushes w/o snapshot.