> With the large new-gen, you were actually seeing fallbacks to full GC?
> You weren't just still experiencing problems because at 10 gig, the new-gen
> will be so slow to compact to effectively be similar to a full gc in terms of
> affecting latency?
Yes, we were seeing fallbacks to full GC with
Sorry about the delay,
> I do believe there is a fundamental issue with compactions allocating too
> much memory and incurring too many garbage collections (at least with 0.6.12).
[snip a lot of good info]
You certainly seem to have a real issue, though I don't get the feel
it's the same as the
> Sent: Thursday, March 10, 2011 10:36 AM
> To: ruslan usifov
> Cc: user@cassandra.apache.org
> Subject: Re: Nodes frozen in GC
>
> I think it would be very useful to get to the bottom of this but without
> further details (like the asked for GC logs) I'm not sure what to do/s
I think it would be very useful to get to the bottom of this but
without further details (like the asked for GC logs) I'm not sure what
to do/suggest.
It's clear that a single CF with a 64 MB memtable flush threshold and
without key cache and row cache and some bulk insertion, should not be
causin
2011/3/8 Peter Schuller
> > $client->batch_mutate($mutations,
> > cassandra_ConsistencyLevel::QUORUM);
>
> Btw, what are the mutations? Are you doing something like inserting
> both very small values and very large ones?
>
> I have big xml file (5 GB) (mysql dump in xml format) an
2011/3/8 Paul Pak
> Hi Ruslan,
>
> Is it possible for you to tell us the details on what you have done which
> measurably helped your situation, so we can start a "best practices" doc on
> growing cassandra systems?
>
> So far, I see that under load, cassandra is rarely "ready" to take heavy
> l
Hi Ruslan,
Is it possible for you to tell us the details on what you have done
which measurably helped your situation, so we can start a "best
practices" doc on growing cassandra systems?
So far, I see that under load, cassandra is rarely "ready" to take heavy
load in it's default configuration
> Also, why is there so much garbage collection to begin with? Memcache
> uses a slab allocator to reuse blocks to prevent allocation/deallocation
> of blocks from consuming all the cpu time. Are there any plans to reuse
> blocks so the garbage collector doesn't have to work so hard?
And to addr
> $client->batch_mutate($mutations,
> cassandra_ConsistencyLevel::QUORUM);
Btw, what are the mutations? Are you doing something like inserting
both very small values and very large ones?
In any case: My main reason to butt back into this thread is that
under normal circumstances y
> Add:
>
> JVM_OPTS="$JVM_OPTS -XX:+PrintGC"
> JVM_OPTS="$JVM_OPTS -XX:+PrintGCDetails"
> JVM_OPTS="$JVM_OPTS -XX:+PrintGCTimeStamps"
>
> And you will see significantly more detail in the GC log.
Maybe you want to add -XX:+PrintGCApplicationConcurrentTime while you're at it.
But the key is to se
> JVM_OPTS="$JVM_OPTS -XX:+PrintGCApplicationStoppedTime"
> JVM_OPTS="$JVM_OPTS -Xloggc:/var/log/cassandra/gc.log"
Add:
JVM_OPTS="$JVM_OPTS -XX:+PrintGC"
JVM_OPTS="$JVM_OPTS -XX:+PrintGCDetails"
JVM_OPTS="$JVM_OPTS -XX:+PrintGCTimeStamps"
And you will see significantly more detail in the GC log.
2011/3/8 Peter Schuller
>
> (1) I cannot stress this one enough: Run with -XX:+PrintGC
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps and collect the output.
> (2) Attach to your process with jconsole or some similar tool.
> (3) Observe the behavior of the heap over time. Preferably post
> screensh
Also:
* What is the frequency of the pauses? Are we talking every few
seconds, minutes, hours, days
* If you say decrease the load down to 25%. Are you seeing the same
effect but at 1/4th the frequency, or does it remain unchanged, or
does the problem go away completely?
--
/ Peter Schuller
> (1) I cannot stress this one enough: Run with -XX:+PrintGC
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps and collect the output.
Actually, I wonder if it's worth someone getting this enabled by
default, with the obvious problems associated with getting the log
output placed appropriately and rota
> So, are you saying this is normal and expected from Cassandra? So,
> under load, we can expect java garbage collection to stop the Cassandra
> process on that server from time to time, essentially taking out the
> node for short periods of time while it does garbage collection?
This thread is g
2011/3/8 Paul Pak
> Hi Ruslan,
>
> It looks like Jonathan and Stu have already been working to reduce garbage
> collection on v.8 The ticket is at
> https://issues.apache.org/jira/browse/CASSANDRA-2252
>
> Jonathan, is there any way to apply the patch to .73 and have ruslan test
> it to see if
If RF=2 and CL= QUORUM, you're getting no benefit from replication. When a
node is in GC it stops everything. Set RF=3, so when one node is busy the
cluster will still work.
On Tue, Mar 8, 2011 at 11:46 AM, ruslan usifov wrote:
>
>
> 2011/3/8 Chris Goffinet
>
>> How large are your SSTables on di
2011/3/8 Chris Goffinet
> How large are your SSTables on disk? My thought was because you have so
> many on disk, we have to store the bloom filter + every 128 keys from index
> in memory.
>
>
0.5GB
But as I understand store in memory happens only when read happens, i do
only inserts. And i thin
How large are your SSTables on disk? My thought was because you have so many
on disk, we have to store the bloom filter + every 128 keys from index in
memory.
On Mon, Mar 7, 2011 at 4:35 PM, ruslan usifov wrote:
>
>
> 2011/3/8 Chris Goffinet
>
>> The rows you are inserting, what is your update r
2011/3/8 Chris Goffinet
> The rows you are inserting, what is your update ratio to those rows?
>
> I doesn't update them only insert, with speed 16000 per second
The rows you are inserting, what is your update ratio to those rows?
On Mon, Mar 7, 2011 at 4:03 PM, ruslan usifov wrote:
>
>
> 2011/3/8 Chris Goffinet
>
> Can you tell me how many SSTables on disk when you see GC pauses? In your 3
>> node cluster, what's the RF factor?
>>
>>
> About 30-40, and
2011/3/8 Chris Goffinet
> Can you tell me how many SSTables on disk when you see GC pauses? In your 3
> node cluster, what's the RF factor?
>
>
About 30-40, and i use RF=2, and insert rows with QUORUM consistency level
Can you tell me how many SSTables on disk when you see GC pauses? In your 3
node cluster, what's the RF factor?
On Mon, Mar 7, 2011 at 1:50 PM, ruslan usifov wrote:
>
>
> 2011/3/8 Jonathan Ellis
>
> It sounds like you're complaining that the JVM sometimes does
>> stop-the-world GC.
>>
>> You can
Hi Ruslan,
It looks like Jonathan and Stu have already been working to reduce
garbage collection on v.8 The ticket is at
https://issues.apache.org/jira/browse/CASSANDRA-2252
Jonathan, is there any way to apply the patch to .73 and have ruslan
test it to see if it fixes his issue with Garbage col
So, are you saying this is normal and expected from Cassandra? So,
under load, we can expect java garbage collection to stop the Cassandra
process on that server from time to time, essentially taking out the
node for short periods of time while it does garbage collection?
Also, why is there so mu
2011/3/8 Jonathan Ellis
> It sounds like you're complaining that the JVM sometimes does
> stop-the-world GC.
>
> You can mitigate this but not (for most workloads) eliminate it with
> GC option tuning. That's simply the state of the art for Java garbage
> collection right now.
>
>
Hm, but what t
It sounds like you're complaining that the JVM sometimes does stop-the-world GC.
You can mitigate this but not (for most workloads) eliminate it with
GC option tuning. That's simply the state of the art for Java garbage
collection right now.
On Sun, Mar 6, 2011 at 2:18 AM, ruslan usifov wrote:
Hello thanks for reply
2011/3/7 Aaron Morton
> It's always possible to run out of memory. Can you provide...
>
> - number cf's and their Memtable settings
>
1 CF with memtable 64MB, other settings as default
> - any row or key cache settings
>
Its stay default e.g 20, but i don't do any r
It's always possible to run out of memory. Can you provide...
- number cf's and their Memtable settings
- any row or key cache settings
- any other buffer or memory settings you may have changed in Cassandra.yaml.
- what load you are putting on the cluster, e.g. Inserting x rows/columns per
secon
2011/3/6 aaron morton
> Your node is under memory pressure, after the GC there is still 5.7GB in
> use. In fact it looks like memory usage went up during the GC process.
>
> Can you reduce the memtable size, caches or the number of CF's or increase
> the JVM size? Also is this happening under hea
Do you have row cache enabled? Disable it. If it fixes it and you want
it, re-enable but consider row sizes and the cap on the cache size..
--
/ Peter Schuller
2011/3/6 aaron morton
> Your node is under memory pressure, after the GC there is still 5.7GB in
> use. In fact it looks like memory usage went up during the GC process.
>
> Can you reduce the memtable size, caches or the number of CF's or increase
> the JVM size? Also is this happening under hea
Your node is under memory pressure, after the GC there is still 5.7GB in use.
In fact it looks like memory usage went up during the GC process.
Can you reduce the memtable size, caches or the number of CF's or increase the
JVM size? Also is this happening under heavy load ?
Aaron
On 6/03/201
33 matches
Mail list logo
