Re: flush_largest_memtables_at messages in 7.4
Peter Schuller wrote: Saturated. But read latency is still something like 30ms which I would think would be much higher if it's saturated. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/flush-largest-memtables-at-messages-in-7-4-tp6266221p6269655.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: flush_largest_memtables_at messages in 7.4
One correction qu size in iostat ranges between 6-120. But still this doesn't explain why read latency is low in cfstats. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/flush-largest-memtables-at-messages-in-7-4-tp6266221p6269875.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: flush_largest_memtables_at messages in 7.4
But read latency is still something like 30ms which I would think would be much higher if it's saturated. No. You're using stress, so you have some total cap on concurrency. Given a fixed concurrency, you'll saturate at some particular average latency which is mostly a function of the backlog implies by the concurrency and the average time for each actual request to process. If you double the concurrency of your stress clients, you should expect roughly twice the average latency. Running a fixed-concurrency benchmark against a saturated cluster will have vastly different effects on latency than trying to serve real live traffic without a feedback mechanism with a system which is processing fewer requests than incoming. -- / Peter Schuller
Re: flush_largest_memtables_at messages in 7.4
I still don't understand. You would expect read latency to increase drastically when it's fully saturated and lot of READ drop messages also, correct? I don't see that in cfstats or system.log which I don't really understand why. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/flush-largest-memtables-at-messages-in-7-4-tp6266221p6270244.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: flush_largest_memtables_at messages in 7.4
I still don't understand. You would expect read latency to increase drastically when it's fully saturated and lot of READ drop messages also, correct? I don't see that in cfstats or system.log which I don't really understand why. No. With a fixed concurrency there is only so many outstanding requests at any given moment. Unless the concurrency is sufficiently huge to exhaust queue sizes that's not what you see. It's just the way it works. An analogous situation is this: You have N threads all doing random reads from your disk. If each thread is responsible for exactly one outstanding I/O operation at a time, you will never have more than N outstanding requests to the I/O device in total. On average then, when a request starts it will have N - 1 requests in front of it waiting to be processes, so the latency will increase linearly with N. Double the concurrency, double the latency. Something like Cassandra/postgresql/etc behaves similarly. If you're running a benchmark at a fixed concurrency of N, you won't have more than N total requests outstanding at any given moment. -- / Peter Schuller
Re: flush_largest_memtables_at messages in 7.4
Actually when I run 2 stress clients in parallel I see Read Latency stay the same. I wonder if cassandra is reporting accurate nos. I understand your analogy but for some reason I don't see that happening with the results I am seeing with multiple stress clients running. So I am just confused where the real bottleneck is. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/flush-largest-memtables-at-messages-in-7-4-tp6266221p6270942.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
flush_largest_memtables_at messages in 7.4
I am using cassandra 7.4 and getting these messages. Heap is 0.7802529021498031 full. You may need to reduce memtable and/or cache sizes Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically How do I verify that I need to adjust any thresholds? And how to calculate correct value? When I got this message only reads were occuring. create keyspace StressKeyspace with replication_factor = 3 and placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy'; use StressKeyspace; drop column family StressStandard; create column family StressStandard with comparator = UTF8Type and keys_cached = 100 and memtable_flush_after = 1440 and memtable_throughput = 128; nodetool -h dsdb4 tpstats Pool NameActive Pending Completed ReadStage32 281 456598 RequestResponseStage 0 0 797237 MutationStage 0 0 499205 ReadRepairStage 0 0 149077 GossipStage 0 0 217227 AntiEntropyStage 0 0 0 MigrationStage0 0201 MemtablePostFlusher 0 0 1842 StreamStage 0 0 0 FlushWriter 0 0 1841 FILEUTILS-DELETE-POOL 0 0 3670 MiscStage 0 0 0 FlushSorter 0 0 0 InternalResponseStage 0 0 0 HintedHandoff 0 0 15 cfstats Keyspace: StressKeyspace Read Count: 460988 Read Latency: 38.07654727454945 ms. Write Count: 499205 Write Latency: 0.007409593253272703 ms. Pending Tasks: 0 Column Family: StressStandard SSTable count: 9 Space used (live): 247408645485 Space used (total): 247408645485 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 1878 Read Count: 460989 Read Latency: 28.237 ms. Write Count: 499205 Write Latency: NaN ms. Pending Tasks: 0 Key cache capacity: 100 Key cache size: 299862 Key cache hit rate: 0.6031833150384193 Row cache: disabled Compacted row minimum size: 219343 Compacted row maximum size: 5839588 Compacted row mean size: 497474 -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/flush-largest-memtables-at-messages-in-7-4-tp6266221p6266221.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: flush_largest_memtables_at messages in 7.4
your jvm heap has reached 78% so cassandra automatically flushes its memtables. you need to explain more about your configuration. 32 or 64 bit OS, what is max heap, how much ram installed? If this happens under stress test conditions its probably understandable. you should look into graphing your memory usage, or use the jconsole to graph heap during your tests. On Apr 12, 2011, at 8:36 PM, mcasandra wrote: I am using cassandra 7.4 and getting these messages. Heap is 0.7802529021498031 full. You may need to reduce memtable and/or cache sizes Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically How do I verify that I need to adjust any thresholds? And how to calculate correct value? When I got this message only reads were occuring. create keyspace StressKeyspace with replication_factor = 3 and placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy'; use StressKeyspace; drop column family StressStandard; create column family StressStandard with comparator = UTF8Type and keys_cached = 100 and memtable_flush_after = 1440 and memtable_throughput = 128; nodetool -h dsdb4 tpstats Pool NameActive Pending Completed ReadStage32 281 456598 RequestResponseStage 0 0 797237 MutationStage 0 0 499205 ReadRepairStage 0 0 149077 GossipStage 0 0 217227 AntiEntropyStage 0 0 0 MigrationStage0 0201 MemtablePostFlusher 0 0 1842 StreamStage 0 0 0 FlushWriter 0 0 1841 FILEUTILS-DELETE-POOL 0 0 3670 MiscStage 0 0 0 FlushSorter 0 0 0 InternalResponseStage 0 0 0 HintedHandoff 0 0 15 cfstats Keyspace: StressKeyspace Read Count: 460988 Read Latency: 38.07654727454945 ms. Write Count: 499205 Write Latency: 0.007409593253272703 ms. Pending Tasks: 0 Column Family: StressStandard SSTable count: 9 Space used (live): 247408645485 Space used (total): 247408645485 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 1878 Read Count: 460989 Read Latency: 28.237 ms. Write Count: 499205 Write Latency: NaN ms. Pending Tasks: 0 Key cache capacity: 100 Key cache size: 299862 Key cache hit rate: 0.6031833150384193 Row cache: disabled Compacted row minimum size: 219343 Compacted row maximum size: 5839588 Compacted row mean size: 497474 -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/flush-largest-memtables-at-messages-in-7-4-tp6266221p6266221.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: flush_largest_memtables_at messages in 7.4
64 bit 12 core 96 GB RAM -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/flush-largest-memtables-at-messages-in-7-4-tp6266221p6266400.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: flush_largest_memtables_at messages in 7.4
Heap is 0.7802529021498031 full. You may need to reduce memtable and/or cache sizes Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically How do I verify that I need to adjust any thresholds? And how to calculate correct value? Is this on the same cluster/nodes that you're doing your 250k column stresses (the other thread)? In any case, for typical cases there is: http://www.datastax.com/docs/0.7/operations/tuning -- / Peter Schuller
Re: flush_largest_memtables_at messages in 7.4
Yes -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/flush-largest-memtables-at-messages-in-7-4-tp6266221p6266726.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: flush_largest_memtables_at messages in 7.4
Yes Without checking I don't know the details of the memtable threshold calculations enough to be sure whether large columns are somehow causing the size estimations to be ineffective (off hand I would expect the reverse since the overhead of the Java object structures become much less significant); but if this is not the case, then this particular problem should be a matter of adjusting heap size according to your memtable thresholds. I.e., increase heap size and/or decrease memtable flush thresholds. -- / Peter Schuller
Re: flush_largest_memtables_at messages in 7.4
One thing I am noticing is that cache hit rate is very low even though my cache key size is 1M and I have less than 1M rows. Not sure why so many cache miss? Keyspace: StressKeyspace Read Count: 162506 Read Latency: 45.22479006928975 ms. Write Count: 247180 Write Latency: 0.011610943442026053 ms. Pending Tasks: 0 Column Family: StressStandard SSTable count: 184 Space used (live): 99616537894 Space used (total): 99616537894 Memtable Columns Count: 351 Memtable Data Size: 171716049 Memtable Switch Count: 543 Read Count: 162507 Read Latency: 317.892 ms. Write Count: 247180 Write Latency: 0.006 ms. Pending Tasks: 0 Key cache capacity: 100 Key cache size: 256013 Key cache hit rate: 0.33801452784503633 Row cache: disabled Compacted row minimum size: 182786 Compacted row maximum size: 5839588 Compacted row mean size: 537470 -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/flush-largest-memtables-at-messages-in-7-4-tp6266221p6267234.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: flush_largest_memtables_at messages in 7.4
One thing I am noticing is that cache hit rate is very low even though my cache key size is 1M and I have less than 1M rows. Not sure why so many cache miss? The key cache should be strictly LRU for read-only workloads. For write/read workloads it may not be strictly LRU because compaction causes key cache migration. In your case: Key cache capacity: 100 Key cache size: 256013 Key cache hit rate: 0.33801452784503633 So you have only 256k in the cache. Have you run for long enough after enabling it for it to actually be fully populated? -- / Peter Schuller
Re: flush_largest_memtables_at messages in 7.4
Does it really matter how long cassandra has been running? I thought it will keep keys of 1M at least. Regarding your previous question about queue size in iostat I see it ranging from 114-300. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/flush-largest-memtables-at-messages-in-7-4-tp6266221p6267728.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: flush_largest_memtables_at messages in 7.4
Does it really matter how long cassandra has been running? I thought it will keep keys of 1M at least. It will keep up to the limit, and it will save caches periodically and reload them on start. But the cache needs to be populated by traffic first. If you wrote a bunch of data, enabled the row cache, and began reading you have to first wait for population of the cache prior to looking at cache locality. Note that the saving of caches is periodic and if you were constantly restarting nodes during testing maybe it never got saved with the full set of keys. Regarding your previous question about queue size in iostat I see it ranging from 114-300. Saturated. -- / Peter Schuller
