RE: Consurrent compactors
My concern is not anout cleanup, but about supposed tendency of small sstables to accumulate during a single long running compactions. When next task is for the same column family as currently long-running compaction, other column families compactions are freezed and concurrent_compactors 1 setting just not working. Best regards/ Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.com Phone: +370 5 212 3063. Fax: +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania [Adform news]http://www.adform.com/ [Visit us!] Follow: [twitter]http://twitter.com/#!/adforminsider Visit our bloghttp://www.adform.com/site/blog Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. From: aaron morton [mailto:aa...@thelastpickle.com] Sent: Wednesday, February 01, 2012 21:51 To: user@cassandra.apache.org Subject: Re: Consurrent compactors (Assuming 1.0* release) From the comments in cassandra.yaml # Number of simultaneous compactions to allow, NOT including # validation compactions for anti-entropy repair. Simultaneous # compactions can help preserve read performance in a mixed read/write # workload, by mitigating the tendency of small sstables to accumulate # during a single long running compactions. The default is usually # fine and if you experience problems with compaction running too # slowly or too fast, you should look at # compaction_throughput_mb_per_sec first. # # This setting has no effect on LeveledCompactionStrategy. # # concurrent_compactors defaults to the number of cores. # Uncomment to make compaction mono-threaded, the pre-0.8 default. #concurrent_compactors: 1 If you set it to 1 then only 1 compaction should run at a time, excluding validation. How often do you run a cleanup compaction ? They are only necessary when you perform a token move. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 1/02/2012, at 9:48 PM, Viktor Jevdokimov wrote: Hi, When concurrent compactors are set to more then 1, it's rare when more than 1 compaction is running in parallel. Didn't checked the source code, but it looks like when next compaction task (any of minor, major, or cleanup) is for the same CF, it will not start in parallel and next tasks are not checked. Will it be possible to check all tasks, not only the next one, to find which of them can be started? This is actual especially when nightly cleanup is running, a lot of cleanup tasks are pending, regular minor compactions are waiting until all cleanup compactions are finished. Best regards/ Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.commailto:viktor.jevdoki...@adform.com Phone: +370 5 212 3063. Fax: +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania signature-logo29.pnghttp://www.adform.com/ dm-exco4823.png Follow: tweet18be.pnghttp://twitter.com/#!/adforminsider Visit our bloghttp://www.adform.com/site/blog Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. inline: signature-logo29.pnginline: dm-exco4823.pnginline: tweet18be.png
Write latency of counter updates across multiple rows
Hi, In our use case, we maintain minute-wise roll ups for different metrics. These are stored in a counter column family where the row key is a composite containing the timestamp rounded to the last minute and an integer between 0-9 (This integer is calculated as the MD5 hash of the metric mod 10). The column names are the metrics we wish to track. Typically, each row has about 100,000 counters. We tested two scenarios. The first one is as mentioned above. In this case we got a per write latency of about 80 micro-seconds to 100 micro-seconds. In the other scenario, we calculated the integer in the row key as mod 100. In this case we observed a per write latency of 50 micro-seconds to 70 micro-seconds. I wish to understand why updates to counters were faster as they got spread across multiple rows? Cluster summary : 4 nodes running Cassandra 1.0.5. Each with 8 cores, 32G RAM, 10G Cassandra heap. We are using replication factor of 2. -- Thanks! Amit Chavan
Re: Unsubscribe
Help yourself : http://wiki.apache.org/cassandra/FAQ#unsubscribe Alain 2012/2/3 francesco.tangari@gmail.com lol -- francesco.tangari@gmail.com Inviato con Sparrow http://www.sparrowmailapp.com/?sig Il giorno venerdì 3 febbraio 2012, alle ore 14.51, Larry Liu ha scritto: Please unsubscribe me. Thanks!
Re: Unsubscribe
On Fri, Feb 3, 2012 at 7:51 AM, Larry Liu larryliu...@gmail.com wrote: Please unsubscribe me. http://goo.gl/lQJC2 -- Eric Evans Acunu | http://www.acunu.com | @acunu
Re: WARN [Memtable] live ratio
It's a warn because it's nonsense for the JVM to report that an column + overhead, takes less space than just the column data itself. On Tue, Jan 31, 2012 at 9:41 PM, Mohit Anchlia mohitanch...@gmail.com wrote: I guess this is not really a WARN in that case. On Tue, Jan 31, 2012 at 4:29 PM, aaron morton aa...@thelastpickle.com wrote: The ratio is the ratio of serialised bytes for a memtable to actual JVM allocated memory. Using a ratio below 1 would imply the JVM is using less bytes to store the memtable in memory than it takes to store it on disk (without compression). The ceiling for the ratio is 64. The ratio is calculated periodically so if the workload changes, such as system start up, the number will lag behind. I would guess numbers less than 1 mean the memtable does not have any data. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 1/02/2012, at 8:27 AM, Radim Kolar wrote: but a ration of 1 may occur for column families with a very high update to insert ratio. better to ask why minimum ratio is 1.0. What harm can be done with using 1.0 ratio? -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: Recommended configuration for good streaming performance?
On Thu, Feb 2, 2012 at 11:52 AM, Erik Forsberg forsb...@opera.com wrote: Our Cassandra hardware are 16 core (including HT cores) with 24GiB of RAM. They have two disks each. So far we've configured them with commitlog on one disk and sstables on the other, but with streaming not using commitlog (correct?) Yes. maybe it makes sense to have sstables on both disks, doubling available I/O? That would make sense (raid0) if streaming performance is your main priority. But if you have a more balanced workload involving lots of reads and [non-bulk] updates, then the commitlog on a separate spindle is going to make a big difference there. Thoughts on number of parallel streaming clients? Good question. I don't know that I have any useful intuition here, other than more than 1. :) Let us know if you do some experimenting! -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Internal error processing batch_mutate java.util.ConcurrentModificationException
What may be cause of the following exception in 1.0.7 Cassandra: ERROR [Thrift:134] 2012-02-03 15:51:02,800 Cassandra.java (line 3462) Internal error processing batch_mutate java.util.ConcurrentModificationException at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372) at java.util.AbstractList$Itr.next(AbstractList.java:343) at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:532) at org.apache.cassandra.service.AbstractWriteResponseHandler.waitForHints(AbstractWriteResponseHandler.java:89) at org.apache.cassandra.service.AbstractWriteResponseHandler.get(AbstractWriteResponseHandler.java:58) at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:201) at org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:639) at org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:590) at org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:598) at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3454) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Best regards/ Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.com Phone: +370 5 212 3063. Fax: +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania [Adform news]http://www.adform.com/ [Visit us!] Follow: [twitter]http://twitter.com/#!/adforminsider Visit our bloghttp://www.adform.com/site/blog Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. inline: signature-logo4913.pnginline: dm-exco199c.pnginline: tweet3130.png
Re: WARN [Memtable] live ratio
On Fri, Feb 3, 2012 at 7:32 AM, Jonathan Ellis jbel...@gmail.com wrote: It's a warn because it's nonsense for the JVM to report that an column + overhead, takes less space than just the column data itself. But is there any action we need to take, or worry about? We trigger alerts based on WARN and ERROR. But if there is nothing to do then it probably is just an INFO. On Tue, Jan 31, 2012 at 9:41 PM, Mohit Anchlia mohitanch...@gmail.com wrote: I guess this is not really a WARN in that case. On Tue, Jan 31, 2012 at 4:29 PM, aaron morton aa...@thelastpickle.com wrote: The ratio is the ratio of serialised bytes for a memtable to actual JVM allocated memory. Using a ratio below 1 would imply the JVM is using less bytes to store the memtable in memory than it takes to store it on disk (without compression). The ceiling for the ratio is 64. The ratio is calculated periodically so if the workload changes, such as system start up, the number will lag behind. I would guess numbers less than 1 mean the memtable does not have any data. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 1/02/2012, at 8:27 AM, Radim Kolar wrote: but a ration of 1 may occur for column families with a very high update to insert ratio. better to ask why minimum ratio is 1.0. What harm can be done with using 1.0 ratio? -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: WARN [Memtable] live ratio
You should come up with a way to reproduce so we can fix it. :) On Fri, Feb 3, 2012 at 10:31 AM, Mohit Anchlia mohitanch...@gmail.com wrote: On Fri, Feb 3, 2012 at 7:32 AM, Jonathan Ellis jbel...@gmail.com wrote: It's a warn because it's nonsense for the JVM to report that an column + overhead, takes less space than just the column data itself. But is there any action we need to take, or worry about? We trigger alerts based on WARN and ERROR. But if there is nothing to do then it probably is just an INFO. On Tue, Jan 31, 2012 at 9:41 PM, Mohit Anchlia mohitanch...@gmail.com wrote: I guess this is not really a WARN in that case. On Tue, Jan 31, 2012 at 4:29 PM, aaron morton aa...@thelastpickle.com wrote: The ratio is the ratio of serialised bytes for a memtable to actual JVM allocated memory. Using a ratio below 1 would imply the JVM is using less bytes to store the memtable in memory than it takes to store it on disk (without compression). The ceiling for the ratio is 64. The ratio is calculated periodically so if the workload changes, such as system start up, the number will lag behind. I would guess numbers less than 1 mean the memtable does not have any data. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 1/02/2012, at 8:27 AM, Radim Kolar wrote: but a ration of 1 may occur for column families with a very high update to insert ratio. better to ask why minimum ratio is 1.0. What harm can be done with using 1.0 ratio? -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: WARN [Memtable] live ratio
Dne 3.2.2012 17:46, Jonathan Ellis napsal(a): You should come up with a way to reproduce so we can fix it. :) it happens after HH delivery when memtable contains lot of deletes
Re: WARN [Memtable] live ratio
On Fri, Feb 3, 2012 at 8:46 AM, Jonathan Ellis jbel...@gmail.com wrote: You should come up with a way to reproduce so we can fix it. :) This happens when server is idle for a while with not so much data and then you start a load like 50 write and 50 read requests per sec. We have 4 CF that we write to and then read from. On Fri, Feb 3, 2012 at 10:31 AM, Mohit Anchlia mohitanch...@gmail.com wrote: On Fri, Feb 3, 2012 at 7:32 AM, Jonathan Ellis jbel...@gmail.com wrote: It's a warn because it's nonsense for the JVM to report that an column + overhead, takes less space than just the column data itself. But is there any action we need to take, or worry about? We trigger alerts based on WARN and ERROR. But if there is nothing to do then it probably is just an INFO. On Tue, Jan 31, 2012 at 9:41 PM, Mohit Anchlia mohitanch...@gmail.com wrote: I guess this is not really a WARN in that case. On Tue, Jan 31, 2012 at 4:29 PM, aaron morton aa...@thelastpickle.com wrote: The ratio is the ratio of serialised bytes for a memtable to actual JVM allocated memory. Using a ratio below 1 would imply the JVM is using less bytes to store the memtable in memory than it takes to store it on disk (without compression). The ceiling for the ratio is 64. The ratio is calculated periodically so if the workload changes, such as system start up, the number will lag behind. I would guess numbers less than 1 mean the memtable does not have any data. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 1/02/2012, at 8:27 AM, Radim Kolar wrote: but a ration of 1 may occur for column families with a very high update to insert ratio. better to ask why minimum ratio is 1.0. What harm can be done with using 1.0 ratio? -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
yet a couple more questions on composite columns
Hi,
We are getting close to replacing our super-column based schema to
something more efficient, and I am trying to wrap my heads around composite
columns.
An email from another list member has clarified that composite and
non-composite columns should not be mixed in the same CF because only one
comparator is used for all columns. Does this also mean all composite
columns within the same CF must be homogeneous types? and I cannot have
one composite column name with 3 components while another with 4 components?
Another question I have is how flexible composite columns actually are. If
my data model has a CF containing US zip codes with the following composite
columns:
{OH:Spring Field} : 45503
{OH:Columbus} : 43085
{FL:Spring Field} : 32401
{FL:Key West} : 33040
I know I can ask cassandra to give me the zip codes of all cities in OH.
But can I ask it to give me the zip codes of all cities named Spring
Field using this model? Thanks.
-- Y.
Cassandra OOM - 1.0.2
Hey guys, I am getting an out of memory (mmap failed) error with Cassandra 1.0.2. The relevant log lines are pasted at http://pastebin.com/UM28ZC1g. Cassandra works fine until it reaches about 300-400GB of load (on one instance, I have 12 nodes RF=2). Then nodes start failing with such errors. The nodes are pretty beefy, 32GB of ram, 8 cores. Increasing the JVM heap size does not help. I am running on a 64bit jvm. I am using jna. I have memlock unlimited for the user. (I confirmed this by looking at /proc/pid/limits). I also tried restarting the process as root, but it crashes with the same error. Also the number of files that I have in the data directory is about ~300, so it should not be exceeding the open files limit. I don't know if this is relevant. I just have two column families, counter_object and counter_time. I am using very wide columns, so row sizes can be huge. You can see from the log link, that the *.db files are sometimes pretty big. Please help! Thank you! -- Regards, Ajeet
Re: yet a couple more questions on composite columns
On 4 February 2012 06:21, Yiming Sun yiming@gmail.com wrote:
I cannot have one composite column name with 3 components while another
with 4 components?
Just put 4 components and left last empty (if it is same type)?!
Another question I have is how flexible composite columns actually are. If
my data model has a CF containing US zip codes with the following composite
columns:
{OH:Spring Field} : 45503
{OH:Columbus} : 43085
{FL:Spring Field} : 32401
{FL:Key West} : 33040
I know I can ask cassandra to give me the zip codes of all cities in OH.
But can I ask it to give me the zip codes of all cities named Spring
Field using this model? Thanks.
No. You set first composite component at first.
Re: Can you query Cassandra while it's doing major compaction
At Netflix we rotate the major compactions around the cluster, don't run them all at once. We also either take that node out of client traffic so it doesn't get used as a coordinator or use the Astyanax client that is latency and token aware to steer traffic to the other replicas. We are running on EC2 with lots of CPU and RAM but only two internal disk spindles, so if you have lots of IOPS available in your config, you may find that it doesn't affect read times much. Adrian On Thu, Feb 2, 2012 at 11:44 PM, Peter Schuller peter.schul...@infidyne.com wrote: If every node in the cluster is running major compaction, would it be able to answer any read request? And is it wise to write anything to a cluster while it's doing major compaction? Compaction is something that is supposed to be continuously running in the background. As noted, it will have a performance impact in that it (1) generates I/O, (2) leads to cache eviction, and (if you're CPU bound rather than disk bound) (3) adds CPU load. But there is no intention that clients should have to care about compaction; it's to be viewed as a background operation continuously happening. A good rule of thumb is that an individual node should be able to handle your traffic when doing compaction; you don't want to be in the position where you're just barely dealing with the traffic, and a node doing compaction not being able to handle it. -- / Peter Schuller (@scode, http://worldmodscode.wordpress.com)
Re: yet a couple more questions on composite columns
On 4 February 2012 06:21, Yiming Sun yiming@gmail.com wrote:
I cannot have one composite column name with 3 components while another with
4 components?
Just put 4 components and left last empty (if it is same type)?!
Another question I have is how flexible composite columns actually are. If
my data model has a CF containing US zip codes with the following composite
columns:
{OH:Spring Field} : 45503
{OH:Columbus} : 43085
{FL:Spring Field} : 32401
{FL:Key West} : 33040
I know I can ask cassandra to give me the zip codes of all cities in OH.
But can I ask it to give me the zip codes of all cities named Spring Field
using this model? Thanks.
No. You set first composite component at first.
I'd use a dynamic CF:
row key = state abbreviation
column name = city name
column value = zip code (or a complex object, one of whose properties is zip
code)
you can iterate over the columns in a single row to get a state's city names
and their zip code and you can do a get_range_slices on all keys for the
columns starting and ending on the city name to find out the zip codes for a
cities with the given name.
I think
- Chris
