strange behaviour in cassandra
Hi All, I stop my cassandra node and restarted it,this start and stop resulted in decrease in the space used by cassandra by 20%. I am not sure what is happening?Can Anyone please shower some light on this behaviour. Thanks Anurag
Re: strange behaviour in cassandra
I stop my cassandra node and restarted it,this start and stop resulted in decrease in the space used by cassandra by 20%. I am not sure what is happening?Can Anyone please shower some light on this Probably because unused sstables were deleted on start-up that were otherwise not used, and waiting on full GC to be removed. Check system.log for deleted sstable messages. I didn't find a wiki ref right now, but in short cassandra uses weak references to discover when an sstable is guaranteed to no longer be used, in order to perform removals of the actual sstable files. This can lead to a delay of sstable removal which will vary with whatever else is happening (the more busy the node, the more often a concurrent mark/sweep gc phase is triggered, and the more frequently obsolete sstables are deleted). -- / Peter Schuller
Re: strange behaviour in cassandra
Thanks for your response I think the reference you are talking about is http://wiki.apache.org/cassandra/MemtableSSTable Regards, Anurag On Sun, May 8, 2011 at 11:01 AM, Peter Schuller peter.schul...@infidyne.com wrote: I stop my cassandra node and restarted it,this start and stop resulted in decrease in the space used by cassandra by 20%. I am not sure what is happening?Can Anyone please shower some light on this Probably because unused sstables were deleted on start-up that were otherwise not used, and waiting on full GC to be removed. Check system.log for deleted sstable messages. I didn't find a wiki ref right now, but in short cassandra uses weak references to discover when an sstable is guaranteed to no longer be used, in order to perform removals of the actual sstable files. This can lead to a delay of sstable removal which will vary with whatever else is happening (the more busy the node, the more often a concurrent mark/sweep gc phase is triggered, and the more frequently obsolete sstables are deleted). -- / Peter Schuller
Re: Filter on row iterator
AFAIK Token suport is there for the hadoop integration and is not something you should normally need to use. It will be easier to use a KeyRange with a start_key if you have a consistent key format, with the most significant discriminator first. e.g. category/id Then you can use a range start such as foo/ If you are using the RandomParitioner don't forget http://wiki.apache.org/cassandra/FAQ#range_rp Hope that helps. Aaron - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 6 May 2011, at 22:34, cbert...@libero.it wrote: Hi all, I have a column family with about 300 rows. Rows name are of 2 categories: number (eg: 12345) e_number (eg: e_12345) is there any way to extract only rows that are numbers? For the moment I'm iterating over all rows with a KeyRange and filtering client-side but I don't like this solution. I've seen that KeyRange can be created using tokens instead of keys but I don't understand how they works and did not find any working example. (Java/Pelops/Cassandra 0.7.5) TIA Carlo
RequestResponseStage Assertion Error
I have a 4 node ring that was setup with tokens a,b,c,d using NTS and 2 nodes in each of 2 datacenters with a replication of DC1:1, DC2:1. I was getting uneven replica placement so I did a drop keyspace, followed by a nodetool move to DC1 having tokens (a,b) and DC2 having tokens (a+1,b+1) , then I removed the old data directory and recreated the keyspace. This has resolved my uneven replication, but now on one of my nodes I consistently get these errors. ERROR [RequestResponseStage:1] 2011-05-08 20:52:28,824 DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor java.lang.AssertionError at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) I dont know if i ... shutdown the node in the middle of one of the earlier operations or what. It seems to insert data fine, and my distribution is very even. What is this error, what is causing it, and how do i fix it? Thanks -Eric
Re: CFHistograms?
Timeouts (hence errors) will register at the rpc timeout value. On Sat, May 7, 2011 at 1:27 PM, Josep Blanquer blanq...@rightscale.com wrote: I believe the offset value of Writes and Reads are in *micro*seconds right? (that page talks about *milli*seconds) Also, are any timeouts or errors reflected in those times or just successful operations? if not, is there any JMX or other tool to keep track of them? Josep M. On Fri, May 6, 2011 at 9:09 AM, Jonathan Ellis jbel...@gmail.com wrote: Those are updated at compaction time. On Thu, May 5, 2011 at 11:38 PM, Xaero S xaeros...@gmail.com wrote: Can someone point me to a document that explains how to interpret CFHistograms output? i went through http://narendrasharma.blogspot.com/2011/04/cassandra-07x-understanding-output-of.html which is a good beginning, but was wondering if there was anything more detailed. For e.g when i run CFHistograms, i see rowsize and columncount items in the table always 0 (which cant be right?) -Xaero -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: New node not joining
Ah, I see the case you are talking about. If the node will auto bootstrap on startup if when it joins the ring: it is not already bootstrapped, auto bootstrap is enabled, and the node is not in it's own seed list. In the auto bootstrap process then finds the token it wants, but aborts the process if there are no non system tables defined.That may happen because the bootstrap code finds the node with the highest load and splits it's range, if all the nodes have zero load (no user data) then that process is unreliable. But it's also unreliable if there is a schema and no data. Created https://issues.apache.org/jira/browse/CASSANDRA-2625 to see if it can be changed. Thanks - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 7 May 2011, at 05:25, Len Bucchino wrote: While I agree that what you suggested is a very good idea the bootstrapping process _should_ work properly. Here is some additional detail on the original problem. If the current node that you are trying to bootstrap has itself listed in seeds in its yaml then it will be able to bootstrap on an empty schema. If it does not have itself listed in seeds in its yaml and you have and empty schema then the bootstrap process will not complete and no errors will be reported in the logs even with debug enabled. From: aaron morton [mailto:aa...@thelastpickle.com] Sent: Thursday, May 05, 2011 6:51 PM To: user@cassandra.apache.org Subject: Re: New node not joining When adding nodes it is a *very* good idea to manually set the tokens, see http://wiki.apache.org/cassandra/Operations#Load_balancing bootstrap is a process that happens only once on a node, where as well as telling the other nodes it's around it asks them to stream over the data it will no be responsible for. nodetool loadbalance is an old utility that should have better warnings not to use it. The best way to load balance the cluster is manually creating the tokens and assigning them either using the initial_token config param or using nodetool move. Hope that helps. - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 6 May 2011, at 08:37, Sanjeev Kulkarni wrote: Here is what I did. I booted up the first one. After that I started the second one with bootstrap turned off. Then I did a nodetool loadbalance on the second node. After which I added the third node again with bootstrap turned off. Then did the loadbalance again on the third node. This seems to have successfully completed and I am now able to read/write into my system. Thanks! On Thu, May 5, 2011 at 1:22 PM, Len Bucchino len.bucch...@veritix.com wrote: I just rebuilt the cluster in the same manner as I did originally except after I setup the first node I added a keyspace and column family before adding any new nodes. This time the 3rd node auto bootstrapped successfully. From: Len Bucchino [mailto:len.bucch...@veritix.com] Sent: Thursday, May 05, 2011 1:31 PM To: user@cassandra.apache.org Subject: RE: New node not joining Also, setting auto_bootstrap to false and setting token to the one that it said it would use in the logs allows the new node to join the ring. From: Len Bucchino [mailto:len.bucch...@veritix.com] Sent: Thursday, May 05, 2011 1:25 PM To: user@cassandra.apache.org Subject: RE: New node not joining Adding the fourth node to the cluster with an empty schema using auto_bootstrap was not successful. A nodetool netstats on the new node shows “Mode: Joining: getting bootstrap token” similar to what the third node did before it was manually added. Also, there are no exceptions in the logs but it never joins the ring. From: Sanjeev Kulkarni [mailto:sanj...@locomatix.com] Sent: Thursday, May 05, 2011 11:47 AM To: user@cassandra.apache.org Subject: Re: New node not joining Hi Len, This looks like a decent workaround. I would be very interested to see how the addition of the 4th node went. Please post it whenever you get a chance. Thanks! On Thu, May 5, 2011 at 6:47 AM, Len Bucchino len.bucch...@veritix.com wrote: I have the same problem on 0.7.5 auto bootstrapping a 3rd node onto an empty 2 node test cluster (the two nodes were manually added) and the it currently has an empty schema. My log entries look similar to yours. I took the new token it says its going to use from the log file added it to the yaml and turned off auto bootstrap and the node added fine. I'm bringing up a 4th node now and will see if it has the same problem auto bootstrapping. From: Sanjeev Kulkarni [sanj...@locomatix.com] Sent: Thursday, May 05, 2011 2:18 AM To: user@cassandra.apache.org Subject: New node not joining Hey guys, I'm running into what seems like a very basic problem. I have a one node cassandra instance. Version 0.7.5. Freshly installed.
Re: RequestResponseStage Assertion Error
What version are you on ? Check the nodetool ring from each node in your cluster to check they have the same view. Hope that helps. - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 9 May 2011, at 09:37, Eric tamme wrote: I have a 4 node ring that was setup with tokens a,b,c,d using NTS and 2 nodes in each of 2 datacenters with a replication of DC1:1, DC2:1. I was getting uneven replica placement so I did a drop keyspace, followed by a nodetool move to DC1 having tokens (a,b) and DC2 having tokens (a+1,b+1) , then I removed the old data directory and recreated the keyspace. This has resolved my uneven replication, but now on one of my nodes I consistently get these errors. ERROR [RequestResponseStage:1] 2011-05-08 20:52:28,824 DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor java.lang.AssertionError at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) I dont know if i ... shutdown the node in the middle of one of the earlier operations or what. It seems to insert data fine, and my distribution is very even. What is this error, what is causing it, and how do i fix it? Thanks -Eric
Re: Does anybody have experience with running Cassandra in Amazon's Virtual Private Cloud (VPC)?
We are planning to use EIP's + https://issues.apache.org/jira/browse/CASSANDRA-2452 for AWS multi region deployment. Regards, /VJ On Thu, May 5, 2011 at 5:01 PM, aaron morton aa...@thelastpickle.comwrote: There have been some recent discussions about different EC2 deployments, may be be exactly what you are looking for but try start here http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Re-IP-address-resolution-in-MultiDC-setup-EC2-VIP-td6306635.html http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Re-IP-address-resolution-in-MultiDC-setup-EC2-VIP-td6306635.html - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 6 May 2011, at 09:21, Sameer Farooqui wrote: Here is an image that shows what the Amazon VPC we're thinking about using looks like: http://i.imgur.com/OUe1i.png We would like to configure a 2 node Cassandra cluster in the private subnet and a read/write web application service in the public subnet. However, we also want to span the Cassandra cluster across from the Virginia VPC to a California VPC and have 2 Cassandra nodes in the private subnet in Virginia and 1 node in the private subnet in California. Amazon states that EC2 instances in Virginia’s VPC can communicate with EC2 instances in California’s VPC as long as all communication takes place over the internet gateway of each VPC and uses Elastic IP addresses as the specified source or destination address. But Amazon also says that “EIP addresses should only be used on instances in subnets configured to route their traffic directly to the Internet Gateway”. So, by this they must mean the public subnet in the diagram above. Amazon goes on to say “EIPs cannot be used on instances in subnets configured to use a NAT instance to access the Internet.” It looks like we cannot have a Cassandra cluster spanning two Private Subnets in two different regions. Can somebody with VPC experience confirm this? Also has anybody successfully deployed a Cassandra cluster in private subnets spanning multiple regions? Any tips from your experience? How did you find the latency over VPCs and gateways across regions? - Sameer Accenture Technology Labs
Adding a new node
Hi, I am trying to bring up a new node (with different IP) to replace a dead node on cassandra 0.7.5. Rather than bootstrap, I am copying the SSTable files to the new node(backed up files) as my data runs into several GB. Although the node successfully joins the ring, some of the ring nodes still seem to point to the old dead node as seen from ring command. Is there a way to notify all nodes about the new node? Am looking for options that can bring the cluster back to it original state in a faster and reliable manner since I do have all the SSTable files. One option I looked at was to remove all system table and restart the entire cluster. But I loose the schemas with this approach. Thanks in advance for your reply. VR
Re: CFHistograms?
Is there a more detailed document available on CF Histograms? On Sun, May 8, 2011 at 2:39 PM, Jonathan Ellis jbel...@gmail.com wrote: Timeouts (hence errors) will register at the rpc timeout value. On Sat, May 7, 2011 at 1:27 PM, Josep Blanquer blanq...@rightscale.com wrote: I believe the offset value of Writes and Reads are in *micro*seconds right? (that page talks about *milli*seconds) Also, are any timeouts or errors reflected in those times or just successful operations? if not, is there any JMX or other tool to keep track of them? Josep M. On Fri, May 6, 2011 at 9:09 AM, Jonathan Ellis jbel...@gmail.com wrote: Those are updated at compaction time. On Thu, May 5, 2011 at 11:38 PM, Xaero S xaeros...@gmail.com wrote: Can someone point me to a document that explains how to interpret CFHistograms output? i went through http://narendrasharma.blogspot.com/2011/04/cassandra-07x-understanding-output-of.html which is a good beginning, but was wondering if there was anything more detailed. For e.g when i run CFHistograms, i see rowsize and columncount items in the table always 0 (which cant be right?) -Xaero -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
datacenter ShardStrategy
Hi All, I want to use datacenter ShardStrategy in my cassandra setup .Can someone please let me know what steps / configuration changes I need to make. Thanks Anurag
Re: Migrating all rows from 0.6.13 to 0.7.5 over thrift?
Out of interest i've done some more digging. Not sure how much more I've
contributed but here goes...
Ran this against an clean v 0.6.12 and it works (I expected it to fail on the
first read)
client = pycassa.connect()
standard1 = pycassa.ColumnFamily(client, 'Keyspace1', 'Standard1')
uni_str = u数時間
uni_str = uni_str.encode(utf-8)
print Insert row, uni_str
print uni_str, standard1.insert(uni_str, {bar : baz})
print Read rows
print ???, standard1.get(???)
print uni_str, standard1.get(uni_str)
Ran that against the current 0.6 head from the command line and it works. Run
against the code running in intelli J and the code fails as expected. Code also
fails as expected on 0.7.5
At one stage I grabbed the buffer created by fastbinary.encode_binary in the
python generated batch_mutate_args.write() and it looked like the key was
correctly utf-8 encoded (matching bytes to the previous utf-8 encoding of that
string).
I've updated the git project https://github.com/amorton/cassandra-unicode-bug
Am going to leave it there unless there is interest to keep looking into it.
-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 8 May 2011, at 13:31, Jonathan Ellis wrote:
Right, that's sort of a half-repair: it will repair differences in
replies it got, but it won't doublecheck md5s on the rest in the
background. So if you're doing CL.ONE reads this is a no-op.
On Sat, May 7, 2011 at 4:25 PM, aaron morton aa...@thelastpickle.com wrote:
I remembered something like that so had a look at
RangeSliceResponseResolver.resolve() in 0.6.12 and it looks like it
schedules the repairs...
protected Row getReduced()
{
ColumnFamily resolved =
ReadResponseResolver.resolveSuperset(versions);
ReadResponseResolver.maybeScheduleRepairs(resolved, table,
key, versions, versionSources);
versions.clear();
versionSources.clear();
return new Row(key, resolved);
}
Is that right?
-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 8 May 2011, at 00:48, Jonathan Ellis wrote:
range_slices respects consistencylevel, but only single-row reads and
multiget do the *repair* part of RR.
On Sat, May 7, 2011 at 1:44 AM, aaron morton aa...@thelastpickle.com
wrote:
get_range_slices() does read repair if enabled (checked
DoConsistencyChecksBoolean in the config, it's on by default) so you
should be getting good reads. If you want belt-and-braces run nodetool
repair first.
Hope that helps.
On 7 May 2011, at 11:46, Jeremy Hanna wrote:
Great! I just wanted to make sure you were getting the information you
needed.
On May 6, 2011, at 6:42 PM, Henrik Schröder wrote:
Well, I already completed the migration program. Using get_range_slices
I could migrate a few thousand rows per second, which means that
migrating all of our data would take a few minutes, and we'll end up
with pristine datafiles for the new cluster. Problem solved!
I'll see if I can create datafiles in 0.6 that are uncleanable in 0.7 so
that you all can repeat this and hopefully fix it.
/Henrik Schröder
On Sat, May 7, 2011 at 00:35, Jeremy Hanna jeremy.hanna1...@gmail.com
wrote:
If you're able, go into the #cassandra channel on freenode (IRC) and
talk to driftx or jbellis or aaron_morton about your problem. It could
be that you don't have to do all of this based on a conversation there.
On May 6, 2011, at 5:04 AM, Henrik Schröder wrote:
I'll see if I can make some example broken files this weekend.
/Henrik Schröder
On Fri, May 6, 2011 at 02:10, aaron morton aa...@thelastpickle.com
wrote:
The difficulty is the different thrift clients between 0.6 and 0.7.
If you want to roll your own solution I would consider:
- write an app to talk to 0.6 and pull out the data using keys from the
other system (so you know can check referential integrity while you are
at it). Dump the data to flat file.
- write an app to talk to 0.7 to load the data back in.
I've not given up digging on your migration problem, having to manually
dump and reload if you've done nothing wrong is not the best solution.
I'll try to find some time this weekend to test with:
- 0.6 server, random paritioner, standard CF's, byte column
- load with python or the cli on osx or ubuntu (dont have a window
machine any more)
- migrate and see whats going on.
If you can spare some sample data to load please send it over in the
user group or my email address.
Cheers
-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 6 May 2011, at 05:52, Henrik Schröder wrote:
We can't do a straight upgrade from 0.6.13 to 0.7.5 because we have
rows stored that have unicode keys, and
Re: Adding a new node
It is possible to change IP address of a node, background http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/change-node-IP-address-td6197607.html If you have already bought a new node back with a different IP and the nodes in the cluster have different views of the ring (nodetool ring) you should see http://www.datastax.com/docs/0.7/troubleshooting/index#view-of-ring-differs-between-some-nodes What version are you on and what does nodetool ring say? Hope that helps. - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 9 May 2011, at 12:24, Venkat Rama wrote: Hi, I am trying to bring up a new node (with different IP) to replace a dead node on cassandra 0.7.5. Rather than bootstrap, I am copying the SSTable files to the new node(backed up files) as my data runs into several GB. Although the node successfully joins the ring, some of the ring nodes still seem to point to the old dead node as seen from ring command. Is there a way to notify all nodes about the new node? Am looking for options that can bring the cluster back to it original state in a faster and reliable manner since I do have all the SSTable files. One option I looked at was to remove all system table and restart the entire cluster. But I loose the schemas with this approach. Thanks in advance for your reply. VR
This is an easy way to help Lake Dolloff students
.and it won't cost you or tax-payers a dime. Go to: www.refresheverything.com/federal-way-kids-at-play and vote for Lake Dolloff to get new play equipment. It has been many years since I taught at Lake Dolloff (1987-1992), so I know when the new equipment was installed. Now that equipment is unsafe and needs to be replaced. This grant would go a long way to get new playground equipment for the kids to play on during recess. Let's give them a hand and take a moment to make an account at the website and vote.a Thank you. Marie-Anne Harkness maharkn...@comcast.net Peace It does not mean to be in a place where there is no noise, no trouble or hard work, it means to be in the midst of those things and still be calm in your heart. Unknown
