nodetool decommission unable to do its job successfully
I ran nodetool decommission on a node from a 4 nodes cluster as follows: nodetool -h 192.168.26.130 decommission But it never completes doesn't show any messages, on seeing the logs I noticed: java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecuto r.java:767) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecuto r.java:658) on viewing netstats on this node I can see the node in LEAVING mode mode: LEAVING Nothing streaming to /192.168.26.131 Nothing streaming to /192.168.26.129 Not receiving any streams.
How to reduce the memory consumed by cassandra (so as to prevent crashes OOMs) ?
I'm running a 4 nodes cassandra cluster of VMware ubuntu instances each 768MB memory (on a single machine for development purposes). I need to reduce heap size appropriately as my nodes have been crashing at times with OOMs. How do I configure for this ? I think I would need to make some tweaks with MAX_HEAP_SIZE HEAP_NEWSIZE in cassandra-env.sh !? but I not sure what should be the correct values I should put here for my case. What would have been the values for these parameters in case I had just 512mb for each node?
Only the last added node is visible in the cluster
While creating a multinode cluster, my nodes are unable to identify all the nodes in the cluster. Only the *last added' node is visible when I do: ./nodetool -h localhost ring I am trying to create a 4 nodes cluster. On starting the seed node, the above command shows just itself(ok.. good), then when I start the 2nd node the first one disappears there is only 2nd visible in the ring. On starting 3rd one, just the 3rd one remains. In cassandra.yaml of each node, I configured the listen_adress equal to ip address of that node for seeds I just put the ip address of 1st node everywhere. Can anyone point to me what may be causing this ?
Re: Only the last added node is visible in the cluster
yes, exactly I did cloned a single VMware machine to make other instances.. so how do I correct this now ? On Sun, Feb 26, 2012 at 2:18 AM, Brandon Williams dri...@gmail.com wrote: Then my next guess is you cloned one system to make the others in a virtual env, and the token is recorded in the system keyspace. In any case, some nodetool ring output at each node addition will clarify this. -Brandon On Sat, Feb 25, 2012 at 1:20 PM, Aditya Gupta ady...@gmail.com wrote: Nope, I just re-verified :) I have split up the range into 4 parts for 4 nodes. I have specified that in the intial_token On Sun, Feb 26, 2012 at 12:33 AM, Brandon Williams dri...@gmail.com wrote: My guess would be you're using the same token everywhere. -Brandon On Sat, Feb 25, 2012 at 9:48 AM, Aditya Gupta ady...@gmail.com wrote: While creating a multinode cluster, my nodes are unable to identify all the nodes in the cluster. Only the *last added' node is visible when I do: ./nodetool -h localhost ring I am trying to create a 4 nodes cluster. On starting the seed node, the above command shows just itself(ok.. good), then when I start the 2nd node the first one disappears there is only 2nd visible in the ring. On starting 3rd one, just the 3rd one remains. In cassandra.yaml of each node, I configured the listen_adress equal to ip address of that node for seeds I just put the ip address of 1st node everywhere. Can anyone point to me what may be causing this ?
Re: Only the last added node is visible in the cluster
The output of nodetool ring after each addition of nodes, make just the last added node visible in the ring. When I retry to add the node(which are not visible) it says that it is already a part of the ring. Could you indicate how should I rectify this now, as you seem to have figured out the issue ? On Sun, Feb 26, 2012 at 2:49 AM, Aditya Gupta ady...@gmail.com wrote: yes, exactly I did cloned a single VMware machine to make other instances.. so how do I correct this now ? On Sun, Feb 26, 2012 at 2:18 AM, Brandon Williams dri...@gmail.comwrote: Then my next guess is you cloned one system to make the others in a virtual env, and the token is recorded in the system keyspace. In any case, some nodetool ring output at each node addition will clarify this. -Brandon On Sat, Feb 25, 2012 at 1:20 PM, Aditya Gupta ady...@gmail.com wrote: Nope, I just re-verified :) I have split up the range into 4 parts for 4 nodes. I have specified that in the intial_token On Sun, Feb 26, 2012 at 12:33 AM, Brandon Williams dri...@gmail.com wrote: My guess would be you're using the same token everywhere. -Brandon On Sat, Feb 25, 2012 at 9:48 AM, Aditya Gupta ady...@gmail.com wrote: While creating a multinode cluster, my nodes are unable to identify all the nodes in the cluster. Only the *last added' node is visible when I do: ./nodetool -h localhost ring I am trying to create a 4 nodes cluster. On starting the seed node, the above command shows just itself(ok.. good), then when I start the 2nd node the first one disappears there is only 2nd visible in the ring. On starting 3rd one, just the 3rd one remains. In cassandra.yaml of each node, I configured the listen_adress equal to ip address of that node for seeds I just put the ip address of 1st node everywhere. Can anyone point to me what may be causing this ?
Re: Only the last added node is visible in the cluster
Thanks removing the system keyspace worked! Thanks! On Sun, Feb 26, 2012 at 3:39 AM, Brandon Williams dri...@gmail.com wrote: On Sat, Feb 25, 2012 at 3:39 PM, Aditya Gupta ady...@gmail.com wrote: The output of nodetool ring after each addition of nodes, make just the last added node visible in the ring. When I retry to add the node(which are not visible) it says that it is already a part of the ring. Could you indicate how should I rectify this now, as you seem to have figured out the issue ? The simplest thing to do is rm -rf /var/lib/cassandra on all the nodes. -Brandon
What linux distro for the Cassandra nodes ?
I am about to choose a linux distro to be installed on Cassandra nodes. Which are the most popular recommended ones by Cassandra community? (Not interested in paying licensing fees)
Re: What linux distro for the Cassandra nodes ?
@Edward: That's nice to know, but I also heard that there are some bugs in some recent Ubuntu version that bubble up with Cassandra specifically, so I was thinking I should probably go with what's more widely used by other Cassandra users for production needs. On Tue, Feb 21, 2012 at 8:57 PM, Edward Capriolo edlinuxg...@gmail.comwrote: distro=wallpaper. Pick the one you think looks best :) On Tue, Feb 21, 2012 at 8:14 AM, R. Verlangen ro...@us2.nl wrote: We use Centos 5.0 and every runs like a train :-) I actually think the OS will not really affect any Cassandra performance. However I read something about bugs with a certain Ubuntu version in combination with a specific Cassandra version. 2012/2/21 Rishabh Agrawal rishabh.agra...@impetus.co.in Hello Aditya, I am using Ubuntu 11.10 and it really good. Try it out. Regards Rishabh Agrawal Impetus’ Head of Innovation labs, Vineet Tyagi will be presenting on ‘Big Data Big Costs?’ at the Strata Conference, CA (Feb 28 - Mar 1) http://bit.ly/bSMWd7. Listen to our webcast ‘Hybrid Approach to Extend Web Apps to Tablets Smartphones’ available at http://bit.ly/yQC1oD. NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Re: Newbie Question: Cassandra consuming 100% CPU on ubuntu server
@Tamar: Thanks a ton! works perfect now! On Mon, Feb 20, 2012 at 12:02 PM, Tamar Fraenkel ta...@tok-media.comwrote: On 19/02/2012 21:55, Aditya Gupta wrote: Is there anything to do with running cassandra on a VMware ubuntu instance !? I am trying Cassandra on VMware ubuntu server instance. I am doing that for the past couple of weeks both on VMWare player and on ESXi in development and it works fine. My Ubuntu is 11.10, cassandra 1.0.7. I followed the directions in datastax site for installation. Tamar On Sun, Feb 19, 2012 at 2:45 AM, Aditya Gupta ady...@gmail.com wrote: In my case, after installing cassandra, as soon as the server is started it hangs on it own (totally unresponsive). It had openJDK-6's jdk jre implementations. On Sat, Feb 18, 2012 at 10:30 PM, Guy Incognito dnd1...@gmail.comwrote: perhaps entirely unrelated, but somebody was asking about lockups on EC2 yesterday and found: http://wiki.apache.org/cassandra/FAQ#ubuntu_hangs On 18/02/2012 14:58, Aditya Gupta wrote: Am I installing it the right way ? While installing I didn't verify the signatures using public key. On Sat, Feb 18, 2012 at 8:21 PM, Aditya Gupta ady...@gmail.com wrote: No data at all. just a fresh installation On Sat, Feb 18, 2012 at 6:57 PM, R. Verlangen ro...@us2.nl wrote: You might want to check your Cassandra logs, they contain important information that might lead you to the actual cause of the problems. 2012/2/18 Aditya Gupta ady...@gmail.com Thanks! But what about the 100% cpu consumption that is causing the server to hang? On Sat, Feb 18, 2012 at 6:19 PM, Watanabe Maki watanabe.m...@gmail.com wrote: I haven't use the packaged kit, but Cassandra uses half of physical memory on your system by default. You need to edit cassandra-env.sh to decrease heap size. Update MAX_HEAP_SIZE and NEW_HEAP_SIZE and restart. From iPhone On 2012/02/18, at 20:40, Aditya Gupta ady...@gmail.com wrote: I just installed Cassandra on my ubuntu server by adding the following to the sources list: deb http://www.apache.org/dist/cassandra/debian 10x main deb-src http://www.apache.org/dist/cassandra/debian 10x main Soon after install I started getting OOM errors then the server became unresponsive. I added more RAM to the server but found that cassandra was consuming 100% CPU 1GB RAM as soon the server was being started. Why is this happening how can get it to normal conditions ? -- Tamar Fraenkel Senior Software Engineer, TOK Media Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956
Re: Newbie Question: Cassandra consuming 100% CPU on ubuntu server
Is there anything to do with running cassandra on a VMware ubuntu instance !? I am trying Cassandra on VMware ubuntu server instance. On Sun, Feb 19, 2012 at 2:45 AM, Aditya Gupta ady...@gmail.com wrote: In my case, after installing cassandra, as soon as the server is started it hangs on it own (totally unresponsive). It had openJDK-6's jdk jre implementations. On Sat, Feb 18, 2012 at 10:30 PM, Guy Incognito dnd1...@gmail.com wrote: perhaps entirely unrelated, but somebody was asking about lockups on EC2 yesterday and found: http://wiki.apache.org/cassandra/FAQ#ubuntu_hangs On 18/02/2012 14:58, Aditya Gupta wrote: Am I installing it the right way ? While installing I didn't verify the signatures using public key. On Sat, Feb 18, 2012 at 8:21 PM, Aditya Gupta ady...@gmail.com wrote: No data at all. just a fresh installation On Sat, Feb 18, 2012 at 6:57 PM, R. Verlangen ro...@us2.nl wrote: You might want to check your Cassandra logs, they contain important information that might lead you to the actual cause of the problems. 2012/2/18 Aditya Gupta ady...@gmail.com Thanks! But what about the 100% cpu consumption that is causing the server to hang? On Sat, Feb 18, 2012 at 6:19 PM, Watanabe Maki watanabe.m...@gmail.com wrote: I haven't use the packaged kit, but Cassandra uses half of physical memory on your system by default. You need to edit cassandra-env.sh to decrease heap size. Update MAX_HEAP_SIZE and NEW_HEAP_SIZE and restart. From iPhone On 2012/02/18, at 20:40, Aditya Gupta ady...@gmail.com wrote: I just installed Cassandra on my ubuntu server by adding the following to the sources list: deb http://www.apache.org/dist/cassandra/debian 10x main deb-src http://www.apache.org/dist/cassandra/debian 10x main Soon after install I started getting OOM errors then the server became unresponsive. I added more RAM to the server but found that cassandra was consuming 100% CPU 1GB RAM as soon the server was being started. Why is this happening how can get it to normal conditions ?
Newbie Question: Cassandra consuming 100% CPU on ubuntu server
I just installed Cassandra on my ubuntu server by adding the following to the sources list: deb http://www.apache.org/dist/cassandra/debian 10x main deb-src http://www.apache.org/dist/cassandra/debian 10x main Soon after install I started getting OOM errors then the server became unresponsive. I added more RAM to the server but found that cassandra was consuming 100% CPU 1GB RAM as soon the server was being started. Why is this happening how can get it to normal conditions ?
Re: Newbie Question: Cassandra consuming 100% CPU on ubuntu server
Thanks! But what about the 100% cpu consumption that is causing the server to hang? On Sat, Feb 18, 2012 at 6:19 PM, Watanabe Maki watanabe.m...@gmail.comwrote: I haven't use the packaged kit, but Cassandra uses half of physical memory on your system by default. You need to edit cassandra-env.sh to decrease heap size. Update MAX_HEAP_SIZE and NEW_HEAP_SIZE and restart. From iPhone On 2012/02/18, at 20:40, Aditya Gupta ady...@gmail.com wrote: I just installed Cassandra on my ubuntu server by adding the following to the sources list: deb http://www.apache.org/dist/cassandra/debian 10x main deb-src http://www.apache.org/dist/cassandra/debian 10x main Soon after install I started getting OOM errors then the server became unresponsive. I added more RAM to the server but found that cassandra was consuming 100% CPU 1GB RAM as soon the server was being started. Why is this happening how can get it to normal conditions ?
Re: Newbie Question: Cassandra consuming 100% CPU on ubuntu server
For clarification, I'm running Cassandra on a VMware ubuntu server instance. On Sat, Feb 18, 2012 at 6:44 PM, Aditya Gupta ady...@gmail.com wrote: Thanks! But what about the 100% cpu consumption that is causing the server to hang? On Sat, Feb 18, 2012 at 6:19 PM, Watanabe Maki watanabe.m...@gmail.comwrote: I haven't use the packaged kit, but Cassandra uses half of physical memory on your system by default. You need to edit cassandra-env.sh to decrease heap size. Update MAX_HEAP_SIZE and NEW_HEAP_SIZE and restart. From iPhone On 2012/02/18, at 20:40, Aditya Gupta ady...@gmail.com wrote: I just installed Cassandra on my ubuntu server by adding the following to the sources list: deb http://www.apache.org/dist/cassandra/debian 10x main deb-src http://www.apache.org/dist/cassandra/debian 10x main Soon after install I started getting OOM errors then the server became unresponsive. I added more RAM to the server but found that cassandra was consuming 100% CPU 1GB RAM as soon the server was being started. Why is this happening how can get it to normal conditions ?
Re: Newbie Question: Cassandra consuming 100% CPU on ubuntu server
No data at all. just a fresh installation On Sat, Feb 18, 2012 at 6:57 PM, R. Verlangen ro...@us2.nl wrote: You might want to check your Cassandra logs, they contain important information that might lead you to the actual cause of the problems. 2012/2/18 Aditya Gupta ady...@gmail.com Thanks! But what about the 100% cpu consumption that is causing the server to hang? On Sat, Feb 18, 2012 at 6:19 PM, Watanabe Maki watanabe.m...@gmail.comwrote: I haven't use the packaged kit, but Cassandra uses half of physical memory on your system by default. You need to edit cassandra-env.sh to decrease heap size. Update MAX_HEAP_SIZE and NEW_HEAP_SIZE and restart. From iPhone On 2012/02/18, at 20:40, Aditya Gupta ady...@gmail.com wrote: I just installed Cassandra on my ubuntu server by adding the following to the sources list: deb http://www.apache.org/dist/cassandra/debian 10x main deb-src http://www.apache.org/dist/cassandra/debian 10x main Soon after install I started getting OOM errors then the server became unresponsive. I added more RAM to the server but found that cassandra was consuming 100% CPU 1GB RAM as soon the server was being started. Why is this happening how can get it to normal conditions ?
Re: Newbie Question: Cassandra consuming 100% CPU on ubuntu server
Am I installing it the right way ? While installing I didn't verify the signatures using public key. On Sat, Feb 18, 2012 at 8:21 PM, Aditya Gupta ady...@gmail.com wrote: No data at all. just a fresh installation On Sat, Feb 18, 2012 at 6:57 PM, R. Verlangen ro...@us2.nl wrote: You might want to check your Cassandra logs, they contain important information that might lead you to the actual cause of the problems. 2012/2/18 Aditya Gupta ady...@gmail.com Thanks! But what about the 100% cpu consumption that is causing the server to hang? On Sat, Feb 18, 2012 at 6:19 PM, Watanabe Maki watanabe.m...@gmail.comwrote: I haven't use the packaged kit, but Cassandra uses half of physical memory on your system by default. You need to edit cassandra-env.sh to decrease heap size. Update MAX_HEAP_SIZE and NEW_HEAP_SIZE and restart. From iPhone On 2012/02/18, at 20:40, Aditya Gupta ady...@gmail.com wrote: I just installed Cassandra on my ubuntu server by adding the following to the sources list: deb http://www.apache.org/dist/cassandra/debian 10x main deb-src http://www.apache.org/dist/cassandra/debian 10x main Soon after install I started getting OOM errors then the server became unresponsive. I added more RAM to the server but found that cassandra was consuming 100% CPU 1GB RAM as soon the server was being started. Why is this happening how can get it to normal conditions ?
Re: Newbie Question: Cassandra consuming 100% CPU on ubuntu server
In my case, after installing cassandra, as soon as the server is started it hangs on it own (totally unresponsive). It had openJDK-6's jdk jre implementations. On Sat, Feb 18, 2012 at 10:30 PM, Guy Incognito dnd1...@gmail.com wrote: perhaps entirely unrelated, but somebody was asking about lockups on EC2 yesterday and found: http://wiki.apache.org/cassandra/FAQ#ubuntu_hangs On 18/02/2012 14:58, Aditya Gupta wrote: Am I installing it the right way ? While installing I didn't verify the signatures using public key. On Sat, Feb 18, 2012 at 8:21 PM, Aditya Gupta ady...@gmail.com wrote: No data at all. just a fresh installation On Sat, Feb 18, 2012 at 6:57 PM, R. Verlangen ro...@us2.nl wrote: You might want to check your Cassandra logs, they contain important information that might lead you to the actual cause of the problems. 2012/2/18 Aditya Gupta ady...@gmail.com Thanks! But what about the 100% cpu consumption that is causing the server to hang? On Sat, Feb 18, 2012 at 6:19 PM, Watanabe Maki watanabe.m...@gmail.com wrote: I haven't use the packaged kit, but Cassandra uses half of physical memory on your system by default. You need to edit cassandra-env.sh to decrease heap size. Update MAX_HEAP_SIZE and NEW_HEAP_SIZE and restart. From iPhone On 2012/02/18, at 20:40, Aditya Gupta ady...@gmail.com wrote: I just installed Cassandra on my ubuntu server by adding the following to the sources list: deb http://www.apache.org/dist/cassandra/debian 10x main deb-src http://www.apache.org/dist/cassandra/debian 10x main Soon after install I started getting OOM errors then the server became unresponsive. I added more RAM to the server but found that cassandra was consuming 100% CPU 1GB RAM as soon the server was being started. Why is this happening how can get it to normal conditions ?
