Re: Can't Add AWS Node due to /mnt/cassandra/data directory
Awesome, thanks Ben! On Wed, Aug 27, 2014 at 9:14 PM, Ben Bromhead b...@instaclustr.com wrote: Make sure you have also setup the ephemeral drives as a raid device (use mdadm) and mounted it under /mnt/cassandra otherwise your data dir is the os partition which is usually very small. Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr http://twitter.com/instaclustr | +61 415 936 359 On 27 Aug 2014, at 8:21 pm, Stephen Portanova sport...@gmail.com wrote: Worked great! Thanks Mark! On Wed, Aug 27, 2014 at 2:00 AM, Mark Reddy mark.l.re...@gmail.com wrote: Hi stephen, I have never added a node via OpsCenter, so this may be a short coming of that process. However in non OpsCenter installs you would have to create the data directories first: sudo mkdir -p /mnt/cassandra/commitlog sudo mkdir -p /mnt/cassandra/data sudo mkdir -p /mnt/cassandra/saved_caches And then give the cassandra user ownership of those directories: sudo chown -R cassandra:cassandra /mnt/cassandra Once this is done Cassandra will have the correct directories and permission to start up. Mark On 27 August 2014 09:50, Stephen Portanova sport...@gmail.com wrote: I already have a 3node m3.large DSE cluster, but I can't seem to add another m3.large node. I'm using the ubuntu-trusty-14.04-amd64-server-20140607.1 (ami-a7fdfee2) AMI (instance-store backed, PV) on AWS, I install java 7 and the JNA, then I go into opscenter to add a node. Things look good for 3 or 4 green circles, until I either get this error: Start Errored: Timed out waiting for Cassandra to start. or this error: Agent Connection Errored: Timed out waiting for agent to connect. I check the system.log and output.log, and they both say: INFO [main] 2014-08-27 08:17:24,642 CLibrary.java (line 121) JNA mlockall successful ERROR [main] 2014-08-27 08:17:24,644 CassandraDaemon.java (line 235) *Directory /mnt/cassandra/data doesn't exist* *ERROR [main] 2014-08-27 08:17:24,645 CassandraDaemon.java (line 239) Has no permission to create /mnt/cassandra/data directory* INFO [Thread-1] 2014-08-27 08:17:24,646 DseDaemon.java (line 477) DSE shutting down... ERROR [Thread-1] 2014-08-27 08:17:24,725 CassandraDaemon.java (line 199) Exception in thread Thread[Thread-1,5,main] java.lang.AssertionError at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1263) at com.datastax.bdp.gms.DseState.setActiveStatus(DseState.java:171) at com.datastax.bdp.server.DseDaemon.stop(DseDaemon.java:478) at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:384) My agent.log file says: Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:40:57,848 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:17,849 Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:41:17,849 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:37,849 Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:41:37,850 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:57,850 Node is still provisioning, not attempting to determine ip. I feel like I'm missing something easy with the mount, so if you could point me in the right direction, I would really appreciate it! -- Stephen Portanova (480) 495-2634 -- Stephen Portanova (480) 495-2634 -- Stephen Portanova (480) 495-2634
Can't Add AWS Node due to /mnt/cassandra/data directory
I already have a 3node m3.large DSE cluster, but I can't seem to add another m3.large node. I'm using the ubuntu-trusty-14.04-amd64-server-20140607.1 (ami-a7fdfee2) AMI (instance-store backed, PV) on AWS, I install java 7 and the JNA, then I go into opscenter to add a node. Things look good for 3 or 4 green circles, until I either get this error: Start Errored: Timed out waiting for Cassandra to start. or this error: Agent Connection Errored: Timed out waiting for agent to connect. I check the system.log and output.log, and they both say: INFO [main] 2014-08-27 08:17:24,642 CLibrary.java (line 121) JNA mlockall successful ERROR [main] 2014-08-27 08:17:24,644 CassandraDaemon.java (line 235) *Directory /mnt/cassandra/data doesn't exist* *ERROR [main] 2014-08-27 08:17:24,645 CassandraDaemon.java (line 239) Has no permission to create /mnt/cassandra/data directory* INFO [Thread-1] 2014-08-27 08:17:24,646 DseDaemon.java (line 477) DSE shutting down... ERROR [Thread-1] 2014-08-27 08:17:24,725 CassandraDaemon.java (line 199) Exception in thread Thread[Thread-1,5,main] java.lang.AssertionError at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1263) at com.datastax.bdp.gms.DseState.setActiveStatus(DseState.java:171) at com.datastax.bdp.server.DseDaemon.stop(DseDaemon.java:478) at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:384) My agent.log file says: Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:40:57,848 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:17,849 Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:41:17,849 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:37,849 Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:41:37,850 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:57,850 Node is still provisioning, not attempting to determine ip. I feel like I'm missing something easy with the mount, so if you could point me in the right direction, I would really appreciate it! -- Stephen Portanova (480) 495-2634
Re: Can't Add AWS Node due to /mnt/cassandra/data directory
Hi stephen, I have never added a node via OpsCenter, so this may be a short coming of that process. However in non OpsCenter installs you would have to create the data directories first: sudo mkdir -p /mnt/cassandra/commitlog sudo mkdir -p /mnt/cassandra/data sudo mkdir -p /mnt/cassandra/saved_caches And then give the cassandra user ownership of those directories: sudo chown -R cassandra:cassandra /mnt/cassandra Once this is done Cassandra will have the correct directories and permission to start up. Mark On 27 August 2014 09:50, Stephen Portanova sport...@gmail.com wrote: I already have a 3node m3.large DSE cluster, but I can't seem to add another m3.large node. I'm using the ubuntu-trusty-14.04-amd64-server-20140607.1 (ami-a7fdfee2) AMI (instance-store backed, PV) on AWS, I install java 7 and the JNA, then I go into opscenter to add a node. Things look good for 3 or 4 green circles, until I either get this error: Start Errored: Timed out waiting for Cassandra to start. or this error: Agent Connection Errored: Timed out waiting for agent to connect. I check the system.log and output.log, and they both say: INFO [main] 2014-08-27 08:17:24,642 CLibrary.java (line 121) JNA mlockall successful ERROR [main] 2014-08-27 08:17:24,644 CassandraDaemon.java (line 235) *Directory /mnt/cassandra/data doesn't exist* *ERROR [main] 2014-08-27 08:17:24,645 CassandraDaemon.java (line 239) Has no permission to create /mnt/cassandra/data directory* INFO [Thread-1] 2014-08-27 08:17:24,646 DseDaemon.java (line 477) DSE shutting down... ERROR [Thread-1] 2014-08-27 08:17:24,725 CassandraDaemon.java (line 199) Exception in thread Thread[Thread-1,5,main] java.lang.AssertionError at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1263) at com.datastax.bdp.gms.DseState.setActiveStatus(DseState.java:171) at com.datastax.bdp.server.DseDaemon.stop(DseDaemon.java:478) at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:384) My agent.log file says: Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:40:57,848 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:17,849 Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:41:17,849 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:37,849 Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:41:37,850 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:57,850 Node is still provisioning, not attempting to determine ip. I feel like I'm missing something easy with the mount, so if you could point me in the right direction, I would really appreciate it! -- Stephen Portanova (480) 495-2634
Re: Can't Add AWS Node due to /mnt/cassandra/data directory
Worked great! Thanks Mark! On Wed, Aug 27, 2014 at 2:00 AM, Mark Reddy mark.l.re...@gmail.com wrote: Hi stephen, I have never added a node via OpsCenter, so this may be a short coming of that process. However in non OpsCenter installs you would have to create the data directories first: sudo mkdir -p /mnt/cassandra/commitlog sudo mkdir -p /mnt/cassandra/data sudo mkdir -p /mnt/cassandra/saved_caches And then give the cassandra user ownership of those directories: sudo chown -R cassandra:cassandra /mnt/cassandra Once this is done Cassandra will have the correct directories and permission to start up. Mark On 27 August 2014 09:50, Stephen Portanova sport...@gmail.com wrote: I already have a 3node m3.large DSE cluster, but I can't seem to add another m3.large node. I'm using the ubuntu-trusty-14.04-amd64-server-20140607.1 (ami-a7fdfee2) AMI (instance-store backed, PV) on AWS, I install java 7 and the JNA, then I go into opscenter to add a node. Things look good for 3 or 4 green circles, until I either get this error: Start Errored: Timed out waiting for Cassandra to start. or this error: Agent Connection Errored: Timed out waiting for agent to connect. I check the system.log and output.log, and they both say: INFO [main] 2014-08-27 08:17:24,642 CLibrary.java (line 121) JNA mlockall successful ERROR [main] 2014-08-27 08:17:24,644 CassandraDaemon.java (line 235) *Directory /mnt/cassandra/data doesn't exist* *ERROR [main] 2014-08-27 08:17:24,645 CassandraDaemon.java (line 239) Has no permission to create /mnt/cassandra/data directory* INFO [Thread-1] 2014-08-27 08:17:24,646 DseDaemon.java (line 477) DSE shutting down... ERROR [Thread-1] 2014-08-27 08:17:24,725 CassandraDaemon.java (line 199) Exception in thread Thread[Thread-1,5,main] java.lang.AssertionError at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1263) at com.datastax.bdp.gms.DseState.setActiveStatus(DseState.java:171) at com.datastax.bdp.server.DseDaemon.stop(DseDaemon.java:478) at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:384) My agent.log file says: Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:40:57,848 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:17,849 Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:41:17,849 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:37,849 Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:41:37,850 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:57,850 Node is still provisioning, not attempting to determine ip. I feel like I'm missing something easy with the mount, so if you could point me in the right direction, I would really appreciate it! -- Stephen Portanova (480) 495-2634 -- Stephen Portanova (480) 495-2634
Re: Can't Add AWS Node due to /mnt/cassandra/data directory
Make sure you have also setup the ephemeral drives as a raid device (use mdadm) and mounted it under /mnt/cassandra otherwise your data dir is the os partition which is usually very small. Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359 On 27 Aug 2014, at 8:21 pm, Stephen Portanova sport...@gmail.com wrote: Worked great! Thanks Mark! On Wed, Aug 27, 2014 at 2:00 AM, Mark Reddy mark.l.re...@gmail.com wrote: Hi stephen, I have never added a node via OpsCenter, so this may be a short coming of that process. However in non OpsCenter installs you would have to create the data directories first: sudo mkdir -p /mnt/cassandra/commitlog sudo mkdir -p /mnt/cassandra/data sudo mkdir -p /mnt/cassandra/saved_caches And then give the cassandra user ownership of those directories: sudo chown -R cassandra:cassandra /mnt/cassandra Once this is done Cassandra will have the correct directories and permission to start up. Mark On 27 August 2014 09:50, Stephen Portanova sport...@gmail.com wrote: I already have a 3node m3.large DSE cluster, but I can't seem to add another m3.large node. I'm using the ubuntu-trusty-14.04-amd64-server-20140607.1 (ami-a7fdfee2) AMI (instance-store backed, PV) on AWS, I install java 7 and the JNA, then I go into opscenter to add a node. Things look good for 3 or 4 green circles, until I either get this error: Start Errored: Timed out waiting for Cassandra to start. or this error: Agent Connection Errored: Timed out waiting for agent to connect. I check the system.log and output.log, and they both say: INFO [main] 2014-08-27 08:17:24,642 CLibrary.java (line 121) JNA mlockall successful ERROR [main] 2014-08-27 08:17:24,644 CassandraDaemon.java (line 235) Directory /mnt/cassandra/data doesn't exist ERROR [main] 2014-08-27 08:17:24,645 CassandraDaemon.java (line 239) Has no permission to create /mnt/cassandra/data directory INFO [Thread-1] 2014-08-27 08:17:24,646 DseDaemon.java (line 477) DSE shutting down... ERROR [Thread-1] 2014-08-27 08:17:24,725 CassandraDaemon.java (line 199) Exception in thread Thread[Thread-1,5,main] java.lang.AssertionError at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1263) at com.datastax.bdp.gms.DseState.setActiveStatus(DseState.java:171) at com.datastax.bdp.server.DseDaemon.stop(DseDaemon.java:478) at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:384) My agent.log file says: Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:40:57,848 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:17,849 Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:41:17,849 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:37,849 Node is still provisioning, not attempting to determine ip. INFO [Initialization] 2014-08-27 08:41:37,850 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2014-08-27 08:41:57,850 Node is still provisioning, not attempting to determine ip. I feel like I'm missing something easy with the mount, so if you could point me in the right direction, I would really appreciate it! -- Stephen Portanova (480) 495-2634 -- Stephen Portanova (480) 495-2634
