Re: Any tips on how to track down why Cassandra won't cluster?

[Richard L. Burton III]

Thanks everyone. The issue was a missing firewall entry in the security
groups that prevented Cassandra from clustering.

So Alain, initially I was using the private IP. I went to irc and someone
had mentioned to use the public IP, although I wasnt doing multi region
clustering.

I should at some point write a santy tool that checks all the configuration
options, validates ports, etc to help automate the validation process.

Cheers,

On Thu, Feb 4, 2016 at 6:37 PM, Alain RODRIGUEZ  wrote:

> Hi Richard,
>
> I think you just can't use EC2Snitch with public IPs.
>
> See
> https://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architectureSnitchEC2_t.html
>
> Precisely "Because private IPs are used, this snitch does not work across
> multiple regions"
>
> 54.*.*.* looks like a public one.
>
> You can stick with the private IPs (with limitation written above, even if
> you can workaround with a VPN tunnel across Regions). In this case set
> listen address to private IP and comment broadcast_address. You can also
> use the EC2MultiRegionSnitch, but then be careful with broadcast_address
> (public IP) and listen-address (private IP) configuration on the
> cassandra.yaml files and also with ports management on AWS console.
>
> Also, as you nodes already bootstrapped, you might have to clean the
> cassandra folder, usually something like rm -rf /var/lib/cassandra/*
> *warning: *you will loose all the data, but this "cluster" doesn't look
> like a running cluster, only you can know :-).
>
> Any suggestions on how to track down what might trigger this problem
>
>
> This kind of issue might be due to:
>
> - Different cluster names
> - *Bad configuration* (IPs, Snitch + configuration files, ...) <--
> probably your case
> - Ports (firewall, AWS rules...) <-- telnet might be useful here
> - Seeds being differents on the nodes <-- make sure that your seeds are
> the same on every node
>
> Hope this will be enough to get you out of this,
>
> C*heers,
> -
> Alain Rodriguez
> France
>
> The Last Pickle
> http://www.thelastpickle.com
>
>
>
> 2016-02-04 16:35 GMT+00:00 Victor Chen :
>
>> Along the lines of what Ben and Bryan suggested, what are you using to
>> verify ports are open? If you do something like:
>>
>> node1$ nc -zv node2 9042
>> node2$ nc -zv node1 9042
>>
>> does it succeed from both nodes?
>> Does the first node 'know' that it is a seed? i.e. do you have first node
>> listed in its own seed's list?
>> What does the system.log show as both nodes are spun up?
>>
>>
>> On Wed, Feb 3, 2016 at 7:20 PM, Bryan Cheng 
>> wrote:
>>
>>>
>>>
>>>
 On Wed, 3 Feb 2016 at 11:49 Richard L. Burton III 
 wrote:

>
> Any suggestions on how to track down what might trigger this problem?
> I'm not receiving any exceptions.
>

>>> You're not getting "Unable to gossip with any seeds" on the second node?
>>> What does nodetool status show on both machines?
>>>
>>
>>
>


-- 
-Richard L. Burton III
@rburton

Re: Any tips on how to track down why Cassandra won't cluster?

[Alain RODRIGUEZ]

Hi Richard,

I think you just can't use EC2Snitch with public IPs.

See
https://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architectureSnitchEC2_t.html

Precisely "Because private IPs are used, this snitch does not work across
multiple regions"

54.*.*.* looks like a public one.

You can stick with the private IPs (with limitation written above, even if
you can workaround with a VPN tunnel across Regions). In this case set
listen address to private IP and comment broadcast_address. You can also
use the EC2MultiRegionSnitch, but then be careful with broadcast_address
(public IP) and listen-address (private IP) configuration on the
cassandra.yaml files and also with ports management on AWS console.

Also, as you nodes already bootstrapped, you might have to clean the
cassandra folder, usually something like rm -rf /var/lib/cassandra/*
*warning: *you will loose all the data, but this "cluster" doesn't look
like a running cluster, only you can know :-).

Any suggestions on how to track down what might trigger this problem


This kind of issue might be due to:

- Different cluster names
- *Bad configuration* (IPs, Snitch + configuration files, ...) <-- probably
your case
- Ports (firewall, AWS rules...) <-- telnet might be useful here
- Seeds being differents on the nodes <-- make sure that your seeds are the
same on every node

Hope this will be enough to get you out of this,

C*heers,
-
Alain Rodriguez
France

The Last Pickle
http://www.thelastpickle.com



2016-02-04 16:35 GMT+00:00 Victor Chen :

> Along the lines of what Ben and Bryan suggested, what are you using to
> verify ports are open? If you do something like:
>
> node1$ nc -zv node2 9042
> node2$ nc -zv node1 9042
>
> does it succeed from both nodes?
> Does the first node 'know' that it is a seed? i.e. do you have first node
> listed in its own seed's list?
> What does the system.log show as both nodes are spun up?
>
>
> On Wed, Feb 3, 2016 at 7:20 PM, Bryan Cheng  wrote:
>
>>
>>
>>
>>> On Wed, 3 Feb 2016 at 11:49 Richard L. Burton III 
>>> wrote:
>>>

 Any suggestions on how to track down what might trigger this problem?
 I'm not receiving any exceptions.

>>>
>> You're not getting "Unable to gossip with any seeds" on the second node?
>> What does nodetool status show on both machines?
>>
>
>

Re: Any tips on how to track down why Cassandra won't cluster?

[Victor Chen]

Along the lines of what Ben and Bryan suggested, what are you using to
verify ports are open? If you do something like:

node1$ nc -zv node2 9042
node2$ nc -zv node1 9042

does it succeed from both nodes?
Does the first node 'know' that it is a seed? i.e. do you have first node
listed in its own seed's list?
What does the system.log show as both nodes are spun up?


On Wed, Feb 3, 2016 at 7:20 PM, Bryan Cheng  wrote:

>
>
>
>> On Wed, 3 Feb 2016 at 11:49 Richard L. Burton III 
>> wrote:
>>
>>>
>>> Any suggestions on how to track down what might trigger this problem?
>>> I'm not receiving any exceptions.
>>>
>>
> You're not getting "Unable to gossip with any seeds" on the second node?
> What does nodetool status show on both machines?
>

Re: Any tips on how to track down why Cassandra won't cluster?

[Bryan Cheng]

> On Wed, 3 Feb 2016 at 11:49 Richard L. Burton III 
> wrote:
>
>>
>> Any suggestions on how to track down what might trigger this problem? I'm
>> not receiving any exceptions.
>>
>
You're not getting "Unable to gossip with any seeds" on the second node?
What does nodetool status show on both machines?

Any tips on how to track down why Cassandra won't cluster?

[Richard L. Burton III]

I'm deploying 2 nodes at the moment using cassandra-dse on Amazon. I
configured it to use EC2Snitch and configured rackdc to use us-east with
rack "1".

The second node points to the first node as the seed e.g., "seeds":
["54.*.*.*"] and all of the ports are open.

Any suggestions on how to track down what might trigger this problem? I'm
not receiving any exceptions.

-- 
-Richard L. Burton III
@rburton

Re: Any tips on how to track down why Cassandra won't cluster?

[Ben Bromhead]

Check network connectivity. If you are using public addresses as the
broadcast, make sure you can telnet from one node to the other nodes public
address using the internode port.

Last time I looked into something like this, for some reason if you only
add a security group id to the allowed traffic in a security group you
still need to add public IP addresses for each node in a security groups
allowed inbound traffic as well.

On Wed, 3 Feb 2016 at 11:49 Richard L. Burton III 
wrote:

> I'm deploying 2 nodes at the moment using cassandra-dse on Amazon. I
> configured it to use EC2Snitch and configured rackdc to use us-east with
> rack "1".
>
> The second node points to the first node as the seed e.g., "seeds":
> ["54.*.*.*"] and all of the ports are open.
>
> Any suggestions on how to track down what might trigger this problem? I'm
> not receiving any exceptions.
>
>
> --
> -Richard L. Burton III
> @rburton
>
-- 
Ben Bromhead
CTO | Instaclustr
+1 650 284 9692