[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[
https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039420#comment-16039420
]
ASF GitHub Bot commented on CASSANDRA-9748:
---
Github user aweisberg commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/111#discussion_r120443366
--- Diff: src/java/org/apache/cassandra/net/async/NettyFactory.java ---
@@ -125,10 +126,19 @@
* Determine the number of accept threads we need, which is based upon
the number of listening sockets we will have.
* We'll have either 1 or 2 listen sockets, depending on if we use SSL
or not in combination with non-SSL. If we have both,
* we'll have two sockets, and thus need two threads; else one socket
and one thread.
+ *
+ * If the operator has configured multiple IP addresses (both {@link
org.apache.cassandra.config.Config#broadcast_address}
+ * and {@link org.apache.cassandra.config.Config#listen_address} are
configured), then we listen on another set of sockets
+ * - basically doubling the count. See CASSANDRA-9748 for more details.
*/
static int determineAcceptGroupSize(InternodeEncryption
internode_encryption)
{
-return internode_encryption == InternodeEncryption.dc ||
internode_encryption == InternodeEncryption.rack ? 2 : 1;
+int listenSocketCount = internode_encryption ==
InternodeEncryption.dc || internode_encryption == InternodeEncryption.rack ? 2
: 1;
+
+if (MessagingService.shouldListenOnBroadcastAddress())
+listenSocketCount *= 2;
--- End diff --
Why do you need more threads to listen on more sockets? It's a non-blocking
operation and more sockets doesn't indicate a need for more throughput?
> Can't see other nodes when using multiple network interfaces
>
>
> Key: CASSANDRA-9748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9748
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
> Environment: Cassandra 2.0.16; multi-DC configuration
>Reporter: Roman Bielik
>Assignee: Paulo Motta
>Priority: Minor
> Labels: docs-impacting
> Fix For: 2.2.5, 3.0.3, 3.2
>
> Attachments: system_node1.log, system_node2.log
>
>
> The idea is to setup a multi-DC environment across 2 different networks based
> on the following configuration recommendations:
> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html
> Each node has 2 network interfaces. One used as a private network (DC1:
> 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all
> nodes can see each other (this one has a higher latency).
> Using the following settings in cassandra.yaml:
> *seeds:* public IP (same as used in broadcast_address)
> *listen_address:* private IP
> *broadcast_address:* public IP
> *rpc_address:* 0.0.0.0
> *endpoint_snitch:* GossipingPropertyFileSnitch
> _(tried different combinations with no luck)_
> No firewall and no SSL/encryption used.
> The problem is that nodes do not see each other (a gossip problem I guess).
> The nodetool ring/status shows only the local node but not the other ones
> (even from the same DC).
> When I set listen_address to public IP, then everything works fine, but that
> is not the required configuration.
> _Note: Not using EC2 cloud!_
> netstat -anp | grep -E "(7199|9160|9042|7000)"
> tcp0 0 0.0.0.0:71990.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9160 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9042 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:7000 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 127.0.0.1:7199 127.0.0.1:52874
> ESTABLISHED 3587/java
> tcp0 0 10.0.1.1:7199 10.0.1.1:39650
> ESTABLISHED 3587/java
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046773#comment-15046773 ] Paulo Motta commented on CASSANDRA-9748: Tests look good, can we mark as ready to commit? > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta >Priority: Minor > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045210#comment-15045210 ] Ariel Weisberg commented on CASSANDRA-9748: --- +1 to the logging changes. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta >Priority: Minor > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041871#comment-15041871 ] Ariel Weisberg commented on CASSANDRA-9748: --- I am +1 on this. One nice to have that occurred to me would be to log the interfaces we end up binding on. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta >Priority: Minor > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[
https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041884#comment-15041884
]
Paulo Motta commented on CASSANDRA-9748:
bq. One nice to have that occurred to me would be to log the interfaces we end
up binding on.
we could do that, but why exactly? since the {{listen_address}} and
{{broadcast_address}} are explicitly defined, the admin should know which
interfaces are they bound to. We would probably need to do additional work to
fetch this, so I'd prefer to keep it as is, unless there's a good reason for it.
> Can't see other nodes when using multiple network interfaces
>
>
> Key: CASSANDRA-9748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9748
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
> Environment: Cassandra 2.0.16; multi-DC configuration
>Reporter: Roman Bielik
>Assignee: Paulo Motta
>Priority: Minor
> Labels: docs-impacting
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
> Attachments: system_node1.log, system_node2.log
>
>
> The idea is to setup a multi-DC environment across 2 different networks based
> on the following configuration recommendations:
> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html
> Each node has 2 network interfaces. One used as a private network (DC1:
> 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all
> nodes can see each other (this one has a higher latency).
> Using the following settings in cassandra.yaml:
> *seeds:* public IP (same as used in broadcast_address)
> *listen_address:* private IP
> *broadcast_address:* public IP
> *rpc_address:* 0.0.0.0
> *endpoint_snitch:* GossipingPropertyFileSnitch
> _(tried different combinations with no luck)_
> No firewall and no SSL/encryption used.
> The problem is that nodes do not see each other (a gossip problem I guess).
> The nodetool ring/status shows only the local node but not the other ones
> (even from the same DC).
> When I set listen_address to public IP, then everything works fine, but that
> is not the required configuration.
> _Note: Not using EC2 cloud!_
> netstat -anp | grep -E "(7199|9160|9042|7000)"
> tcp0 0 0.0.0.0:71990.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9160 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9042 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:7000 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 127.0.0.1:7199 127.0.0.1:52874
> ESTABLISHED 3587/java
> tcp0 0 10.0.1.1:7199 10.0.1.1:39650
> ESTABLISHED 3587/java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041890#comment-15041890 ] Ariel Weisberg commented on CASSANDRA-9748: --- If they specify the interface via a hostname it could end up binding to an IP other than what you intended. Also it saves you having to open up another file. For instance if someone uploads their log but not the yaml you won't have to go ask for the yaml. In that vein I've also implemented dumping the config into log files at the beginning of each one so when someone submits a log it always has the config in it even when the log file rolls. I'll think about suggesting that separately. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta >Priority: Minor > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[
https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042066#comment-15042066
]
Paulo Motta commented on CASSANDRA-9748:
Updated patch printing interface it binds to. In case it's not possible to
determine the interface, it doesn't print as before.
With the following yaml configuration:
{noformat}
listen_address: 127.0.0.1
broadcast_address: 192.168.1.116
listen_on_broadcast_address: true
{noformat}
It prints:
{noformat}
INFO [main] 2015-12-04 11:09:45,337 MessagingService.java:518 - Starting
Messaging Service on /127.0.0.1:7000 (lo)
INFO [main] 2015-12-04 11:09:45,339 MessagingService.java:518 - Starting
Messaging Service on /192.168.1.116:7000 (wlan0)
{noformat}
Resubmitted tests to make sure everything is fine with CI.
> Can't see other nodes when using multiple network interfaces
>
>
> Key: CASSANDRA-9748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9748
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
> Environment: Cassandra 2.0.16; multi-DC configuration
>Reporter: Roman Bielik
>Assignee: Paulo Motta
>Priority: Minor
> Labels: docs-impacting
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
> Attachments: system_node1.log, system_node2.log
>
>
> The idea is to setup a multi-DC environment across 2 different networks based
> on the following configuration recommendations:
> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html
> Each node has 2 network interfaces. One used as a private network (DC1:
> 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all
> nodes can see each other (this one has a higher latency).
> Using the following settings in cassandra.yaml:
> *seeds:* public IP (same as used in broadcast_address)
> *listen_address:* private IP
> *broadcast_address:* public IP
> *rpc_address:* 0.0.0.0
> *endpoint_snitch:* GossipingPropertyFileSnitch
> _(tried different combinations with no luck)_
> No firewall and no SSL/encryption used.
> The problem is that nodes do not see each other (a gossip problem I guess).
> The nodetool ring/status shows only the local node but not the other ones
> (even from the same DC).
> When I set listen_address to public IP, then everything works fine, but that
> is not the required configuration.
> _Note: Not using EC2 cloud!_
> netstat -anp | grep -E "(7199|9160|9042|7000)"
> tcp0 0 0.0.0.0:71990.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9160 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9042 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:7000 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 127.0.0.1:7199 127.0.0.1:52874
> ESTABLISHED 3587/java
> tcp0 0 10.0.1.1:7199 10.0.1.1:39650
> ESTABLISHED 3587/java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037592#comment-15037592 ] Roman Bielik commented on CASSANDRA-9748: - Thank you very much, the fix seems to be working fine. Tested on C2.2. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta >Priority: Minor > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036022#comment-15036022 ] Paulo Motta commented on CASSANDRA-9748: This is a 2.2 build, did you intend to compile the 2.2 version? It seems to compile fine here. What's the exact error? Did you try "ant clean jar" ? You could also try removing your "build" directory before "ant clean jar". > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta >Priority: Minor > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035929#comment-15035929 ] Roman Bielik commented on CASSANDRA-9748: - Hi Paulo, which git branch can I use? I tried this one: https://github.com/apache/cassandra/tree/204bd3af266c36d1c83d8cedb5daaa8124342178 but there seems to be a problem with some ANSI characters during the compilation. Thank you. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta >Priority: Minor > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[
https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036071#comment-15036071
]
Paulo Motta commented on CASSANDRA-9748:
It seems your default JDK is configured to use ASCII, but you need to configure
it to use UTF-8.
Could you try setting {{export JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8}} before
running ant?
> Can't see other nodes when using multiple network interfaces
>
>
> Key: CASSANDRA-9748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9748
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
> Environment: Cassandra 2.0.16; multi-DC configuration
>Reporter: Roman Bielik
>Assignee: Paulo Motta
>Priority: Minor
> Labels: docs-impacting
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
> Attachments: system_node1.log, system_node2.log
>
>
> The idea is to setup a multi-DC environment across 2 different networks based
> on the following configuration recommendations:
> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html
> Each node has 2 network interfaces. One used as a private network (DC1:
> 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all
> nodes can see each other (this one has a higher latency).
> Using the following settings in cassandra.yaml:
> *seeds:* public IP (same as used in broadcast_address)
> *listen_address:* private IP
> *broadcast_address:* public IP
> *rpc_address:* 0.0.0.0
> *endpoint_snitch:* GossipingPropertyFileSnitch
> _(tried different combinations with no luck)_
> No firewall and no SSL/encryption used.
> The problem is that nodes do not see each other (a gossip problem I guess).
> The nodetool ring/status shows only the local node but not the other ones
> (even from the same DC).
> When I set listen_address to public IP, then everything works fine, but that
> is not the required configuration.
> _Note: Not using EC2 cloud!_
> netstat -anp | grep -E "(7199|9160|9042|7000)"
> tcp0 0 0.0.0.0:71990.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9160 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9042 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:7000 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 127.0.0.1:7199 127.0.0.1:52874
> ESTABLISHED 3587/java
> tcp0 0 10.0.1.1:7199 10.0.1.1:39650
> ESTABLISHED 3587/java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[
https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036060#comment-15036060
]
Roman Bielik commented on CASSANDRA-9748:
-
Yeah, I can test on both 2.1 or 2.2 and both fail with the same errors.
Compiled under CentOS 6 + Java 1.7.
Perhaps I'm doing something wrong - the last time I tried your branch as
instructed it worked fine.
[javac]
/cassandra/cassandra-256bd77fa835486fab6bce9583e55a6a4ae9aa0a/src/java/org/apache/cassandra/utils/CoalescingStrategies.java:172:
error: unmappable character for encoding ASCII
[javac] logger.info(toString() + " gap " +
TimeUnit.NANOSECONDS.toMicros(averageGap) + "??s");
[javac]
^
[javac] 4 errors
[javac]
/cassandra/cassandra-204bd3af266c36d1c83d8cedb5daaa8124342178/test/unit/org/apache/cassandra/cql3/validation/operations/UpdateTest.java:57:
error: unmappable character for encoding ASCII
[javac]row(ByteBufferUtil.bytes("??"),
ByteBufferUtil.bytes("??"), set(ByteBufferUtil.bytes("??")),
ByteBufferUtil.bytes("??")));
[javac]
> Can't see other nodes when using multiple network interfaces
>
>
> Key: CASSANDRA-9748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9748
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
> Environment: Cassandra 2.0.16; multi-DC configuration
>Reporter: Roman Bielik
>Assignee: Paulo Motta
>Priority: Minor
> Labels: docs-impacting
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
> Attachments: system_node1.log, system_node2.log
>
>
> The idea is to setup a multi-DC environment across 2 different networks based
> on the following configuration recommendations:
> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html
> Each node has 2 network interfaces. One used as a private network (DC1:
> 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all
> nodes can see each other (this one has a higher latency).
> Using the following settings in cassandra.yaml:
> *seeds:* public IP (same as used in broadcast_address)
> *listen_address:* private IP
> *broadcast_address:* public IP
> *rpc_address:* 0.0.0.0
> *endpoint_snitch:* GossipingPropertyFileSnitch
> _(tried different combinations with no luck)_
> No firewall and no SSL/encryption used.
> The problem is that nodes do not see each other (a gossip problem I guess).
> The nodetool ring/status shows only the local node but not the other ones
> (even from the same DC).
> When I set listen_address to public IP, then everything works fine, but that
> is not the required configuration.
> _Note: Not using EC2 cloud!_
> netstat -anp | grep -E "(7199|9160|9042|7000)"
> tcp0 0 0.0.0.0:71990.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9160 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9042 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:7000 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 127.0.0.1:7199 127.0.0.1:52874
> ESTABLISHED 3587/java
> tcp0 0 10.0.1.1:7199 10.0.1.1:39650
> ESTABLISHED 3587/java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036300#comment-15036300 ] Roman Bielik commented on CASSANDRA-9748: - Thank you, that worked like a charm :) I'll let you know the test results tomorrow. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta >Priority: Minor > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034813#comment-15034813 ] Paulo Motta commented on CASSANDRA-9748: Updated dtests to check if ports are bound, in addition to logs. I guess in case log changes we can just update tests. Tested on Windows and works fine. Let's wait [~RomanB]'s new patch results in his environment before marking this as ready to commit. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta >Priority: Minor > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15032129#comment-15032129 ] Ariel Weisberg commented on CASSANDRA-9748: --- Nice! Way to make it less code. I reviewed the code and dtests and it looks good. Parsing the logs makes it a little fragile to changes in log statements. Moderately better might be to look if the interface and port are bound (simple on Linux, less so on Windows), or to connect to it and make sure the protocol messages returned are acceptable. I don't feel strongly about it, but it's one of those when not if things where this test is going to need to be updated. The tests appear to pass in Cassci so that looks good. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta >Priority: Minor > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029903#comment-15029903 ] Roman Bielik commented on CASSANDRA-9748: - Thank you, for me this feature is really important and I would be glad if it could be backported to the 3.x branch as well (for future compatibility). > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta >Priority: Minor > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[
https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029971#comment-15029971
]
Paulo Motta commented on CASSANDRA-9748:
[~RomanB] meanwhile would you mind trying the new patch with the
{{listen_on_broadcast_address}} option enabled following the instructions on
[configuring multiple
networks|http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html]
and check if that also works?
> Can't see other nodes when using multiple network interfaces
>
>
> Key: CASSANDRA-9748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9748
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
> Environment: Cassandra 2.0.16; multi-DC configuration
>Reporter: Roman Bielik
>Assignee: Paulo Motta
>Priority: Minor
> Labels: docs-impacting
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
> Attachments: system_node1.log, system_node2.log
>
>
> The idea is to setup a multi-DC environment across 2 different networks based
> on the following configuration recommendations:
> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html
> Each node has 2 network interfaces. One used as a private network (DC1:
> 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all
> nodes can see each other (this one has a higher latency).
> Using the following settings in cassandra.yaml:
> *seeds:* public IP (same as used in broadcast_address)
> *listen_address:* private IP
> *broadcast_address:* public IP
> *rpc_address:* 0.0.0.0
> *endpoint_snitch:* GossipingPropertyFileSnitch
> _(tried different combinations with no luck)_
> No firewall and no SSL/encryption used.
> The problem is that nodes do not see each other (a gossip problem I guess).
> The nodetool ring/status shows only the local node but not the other ones
> (even from the same DC).
> When I set listen_address to public IP, then everything works fine, but that
> is not the required configuration.
> _Note: Not using EC2 cloud!_
> netstat -anp | grep -E "(7199|9160|9042|7000)"
> tcp0 0 0.0.0.0:71990.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9160 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9042 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:7000 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 127.0.0.1:7199 127.0.0.1:52874
> ESTABLISHED 3587/java
> tcp0 0 10.0.1.1:7199 10.0.1.1:39650
> ESTABLISHED 3587/java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[
https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029976#comment-15029976
]
Paulo Motta commented on CASSANDRA-9748:
Don't forget to enable the {{prefer_local}} option of the
{{GossipingPropertyFileSnitch}} (that is not mentioned in those docs
unfortunately), the {{local_address}} option is not needed anymore.
> Can't see other nodes when using multiple network interfaces
>
>
> Key: CASSANDRA-9748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9748
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
> Environment: Cassandra 2.0.16; multi-DC configuration
>Reporter: Roman Bielik
>Assignee: Paulo Motta
>Priority: Minor
> Labels: docs-impacting
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
> Attachments: system_node1.log, system_node2.log
>
>
> The idea is to setup a multi-DC environment across 2 different networks based
> on the following configuration recommendations:
> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html
> Each node has 2 network interfaces. One used as a private network (DC1:
> 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all
> nodes can see each other (this one has a higher latency).
> Using the following settings in cassandra.yaml:
> *seeds:* public IP (same as used in broadcast_address)
> *listen_address:* private IP
> *broadcast_address:* public IP
> *rpc_address:* 0.0.0.0
> *endpoint_snitch:* GossipingPropertyFileSnitch
> _(tried different combinations with no luck)_
> No firewall and no SSL/encryption used.
> The problem is that nodes do not see each other (a gossip problem I guess).
> The nodetool ring/status shows only the local node but not the other ones
> (even from the same DC).
> When I set listen_address to public IP, then everything works fine, but that
> is not the required configuration.
> _Note: Not using EC2 cloud!_
> netstat -anp | grep -E "(7199|9160|9042|7000)"
> tcp0 0 0.0.0.0:71990.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9160 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9042 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:7000 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 127.0.0.1:7199 127.0.0.1:52874
> ESTABLISHED 3587/java
> tcp0 0 10.0.1.1:7199 10.0.1.1:39650
> ESTABLISHED 3587/java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029980#comment-15029980 ] Roman Bielik commented on CASSANDRA-9748: - Will do - early next week and will let you know the results. Thank you! > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta >Priority: Minor > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[
https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029271#comment-15029271
]
Paulo Motta commented on CASSANDRA-9748:
Actually the solution is much simpler than initially thought, and requires
little change in code and documentation.
The idea is to provide a new property {{listen_on_broadcast_address}}, which
should be enabled for who wants to listen on both {{listen_address}} and
{{broadcast_address}}, in network environments which do not automatically route
from the public to the private address, such as outside EC2. The benefits of
this approach is that the [multiple network interfaces
documentation|http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html]
requires little change, just a note asking users to enable
{{listen_on_broadcast_address}} property if they're using multiple physical
interfaces. Furthermore, no changes are required to
{{GossipingPropertyFileSnitch}} which already re-routes traffic from
{{broadcast_address}} to {{listen_address}} when the {{prefer_local}} is
enabled, so it's done.
Changing {{listen_address}} to receive a list wouldn't be a good idea because
it would require a change to system keyspace and associated schema migrations.
Furthermore there are other places around the code that use this property so
this would break them (and maybe drivers as well). Adding a
{{local_listen_address}} would require a different documentation for EC2 and
multiple physical network interfaces, and would add more complexity, since we
already have {{listen_address}} and {{broadcast_address}} exactly for this use
case already, so no need to introduce a third address in addition to those.
I created a [dtest|https://github.com/riptano/cassandra-dtest/pull/687] that
tests {{listen_on_broadcast_address}} and also the {{prefer_local}} property of
{{GossipingPropertyFileSnitch}}.
The change is so small and independent that I think we can include it in 2.1.
I'm not sure if it should go into 3.0.1/3.1, but submitted tests anyway, so
will let the committer/reviewer decide.
There was another user recently [on the mailing
list|https://www.mail-archive.com/user@cassandra.apache.org/msg44987.html]
unable to setup multiple network interfaces due to this limitation, so I think
it has non-negligible demand.
||2.1||2.2||3.0||trunk||dtest||
|[branch|https://github.com/apache/cassandra/compare/cassandra-2.1...pauloricardomg:9478-2.1-v2]|[branch|https://github.com/apache/cassandra/compare/cassandra-2.2...pauloricardomg:9478-2.2-v2]|[branch|https://github.com/apache/cassandra/compare/cassandra-3.0...pauloricardomg:9478-3.0-v2]|[branch|https://github.com/apache/cassandra/compare/trunk...pauloricardomg:9478-trunk-v2]|[branch|https://github.com/riptano/cassandra-dtest/compare/master...pauloricardomg:9748]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-2.1-v2-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-2.2-v2-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-3.0-v2-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-trunk-v2-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-2.1-v2-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-2.2-v2-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-3.0-v2-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-trunk-v2-dtest/lastCompletedBuild/testReport/]|
> Can't see other nodes when using multiple network interfaces
>
>
> Key: CASSANDRA-9748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9748
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
> Environment: Cassandra 2.0.16; multi-DC configuration
>Reporter: Roman Bielik
>Assignee: Paulo Motta
>Priority: Minor
> Labels: docs-impacting
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
> Attachments: system_node1.log, system_node2.log
>
>
> The idea is to setup a multi-DC environment across 2 different networks based
> on the following configuration recommendations:
> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html
> Each node has 2 network interfaces. One used as a private network (DC1:
> 10.0.1.x and DC2: 10.0.2.x). The second one a
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990223#comment-14990223 ] Ariel Weisberg commented on CASSANDRA-9748: --- [~pauloricardomg] can you make it so they can configure multiple interfaces for listening without binding to all interfaces? At some point in the future we will genuinely need to support multiple interfaces for performance in virtualized environments. We could go with a UI that accommodates that although we will probably get it wrong anyways. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988310#comment-14988310 ] Jeremiah Jordan commented on CASSANDRA-9748: I definitely wouldn't allow 0.0.0.0 in there, but add a new setting for "local listen" or some such, or support a list. As we would still only support a max of two here yes? The main one and the local one? > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988261#comment-14988261 ] Ariel Weisberg commented on CASSANDRA-9748: --- Summing up our private conversation. The use case is where someone has two interfaces they want to use for intra-cluster traffic. One is accessible from other DCs the other is intra-DC. Right now they can only specify one broadcast address and that has to be the one reachable everywhere. This was fine for EC2 with it's weird private/public IP magic. For real separate interfaces it doesn't work because you need to select between one of two IPs. The addition of a local IP to the GossipingPropertyFile snitch means that instead of advertising the broadcast address via gossip (as the internal interface) it can advertise a second interface. My question is why is this necessary at all? My ops fu is pretty weak but I thought that a single interface is generally fine (unless there is a performance issue) because you can configure your network to do the right thing WRT to reachability and efficiently routing traffic between and intra DCs with a single IP. One thing that seems suboptimal is binding to all interfaces instead of a list of interfaces. This forces people to manually block connections on interfaces we really shouldn't be accepting on. That said this is mechanism not policy and it enables people to do something they couldn't before so it's an improvement and they have options outside of C*. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985995#comment-14985995 ] Ariel Weisberg commented on CASSANDRA-9748: --- I'm a little confused as to why binding to the private IP for the interface isn't working? I thought you can't bind to the public IP. You have to bind to the private one and I thought that was how you accepted connections on the public IP and private IP. It's not two different interfaces it's one interface. I just did a quick test. I bound with socat using the private IP and then connected using the public IP and it worked fine. Is there an explanation for that? I also thought that it was always safe to use the public IP when advertising and not the local one and that EC2 magically takes care of not routing everything over the wider network. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[
https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14967887#comment-14967887
]
Paulo Motta commented on CASSANDRA-9748:
Attached [2.1|https://github.com/pauloricardomg/cassandra/tree/9478-2.1],
[2.2|https://github.com/pauloricardomg/cassandra/tree/9478-2.2] and
[3.0|https://github.com/pauloricardomg/cassandra/tree/9478-3.0] patches. Merge
conflicts are basically test formatting, so not much difference between
different version patches.
Main changes are:
* Allow specifying {{listen_address=0.0.0.0}} (in order to support listening on
multiple interfaces), as long as {{broadcast_address != 0.0.0.0}} (similar to
what is done in {{rpc_address}} and {{rpc_broadcast_address}}). AFAIK
{{listen_address=0.0.0.0}} was disallowed for [historic
reasons|http://wiki.apache.org/cassandra/FAQ#cant_listen_on_ip_any], mostly
because this was used as a node primary id, but since the introduction of
{{broadcast_address}} on CASSANDRA-2491 I believe this is not a concern anymore.
* Added {{local_address}} option to {{GossipingPropertyFileSnitch}} in order to
specify preferred local address to use when {{prefer_local=true}} and
{{listen_address=0.0.0.0}} (since currently the value of {{listen_address}} is
used as preferred local address).
** Added warning when {{prefer_local==true}}, {{listen_address=0.0.0.0}} and
{{local_address}} is not specified on {{cassandra-rackdc.properties}}.
* {{DatabaseDescriptor}} tests with validations when {{listen_address=0.0.0.0}}
and {{GossipingPropertyFileSnitch}} tests for new {{local_address}} option.
Tests:
* 2.1
[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-2.1-testall/]
[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-2.1-dtest/]
* 2.2
[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-2.2-testall/]
[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-2.2-dtest/]
* 3.0
[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-3.0-testall/]
[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-3.0-dtest/]
* trunk
[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-trunk-testall/]
[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-trunk-dtest/]
TODO after merged:
* Update [Using multiple network
interfaces|http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configMultiNetworks.html]
doc entry to refer to the new options (outside EC2).
* Update [wiki
entry|http://wiki.apache.org/cassandra/FAQ#cant_listen_on_ip_any] disallowing
0.0.0.0
> Can't see other nodes when using multiple network interfaces
>
>
> Key: CASSANDRA-9748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9748
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.0.16; multi-DC configuration
>Reporter: Roman Bielik
>Assignee: Paulo Motta
> Attachments: system_node1.log, system_node2.log
>
>
> The idea is to setup a multi-DC environment across 2 different networks based
> on the following configuration recommendations:
> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html
> Each node has 2 network interfaces. One used as a private network (DC1:
> 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all
> nodes can see each other (this one has a higher latency).
> Using the following settings in cassandra.yaml:
> *seeds:* public IP (same as used in broadcast_address)
> *listen_address:* private IP
> *broadcast_address:* public IP
> *rpc_address:* 0.0.0.0
> *endpoint_snitch:* GossipingPropertyFileSnitch
> _(tried different combinations with no luck)_
> No firewall and no SSL/encryption used.
> The problem is that nodes do not see each other (a gossip problem I guess).
> The nodetool ring/status shows only the local node but not the other ones
> (even from the same DC).
> When I set listen_address to public IP, then everything works fine, but that
> is not the required configuration.
> _Note: Not using EC2 cloud!_
> netstat -anp | grep -E "(7199|9160|9042|7000)"
> tcp0 0 0.0.0.0:71990.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9160 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9042 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:7000 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 127.0.0.1:7199
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953165#comment-14953165 ] Roman Bielik commented on CASSANDRA-9748: - Hi Paulo, thank you very much, the provided github version works absolutely well!! I used the suggested configuration and the nodes started smoothly seeing both nodes in a ring. I run few tests with my Thrift client and all looks good. :) > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[
https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943965#comment-14943965
]
Paulo Motta commented on CASSANDRA-9748:
[~RomanB] I've attached a [github
branch|https://github.com/pauloricardomg/cassandra/tree/9478-2.1] that allows
setting listen_address to 0.0.0.0, could you please test that? I've tested with
my wlan0 and eth0 interfaces and it seems to work. After cloning the repository
you may use {{ant jar}} to create jars (and replace them manually), or {{ant
artifacts}} to create a cassandra tarball which you can then decompress to
perform the test.
In addition to setting the {{listen_address}} to {{0.0.0.0}} you should enable
the {{GossipingPropertyFileSnitch}} option {{prefer_local}} and also set the
new option of this snitch {{local_address}} to the local ip adress the node
should be contacted in the private network.
In summary, your configuration should look like:
* *seeds*: public IP (same as used in broadcast_address)
* *listen_address*: 0.0.0.0
* *broadcast_address*: public IP
* *rpc_address*: 0.0.0.0
* *broadcast_rpc_address*: public IP
* *endpoint_snitch*: GossipingPropertyFileSnitch
** *prefer_local*: true
** *local_address*: private IP
> Can't see other nodes when using multiple network interfaces
>
>
> Key: CASSANDRA-9748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9748
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.0.16; multi-DC configuration
>Reporter: Roman Bielik
>Assignee: Paulo Motta
> Attachments: system_node1.log, system_node2.log
>
>
> The idea is to setup a multi-DC environment across 2 different networks based
> on the following configuration recommendations:
> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html
> Each node has 2 network interfaces. One used as a private network (DC1:
> 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all
> nodes can see each other (this one has a higher latency).
> Using the following settings in cassandra.yaml:
> *seeds:* public IP (same as used in broadcast_address)
> *listen_address:* private IP
> *broadcast_address:* public IP
> *rpc_address:* 0.0.0.0
> *endpoint_snitch:* GossipingPropertyFileSnitch
> _(tried different combinations with no luck)_
> No firewall and no SSL/encryption used.
> The problem is that nodes do not see each other (a gossip problem I guess).
> The nodetool ring/status shows only the local node but not the other ones
> (even from the same DC).
> When I set listen_address to public IP, then everything works fine, but that
> is not the required configuration.
> _Note: Not using EC2 cloud!_
> netstat -anp | grep -E "(7199|9160|9042|7000)"
> tcp0 0 0.0.0.0:71990.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9160 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9042 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:7000 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 127.0.0.1:7199 127.0.0.1:52874
> ESTABLISHED 3587/java
> tcp0 0 10.0.1.1:7199 10.0.1.1:39650
> ESTABLISHED 3587/java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905963#comment-14905963 ] Roman Bielik commented on CASSANDRA-9748: - Hi, I tried the port forwarding and the telnet to public IP port 7000 works OK, but I still get the "Unable to gossip..." error on the second node. Anyway the analysis seems perfectly reasonable, thank you. That would explain why in all tutorials the private/public IP configuration was always used in combination with EC2MultiRegionSnitch. Thank you, it would be great if the listen_address could be set to all interfaces (0.0.0.0) as it would probably solve this issue. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903515#comment-14903515 ] Paulo Motta commented on CASSANDRA-9748: Thanks for the logs! Since listen_address is set to the private ip, the node only listen to connections on the private interface, so nodes cannot connect to the public broadcast_address (since the node is not listening to it). EC2 peforms the translation automatically from public elastic IPs to the internal private IP, so this problem doesn't happen when using the EC2MultiRegionSnitch (CASSANDRA-2452 added the broadcast_address parameter to support the EC2MultiRegionSnitch). As a temporary workaround, could you add an iptables NAT rule to forward all packets on the public interface to the private interface on port 7000 and check if that fixes the problem? You can use telnet to verify if the forwarding is being performed correctly. A more permanent solution would be to allow the messaging service to listen on multiple interfaces (listen_address=0.0.0.0), but currently this is not supported. As long as the broadcast_address is set, I don't see a problem on enabling listen_address=0.0.0.0, but I will have to check with other developers if there's any problem with that approach. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14902453#comment-14902453 ] Roman Bielik commented on CASSANDRA-9748: - Please find the systen.logs attached below. The configuration was following: Node 1 == seeds: "192.168.24.153" listen_address: 10.0.2.1 broadcast_address: 192.168.24.153 Gossipinfo: /192.168.24.153 generation:1442922252 heartbeat:298 STATUS:NORMAL,-9223372036854775708 INTERNAL_IP:10.0.2.1 DC:DC2 HOST_ID:443c4534-3b1c-469b-9abc-8ed8e19db09e RELEASE_VERSION:2.2.1 LOAD:93969.0 RPC_READY:true RPC_ADDRESS:10.0.2.1 NET_VERSION:9 SEVERITY:0.0 RACK:RAC1 SCHEMA:040e7755-de2c-3516-8cf0-84abcdece3f7 Node 2 == seeds: "192.168.24.153" listen_address: 10.0.2.2 broadcast_address: 192.168.24.154 Node did not start, startup exception: "Unable to gossip with any seeds" > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747206#comment-14747206 ] Roman Bielik commented on CASSANDRA-9748: - Retested with the suggested settings, but it had no effect. The result is still the same. How does "prefer_local" effect the gossip? I mean how does Cassandra identify which interface is local? > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[
https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14790840#comment-14790840
]
Paulo Motta commented on CASSANDRA-9748:
{{prefer_local}} basically makes connections in the local datacenter use
{{listen_address}} (which is typically a private IP), instead of
{{broadcast_address}} (which is typically a public IP), but that was just a
wild guess which didn't seem to work.
ok, could you try to reproduce the issue again with only 2 nodes (a seed, and a
non-seed) with version 2.1 or 2.2, and change the root level from {{INFO}} to
{{DEBUG}} on {{conf/logback.xml}}, and also add the following lines to the end
of your {{conf/logback.xml}} (within the configuration section):
{noformat}
...
{noformat}
Please attach the corresponding {{log/system.log}} of both nodes here so we can
investigate what's going on with more detail. Also, please attach the output of
{{nodetool gossipinfo}} of both nodes.
> Can't see other nodes when using multiple network interfaces
>
>
> Key: CASSANDRA-9748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9748
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.0.16; multi-DC configuration
>Reporter: Roman Bielik
>Assignee: Paulo Motta
>
> The idea is to setup a multi-DC environment across 2 different networks based
> on the following configuration recommendations:
> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html
> Each node has 2 network interfaces. One used as a private network (DC1:
> 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all
> nodes can see each other (this one has a higher latency).
> Using the following settings in cassandra.yaml:
> *seeds:* public IP (same as used in broadcast_address)
> *listen_address:* private IP
> *broadcast_address:* public IP
> *rpc_address:* 0.0.0.0
> *endpoint_snitch:* GossipingPropertyFileSnitch
> _(tried different combinations with no luck)_
> No firewall and no SSL/encryption used.
> The problem is that nodes do not see each other (a gossip problem I guess).
> The nodetool ring/status shows only the local node but not the other ones
> (even from the same DC).
> When I set listen_address to public IP, then everything works fine, but that
> is not the required configuration.
> _Note: Not using EC2 cloud!_
> netstat -anp | grep -E "(7199|9160|9042|7000)"
> tcp0 0 0.0.0.0:71990.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9160 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9042 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:7000 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 127.0.0.1:7199 127.0.0.1:52874
> ESTABLISHED 3587/java
> tcp0 0 10.0.1.1:7199 10.0.1.1:39650
> ESTABLISHED 3587/java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14745299#comment-14745299 ] Roman Bielik commented on CASSANDRA-9748: - I have retested on Cassandra 2.2.1 and the results are pretty much the same. If my premise is correct, I need to use public_IP as seeds as this is the interface/network where different datacenters can connect to each other. I have tested all possible permutations of listen_address, broadcast_address and broadcast_rpc_address. The seeds were set to public IP and rpc_address to 0.0.0.0 in all tests. Results: listen_address - if set to private_IP, the ring showed only one node broadcast_address - if set to private_IP, there was always problem with gossip broadcast_rpc_address - it did not matter in terms of connection or ring, so I could use private IP here. To sum up: I still needed to use public IP in seeds, listen_address and broadcast_address in order to start a Cassandra cluster. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[
https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14745614#comment-14745614
]
Paulo Motta commented on CASSANDRA-9748:
Could you try using the {{GossipingPropertyFileSnitch}} option
{{prefer_local=true}}, if you haven't already, and check if that fixes the
problem?
> Can't see other nodes when using multiple network interfaces
>
>
> Key: CASSANDRA-9748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9748
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.0.16; multi-DC configuration
>Reporter: Roman Bielik
>
> The idea is to setup a multi-DC environment across 2 different networks based
> on the following configuration recommendations:
> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html
> Each node has 2 network interfaces. One used as a private network (DC1:
> 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all
> nodes can see each other (this one has a higher latency).
> Using the following settings in cassandra.yaml:
> *seeds:* public IP (same as used in broadcast_address)
> *listen_address:* private IP
> *broadcast_address:* public IP
> *rpc_address:* 0.0.0.0
> *endpoint_snitch:* GossipingPropertyFileSnitch
> _(tried different combinations with no luck)_
> No firewall and no SSL/encryption used.
> The problem is that nodes do not see each other (a gossip problem I guess).
> The nodetool ring/status shows only the local node but not the other ones
> (even from the same DC).
> When I set listen_address to public IP, then everything works fine, but that
> is not the required configuration.
> _Note: Not using EC2 cloud!_
> netstat -anp | grep -E "(7199|9160|9042|7000)"
> tcp0 0 0.0.0.0:71990.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9160 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9042 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:7000 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 127.0.0.1:7199 127.0.0.1:52874
> ESTABLISHED 3587/java
> tcp0 0 10.0.1.1:7199 10.0.1.1:39650
> ESTABLISHED 3587/java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736009#comment-14736009 ] Jonathan Ellis commented on CASSANDRA-9748: --- Can you reproduce with 2.1 or 2.2? > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14696772#comment-14696772 ] Richard Dawe commented on CASSANDRA-9748: - I'm seeing a similar issue with Cassandra 2.0.15. I strace'd the java process, and I saw that it's trying to connect to port 7000 on the public IP address, but it's getting a ECONNREFUSED. My C* nodes are only listening on port 7000 on the private IP address. I have broadcast_address = public IP and listen_address = public IP. Can't see other nodes when using multiple network interfaces Key: CASSANDRA-9748 URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.0.16; multi-DC configuration Reporter: Roman Bielik The idea is to setup a multi-DC environment across 2 different networks based on the following configuration recommendations: http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html Each node has 2 network interfaces. One used as a private network (DC1: 10.0.1.x and DC2: 10.0.2.x). The second one a public network where all nodes can see each other (this one has a higher latency). Using the following settings in cassandra.yaml: *seeds:* public IP (same as used in broadcast_address) *listen_address:* private IP *broadcast_address:* public IP *rpc_address:* 0.0.0.0 *endpoint_snitch:* GossipingPropertyFileSnitch _(tried different combinations with no luck)_ No firewall and no SSL/encryption used. The problem is that nodes do not see each other (a gossip problem I guess). The nodetool ring/status shows only the local node but not the other ones (even from the same DC). When I set listen_address to public IP, then everything works fine, but that is not the required configuration. _Note: Not using EC2 cloud!_ netstat -anp | grep -E (7199|9160|9042|7000) tcp0 0 0.0.0.0:71990.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9160 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9042 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:7000 0.0.0.0:* LISTEN 3587/java tcp0 0 127.0.0.1:7199 127.0.0.1:52874 ESTABLISHED 3587/java tcp0 0 10.0.1.1:7199 10.0.1.1:39650 ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679714#comment-14679714 ] Roman Bielik commented on CASSANDRA-9748: - Yeah, the problem arises even with the first DC. It might be some configuration problem indeed. But I can't figure out what. I followed the recommended setting for seeds, listen and broadcast addresses plus tried different variations with no luck. OK, I will post a question using the suggested mailing list. Thank you for the tip. Can't see other nodes when using multiple network interfaces Key: CASSANDRA-9748 URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.0.16; multi-DC configuration Reporter: Roman Bielik The idea is to setup a multi-DC environment across 2 different networks based on the following configuration recommendations: http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html Each node has 2 network interfaces. One used as a private network (DC1: 10.0.1.x and DC2: 10.0.2.x). The second one a public network where all nodes can see each other (this one has a higher latency). Using the following settings in cassandra.yaml: *seeds:* public IP (same as used in broadcast_address) *listen_address:* private IP *broadcast_address:* public IP *rpc_address:* 0.0.0.0 *endpoint_snitch:* GossipingPropertyFileSnitch _(tried different combinations with no luck)_ No firewall and no SSL/encryption used. The problem is that nodes do not see each other (a gossip problem I guess). The nodetool ring/status shows only the local node but not the other ones (even from the same DC). When I set listen_address to public IP, then everything works fine, but that is not the required configuration. _Note: Not using EC2 cloud!_ netstat -anp | grep -E (7199|9160|9042|7000) tcp0 0 0.0.0.0:71990.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9160 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9042 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:7000 0.0.0.0:* LISTEN 3587/java tcp0 0 127.0.0.1:7199 127.0.0.1:52874 ESTABLISHED 3587/java tcp0 0 10.0.1.1:7199 10.0.1.1:39650 ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14659898#comment-14659898 ] Paulo Motta commented on CASSANDRA-9748: Hmm, I'll reopen the ticket then, but this looks like some kind of configuration problem. Did you try doing a similar configuration with only one DC first? Also, you may get a faster responser by asking on the user mailing list (u...@cassandra.apache.org), since you may expose the problem to users more familiar with this kind of setup. Can't see other nodes when using multiple network interfaces Key: CASSANDRA-9748 URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.0.16; multi-DC configuration Reporter: Roman Bielik The idea is to setup a multi-DC environment across 2 different networks based on the following configuration recommendations: http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html Each node has 2 network interfaces. One used as a private network (DC1: 10.0.1.x and DC2: 10.0.2.x). The second one a public network where all nodes can see each other (this one has a higher latency). Using the following settings in cassandra.yaml: *seeds:* public IP (same as used in broadcast_address) *listen_address:* private IP *broadcast_address:* public IP *rpc_address:* 0.0.0.0 *endpoint_snitch:* GossipingPropertyFileSnitch _(tried different combinations with no luck)_ No firewall and no SSL/encryption used. The problem is that nodes do not see each other (a gossip problem I guess). The nodetool ring/status shows only the local node but not the other ones (even from the same DC). When I set listen_address to public IP, then everything works fine, but that is not the required configuration. _Note: Not using EC2 cloud!_ netstat -anp | grep -E (7199|9160|9042|7000) tcp0 0 0.0.0.0:71990.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9160 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9042 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:7000 0.0.0.0:* LISTEN 3587/java tcp0 0 127.0.0.1:7199 127.0.0.1:52874 ESTABLISHED 3587/java tcp0 0 10.0.1.1:7199 10.0.1.1:39650 ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14653519#comment-14653519 ] Paulo Motta commented on CASSANDRA-9748: You must use the same list of seeds for all nodes, otherwise the nodes will not be able to see each other during cluster bootstrap. Find more information about seed nodes here: http://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architectureGossipAbout_c.html Can't see other nodes when using multiple network interfaces Key: CASSANDRA-9748 URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.0.16; multi-DC configuration Reporter: Roman Bielik The idea is to setup a multi-DC environment across 2 different networks based on the following configuration recommendations: http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html Each node has 2 network interfaces. One used as a private network (DC1: 10.0.1.x and DC2: 10.0.2.x). The second one a public network where all nodes can see each other (this one has a higher latency). Using the following settings in cassandra.yaml: *seeds:* public IP (same as used in broadcast_address) *listen_address:* private IP *broadcast_address:* public IP *rpc_address:* 0.0.0.0 *endpoint_snitch:* GossipingPropertyFileSnitch _(tried different combinations with no luck)_ No firewall and no SSL/encryption used. The problem is that nodes do not see each other (a gossip problem I guess). The nodetool ring/status shows only the local node but not the other ones (even from the same DC). When I set listen_address to public IP, then everything works fine, but that is not the required configuration. _Note: Not using EC2 cloud!_ netstat -anp | grep -E (7199|9160|9042|7000) tcp0 0 0.0.0.0:71990.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9160 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9042 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:7000 0.0.0.0:* LISTEN 3587/java tcp0 0 127.0.0.1:7199 127.0.0.1:52874 ESTABLISHED 3587/java tcp0 0 10.0.1.1:7199 10.0.1.1:39650 ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14653294#comment-14653294 ] Roman Bielik commented on CASSANDRA-9748: - No, each node uses its own public IP. So each seed belongs to a particular node. Can't see other nodes when using multiple network interfaces Key: CASSANDRA-9748 URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.0.16; multi-DC configuration Reporter: Roman Bielik The idea is to setup a multi-DC environment across 2 different networks based on the following configuration recommendations: http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html Each node has 2 network interfaces. One used as a private network (DC1: 10.0.1.x and DC2: 10.0.2.x). The second one a public network where all nodes can see each other (this one has a higher latency). Using the following settings in cassandra.yaml: *seeds:* public IP (same as used in broadcast_address) *listen_address:* private IP *broadcast_address:* public IP *rpc_address:* 0.0.0.0 *endpoint_snitch:* GossipingPropertyFileSnitch _(tried different combinations with no luck)_ No firewall and no SSL/encryption used. The problem is that nodes do not see each other (a gossip problem I guess). The nodetool ring/status shows only the local node but not the other ones (even from the same DC). When I set listen_address to public IP, then everything works fine, but that is not the required configuration. _Note: Not using EC2 cloud!_ netstat -anp | grep -E (7199|9160|9042|7000) tcp0 0 0.0.0.0:71990.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9160 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9042 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:7000 0.0.0.0:* LISTEN 3587/java tcp0 0 127.0.0.1:7199 127.0.0.1:52874 ESTABLISHED 3587/java tcp0 0 10.0.1.1:7199 10.0.1.1:39650 ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14653529#comment-14653529 ] Roman Bielik commented on CASSANDRA-9748: - Yeah this should be correct. Perhaps I misunderstood the question. I'm using public IPs in seeds configuration and the seeds list is the same on all nodes. The list contains seeds from both datacenters too. So in my case the seeds looked like: 192.168.24.151,192.168.24.152,192.168.24.153,192.168.24.154 on all nodes. Can't see other nodes when using multiple network interfaces Key: CASSANDRA-9748 URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.0.16; multi-DC configuration Reporter: Roman Bielik The idea is to setup a multi-DC environment across 2 different networks based on the following configuration recommendations: http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html Each node has 2 network interfaces. One used as a private network (DC1: 10.0.1.x and DC2: 10.0.2.x). The second one a public network where all nodes can see each other (this one has a higher latency). Using the following settings in cassandra.yaml: *seeds:* public IP (same as used in broadcast_address) *listen_address:* private IP *broadcast_address:* public IP *rpc_address:* 0.0.0.0 *endpoint_snitch:* GossipingPropertyFileSnitch _(tried different combinations with no luck)_ No firewall and no SSL/encryption used. The problem is that nodes do not see each other (a gossip problem I guess). The nodetool ring/status shows only the local node but not the other ones (even from the same DC). When I set listen_address to public IP, then everything works fine, but that is not the required configuration. _Note: Not using EC2 cloud!_ netstat -anp | grep -E (7199|9160|9042|7000) tcp0 0 0.0.0.0:71990.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9160 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9042 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:7000 0.0.0.0:* LISTEN 3587/java tcp0 0 127.0.0.1:7199 127.0.0.1:52874 ESTABLISHED 3587/java tcp0 0 10.0.1.1:7199 10.0.1.1:39650 ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14652683#comment-14652683 ] Paulo Motta commented on CASSANDRA-9748: bq. seeds: public IP (same as used in broadcast_address) are you using the same seed public IP for all nodes, or each node uses its own public IP as seed? Can't see other nodes when using multiple network interfaces Key: CASSANDRA-9748 URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.0.16; multi-DC configuration Reporter: Roman Bielik The idea is to setup a multi-DC environment across 2 different networks based on the following configuration recommendations: http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html Each node has 2 network interfaces. One used as a private network (DC1: 10.0.1.x and DC2: 10.0.2.x). The second one a public network where all nodes can see each other (this one has a higher latency). Using the following settings in cassandra.yaml: *seeds:* public IP (same as used in broadcast_address) *listen_address:* private IP *broadcast_address:* public IP *rpc_address:* 0.0.0.0 *endpoint_snitch:* GossipingPropertyFileSnitch _(tried different combinations with no luck)_ No firewall and no SSL/encryption used. The problem is that nodes do not see each other (a gossip problem I guess). The nodetool ring/status shows only the local node but not the other ones (even from the same DC). When I set listen_address to public IP, then everything works fine, but that is not the required configuration. _Note: Not using EC2 cloud!_ netstat -anp | grep -E (7199|9160|9042|7000) tcp0 0 0.0.0.0:71990.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9160 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9042 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:7000 0.0.0.0:* LISTEN 3587/java tcp0 0 127.0.0.1:7199 127.0.0.1:52874 ESTABLISHED 3587/java tcp0 0 10.0.1.1:7199 10.0.1.1:39650 ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14620125#comment-14620125 ] Roman Bielik commented on CASSANDRA-9748: - 10.0.1.x = DC1 subnet 10.0.2.x = DC2 subnet 192.168.24.x = shared (public) network, all nodes can see each other here *Node 1 (DC1)* IP: 192.168.24.151 (eth0), 10.0.1.1 (eth2) $ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface 10.0.1.00.0.0.0 255.255.255.0 U 0 00 eth2 192.168.24.00.0.0.0 255.255.252.0 U 0 00 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1002 00 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1003 00 eth2 0.0.0.0 192.168.24.254 0.0.0.0 UG0 00 eth0 $ ip route list 10.0.1.0/24 dev eth2 proto kernel scope link src 10.0.1.1 192.168.24.0/22 dev eth0 proto kernel scope link src 192.168.24.151 169.254.0.0/16 dev eth0 scope link metric 1002 169.254.0.0/16 dev eth2 scope link metric 1003 default via 192.168.24.254 dev eth0 *Node 2 (DC1)* IP: 192.168.24.152 (eth0), 10.0.1.2 (eth2) $ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface 10.0.1.00.0.0.0 255.255.255.0 U 0 00 eth2 192.168.24.00.0.0.0 255.255.252.0 U 0 00 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1002 00 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1003 00 eth2 0.0.0.0 192.168.24.254 0.0.0.0 UG0 00 eth0 $ ip route list 10.0.1.0/24 dev eth2 proto kernel scope link src 10.0.1.2 192.168.24.0/22 dev eth0 proto kernel scope link src 192.168.24.152 169.254.0.0/16 dev eth0 scope link metric 1002 169.254.0.0/16 dev eth2 scope link metric 1003 default via 192.168.24.254 dev eth0 *Node3 (DC2)* IP: 192.168.24.153 (eth0), 10.0.2.1 (eth2) $ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface 10.0.2.00.0.0.0 255.255.255.0 U 0 00 eth2 192.168.24.00.0.0.0 255.255.252.0 U 0 00 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1002 00 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1003 00 eth2 0.0.0.0 192.168.24.254 0.0.0.0 UG0 00 eth0 $ ip route list 10.0.2.0/24 dev eth2 proto kernel scope link src 10.0.2.1 192.168.24.0/22 dev eth0 proto kernel scope link src 192.168.24.153 169.254.0.0/16 dev eth0 scope link metric 1002 169.254.0.0/16 dev eth2 scope link metric 1003 default via 192.168.24.254 dev eth0 *Node4 (DC2)* IP: 192.168.24.154 (eth0), 10.0.2.2 (eth2) $ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface 10.0.2.00.0.0.0 255.255.255.0 U 0 00 eth2 192.168.24.00.0.0.0 255.255.252.0 U 0 00 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1002 00 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1003 00 eth2 0.0.0.0 192.168.24.254 0.0.0.0 UG0 00 eth0 $ ip route list 10.0.2.0/24 dev eth2 proto kernel scope link src 10.0.2.2 192.168.24.0/22 dev eth0 proto kernel scope link src 192.168.24.154 169.254.0.0/16 dev eth0 scope link metric 1002 169.254.0.0/16 dev eth2 scope link metric 1003 default via 192.168.24.254 dev eth0 Can't see other nodes when using multiple network interfaces Key: CASSANDRA-9748 URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.0.16; multi-DC configuration Reporter: Roman Bielik The idea is to setup a multi-DC environment across 2 different networks based on the following configuration recommendations: http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html Each node has 2 network interfaces. One used as a private network (DC1: 10.0.1.x and DC2: 10.0.2.x). The second one a public network where all nodes can see each other (this one has a higher latency). Using the following settings in cassandra.yaml: *seeds:* public IP (same as used in broadcast_address) *listen_address:* private IP *broadcast_address:* public IP *rpc_address:* 0.0.0.0 *endpoint_snitch:* GossipingPropertyFileSnitch _(tried different combinations with no luck)_ No firewall and no SSL/encryption used. The problem is that nodes do not see each other (a gossip problem I guess). The
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[
https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14618865#comment-14618865
]
Michael Shuler commented on CASSANDRA-9748:
---
What does the routing table on your nodes look like? {{route -n}} and/or {{ip
route list}}
Can't see other nodes when using multiple network interfaces
Key: CASSANDRA-9748
URL: https://issues.apache.org/jira/browse/CASSANDRA-9748
Project: Cassandra
Issue Type: Bug
Environment: Cassandra 2.0.16; multi-DC configuration
Reporter: Roman Bielik
The idea is to setup a multi-DC environment across 2 different networks based
on the following configuration recommendations:
http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html
Each node has 2 network interfaces. One used as a private network (DC1:
10.0.1.x and DC2: 10.0.2.x). The second one a public network where all
nodes can see each other (this one has a higher latency).
Using the following settings in cassandra.yaml:
*seeds:* public IP (same as used in broadcast_address)
*listen_address:* private IP
*broadcast_address:* public IP
*rpc_address:* 0.0.0.0
*endpoint_snitch:* GossipingPropertyFileSnitch
_(tried different combinations with no luck)_
No firewall and no SSL/encryption used.
The problem is that nodes do not see each other (a gossip problem I guess).
The nodetool ring/status shows only the local node but not the other ones
(even from the same DC).
When I set listen_address to public IP, then everything works fine, but that
is not the required configuration.
_Note: Not using EC2 cloud!_
netstat -anp | grep -E (7199|9160|9042|7000)
tcp0 0 0.0.0.0:71990.0.0.0:*
LISTEN 3587/java
tcp0 0 10.0.1.1:9160 0.0.0.0:*
LISTEN 3587/java
tcp0 0 10.0.1.1:9042 0.0.0.0:*
LISTEN 3587/java
tcp0 0 10.0.1.1:7000 0.0.0.0:*
LISTEN 3587/java
tcp0 0 127.0.0.1:7199 127.0.0.1:52874
ESTABLISHED 3587/java
tcp0 0 10.0.1.1:7199 10.0.1.1:39650
ESTABLISHED 3587/java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
