[
https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15029271#comment-15029271
]
Paulo Motta commented on CASSANDRA-9748:
----------------------------------------
Actually the solution is much simpler than initially thought, and requires
little change in code and documentation.
The idea is to provide a new property {{listen_on_broadcast_address}}, which
should be enabled for who wants to listen on both {{listen_address}} and
{{broadcast_address}}, in network environments which do not automatically route
from the public to the private address, such as outside EC2. The benefits of
this approach is that the [multiple network interfaces
documentation|http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html]
requires little change, just a note asking users to enable
{{listen_on_broadcast_address}} property if they're using multiple physical
interfaces. Furthermore, no changes are required to
{{GossipingPropertyFileSnitch}} which already re-routes traffic from
{{broadcast_address}} to {{listen_address}} when the {{prefer_local}} is
enabled, so it's done.
Changing {{listen_address}} to receive a list wouldn't be a good idea because
it would require a change to system keyspace and associated schema migrations.
Furthermore there are other places around the code that use this property so
this would break them (and maybe drivers as well). Adding a
{{local_listen_address}} would require a different documentation for EC2 and
multiple physical network interfaces, and would add more complexity, since we
already have {{listen_address}} and {{broadcast_address}} exactly for this use
case already, so no need to introduce a third address in addition to those.
I created a [dtest|https://github.com/riptano/cassandra-dtest/pull/687] that
tests {{listen_on_broadcast_address}} and also the {{prefer_local}} property of
{{GossipingPropertyFileSnitch}}.
The change is so small and independent that I think we can include it in 2.1.
I'm not sure if it should go into 3.0.1/3.1, but submitted tests anyway, so
will let the committer/reviewer decide.
There was another user recently [on the mailing
list|https://www.mail-archive.com/[email protected]/msg44987.html]
unable to setup multiple network interfaces due to this limitation, so I think
it has non-negligible demand.
||2.1||2.2||3.0||trunk||dtest||
|[branch|https://github.com/apache/cassandra/compare/cassandra-2.1...pauloricardomg:9478-2.1-v2]|[branch|https://github.com/apache/cassandra/compare/cassandra-2.2...pauloricardomg:9478-2.2-v2]|[branch|https://github.com/apache/cassandra/compare/cassandra-3.0...pauloricardomg:9478-3.0-v2]|[branch|https://github.com/apache/cassandra/compare/trunk...pauloricardomg:9478-trunk-v2]|[branch|https://github.com/riptano/cassandra-dtest/compare/master...pauloricardomg:9748]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-2.1-v2-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-2.2-v2-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-3.0-v2-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-trunk-v2-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-2.1-v2-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-2.2-v2-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-3.0-v2-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-9478-trunk-v2-dtest/lastCompletedBuild/testReport/]|
> Can't see other nodes when using multiple network interfaces
> ------------------------------------------------------------
>
> Key: CASSANDRA-9748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9748
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
> Environment: Cassandra 2.0.16; multi-DC configuration
> Reporter: Roman Bielik
> Assignee: Paulo Motta
> Priority: Minor
> Labels: docs-impacting
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
> Attachments: system_node1.log, system_node2.log
>
>
> The idea is to setup a multi-DC environment across 2 different networks based
> on the following configuration recommendations:
> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html
> Each node has 2 network interfaces. One used as a private network (DC1:
> 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all
> nodes can see each other (this one has a higher latency).
> Using the following settings in cassandra.yaml:
> *seeds:* public IP (same as used in broadcast_address)
> *listen_address:* private IP
> *broadcast_address:* public IP
> *rpc_address:* 0.0.0.0
> *endpoint_snitch:* GossipingPropertyFileSnitch
> _(tried different combinations with no luck)_
> No firewall and no SSL/encryption used.
> The problem is that nodes do not see each other (a gossip problem I guess).
> The nodetool ring/status shows only the local node but not the other ones
> (even from the same DC).
> When I set listen_address to public IP, then everything works fine, but that
> is not the required configuration.
> _Note: Not using EC2 cloud!_
> netstat -anp | grep -E "(7199|9160|9042|7000)"
> tcp 0 0 0.0.0.0:7199 0.0.0.0:*
> LISTEN 3587/java
> tcp 0 0 10.0.1.1:9160 0.0.0.0:*
> LISTEN 3587/java
> tcp 0 0 10.0.1.1:9042 0.0.0.0:*
> LISTEN 3587/java
> tcp 0 0 10.0.1.1:7000 0.0.0.0:*
> LISTEN 3587/java
> tcp 0 0 127.0.0.1:7199 127.0.0.1:52874
> ESTABLISHED 3587/java
> tcp 0 0 10.0.1.1:7199 10.0.1.1:39650
> ESTABLISHED 3587/java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
