[jira] [Commented] (CASSANDRA-9747) JNA makes Cassandra slower
[ https://issues.apache.org/jira/browse/CASSANDRA-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15087103#comment-15087103 ] Roman Bielik commented on CASSANDRA-9747: - Hi, I have performed testing on HP Gen9 servers (2 node installation) under heavy load with various HW and Cassandra configuration. For some reason the combination of Cassandra and JNA performed worse. I could see high IO waits (using 'top') and also my test application got worse results (in matter of response times and overall performance). I have no idea how this is possible. The test performed roughly 2*6500 updates/s (writes+deletes) and 1500 reads/s over 5 million records using only primary key. > JNA makes Cassandra slower > -- > > Key: CASSANDRA-9747 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9747 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.0.14; JNA 4.x; 32-core CPU; 64GB RAM (8GB > heap); 2 physical HDDs used as commit and data; Java 1.7 >Reporter: Roman Bielik >Priority: Minor > > When JNA is loaded (as recommended in the production settings), it has > negative effect on performance, especially when used with HDD => it > significantly increases the disk IO wait. > It looks like JNA works fine only when used in combination with SSD disk. > Otherwise it is better not to use JNA at all. This is very unexpected > behavior, can you please analyse? > Test description: > Roughly 50% reads and 50% writes of about 1.5kB records using Thrift > interface. > Test results (sorted by performance): > *HDD + JNA* > Worst performance. High IO wait. > *HDD, no JNA* > Good performance. Low IO wait, but the Cassandra CPU usage is quite high. > *SSD + JNA* > Seems to provide the best performance. Low IO wait. Low Cassandra CPU usage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037592#comment-15037592 ] Roman Bielik commented on CASSANDRA-9748: - Thank you very much, the fix seems to be working fine. Tested on C2.2. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta >Priority: Minor > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035929#comment-15035929 ] Roman Bielik commented on CASSANDRA-9748: - Hi Paulo, which git branch can I use? I tried this one: https://github.com/apache/cassandra/tree/204bd3af266c36d1c83d8cedb5daaa8124342178 but there seems to be a problem with some ANSI characters during the compilation. Thank you. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta >Priority: Minor > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[
https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036060#comment-15036060
]
Roman Bielik commented on CASSANDRA-9748:
-
Yeah, I can test on both 2.1 or 2.2 and both fail with the same errors.
Compiled under CentOS 6 + Java 1.7.
Perhaps I'm doing something wrong - the last time I tried your branch as
instructed it worked fine.
[javac]
/cassandra/cassandra-256bd77fa835486fab6bce9583e55a6a4ae9aa0a/src/java/org/apache/cassandra/utils/CoalescingStrategies.java:172:
error: unmappable character for encoding ASCII
[javac] logger.info(toString() + " gap " +
TimeUnit.NANOSECONDS.toMicros(averageGap) + "??s");
[javac]
^
[javac] 4 errors
[javac]
/cassandra/cassandra-204bd3af266c36d1c83d8cedb5daaa8124342178/test/unit/org/apache/cassandra/cql3/validation/operations/UpdateTest.java:57:
error: unmappable character for encoding ASCII
[javac]row(ByteBufferUtil.bytes("??"),
ByteBufferUtil.bytes("??"), set(ByteBufferUtil.bytes("??")),
ByteBufferUtil.bytes("??")));
[javac]
> Can't see other nodes when using multiple network interfaces
>
>
> Key: CASSANDRA-9748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9748
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
> Environment: Cassandra 2.0.16; multi-DC configuration
>Reporter: Roman Bielik
>Assignee: Paulo Motta
>Priority: Minor
> Labels: docs-impacting
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
> Attachments: system_node1.log, system_node2.log
>
>
> The idea is to setup a multi-DC environment across 2 different networks based
> on the following configuration recommendations:
> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html
> Each node has 2 network interfaces. One used as a private network (DC1:
> 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all
> nodes can see each other (this one has a higher latency).
> Using the following settings in cassandra.yaml:
> *seeds:* public IP (same as used in broadcast_address)
> *listen_address:* private IP
> *broadcast_address:* public IP
> *rpc_address:* 0.0.0.0
> *endpoint_snitch:* GossipingPropertyFileSnitch
> _(tried different combinations with no luck)_
> No firewall and no SSL/encryption used.
> The problem is that nodes do not see each other (a gossip problem I guess).
> The nodetool ring/status shows only the local node but not the other ones
> (even from the same DC).
> When I set listen_address to public IP, then everything works fine, but that
> is not the required configuration.
> _Note: Not using EC2 cloud!_
> netstat -anp | grep -E "(7199|9160|9042|7000)"
> tcp0 0 0.0.0.0:71990.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9160 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9042 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:7000 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 127.0.0.1:7199 127.0.0.1:52874
> ESTABLISHED 3587/java
> tcp0 0 10.0.1.1:7199 10.0.1.1:39650
> ESTABLISHED 3587/java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036300#comment-15036300 ] Roman Bielik commented on CASSANDRA-9748: - Thank you, that worked like a charm :) I'll let you know the test results tomorrow. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta >Priority: Minor > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029903#comment-15029903 ] Roman Bielik commented on CASSANDRA-9748: - Thank you, for me this feature is really important and I would be glad if it could be backported to the 3.x branch as well (for future compatibility). > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta >Priority: Minor > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029980#comment-15029980 ] Roman Bielik commented on CASSANDRA-9748: - Will do - early next week and will let you know the results. Thank you! > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta >Priority: Minor > Labels: docs-impacting > Fix For: 2.1.x, 2.2.x, 3.0.x > > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953165#comment-14953165 ] Roman Bielik commented on CASSANDRA-9748: - Hi Paulo, thank you very much, the provided github version works absolutely well!! I used the suggested configuration and the nodes started smoothly seeing both nodes in a ring. I run few tests with my Thrift client and all looks good. :) > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905963#comment-14905963 ] Roman Bielik commented on CASSANDRA-9748: - Hi, I tried the port forwarding and the telnet to public IP port 7000 works OK, but I still get the "Unable to gossip..." error on the second node. Anyway the analysis seems perfectly reasonable, thank you. That would explain why in all tutorials the private/public IP configuration was always used in combination with EC2MultiRegionSnitch. Thank you, it would be great if the listen_address could be set to all interfaces (0.0.0.0) as it would probably solve this issue. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14902453#comment-14902453 ] Roman Bielik commented on CASSANDRA-9748: - Please find the systen.logs attached below. The configuration was following: Node 1 == seeds: "192.168.24.153" listen_address: 10.0.2.1 broadcast_address: 192.168.24.153 Gossipinfo: /192.168.24.153 generation:1442922252 heartbeat:298 STATUS:NORMAL,-9223372036854775708 INTERNAL_IP:10.0.2.1 DC:DC2 HOST_ID:443c4534-3b1c-469b-9abc-8ed8e19db09e RELEASE_VERSION:2.2.1 LOAD:93969.0 RPC_READY:true RPC_ADDRESS:10.0.2.1 NET_VERSION:9 SEVERITY:0.0 RACK:RAC1 SCHEMA:040e7755-de2c-3516-8cf0-84abcdece3f7 Node 2 == seeds: "192.168.24.153" listen_address: 10.0.2.2 broadcast_address: 192.168.24.154 Node did not start, startup exception: "Unable to gossip with any seeds" > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik >Assignee: Paulo Motta > Attachments: system_node1.log, system_node2.log > > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747206#comment-14747206 ] Roman Bielik commented on CASSANDRA-9748: - Retested with the suggested settings, but it had no effect. The result is still the same. How does "prefer_local" effect the gossip? I mean how does Cassandra identify which interface is local? > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14745299#comment-14745299 ] Roman Bielik commented on CASSANDRA-9748: - I have retested on Cassandra 2.2.1 and the results are pretty much the same. If my premise is correct, I need to use public_IP as seeds as this is the interface/network where different datacenters can connect to each other. I have tested all possible permutations of listen_address, broadcast_address and broadcast_rpc_address. The seeds were set to public IP and rpc_address to 0.0.0.0 in all tests. Results: listen_address - if set to private_IP, the ring showed only one node broadcast_address - if set to private_IP, there was always problem with gossip broadcast_rpc_address - it did not matter in terms of connection or ring, so I could use private IP here. To sum up: I still needed to use public IP in seeds, listen_address and broadcast_address in order to start a Cassandra cluster. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679714#comment-14679714 ] Roman Bielik commented on CASSANDRA-9748: - Yeah, the problem arises even with the first DC. It might be some configuration problem indeed. But I can't figure out what. I followed the recommended setting for seeds, listen and broadcast addresses plus tried different variations with no luck. OK, I will post a question using the suggested mailing list. Thank you for the tip. Can't see other nodes when using multiple network interfaces Key: CASSANDRA-9748 URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.0.16; multi-DC configuration Reporter: Roman Bielik The idea is to setup a multi-DC environment across 2 different networks based on the following configuration recommendations: http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html Each node has 2 network interfaces. One used as a private network (DC1: 10.0.1.x and DC2: 10.0.2.x). The second one a public network where all nodes can see each other (this one has a higher latency). Using the following settings in cassandra.yaml: *seeds:* public IP (same as used in broadcast_address) *listen_address:* private IP *broadcast_address:* public IP *rpc_address:* 0.0.0.0 *endpoint_snitch:* GossipingPropertyFileSnitch _(tried different combinations with no luck)_ No firewall and no SSL/encryption used. The problem is that nodes do not see each other (a gossip problem I guess). The nodetool ring/status shows only the local node but not the other ones (even from the same DC). When I set listen_address to public IP, then everything works fine, but that is not the required configuration. _Note: Not using EC2 cloud!_ netstat -anp | grep -E (7199|9160|9042|7000) tcp0 0 0.0.0.0:71990.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9160 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9042 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:7000 0.0.0.0:* LISTEN 3587/java tcp0 0 127.0.0.1:7199 127.0.0.1:52874 ESTABLISHED 3587/java tcp0 0 10.0.1.1:7199 10.0.1.1:39650 ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14653294#comment-14653294 ] Roman Bielik commented on CASSANDRA-9748: - No, each node uses its own public IP. So each seed belongs to a particular node. Can't see other nodes when using multiple network interfaces Key: CASSANDRA-9748 URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.0.16; multi-DC configuration Reporter: Roman Bielik The idea is to setup a multi-DC environment across 2 different networks based on the following configuration recommendations: http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html Each node has 2 network interfaces. One used as a private network (DC1: 10.0.1.x and DC2: 10.0.2.x). The second one a public network where all nodes can see each other (this one has a higher latency). Using the following settings in cassandra.yaml: *seeds:* public IP (same as used in broadcast_address) *listen_address:* private IP *broadcast_address:* public IP *rpc_address:* 0.0.0.0 *endpoint_snitch:* GossipingPropertyFileSnitch _(tried different combinations with no luck)_ No firewall and no SSL/encryption used. The problem is that nodes do not see each other (a gossip problem I guess). The nodetool ring/status shows only the local node but not the other ones (even from the same DC). When I set listen_address to public IP, then everything works fine, but that is not the required configuration. _Note: Not using EC2 cloud!_ netstat -anp | grep -E (7199|9160|9042|7000) tcp0 0 0.0.0.0:71990.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9160 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9042 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:7000 0.0.0.0:* LISTEN 3587/java tcp0 0 127.0.0.1:7199 127.0.0.1:52874 ESTABLISHED 3587/java tcp0 0 10.0.1.1:7199 10.0.1.1:39650 ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14653529#comment-14653529 ] Roman Bielik commented on CASSANDRA-9748: - Yeah this should be correct. Perhaps I misunderstood the question. I'm using public IPs in seeds configuration and the seeds list is the same on all nodes. The list contains seeds from both datacenters too. So in my case the seeds looked like: 192.168.24.151,192.168.24.152,192.168.24.153,192.168.24.154 on all nodes. Can't see other nodes when using multiple network interfaces Key: CASSANDRA-9748 URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.0.16; multi-DC configuration Reporter: Roman Bielik The idea is to setup a multi-DC environment across 2 different networks based on the following configuration recommendations: http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html Each node has 2 network interfaces. One used as a private network (DC1: 10.0.1.x and DC2: 10.0.2.x). The second one a public network where all nodes can see each other (this one has a higher latency). Using the following settings in cassandra.yaml: *seeds:* public IP (same as used in broadcast_address) *listen_address:* private IP *broadcast_address:* public IP *rpc_address:* 0.0.0.0 *endpoint_snitch:* GossipingPropertyFileSnitch _(tried different combinations with no luck)_ No firewall and no SSL/encryption used. The problem is that nodes do not see each other (a gossip problem I guess). The nodetool ring/status shows only the local node but not the other ones (even from the same DC). When I set listen_address to public IP, then everything works fine, but that is not the required configuration. _Note: Not using EC2 cloud!_ netstat -anp | grep -E (7199|9160|9042|7000) tcp0 0 0.0.0.0:71990.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9160 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9042 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:7000 0.0.0.0:* LISTEN 3587/java tcp0 0 127.0.0.1:7199 127.0.0.1:52874 ESTABLISHED 3587/java tcp0 0 10.0.1.1:7199 10.0.1.1:39650 ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14620125#comment-14620125 ] Roman Bielik commented on CASSANDRA-9748: - 10.0.1.x = DC1 subnet 10.0.2.x = DC2 subnet 192.168.24.x = shared (public) network, all nodes can see each other here *Node 1 (DC1)* IP: 192.168.24.151 (eth0), 10.0.1.1 (eth2) $ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface 10.0.1.00.0.0.0 255.255.255.0 U 0 00 eth2 192.168.24.00.0.0.0 255.255.252.0 U 0 00 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1002 00 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1003 00 eth2 0.0.0.0 192.168.24.254 0.0.0.0 UG0 00 eth0 $ ip route list 10.0.1.0/24 dev eth2 proto kernel scope link src 10.0.1.1 192.168.24.0/22 dev eth0 proto kernel scope link src 192.168.24.151 169.254.0.0/16 dev eth0 scope link metric 1002 169.254.0.0/16 dev eth2 scope link metric 1003 default via 192.168.24.254 dev eth0 *Node 2 (DC1)* IP: 192.168.24.152 (eth0), 10.0.1.2 (eth2) $ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface 10.0.1.00.0.0.0 255.255.255.0 U 0 00 eth2 192.168.24.00.0.0.0 255.255.252.0 U 0 00 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1002 00 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1003 00 eth2 0.0.0.0 192.168.24.254 0.0.0.0 UG0 00 eth0 $ ip route list 10.0.1.0/24 dev eth2 proto kernel scope link src 10.0.1.2 192.168.24.0/22 dev eth0 proto kernel scope link src 192.168.24.152 169.254.0.0/16 dev eth0 scope link metric 1002 169.254.0.0/16 dev eth2 scope link metric 1003 default via 192.168.24.254 dev eth0 *Node3 (DC2)* IP: 192.168.24.153 (eth0), 10.0.2.1 (eth2) $ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface 10.0.2.00.0.0.0 255.255.255.0 U 0 00 eth2 192.168.24.00.0.0.0 255.255.252.0 U 0 00 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1002 00 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1003 00 eth2 0.0.0.0 192.168.24.254 0.0.0.0 UG0 00 eth0 $ ip route list 10.0.2.0/24 dev eth2 proto kernel scope link src 10.0.2.1 192.168.24.0/22 dev eth0 proto kernel scope link src 192.168.24.153 169.254.0.0/16 dev eth0 scope link metric 1002 169.254.0.0/16 dev eth2 scope link metric 1003 default via 192.168.24.254 dev eth0 *Node4 (DC2)* IP: 192.168.24.154 (eth0), 10.0.2.2 (eth2) $ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface 10.0.2.00.0.0.0 255.255.255.0 U 0 00 eth2 192.168.24.00.0.0.0 255.255.252.0 U 0 00 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1002 00 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1003 00 eth2 0.0.0.0 192.168.24.254 0.0.0.0 UG0 00 eth0 $ ip route list 10.0.2.0/24 dev eth2 proto kernel scope link src 10.0.2.2 192.168.24.0/22 dev eth0 proto kernel scope link src 192.168.24.154 169.254.0.0/16 dev eth0 scope link metric 1002 169.254.0.0/16 dev eth2 scope link metric 1003 default via 192.168.24.254 dev eth0 Can't see other nodes when using multiple network interfaces Key: CASSANDRA-9748 URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.0.16; multi-DC configuration Reporter: Roman Bielik The idea is to setup a multi-DC environment across 2 different networks based on the following configuration recommendations: http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html Each node has 2 network interfaces. One used as a private network (DC1: 10.0.1.x and DC2: 10.0.2.x). The second one a public network where all nodes can see each other (this one has a higher latency). Using the following settings in cassandra.yaml: *seeds:* public IP (same as used in broadcast_address) *listen_address:* private IP *broadcast_address:* public IP *rpc_address:* 0.0.0.0 *endpoint_snitch:* GossipingPropertyFileSnitch _(tried different combinations with no luck)_ No firewall and no SSL/encryption used. The problem is that nodes do not see each other (a gossip problem I guess). The
[jira] [Created] (CASSANDRA-9747) JNA makes Cassandra slower
Roman Bielik created CASSANDRA-9747: --- Summary: JNA makes Cassandra slower Key: CASSANDRA-9747 URL: https://issues.apache.org/jira/browse/CASSANDRA-9747 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.0.14; JNA 4.x; 32-core CPU; 64GB RAM (8GB heap); 2 physical HDDs used as commit and data; Java 1.7 Reporter: Roman Bielik Priority: Minor When JNA is loaded (as recommended in the production settings), it has negative effect on performance, especially when used with HDD = it significantly increases the disk IO wait. It looks like JNA works fine only when used in combination with SSD disk. Otherwise it is better not to use JNA at all. This is very unexpected behavior, can you please analyse? Test description: Roughly 50% reads and 50% writes of about 1.5kB records using Thrift interface. Test results (sorted by performance): *HDD + JNA* Worst performance. High IO wait. *HDD, no JNA* Good performance. Low IO wait, but the Cassandra CPU usage is quite high. *SSD + JNA* Seems to provide the best performance. Low IO wait. Low Cassandra CPU usage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
Roman Bielik created CASSANDRA-9748: --- Summary: Can't see other nodes when using multiple network interfaces Key: CASSANDRA-9748 URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.0.16; multi-DC configuration Reporter: Roman Bielik The idea is to setup a multi-DC environment across 2 different networks based on the following configuration recommendations: http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html Each node has 2 network interfaces. One used as a private network (DC1: 10.0.1.x and DC2: 10.0.2.x). The second one a public network where all nodes can see each other (this one has a higher latency). Using the following settings in cassandra.yaml: *seeds:* public IP (same as used in broadcast_address) *listen_address:* private IP *broadcast_address:* public IP *rpc_address:* 0.0.0.0 *endpoint_snitch:* GossipingPropertyFileSnitch _(tried different combinations with no luck)_ No firewall and no SSL/encryption used. The problem is that nodes do not see each other (a gossip problem I guess). The nodetool ring/status shows only the local node but not the other ones (even from the same DC). When I set listen_address to public IP, then everything works fine, but that is not the required configuration. _Note: Not using EC2 cloud!_ netstat -anp | grep -E (7199|9160|9042|7000) tcp0 0 0.0.0.0:71990.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9160 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:9042 0.0.0.0:* LISTEN 3587/java tcp0 0 10.0.1.1:7000 0.0.0.0:* LISTEN 3587/java tcp0 0 127.0.0.1:7199 127.0.0.1:52874 ESTABLISHED 3587/java tcp0 0 10.0.1.1:7199 10.0.1.1:39650 ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
