Re: Slowness in C* cluster after implementing multiple network interface configuration.

2017-05-24 Thread Carlos Rolo 
It might be a bug.
Cassandra, AFAIK, scans those files for changes and updates the topology
(So you don't need a restart if you change the files). It might be the case
that the absence of the file, is still noticed by Cassandra even if it is
not really used.

I can do a small test to confirm, if so, it is a question of "expected
behaviour" (as in, always leave the file there) vs Bug (It shouldn't care
for files it doesn't use).

If you can always reproduce, feel free to Open a JIRA.

Thanks for the description.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant / Datastax Certified Architect / Cassandra MVP

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
*linkedin.com/in/carlosjuzarterolo
<http://linkedin.com/in/carlosjuzarterolo>*
Mobile: +351 918 918 100
www.pythian.com

On Wed, May 24, 2017 at 8:12 AM, Prakash Chauhan <
prakash.chau...@ericsson.com> wrote:

> Hi All,
>
>
>
> We have a new observation.
>
>
>
> Earlier for implementing multiple network interfaces, we were deleting
> *cassandra-topologies.properties* in the last step (Steps are mentioned
> in mail trail).
>
> The rationale was that because we are using altogether a new
> endpoint_snitch , we don’t require cassandra-topologies.properties file
> anymore.
>
>
>
> Now we have observed that if we don’t delete cassandra-topologies.properties,
> the slowness is not there in the cluster (Even with multiple restarts)
>
>
>
> Is there some relationship between *GossipingPropertyFileSnitch* and
> *cassandra-topologies.properties* ?
>
>
>
> As per my knowledge,  *cassandra-topologies.properties* file is only used
> as a fallback while doing snitch migration. If that’s the case, why does
> Cassandra becomes slow with time ( and after doing multiple restarts )
> after deleting cassandra-topologies.properties ?
>
>
>
>
>
>
>
>
>
> Regards,
>
> Prakash Chauhan.
>
>
>
> *From:* Cogumelos Maravilha [mailto:cogumelosmaravi...@sapo.pt]
> *Sent:* Wednesday, May 24, 2017 12:15 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Slowness in C* cluster after implementing multiple network
> interface configuration.
>
>
>
> Hi,
>
> I never used version 2.0.x but I think port 7000 isn't enough.
>
> Try enable:
>
> 7000 inter-node
>
> 7001 SSL inter-node
>
> 9042 CQL
>
> 9160 Thrift is enable in that version
>
>
>
> And
>
> In Cassandra.yaml, add property “broadcast_address”.  = local ipv4
>
> In Cassandra.yaml, change “listen_address” to private IP. = local ipv4
>
>
>
> As a starting point.
>
>
>
> Cheers.
>
>
>
> On 22-05-2017 12:36, Prakash Chauhan wrote:
>
> Hi All ,
>
>
>
> Need Help !!!
>
>
>
> *Setup Details:*
>
> Cassandra 2.0.14
>
> Geo Red setup
>
> · DC1 - 3 nodes
>
> · DC2 - 3 nodes
>
>
>
>
>
> We were trying to implement multiple network interfaces with Cassandra
> 2.0.14
>
> After doing all the steps mentioned in DataStax doc
> http://docs.datastax.com/en/archived/cassandra/2.0/
> cassandra/configuration/configMultiNetworks.html, we observed that nodes
> were not able to see each other (checked using nodetool status).
>
>
>
> To resolve this issue, we followed the comment
> <https://issues.apache.org/jira/browse/CASSANDRA-9748?focusedCommentId=14903515=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14903515>
> mentioned in the JIRA : CASSANDRA-9748
> <https://issues.apache.org/jira/browse/CASSANDRA-9748>
>
>
>
> Exact steps that we followed are :
>
> 
>
> *1.   *Stop Cassandra
>
> *2.   *Add rule to “iptables” to forward all packets on the public
> interface to the private interface.
>
>
>
> COMMAND: # iptables -t nat -A PREROUTING -p tcp -m tcp -d 
> --dport 7000 -j DNAT --to-destination :7000
>
>
>
> *3.   *In Cassandra.yaml, add property “broadcast_address”.
>
> *4.   *In Cassandra.yaml, change “listen_address” to private IP.
>
> *5.   *Clear the data from directory “peers”.
>
> *6.   *Change Snitch to GossipingPropertyFileSnitch.
>
> *7.   *Append following property to the file 
> “/etc/cassandra/conf/cassandra-env.sh”
> to purge gossip state.
>
> JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false"
>
>
>
> *8.   *Start Cassandra
>
> *9.   *After node has been started, remove following property from
> the file “/etc/cassandra/conf/cassandra-env.sh” (previously added in step
> 7)
>
> JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false"
>
> *10.   *Delete file “/etc/cassandra/conf/cassandra-topology.properties”
>
>
>
>
>
> Now We have an observation that after multiple restarts of Cassandra on
> multiple nodes, slowness is observed in the cluster.
>
> The problem gets resolved when we revert the steps mentioned above.
>
>
>
> *Do u think there is any step that can cause the problem ?*
>
> We are suspecting Step 2(iptable rule) but not very sure about it.
>
>
>
>
>
> Regards,
>
> Prakash Chauhan.
>
>
>

-- 


--

RE: Slowness in C* cluster after implementing multiple network interface configuration.

2017-05-24 Thread Prakash Chauhan 
Hi All,

We have a new observation.

Earlier for implementing multiple network interfaces, we were deleting 
cassandra-topologies.properties in the last step (Steps are mentioned in mail 
trail).
The rationale was that because we are using altogether a new endpoint_snitch , 
we don't require cassandra-topologies.properties file anymore.

Now we have observed that if we don't delete cassandra-topologies.properties, 
the slowness is not there in the cluster (Even with multiple restarts)

Is there some relationship between GossipingPropertyFileSnitch and 
cassandra-topologies.properties ?

As per my knowledge,  cassandra-topologies.properties file is only used as a 
fallback while doing snitch migration. If that's the case, why does Cassandra 
becomes slow with time ( and after doing multiple restarts ) after deleting 
cassandra-topologies.properties ?




Regards,
Prakash Chauhan.

From: Cogumelos Maravilha [mailto:cogumelosmaravi...@sapo.pt]
Sent: Wednesday, May 24, 2017 12:15 AM
To: user@cassandra.apache.org
Subject: Re: Slowness in C* cluster after implementing multiple network 
interface configuration.


Hi,

I never used version 2.0.x but I think port 7000 isn't enough.

Try enable:

7000 inter-node

7001 SSL inter-node

9042 CQL

9160 Thrift is enable in that version



And

In Cassandra.yaml, add property "broadcast_address".  = local ipv4

In Cassandra.yaml, change "listen_address" to private IP. = local ipv4



As a starting point.



Cheers.

On 22-05-2017 12:36, Prakash Chauhan wrote:
Hi All ,

Need Help !!!

Setup Details:
Cassandra 2.0.14
Geo Red setup

* DC1 - 3 nodes

* DC2 - 3 nodes


We were trying to implement multiple network interfaces with Cassandra 2.0.14
After doing all the steps mentioned in DataStax doc 
http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/configuration/configMultiNetworks.html,
 we observed that nodes were not able to see each other (checked using nodetool 
status).

To resolve this issue, we followed the 
comment<https://issues.apache.org/jira/browse/CASSANDRA-9748?focusedCommentId=14903515=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14903515>
 mentioned in the JIRA : 
CASSANDRA-9748<https://issues.apache.org/jira/browse/CASSANDRA-9748>

Exact steps that we followed are :


1.   Stop Cassandra

2.   Add rule to "iptables" to forward all packets on the public interface 
to the private interface.


COMMAND: # iptables -t nat -A PREROUTING -p tcp -m tcp -d  --dport 
7000 -j DNAT --to-destination :7000



3.   In Cassandra.yaml, add property "broadcast_address".

4.   In Cassandra.yaml, change "listen_address" to private IP.

5.   Clear the data from directory "peers".

6.   Change Snitch to GossipingPropertyFileSnitch.

7.   Append following property to the file 
"/etc/cassandra/conf/cassandra-env.sh" to purge gossip state.

JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false"



8.   Start Cassandra

9.   After node has been started, remove following property from the file 
"/etc/cassandra/conf/cassandra-env.sh" (previously added in step 7)

JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false"

10.   Delete file "/etc/cassandra/conf/cassandra-topology.properties"


Now We have an observation that after multiple restarts of Cassandra on 
multiple nodes, slowness is observed in the cluster.
The problem gets resolved when we revert the steps mentioned above.

Do u think there is any step that can cause the problem ?
We are suspecting Step 2(iptable rule) but not very sure about it.


Regards,
Prakash Chauhan.

Re: Slowness in C* cluster after implementing multiple network interface configuration.

2017-05-23 Thread Cogumelos Maravilha 
Hi,

I never used version 2.0.x but I think port 7000 isn't enough.

Try enable:

7000 inter-node

7001 SSL inter-node

9042 CQL

9160 Thrift is enable in that version


And

**In Cassandra.yaml, add property “broadcast_address”.  = local ipv4

**In Cassandra.yaml, change “listen_address” to private IP. = local ipv4


As a starting point.


Cheers.


On 22-05-2017 12:36, Prakash Chauhan wrote:
>
> Hi All ,
>
>  
>
> Need Help !!!
>
>  
>
> *Setup Details:*
>
> Cassandra 2.0.14
>
> Geo Red setup
>
> · DC1 - 3 nodes
>
> · DC2 - 3 nodes
>
>  
>
>  
>
> We were trying to implement multiple network interfaces with Cassandra
> 2.0.14
>
> After doing all the steps mentioned in DataStax doc
> http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/configuration/configMultiNetworks.html,
> we observed that nodes were not able to see each other (checked using
> nodetool status).
>
>  
>
> To resolve this issue, we followed the comment
> 
> mentioned in the JIRA : CASSANDRA-9748
> 
>
>  
>
> Exact steps that we followed are :
>
> 
>
> *1.   *Stop Cassandra
>
> *2.   *Add rule to “iptables” to forward all packets on the public
> interface to the private interface.
>
>  
>
> COMMAND: # iptables -t nat -A PREROUTING -p tcp -m tcp -d
>  --dport 7000 -j DNAT --to-destination :7000
>
>  
>
> *3.   *In Cassandra.yaml, add property “broadcast_address”.
>
> *4.   *In Cassandra.yaml, change “listen_address” to private IP.
>
> *5.   *Clear the data from directory “peers”.
>
> *6.   *Change Snitch to GossipingPropertyFileSnitch.
>
> *7.   *Append following property to the file
> “/etc/cassandra/conf/cassandra-env.sh” to purge gossip state.
>
> JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false"
>
>  
>
> *8.   *Start Cassandra
>
> *9.   *After node has been started, remove following property from
> the file “/etc/cassandra/conf/cassandra-env.sh” (previously added in
> step 7)
>
> JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false"
>
> *10.   *Delete file “/etc/cassandra/conf/cassandra-topology.properties”
>
>  
>
>  
>
> Now We have an observation that after multiple restarts of Cassandra
> on multiple nodes, slowness is observed in the cluster.
>
> The problem gets resolved when we revert the steps mentioned above.
>
>  
>
> *Do u think there is any step that can cause the problem ?*
>
> We are suspecting Step 2(iptable rule) but not very sure about it.
>
>  
>
>  
>
> Regards,
>
> Prakash Chauhan.
>

Slowness in C* cluster after implementing multiple network interface configuration.

2017-05-22 Thread Prakash Chauhan 
Hi All ,

Need Help !!!

Setup Details:
Cassandra 2.0.14
Geo Red setup

* DC1 - 3 nodes

* DC2 - 3 nodes


We were trying to implement multiple network interfaces with Cassandra 2.0.14
After doing all the steps mentioned in DataStax doc 
http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/configuration/configMultiNetworks.html,
 we observed that nodes were not able to see each other (checked using nodetool 
status).

To resolve this issue, we followed the 
comment
 mentioned in the JIRA : 
CASSANDRA-9748

Exact steps that we followed are :


1.   Stop Cassandra

2.   Add rule to "iptables" to forward all packets on the public interface 
to the private interface.


COMMAND: # iptables -t nat -A PREROUTING -p tcp -m tcp -d  --dport 
7000 -j DNAT --to-destination :7000



3.   In Cassandra.yaml, add property "broadcast_address".

4.   In Cassandra.yaml, change "listen_address" to private IP.

5.   Clear the data from directory "peers".

6.   Change Snitch to GossipingPropertyFileSnitch.

7.   Append following property to the file 
"/etc/cassandra/conf/cassandra-env.sh" to purge gossip state.

JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false"



8.   Start Cassandra

9.   After node has been started, remove following property from the file 
"/etc/cassandra/conf/cassandra-env.sh" (previously added in step 7)

JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false"

10.   Delete file "/etc/cassandra/conf/cassandra-topology.properties"


Now We have an observation that after multiple restarts of Cassandra on 
multiple nodes, slowness is observed in the cluster.
The problem gets resolved when we revert the steps mentioned above.

Do u think there is any step that can cause the problem ?
We are suspecting Step 2(iptable rule) but not very sure about it.


Regards,
Prakash Chauhan.