Re: Cassandra scale-out with no traffic on newly joined nodes

2020-09-14 Thread Sandeep Nethi
Small correction in cassandra driver usage,

Driver used: Spring boot (
org.springframework.boot:spring-boot-starter-data-cassandra)
Version: 2.3.2.RELEASE

Are there any differences between spring boot (I think it's based on
datastax cql driver) and Datastax core driver load balancing policies?

Thanks in advance?

On Tue, Sep 15, 2020 at 12:57 AM Sandeep Nethi 
wrote:

> Thanks Erick.
>
> We are using datastax java driver (
> https://docs.datastax.com/en/developer/java-driver/).
> Driver version number:  3.3.0
> C* version: 3.11.6
>
> Regards,
> Sandeep
>
>
> On Sat, Sep 12, 2020 at 2:41 PM Erick Ramirez 
> wrote:
>
>> That would be my last option to add a new host as contant point but as
>>> per my understanding cassandra should auto discover newly joined nodes and
>>> server or load balance connections automatically but it's not happening.
>>>
>>
>> Yes, that sounds right to me. Sorry, this fell off my radar due to
>> competing priorities. I'm going to reach out to the DataStax Drivers team.
>> In the meantime, can you tell me which driver + version and C* version?
>> Cheers!
>>
>


Re: Cassandra scale-out with no traffic on newly joined nodes

2020-09-14 Thread Sandeep Nethi
Thanks Erick.

We are using datastax java driver (
https://docs.datastax.com/en/developer/java-driver/).
Driver version number:  3.3.0
C* version: 3.11.6

Regards,
Sandeep


On Sat, Sep 12, 2020 at 2:41 PM Erick Ramirez 
wrote:

> That would be my last option to add a new host as contant point but as per
>> my understanding cassandra should auto discover newly joined nodes and
>> server or load balance connections automatically but it's not happening.
>>
>
> Yes, that sounds right to me. Sorry, this fell off my radar due to
> competing priorities. I'm going to reach out to the DataStax Drivers team.
> In the meantime, can you tell me which driver + version and C* version?
> Cheers!
>


Re: Cassandra scale-out with no traffic on newly joined nodes

2020-09-11 Thread Erick Ramirez
>
> That would be my last option to add a new host as contant point but as per
> my understanding cassandra should auto discover newly joined nodes and
> server or load balance connections automatically but it's not happening.
>

Yes, that sounds right to me. Sorry, this fell off my radar due to
competing priorities. I'm going to reach out to the DataStax Drivers team.
In the meantime, can you tell me which driver + version and C* version?
Cheers!


Re: Cassandra scale-out with no traffic on newly joined nodes

2020-09-11 Thread Sandeep Nethi
NetworkTopology strategy and murmer3 partitioner is being used.

Just an update, nodes were able to receive queries from coordinator nodes
but only old nodes are acting as coordinators.

As I explained before number of native connections on new nodes is very
less compared to old nodes.

How can i balance these connections without restarting client application,
any idea?


Thanks,
Sandeep

On Sat, 12 Sep 2020 at 5:02 AM, ZAIDI, ASAD  wrote:

>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Can you share please what replica placement strategy ( in keyspace
> definition) and   partitioner is used across nodes ?
>
>
> ~Asad
>
>
>
>
>
>
>
>
>
>
>
> *From:* Sandeep Nethi 
>
>
> *Sent:* Tuesday, September 8, 2020 1:12 AM
>
>
> *To:* user@cassandra.apache.org
>
>
> *Subject:* Re: Cassandra scale-out with no traffic on newly joined nodes
>
>
>
>
>
>
>
>
>
> That would be my last option to add a new host as contant point but as per
> my understanding cassandra should auto discover newly joined nodes and
> server or
>
> load balance connections automatically but it's not happening. So, i'm
> trying to understand what could be the root cause here.
>
>
>
>
>
>
>
>
>
>
>
>
>
> Another problem that I have noticed is, the number of native client
> connections on newly joined nodes vs old nodes is 1:7 and client
> connections are not balanced
>
> across nodes (old nodes are overloaded compared to newly joined nodes).
> Since i have a 6 node cluster with 3 racks, is it mandatory to have rack
> awareness in driver load balancer?
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Tue, Sep 8, 2020 at 6:02 PM manish khandelwal <
> manishkhandelwa...@gmail.com> wrote:
>
>
>
>
>
>
>
>
> Can you add new host as contact points and see if traffic lands on them or
> not?
>
>
>
>
> Also you can verify new nodes are added in system.peers of host name which
> you are giving as contact points
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Tue, Sep 8, 2020 at 11:27 AM Sandeep Nethi 
> wrote:
>
>
>
>
>
>
>
>
> Yes, all nodes are UN and no issues identified. Infact i could see some
> client connections on new nodes with telnet but not seeing any traffic.
>
>
>
>
>
>
>
>
>
>
>
> Cassandra version: 3.11.6
>
>
>
>
>
>
> Load Balancing policy used is default with no custom policies.
>
>
>
>
>
>
>
>
>
>
>
>
>
> Thanks,
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Tue, Sep 8, 2020 at 5:52 PM Erick Ramirez 
> wrote:
>
>
>
>
>
>
>
>
>
>
>
>
> That shouldn't be a problem for the control connection.
>
>
>
>
>
>
>
>
>
>
>
>
>
> I would suggest looking into the load-balancing policy configured on the
> driver. Also, are all the new nodes fully up and fully joined the cluster?
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>


RE: Cassandra scale-out with no traffic on newly joined nodes

2020-09-11 Thread ZAIDI, ASAD
Can you share please what replica placement strategy ( in keyspace definition) 
and   partitioner is used across nodes ?
~Asad



From: Sandeep Nethi 
Sent: Tuesday, September 8, 2020 1:12 AM
To: user@cassandra.apache.org
Subject: Re: Cassandra scale-out with no traffic on newly joined nodes

That would be my last option to add a new host as contant point but as per my 
understanding cassandra should auto discover newly joined nodes and server or 
load balance connections automatically but it's not happening. So, i'm trying 
to understand what could be the root cause here.

Another problem that I have noticed is, the number of native client connections 
on newly joined nodes vs old nodes is 1:7 and client connections are not 
balanced across nodes (old nodes are overloaded compared to newly joined 
nodes). Since i have a 6 node cluster with 3 racks, is it mandatory to have 
rack awareness in driver load balancer?

On Tue, Sep 8, 2020 at 6:02 PM manish khandelwal 
mailto:manishkhandelwa...@gmail.com>> wrote:
Can you add new host as contact points and see if traffic lands on them or not?
Also you can verify new nodes are added in system.peers of host name which you 
are giving as contact points

On Tue, Sep 8, 2020 at 11:27 AM Sandeep Nethi 
mailto:nethisande...@gmail.com>> wrote:
Yes, all nodes are UN and no issues identified. Infact i could see some client 
connections on new nodes with telnet but not seeing any traffic.

Cassandra version: 3.11.6
Load Balancing policy used is default with no custom policies.

Thanks,

On Tue, Sep 8, 2020 at 5:52 PM Erick Ramirez 
mailto:erick.rami...@datastax.com>> wrote:
That shouldn't be a problem for the control connection.

I would suggest looking into the load-balancing policy configured on the 
driver. Also, are all the new nodes fully up and fully joined the cluster?


Re: Cassandra scale-out with no traffic on newly joined nodes

2020-09-07 Thread Sandeep Nethi
That would be my last option to add a new host as contant point but as per
my understanding cassandra should auto discover newly joined nodes and
server or load balance connections automatically but it's not happening.
So, i'm trying to understand what could be the root cause here.

Another problem that I have noticed is, the number of native client
connections on newly joined nodes vs old nodes is 1:7 and client
connections are not balanced across nodes (old nodes are overloaded
compared to newly joined nodes). Since i have a 6 node cluster with 3
racks, is it mandatory to have rack awareness in driver load balancer?

On Tue, Sep 8, 2020 at 6:02 PM manish khandelwal <
manishkhandelwa...@gmail.com> wrote:

> Can you add new host as contact points and see if traffic lands on them or
> not?
> Also you can verify new nodes are added in system.peers of host name which
> you are giving as contact points
>
> On Tue, Sep 8, 2020 at 11:27 AM Sandeep Nethi 
> wrote:
>
>> Yes, all nodes are UN and no issues identified. Infact i could see some
>> client connections on new nodes with telnet but not seeing any traffic.
>>
>> Cassandra version: 3.11.6
>> Load Balancing policy used is default with no custom policies.
>>
>> Thanks,
>>
>> On Tue, Sep 8, 2020 at 5:52 PM Erick Ramirez 
>> wrote:
>>
>>> That shouldn't be a problem for the control connection.
>>>
>>> I would suggest looking into the load-balancing policy configured on the
>>> driver. Also, are all the new nodes fully up and fully joined the cluster?
>>>



Re: Cassandra scale-out with no traffic on newly joined nodes

2020-09-07 Thread manish khandelwal
Can you add new host as contact points and see if traffic lands on them or
not?
Also you can verify new nodes are added in system.peers of host name which
you are giving as contact points

On Tue, Sep 8, 2020 at 11:27 AM Sandeep Nethi 
wrote:

> Yes, all nodes are UN and no issues identified. Infact i could see some
> client connections on new nodes with telnet but not seeing any traffic.
>
> Cassandra version: 3.11.6
> Load Balancing policy used is default with no custom policies.
>
> Thanks,
>
> On Tue, Sep 8, 2020 at 5:52 PM Erick Ramirez 
> wrote:
>
>> That shouldn't be a problem for the control connection.
>>
>> I would suggest looking into the load-balancing policy configured on the
>> driver. Also, are all the new nodes fully up and fully joined the cluster?
>>
>>>


Re: Cassandra scale-out with no traffic on newly joined nodes

2020-09-07 Thread Sandeep Nethi
Yes, all nodes are UN and no issues identified. Infact i could see some
client connections on new nodes with telnet but not seeing any traffic.

Cassandra version: 3.11.6
Load Balancing policy used is default with no custom policies.

Thanks,

On Tue, Sep 8, 2020 at 5:52 PM Erick Ramirez 
wrote:

> That shouldn't be a problem for the control connection.
>
> I would suggest looking into the load-balancing policy configured on the
> driver. Also, are all the new nodes fully up and fully joined the cluster?
>
>>


Re: Cassandra scale-out with no traffic on newly joined nodes

2020-09-07 Thread Erick Ramirez
That shouldn't be a problem for the control connection.

I would suggest looking into the load-balancing policy configured on the
driver. Also, are all the new nodes fully up and fully joined the cluster?

>


Re: Cassandra scale-out with no traffic on newly joined nodes

2020-09-07 Thread Sandeep Nethi
Thanks Erick.

Just to confirm, my application connection string is using host address
details and not VIP/some other load balancer in between.

So, inorder for my application to successfully establish control connection
and be able to read system related tables to learn about topology
automatically, is it required to give required permissions for each
application user on system keyspace?

Currently my application users don't have permissions on system keyspace?
Is this could be the cause?

Thanks,

On Tue, Sep 8, 2020 at 2:23 PM Erick Ramirez 
wrote:

> If I were to guess I'd say that your app is hitting a hardware/software
> load-balancer or DNS VIP or DNS round-robin to get to the cluster and so
> it's only routing traffic to the old nodes.
>
> If this is the case, I would recommend you use the built-in load-balancing
> policy on the C* drivers instead. I've explained the reasons in detail on
> https://community.datastax.com/questions/7702/ and
> https://community.datastax.com/questions/4390/. Cheers!
>
>>


Re: Cassandra scale-out with no traffic on newly joined nodes

2020-09-07 Thread Erick Ramirez
If I were to guess I'd say that your app is hitting a hardware/software
load-balancer or DNS VIP or DNS round-robin to get to the cluster and so
it's only routing traffic to the old nodes.

If this is the case, I would recommend you use the built-in load-balancing
policy on the C* drivers instead. I've explained the reasons in detail on
https://community.datastax.com/questions/7702/ and
https://community.datastax.com/questions/4390/. Cheers!

>