RE: Connection reset by peer

[Hanauer, Arnulf, Vodacom South Africa (External)]

Thanks to both Erik/Shaun for your responses,

Both your explanations are plausible in my scenario, this is what I have done 
subsequently which seems to have improved the situation,



  1.  The cluster was very busy trying to run repairs/sync the new replicas 
(about 350GB)  in the new DC (Gossip was temporarily marking down the source 
nodes at different points in time)

  *   Disabled Reaper, stopped all validation/repairs



  1.  I removed the new replica’s to stop any potential read_repair across the 
WAN

  *   I will recreate the replica’s over the weekend during quiet time & run 
the repair to sync



  1.  The network ping response time was quite high around 10-15msec at error 
times

  *   This dropped to under 1ms later in the day when some jobs were rerun 
successfully



  1.  I will apply some of the recommended TCP_KEEPALIVE settings Shaun pointed 
me to



Last question: In all your experiences, how high can the latency (simple ping 
response times go) before it becomes a problem? (Obviously the lower the better 
but is there some sort of cut off/formula where problems can be expected 
intermittently like the connection resets)




Kind regards

Arnulf Hanauer


From: Erick Ramirez 
Sent: Thursday, 13 February 2020 03:10
To: user@cassandra.apache.org
Subject: Re: Connection reset by peer

I generally see these exceptions when the cluster is overloaded. I think what's 
happening is that when the app/driver sends a read request, the coordinator 
takes a long time to respond because the nodes are busy serving other requests. 
The driver gives up (client-side timeout reached) and the socket is closed. 
Meanwhile, the coordinator eventually gets results from replicas and tries to 
send the response back to the app/driver but can't because the connection is no 
longer there. Does this scenario sound plausible for your cluster?


Erick Ramirez  |  Developer Relations

erick.rami...@datastax.com | 
datastax.com
[https://lh4.googleusercontent.com/GivrE4j_1bWvQnZP67Zpa5eomhEeKON-X6kFljLxDatL7QPL_aineBJzM_rXzrqNQkENnZt7KyXLROlLTHuMM3OFNlZ8IrW-adjXKRiD7ojG6OjjFoLio3HbKwVwXt7_Qna02H8I][https://lh6.googleusercontent.com/0juOULc-Qhs6qzVY5mN0tzIMZ4w17Jv2fiV5xboewGBH0SFiEwV0uPTO_W5OwGr0jCOXmoJLBq1aNLsr1oChLMgJNvNt1e4bHxO2KJUK-iagQ4jw9SiuTMmpktVSfygdLS_vQe6v][https://lh5.googleusercontent.com/IdGeRVBWRf50wPOny50XQ3O0rtkebOh8e2D9DCanVuy-f3a-wpI3PpQJnGtVFL5aHPOwm4hsginvqhQfTXnP_XT_8fuQWS6Mt0KKRFkRANDhS22T3UiXpGfBkMHJxy48ZQJFaXsZ][https://lh4.googleusercontent.com/PbPMGIQsTltjGio5a_e7dp35l6ysZMG_ib69EUHmIvbXHXzRkrNKNMfMR8uwSS1AAoQaG6xn96PH-L1wLQE8FBLSjN_g10Q8y0n1Tp5SYtKO3L1JDN_T73HgSSQJayqn7YMTFXn-][https://lh5.googleusercontent.com/Rnk5QTWTovfX-z1uRr0FQjt17VnMURyI8rDCi4rTJUY5lnX-QevuQWTFa39GS9fJCMqP0SXSkCLtKf064p0-59f80PmA2hZRqGRFFOlZlbJzXv2EevvdbeKYFq4s9g5zzP54KKQB]

[https://datastax.com/sites/default/files/content/graphics/email/datastax-email-signature-2019.jpg]


On Wed, 12 Feb 2020 at 21:13, Hanauer, Arnulf, Vodacom South Africa (External) 
mailto:arnulf.hana...@vcontractor.co.za>> 
wrote:
Hi Cassandra folks,

We are getting a lot of these errors and transactions are timing out and I was 
wondering if this can be caused by Cassandra itself or if this is a genuine 
Linux network issue only. The client job reports Cassandra node down after this 
occurs but I suspect this is due to the connection failure – need some 
clarification as where to go look for a solution.


INFO  [epollEventLoopGroup-2-10] 2020-02-12 11:53:42,748 Message.java:623 - 
Unexpected exception during request; channel = [id: 0x8a3e6831, 
L:/10.132.65.152:9042 - 
R:/10.132.11.15:48020]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]

INFO  [epollEventLoopGroup-2-15] 2020-02-12 11:42:46,871 Message.java:623 - 
Unexpected exception during request; channel = [id: 0xa071f1c8, 
L:/10.132.65.152:9042 - 
R:/10.132.11.15:45134]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]


Source and Destination IP addresses are in the same DC (LAN).

I did recycle all the Cassandra services on all the nodes in both clusters but 
the problem remains.

The only change made recently was the adding of replicas in the second DC for 
the keyspace that is being written to when these messages occur (not had a 
chance to run a full repair 

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Erick Ramirez]

Yes to the steps. The only thing I would add is to run a nodetool drain
before shutting C* down so all mutations are flushed to SSTables and there
won't be any commit logs to replay on startup.

Also, the usual "backup your cluster and configuration files" boilerplate
applies. 

>

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Sergio]

Should I follow the steps above right?
Thanks Erick!

On Wed, Feb 12, 2020, 6:58 PM Erick Ramirez 
wrote:

> In case you have an hybrid situation with 3.11.3 , 3.11.4 and 3.11.5 that
>> it is working and it is in production what do you recommend?
>
>
> You shouldn't end up in this mixed-version situation at all. I would
> highly recommend you upgrade all the nodes to 3.11.5 or whatever the latest
> version is installed on the nodes. Mixed-versions isn't a tested or
> supported scenario and the cluster's behaviour can be unpredictable. The
> behaviour might not be catastrophic but you don't want to be the one who
> discovers some exotic bug that arises out of that configuration. Cheers!
>
>>

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

Thanks Eric ...
This is helpful...


On Wed, 12 Feb 2020 at 17:46, Erick Ramirez 
wrote:

> There shouldn't be any negative impact from dropping MVs and there's
> certainly no risk to the base table if that is your concern. All it will do
> is remove all the data in the respective views plus drop any pending view
> mutations from the batch log. If anything, you should see some performance
> gain since updates to the base table will only trigger 4 view updates
> instead of the previous 11. Cheers!
>
> Erick Ramirez  |  Developer Relations
>
> erick.rami...@datastax.com | datastax.com 
> 
>  
>  
>
> 
>
>
>
> On Thu, 13 Feb 2020 at 04:26, Surbhi Gupta 
> wrote:
>
>> Hi,
>>
>> So application team created 11 materialized views on a base table in
>> production and we need to drop 7 Materialized views as they are not in use.
>> Wanted to understand the impact of dropping the materialized views.
>> We are on Cassandra 3.11.1 , multi datacenter with replication factor of
>> 3 in each datacenter.
>> We are using LOCAL_QUORUM for write consistency and LOCAL_ONE for read
>> consistency.
>>
>> Any thoughts or suggestion to keep in mind before dropping the
>> Materialized views.
>>
>> Thanks
>> Surbhi
>>
>>
>>
>>

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Erick Ramirez]

>
> In case you have an hybrid situation with 3.11.3 , 3.11.4 and 3.11.5 that
> it is working and it is in production what do you recommend?


You shouldn't end up in this mixed-version situation at all. I would highly
recommend you upgrade all the nodes to 3.11.5 or whatever the latest
version is installed on the nodes. Mixed-versions isn't a tested or
supported scenario and the cluster's behaviour can be unpredictable. The
behaviour might not be catastrophic but you don't want to be the one who
discovers some exotic bug that arises out of that configuration. Cheers!

>

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Sergio]

Thanks everyone!

In case you have an hybrid situation with 3.11.3 , 3.11.4 and 3.11.5 that
it is working and it is in production what do you recommend?



On Wed, Feb 12, 2020, 5:55 PM Erick Ramirez 
wrote:

> So unless the sstable format has not been changed I can avoid to do that.
>
>
> Just to reinforce what Jon and Sean already said, the above assumption is
> dangerous. It is always best to follow the recommended upgrade procedure
> and mixed-versions is never a good idea unless you've received instructions
> from a qualified source to address a specific issue. But as Jon said, we
> wouldn't be on this mailing list otherwise. 
>
> Erick Ramirez  |  Developer Relations
>
> erick.rami...@datastax.com | datastax.com 
> 
>  
>  
>
> 
>
>>

Re: Cassandra Encyrption between DC

[Erick Ramirez]

I've just seen your questions on ASF Slack and didn't immediately make the
connection that this post in the mailing list is one and the same. I
understand what you're doing now -- you have an existing DC with no
encryption and you want to add a new DC with encryption enabled but don't
want the downtime associated with enabling encryption on the existing DC.

As driftx, exlt, myself & co pointed out, there isn't a "transitional path"
of implementing it without downtime in the current (released) versions of
C*. Cheers!

>

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Erick Ramirez]

>
> So unless the sstable format has not been changed I can avoid to do that.


Just to reinforce what Jon and Sean already said, the above assumption is
dangerous. It is always best to follow the recommended upgrade procedure
and mixed-versions is never a good idea unless you've received instructions
from a qualified source to address a specific issue. But as Jon said, we
wouldn't be on this mailing list otherwise. 

Erick Ramirez  |  Developer Relations

erick.rami...@datastax.com | datastax.com 

 
 



>

Re: Consequences of dropping Materialized views

[Erick Ramirez]

There shouldn't be any negative impact from dropping MVs and there's
certainly no risk to the base table if that is your concern. All it will do
is remove all the data in the respective views plus drop any pending view
mutations from the batch log. If anything, you should see some performance
gain since updates to the base table will only trigger 4 view updates
instead of the previous 11. Cheers!

Erick Ramirez  |  Developer Relations

erick.rami...@datastax.com | datastax.com 

 
 





On Thu, 13 Feb 2020 at 04:26, Surbhi Gupta  wrote:

> Hi,
>
> So application team created 11 materialized views on a base table in
> production and we need to drop 7 Materialized views as they are not in use.
> Wanted to understand the impact of dropping the materialized views.
> We are on Cassandra 3.11.1 , multi datacenter with replication factor of 3
> in each datacenter.
> We are using LOCAL_QUORUM for write consistency and LOCAL_ONE for read
> consistency.
>
> Any thoughts or suggestion to keep in mind before dropping the
> Materialized views.
>
> Thanks
> Surbhi
>
>
>
>

Re: Cassandra Encyrption between DC

[Erick Ramirez]

>
> ... where dc-1 have encryption enabled and dc-2 does't have encryption?

... is there a way to specify encrypt within DC?


The quick answer to your question is no. But you've got me really curious
now because you have a very strange setup which makes no sense to me and
I'm hoping you could elaborate. Effectively you're saying:
(a) you trust the network between DC1 and DC2 so you don't encrypt intra-DC
comms,
(b) but you don't trust the network in DC2 so comms between nodes in DC2
are encrypted.

That doesn't make sense to me. Is DC2 in some untrusted DMZ?

This statement also begs the question:

... where existing dc don't have encryption between the nodes but the new
> DC have encryption enabled


How do you plan to bootstrap the new nodes? And if they're already
bootstrapped, are you trying to join 2 separate clusters together? Because
if you are, that's a very bad idea. You'll end up corrupting the schema
assuming you know enough to override the cluster_name in system.local. I
would strongly advise against it if that's what you're trying to achieve.
Cheers!

Erick Ramirez  |  Developer Relations

erick.rami...@datastax.com | datastax.com 

 
 





On Thu, 13 Feb 2020 at 02:54, Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Hello,
>
> Is there a way we can have a multi DC Cassandra cluster, where dc-1 have
> encryption enabled and dc-2 does't have encryption?
>
> I am trying to add a new DC to the existing cluster, where existing dc
> don't have encryption between the nodes but the new DC have encryption
> enabled?
>
> I see the below options, is there a way to specify encrypt within DC?
>
>- all - Encrypt all inter-node communications
>- none - No encryption
>- dc - Encrypt the traffic between the datacenters
>- rack - Encrypt the traffic between the racks
>
>

Re: Connection reset by peer

[Erick Ramirez]

I generally see these exceptions when the cluster is overloaded. I think
what's happening is that when the app/driver sends a read request, the
coordinator takes a long time to respond because the nodes are busy serving
other requests. The driver gives up (client-side timeout reached) and the
socket is closed. Meanwhile, the coordinator eventually gets results from
replicas and tries to send the response back to the app/driver but can't
because the connection is no longer there. Does this scenario sound
plausible for your cluster?

Erick Ramirez  |  Developer Relations

erick.rami...@datastax.com | datastax.com 

 
 





On Wed, 12 Feb 2020 at 21:13, Hanauer, Arnulf, Vodacom South Africa
(External)  wrote:

> Hi Cassandra folks,
>
>
>
> We are getting a lot of these errors and transactions are timing out and I
> was wondering if this can be caused by Cassandra itself or if this is a
> genuine Linux network issue only. The client job reports Cassandra node
> down after this occurs but I suspect this is due to the connection failure
> – need some clarification as where to go look for a solution.
>
>
>
>
>
> *INFO  [epollEventLoopGroup-2-10] 2020-02-12 11:53:42,748 Message.java:623
> - Unexpected exception during request; channel = [id: 0x8a3e6831,
> L:/10.132.65.152:9042  - R:/10.132.11.15:48020
> ]*
>
> *io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
> failed: Connection reset by peer*
>
> *at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown
> Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]*
>
>
>
> *INFO  [epollEventLoopGroup-2-15] 2020-02-12 11:42:46,871 Message.java:623
> - Unexpected exception during request; channel = [id: 0xa071f1c8,
> L:/10.132.65.152:9042  - R:/10.132.11.15:45134
> ]*
>
> *io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
> failed: Connection reset by peer*
>
> *at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown
> Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]*
>
>
>
>
>
> Source and Destination IP addresses are in the same DC (LAN).
>
>
>
> I did recycle all the Cassandra services on all the nodes in both clusters
> but the problem remains.
>
>
>
> The only change made recently was the adding of replicas in the second DC
> for the keyspace that is being written to when these messages occur (not
> had a chance to run a full repair yet to sync the replicas)
>
>
>
>
>
> FYI:
>
> Cassandra 3.11.2
>
> 5 Node cluster each in 2 DC’s
>
>
>
>
>
> Kind regards
> Arnulf Hanauer
>
>
>
>
>
>
>
>
>
>
> "This e-mail is sent on the Terms and Conditions that can be accessed by
> Clicking on this link https://webmail.vodacom.co.za/tc/default.html
> 
> "
>

RE: [EXTERNAL] Cassandra 3.11.X upgrades

[Durity, Sean R]

Ah - I should have looked it up! Thank you for fixing my mistake.

Sean Durity

-Original Message-
From: Michael Shuler 
Sent: Wednesday, February 12, 2020 3:17 PM
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Cassandra 3.11.X upgrades

On 2/12/20 12:58 PM, Durity, Sean R wrote:
> Check the readme.txt for any upgrade notes

Just a quick correction:

NEWS.txt (upgrade (and other important) notes)
CHANGES.txt (changelog with JIRAs)

This is why we list links to these two files in the release announcements.

--
Kind regards,
Michael

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Reid Pinchback]

Hi Sergio,

We have a production cluster with vnodes=4 that is a bit larger than that, so 
yes it is possible to do so.  That said, we aren’t wedded to vnodes=4 and are 
paying attention to discussions happening around the 4.0 work and mulling the 
possibility of shifting to 16.

Note though, we didn’t just pick 4 based on blind faith.  There was work done 
to evaluate the state of our token distribution before and after.  It’s pretty 
much true of all the settings we have for C* (or anything else for that 
matter).  We started from a lot of helpful articles and docs and mail threads 
and JIRA issues, but then we evaluated to see what we thought of the results.  
It was the only way for us to start building up some understanding of the 
details.

From: Sergio 
Reply-To: "user@cassandra.apache.org" 
Date: Wednesday, February 12, 2020 at 2:50 PM
To: "user@cassandra.apache.org" 
Subject: Re: [EXTERNAL] Cassandra 3.11.X upgrades

Message from External Sender
Thanks, everyone! @Jon 
https://lists.apache.org/thread.html/rd18814bfba487824ca95a58191f4dcdb86f15c9bb66cf2bcc29ddf0b%40%3Cuser.cassandra.apache.org%3E
I have a side response to something that looks to be controversial with the 
response from Anthony.
So is it safe to go to production in a 1TB cluster with vnodes = 4?
Do we need to follow these steps 
https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html?
 because from Anthony response what I got that this is just an example and 
vnodes = 4 it is not ready for production.
https://lists.apache.org/thread.html/r21cd99fa269076d186a82a8b466eb925681373302dd7aa6bb26e5bde%40%3Cuser.cassandra.apache.org%3E

Best,

Sergio


Il giorno mer 12 feb 2020 alle ore 11:42 Durity, Sean R 
mailto:sean_r_dur...@homedepot.com>> ha scritto:
>>A while ago, on my first cluster

Understatement used so effectively. Jon is a master.



On Wed, Feb 12, 2020 at 11:02 AM Sergio 
mailto:lapostadiser...@gmail.com>> wrote:
Thanks for your reply!

So unless the sstable format has not been changed I can avoid to do that.

Correct?

Best,

Sergio

On Wed, Feb 12, 2020, 10:58 AM Durity, Sean R 
mailto:sean_r_dur...@homedepot.com>> wrote:
Check the readme.txt for any upgrade notes, but the basic procedure is to:

  *   Verify that nodetool upgradesstables has completed successfully on all 
nodes from any previous upgrade
  *   Turn off repairs and any other streaming operations (add/remove nodes)
  *   Stop an un-upgraded node (seeds first, preferably)
  *   Install new binaries and configs on the down node
  *   Restart that node and make sure it comes up clean (it will function 
normally in the cluster – even with mixed versions)
  *   Repeat for all nodes
  *   Run upgradesstables on each node (as many at a time as your load will 
allow). Minor upgrades usually don’t require this step (only if the sstable 
format has changed), but it is good to check.
  *   NOTE: in most cases applications can keep running and will not notice 
much impact – unless the cluster is overloaded and a single node down causes 
impact.



Sean Durity – Staff Systems Engineer, Cassandra

From: Sergio mailto:lapostadiser...@gmail.com>>
Sent: Wednesday, February 12, 2020 11:36 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Cassandra 3.11.X upgrades

Hi guys!

How do you usually upgrade your cluster for minor version upgrades?

I tried to add a node with 3.11.5 version to a test cluster with 3.11.4 nodes.

Is there any restriction?

Best,

Sergio



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When 

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Michael Shuler]

On 2/12/20 12:58 PM, Durity, Sean R wrote:

Check the readme.txt for any upgrade notes


Just a quick correction:

NEWS.txt (upgrade (and other important) notes)
CHANGES.txt (changelog with JIRAs)

This is why we list links to these two files in the release announcements.

--
Kind regards,
Michael

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Sergio]

Thanks, everyone! @Jon
https://lists.apache.org/thread.html/rd18814bfba487824ca95a58191f4dcdb86f15c9bb66cf2bcc29ddf0b%40%3Cuser.cassandra.apache.org%3E

I have a side response to something that looks to be controversial with the
response from Anthony.
So is it safe to go to production in a 1TB cluster with vnodes = 4?
Do we need to follow these steps
https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html?
because
from Anthony response what I got that this is just an example and vnodes =
4 it is not ready for production.
https://lists.apache.org/thread.html/r21cd99fa269076d186a82a8b466eb925681373302dd7aa6bb26e5bde%40%3Cuser.cassandra.apache.org%3E

Best,

Sergio



Il giorno mer 12 feb 2020 alle ore 11:42 Durity, Sean R <
sean_r_dur...@homedepot.com> ha scritto:

> >>A while ago, on my first cluster
>
>
>
> Understatement used so effectively. Jon is a master.
>
>
>
>
>
>
>
> On Wed, Feb 12, 2020 at 11:02 AM Sergio  wrote:
>
> Thanks for your reply!
>
>
>
> So unless the sstable format has not been changed I can avoid to do that.
>
>
>
> Correct?
>
>
>
> Best,
>
>
>
> Sergio
>
>
>
> On Wed, Feb 12, 2020, 10:58 AM Durity, Sean R 
> wrote:
>
> Check the readme.txt for any upgrade notes, but the basic procedure is to:
>
>- Verify that nodetool upgradesstables has completed successfully on
>all nodes from any previous upgrade
>- Turn off repairs and any other streaming operations (add/remove
>nodes)
>- Stop an un-upgraded node (seeds first, preferably)
>- Install new binaries and configs on the down node
>- Restart that node and make sure it comes up clean (it will function
>normally in the cluster – even with mixed versions)
>- Repeat for all nodes
>- Run upgradesstables on each node (as many at a time as your load
>will allow). Minor upgrades usually don’t require this step (only if the
>sstable format has changed), but it is good to check.
>- NOTE: in most cases applications can keep running and will not
>notice much impact – unless the cluster is overloaded and a single node
>down causes impact.
>
>
>
>
>
>
>
> Sean Durity – Staff Systems Engineer, Cassandra
>
>
>
> *From:* Sergio 
> *Sent:* Wednesday, February 12, 2020 11:36 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Cassandra 3.11.X upgrades
>
>
>
> Hi guys!
>
> How do you usually upgrade your cluster for minor version upgrades?
>
> I tried to add a node with 3.11.5 version to a test cluster with 3.11.4
> nodes.
>
> Is there any restriction?
>
> Best,
>
> Sergio
>
>
> --
>
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>
>
> --
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>

RE: [EXTERNAL] Cassandra 3.11.X upgrades

[Durity, Sean R]

>>A while ago, on my first cluster

Understatement used so effectively. Jon is a master.



On Wed, Feb 12, 2020 at 11:02 AM Sergio 
mailto:lapostadiser...@gmail.com>> wrote:
Thanks for your reply!

So unless the sstable format has not been changed I can avoid to do that.

Correct?

Best,

Sergio

On Wed, Feb 12, 2020, 10:58 AM Durity, Sean R 
mailto:sean_r_dur...@homedepot.com>> wrote:
Check the readme.txt for any upgrade notes, but the basic procedure is to:

  *   Verify that nodetool upgradesstables has completed successfully on all 
nodes from any previous upgrade
  *   Turn off repairs and any other streaming operations (add/remove nodes)
  *   Stop an un-upgraded node (seeds first, preferably)
  *   Install new binaries and configs on the down node
  *   Restart that node and make sure it comes up clean (it will function 
normally in the cluster – even with mixed versions)
  *   Repeat for all nodes
  *   Run upgradesstables on each node (as many at a time as your load will 
allow). Minor upgrades usually don’t require this step (only if the sstable 
format has changed), but it is good to check.
  *   NOTE: in most cases applications can keep running and will not notice 
much impact – unless the cluster is overloaded and a single node down causes 
impact.



Sean Durity – Staff Systems Engineer, Cassandra

From: Sergio mailto:lapostadiser...@gmail.com>>
Sent: Wednesday, February 12, 2020 11:36 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Cassandra 3.11.X upgrades

Hi guys!

How do you usually upgrade your cluster for minor version upgrades?

I tried to add a node with 3.11.5 version to a test cluster with 3.11.4 nodes.

Is there any restriction?

Best,

Sergio



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Jon Haddad]

A while ago, on my first cluster, I decided to do an upgrade by adding
nodes running 1.2 to an existing cluster running version 1.1.  This was a
bad decision, and at that point I decided to always play it safe and always
stick to a single version, and never bootstrap in a node running different
version to a cluster.

I would do this even when it's a trivial upgrade, or a single commit is
different only touching a few lines of code.

Why?

I do this because in my opinion it's better to have a single way of doing
things.  Getting in the routine of doing minor upgrades by following Sean's
upgrade checklist will always work.  It's the supported, correct way of
doing things, and if you decide to always follow the same checklist, the
possibility of doing things wrong when there is an
incompatibility decreases.  Think of it as practice.

The only time you should even consider mixed version bootstrap upgrades is
if you know enough to not have to ask the list.  Even then, I go back to my
previous point about practice.  No point in practicing the less safe
version of things.

My 2 cents.

Jon



On Wed, Feb 12, 2020 at 11:02 AM Sergio  wrote:

> Thanks for your reply!
>
> So unless the sstable format has not been changed I can avoid to do that.
>
> Correct?
>
> Best,
>
> Sergio
>
> On Wed, Feb 12, 2020, 10:58 AM Durity, Sean R 
> wrote:
>
>> Check the readme.txt for any upgrade notes, but the basic procedure is to:
>>
>>- Verify that nodetool upgradesstables has completed successfully on
>>all nodes from any previous upgrade
>>- Turn off repairs and any other streaming operations (add/remove
>>nodes)
>>- Stop an un-upgraded node (seeds first, preferably)
>>- Install new binaries and configs on the down node
>>- Restart that node and make sure it comes up clean (it will function
>>normally in the cluster – even with mixed versions)
>>- Repeat for all nodes
>>- Run upgradesstables on each node (as many at a time as your load
>>will allow). Minor upgrades usually don’t require this step (only if the
>>sstable format has changed), but it is good to check.
>>- NOTE: in most cases applications can keep running and will not
>>notice much impact – unless the cluster is overloaded and a single node
>>down causes impact.
>>
>>
>>
>>
>>
>>
>>
>> Sean Durity – Staff Systems Engineer, Cassandra
>>
>>
>>
>> *From:* Sergio 
>> *Sent:* Wednesday, February 12, 2020 11:36 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* [EXTERNAL] Cassandra 3.11.X upgrades
>>
>>
>>
>> Hi guys!
>>
>> How do you usually upgrade your cluster for minor version upgrades?
>>
>> I tried to add a node with 3.11.5 version to a test cluster with 3.11.4
>> nodes.
>>
>> Is there any restriction?
>>
>> Best,
>>
>> Sergio
>>
>> --
>>
>> The information in this Internet Email is confidential and may be legally
>> privileged. It is intended solely for the addressee. Access to this Email
>> by anyone else is unauthorized. If you are not the intended recipient, any
>> disclosure, copying, distribution or any action taken or omitted to be
>> taken in reliance on it, is prohibited and may be unlawful. When addressed
>> to our clients any opinions or advice contained in this Email are subject
>> to the terms and conditions expressed in any applicable governing The Home
>> Depot terms of business or client engagement letter. The Home Depot
>> disclaims all responsibility and liability for the accuracy and content of
>> this attachment and for any damages or losses arising from any
>> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
>> items of a destructive nature, which may be contained in this attachment
>> and shall not be liable for direct, indirect, consequential or special
>> damages in connection with this e-mail message or its attachment.
>>
>

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Sergio]

Thanks for your reply!

So unless the sstable format has not been changed I can avoid to do that.

Correct?

Best,

Sergio

On Wed, Feb 12, 2020, 10:58 AM Durity, Sean R 
wrote:

> Check the readme.txt for any upgrade notes, but the basic procedure is to:
>
>- Verify that nodetool upgradesstables has completed successfully on
>all nodes from any previous upgrade
>- Turn off repairs and any other streaming operations (add/remove
>nodes)
>- Stop an un-upgraded node (seeds first, preferably)
>- Install new binaries and configs on the down node
>- Restart that node and make sure it comes up clean (it will function
>normally in the cluster – even with mixed versions)
>- Repeat for all nodes
>- Run upgradesstables on each node (as many at a time as your load
>will allow). Minor upgrades usually don’t require this step (only if the
>sstable format has changed), but it is good to check.
>- NOTE: in most cases applications can keep running and will not
>notice much impact – unless the cluster is overloaded and a single node
>down causes impact.
>
>
>
>
>
>
>
> Sean Durity – Staff Systems Engineer, Cassandra
>
>
>
> *From:* Sergio 
> *Sent:* Wednesday, February 12, 2020 11:36 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Cassandra 3.11.X upgrades
>
>
>
> Hi guys!
>
> How do you usually upgrade your cluster for minor version upgrades?
>
> I tried to add a node with 3.11.5 version to a test cluster with 3.11.4
> nodes.
>
> Is there any restriction?
>
> Best,
>
> Sergio
>
> --
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>

RE: [EXTERNAL] Cassandra 3.11.X upgrades

[Durity, Sean R]

Check the readme.txt for any upgrade notes, but the basic procedure is to:

  *   Verify that nodetool upgradesstables has completed successfully on all 
nodes from any previous upgrade
  *   Turn off repairs and any other streaming operations (add/remove nodes)
  *   Stop an un-upgraded node (seeds first, preferably)
  *   Install new binaries and configs on the down node
  *   Restart that node and make sure it comes up clean (it will function 
normally in the cluster – even with mixed versions)
  *   Repeat for all nodes
  *   Run upgradesstables on each node (as many at a time as your load will 
allow). Minor upgrades usually don’t require this step (only if the sstable 
format has changed), but it is good to check.
  *   NOTE: in most cases applications can keep running and will not notice 
much impact – unless the cluster is overloaded and a single node down causes 
impact.



Sean Durity – Staff Systems Engineer, Cassandra

From: Sergio 
Sent: Wednesday, February 12, 2020 11:36 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Cassandra 3.11.X upgrades

Hi guys!

How do you usually upgrade your cluster for minor version upgrades?

I tried to add a node with 3.11.5 version to a test cluster with 3.11.4 nodes.

Is there any restriction?

Best,

Sergio



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: How to elect a normal node to a seed node

[Voytek Jarnot]

>This means that from the client driver perspective when I define the
contact points I can specify any node in the cluster as contact point and
not necessary a seed node?

Correct.



On Wed, Feb 12, 2020 at 11:48 AM Sergio  wrote:

> So if
> 1) I stop the a Cassandra node that doesn't have in the seeds IP list
> itself
> 2) I change the cassandra.yaml of this node and I add it to the seed list
> 3) I restart the node
>
> It will work completely fine and this is not even necessary.
>
> This means that from the client driver perspective when I define the
> contact points I can specify any node in the cluster as contact point and
> not necessary a seed node?
>
> Best,
>
> Sergio
>
>
> On Wed, Feb 12, 2020, 9:08 AM Arvinder Dhillon 
> wrote:
>
>> I believe seed nodes are not special nodes, it's just that you choose a
>> few nodes from cluster that helps to bootstrap new joining nodes. You can
>> change Cassandra.yaml to make any other node as seed node. There's nothing
>> like promotion.
>>
>> -Arvinder
>>
>> On Wed, Feb 12, 2020, 8:37 AM Sergio  wrote:
>>
>>> Hi guys!
>>>
>>> Is there a way to promote a not seed node to a seed node?
>>>
>>> If yes, how do you do it?
>>>
>>> Thanks!
>>>
>>

Re: How to elect a normal node to a seed node

[Alexander Dejanovski]

Seed nodes are special in the sense that other nodes need them for
bootstrap (first startup only) and they have a special place in the Gossip
system. Odds of gossiping to a seed node are higher than other nodes, which
makes them "hubs" of gossip messaging.
Also, they do not bootstrap, so they won't stream data in on their first
start.

Aside from that, any node can become a seed node at anytime. Just update
the seed list on all nodes, roll restart the cluster and you'll have a new
set of seed nodes.

-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com


On Wed, Feb 12, 2020 at 6:48 PM Sergio  wrote:

> So if
> 1) I stop the a Cassandra node that doesn't have in the seeds IP list
> itself
> 2) I change the cassandra.yaml of this node and I add it to the seed list
> 3) I restart the node
>
> It will work completely fine and this is not even necessary.
>
> This means that from the client driver perspective when I define the
> contact points I can specify any node in the cluster as contact point and
> not necessary a seed node?
>
> Best,
>
> Sergio
>
>
> On Wed, Feb 12, 2020, 9:08 AM Arvinder Dhillon 
> wrote:
>
>> I believe seed nodes are not special nodes, it's just that you choose a
>> few nodes from cluster that helps to bootstrap new joining nodes. You can
>> change Cassandra.yaml to make any other node as seed node. There's nothing
>> like promotion.
>>
>> -Arvinder
>>
>> On Wed, Feb 12, 2020, 8:37 AM Sergio  wrote:
>>
>>> Hi guys!
>>>
>>> Is there a way to promote a not seed node to a seed node?
>>>
>>> If yes, how do you do it?
>>>
>>> Thanks!
>>>
>>

Re: How to elect a normal node to a seed node

[Sergio]

So if
1) I stop the a Cassandra node that doesn't have in the seeds IP list
itself
2) I change the cassandra.yaml of this node and I add it to the seed list
3) I restart the node

It will work completely fine and this is not even necessary.

This means that from the client driver perspective when I define the
contact points I can specify any node in the cluster as contact point and
not necessary a seed node?

Best,

Sergio


On Wed, Feb 12, 2020, 9:08 AM Arvinder Dhillon 
wrote:

> I believe seed nodes are not special nodes, it's just that you choose a
> few nodes from cluster that helps to bootstrap new joining nodes. You can
> change Cassandra.yaml to make any other node as seed node. There's nothing
> like promotion.
>
> -Arvinder
>
> On Wed, Feb 12, 2020, 8:37 AM Sergio  wrote:
>
>> Hi guys!
>>
>> Is there a way to promote a not seed node to a seed node?
>>
>> If yes, how do you do it?
>>
>> Thanks!
>>
>

Consequences of dropping Materialized views

[Surbhi Gupta]

Hi,

So application team created 11 materialized views on a base table in
production and we need to drop 7 Materialized views as they are not in use.
Wanted to understand the impact of dropping the materialized views.
We are on Cassandra 3.11.1 , multi datacenter with replication factor of 3
in each datacenter.
We are using LOCAL_QUORUM for write consistency and LOCAL_ONE for read
consistency.

Any thoughts or suggestion to keep in mind before dropping the Materialized
views.

Thanks
Surbhi

Re: How to elect a normal node to a seed node

[Arvinder Dhillon]

I believe seed nodes are not special nodes, it's just that you choose a few
nodes from cluster that helps to bootstrap new joining nodes. You can
change Cassandra.yaml to make any other node as seed node. There's nothing
like promotion.

-Arvinder

On Wed, Feb 12, 2020, 8:37 AM Sergio  wrote:

> Hi guys!
>
> Is there a way to promote a not seed node to a seed node?
>
> If yes, how do you do it?
>
> Thanks!
>

How to elect a normal node to a seed node

[Sergio]

Hi guys!

Is there a way to promote a not seed node to a seed node?

If yes, how do you do it?

Thanks!

Cassandra 3.11.X upgrades

[Sergio]

Hi guys!

How do you usually upgrade your cluster for minor version upgrades?

I tried to add a node with 3.11.5 version to a test cluster with 3.11.4
nodes.

Is there any restriction?

Best,

Sergio

Cassandra Encyrption between DC

[Jai Bheemsen Rao Dhanwada]

Hello,

Is there a way we can have a multi DC Cassandra cluster, where dc-1 have
encryption enabled and dc-2 does't have encryption?

I am trying to add a new DC to the existing cluster, where existing dc
don't have encryption between the nodes but the new DC have encryption
enabled?

I see the below options, is there a way to specify encrypt within DC?

   - all - Encrypt all inter-node communications
   - none - No encryption
   - dc - Encrypt the traffic between the datacenters
   - rack - Encrypt the traffic between the racks

Re: cassandra-cli on 3.x

[Jai Bheemsen Rao Dhanwada]

thank you

On Tue, Feb 11, 2020 at 6:38 PM Erick Ramirez 
wrote:

> I am using astyanax client
>
>
> Right. It was announced as being retired back in 2016 [1] which ended in
> 2018 [2]:
>
>
>>
>> *DeprecationAstyanax has been retired and is no longer under active
>> development but may receive dependency updates to ease migration away from
>> Astyanax.In place of Astyanax consider using DataStax Java Driver for
>> Apache Cassandra which is under active development and encapsulates a lot
>> of lessons learned from Astyanax. Since the DataStax driver supports only
>> CQL protocol (because Apache Cassandra 4.x will drop the Thrift protocol),
>> you’ll have to update all of your Thrift-based queries to CQL queries.*
>
>
> I'm sure you've heard it a lot by now that you will need to migrate off
> Thrift. C* 2.1 is on maintenance mode and will no longer be supported when
> C* 4.0 is released [3]. Cheers!
>
> [1]
> https://netflixtechblog.com/astyanax-retiring-an-old-friend-6cca1de9ac4
> [2] https://github.com/Netflix/astyanax
> [3] http://cassandra.apache.org/download/
>

RE: Connection reset by peer

[Durity, Sean R]

This looks like an error between your client and the cluster. Is the other ip 
address your client app? I have typically seen this when there are network 
issues between the client and the cluster. Cassandra driver connections are 
typically very long-lived. If something like a switch or firewall times out the 
connection you can get errors like this. Tcp_keepalive settings on the cluster 
nodes can help. See here: 
https://docs.datastax.com/en/dse/6.7/dse-admin/datastax_enterprise/config/configRecommendedSettings.html



Sean Durity

From: Hanauer, Arnulf, Vodacom South Africa (External) 

Sent: Wednesday, February 12, 2020 7:06 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Connection reset by peer


Hi Cassandra folks,

We are getting a lot of these errors and transactions are timing out and I was 
wondering if this can be caused by Cassandra itself or if this is a Linux 
network issue only. The client job reports Cassandra node down after this 
occurs but I suspect this is due to the connection failure - need some 
clarification as where to go look for a solution.


INFO  [epollEventLoopGroup-2-10] 2020-02-12 11:53:42,748 Message.java:623 - 
Unexpected exception during request; channel = [id: 0x8a3e6831, 
L:/10.132.65.152:9042 - R:/10.132.11.15:48020]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]

INFO  [epollEventLoopGroup-2-15] 2020-02-12 11:42:46,871 Message.java:623 - 
Unexpected exception during request; channel = [id: 0xa071f1c8, 
L:/10.132.65.152:9042 - R:/10.132.11.15:45134]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]


Source and Destination IP addresses are in the same DC (LAN).

I did recycle all the Cassandra services on all the nodes in both clusters but 
the problem remains.

The only change made recently was the adding of replicas in the second DC for 
the keyspace that is being written to when these messages occur (not had a 
chance to run a full repair yet to sync the replicas)


FYI:
Cassandra 3.11.2
5 Node cluster each in 2 DC's


Kind regards
Arnulf Hanauer









"This e-mail is sent on the Terms and Conditions that can be accessed by 
Clicking on this link https://webmail.vodacom.co.za/tc/default.html 
[vodacom.co.za]
 "



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Connection reset by peer

[Hanauer, Arnulf, Vodacom South Africa (External)]

Hi Cassandra folks,

We are getting a lot of these errors and transactions are timing out and I was 
wondering if this can be caused by Cassandra itself or if this is a Linux 
network issue only. The client job reports Cassandra node down after this 
occurs but I suspect this is due to the connection failure - need some 
clarification as where to go look for a solution.


INFO  [epollEventLoopGroup-2-10] 2020-02-12 11:53:42,748 Message.java:623 - 
Unexpected exception during request; channel = [id: 0x8a3e6831, 
L:/10.132.65.152:9042 - R:/10.132.11.15:48020]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]

INFO  [epollEventLoopGroup-2-15] 2020-02-12 11:42:46,871 Message.java:623 - 
Unexpected exception during request; channel = [id: 0xa071f1c8, 
L:/10.132.65.152:9042 - R:/10.132.11.15:45134]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]


Source and Destination IP addresses are in the same DC (LAN).

I did recycle all the Cassandra services on all the nodes in both clusters but 
the problem remains.

The only change made recently was the adding of replicas in the second DC for 
the keyspace that is being written to when these messages occur (not had a 
chance to run a full repair yet to sync the replicas)


FYI:
Cassandra 3.11.2
5 Node cluster each in 2 DC's


Kind regards
Arnulf Hanauer


"This e-mail is sent on the Terms and Conditions that can be accessed by 
Clicking on this 
linkhttps://www.vodacom.co.za/vodacom/terms/email-acceptable-user-policy; 

Connection reset by peer

[Hanauer, Arnulf, Vodacom South Africa (External)]

Hi Cassandra folks,

We are getting a lot of these errors and transactions are timing out and I was 
wondering if this can be caused by Cassandra itself or if this is a genuine 
Linux network issue only. The client job reports Cassandra node down after this 
occurs but I suspect this is due to the connection failure - need some 
clarification as where to go look for a solution.


INFO  [epollEventLoopGroup-2-10] 2020-02-12 11:53:42,748 Message.java:623 - 
Unexpected exception during request; channel = [id: 0x8a3e6831, 
L:/10.132.65.152:9042 - R:/10.132.11.15:48020]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]

INFO  [epollEventLoopGroup-2-15] 2020-02-12 11:42:46,871 Message.java:623 - 
Unexpected exception during request; channel = [id: 0xa071f1c8, 
L:/10.132.65.152:9042 - R:/10.132.11.15:45134]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]


Source and Destination IP addresses are in the same DC (LAN).

I did recycle all the Cassandra services on all the nodes in both clusters but 
the problem remains.

The only change made recently was the adding of replicas in the second DC for 
the keyspace that is being written to when these messages occur (not had a 
chance to run a full repair yet to sync the replicas)


FYI:
Cassandra 3.11.2
5 Node cluster each in 2 DC's


Kind regards
Arnulf Hanauer



"This e-mail is sent on the Terms and Conditions that can be accessed by 
Clicking on this 
linkhttps://www.vodacom.co.za/vodacom/terms/email-acceptable-user-policy;