Hi Cassandra folks, We are getting a lot of these errors and transactions are timing out and I was wondering if this can be caused by Cassandra itself or if this is a Linux network issue only. The client job reports Cassandra node down after this occurs but I suspect this is due to the connection failure - need some clarification as where to go look for a solution.
INFO [epollEventLoopGroup-2-10] 2020-02-12 11:53:42,748 Message.java:623 - Unexpected exception during request; channel = [id: 0x8a3e6831, L:/10.132.65.152:9042 - R:/10.132.11.15:48020] io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] INFO [epollEventLoopGroup-2-15] 2020-02-12 11:42:46,871 Message.java:623 - Unexpected exception during request; channel = [id: 0xa071f1c8, L:/10.132.65.152:9042 - R:/10.132.11.15:45134] io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] Source and Destination IP addresses are in the same DC (LAN). I did recycle all the Cassandra services on all the nodes in both clusters but the problem remains. The only change made recently was the adding of replicas in the second DC for the keyspace that is being written to when these messages occur (not had a chance to run a full repair yet to sync the replicas) FYI: Cassandra 3.11.2 5 Node cluster each in 2 DC's Kind regards Arnulf Hanauer "This e-mail is sent on the Terms and Conditions that can be accessed by Clicking on this linkhttps://www.vodacom.co.za/vodacom/terms/email-acceptable-user-policy"