Surbhi, The hints could be getting accumulated for one or both of the following reasons: - Some node is becoming unavailable very routinely, which is unlikely- The hints are getting replayed very slowly due to network bandwidth issues, which is more likely Increase the max_hint_window_in_ms setting in cassandra.yaml to more than 3 hours, perhaps 6 hours. If the issue still persists networking may need to be tested for bandwidth issues. regards,Deepak On Tuesday, January 28, 2020, 01:01:51 a.m. UTC, Surbhi Gupta <surbhi.gupt...@gmail.com> wrote: Why we think it might be related to hints is , because if we truncate the hints then load goes normal on the nodes.FYI , We had to run repair after truncating hints. Any thoughts ?
On Mon, 27 Jan 2020 at 15:27, Deepak Vohra <dvohr...@yahoo.com.invalid> wrote: Hints are a stopgap measure and not a fix to the underlying issue. Run a full repair. On Monday, January 27, 2020, 10:17:01 p.m. UTC, Surbhi Gupta <surbhi.gupt...@gmail.com> wrote: Hi, We are on Open source 3.11 .We have a issue in one of the cluster where lots of hints gets piled up and they don't get applied within hinted handoff period ( 3 hour in our case) . And load and CPU of the server goes very high.We see lot of messages in system.log and debug.log . Our read repair chance and dc_local_repair chance is 0.1 . Any pointers are welcome . ERROR [ReadRepairStage:83] 2020-01-27 13:08:43,695 CassandraDaemon.java:228 - Exception in thread Thread[ReadRepairStage:83,5,main] org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. DEBUG [ReadRepairStage:111] 2020-01-27 13:10:06,663 ReadCallback.java:242 - Digest mismatch: org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(4759131696153881383, 9a21276d0af64de28eeeed5d3023b69e) (142a55e1e28de7daa2ddc34a361 474a0 vs fcba30f022ef25f456914c341022963d)
