Hi, We are on Open source 3.11 . We have a issue in one of the cluster where lots of hints gets piled up and they don't get applied within hinted handoff period ( 3 hour in our case) . And load and CPU of the server goes very high. We see lot of messages in system.log and debug.log . Our read repair chance and dc_local_repair chance is 0.1 . Any pointers are welcome .
ERROR [ReadRepairStage:83] 2020-01-27 13:08:43,695 CassandraDaemon.java:228 - Exception in thread Thread[ReadRepairStage:83,5,main] org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. DEBUG [ReadRepairStage:111] 2020-01-27 13:10:06,663 ReadCallback.java:242 - Digest mismatch: org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(4759131696153881383, 9a21276d0af64de28eeeed5d3023b69e) (142a55e1e28de7daa2ddc34a361 474a0 vs fcba30f022ef25f456914c341022963d)
