My general advice for any time you see hints accumulating, consider that smoke for the more pressing fire happening somewhere else. You correctly identified the right path to consider, which is some sort of scheduled activity. Cassandra doesn't have any scheduled internal jobs. Compactions happen as needed, and maintenance jobs such as repair are scheduled by the operator. The thing to investigate at this point is what other spikes are happening in the system at this point. Network? Disk activity? CPU?
Other things to investigate Unrelated systems consuming network bandwidth VPN tunnels resetting nightly (yes, I've seen that one) You mentioned virtualization, noisy neighbor? Good luck with the sleuthing! Patrick On Mon, Jan 16, 2023 at 6:33 AM Yakir Gibraltar <yaki...@gmail.com> wrote: > Check if you see packet loss at this time > > On Mon, Jan 16, 2023 at 4:08 PM Loïc CHANEL via user < > user@cassandra.apache.org> wrote: > >> Hi team, >> >> I am currently running a 2-nodes Cassandra database. Although that's not >> the best setup, the cluster is doing pretty fine. >> Still, I noticed that for (at least) 5 days now, one of my two nodes is >> writing hints during the night, and then it recovers the data-sync with the >> other node in the morning. What's interesting here is that we can see that >> the hints writing stuff happens at 0:00 every day, then it starts to >> recover for a few minutes before 2:00, then the node starts writing hints >> again at 2:00, then it (sometimes) starts to recover a few minutes before >> 4:00 and then it start writing hints again at 4:00 (and sometimes same >> thing happens once more around 6:00). >> See the following graph that shows the number of hints files in hints >> directory : >> >> [image: image.png] >> >> I am not aware of any nightly sync or any such process in Cassandra >> database, but I checked cron logs on the servers and nothing seems to >> happen at these times, and there is no particular activity at that time on >> the virtualization platform (the daily snapshot is taken at 1:30). So my >> question here is : am I missing a Cassandra internal process that is >> triggered on a daily basis at 0:00 and 2:00 ? >> Thanks, >> >> >> Loïc CHANEL >> System Big Data engineer >> SoftAtHome (Lyon, France) >> > > > -- > *בברכה,* > *יקיר גיברלטר* >
