599166320 commented on issue #13080: URL: https://github.com/apache/druid/issues/13080#issuecomment-1295842146
Today, another cluster of ours (without any modification) once again experienced the problem that the expired data of the hot node was not deleted, causing the storage space to be quickly used up.   It looks like the storage space of the hot node is almost full, and the cold node still has enough storage space.   Here are some monitoring:   After this problem occurred, I quickly emitted the monitoring data to promethues. After observing for two hours, I upgraded the master node and applied `LoadDropByPeriod`, and the cluster quickly returned to normal. Of course, our cluster also found some problems in the process of applying `LoadDropByPeriod`, such as the need to limit the speed of deletion. I still have a few hours of monitoring data on my side that I can share with you if you need it. @kfaraz -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
