egor-ryashin commented on issue #5577: URL: https://github.com/apache/druid/issues/5577#issuecomment-848585038
I can confirm that this is still present in Druid 0.21. I see logs like: ``` [WARN ] 2021-05-25 14:39:20.718 [SimpleDataSegmentChangeHandler-2] StorageLocation - Segment[<segment_name>:260,453,366] too large for storage/opt/druid/var/segmentCache:7,621,518. Check your druid.segmentCache.locations maxSize param ``` To eliminate confusion I can confirm that 7,621,518 number is what Druid thinks about available space in storage. You can check it here `org/apache/druid/segment/loading/StorageLocation.java:155` Meanwhile there are at least several GBs available: ``` df -h ... Avail Use% Mounted on 17G 94% /opt/druid/var ``` Metrics say the same (the correct used size), actually, metrics report from a different in-memory variable, you can find it here `org/apache/druid/server/metrics/HistoricalMetricsMonitor.java:88` Right now I cannot find a bug in source code, I can only speculate that some exception is uncaught and the available size is not decreased properly, but looking through logs doesn't show there were uncaught exceptions. I propose to log or send as metrics this variable `org.apache.druid.segment.loading.StorageLocation#availableSizeBytes` so we can track when the issue starts taking place, other debugging ideas are welcome. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
