Oh and I should add, the other place to look is prometheus metrics :-) node_exporter reports both "node_filesystem_avail_bytes" and "node_filesystem_free_bytes". The former excludes space reserved for root, so you'll see that hit zero sooner, and that'll be when prometheus thinks the disk is "full".
You can of course graph: node_filesystem_avail_bytes / node_filesystem_size_bytes node_filesystem_free_bytes / node_filesystem_size_bytes On Friday, 9 September 2022 at 10:05:48 UTC+1 Brian Candler wrote: > Which filesystem are you using on the docker host? > > If it's ext4: many systems by default configure it to reserve a minimum of > 5% free space (i.e. space that only 'root' can use). > > Check with: > tune2fs -l /dev/sda1 # or whatever device your root partition is in > and look at the ratio of "Reserved block count" to "Block count". e.g. on > one system I have here, I see > > Block count: 5242880 > Reserved block count: 262144 > > 262144/5242880 = 0.05 > > It can be changed with the -m option to tune2fs. > > If it's btrfs: there's a whole can of worms around what constitutes "free > space" in btrfs :-) > > But looking at your figures, where you're at 94% full, I think it's most > likely you're hitting the ext4 reserved blocks limit. > > On Friday, 9 September 2022 at 08:36:07 UTC+1 [email protected] wrote: > >> Hi all, >> >> I'm running a Prometheus docker container, and I've run into an issue >> regarding disk space. Despite having hundreds of MB of free disk space, if >> I attempt to call the API snapshot endpoint `curl -XPOST >> http://localhost:9090/api/v1/admin/tsdb/snapshot` I receive an error >> message: >> >> `create snapshot: snapshot head block: populate block: write chunks: >> preallocate: no space left on device` >> >> Output of `df -h` inside my prometheus container: >> [image: space_left.PNG] >> With around 518 MB free, I am able to call the snapshot endpoint. >> However, with less free space than this, the snapshot endpoint returns the >> error regarding no space left on device. >> >> size of my /data folder: >> [image: data_folder.PNG] >> >> I see from this thread >> <https://github.com/prometheus/prometheus/issues/8406> that potentially >> 100's of MB free disk space is required to take a snapshot. I am a little >> surprised that (from what it looks like) at least 500 MB are required to >> take a single snapshot. Is this expected/intended behaviour of prometheus, >> or could there be something on my end (perhaps docker related) that is >> contributing to this issue? >> >> Thank you! >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ab88ce95-1f29-4017-a8d0-cdc2347a9831n%40googlegroups.com.
