Hi Brian, Thank you for the reply, I did some investigating as you suggested:
My partition is using ext4 filesystem, and with `tune2fs` I confirmed that 5% free space is reserved. However, my understanding of the `df` output is that it is not counting this reserved disk space under "available". I confirmed this by running the suggested prometheus metrics as well: "node_filesystem_avail_bytes" gives me 518 MiB as well for the above, which correlates to the "available" space according to `df`. Accordingly, "node_filesystem_free_bytes" shows a larger remaining disk space, as would be expected since this includes reserved space as well. So according to this, it seems to me prometheus is indeed seeing 518 MiB remaining, and yet for values just a bit lower than this (i.e. below 500 MiB), I start getting the 'no space left on device' error. On Friday, September 9, 2022 at 2:24:17 AM UTC-7 Brian Candler wrote: > Oh and I should add, the other place to look is prometheus metrics :-) > > node_exporter reports both "node_filesystem_avail_bytes" and > "node_filesystem_free_bytes". The former excludes space reserved for root, > so you'll see that hit zero sooner, and that'll be when prometheus thinks > the disk is "full". > > You can of course graph: > node_filesystem_avail_bytes / node_filesystem_size_bytes > node_filesystem_free_bytes / node_filesystem_size_bytes > > On Friday, 9 September 2022 at 10:05:48 UTC+1 Brian Candler wrote: > >> Which filesystem are you using on the docker host? >> >> If it's ext4: many systems by default configure it to reserve a minimum >> of 5% free space (i.e. space that only 'root' can use). >> >> Check with: >> tune2fs -l /dev/sda1 # or whatever device your root partition is in >> and look at the ratio of "Reserved block count" to "Block count". e.g. >> on one system I have here, I see >> >> Block count: 5242880 >> Reserved block count: 262144 >> >> 262144/5242880 = 0.05 >> >> It can be changed with the -m option to tune2fs. >> >> If it's btrfs: there's a whole can of worms around what constitutes "free >> space" in btrfs :-) >> >> But looking at your figures, where you're at 94% full, I think it's most >> likely you're hitting the ext4 reserved blocks limit. >> >> On Friday, 9 September 2022 at 08:36:07 UTC+1 [email protected] wrote: >> >>> Hi all, >>> >>> I'm running a Prometheus docker container, and I've run into an issue >>> regarding disk space. Despite having hundreds of MB of free disk space, if >>> I attempt to call the API snapshot endpoint `curl -XPOST >>> http://localhost:9090/api/v1/admin/tsdb/snapshot` I receive an error >>> message: >>> >>> `create snapshot: snapshot head block: populate block: write chunks: >>> preallocate: no space left on device` >>> >>> Output of `df -h` inside my prometheus container: >>> [image: space_left.PNG] >>> With around 518 MB free, I am able to call the snapshot endpoint. >>> However, with less free space than this, the snapshot endpoint returns the >>> error regarding no space left on device. >>> >>> size of my /data folder: >>> [image: data_folder.PNG] >>> >>> I see from this thread >>> <https://github.com/prometheus/prometheus/issues/8406> that potentially >>> 100's of MB free disk space is required to take a snapshot. I am a little >>> surprised that (from what it looks like) at least 500 MB are required to >>> take a single snapshot. Is this expected/intended behaviour of prometheus, >>> or could there be something on my end (perhaps docker related) that is >>> contributing to this issue? >>> >>> Thank you! >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6d7984a1-d01c-47ff-ab1c-6cc734f2e575n%40googlegroups.com.
