Which filesystem are you using on the docker host? If it's ext4: many systems by default configure it to reserve a minimum of 5% free space (i.e. space that only 'root' can use).
Check with: tune2fs -l /dev/sda1 # or whatever device your root partition is in and look at the ratio of "Reserved block count" to "Block count". e.g. on one system I have here, I see Block count: 5242880 Reserved block count: 262144 262144/5242880 = 0.05 It can be changed with the -m option to tune2fs. If it's btrfs: there's a whole can of worms around what constitutes "free space" in btrfs :-) But looking at your figures, where you're at 94% full, I think it's most likely you're hitting the ext4 reserved blocks limit. On Friday, 9 September 2022 at 08:36:07 UTC+1 [email protected] wrote: > Hi all, > > I'm running a Prometheus docker container, and I've run into an issue > regarding disk space. Despite having hundreds of MB of free disk space, if > I attempt to call the API snapshot endpoint `curl -XPOST > http://localhost:9090/api/v1/admin/tsdb/snapshot` I receive an error > message: > > `create snapshot: snapshot head block: populate block: write chunks: > preallocate: no space left on device` > > Output of `df -h` inside my prometheus container: > [image: space_left.PNG] > With around 518 MB free, I am able to call the snapshot endpoint. However, > with less free space than this, the snapshot endpoint returns the error > regarding no space left on device. > > size of my /data folder: > [image: data_folder.PNG] > > I see from this thread > <https://github.com/prometheus/prometheus/issues/8406> that potentially > 100's of MB free disk space is required to take a snapshot. I am a little > surprised that (from what it looks like) at least 500 MB are required to > take a single snapshot. Is this expected/intended behaviour of prometheus, > or could there be something on my end (perhaps docker related) that is > contributing to this issue? > > Thank you! > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2ba6284e-0657-4fd1-a135-f9421e8ec413n%40googlegroups.com.
