Thanks for the response Brain, it was 175GB and with the last restart, I increased it to 500 GB, sorry I did a typo there.
It's not generating that much data in 2 ~ 4 hrs, but it is doing over a week. The data is stored on the local filesystem ext4. not on nas. # ls 01FTRRKBEQV1QX0HAM2MXG1W3Q 01FTTWSEGZ4MQN3NXAR12R8GSD 01FTTX6H5GM758FVPHK1DTSH9S chunks_head lock queries.active wal # ls chunks_head/ 000175 000176 I don't see any errors in the logs, but still checking on the issue. Thanks On Tuesday, February 1, 2022 at 6:37:29 AM UTC-5 Brian Candler wrote: > I am having issues with chunks_head directory. I have the tsdb retention >> size to 175 GB >> > ... > >> --storage.tsdb.retention.size=500GB >> > > That doesn't appear to be consistent with what you said above. > > However, in any case, the "chunks_head" directory should only contain data > created in the last 2-4 hours, before it gets moved to its final location. > > Is it possible that you're generating 689GB of data in 2-4 hours? That's > a *lot* of metrics, and it seems unlikely - but if so, this would be > expected behaviour. > > If not, then there's some problem moving the head chunks into their final > location. Your storage directory should look something like this (from > v2.30.3): > > root@prometheus:~# ls /var/lib/prometheus/data/ > 01FRDMW4G1T19JM989NE1VKZ32 01FSGD7TFJEWA6FJ08PJ2TZZPG > 01FTK5KH12W0RDWZA85NSTP4TX chunks_head > 01FRKE8RPDW2ZYSJFXEDD7QK3S 01FSP6MG73NQQ996C7EWH49MQF > 01FTRZ05RH3C8RKJ4PQ162QJT8 lock > 01FRS7NBJST1MW7R3XDBE9592M 01FSW012JNK10VV5WWZ4C2H3V9 > 01FTSKK60DX4D8C908BYHZTR8F queries.active > 01FRZ120RQQZJW24T3BVP9NFTT 01FT1SDN5R9JYEQFQ90JCS2Y2Z > 01FTT869E3DW1V0QYSPZ921CKQ wal > 01FS4TEKSMZ3NJ2M6NNEAH9K63 01FT7JTABCH3D1JDQYZAPJ9QXK > 01FTT86AK2N6VA00RSCYB9AC6J > 01FSAKV7VSS6VMPDCJAWNZGT6V 01FTDC6WGMY25WCQ3JTKZECGPX > 01FTTF20P47NVE06W79PKKFXFJ > > The majority of storage will be in those other directories, which are > created automatically, and are what the retention setting will control. If > this isn't happening, then you have a problem. > > I suggest you check the log output from prometheus for errors (e.g. > "journalctl -eu prometheus" if you're running it under systemd). It could > be that it's failing to create these directories, and data is sticking > around in the head. Are permissions on /apps/prometheus/prometheus_data > correct? Are you storing data on a block filesystem (e.g. local disk or a > filesystem in Amazon EBS)? Prometheus does not support writing data to any > sort of NAS filesystem, e.g. NFS / Samba / EFS. > > There were a couple of minor fixes in v2.31.1 and v2.31.2, but I don't > think they are relevant here. > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b8c0105d-6ba3-4e26-916d-bf263822c0b7n%40googlegroups.com.
