Something is going wrong. –storage.tsdb.retention.size controls the *total* storage retained, not the WAL - and the WAL is only about 2 hours worth of data.
What filesystem is /apps/prometheus/prometheus_data stored on? What do you see inside the wal directory? e.g. on a system here: # *ls -lhR wal* wal: total 501M -rw-r--r-- 1 prometheus prometheus 128M Dec 1 17:45 00031696 -rw-r--r-- 1 prometheus prometheus 128M Dec 1 18:30 00031697 -rw-r--r-- 1 prometheus prometheus 85M Dec 1 19:00 00031698 -rw-r--r-- 1 prometheus prometheus 128M Dec 1 19:45 00031699 -rw-r--r-- 1 prometheus prometheus 32M Dec 1 19:56 00031700 drwxr-xr-x 1 prometheus prometheus 16 Dec 1 19:00 checkpoint.00031695 wal/checkpoint.00031695: total 7.6M -rw-r--r-- 1 prometheus prometheus 7.6M Dec 1 19:00 00000000 Do you have a massive churn in timeseries? Go to the web interface, select Status > TSDB Status, and what does the Head Stats section say? e.g. Number of Series Number of Chunks Number of Label Pairs Current Min Time Current Max Time => 499826 499826 13719 2021-12-01T18:00:00.410Z (1638381600410) 2021-12-01T19:58:08.988Z (1638388688988) -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ea067387-0dc7-41fa-bed1-8d4cefa1bfb8n%40googlegroups.com.
