Hello everyone,
1. I'm running load testing:
- case 1 - reboot Prometheus server before starting a new round test,
then general memory usage is around 45%*32GB for 1700000 series.
- case 2 - without rebooting, direct run the next round test, then
memory usage is around 65%*32GB for the same amount of series. I understand
if without rebooting, in memory there will be data for last round, but the
data will be flushed to disk after 2-3 hours(?), then memory consumption
should be reduced to 45%, but why this hadn't happened?
Could some experts help to check?
- case 3 - at 10:28AM prometheus_tsdb_head_series is 1680000, but after
one hour, the series was almost doubled to 3100000, and memory consumption
increased to 85%*32. Why the time series was doubled? Checked one more
hour, prometheus_tsdb_head_series was back to 1680000. May I ask what
happened probably during this 1 hour?
*2. *Local disk space which attach to Prometheus server, retention period
is 1h.(at 3:35, the test started.)
3:35 used 655.2M
4:35 used 1.2G
5:38 used 1.5G
6:38 used 2.4G
7:38 used 1.7G
8:38 used 2.6G
9:38 used 1.7G
10:38 used 2.7G
11:38 used 1.7G
>From the above record, I have several questions:
- I scrapped 1,700,000 metrics every 60s, but why the data increased
differently, 3:35-4:35, 4:35-5:38, 5:38-6:38, 7:38-8:38，9:38-10:38.
- I tried to use this formula to estimate the disk space first
"needed_disk_space = retention_time_seconds * ingested_samples_per_second *
bytes_per_sample" -> 60*60*(1700000/60)*2 = 201,600,000bytes
This is different with my testing record, from my testing record, the
increased is between 400M - 1G.
Why the result is not consistent between the expect and actual result?
- I understand the data reduced due to the 1h retention period, am I
right?
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/fddc0363-cbc7-43f3-a2bf-a1df5d3883c5n%40googlegroups.com.
