andreinnanu commented on PR #4488:
URL: https://github.com/apache/iggy/pull/4488#issuecomment-6276044877
Tested by manually freezing the sampler interval (adding a big extra sleep
in the binary for testing purposes):
Test scripts:
- measuring script, with a 0.2s interval, checking the /metrics endpoint and
`du` for data dir size
- 5 producers, producing data over 5 different topics
```
sending 32 x 65536 B per request to disk-lab/{burst-0, burst-1, burst-2,
burst-3, burst-4}, partition 0, rate 64.00 MiB/s
batches 976 sent 1952.00 MiB rate 16.27 MiB/s
done: 976 requests, 1952.00 MiB in 120.01 s (16.27 MiB/s)
```
- 10 consumers, consuming the produced data
```
10 consumers polling disk-lab/burst-0..4, 1 partition(s) each, 16 messages
per poll
polls 8246 messages 131936 read 8246.00 MiB 68.52 MiB/s
done: 8246 polls, 131936 messages, 8246.00 MiB in 120.34 s
```
Results:
- Preallocation off
```
largest overcount (estimate grew more than the data directory): 34.18
MiB
largest undercount (estimate grew less than the data directory): 0.05
MiB
```
- Preallocation on
```
largest overcount (estimate grew more than the data directory): 209.39
MiB
largest undercount (estimate grew less than the data directory): 0.04
MiB
```
Because of different frees that are not taken into account, the overcount
has the tendency to grow between samples, when it gets reseted. Is this
something we should consider at this point?
Also, the large overcounts are usually spikes created by the measuring
script, the diff looks like this over time:
```
23:28:42.155 983.45 966.20 17.25
23:28:42.475 983.50 966.22 17.28
23:28:42.772 983.59 966.32 17.27
23:28:43.033 983.64 966.38 17.26
23:28:43.369 1175.90 966.52 209.39
23:28:43.768 1175.99 1158.70 17.30
23:28:44.053 1176.04 1158.79 17.26
23:28:44.397 1176.19 1158.93 17.26
23:28:44.659 1176.24 1158.99 17.26
23:28:44.947 1176.35 1159.09 17.26
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
