GitHub user yatriks added a comment to the discussion: Memory leak and confusing logs
@git-hulk Thank you for the explanations, that's very helpful. Some mixed results from testing the `rocksdb.max_compaction_bytes`, I will try to be as rigourous as I can in my reporting. I have now run two detailed load tests over the past night, where I capture: 1) memory reported by `docker stats` 2) memory reported by `redis-cli -p 6666 info` 3) all the stats collected by `kvrocks-exporter` I am running relatively heavy read/write workloads, and performance is at 15-20k ops/s. ### First test For the first test, I had only the change to the `max_compaction_bytes`. For 12 hours, this has run continuously without any issue. However, I notice the following: 1) lots of logs `E20250522 11:59:05.299835 139759830681280 compaction_checker.cc:88] [compaction checker] Parse total_keys error: out of range of integer type`. Not sure if this matters. 2) the `docker stats` memory is consistently higher than reported by `kvrocks`. While `docker stats` sees memory going from ~5-12 GB (though consistently under 12, so no oom kills), `kvrocks` shows it consistently around 5-7GB. **All in all, this was good** -- no issues after 12 hours of significant load, though I was a bit concerned about the `docker stats` memory being higher, my worry being if `kvrocks` had some kind of momentary spike maybe it would cause an issue with docker...don't know enough to know if this is true. | docker stats | kvrocks | | :----------- | :--------------: | | <img src="https://github.com/user-attachments/assets/5ff6d21c-9883-428e-91e0-b7d835e745be"; width="400" /><br/> *note how it constantly almost reaches 12 but then backs off* | <img src="https://github.com/user-attachments/assets/390b1535-0c16-4928-b2b2-0838cf35429d"; width="400" /><br/>*consistently <7.5GB* | ### Second test Just in case, I wanted to see if the change to `MALLOC_CONF=retain:false` would get the `docker stats` and `kvrocks` reported memory to agree. Results: 1) Lots of logs, pretty much constant, of `<jemalloc>: Error in munmap(): Cannot allocate memory`, pretty much constantly. 2) The docker stats memory report and kvrocks agree _more_, to an extent, but the docker memory is a bit jumpier. 2) The container is oom-killed after it reaches 12GB of memory usage. | docker stats | kvrocks | | :----------- | :--------------: | | <img src="https://github.com/user-attachments/assets/8623489f-9076-45d6-a691-5bfff72d6272"; width="400" /> | <img src="https://github.com/user-attachments/assets/0bb5c750-2a97-42c5-92ad-57ffc14e07ed"; width="400" /> | Conclusions: 1) It seems that with the `max_compaction_bytes` set, even with auto compaction, there were no issues (though I am still concerned about the disagreement between `kvrocks` RSS, which strangely agrees with `top` but not with `docker`. 2) The `MALLOC_CONF` change did make the numbers reported by `docker stats` and `kvrocks` more in the same general area, but leads to an oom kill. Questions: 1) Do the `Parse total_keys error` matter/should I be concerned? I will check and see `disable_auto_compaction` makes a difference in performance/whether it will strain the server too much during the compaction window. Please let me know if I can investigate anything else that would be helpful. GitHub link: https://github.com/apache/kvrocks/discussions/2984#discussioncomment-13237586 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
