Hi, Quan I greatly appreciate your effort and detailed information.
> Maybe the gradual memory growth issue is more a Jemalloc's nature than > Kvrocks issue itself? I think so. From your information, it did look like most of the memory are reclaimable cache since they cannot be returned to the OS when running with memory-intensive processes I will repost your post link in GitHub Discussions to let other users have a try. Thank you! On Thu, 8 May 2025 at 12:44, Hong Quan TRAN <hqt...@linagora.com> wrote: > > Hi Hulk, > I’d like to share some results from an experiment I ran related to Jemalloc > memory tuning in Kvrocks, which may be helpful to the gradual memory growth > over time.I read a Apache Doris jemalloc analysis article, where they seems > to claim more or the the same issue as our. Jemalloc retained memory > aggressively by default, leading to high memory usage that the OS could not > easily reclaim. They suggest to restart Apache Doris after sometime to > reclaim the memory, or use the retain:false Jemalloc option to force > Jemalloc return the unused memory to OS. > > I tried the option with Kvrocks using the environment variable > MALLOC_CONF=retain:false. > > After applying retain=false and stress tests Kvrocks using redis-benchmark a > few dozen times, I observed: > > > - Docker stats memory: consistently under 800MB (was more than 1.3GB if > retain:true) > > > - Kvrocks-reported RSS memory: consistently under 500MB > > > - When running other memory-intensive processes, Kvrocks memory usage > dropped by ~150MB, suggesting part of the usage was reclaimable cache. > > > > This aligns with Jemalloc behavior where disabling retain makes memory more > readily returnable to the OS. > > Maybe the gradual memory growth issue is more a Jemalloc's nature than > Kvrocks issue itself? > > Regarding performance metrics, I did not deeply dig into it yet. A quick > benchmark on my side showed no significant difference between the two > settings. > > Please let me know your thoughts on this matter. > > Quan > > > > On May 3, 2025 9:10 PM, from Hulk <hulk.webs...@gmail.com>Hi Quan, > > I find a gap in the memory usage in Docker stats, which is about double > that of Kvrocks's RSS. > > You might have a check as well using the command `info memory` via > redis-cli. > > Not sure if it's caused by the different algorithm when counting the memory > usage. > > > *docker stats a68a50cf4a82* > > a68a50cf4a82 tender_jackson 143.39% 635MiB / 14.63GiB 4.24% > 22.8GB / 4.7GB 844kB / 21GB 18 > > *hulk@instance-20241021-041032:~$ redis-cli -p 6666 info memory* > > # Memory > used_memory_rss:247369728 > used_memory_rss_human:235.91M > used_memory_lua:319488 > used_memory_lua_human:312.00K > used_memory_startup:33468416 > > > On Sat, 3 May 2025 at 19:02, hulk <hulk.webs...@gmail.com> wrote: > > > Hi, Quan > > > > Thanks for your detailed information. I can reproduce it on my side. I > > ran the benchmark a few times, and its memory is around the same(about > > 1.3GiB), > > but I haven't dived deep into the Jemalloc stats yet. > > > > On Fri, 2 May 2025 at 17:31, Hong Quan TRAN <hqt...@linagora.com> wrote: > > > > > > Hi Hulk, > > > > > > Sorry for the late reply (we were on holiday). > > > > > > > I have no clue about why it's up to 1.3 GiB, will try to reproduce > > > this on my side and analyze the memory in Jemalloc. > > > > > > We use Kvrocks as Rspamd's bayes training database. I would attach the > > Kvrocks used command stats for more details. > > > > > > BTW on my side, I tried to run the redis-benchmark tool against Kvrocks > > (still the same configuration). After a dozen times running > > `redis-benchmark -n 2000000 -c 50` command, Kvrocks memory is upto around > > 1.1GB and does not seems to be released automatically after a while. > > > > > > I hope this would help you ease reproducing the issue. I would be happy > > to do more diagnostic and share the output to you if needed. > > > > > > Regards, > > > > > > Quan > > > > > > > > > On Apr 27, 2025 4:27 PM, from Hulk <hulk.webs...@gmail.com> > > > > > > Hi Benoit, Quan > > > > > > Sorry for the late reply. I took a look at your configuration, and it > > > all looks good. > > > > > > I have no clue about why it's up to 1.3 GiB, will try to reproduce > > > this on my side and analyze the memory in Jemalloc. > > > > > > Will update you once we have any progress. > > > > > > On Mon, 21 Apr 2025 at 11:00, hulk <hulk.webs...@gmail.com> wrote: > > > > > > > > Thanks all for taking care of this, will have a look. > > > > > > > > On Mon, 21 Apr 2025 at 10:56, Hong Quan TRAN <hqt...@linagora.com> > > wrote: > > > >> > > > >> Hi, > > > >> > > > >> Somehow the format is still bad (although I tested sending it to my > > personal mail). > > > >> > > > >> I would send the Kvrocks config output via an attachment this time. > > > >> > > > >> Quan > > > >> > > > >> Vào ngày thg 4 21, 2025 9:39 SA, từ Hong Quan Tran < > > hqt...@linagora.com> > > > >> > > > >> Hi everyone, > > > >> Sorry for the bad formatting output. > > > >> I resend the config output. I hope it is fine this time. > > > >> > > > >> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) > > "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) > > "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9) > > "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) > > "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) > > "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21) > > "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) > > "force-compact-file-age" 26) "172800" 27) > > "force-compact-file-min-deleted-percentage" 28) "10" 29) > > "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" > > 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) > > "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) > > "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) > > "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) > > "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) > > "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) > > "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" > > 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" > > 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) > > "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) > > "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78) > > "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" > > 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85) > > "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" > > 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) > > "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" > > 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) > > "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) > > "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) > > "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) > > "yes"111) "rocksdb.blob_file_size"112) "268435456"113) > > "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) > > "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) > > "lru"119) "rocksdb.block_size"120) "16384"121) > > "rocksdb.cache_index_and_filter_blocks"122) "yes"123) > > "rocksdb.compaction_readahead_size"124) "2097152"125) > > "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) > > "32767"129) "rocksdb.compression_start_level"130) "2"131) > > "rocksdb.delayed_write_rate"132) "0"133) > > "rocksdb.disable_auto_compactions"134) "no"135) > > "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) > > "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) > > "rocksdb.enable_pipelined_write"142) "no"143) > > "rocksdb.level0_file_num_compaction_trigger"144) "4"145) > > "rocksdb.level0_slowdown_writes_trigger"146) "20"147) > > "rocksdb.level0_stop_writes_trigger"148) "40"149) > > "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) > > "rocksdb.max_background_compactions"152) "2"153) > > "rocksdb.max_background_flushes"154) "2"155) > > "rocksdb.max_background_jobs"156) "4"157) > > "rocksdb.max_bytes_for_level_base"158) "268435456"159) > > "rocksdb.max_bytes_for_level_multiplier"160) "10"161) > > "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) > > "2"165) "rocksdb.max_total_wal_size"166) "256"167) > > "rocksdb.max_write_buffer_number"168) "2"169) > > "rocksdb.metadata_block_cache_size"170) "2048"171) > > "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) > > "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) > > "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) > > "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) > > "rocksdb.stats_dump_period_sec"184) "0"185) > > "rocksdb.subkey_block_cache_size"186) "2048"187) > > "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) > > "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) > > "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) > > "32"197) "rocksdb.write_options.disable_wal"198) "no"199) > > "rocksdb.write_options.low_pri"200) "no"201) > > "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) > > "rocksdb.write_options.no_slowdown"204) "no"205) > > "rocksdb.write_options.sync"206) "no"207) > > "rocksdb.write_options.write_batch_max_bytes"208) "0"209) > > "skip-block-cache-deallocation-on-close"210) "no"211) > > "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) > > "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) > > "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) > > "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) > > "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) > > "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) > > "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) > > ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) > > "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) > > "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) > > "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) > > "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) > > "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) > > "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) > > "8" > > > >> Quan > > > >> Vào ngày thg 4 21, 2025 1:52 SA, từ Benoit Tellier < > > btell...@linagora.com>Careful > > > >> > > > >> Formating sucks > > > >> > > > >> Maybe resent it as a plain/text version to avoid this pitfall? > > > >> > > > >> -- > > > >> > > > >> Best regards, > > > >> > > > >> Benoit TELLIER > > > >> > > > >> General manager of Linagora VIETNAM. > > > >> Product owner for Team-Mail product. > > > >> Chairman of the Apache James project. > > > >> > > > >> Mail: btell...@linagora.com > > > >> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal) > > > >> > > > >> > > > >> On Apr 20, 2025 3:43 PM, from Hong Quan Tran <hqt...@linagora.com>Hi > > Hulk, > > > >> > Thanks for your information, could you please provide the output by > > running the command in redis-cli `config get *`? > > > >> Please see the output below. > > > >> > > > >> kvrocks@561e141f9b72:/$ redis-cli -p 6379 CONFIG GET '*' 1) > > "auto-resize-block-and-sst" 2) "yes" 3) "backup-dir" 4) > > "/var/lib/kvrocks/backup" 5) "bgsave-cron" 6) "" 7) "bind" 8) "0.0.0.0" 9) > > "cluster-enabled" 10) "no" 11) "compact-cron" 12) "" 13) > > "compaction-checker-cron" 14) "" 15) "compaction-checker-range" 16) "" 17) > > "daemonize" 18) "no" 19) "db-name" 20) "change.me.db" 21) > > "dbsize-scan-cron" 22) "" 23) "dir" 24) "/var/lib/kvrocks" 25) > > "force-compact-file-age" 26) "172800" 27) > > "force-compact-file-min-deleted-percentage" 28) "10" 29) > > "fullsync-recv-file-delay" 30) "0" 31) "histogram-bucket-boundaries" 32) "" > > 33) "json-max-nesting-depth" 34) "1024" 35) "json-storage-format" 36) > > "json" 37) "log-dir" 38) "/var/lib/kvrocks" 39) "log-level" 40) "info" 41) > > "log-retention-days" 42) "-1" 43) "master-use-repl-port" 44) "no" 45) > > "masterauth" 46) "" 47) "max-backup-keep-hours" 48) "0" 49) > > "max-backup-to-keep" 50) "1" 51) "max-bitmap-to-string-mb" 52) "16" 53) > > "max-db-size" 54) "0" 55) "max-io-mb" 56) "0" 57) "max-replication-mb" 58) > > "0" 59) "maxclients" 60) "10240" 61) "migrate-batch-rate-limit-mb" 62) "16" > > 63) "migrate-batch-size-kb" 64) "16" 65) "migrate-pipeline-size" 66) "16" > > 67) "migrate-sequence-gap" 68) "10000" 69) "migrate-speed" 70) "4096" 71) > > "migrate-type" 72) "redis-command" 73) "persist-cluster-nodes-enabled" 74) > > "yes" 75) "pidfile" 76) "/var/run/kvrocks/kvrocks.pid" 77) "port" 78) > > "6379" 79) "profiling-sample-commands" 80) "" 81) "profiling-sample-ratio" > > 82) "0" 83) "profiling-sample-record-max-len" 84) "256" 85) > > "profiling-sample-record-threshold-ms" 86) "100" 87) "proto-max-bulk-len" > > 88) "536870912" 89) "purge-backup-on-fullsync" 90) "no" 91) > > "redis-cursor-compatible" 92) "yes" 93) "repl-namespace-enabled" 94) "no" > > 95) "replica-announce-ip" 96) "" 97) "replica-announce-port" 98) "0" 99) > > "replicaof"100) ""101) "replication-connect-timeout-ms"102) "3100"103) > > "replication-recv-timeout-ms"104) "3200"105) "requirepass"106) ""107) > > "resp3-enabled"108) "no"109) "rocksdb.avoid_unnecessary_blocking_io"110) > > "yes"111) "rocksdb.blob_file_size"112) "268435456"113) > > "rocksdb.blob_garbage_collection_age_cutoff"114) "25"115) > > "rocksdb.block_cache_size"116) "256"117) "rocksdb.block_cache_type"118) > > "lru"119) "rocksdb.block_size"120) "16384"121) > > "rocksdb.cache_index_and_filter_blocks"122) "yes"123) > > "rocksdb.compaction_readahead_size"124) "2097152"125) > > "rocksdb.compression"126) "no"127) "rocksdb.compression_level"128) > > "32767"129) "rocksdb.compression_start_level"130) "2"131) > > "rocksdb.delayed_write_rate"132) "0"133) > > "rocksdb.disable_auto_compactions"134) "no"135) > > "rocksdb.dump_malloc_stats"136) "yes"137) "rocksdb.enable_blob_files"138) > > "no"139) "rocksdb.enable_blob_garbage_collection"140) "yes"141) > > "rocksdb.enable_pipelined_write"142) "no"143) > > "rocksdb.level0_file_num_compaction_trigger"144) "4"145) > > "rocksdb.level0_slowdown_writes_trigger"146) "20"147) > > "rocksdb.level0_stop_writes_trigger"148) "40"149) > > "rocksdb.level_compaction_dynamic_level_bytes"150) "no"151) > > "rocksdb.max_background_compactions"152) "2"153) > > "rocksdb.max_background_flushes"154) "2"155) > > "rocksdb.max_background_jobs"156) "4"157) > > "rocksdb.max_bytes_for_level_base"158) "268435456"159) > > "rocksdb.max_bytes_for_level_multiplier"160) "10"161) > > "rocksdb.max_open_files"162) "8096"163) "rocksdb.max_subcompactions"164) > > "2"165) "rocksdb.max_total_wal_size"166) "256"167) > > "rocksdb.max_write_buffer_number"168) "2"169) > > "rocksdb.metadata_block_cache_size"170) "2048"171) > > "rocksdb.min_blob_size"172) "4096"173) "rocksdb.partition_filters"174) > > "yes"175) "rocksdb.rate_limiter_auto_tuned"176) "yes"177) > > "rocksdb.read_options.async_io"178) "yes"179) "rocksdb.row_cache_size"180) > > "0"181) "rocksdb.share_metadata_and_subkey_block_cache"182) "yes"183) > > "rocksdb.stats_dump_period_sec"184) "0"185) > > "rocksdb.subkey_block_cache_size"186) "2048"187) > > "rocksdb.target_file_size_base"188) "128"189) "rocksdb.wal_compression"190) > > "no"191) "rocksdb.wal_size_limit_mb"192) "16384"193) > > "rocksdb.wal_ttl_seconds"194) "10800"195) "rocksdb.write_buffer_size"196) > > "32"197) "rocksdb.write_options.disable_wal"198) "no"199) > > "rocksdb.write_options.low_pri"200) "no"201) > > "rocksdb.write_options.memtable_insert_hint_per_batch"202) "no"203) > > "rocksdb.write_options.no_slowdown"204) "no"205) > > "rocksdb.write_options.sync"206) "no"207) > > "rocksdb.write_options.write_batch_max_bytes"208) "0"209) > > "skip-block-cache-deallocation-on-close"210) "no"211) > > "slave-empty-db-before-fullsync"212) "no"213) "slave-priority"214) > > "100"215) "slave-read-only"216) "yes"217) "slave-serve-stale-data"218) > > "yes"219) "slaveof"220) ""221) "slowlog-log-slower-than"222) "200000"223) > > "slowlog-max-len"224) "128"225) "socket-fd"226) "-1"227) "supervised"228) > > "no"229) "tcp-backlog"230) "511"231) "timeout"232) "0"233) > > "tls-auth-clients"234) ""235) "tls-ca-cert-dir"236) ""237) > > "tls-ca-cert-file"238) ""239) "tls-cert-file"240) ""241) "tls-ciphers"242) > > ""243) "tls-ciphersuites"244) ""245) "tls-key-file"246) ""247) > > "tls-key-file-pass"248) ""249) "tls-port"250) "0"251) > > "tls-prefer-server-ciphers"252) "no"253) "tls-protocols"254) ""255) > > "tls-replication"256) "no"257) "tls-session-cache-size"258) "20480"259) > > "tls-session-cache-timeout"260) "300"261) "tls-session-caching"262) > > "yes"263) "txn-context-enabled"264) "no"265) "unixsocket"266) ""267) > > "unixsocketperm"268) "777"269) "use-rsid-psync"270) "no"271) "workers"272) > > "8" > > > >> Vào ngày thg 4 18, 2025 5:49 CH, từ Hulk <hulk.webs...@gmail.com>Hi > > Quan, > > > >> > > > >> Thanks for your information, could you please provide the output by > > running > > > >> the command in redis-cli `config get *`? > > > >> > > > >> On Fri, 18 Apr 2025 at 18:24, Hong Quan TRAN <hqt...@linagora.com> > > wrote: > > > >> > > > >> > Hi the Kvrocks community, > > > >> > We would like to update you our progress on the integration topic. > > > >> > We have started to migrate a Redis deployment to Kvrocks for a > > week. The > > > >> > Kvrocks is running stable I would say. However, we have concern > > after > > > >> > observing the Kvrocks's memory comsuption as it keeps slowly > > consume more > > > >> > memory without releasing them back to the OS. At first the memory > > usage was > > > >> > 500MB, then it is up to 700MB, 1GB, 1.3GB... > > > >> > Please note that we already tuning the memory usage using the > > > >> > recommendations at github.com/apache/kvrocks/discussions/2767. > > > >> > ```kvrocks.conflog-level > > > >> > <github.com/apache/kvrocks/discussions/2767.kvrocks.conflog-level> > > > >> > infoport 6379rocksdb.max_total_wal_size > > 256rocksdb.max_write_buffer_number > > > >> > 2rocksdb.write_buffer_size 32rocksdb.block_cache_size 256``` > > > >> > We expect the memory consumption would be low and stable with this > > applied > > > >> > configuration. However, it keeps slowly increasing, which makes us > > worry. > > > >> > Our memory stats so far: > > > >> > ```docker statsCONTAINER ID NAME CPU % MEM USAGE / > > > >> > LIMIT MEM % NET I/O BLOCK I/O PIDS561e141f9b72 > > > >> > kvrocks 0.14% 1.292GiB / 14.3GiB 9.03% 4.77GB / > > > >> > 1.51GB 270kB / 8.08GB 19``` > > > >> > ```kvrocks@561e141f9b72:/$ redis-cli -p 6379 info memory# > > > >> > > > Memoryused_memory_rss:983244800used_memory_rss_human:937.70Mused_memory_lua:59392used_memory_lua_human:58.00Kused_memory_startup:63860736``` > > > >> > FYI we are using Kvrocks version 2.11.1. > > > >> > Could you help provide insights and advice on the Kvrocks memory > > > >> > management topic? How to limit the Kvrocks memory usage effectively? > > > >> > > > > >> > Best regards, > > > >> > Quan > > > >> > > > > >> > > > > >> > > > >> -- > > > >> Best Regards, > > > >> - *Hulk Lin* > > > > > > > > > > > > > > > > -- > > > > Best Regards, > > > > - Hulk Lin > > > > > > > > > > > > -- > > > Best Regards, > > > - Hulk Lin > > > > > > > > > -- > > Best Regards, > > - Hulk Lin > > > > > -- > Best Regards, > - *Hulk Lin* -- Best Regards, - Hulk Lin
