byteayan commented on code in PR #3355: URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4386515984
########## hugegraph-store/docs/deployment-guide.md: ########## @@ -356,14 +356,76 @@ rocksdb: level0_file_num_compaction_trigger: 4 max_background_jobs: 8 # Background compaction/flush threads - # Block Cache - block_cache_size: 16000000000 # Block cache (16GB) + # Budget split: this share goes to memtables, the rest to the block cache + write_buffer_ratio: 0.66 # SST File Size target_file_size_base: 134217728 # Target SST size (128MB) max_bytes_for_level_base: 1073741824 # L1 size (1GB) ``` +`total_memory_size` bounds the memtables (`write_buffer_ratio` of it) and the block cache +(the rest). Both live outside the Java heap, so `-Xmx` does not limit them. A fixed 1 GiB +cache for the Raft log comes on top of this budget; the Raft log memtables of all partitions +share a 256 MiB write buffer charged to that cache, so they don't add to it. + +With `total_memory_size: auto` (the shipped value; a missing, blank or `0` value means the same) +the Store sizes the budget at startup as half of what is left after the heap, the maximum direct +memory and a 1.5 GiB reserve (the 1 GiB Raft log cache plus 512 MiB for other native memory), +clamped to 256 MiB–32 GB. Inside a container the starting point is the cgroup memory limit, including a +limit set only on a parent cgroup when the Store can see that cgroup; inside a private cgroup +namespace, or when only a subtree of the cgroup2 hierarchy is mounted, a limit above it is not +visible, so pin the value there if only the pod or slice is limited. The Store uses the lowest +limit it can see and logs a warning when it can't see the whole hierarchy; it doesn't guess at +limits it can't read. A Store with a 4 GiB limit, `-Xmx1g` and `-XX:MaxDirectMemorySize=512m` +gets 512 MiB. Without +`-XX:MaxDirectMemorySize` the JVM allows as much direct memory as heap, and the budget counts +it that way. `bin/start-hugegraph-store.sh` sets the flag to the heap it picks; when `-j` sets +`-Xmx`, `-XX:MaxHeapSize` or `-XX:MaxDirectMemorySize` itself, the launcher leaves the flag out and the JVM's +default (that heap) or the given cap applies. Either way the budget counts a limit the JVM +enforces. In small containers set a +cap below the heap, or the budget falls to the 256 MiB floor. + +On a host without a container limit the starting point is physical memory. The default +`bin/start-hugegraph-store.sh` caps the heap at 2 GiB and direct memory at the same 2 GiB, so a +16 GiB host gets about 5.25 GiB. When you raise the heap, pass a direct-memory cap with it, +for example `-j "-Xmx36g -XX:MaxDirectMemorySize=2g"`; otherwise direct memory counts as a +second heap. With a 2 GiB cap a 16 GiB host with `-Xmx8g` gets 2.25 GiB and a 64 GiB host with +`-Xmx36g` gets 12.25 GiB. Before this default every host got a fixed `32000000000`, so pin the +value after upgrading if a large node relied on it. + +At startup the Store logs the budget and what it was computed from, for example: + +```text +RocksDB memory budget: 512 MiB (536870912 bytes), mode=auto, physical-limit=4 GiB, source=cgroup-v2, heap=1 GiB, direct=512 MiB, reserve=1.5 GiB, cgroup-view=complete +``` + +It warns when the floor applies, or when an explicit value is under 256 MiB or exceeds the +memory left after the heap, direct memory and the reserve. Set a number of bytes, at least +1048576 (1 MiB), to pin the budget; a smaller value or a unit suffix such as `4GB` is rejected. Before this default existed the +shipped value was `32000000000`, which can exceed a container limit and get the Store +OOM-killed. Both caps are soft: RocksDB can briefly go over them under heavy writes, and a +small budget with many partitions means frequent memtable flushes. + +**Upgrading an existing deployment.** Auto-sizing applies only when the loaded configuration +has `total_memory_size: auto` (or `0`, or no value). A Store that mounts its own `conf/application-pd.yml`, +or a tarball install that kept its `conf/` from an earlier release, still has `32000000000` and +keeps that fixed budget; the Store logs a warning naming the old value at startup. Set the key +to `auto`, or set `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` in Docker. + +PD can also hold the old value. When PD has no shared copy yet, the first Store to start uploads +its local `application-pd.yml` to PD (KV key `S:FS`). At startup a Store loads that copy over its +local file only when it has no local file or the file is no newer than PD's `S:Timestamp`. The +upload sets no timestamp and a missing one counts as 0, so changing `S:FS` alone changes nothing. +The simplest upgrade is to set `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE=auto` (or +`-Drocksdb.total_memory_size=auto`) on every Store and restart it; the Store keeps this value +above both its local file and PD's copy. To change the shared copy instead, set +`rocksdb.total_memory_size` to `auto` in `S:FS`, then put the current time in Review Comment: Right, earlier releases run `Long.parseLong` on anything but `0`. Fixed in 63856ac31. The guide now says to set `0` or remove the key in `S:FS` (earlier releases read `0` as their old heap-sized default), and to put `auto` there only once every Store runs this release. It also covers a copy uploaded from this release's shipped file, which already says `auto`. The same commit notes that the `S:Timestamp` self-restart drops `-j`/`-g`, the point you raised on apache/hugegraph-doc#535. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
