byteayan commented on code in PR #3355: URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4392073656
########## hugegraph-store/docs/deployment-guide.md: ########## @@ -356,14 +356,82 @@ rocksdb: level0_file_num_compaction_trigger: 4 max_background_jobs: 8 # Background compaction/flush threads - # Block Cache - block_cache_size: 16000000000 # Block cache (16GB) + # Budget split: this share goes to memtables, the rest to the block cache + write_buffer_ratio: 0.66 # SST File Size target_file_size_base: 134217728 # Target SST size (128MB) max_bytes_for_level_base: 1073741824 # L1 size (1GB) ``` +`total_memory_size` bounds the memtables (`write_buffer_ratio` of it) and the block cache +(the rest). Both live outside the Java heap, so `-Xmx` does not limit them. A fixed 1 GiB +cache for the Raft log comes on top of this budget; the Raft log memtables of all partitions +share a 256 MiB write buffer charged to that cache, so they don't add to it. + +With `total_memory_size: auto` (the shipped value; a missing, blank or `0` value means the same) +the Store sizes the budget at startup as half of what is left after the heap, the maximum direct +memory and a 1.5 GiB reserve (the 1 GiB Raft log cache plus 512 MiB for other native memory), +clamped to 256 MiB–32 GB. Inside a container the starting point is the cgroup memory limit, including a +limit set only on a parent cgroup when the Store can see that cgroup; inside a private cgroup +namespace, or when only a subtree of the cgroup2 hierarchy is mounted, a limit above it is not +visible, so pin the value there if only the pod or slice is limited. The Store uses the lowest +limit it can see and logs a warning when it can't see the whole hierarchy; it doesn't guess at +limits it can't read. A Store with a 4 GiB limit, `-Xmx1g` and `-XX:MaxDirectMemorySize=512m` +gets 512 MiB. Without +`-XX:MaxDirectMemorySize` the JVM allows as much direct memory as heap, and the budget counts +it that way. `bin/start-hugegraph-store.sh` sets the flag to the heap it picks; when `-j` sets +`-Xmx`, `-XX:MaxHeapSize` or `-XX:MaxDirectMemorySize` itself, the launcher leaves the flag out and the JVM's +default (that heap) or the given cap applies. Either way the budget counts a limit the JVM +enforces. In small containers set a +cap below the heap, or the budget falls to the 256 MiB floor. + +On a host without a container limit the starting point is physical memory. The default +`bin/start-hugegraph-store.sh` caps the heap at 2 GiB and direct memory at the same 2 GiB, so a +16 GiB host gets about 5.25 GiB. When you raise the heap, pass a direct-memory cap with it, +for example `-j "-Xmx36g -XX:MaxDirectMemorySize=2g"`; otherwise direct memory counts as a +second heap. With a 2 GiB cap a 16 GiB host with `-Xmx8g` gets 2.25 GiB and a 64 GiB host with +`-Xmx36g` gets 12.25 GiB. Before this default every host got a fixed `32000000000`, so pin the +value after upgrading if a large node relied on it. + +At startup the Store logs the budget and what it was computed from, for example: + +```text +RocksDB memory budget: 512 MiB (536870912 bytes), mode=auto, physical-limit=4 GiB, source=cgroup-v2, heap=1 GiB, direct=512 MiB, reserve=1.5 GiB, cgroup-view=complete +``` + +It warns when the floor applies, or when an explicit value is under 256 MiB or exceeds the +memory left after the heap, direct memory and the reserve. Set a number of bytes, at least +1048576 (1 MiB), to pin the budget; a smaller value or a unit suffix such as `4GB` is rejected. Before this default existed the +shipped value was `32000000000`, which can exceed a container limit and get the Store +OOM-killed. Both caps are soft: RocksDB can briefly go over them under heavy writes, and a +small budget with many partitions means frequent memtable flushes. + +**Upgrading an existing deployment.** Auto-sizing applies only when the loaded configuration +has `total_memory_size: auto` (or `0`, or no value). A Store that mounts its own `conf/application-pd.yml`, +or a tarball install that kept its `conf/` from an earlier release, still has `32000000000` and +keeps that fixed budget; the Store logs a warning naming the old value at startup. Set the key +to `auto`, or set `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` in Docker. + +PD can also hold the old value. When PD has no shared copy yet, the first Store to start uploads +its local `application-pd.yml` to PD (KV key `S:FS`). At startup a Store loads that copy over its +local file only when it has no local file or the file is no newer than PD's `S:Timestamp`. The +upload sets no timestamp and a missing one counts as 0, so changing `S:FS` alone changes nothing. +The simplest upgrade is to set `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE=auto` (or +`-Drocksdb.total_memory_size=auto`) on every Store and restart it; the Store keeps this value +above both its local file and PD's copy. To change the shared copy instead, set +`rocksdb.total_memory_size` to `0` or remove it in `S:FS`, then put the current time in +milliseconds into `S:Timestamp` through PD's KV service (gRPC `KvService.put`). Use `auto` there +only once every Store runs this release: an earlier release can't parse `auto` and fails to start, +while it reads `0` as its old heap-sized default. This also applies to a copy that a Store of this +release uploaded when PD had none, since the shipped file says `auto`. Every running Store watches +`S:Timestamp`: it reloads PD's copy, writes it over its local file and restarts itself with +`bin/restart-hugegraph-store.sh` within 60 seconds. A stopped Store loads the copy at its next +start. That self-restart passes no arguments, so `-j` and `-g` from the original start are +dropped, including the `JAVA_OPTS` the Docker entrypoint passes as `-j`, and the launcher's Review Comment: Confirmed: with `-d false` the launcher execs Java as the entrypoint's last process, so the stop step ends the container before the start step runs. The guide now says a Docker Store exits with 143 and stays down without a restart policy, and points Docker users to `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` instead of `S:Timestamp`. It also notes that all Stores restart within the same minute. Fixed in 3cbb58fe9; hugegraph-doc#535 has the same change in eab98044. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
