bitflicker64 opened a new issue, #3254:
URL: https://github.com/apache/hugegraph/issues/3254

   ### Bug Type (问题类型)
   
   server status (启动/运行异常)
   
   ### Before submit
   
   - [x] 我已经确认现有的 [Issues](https://github.com/apache/hugegraph/issues) 与 
[FAQ](https://hugegraph.apache.org/docs/guides/faq/) 中没有相同 / 重复问题 (I have 
confirmed and searched that there are no similar problems in the historical 
issue and documents)
   
   Searched for `total_memory_size`, `store OOM`, `SPRING_APPLICATION_JSON` and 
`application-pd.yml`. #2847 contains `total_memory_size: 64GB`, but that is the 
reporter's own config pasted into a data-sync bug, not this problem.
   
   ### Environment (环境信息)
   
   - Code: apache/hugegraph master `176fb56dd` (2026-10-02). All line 
references below are at that commit.
   - Backend: HStore, 3 Stores on k3s, cluster preset of the Helm chart in 
hugegraph/hugegraph#221 (chart `05f3d9e`, PD/Store images of 2026-09-18), Store 
started with `-Xmx1024m -XX:MaxDirectMemorySize=512m`
   - Data Size: load of 5M vertices with a 1 KB text property each; the Stores 
died from vertex 2.36M on, holding about 1.0-1.3 GB of RocksDB data and 4.2 GB 
of raft log each
   - Run and measurements by Sebastian Gruza, 2026-09-19: 
https://github.com/hugegraph/hugegraph/pull/221#issuecomment-5740338545
   - All three Stores in that run log profiles `"pd", "default"` and 
`total_memory_size:32000000000` at startup ([store-1, lines 28 and 
44-48](https://github.com/SebastianGruza/hugegraph-validation/blob/52c52778263e4ba81686a86de8eac2cf4b0df7ae/results/store-oom-cluster-preset/store-1-previous.log#L28-L48)).
 The comment's memory accounting assumes the key is unset and falls back to the 
heap size; these logs show it is set.
   
   ### Expected & Actual behavior (期望与实际表现)
   
   **Expected:** the Store's RocksDB budget fits inside the memory the process 
is given, or a Docker user can set it.
   
   **Actual:** loading the data above OOM-killed all three Stores at a 4Gi 
container limit (`Memory cgroup out of memory: Killed process (java) 
anon-rss:4154896kB`), and they crash-looped afterwards. With the limit raised 
to 8Gi they stayed up, and store-1 measured 4.42 GiB anonymous RSS. The 8Gi 
limit only adds headroom; nothing bounds RocksDB below 32 GB.
   
   Why nothing keeps RocksDB inside the limit, in code (the 4.42 GiB was not 
broken down per allocator, so the RocksDB share of it is not measured):
   
   1. 
[`application-pd.yml:28-30`](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-store/hg-store-dist/src/assembly/static/conf/application-pd.yml#L28-L30)
 sets `rocksdb.total_memory_size: 32000000000`. The shipped `application.yml` 
includes the `pd` profile 
([`application.yml:57-59`](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-store/hg-store-dist/src/assembly/static/conf/application.yml#L57-L59)).
 The copy inside the jar has the same value 
(`hg-store-node/src/main/resources/application-pd.yml:37`).
   2. Because the key is set, the existing fallback in 
[`AppConfig.java:104-107`](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/AppConfig.java#L104-L107)
 (use `Runtime.maxMemory()` when the key is missing or `0`) never runs.
   3. 
[`RaftRocksdbOptions.java:155-167`](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RaftRocksdbOptions.java#L155-L167)
 splits the value into a write cache for the `WriteBufferManager` and a block 
cache. With the default `write_buffer_ratio` of 0.66 
(`hg-store-rocksdb/.../RocksDBOptions.java:43-49`) that is about 21.1 GB and 
10.9 GB. Both are native `LRUCache`s, off-heap, so together they grow with data 
up to 32 GB and are not limited by the heap or the cgroup. 
`RaftRocksdbOptions.java:69` adds a separate 1 GB `LRUCache` for the raft log 
storage on top.
   4. The heap is limited separately. `hugegraph-store/Dockerfile:43` sets 
`-XX:MaxRAMPercentage=50`, but `start-hugegraph-store.sh:145-152` also passes 
`-Xmx` from `calc_xmx` (`util.sh:179-197`: 512 MB to 2 GB, half of free memory 
read from `/proc/meminfo`), and an explicit `-Xmx` takes precedence. Neither 
value feeds into `total_memory_size`.
   5. Docker users cannot change it. 
[`docker-entrypoint.sh:58-70`](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-store/hg-store-dist/docker/docker-entrypoint.sh#L58-L70)
 builds `SPRING_APPLICATION_JSON` from the five `HG_STORE_*` variables only and 
exports it, so a `SPRING_APPLICATION_JSON` passed to the container is 
overwritten, and no `HG_STORE_*` variable maps to a `rocksdb.*` key.
   
   Possible workaround, untested: `AppConfig.java:307-313` binds `rocksdb` as a 
map with `@ConfigurationProperties(prefix = "")`, so `JAVA_OPTS="... 
-Drocksdb.total_memory_size=<bytes>"` may reach the map key. Nobody has 
verified that it binds.
   
   **Fix direction (proposal):**
   
   - Derive the default from the memory the process actually has (cgroup limit, 
or the limit minus heap and direct memory) instead of a fixed 32 GB. Dropping 
the line from `application-pd.yml` alone would hand the budget to the 
`Runtime.maxMemory()` fallback, which sizes RocksDB equal to the heap; that is 
a smaller number but still a second heap-sized allocation on top of the first.
   - Add an `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` (or similar) variable to the 
entrypoint's JSON, so container and Helm users can set it explicitly.
   - `hugegraph-store/docs/deployment-guide.md:348` documents the 32 GB value 
and would need the same change.
   
   ### Vertex/Edge example (问题点 / 边数据举例)
   
   Not data specific. In the reported run the OOM came once each Store held 
about 1.0-1.3 GB of RocksDB data.
   
   ### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)
   
   Not schema specific. The reported run used one vertex label with a 1 KB text 
property.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to