bitflicker64 opened a new issue, #3256: URL: https://github.com/apache/hugegraph/issues/3256
### Bug Type (问题类型) server status (启动/运行异常) ### Before submit - [x] I have confirmed and searched that there are no similar problems in the historical issue and documents Searched open and closed issues and PRs for `hs_err`, `ErrorFile`, `HeapDumpPath`, `additivity`, `STDOUT_MODE`, `Dockerfile-hstore`, `docker logs`, `kubectl logs` and `Starting HugeGraphServer failed`. #2979 / #2980 wired the `console` appender and set `STDOUT_MODE=true` in `hugegraph-server/Dockerfile` (the standalone `hugegraph/hugegraph` image), but not in `hugegraph-server/Dockerfile-hstore`, which builds `hugegraph/server` ([docker/README.md:240](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/docker/README.md#L240)). #3203 hit the same wall during a cold-start failure but tracked the retry bug, not the logging. ### Environment (环境信息) - Server Version: `hugegraph/server` images. 2026-09-10: published `latest` (built 2026-09-08) and images built from master `60c8803d` (#3203). 2026-09-22: images built from master `83ef9f3`. All line numbers below are from master `176fb56dd` (2026-10-02). - Backend: hstore, 3 PD + 3 Store + 3 Server - OS: Kubernetes (kind), HugeGraph Helm chart (#3218, earlier #3132) - Data Size: empty (2026-09-10) or a few test writes (2026-09-22); the failure is at Server startup ### Expected & Actual behavior (期望与实际表现) **Observed.** In the 2026-09-10 run (#3203), `kubectl logs --previous` for a Server that exited during startup ended with these lines and held no stack trace: ``` Connecting to HugeGraphServer (http://0.0.0.0:8080/graphs)...........Starting HugeGraphServer failed See /hugegraph-server/logs/hugegraph-server.log for HugeGraphServer log output. ``` (printed by [`util.sh:362`](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh#L362) and [`start-hugegraph.sh:117`](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/bin/start-hugegraph.sh#L117)). The second line is the non-`STDOUT_MODE` branch. The cause was in `logs/hugegraph-server.log`, which went away with the container; the trace in #3203 was only recovered by mounting a volume at `/hugegraph-server/logs` before the first boot. On 2026-09-22 one Server started during a `helm upgrade` exited once with `Starting HugeGraphServer failed`, and its cause was lost the same way. **Expected.** After a Server container exits, its stdout (`kubectl logs --previous`) shows the error, and the JVM crash files sit in a documented directory an operator can mount. **Actual.** Four things keep diagnostics inside the container filesystem. 1. The HStore image does not set `STDOUT_MODE`. `Dockerfile-hstore` sets only `JAVA_OPTS` and `HUGEGRAPH_HOME` ([:43-44](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/Dockerfile-hstore#L43-L44)), and the published `hugegraph/server:latest` (amd64 config, created 2026-10-01, read from the Docker Hub registry on 2026-10-02) has no `STDOUT_MODE` in its `Env`. Without it, `hugegraph-server.sh` sends the JVM's stdout and stderr, `console` appender included, to `logs/hugegraph-server-stdout.log` ([:266-270](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh#L266-L270)), so nothing the JVM logs reaches `kubectl logs`. 2. Even with `STDOUT_MODE=true`, five loggers write only to the file. `log4j2.xml` sets `LOG_PATH` to `logs` ([:21](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml#L21)) and the `file` appender writes `${LOG_PATH}/hugegraph-server.log` ([:32](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml#L32)). Since #2980 the root logger and `org.apache.hugegraph` also go to `console` ([:106-109](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml#L106-L109), [:126-129](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml#L126-L129)). `org.apache.hadoop`, `org.apache.zookeeper`, `com.alipay.sofa`, `io.n etty` and `org.apache.commons` are `additivity="false"` with only `file` ([:110-124](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml#L110-L124)), so their errors never reach stdout. The `file` appender also has `immediateFlush="false"` ([:34](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml#L34)), so a killed JVM can lose its last lines from the file as well. The audit loggers ([:130-135](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml#L130-L135)) and the slow-query log ([:136-138](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml#L136-L138)) are file-only too. They are separate stream s and out of scope here. 3. JVM crash files land inside the install directory. `hugegraph-server.sh` sets `LOGS="$TOP/logs"` ([:41](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh#L41)), passes `-XX:HeapDumpPath=${LOGS}` ([:124](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh#L124)) and runs `cd "${TOP}"` before `exec java` ([:91](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh#L91)). Nothing in the repository sets `-XX:ErrorFile` (`git grep ErrorFile` on `176fb56dd` returns nothing), so HotSpot writes `hs_err_pid<pid>.log` to the working directory, `/hugegraph-server` in the image. The image adds nothing here. `JAVA_OPTS` ([Dockerfile-hstore:43](https://github.com/apache/hugegraph /blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/Dockerfile-hstore#L43)) reaches the script through `-j` ([docker-entrypoint.sh:243](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/docker/docker-entrypoint.sh#L243)) and is appended on line 124. Line 124 sits inside the `JAVA_OPTIONS` default block ([:118-129](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh#L118-L129)), so a user who sets `JAVA_OPTIONS` also loses `-XX:+HeapDumpOnOutOfMemoryError`. 4. The only persistence is an image `VOLUME`. `Dockerfile-hstore` has `WORKDIR /hugegraph-server/` ([:46](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/Dockerfile-hstore#L46)) and `VOLUME /hugegraph-server` ([:78](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/Dockerfile-hstore#L78)). Docker keeps that anonymous volume across `docker restart`. Kubernetes does not turn an image `VOLUME` into a pod volume. A restart starts a new container, and `logs/`, `hs_err_pid*.log` and heap dumps go with the old one. **Fix direction.** - Set `STDOUT_MODE="true"` in `Dockerfile-hstore`, as #2980 did in `Dockerfile`. - In `log4j2.xml`, also send WARN and above from the five loggers on lines 110-124 to `console` (a `console` ref with a WARN threshold, or a second console appender). INFO from netty and hadoop stays in the file. Audit and slow-query stay file-only. - In `hugegraph-server.sh`, add `-XX:ErrorFile=${LOGS}/hs_err_pid%p.log` and move `-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${LOGS}` out of the default block, so both apply whatever `JAVA_OPTIONS` holds. - Document `/hugegraph-server/logs` as the directory to mount on Kubernetes. If #3253 (for #3236) lands, these flags should follow its `LOGS_OVERRIDE` instead of `$TOP/logs`. Related: #2979 / #2980, #3203, #3236 / #3253, #3211 (non-root image, which would `chown` the same log directory). ### Vertex/Edge example (问题点 / 边数据举例) Not applicable. ### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构) Not applicable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
