bitflicker64 opened a new issue, #3256:
URL: https://github.com/apache/hugegraph/issues/3256

   ### Bug Type (问题类型)
   
   server status (启动/运行异常)
   
   ### Before submit
   
   - [x] I have confirmed and searched that there are no similar problems in 
the historical issue and documents
   
   Searched open and closed issues and PRs for `hs_err`, `ErrorFile`, 
`HeapDumpPath`, `additivity`, `STDOUT_MODE`, `Dockerfile-hstore`, `docker 
logs`, `kubectl logs` and `Starting HugeGraphServer failed`. #2979 / #2980 
wired the `console` appender and set `STDOUT_MODE=true` in 
`hugegraph-server/Dockerfile` (the standalone `hugegraph/hugegraph` image), but 
not in `hugegraph-server/Dockerfile-hstore`, which builds `hugegraph/server` 
([docker/README.md:240](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/docker/README.md#L240)).
 #3203 hit the same wall during a cold-start failure but tracked the retry bug, 
not the logging.
   
   ### Environment (环境信息)
   
   - Server Version: `hugegraph/server` images. 2026-09-10: published `latest` 
(built 2026-09-08) and images built from master `60c8803d` (#3203). 2026-09-22: 
images built from master `83ef9f3`. All line numbers below are from master 
`176fb56dd` (2026-10-02).
   - Backend: hstore, 3 PD + 3 Store + 3 Server
   - OS: Kubernetes (kind), HugeGraph Helm chart (#3218, earlier #3132)
   - Data Size: empty (2026-09-10) or a few test writes (2026-09-22); the 
failure is at Server startup
   
   ### Expected & Actual behavior (期望与实际表现)
   
   **Observed.** In the 2026-09-10 run (#3203), `kubectl logs --previous` for a 
Server that exited during startup ended with these lines and held no stack 
trace:
   
   ```
   Connecting to HugeGraphServer 
(http://0.0.0.0:8080/graphs)...........Starting HugeGraphServer failed
   See /hugegraph-server/logs/hugegraph-server.log for HugeGraphServer log 
output.
   ```
   
   (printed by 
[`util.sh:362`](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh#L362)
 and 
[`start-hugegraph.sh:117`](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/bin/start-hugegraph.sh#L117)).
 The second line is the non-`STDOUT_MODE` branch. The cause was in 
`logs/hugegraph-server.log`, which went away with the container; the trace in 
#3203 was only recovered by mounting a volume at `/hugegraph-server/logs` 
before the first boot. On 2026-09-22 one Server started during a `helm upgrade` 
exited once with `Starting HugeGraphServer failed`, and its cause was lost the 
same way.
   
   **Expected.** After a Server container exits, its stdout (`kubectl logs 
--previous`) shows the error, and the JVM crash files sit in a documented 
directory an operator can mount.
   
   **Actual.** Four things keep diagnostics inside the container filesystem.
   
   1. The HStore image does not set `STDOUT_MODE`. `Dockerfile-hstore` sets 
only `JAVA_OPTS` and `HUGEGRAPH_HOME` 
([:43-44](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/Dockerfile-hstore#L43-L44)),
 and the published `hugegraph/server:latest` (amd64 config, created 2026-10-01, 
read from the Docker Hub registry on 2026-10-02) has no `STDOUT_MODE` in its 
`Env`. Without it, `hugegraph-server.sh` sends the JVM's stdout and stderr, 
`console` appender included, to `logs/hugegraph-server-stdout.log` 
([:266-270](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh#L266-L270)),
 so nothing the JVM logs reaches `kubectl logs`.
   
   2. Even with `STDOUT_MODE=true`, five loggers write only to the file. 
`log4j2.xml` sets `LOG_PATH` to `logs` 
([:21](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml#L21))
 and the `file` appender writes `${LOG_PATH}/hugegraph-server.log` 
([:32](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml#L32)).
 Since #2980 the root logger and `org.apache.hugegraph` also go to `console` 
([:106-109](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml#L106-L109),
 
[:126-129](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml#L126-L129)).
 `org.apache.hadoop`, `org.apache.zookeeper`, `com.alipay.sofa`, `io.n
 etty` and `org.apache.commons` are `additivity="false"` with only `file` 
([:110-124](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml#L110-L124)),
 so their errors never reach stdout. The `file` appender also has 
`immediateFlush="false"` 
([:34](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml#L34)),
 so a killed JVM can lose its last lines from the file as well. The audit 
loggers 
([:130-135](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml#L130-L135))
 and the slow-query log 
([:136-138](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml#L136-L138))
 are file-only too. They are separate stream
 s and out of scope here.
   
   3. JVM crash files land inside the install directory. `hugegraph-server.sh` 
sets `LOGS="$TOP/logs"` 
([:41](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh#L41)),
 passes `-XX:HeapDumpPath=${LOGS}` 
([:124](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh#L124))
 and runs `cd "${TOP}"` before `exec java` 
([:91](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh#L91)).
 Nothing in the repository sets `-XX:ErrorFile` (`git grep ErrorFile` on 
`176fb56dd` returns nothing), so HotSpot writes `hs_err_pid<pid>.log` to the 
working directory, `/hugegraph-server` in the image. The image adds nothing 
here. `JAVA_OPTS` ([Dockerfile-hstore:43](https://github.com/apache/hugegraph
 
/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/Dockerfile-hstore#L43))
 reaches the script through `-j` 
([docker-entrypoint.sh:243](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/docker/docker-entrypoint.sh#L243))
 and is appended on line 124. Line 124 sits inside the `JAVA_OPTIONS` default 
block 
([:118-129](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh#L118-L129)),
 so a user who sets `JAVA_OPTIONS` also loses `-XX:+HeapDumpOnOutOfMemoryError`.
   
   4. The only persistence is an image `VOLUME`. `Dockerfile-hstore` has 
`WORKDIR /hugegraph-server/` 
([:46](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/Dockerfile-hstore#L46))
 and `VOLUME /hugegraph-server` 
([:78](https://github.com/apache/hugegraph/blob/176fb56dd747ef0f60a126cf721aa12d17627c31/hugegraph-server/Dockerfile-hstore#L78)).
 Docker keeps that anonymous volume across `docker restart`. Kubernetes does 
not turn an image `VOLUME` into a pod volume. A restart starts a new container, 
and `logs/`, `hs_err_pid*.log` and heap dumps go with the old one.
   
   **Fix direction.**
   
   - Set `STDOUT_MODE="true"` in `Dockerfile-hstore`, as #2980 did in 
`Dockerfile`.
   - In `log4j2.xml`, also send WARN and above from the five loggers on lines 
110-124 to `console` (a `console` ref with a WARN threshold, or a second 
console appender). INFO from netty and hadoop stays in the file. Audit and 
slow-query stay file-only.
   - In `hugegraph-server.sh`, add `-XX:ErrorFile=${LOGS}/hs_err_pid%p.log` and 
move `-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${LOGS}` out of the 
default block, so both apply whatever `JAVA_OPTIONS` holds.
   - Document `/hugegraph-server/logs` as the directory to mount on Kubernetes. 
If #3253 (for #3236) lands, these flags should follow its `LOGS_OVERRIDE` 
instead of `$TOP/logs`.
   
   Related: #2979 / #2980, #3203, #3236 / #3253, #3211 (non-root image, which 
would `chown` the same log directory).
   
   ### Vertex/Edge example (问题点 / 边数据举例)
   
   Not applicable.
   
   ### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)
   
   Not applicable.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to