byteayan opened a new pull request, #3258:
URL: https://github.com/apache/hugegraph/pull/3258
## Purpose of the PR
- close #3256
When a `hugegraph/server` (HStore) container exits during startup, `kubectl
logs --previous` shows only `Starting HugeGraphServer failed` and a pointer to
`logs/hugegraph-server.log`. That file, any `hs_err_pid*.log` and any heap dump
live inside the container, and Kubernetes discards them when it restarts the
pod. In #3203 the stack trace was only recovered by mounting a volume at
`/hugegraph-server/logs` before the first boot.
## Main Changes
- `Dockerfile-hstore` sets `STDOUT_MODE=true`. #2980 added it to the
standalone, PD and Store Dockerfiles and missed this one.
- `conf/log4j2.xml`: the `org.apache.hadoop`, `org.apache.zookeeper`,
`com.alipay.sofa`, `io.netty` and `org.apache.commons` loggers are
`additivity="false"` with only the `file` appender. Each gets `<appender-ref
ref="console" level="WARN"/>`, so WARN and above reaches stdout and INFO stays
in the file. Root and `org.apache.hugegraph` already log to console. Audit and
slow-query logs are unchanged.
- `hugegraph-server.sh`:
- `-XX:+HeapDumpOnOutOfMemoryError` and `-XX:HeapDumpPath` used to sit
inside the `JAVA_OPTIONS` default block, so setting `JAVA_OPTIONS` dropped
them. Nothing set `-XX:ErrorFile`, so `hs_err` files went to the working
directory (the install root). Both are now set on every start, pointing at
`$LOGS`.
- File names are `java_pid<pid>_<launch time>.hprof` and
`hs_err_pid%p_<launch time>.log`. A restarted container often gets the same
PID, and HotSpot neither overwrites an existing `hs_err` file (it falls back to
the working directory) nor writes a heap dump over an existing one. The script
`exec`s java, so `$$` is the JVM's PID.
- The flags go before `JAVA_OPTIONS` and `-j`, so a value given there wins
(HotSpot takes the last occurrence). `JAVA_TOOL_OPTIONS` and `JDK_JAVA_OPTIONS`
are read before the command line, so if either already sets one of these flags,
the launcher does not pass it.
- They are built from `$LOGS`, so they follow `LOGS_OVERRIDE` if #3253
lands. The two PRs touch nearby lines in this script.
- `start-hugegraph.sh`: in `STDOUT_MODE` the failure message now points at
"container logs ('docker logs' or 'kubectl logs')", since the same image runs
on Kubernetes.
- `docker/README.md`: new "Server logs and crash files" section covering
what reaches stdout, the file names under `/hugegraph-server/logs`, mounting an
`emptyDir` or PVC there on Kubernetes, heap-dump size, and turning dumps off
with `-XX:-HeapDumpOnOutOfMemoryError` in `JAVA_OPTS`.
Outside containers, a bare-metal install that sets `JAVA_OPTIONS` now also
writes a heap dump to `logs/` on OOM. The dump can be as large as the heap.
Not in this PR:
- `immediateFlush="false"` on the `file` appender (mentioned in the issue).
WARN and above now also reach the console, which flushes each line.
- With `STDOUT_MODE=true` the launcher omits
`-Dhugegraph.bootstrap.error.log`, so bootstrap fatal errors reach stdout only,
not a mounted `hugegraph-server.log`.
- The PD and Store start scripts also lack `-XX:ErrorFile`, which is left
for a separate PR.
- With telemetry enabled (off by default), the script replaces
`JAVA_TOOL_OPTIONS` with the agent options, which drops an operator's
`-XX:ErrorFile` set there. That replacement predates this PR.
## Verifying these changes
- [ ] Trivial rework / code cleanup without any test coverage. (No Need)
- [x] Already covered by existing tests, such as *(please modify tests
here)*.
- [ ] Need tests and can be verified as follows:
`test-java-security-properties.sh` (run by server-ci) now checks the three
flags and their file names, that `JAVA_OPTIONS` values for `-XX:ErrorFile` and
`-XX:HeapDumpPath` come last, and that flags already in `JAVA_TOOL_OPTIONS` or
`JDK_JAVA_OPTIONS` are not passed again. It passes with JDK 11 against a dist
built from this branch and fails against master's `hugegraph-server.sh`. I
checked the flag syntax with JDK 11 `-XX:+PrintFlagsFinal`. I did not build or
run the Docker images.
## Does this PR potentially affect the following parts?
- [ ] Dependencies ([add/update
license](https://hugegraph.apache.org/docs/contribution-guidelines/contribute/#321-check-licenses)
info &
[regenerate_known_dependencies.sh](../install-dist/scripts/dependency/regenerate_known_dependencies.sh))
- [x] Modify configurations
- [ ] The public API
- [x] Other affects (typed here): JVM startup flags in
`hugegraph-server.sh`, and the `hugegraph/server` image now logs to stdout
- [ ] Nope
## Documentation Status
- [x] `Doc - TODO`: required documentation is pending; complete it before
merging.
- [ ] `Doc - Done`: documentation is included here or linked below.
- [ ] `Doc - No Need`: no user-visible documentation is affected.
Documentation files in this PR or paired hugegraph-doc PR:
- `docker/README.md` (in this PR)
- Pending: hugegraph-doc note that heap dumps and crash logs now go to
`logs/` even when `JAVA_OPTIONS` is set
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
