byteayan commented on code in PR #3258: URL: https://github.com/apache/hugegraph/pull/3258#discussion_r4209606155
########## docker/README.md: ########## @@ -230,6 +230,12 @@ Standalone stores RocksDB data at `/hugegraph-server/rocksdb-data`. The HStore t `docker compose down` keeps named-volume data. `docker compose down -v` intentionally deletes it. +### Server logs and crash files + +Both Server images (`hugegraph/hugegraph` and `hugegraph/server`) write the HugeGraph log to container stdout. WARN and above from Hadoop, ZooKeeper, SOFA, Netty and Commons also reaches stdout. INFO from Hadoop, Netty and Commons, the audit log and the slow-query log stay in files only; ZooKeeper and SOFA log nothing below WARN. Launcher errors that stop the Server before Java starts, such as an unsupported JDK or too little free memory, go to stderr as well as `hugegraph-server.log`, so when a Server exits during startup, `docker logs` or `kubectl logs --previous` shows why. Review Comment: Fixed in 17d88225 by routing the error through the log helper. `configure_riscv64_libatomic` now takes an optional error handler, like `ensure_path_writable`, and the launcher defines `report_error` before calling it, so a missing `libatomic.so.1` reaches `hugegraph-server.log` as well as stderr (when `logs/` is writable). `dump-store.sh`, `init-store.sh` and the smoke tests call it without the argument and keep the stderr message. The test mocks `uname` and `ldconfig` and checks both destinations. ########## docker/README.md: ########## @@ -230,6 +230,12 @@ Standalone stores RocksDB data at `/hugegraph-server/rocksdb-data`. The HStore t `docker compose down` keeps named-volume data. `docker compose down -v` intentionally deletes it. +### Server logs and crash files + +Both Server images (`hugegraph/hugegraph` and `hugegraph/server`) write the HugeGraph log to container stdout. WARN and above from Hadoop, ZooKeeper, SOFA, Netty and Commons also reaches stdout. INFO from Hadoop, Netty and Commons, the audit log and the slow-query log stay in files only; ZooKeeper and SOFA log nothing below WARN. Launcher errors that stop the Server before Java starts, such as an unsupported JDK or too little free memory, go to stderr as well as `hugegraph-server.log`, so when a Server exits during startup, `docker logs` or `kubectl logs --previous` shows why. + +Files stay under `/hugegraph-server/logs`: the full `hugegraph-server.log`, JVM crash logs (`hs_err_pid<pid>_<host>_<launch time>.log`) and out-of-memory heap dumps (`heapdump_<host>_<launch time>/java_pid<pid>.hprof`). Each launch gets its own `heapdump_*` directory, so the Server and any JVM it starts, such as a computer job, write separate dumps. The host name (the pod name on Kubernetes) keeps pods that share one log volume from picking the same name, and a `-1`, `-2`, ... counter is added when a name is already taken. `docker restart` keeps the container filesystem, so the files survive it. Kubernetes starts a new container on every restart and does not turn the image's `VOLUME` into a pod volume, so mount an `emptyDir` or a PersistentVolumeClaim at `/hugegraph-server/logs` to keep them. A heap dump can be as large as the JVM heap, and every launch writes to a new file, so a Server that keeps running out of memory adds one heap-sized dump per restart until the volume is full. T he launcher never removes old dumps or `heapdump_*` directories, whether the volume is an `emptyDir` (which also survives container restarts) or a PVC. Each start leaves a directory behind, empty unless something ran out of memory, because a computer-job JVM can keep using it after its Server exits; old ones can be deleted once no HugeGraph JVM from that launch is running. Size the volume for the number of dumps you want to keep plus the logs, move or delete old `heapdump_*` directories after each incident, or turn heap dumps off with `JAVA_TOOL_OPTIONS=-XX:-HeapDumpOnOutOfMemoryError`, which keeps the image's default `JAVA_OPTS`. Adding the flag to `JAVA_OPTS` also works, but setting `JAVA_OPTS` replaces that default (`-XX:+UseContainerSupport -XX:MaxRAMPercentage=50 ...`), so repeat those flags. Review Comment: Fixed in 17d88225 and 87e45f74: `docker/README.md` now says `JAVA_TOOL_OPTIONS` must be set in the container's environment, via `docker run -e`, an `environment:` entry on the Server service in Compose (the shipped Compose files do not pass it through), or `env` in Kubernetes, and that exporting it in the host shell has no effect. apache/hugegraph-doc#509 says the same in English and Chinese. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
