bitflicker64 commented on code in PR #3360:
URL: https://github.com/apache/hugegraph/pull/3360#discussion_r4294281826
##########
hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh:
##########
@@ -16,6 +16,50 @@
# limitations under the License.
#
+function record_jvm_startup_diagnostics() {
+ JVM_STARTUP_TIME=$(date -u +'%Y-%m-%d %H:%M:%S.%N UTC')
+ JVM_STARTUP_LOG_OFFSET=$(stat -c %s ./logs/hugegraph-server.log
2>/dev/null || echo 0)
+}
+
+function print_jvm_thread_hint() {
+ local kind
+ kind=$(awk '
+ /pthread_create.*\(EPERM\)|pthread_create.*Operation not permitted/ {
denied = 1 }
+ /[Tt]hread.*Operation not permitted/ { denied = 1 }
+ /Cannot create worker GC thread/ { resources = 1 }
+ END { if (denied) print "denied"; else if (resources) print
"resources" }
+ ')
+ case "$kind" in
+ denied)
+ echo 'ERROR: JVM thread creation was denied (EPERM). Check the
Docker host seccomp profile and thread limits.' >&2
+ echo 'To diagnose seccomp restrictions, retry with --security-opt
seccomp=unconfined; use a compatible host/runtime profile for normal
operation.' >&2
+ echo 'If the process limit is too low, set --ulimit
nproc=65535:65535.' >&2
Review Comment:
Minor: The EPERM branch tells users to raise `--ulimit nproc`, and line 34
points them at thread limits too, but a process or thread limit cannot cause
this error. When `RLIMIT_NPROC` or a cgroup `pids.max` is hit,
`clone`/`pthread_create` fail with EAGAIN (pthread_create(3): "A system-imposed
limit on the number of threads was encountered"), and the JVM then logs
`pthread_create failed (EAGAIN)`. EPERM points at a seccomp or LSM denial, so
the `nproc` advice sends people after a cause that cannot produce this message.
Requested change: drop the `nproc` line and the "thread limits" wording from
the `denied` branch, keep them in the `resources` branch, and match the README
paragraph to that.
##########
hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh:
##########
@@ -16,6 +16,50 @@
# limitations under the License.
#
+function record_jvm_startup_diagnostics() {
+ JVM_STARTUP_TIME=$(date -u +'%Y-%m-%d %H:%M:%S.%N UTC')
+ JVM_STARTUP_LOG_OFFSET=$(stat -c %s ./logs/hugegraph-server.log
2>/dev/null || echo 0)
+}
+
+function print_jvm_thread_hint() {
+ local kind
+ kind=$(awk '
+ /pthread_create.*\(EPERM\)|pthread_create.*Operation not permitted/ {
denied = 1 }
+ /[Tt]hread.*Operation not permitted/ { denied = 1 }
+ /Cannot create worker GC thread/ { resources = 1 }
+ END { if (denied) print "denied"; else if (resources) print
"resources" }
+ ')
+ case "$kind" in
+ denied)
+ echo 'ERROR: JVM thread creation was denied (EPERM). Check the
Docker host seccomp profile and thread limits.' >&2
+ echo 'To diagnose seccomp restrictions, retry with --security-opt
seccomp=unconfined; use a compatible host/runtime profile for normal
operation.' >&2
+ echo 'If the process limit is too low, set --ulimit
nproc=65535:65535.' >&2
+ ;;
+ resources)
+ echo 'ERROR: JVM could not create a GC worker thread. Check
available memory and process/thread limits.' >&2
+ echo 'If the process limit is too low, set --ulimit
nproc=65535:65535.' >&2
+ ;;
+ *) return 1 ;;
+ esac
+}
+
+function diagnose_jvm_startup_failure() {
+ local crash size offset
+ while IFS= read -r -d '' crash; do
+ if [[ -r "$crash" ]] && print_jvm_thread_hint < "$crash"; then
Review Comment:
Important: In the #3072 failure the crash file never contains the EPERM
text, so this loop takes the `resources` branch and returns. Users get the
memory hint and never see the seccomp one.
HotSpot prints `Failed to start thread "GC Thread#0" - pthread_create failed
(EPERM)` as a unified-logging warning on its stdout. The `hs_err_pid*.log` file
only carries the generic header. I reproduced it on Docker 29.5.3 with a
seccomp profile that returns EPERM for `clone3`:
```
docker run --rm --security-opt seccomp=$d/seccomp.json -v $d/out:/out -w
/out \
eclipse-temurin:17-jre-jammy java -XX:ErrorFile=/out/hs_err_pid%p.log
-version
```
stdout had the EPERM warning, the same output as in the issue. `grep -E
'pthread_create|EPERM|Operation not permitted|Cannot create worker GC thread'
hs_err_pid7.log` matched one line, `# Cannot create worker GC thread. Out of
system resources.`
The log fallback below can't catch the EPERM line either. `init-store.sh`
sends the JVM's stdout to the container log. `hugegraph-server.sh` sends it to
container stdout under `STDOUT_MODE=true`, which both Dockerfiles set, and to
`hugegraph-server-stdout.log` otherwise. Neither path writes
`hugegraph-server.log`. And this loop returns before the log is read anyway.
The `early42-eperm-file` and `early42-eperm-log` fixtures pass only because
they write the EPERM text into files a real JVM never puts it in.
`early42-resource` already has the real crash-file shape, and it asserts that
the seccomp hint is absent. To check this, I changed that fixture so the JVM
substitute prints the real warning to stdout and writes the real `hs_err`
header, then asserted the seccomp hint. The test failed with `ERROR: JVM could
not create a GC worker thread. Check available memory and process/thread
limits.`
Requested change: scan the failing JVM's own output. For example, tee
`init-store.sh` and the launcher output to a temp file and run
`print_jvm_thread_hint` on it. Or, if that is too invasive, make the `Cannot
create worker GC thread` hint name both causes (seccomp denial and memory or
limits) and tell users to look for `pthread_create failed (EPERM)` just above
it in the container log. Either way, base the fixtures on the real JVM output
(EPERM on stdout, generic text in `hs_err`), and update the README paragraph
that says the crash files and Server log are checked for EPERM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
