Your message dated Fri, 26 Jan 2024 20:43:40 +0000
with message-id <[email protected]>
and subject line Bug#1034600: fixed in openjdk-11 11.0.22+7-2
has caused the Debian Bug report #1034600,
regarding tomcat9: jstack/jcmd broken for non-root users with tomcat9+jdk11 or
greater
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact [email protected]
immediately.)
--
1034600: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1034600
Debian Bug Tracking System
Contact [email protected] with problems
--- Begin Message ---
Package: tomcat9
Version: 9.0.43-2~deb11u6
Severity: normal
X-Debbugs-Cc: [email protected]
Hi,
We noticed while rolling out JDK 17 support for our in-house application
that the following command is "broken" (moral-martin is an LXD container
in my examples below, PID 4108 is the tomcat9 java process):
root@moral-martin:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 11 (bullseye)
Release: 11
Codename: bullseye
root@moral-martin:~# sudo -u tomcat jstack 4108
4108: Unable to open socket file /proc/4108/root/tmp/.java_pid4108: target
process 4108 doesn't respond within 10500ms or HotSpot VM not loaded
...when all following conditions are met:
* tomcat9 is running from systemd, _and_
* the JDK is of version 11 or greater, _and_
* the systemd unit (/lib/systemd/system/tomcat9.service) sets
AmbientCapabilities=CAP_NET_BIND_SERVICE (which is done by the Debian
package)
We have spent a significant amount of time debugging this and I'll try
to do my best to summarize our findings here:
The problem is that the way jstack and similar tools work have changed
from JDK8 to JDK11. In JDK8, it simply uses /tmp to try and communicate
with the target process:
https://github.com/AdoptOpenJDK/openjdk-jdk8u/blob/master/jdk/src/solaris/classes/sun/tools/attach/LinuxVirtualMachine.java#L40-L45
and
https://github.com/AdoptOpenJDK/openjdk-jdk8u/blob/master/jdk/src/solaris/classes/sun/tools/attach/LinuxVirtualMachine.java#L293
In newer JDK versions (JDK 17 as an example), the code has been made
"smarter" to support mount namespaces:
https://github.com/openjdk/jdk17u/blob/master/src/jdk.attach/linux/classes/sun/tools/attach/VirtualMachineImpl.java#L299-L302
_However_... bear with me, this is where it gets interesting: this
presumes that the calling process can access /proc/<pid>/root/tmp. When
AmbientCapabilities=CAP_NET_BIND_SERVICE is set in the systemd unit,
this is not the case:
root@moral-martin:~# sudo -u tomcat ls -l /proc/4108/root
ls: cannot read symbolic link '/proc/4108/root': Permission denied
lrwxrwxrwx 1 tomcat tomcat 0 Apr 13 12:55 /proc/4108/root
We have tested this and concluded that:
1. This happens whever _any_ capability is set in the systemd unit; it's
not limited to CAP_NET_BIND_SERVICE. (Note: I haven't tested adding
all possible capabilities yet; I believed I had but when writing this
bug report I realize that my attempt at setting all of them didn't
actually list all of them in `getpcaps pid`; will test this a bit
more and see if it makes any difference)
2. When you remove AmbientCapabilities or set it to AmbientCapabilities=
(empty string), it also works correctly.
I honestly don't know if tomcat9 is the correct package to report this
to; it can also be seen as a bug in the JDK. (We will work with the JDK
maintainers to get this reported to them as well.) Feel free to reassign
the bug report to another package.
With JDK 8, this works correctly. Some of our tooling/monitoring is
dependent on being able to connect to Tomcat (running on JDK 8 or 17) at
runtime. That's why this is imporant for us.
Workaround
What we have seen that on JDK 17, running `jstack` as root works; this
will connect to the target process correctly. However, this does _not_
work on JDK 8 and doesn't seem to work properly on JDK 11 either (I
think this has been fixed upstream in JDK for more recent JDK versions,
which is why it behaves differently on JDK 17).
Our application supports both JDK 8 and 17 for now, and running `jstack`
as root *does not* work on JDK 8. Hence, having to run it as root with
our JDK 17-based installations (only) makes things unnecessarily
complex.
Conclusions
It puzzles me why setting the ambient capabilities for a process breaks
this. It's uncertain whether this is a "feature" by the kernel or
elsewhere. We have tried to find more details about this by studying the
systemd and dbus code to a certain extent, but have yet been unable to
find anything. If anyone reading this knows the prctl and cap_set_proc
semantics by heart, your help would be greatly appreciated.
Best regards,
Per
-- System Information:
Debian Release: bookworm/sid
APT prefers testing-security
APT policy: (500, 'testing-security'), (500, 'testing')
Architecture: amd64 (x86_64)
Kernel: Linux 6.1.0-6-amd64 (SMP w/20 CPU threads; PREEMPT)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE,
TAINT_UNSIGNED_MODULE
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8),
LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages tomcat9 depends on:
ii lsb-base 11.6
ii systemd [systemd-tmpfiles] 252.6-1
ii sysvinit-utils [lsb-base] 3.06-2
pn tomcat9-common <none>
ii ucf 3.0043+nmu1
Versions of packages tomcat9 recommends:
ii libtcnative-1 1.2.35-1
Versions of packages tomcat9 suggests:
pn tomcat9-admin <none>
pn tomcat9-docs <none>
pn tomcat9-examples <none>
pn tomcat9-user <none>
--- End Message ---
--- Begin Message ---
Source: openjdk-11
Source-Version: 11.0.22+7-2
Done: Matthias Klose <[email protected]>
We believe that the bug you reported is fixed in the latest version of
openjdk-11, which is due to be installed in the Debian FTP archive.
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to [email protected],
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Matthias Klose <[email protected]> (supplier of updated openjdk-11 package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing [email protected])
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Format: 1.8
Date: Fri, 26 Jan 2024 21:04:10 +0100
Source: openjdk-11
Architecture: source
Version: 11.0.22+7-2
Distribution: unstable
Urgency: medium
Maintainer: OpenJDK Team <[email protected]>
Changed-By: Matthias Klose <[email protected]>
Closes: 1034600 1060821
Changes:
openjdk-11 (11.0.22+7-2) unstable; urgency=medium
.
[ Matthias Klose ]
* Add zero support for loong64 (Leslie Zhai). Closes: #1060821.
* d/changelog: Whitespace cleanup.
* Update build dependency on libfontconfig-dev.
* Apply proposed patch for JDK-8307977. Closes: #1034600.
.
[ Vladimir Petko ]
* d/t/jtreg-autopkgtest.*: Set jtreg home property correctly.
Checksums-Sha1:
a8d0cf7cffd2a03d372be8a5067686279878b52a 4652 openjdk-11_11.0.22+7-2.dsc
12321b5f795cbccb6b71898d2d0517b6be244966 69141956
openjdk-11_11.0.22+7.orig.tar.xz
e98fedd8c62fd5770b9caf1916fcc1b021331094 170616
openjdk-11_11.0.22+7-2.debian.tar.xz
69a8f068d634256888263ce7a047231bee70a29d 14908
openjdk-11_11.0.22+7-2_source.buildinfo
Checksums-Sha256:
f40e310ff90cfa576409546c94efec733790369df817261f92d3077b4e177045 4652
openjdk-11_11.0.22+7-2.dsc
c08bca2c534214460f5853fd61d435916f7c61390ab1ffaea099bab503cadea4 69141956
openjdk-11_11.0.22+7.orig.tar.xz
ef064ac0c8cc743e78235cacb863989c13524b9e797be02e789e0b8ef4bb9028 170616
openjdk-11_11.0.22+7-2.debian.tar.xz
cc998c114b56ff615cb5dc361ff841dfd81fe9d962cff3d275836323545f9947 14908
openjdk-11_11.0.22+7-2_source.buildinfo
Files:
20a7d688bc95386e2cf043f5b4650688 4652 java optional openjdk-11_11.0.22+7-2.dsc
1e41b35e5e26b875c31cce3b1369cb60 69141956 java optional
openjdk-11_11.0.22+7.orig.tar.xz
f4d786eb4fd058f0e8d59304cb0fd92f 170616 java optional
openjdk-11_11.0.22+7-2.debian.tar.xz
e27795b0414b6a3a65cd8bcb3e70fa04 14908 java optional
openjdk-11_11.0.22+7-2_source.buildinfo
-----BEGIN PGP SIGNATURE-----
iQJEBAEBCgAuFiEE1WVxuIqLuvFAv2PWvX6qYHePpvUFAmW0EYYQHGRva29AdWJ1
bnR1LmNvbQAKCRC9fqpgd4+m9XvlEADcMauaLbF4X4Naqnq7sj3ZJxqtBPIU/cdj
9TGQnJxP4jQG+1/UF2hv7Io4R9UmhjMB76FzX3CnRdpVI/h4lCK7qPbU0/x5dxiA
rNXbTSEOSGnWD1m0sm4v5kzVC5Q/RPr6Y8C5hwGI6/mgjW6wMyeyFzEVBhon1vTT
kcM9kQKSTDXL7VyFjJc3pTKOk+YDIQyw29feyR1dUtyz5/galiW65g0YqD2IJOCr
PSIUxbsJNnEkam0KE7t8G+2SD6NB+ogSshWnlzrc/UXHUlSofAHdmGwEZmYrpLms
AKlzwrO94+LRztDt+XSDwQQms55XHkBg3kZXcdYFV9IcU21hKSP48op44UAytfRX
aOa/J9lb2dAxghIq5Ysq5g3gokEsnCI6RvfqGBbh8uiuKBH9REjh/RW09GWqZM84
jK05IfKMuf2jOYSudIMpjYAiP6+jr/irqiATL7BNQiJ6r6ADDranh3wLmwXeDNxk
oGDAEup0NH1omCPoC3O7WBnIeDqoKhitaskALk8Vq7rC+Okg2m1UbYEkE1HftiXn
6CB50tyIvq/M+s4CDhKg8yahV2pgD5Ad5vv4AUbaT+eqAbRwc0D1IXq4FS4fao7L
KfhdAgDzQLatg4umg1wVqrshiMoclBPlaYUm20FJpgFocTJIJLSnWNju3WAdxaI5
M3KWF4OeUQ==
=aWlE
-----END PGP SIGNATURE-----
--- End Message ---
