Re: TEST build OK: openjdk8.spec - openjdk 8 and 9 hangs on glibc 2.34 in vserver guest
W dniu 24.11.2021 o 21:30, Jan Palus pisze: > I suppose this is the commit to be blamed for this regression: > > https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=f13fb81ad3159543741e9132685335002a6d5df2 > > glibc 2.33 used to look at /proc too, but it no longer does in 2.34. Yes, f13fb81ad3159543741e9132685335002a6d5df2 is the first bad commit commit f13fb81ad3159543741e9132685335002a6d5df2 Author: Adhemerval Zanella Date: Thu Mar 25 14:04:37 2021 -0300 linux: Remove /proc/cpuinfo fallback on alpha and sparc There is no much gain in fallback to cpuinfo if sysfs is no present, usually on restricted environment neither will be present. It also simplifies the code and make all architecture use the sched_getaffinity as the sysfs fallback. Checked on sparc64-linux-gnu. sysdeps/unix/sysv/linux/alpha/getsysstats.c | 38 - sysdeps/unix/sysv/linux/getsysstats.c | 22 + sysdeps/unix/sysv/linux/sparc/getsysstats.c | 38 - 3 files changed, 1 insertion(+), 97 deletions(-) delete mode 100644 sysdeps/unix/sysv/linux/alpha/getsysstats.c delete mode 100644 sysdeps/unix/sysv/linux/sparc/getsysstats.c glibc people will reintroduce fallback (but based on /proc/stat). In mean time sysfs got mounted on builders and I also saw your changes to make multiprocessor default in openjdk 8 -- Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) ___ pld-devel-en mailing list pld-devel-en@lists.pld-linux.org http://lists.pld-linux.org/mailman/listinfo/pld-devel-en
Re: TEST build OK: openjdk8.spec - openjdk 8 and 9 hangs on glibc 2.34 in vserver guest
On 24.11.2021 21:04, Jan Palus wrote:
> > More fun with that.
> >
> > working system, glibc 2.34, on any kernel
> >
> > mkdir /test/
> > rsync -avPH / /test/ --exclude /test/ --exclude /proc --exclude /sys
> > mount /proc /test/proc -o bind
> > chroot /test/; java -version - hangs at some retry
> >
> > mkdir -p /test/sys/devices/system/cpu
> > chroot /test/; java -version - no hangs
> > (so proc is mounted but /sys is not; just dir exists)
>
> Great finding! Indeed the following is a quick reproducer for any java8
> version -- openjdk8 302/312, icedtea8 and even Oracle JDK8 202:
>
> systemd-run --wait -t -p InaccessiblePaths=/sys
> /usr/lib64/jvm/openjdk8/bin/java -version
>
> So far couldn't reproduce with openjdk11 and FWIW it does not reproduce
> on aarch64.
If I had to guess... Code that opens /sys/devices/system/cpu is found
in glibc:
https://sourceware.org/git?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/getsysstats.c;h=1391e360b8f8e86ce39a56d4bf1117f37d07eed9;hb=ae37d06c7d127817ba43850f0f898b793d42aea7#l95
Notice that result is different between case in which directory does not
exist (1) and exists but is empty (0). Now looking at java8 code there's
a juicy comment right next to getting number of CPUs:
// Most versions of linux have a bug where the number of processors are
// determined by looking at the /proc file system. In a chroot environment,
// the system call returns 1. This causes the VM to act as if it is
// a single processor and elide locking (see is_MP() call).
static bool unsafe_chroot_detected = false;
static const char *unstable_chroot_error = "/proc file system not found.\n"
"Java may be unstable running multithreaded in a
chroot "
"environment on Linux when /proc filesystem is not
mounted.";
void os::Linux::initialize_system_info() {
set_processor_count(sysconf(_SC_NPROCESSORS_CONF));
But... the code in glibc does not look at /proc... at least not in 2.34.
I suppose this is the commit to be blamed for this regression:
https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=f13fb81ad3159543741e9132685335002a6d5df2
glibc 2.33 used to look at /proc too, but it no longer does in 2.34.
Will check how openjdk11 handles that, but if any of this is true I'd say the
easiest fix would be to always lock and not optimize based on cpu count.
___
pld-devel-en mailing list
pld-devel-en@lists.pld-linux.org
http://lists.pld-linux.org/mailman/listinfo/pld-devel-en
Re: TEST build OK: openjdk8.spec - openjdk 8 and 9 hangs on glibc 2.34 in vserver guest
On 24.11.2021 19:55, Arkadiusz Miśkiewicz via pld-devel-en wrote: > W dniu 24.11.2021 o 14:00, Arkadiusz Miśkiewicz pisze: > > W dniu 22.10.2021 o 11:37, Jan Palus pisze: > >> On 22.10.2021 10:44, Jan Rękorajski wrote: > >>> Ah, it wasn't icedtea but openjdk8 that was installed on i686 and x86_64. > >>> I'm uninstalling it and blocking in poldek on builders so that we have > >>> only openjdk11 installable there. > >> > >> Currently openjdk8 requires either JDK 1.7 or 1.8 for build, I guess we > >> can patch it and try to build with openjdk11, no guarantees it would > >> work though. Not sure what is the difference but I had no issues > >> building openjdk8 1.8.0.312 with openjdk8 1.8.0.302 and glibc 2.34 on > >> x86_64, aarch64 and armv7hnl. I'll try to reproduce in fresh VM. > > > > For me our openjdk8 from ftp hangs on just doing "java -version". Not > > always. Sometimes I need to run that 5-10 times but I do get the hang > > easily. kernel 4.9.194, glibc 2.34, x86_64, vserver guest > > > > while (true); do date; java -version; done > > > > openjdk version "1.8.0_302-ga" > > OpenJDK Runtime Environment (build 1.8.0_302-ga-1) > > OpenJDK 64-Bit Server VM (build 25.302-b1, mixed mode) > > > > Build Date : pon, 2 sie 2021, 16:46:33 > > More fun with that. > > working system, glibc 2.34, on any kernel > > mkdir /test/ > rsync -avPH / /test/ --exclude /test/ --exclude /proc --exclude /sys > mount /proc /test/proc -o bind > chroot /test/; java -version - hangs at some retry > > mkdir -p /test/sys/devices/system/cpu > chroot /test/; java -version - no hangs > (so proc is mounted but /sys is not; just dir exists) Great finding! Indeed the following is a quick reproducer for any java8 version -- openjdk8 302/312, icedtea8 and even Oracle JDK8 202: systemd-run --wait -t -p InaccessiblePaths=/sys /usr/lib64/jvm/openjdk8/bin/java -version So far couldn't reproduce with openjdk11 and FWIW it does not reproduce on aarch64. ___ pld-devel-en mailing list pld-devel-en@lists.pld-linux.org http://lists.pld-linux.org/mailman/listinfo/pld-devel-en
Re: TEST build OK: openjdk8.spec - openjdk 8 and 9 hangs on glibc 2.34 in vserver guest
W dniu 24.11.2021 o 14:00, Arkadiusz Miśkiewicz pisze: > W dniu 22.10.2021 o 11:37, Jan Palus pisze: >> On 22.10.2021 10:44, Jan Rękorajski wrote: >>> Ah, it wasn't icedtea but openjdk8 that was installed on i686 and x86_64. >>> I'm uninstalling it and blocking in poldek on builders so that we have >>> only openjdk11 installable there. >> >> Currently openjdk8 requires either JDK 1.7 or 1.8 for build, I guess we >> can patch it and try to build with openjdk11, no guarantees it would >> work though. Not sure what is the difference but I had no issues >> building openjdk8 1.8.0.312 with openjdk8 1.8.0.302 and glibc 2.34 on >> x86_64, aarch64 and armv7hnl. I'll try to reproduce in fresh VM. > > For me our openjdk8 from ftp hangs on just doing "java -version". Not > always. Sometimes I need to run that 5-10 times but I do get the hang > easily. kernel 4.9.194, glibc 2.34, x86_64, vserver guest > > while (true); do date; java -version; done > > openjdk version "1.8.0_302-ga" > OpenJDK Runtime Environment (build 1.8.0_302-ga-1) > OpenJDK 64-Bit Server VM (build 25.302-b1, mixed mode) > > Build Date : pon, 2 sie 2021, 16:46:33 More fun with that. working system, glibc 2.34, on any kernel mkdir /test/ rsync -avPH / /test/ --exclude /test/ --exclude /proc --exclude /sys mount /proc /test/proc -o bind chroot /test/; java -version - hangs at some retry mkdir -p /test/sys/devices/system/cpu chroot /test/; java -version - no hangs (so proc is mounted but /sys is not; just dir exists) -- Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) ___ pld-devel-en mailing list pld-devel-en@lists.pld-linux.org http://lists.pld-linux.org/mailman/listinfo/pld-devel-en
Re: TEST build OK: openjdk8.spec - openjdk 8 and 9 hangs on glibc 2.34 in vserver guest
W dniu 22.10.2021 o 11:37, Jan Palus pisze: > On 22.10.2021 10:44, Jan Rękorajski wrote: >> Ah, it wasn't icedtea but openjdk8 that was installed on i686 and x86_64. >> I'm uninstalling it and blocking in poldek on builders so that we have >> only openjdk11 installable there. > > Currently openjdk8 requires either JDK 1.7 or 1.8 for build, I guess we > can patch it and try to build with openjdk11, no guarantees it would > work though. Not sure what is the difference but I had no issues > building openjdk8 1.8.0.312 with openjdk8 1.8.0.302 and glibc 2.34 on > x86_64, aarch64 and armv7hnl. I'll try to reproduce in fresh VM. For me our openjdk8 from ftp hangs on just doing "java -version". Not always. Sometimes I need to run that 5-10 times but I do get the hang easily. kernel 4.9.194, glibc 2.34, x86_64, vserver guest while (true); do date; java -version; done openjdk version "1.8.0_302-ga" OpenJDK Runtime Environment (build 1.8.0_302-ga-1) OpenJDK 64-Bit Server VM (build 25.302-b1, mixed mode) Build Date : pon, 2 sie 2021, 16:46:33 Doing the same on kernel 5.15.2 (VM on proxmox) and -version works. 4.4.279 on vserver host, glibc 2.34 - works. 4.4.279 butin vserver guest, glibc 2.34 hangs. 4.4.279 but glibc 2.33 in vserver guest - works. openjdk9-9.0.4.12-1.x86_64 also hangs in vserver guest with glibc 2.34 openjdk10-10.0.2.13-1.x86_64 works openjdk11-11.0.13-1.x86_64 works So vserver guest + glibc 2.34 + openjdk 8 and 9 is hanging on futexes. Easy way of testing if you don't have glibc 2.34 installed: - install openjdk8 from ftp (--nofollow --nodeps can be useful to prevent in dragging latest nss and thus glibc 2.3 - only working "java" binary is needed) - copy 2.34 *.so libs to some path - patchelf --set-interpreter /path/to/glibc-2.34-test-libs/ld-linux-x86-64.so.2 /usr/lib64/jvm/openjdk8-1.8.0.302/bin/java - run: while (true); do date; LD_LIBRARY_PATH=/path/to/glibc-2.34-test-libs/ java -version; done ps. builders stopped using vservers on 7 november 2021 and use 5.x kernels -- Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) ___ pld-devel-en mailing list pld-devel-en@lists.pld-linux.org http://lists.pld-linux.org/mailman/listinfo/pld-devel-en
