[Bug 1844195] Re: beegfs-meta lockup with glibc 2.27 on bionic
Any movement? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1844195 Title: beegfs-meta lockup with glibc 2.27 on bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1844195] Re: beegfs-meta lockup with glibc 2.27 on bionic
This bug is affecting our systems as well. I just wanted to note that the torel's workaround of disabling updatedb indexing of beegfs metadata may work for specific uses of beegfs, but it would not fix the problem for us. Our users regularly generate large number of files in cluster jobs, causing the beegfs-meta service to fail. I hope a real fix is available soon, since this critical bug is nearly a year old. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1844195 Title: beegfs-meta lockup with glibc 2.27 on bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1844195] Re: beegfs-meta lockup with glibc 2.27 on bionic
WorkARound is adding beegfs to PRUNEFS and mountpoint to PRUNEPATHS in /etc/updatedb.conf -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1844195 Title: beegfs-meta lockup with glibc 2.27 on bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1844195] Re: beegfs-meta lockup with glibc 2.27 on bionic
Debug of this mornings lockup. ** Attachment added: "beegfs-meta-gdb.log" https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+attachment/5393597/+files/beegfs-meta-gdb.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1844195 Title: beegfs-meta lockup with glibc 2.27 on bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1844195] Re: beegfs-meta lockup with glibc 2.27 on bionic
** Attachment added: "beegfs-meta-gdb2.log" https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+attachment/5393090/+files/beegfs-meta-gdb2.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1844195 Title: beegfs-meta lockup with glibc 2.27 on bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1844195] Re: beegfs-meta lockup with glibc 2.27 on bionic
gdb of locked beegfs-meta. Will have to redo. Forgot stdout. ** Attachment added: "beegfs-meta-gdb.log" https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+attachment/5393089/+files/beegfs-meta-gdb.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1844195 Title: beegfs-meta lockup with glibc 2.27 on bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1844195] Re: beegfs-meta lockup with glibc 2.27 on bionic
Sounds like something possibly was checked into glibc (libc6 on ubuntu) between libc6:amd64 2.27-3ubuntu1 amd64 GNU C Library: Shared libraries and libc6:amd64 2.27-3ubuntu1.2 amd64 Which check-in is the culprit in https://launchpad.net/ubuntu/+source/glibc/2.27-3ubuntu1.2 ? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1844195 Title: beegfs-meta lockup with glibc 2.27 on bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1844195] Re: beegfs-meta lockup with glibc 2.27 on bionic
Been seeing the same, but it has escalated to the point that it occurs every morning at 6:30-6:40am. root@srl-mds1:~# uname -ar Linux srl-mds1 4.15.0-109-generic #110-Ubuntu SMP Tue Jun 23 02:39:32 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux root@srl-mds1:~# dpkg -l libc6 Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version ArchitectureDescription +++--===-===-= ii libc6:amd64 2.27-3ubuntu1.2 amd64 GNU C Library: Shared libraries root@srl-mds1:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description:Ubuntu 18.04.4 LTS Release:18.04 Codename: bionic root@srl-mds1:~# dpkg -l libc6 Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version ArchitectureDescription +++--===-===-= ii libc6:amd64 2.27-3ubuntu1.2 amd64 GNU C Library: Shared libraries root@srl-mds1:~# dpkg -l beegfs-common Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version ArchitectureDescription +++--===-===-= ii beegfs-common20:7.1.5amd64 BeeGFS common files root@srl-mds1:~# dpkg -S /opt/beegfs/sbin/beegfs-meta beegfs-meta: /opt/beegfs/sbin/beegfs-meta root@srl-mds1:~# dpkg -l beegfs-meta Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version ArchitectureDescription +++--===-===-= ii beegfs-meta 20:7.1.5amd64 BeeGFS metadata server daemon root@srl-mds1:~# Debug trace will follow tomorrow morning. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1844195 Title: beegfs-meta lockup with glibc 2.27 on bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1844195] Re: beegfs-meta lockup with glibc 2.27 on bionic
Just a followup as this issue was reported about a month ago, I wonder if there is anyone from the GLIBC team who could have a look at this code? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1844195 Title: beegfs-meta lockup with glibc 2.27 on bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1844195] Re: beegfs-meta lockup with glibc 2.27 on bionic
Really curious if this turns out to be a glibc problem or a beegfs-meta problem, but we were able to hack together something that circumvents this problem by forcing beegfs-meta to use an older glibc6 (from xenial). beegfs-meta is now stable and performing much better! There's probably a simpler way to do this, but we did something along these lines: 1. debootstrap xenial /srv/xenial-chroot 2. chroot into xenial-chroot, add beegfs to apt sources.list, apt install beegfs-meta, exit chroot 3. prepare a systemd pre-exec script "/usr/local/bin/setup-beegfs-meta-chroot.sh": #!/bin/bash set -e cp /etc/beegfs/beegfs-meta.conf /srv/xenial-chroot/etc/beegfs/beegfs-meta.conf mountpoint -q /srv/xenial-chroot/proc || mount --bind /proc /srv/xenial-chroot/proc mountpoint -q /srv/xenial-chroot/sys || mount --bind /sys /srv/xenial-chroot/sys mountpoint -q /srv/xenial-chroot/path/to/metadata || mount --bind /path/to/metadata /srv/xenial-chroot/path/to/metadata 4. copy /lib/systemd/system/beegfs-meta.service to /etc/systemd/system /beegfs-meta.service, adding the following to the [service] section: RootDirectory=/srv/xenial-chroot ExecStartPre=/usr/local/bin/setup-beegfs-meta-chroot.sh RootDirectoryStartOnly=yes 5. daemon-reload systemd and restart beeegfs-meta -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1844195 Title: beegfs-meta lockup with glibc 2.27 on bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1844195] Re: beegfs-meta lockup with glibc 2.27 on bionic
I also experience the issue but not at specific times but on medium/high load situations. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1844195 Title: beegfs-meta lockup with glibc 2.27 on bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1844195] Re: beegfs-meta lockup with glibc 2.27 on bionic
beegfs-meta went into high CPU lock-up again today, at 6:27:07 am, even with daily updates disabled. So apt-daily-upgrade is not directly related to the hangs. It turns out though that there is other stuff scheduled around that time by daily cron: 25 6* * * roottest -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) It is still not clear what triggers the hangs, but it likely is related to a scheduled service, because of the consistency with which this happens at around 6:30am. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1844195 Title: beegfs-meta lockup with glibc 2.27 on bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1844195] Re: beegfs-meta lockup with glibc 2.27 on bionic
Seeing the same issue on the same configuration, but occurs every 24h, to within a few minutes or seconds after the following systemd timer goes off. Sat 2019-10-05 06:41:41 EDT 21h leftFri 2019-10-04 06:33:12 EDT 2h 59min ago apt-daily-upgrade.timer apt-daily-upgrade.service The update started at 06:33:05 (tzdata was updated), the metadata server stopped responding 06:35:55. Similar behavior on 3 different days. Will switch off the updates to see if that fixes the problem -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1844195 Title: beegfs-meta lockup with glibc 2.27 on bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1844195] Re: beegfs-meta lockup with glibc 2.27 on bionic
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: glibc (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1844195 Title: beegfs-meta lockup with glibc 2.27 on bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1844195] Re: beegfs-meta lockup with glibc 2.27 on bionic
Hi, we also have the same issue. Release: Ubuntu 18.04.3 bionic Kernel: 4.15.0-62-generic libc6: 2.27-3ubuntu1 beegfs: 7.1.3 Not ideal, of course, but going to try and run beegfs-meta in a chrooted xenial (using the xenial glibc version). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1844195 Title: beegfs-meta lockup with glibc 2.27 on bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1844195] Re: beegfs-meta lockup with glibc 2.27 on bionic
Hi, we have the same issue. I can confirm that after attaching gdb to the beegfs-meta process, it resumes normal operation. On our system it seems the issue is recuring more or less around 24h. Release: Ubuntu 18.04.1 bionic Kernel: 4.15.0-39-generic libc6: 2.27-3ubuntu1 beegfs: 7.1.1 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1844195 Title: beegfs-meta lockup with glibc 2.27 on bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1844195] Re: beegfs-meta lockup with glibc 2.27 on bionic
Thanks for taking the time to report this bug and helping to make Ubuntu better. We appreciate the difficulties you are facing, but this appears to be a "regular" (non-security) bug. I have unmarked it as a security issue since this bug does not show evidence of allowing attackers to cross privilege boundaries nor directly cause loss of data/privacy. Please feel free to report any other bugs you may find. ** Information type changed from Public Security to Public -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1844195 Title: beegfs-meta lockup with glibc 2.27 on bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1844195] Re: beegfs-meta lockup with glibc 2.27 on bionic
** Information type changed from Private Security to Public Security -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1844195 Title: beegfs-meta lockup with glibc 2.27 on bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1844195/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
