[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
Also I assume the bug is fixed in Ubuntu 20.04 and later as well. Could you please confirm that? ** Changed in: glibc (Ubuntu) Status: Confirmed => Incomplete ** Changed in: glibc (Ubuntu Bionic) Status: New => Incomplete ** Changed in: glibc (Ubuntu Focal) Status: New => Incomplete ** Changed in: openvswitch (Ubuntu Bionic) Status: New => Incomplete ** Changed in: openvswitch (Ubuntu Bionic) Status: Incomplete => Invalid ** Changed in: openvswitch (Ubuntu Focal) Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
2.27-3ubuntu1.3 has been released to bionic-updates. @mnaser do I understand correctly that this bug is fully fixed by it? ** Also affects: glibc (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: openvswitch (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: glibc (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: openvswitch (Ubuntu Bionic) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
For context - 2.27-3ubuntu1.3 is in bionic-proposed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
Thanks for the research @mnaser - interesting that this is only seen on much more powerful machines which I think has been the key blocker to reproduction of this issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
So, it turns out that 2.27-3ubuntu1.3 could fix this because of the fact that not only does it include that patch, but a full rebase to 2.27 branch of upstream glibc, which does include the other fix mentioned above (I found this out when I tried to rebuild locally and found out that quilt was complaining of an already built patch). Upon running the code above on the 2.27-3ubuntu1.2 system, it freezes but works fine under 2.27-3ubuntu1.3. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
I have done some extensive research on this and I am able to confirm that while Bug #1864864 does solve a class of issues, it does not solve the one affecting Open vSwitch, as it does rely on PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP: https://github.com/openvswitch/ovs/blob/0de1b425962db073ebbaa3ddbde445580afda840/lib /ovs-thread.c#L247-L248 And when running the reproducer here: https://sourceware.org/bugzilla/show_bug.cgi?id=23861 By doing the following: ``` curl https://sourceware.org/bugzilla/attachment.cgi?id=11382 -o bug23861.c sed -i 's/do_exit = 0/do_exit(0)/' bug23861.c g++ bug23861.c -lpthread -o bug23861 for ((x=1;x<100;x++)) ; do echo $x;date;./bug23861 --prefer-writer-nonrecursive;done ``` I will add that in my initial tests of doing this inside a small VM with a Docker container, it did not hang after running for 10-15 minutes, but it immediately hung on a much more powerful machine. I will file a bug to try and get the patch above SRU'd into glibc as well and hopefully that will be the end of it. ** Bug watch added: Sourceware.org Bugzilla #23861 https://sourceware.org/bugzilla/show_bug.cgi?id=23861 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
Likely it is not a duplicate, see my comment in #10 and in the bugzilla. So if branch-pthread_rwlock_trywrlock-hang-23844.patch is the patch from the bugzilla, I tested that earlier and it did not solve the issue. The issue here seems to be with pthread_rwlock_rdlock hanging, see the tracebacks in the ovs github issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
is this duplicate of https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1864864 ? does new glibc from bionic-proposed https://launchpad.net/ubuntu/+source/glibc/2.27-3ubuntu1.3 help? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
@salmankh you can solve this by installing this packages from eoan ``` libc6 libc6-dbg libcap-ng0 libc-bin libidn2-0 locales openvswitch-common openvswitch-switch ``` and don't forget to run `systemctl daemon-reexec` afterwards! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
And obviously that restarting of ovs and neutron services disrupt the network traffic massively. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
Is there glic upgrade coming on 18.04? We are hit by this problem very frequently and the only way out is to restart the ovs and neutron services. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
we are also affected by this issue and would appreciate if glibc would be updated to 2.29-0ubuntu2 in bionic repositories. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
I would also like to share our experiences: We have had problems with this bug for a hole year. "ovs is dead..." First we suspected a bug in openvswitch. But, this issue made us aware of glibc: https://github.com/openvswitch/ovs-issues/issues/175 We updated glibc to version 2.29-0ubuntu2 from the disco repository. Now, for more than a month we don't have any problems on 6 compute nodes. It would be nice if the glibc package updates can also be make available in the bionic repositories. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
** Changed in: glibc (Ubuntu) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
So as documented in https://github.com/openvswitch/ovs-issues/issues/175 , this really seems to be a bug in glibc and not in openswitch. After installing libc6 and related pkgs at version 2.29-0ubuntu2 (from disco), the issue is resolved for us. The single patch from https://sourceware.org/bugzilla/show_bug.cgi?id=23844 did not fix the issue. So we would either need to make an effort to get the version from disco into bionic-updates or someone would need to locate a set of possible patches and backport them. ** Changed in: openvswitch (Ubuntu) Status: Confirmed => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
Here a similar portion from the ovs log and gdb trace running openvswitch 2.11.0-0ubuntu2: Thu Oct 31 18:20:23 2019-2019-10-31T17:20:23.521Z|1|ovs_rcu(urcu4)|WARN|blocked 1000 ms waiting for revalidator124 to quiesce Thu Oct 31 18:20:24 2019-2019-10-31T17:20:24.521Z|2|ovs_rcu(urcu4)|WARN|blocked 2000 ms waiting for revalidator124 to quiesce Thu Oct 31 18:20:59 2019-2019-10-31T17:20:26.520Z|3|ovs_rcu(urcu4)|WARN|blocked 4000 ms waiting for revalidator124 to quiesce In the trace: 29 Thread 0x7f72f97fa700 (LWP 26608) "revalidator124" 0x7f734aee237b in futex_abstimed_wait (private=, abstime=0x0, expected=10, futex_word=0x5577bab397c0 ) at ../sysdeps/unix/sysv/linux/futex-internal.h:172 Full trace attached in gdbwrap.1572542426.log.gz The traces are a bit hocus to me, I really don't have a clue whats going on there but I guess it might help you make sense of whats going on here. ** Attachment added: "GDB trace ovs 2.11.0 during when it hangs" https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1839592/+attachment/5308469/+files/gdbwrap.1572542426.log.gz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
** Attachment added: "GDB trace ovs 2.9.2 during when it hangs" https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1839592/+attachment/5308470/+files/gdbwrap.1566706577.log.gz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
I just came across this bug report and would like to share my expierence. I've been having similar issues on 6 servers since we upgraded from 16.04 to 18.04 about 2 years ago with openvswitch. Our biggest problem is our inabilty to reproduce it. We just see Openvswitch hanging from time to time. Sometimes it takes a day to get stuck, sometimes it takes months. The only way to recover from it is to restart openvswitch. Right now we are running with a backport of openvswitch from Disco (2.11.0-0ubuntu2) in Bionic. With that version backported we are having the same issues as with the previously installed 2.9.2-0ubuntu0.18.04.3 version that Bionic has. I have gbd traces from both versions which I will attach. Here a small portion from the ovs log and gdb trace of openvswitch 2.9.2-0ubuntu0.18.04.3: Sun Aug 25 06:16:14 2019-2019-08-25T04:16:14.943Z|1|ovs_rcu(urcu4)|WARN|blocked 1000 ms waiting for revalidator127 to quiesce Sun Aug 25 06:16:15 2019-2019-08-25T04:16:15.943Z|2|ovs_rcu(urcu4)|WARN|blocked 2000 ms waiting for revalidator127 to quiesce Sun Aug 25 06:16:50 2019-2019-08-25T04:16:17.943Z|3|ovs_rcu(urcu4)|WARN|blocked 4001 ms waiting for revalidator127 to quiesce Small portion of the trace: 32 Thread 0x7f1bfa7fc700 (LWP 1461) "revalidator127" 0x7f1c61aeb37b in futex_abstimed_wait (private=, abstime=0x0, expected=10, futex_word=0x55e4ed0aa800 ) at ../sysdeps/unix/sysv/linux/futex-internal.h:172 The full trace is attached in gdbwrap.1566706577.log.gz (Openvswitch 2.9.2) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: glibc (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
bug ref for 2.9.5 SRU - bug 1854360 ** Also affects: glibc (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
2.9.5 test packages in: https://launchpad.net/~ci-train-ppa- service/+archive/ubuntu/3534/+packages -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
>From the discussion in https://github.com/openvswitch/ovs- issues/issues/175 it seems like this may actually be a glibc bug https://sourceware.org/bugzilla/show_bug.cgi?id=23844 , I'll try to get some tests done in order to confirm this. ** Bug watch added: github.com/openvswitch/ovs-issues/issues #175 https://github.com/openvswitch/ovs-issues/issues/175 ** Bug watch added: Sourceware.org Bugzilla #23844 https://sourceware.org/bugzilla/show_bug.cgi?id=23844 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
https://github.com/openvswitch/ovs/commit/2ed1c95e9c80a21a5d81cb760c872b62b32b0733 This seems to be commit for this. and 2.9.3 has it. @Camilo, Do you have reproducer for this issue? ** Tags added: sts -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1839592] Re: Open vSwitch (Version 2.9.2) goes into deadlocked state
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: openvswitch (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1839592 Title: Open vSwitch (Version 2.9.2) goes into deadlocked state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1839592/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
