[Bug 1921562] Re: Intermittent hangs during ldap_search_ext when TLS enabled
This bug was fixed in the package openldap - 2.4.49+dfsg-2ubuntu1.8 --- openldap (2.4.49+dfsg-2ubuntu1.8) focal; urgency=medium * d/p/ITS-8650-loop-on-incomplete-TLS-handshake.patch: Import upstream patch to properly retry gnutls_handshake() after it returns GNUTLS_E_AGAIN. (ITS#8650) (LP: #1921562) -- Utkarsh Gupta Thu, 08 Apr 2021 09:52:01 +0530 ** Changed in: openldap (Ubuntu Focal) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1921562 Title: Intermittent hangs during ldap_search_ext when TLS enabled To manage notifications about this bug go to: https://bugs.launchpad.net/openldap/+bug/1921562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1921562] Re: Intermittent hangs during ldap_search_ext when TLS enabled
** Tags removed: verification-needed ** Tags added: verification-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1921562 Title: Intermittent hangs during ldap_search_ext when TLS enabled To manage notifications about this bug go to: https://bugs.launchpad.net/openldap/+bug/1921562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1921562] Re: Intermittent hangs during ldap_search_ext when TLS enabled
I've been running 2.4.49+dfsg-2ubuntu1.8 from focal-proposed for the past few days and the issue has not returned. As otherwise the issue would occur at least once per day, I consider it fixed. Furthermore, no other issues have cropped up in the meantime. ** Tags removed: verification-needed-focal ** Tags added: verification-done-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1921562 Title: Intermittent hangs during ldap_search_ext when TLS enabled To manage notifications about this bug go to: https://bugs.launchpad.net/openldap/+bug/1921562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1921562] Re: Intermittent hangs during ldap_search_ext when TLS enabled
Hello Vincent, or anyone else affected, Accepted openldap into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openldap/2.4.49+dfsg- 2ubuntu1.8 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-focal. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: openldap (Ubuntu Focal) Status: In Progress => Fix Committed ** Tags added: verification-needed verification-needed-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1921562 Title: Intermittent hangs during ldap_search_ext when TLS enabled To manage notifications about this bug go to: https://bugs.launchpad.net/openldap/+bug/1921562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1921562] Re: Intermittent hangs during ldap_search_ext when TLS enabled
Hi Vincent, > The bug hasn't returned since I installed the fixed package and no > new issues have cropped up. Awesome, thank you for your confirmation! \o/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1921562 Title: Intermittent hangs during ldap_search_ext when TLS enabled To manage notifications about this bug go to: https://bugs.launchpad.net/openldap/+bug/1921562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1921562] Re: Intermittent hangs during ldap_search_ext when TLS enabled
The bug hasn't returned since I installed the fixed package and no new issues have cropped up. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1921562 Title: Intermittent hangs during ldap_search_ext when TLS enabled To manage notifications about this bug go to: https://bugs.launchpad.net/openldap/+bug/1921562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1921562] Re: Intermittent hangs during ldap_search_ext when TLS enabled
I've deployed the patch, I'll let you know whether it works and if any regressions occur. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1921562 Title: Intermittent hangs during ldap_search_ext when TLS enabled To manage notifications about this bug go to: https://bugs.launchpad.net/openldap/+bug/1921562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1921562] Re: Intermittent hangs during ldap_search_ext when TLS enabled
To be clear the fix that was mentioned is in 2.4.49+dfsg-4 and later, 442 openldap (2.4.49+dfsg-4) unstable; urgency=medium ... 453 * Import upstream patch to properly retry gnutls_handshake() after it 454 returns GNUTLS_E_AGAIN. (ITS#8650) (Closes: #861838) Thereby groovy is fixed as well, marking that in the bug tasks. ** Also affects: openldap (Ubuntu Groovy) Importance: Undecided Status: New ** Changed in: openldap (Ubuntu Groovy) Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1921562 Title: Intermittent hangs during ldap_search_ext when TLS enabled To manage notifications about this bug go to: https://bugs.launchpad.net/openldap/+bug/1921562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1921562] Re: Intermittent hangs during ldap_search_ext when TLS enabled
** Merge proposal linked: https://code.launchpad.net/~utkarsh/ubuntu/+source/openldap/+git/openldap/+merge/400754 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1921562 Title: Intermittent hangs during ldap_search_ext when TLS enabled To manage notifications about this bug go to: https://bugs.launchpad.net/openldap/+bug/1921562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1921562] Re: Intermittent hangs during ldap_search_ext when TLS enabled
** Description changed: + [Impact] + + When connecting to an LDAP server with TLS, ldap_search_ext can hang if during the initial TLS handshake a signal is received by the process. The cause of this bug is the same as - https://bugs.openldap.org/show_bug.cgi?id=8650 which was fixed in - https://git.openldap.org/openldap/openldap/-/commit/735e1ab and was - released as part of version 2.4.50. This bug effects Ubuntu 20.04 LTS - and potentially earlier Ubuntu releases. Later Ubuntu releases use an - openldap version that is at least 2.4.50 and are therefore not affected. + https://bugs.openldap.org/show_bug.cgi?id=8650. In our case this bug cause failures in the SSSD LDAP backend at least once per day, resulting in authentication errors followed by a sssd_be + restart after a timeout has been hit. + + + [Test Plan] + === + + When using openldap on 20.04, this bug causes failures in the SSSD LDAP + backend, resulting in authentication errors followed by a sssd_be restart after a timeout has been hit: Mar 19 19:05:31 mail auth[867454]: pam_sss(dovecot:auth): received for user redacted: 4 (System error) Mar 19 19:05:32 mail sssd_be[867455]: Starting up + + With the patched version, this should no longer be a problem. + + + [Where Problems Could Occur] + + + With this patch applied, there may be few edge cases in (and varying + b/w) different versions of GnuTLS. And also some bits that are discussed + in https://bugs.openldap.org/show_bug.cgi?id=8650. + + But that said, the patched version is already being run in production + for over two weeks time (at the time of writing - 07/04/21). So I + believe the SRU will clearly benefit from this and has lower risk of + regression. + + + [More Info] + === A reduced version of the patch linked above can be found attached to this bug report. This patch has been applied to version 2.4.49+dfsg- 2ubuntu1.7 and has been running in production for approximately a week and the issue has no longer occurred. No other issues have appeared during this period. - - As this bug affects all systems using LDAP with TLS, I suggest that the - fix for this bug is ported to Ubuntu 20.04 LTS and potentially earlier - versions. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1921562 Title: Intermittent hangs during ldap_search_ext when TLS enabled To manage notifications about this bug go to: https://bugs.launchpad.net/openldap/+bug/1921562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1921562] Re: Intermittent hangs during ldap_search_ext when TLS enabled
Hello Vincent, I've uploaded a fixed package in my PPA: https://launchpad.net/~utkarsh/+archive/ubuntu/experimental-dump. Could you please test this if it work alright for you before I push this to the official archive? Thanks! ** Changed in: openldap (Ubuntu Focal) Assignee: (unassigned) => Utkarsh Gupta (utkarsh) ** Changed in: openldap (Ubuntu Focal) Status: Triaged => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1921562 Title: Intermittent hangs during ldap_search_ext when TLS enabled To manage notifications about this bug go to: https://bugs.launchpad.net/openldap/+bug/1921562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1921562] Re: Intermittent hangs during ldap_search_ext when TLS enabled
Hello Vincent, Since the Debian version has these fixes incorporated, this would be fixed in Hirsute already (as it's in sync (with a minor delta)). For Focal, it will need somebody affected to commit to doing the necessary QA after the update is prepared (without that QA, we won't be able to land the update). The process is documented at https://wiki.ubuntu.com/StableReleaseUpdates#Procedure. I'll add this task to the server team's backlog. If you'd like to do it sooner, you are welcome to prepare the update yourself following the documented process. Thanks! ** Also affects: openldap (Ubuntu Focal) Importance: Undecided Status: New ** Changed in: openldap (Ubuntu) Status: Confirmed => Fix Released ** Changed in: openldap (Ubuntu Focal) Status: New => Triaged ** Tags added: bitesize server-next -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1921562 Title: Intermittent hangs during ldap_search_ext when TLS enabled To manage notifications about this bug go to: https://bugs.launchpad.net/openldap/+bug/1921562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1921562] Re: Intermittent hangs during ldap_search_ext when TLS enabled
Just to be sure, is there anything that I would need to do in order to have the bugfix applied in a new openldap release for Ubuntu 20.04? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1921562 Title: Intermittent hangs during ldap_search_ext when TLS enabled To manage notifications about this bug go to: https://bugs.launchpad.net/openldap/+bug/1921562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1921562] Re: Intermittent hangs during ldap_search_ext when TLS enabled
The attachment "retry-tls-connect-on-eintr-eagain.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu- reviewers, unsubscribe the team. [This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.] ** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1921562 Title: Intermittent hangs during ldap_search_ext when TLS enabled To manage notifications about this bug go to: https://bugs.launchpad.net/openldap/+bug/1921562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1921562] Re: Intermittent hangs during ldap_search_ext when TLS enabled
Launchpad has imported 19 comments from the remote bug at https://bugs.openldap.org/show_bug.cgi?id=8650. If you reply to an imported comment from within Launchpad, your comment will be sent to the remote bug automatically. Read more about Launchpad's inter-bugtracker facilities at https://help.launchpad.net/InterBugTracking. On 2017-05-06T22:32:26+00:00 Ryan Tandy wrote: Full_Name: Ryan Tandy Version: RE24 OS: Debian URL: Submission from: (NULL) (24.68.41.160) Submitted by: ryan https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=861838 That bug's submitter seems to have unintentionally configured their slapd with the entire list of system CAs. They're fixing it, but we have a bug here too. When the ServerHello is larger than 16kb, gnutls_handshake can return GNUTLS_E_AGAIN. In theory this was always possible, but I'm only seeing it happen with gnutls 3.x and haven't the exact change responsible. We need to loop gnutls_handshake until it completes, like we do already in the re-handshake case. Reply at: https://bugs.launchpad.net/ubuntu/+source/openldap/+bug/1921562/comments/0 On 2017-05-06T22:52:25+00:00 Ryan Tandy wrote: changed notes changed state Open to Test moved from Incoming to Software Bugs Reply at: https://bugs.launchpad.net/ubuntu/+source/openldap/+bug/1921562/comments/1 On 2017-05-06T22:58:54+00:00 Ryan Tandy wrote: Committed the fix, and pinged the submitter to test it. Reply at: https://bugs.launchpad.net/ubuntu/+source/openldap/+bug/1921562/comments/2 On 2018-02-09T17:22:50+00:00 Quanah-x wrote: changed notes changed state Test to Release Reply at: https://bugs.launchpad.net/ubuntu/+source/openldap/+bug/1921562/comments/3 On 2018-03-22T19:25:42+00:00 Quanah-x wrote: changed notes changed state Release to Closed Reply at: https://bugs.launchpad.net/ubuntu/+source/openldap/+bug/1921562/comments/4 On 2018-08-03T15:19:06+00:00 Kartik Subbarao wrote: Hi Ryan, I'm running into a problem with slapd 2.4.46 hanging on Ubuntu 18.04, which seems to be a side effect of the ITS#8650 patch: https://github.com/openldap/openldap/commit/7b5181da8cdd47a13041f9ee36fa9590a0fa6e48 slapd will run fine for a while, but during some periods of high-traffic, it'll hang. It'll peg the CPU at 100% and won't respond to any new LDAP connections. After some time, it'll resume working again, but overall it's fairly unreliable. strace on slapd during the hang shows that it's constantly making read() calls that return EAGAIN. After doing a gdb stack trace on slapd, I realized that these read() calls are happening as part of the busywait for loop in tlsg_session_accept() that repeatedly calls gnutls_handshake() when it gets EAGAIN. When slapd recovers from this hang state, the first message it prints is a TLS negotiation failure error on the culprit file descriptor. If I back out the ITS#8650 patch, the problem goes away. If I insert sleep(1) in the for loop, slapd no longer pegs the CPU at 100%, but it still becomes unresponsive during these high-traffic periods. I don't know what's happening during these high-traffic periods that causes the TLS negotiation to go astray. Unfortunately it's not easy to reproduce this problem outside of this production environment, given the diversity of clients running different OSes with various versions of SSL libraries. I'm wondering if there is a better way to handle EAGAIN returned from gnutls_handshake(), instead of doing a busywait as in ITS#8650, or my simplistic attempt at inserting a sleep() call which doesn't really seem to help. I'm wondering how the GnuTLS developers intend for people to use gnutls_handshake() properly, so as to gracefully handle sessions that involve long packets on the one hand, without opening up a vulnerability to chew up lots of system resources on the other hand. Regards, -Kartik Reply at: https://bugs.launchpad.net/ubuntu/+source/openldap/+bug/1921562/comments/5 On 2018-08-03T16:09:47+00:00 Ryan Tandy wrote: Hi Kartik, On Fri, Aug 03, 2018 at 11:19:06AM -0400, Kartik Subbarao wrote: >I'm running into a problem with slapd 2.4.46 hanging on Ubuntu 18.04, >which seems to be a side effect of the ITS#8650 patch: > >https://github.com/openldap/openldap/commit/7b5181da8cdd47a13041f9ee36fa9590a0fa6e48 > >slapd will run fine for a while, but during some periods of >high-traffic, it'll hang. It'll peg the CPU at 100% and won't respond >to any new LDAP connections. After some time, it'll resume
[Bug 1921562] Re: Intermittent hangs during ldap_search_ext when TLS enabled
** Description changed: When connecting to an LDAP server with TLS, ldap_search_ext can hang if during the initial TLS handshake a signal is received by the process. The cause of this bug is the same as https://bugs.openldap.org/show_bug.cgi?id=8650 which was fixed in https://git.openldap.org/openldap/openldap/-/commit/735e1ab and was released as part of version 2.4.50. This bug effects Ubuntu 20.04 LTS and potentially earlier Ubuntu releases. Later Ubuntu releases use an openldap version that is at least 2.4.50 and are therefore not affected. In our case this bug cause failures in the SSSD LDAP backend at least once per day, resulting in authentication errors followed by a sssd_be restart after a timeout has been hit: Mar 19 19:05:31 mail auth[867454]: pam_sss(dovecot:auth): received for user redacted: 4 (System error) Mar 19 19:05:32 mail sssd_be[867455]: Starting up A reduced version of the patch linked above can be found attached to this bug report. This patch has been applied to version 2.4.49+dfsg- 2ubuntu1.7 and has been running in production for approximately a week and the issue has no longer occurred. No other issues have appeared during this period. - As this bug affects al systems using LDAP with TLS , I suggest that the + As this bug affects all systems using LDAP with TLS, I suggest that the fix for this bug is ported to Ubuntu 20.04 LTS and potentially earlier versions. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1921562 Title: Intermittent hangs during ldap_search_ext when TLS enabled To manage notifications about this bug go to: https://bugs.launchpad.net/openldap/+bug/1921562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1921562] Re: Intermittent hangs during ldap_search_ext when TLS enabled
** Bug watch added: bugs.openldap.org/ #8650 https://bugs.openldap.org/show_bug.cgi?id=8650 ** Also affects: openldap via https://bugs.openldap.org/show_bug.cgi?id=8650 Importance: Unknown Status: Unknown -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1921562 Title: Intermittent hangs during ldap_search_ext when TLS enabled To manage notifications about this bug go to: https://bugs.launchpad.net/openldap/+bug/1921562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1921562] Re: Intermittent hangs during ldap_search_ext when TLS enabled
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: openldap (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1921562 Title: Intermittent hangs during ldap_search_ext when TLS enabled To manage notifications about this bug go to: https://bugs.launchpad.net/openldap/+bug/1921562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
