Peter, Thanks again for your quick reply.
On 2/13/19 15:09, Peter Pentchev wrote: > On Wed, Feb 13, 2019 at 01:45:10PM -0500, Christopher Schultz wrote: >> Peter, >> On 2/13/19 13:09, Peter Pentchev wrote: >>> On Wed, Feb 13, 2019 at 12:17:24PM -0500, Christopher Schultz wrote: >>>> Hello, >>>> >>>> I'm new to the list so I'm sorry if this isn't the right place to report >>>> this. >>>> >>>> I had a server stop responding to stunnel connections sometime yesterday >>>> and the resolution was ultimately to reboot the server and everything >>>> was okay. Restarting the stunnel service was not enough to get things >>>> working again. >>> [snip] >>>> >>>> Looking through the log file, I could see that there were some odd >>>> messages coming from stunnel in the daemon.log file suggesting that >>>> there might be a memory leak. I won't post them here unless requested, >>>> as they may represent a potential security issue. >>> >>> Hi, >>> >>> Unfortunately this is a known problem with the Debian package of >>> stunnel; I did not get around to backporting the fixes to the stunnel >>> version that is in Debian 9.x (stretch); I'm really sorry about that. >> >> Confirmed: I'm running Debian Stretch: >> >> $ lsb_release -a >> No LSB modules are available. >> Distributor ID: Debian >> Description: Debian GNU/Linux 9.7 (stretch) >> Release: 9.7 >> Codename: stretch >> >>> I could make a stretch backport of the stunnel version that is in >>> unstable/testing right now - the one that will be in the upcoming >>> Debian 10.x (buster). Actually I'll build a package later tonight and >>> let you know where you can download it from, then I'll upload it >>> to the stretch-backports suite and wait for the backports admins to >>> accept it. This package will not have the memory exhaustion problem >>> that the version in stretch does have. >> >> That would be great. Will I (eventually) end up getting it via the >> normal apt update mechanism? I only have these apt sources configured: >> >> deb http://ftp.us.debian.org/debian/ stretch main >> deb http://security.debian.org/ stretch/updates main > > Well, to be honest, I'm not sure how easy it will be to backport > the change that fixes the memory exhaustion without bringing in > a lot of other changes in later versions of stunnel. > > I believe your best bet would be to add the stretch-backports suite > as described on https://backports.debian.org/Instructions/ and > then install only the stunnel4 package from that suite: > > apt-get install -t stretch-backports stunnel4 > > This will bring in this package, and only this package, from > the backports suite. The package there is exactly the version that > has been in the testing distribution for two months now (it was > accepted almost immediately after I uploaded it, it didn't really > need to go through human approval), and there have been no reports of > trouble with it since then. I can look into that. I don't exactly need an act of congress to add an apt source and update a package. I might want to do it on a "slow day", though :) >> Is there a safe mitigation I can put into place before a patch? I >> believe Debian's stunnel service scripts completely shut-down one >> stunnel process and then start up a new one, so I might have a (very >> short) interruption. I can schedule that via cron if necessary, but I'd >> prefer to avoid even a small interruption if possible. >> >>> Could you just confirm that you are indeed running Debian 9.x (stretch)? >>> (it seems this way from the kernel version that you are running and >>> from the symptoms; stunnel 5.06 in jessie does not have this problem) >> >> I have 5.39. Is this older than 5.06? Or is that a typo? > > No, it was not a typo; I'm sorry if I was unclear. When I said "jessie", > I meant Debian 8.x (jessie) - the previous release of Debian that had > a much older version of stunnel that did not have the changes that, > unfortunately, led to the memory leak. Duh. I didn't even read "jessie", there. Thanks for the clarification. > To be totally honest, the leak itself is partly my fault, since it > was introduced in a patch for an issue that was fixed in a more > elaborate way in later versions of stunnel.> >>> Sorry again, and thanks for your understanding! >> >> No, it's great to get this kind of feedback right away. >> >> I *am* curious about why a service-restart did not seem to fix things. >> With the above memory-exhaustion, would you have expected a >> service-restart to fix everything? (I certainly did.) Is it likely that >> the "restart" didn't actually restart the service, and I was just >> repeatedly testing a borked server process? > > I think that it is possible that this might have been the case; > there was actually a bug in the stunnel4 init script that was fixed > in a later Debian upload (the one that was in the stretch-backports > suite until now). The bug was that the script did not wait for > the old process to really end before attempting to start a new one - > and the new one would not really start, since the kernel would not > allow it to listen on the same ports that the old one was still > listening on. I didn't think of this bug at once when I read your > first message, but it makes sense now. I almost want to wait until it fails again and then inspect it. If a failing-restart does not explain it, then it suggests some kind of corruption of the kernel's TCP/IP stack which doesn't sound like the kind of thing that should be possible. > So, yeah, it seems that I'm guilty of not fixing not one, but two > serious problems with the version of stunnel in the stable Debian > release... I could try to figure out a way to fix the memory > exhaustion one, but it may take me a couple of days. Right now, > your best bet (and the course of action that I recommend) is to > install stunnel 5.50 from the stretch-backports suite. I'll give that a shot and see how things go. If all else fails, I do know that a server reboot will fix the issue. Of course, if I wanted to reboot servers to get them to work, I'd have built everything on Windows :P Thanks, -chris
signature.asc
Description: OpenPGP digital signature
_______________________________________________ stunnel-users mailing list [email protected] https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users
