Bug#299875: ppp: out-of-memory 30min after LCP terminated by peer
hi again, sorry for the delay, but the bug triggers only when the remote peer disconnects me - and it does it only once a day. Marco d'Itri wrote: reassign 299875 kernel retitle 299875 CAN-2005-0384: Remote Linux DoS on ppp servers tag 299875 patch security yes, it really looks like a pppd DoS, but as i suspected, things are a bit different here: - pppoe receives PADT from peer - pppd in turn gets LCP terminated by peer - pppd tries to re-establish the connection, says Starting link in the logfile [1] and then just hangs. i have to re-adjust the subject line, because OOM was triggered by something else, *because* pppd did not succeed to re-establish the connection. but oom is totally unrelated here (and fixed on my system). but still: pppd hangs and is only killable via kill -9. but it is *NOT* a cpu hog as stated by Paul Mackerras. i've strace'd the hanging pppd: http://nerdbynature.de/bits/sheep/2.6.11/oom/pppd_2.6.11.4.log (the very first line is the only one i get from just doing strace -p `pidof pppd`, then i did killall pppd, then with -9) to summarise: i've noticed the pppd behaviour first with 2.6.11, then with 2.6.11-rc5-bk2, now with 2.6.11.3 and 2.6.11.4. downgrading to ppp_2.4.2+20040428-6_i386 solves it. thank you for your time, Christian. PS: i'll close the issue i reported on lkml: http://www.ussg.iu.edu/hypermail/linux/kernel/0503.2/0316.html [1] http://nerdbynature.de/bits/sheep/2.6.11/oom/oom_2.6.11.3.txt -- BOFH excuse #139: UBNC (user brain not connected) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#299875: ppp: out-of-memory 30min after LCP terminated by peer
Okay. Do I correctly understand that kernel patch + downgrade solves your problem? And, if you have *just* the kernel patch, /usr/sbin/pppd hangs, but doesn't crash the system? Thanks, Justin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#299875: ppp: out-of-memory 30min after LCP terminated by peer
On Mar 20, Christian Kujau [EMAIL PROTECTED] wrote: - pppd tries to re-establish the connection, says Starting link in the logfile [1] and then just hangs. Weird. Maybe you hit something like #298657 and the other similar bugs. -- ciao, Marco signature.asc Description: Digital signature
Bug#299875: ppp: out-of-memory 30min after LCP terminated by peer
Justin Pryzby wrote: Okay. Do I correctly understand that kernel patch + downgrade solves your problem? i don't even have to patch the kernel. i tried with 2.6.11-rc5-bk2, same thing. but (only!) downgrading ppp_2.4.2 (current is ppp_2.4.3) did the trick. And, if you have *just* the kernel patch, /usr/sbin/pppd hangs, but doesn't crash the system? with (current) ppp_2.4.3 pppd hangs, but does not crash the kernel. as a side note: the kernel never really crashed, but the kernel's OOM killer kicked in because i kept running a script (via cron), which was checking internet-connectivity. when pppd decided to hang and was not kill'able via -TERM, the script went nuts and started itsself again and gain. i noticed that always 30min after pppd's hang the machine went OOM. by then, the script must have been started 2000times or so. really, OOM is not longer the issue here, it's just pppd hanging, so i even could set this report from critical not normal, as i seem to be the only one crying here. thanks, Christian. -- BOFH excuse #111: The salesman drove over the CPU board. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Processed: Re: Bug#299875: ppp: out-of-memory 30min after LCP terminated by peer
Processing commands for [EMAIL PROTECTED]: reassign 299875 kernel Bug#299875: ppp: out-of-memory 30min after LCP terminated by peer Bug reassigned from package `ppp' to `kernel'. retitle 299875 CAN-2005-0384: Remote Linux DoS on ppp servers Bug#299875: ppp: out-of-memory 30min after LCP terminated by peer Changed Bug title. tag 299875 patch security Bug#299875: CAN-2005-0384: Remote Linux DoS on ppp servers There were no tags set. Tags added: patch, security thanks Stopping processing here. Please contact me if you need assistance. Debian bug tracking system administrator (administrator, Debian Bugs database) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#299875: ppp: out-of-memory 30min after LCP terminated by peer
reassign 299875 kernel
retitle 299875 CAN-2005-0384: Remote Linux DoS on ppp servers
tag 299875 patch security
thanks
Paul Mackerras says that this bug affects all kernels (2.4 and 2.6) and
can be easily triggered remotely, but is only a CPU DoS.
from 2.6.11.4:
diff -Nru a/drivers/net/ppp_async.c b/drivers/net/ppp_async.c
--- a/drivers/net/ppp_async.c 2005-03-15 16:09:59 -08:00
+++ b/drivers/net/ppp_async.c 2005-03-15 16:09:59 -08:00
@@ -1000,7 +1000,7 @@
data += 4;
dlen -= 4;
/* data[0] is code, data[1] is length */
- while (dlen = 2 dlen = data[1]) {
+ while (dlen = 2 dlen = data[1] data[1] = 2) {
switch (data[0]) {
case LCP_MRU:
val = (data[2] 8) + data[3];
--
ciao,
Marco
signature.asc
Description: Digital signature
Bug#299875: ppp: out-of-memory 30min after LCP terminated by peer
Justin Pryzby wrote: I assume that you have seen this: http://kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.11.4 yes i have*now*. obviously this was a security issue (CAN-2005-0384) and i *guess* that's why the issue was not discussed in public. what pity and what a waste of time in tracking this down when the security guys are already on it and release a fix out-of-the-box. If not .. tag patch:) In less than 100 lines, even! yes, 2.6.11.4 is out including this fix. Marco d'Itri commented: Paul Mackerras says that this bug affects all kernels (2.4 and 2.6) and can be easily triggered remotely, but is only a CPU DoS. --^ actually my problem was *not* about a CPU DoS but a (out-of)memory DoS. cpu was spinning normally. and i was not able to tell the exact kernel version when the problem started - instead my research led to the assumption that the new ppp package was to blame. well, i'm already compiling 2.6.11.4, i'll see what it gives. thank you for your concern, Christian. -- BOFH excuse #188: ..disk or the processor is on fire. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#299875: ppp: out-of-memory 30min after LCP terminated by peer
Package: ppp Version: 2.4.2+20040428-6 Severity: critical Justification: breaks the whole system hello pppeople, i've encountered serious problems with pppd and days of monitoring the issues lead me to the assumption that pppd is to blame. feel free to tell me otherwise and i'll happily forward the problem to the right people - because i am not guru enough to fix it. the problem in short: i am using pppd with pppoe (ADSL dialup) i get disconnected by my ISP once a day and pppd receives LCP terminated by peer then. normally pppd will reconnect and the box would be online again. this does not happen anymore, instead 30min after disconnect all RAM is chewed up by *something* (still don't know by what, please read on) and OOM killer kicks in, killing almost every application. i have reported the issue to the linux-kernel mailinglist first, because i thought it was a kernel issue: http://lkml.org/lkml/2005/3/8/173 http://lkml.org/lkml/2005/3/10/109 maybe the replies are noteworthy too. yesterday i hit the issue again, with more (hopefully helpful) details: http://www.ussg.iu.edu/hypermail/linux/kernel/0503.2/0316.html again, the time between LCP terminated by peer to the first OOM messages are (alsmost exactly) 30min. when this happens, i am usually not around, so when i get to the machine, OOM has already happened. i can recover the machine (no reboot, but SYSRQ-E) and i have to kill -9 pppd, anything else does not work: http://nerdbynature.de/bits/sheep/2.6.11/oom/pppd.log (Terminating on signal 15 does not work, +++ killed by SIGKILL +++ is the only choice.) i first noticed this behaviour with a new kernel, namely 2.6.11, but downgrading to a former (stable for me) kernel (2.6.11-rc5-bk2) did not help. instead, i downgraded from pppd-2.4.3-20041231+2 to ppp-2.4.2+20040428-6 and got rid of the problem! 2.4.3 was uploaded on 26.02.2005, and a couple of days later i switched to a new kernel and did a apt-get upgrade. that could be the cause, i suspected kernel issues first. have a look at http://nerdbynature.de/bits/sheep/2.6.11/oom/ for further details, feel free to ask. i still did not manage to reproduce the problem by hand, e.g. by setting up a pppoe server on the same machine, connecting with pppd call... and killing the pppoe server (in the hope that it might send LCP terminated by peer to the pppd). thank you for your time, Christian. -- System Information: Debian Release: 3.1 APT prefers unstable APT policy: (990, 'unstable'), (1, 'experimental') Architecture: i386 (i686) Kernel: Linux 2.6.11.3 Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968) Versions of packages ppp depends on: ii libc6 2.3.2.ds1-20 GNU C Library: Shared libraries an ii libpam-modules 0.76-22 Pluggable Authentication Modules f ii libpam-runtime 0.76-22 Runtime support for the PAM librar ii libpam0g0.76-22 Pluggable Authentication Modules l ii libpcap0.7 0.7.2-7 System interface for user-level pa ii libssl0.9.7 0.9.7e-3 SSL shared libraries ii makedev 2.3.1-76 creates device files in /dev ii netbase 4.20 Basic TCP/IP networking system ii procps 1:3.2.5-1/proc file system utilities -- no debconf information -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#299875: ppp: out-of-memory 30min after LCP terminated by peer
I assume that you have seen this: http://kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.11.4 If not .. tag patch:) In less than 100 lines, even! Justin On Thu, Mar 17, 2005 at 02:54:56AM +0100, Christian wrote: Package: ppp Version: 2.4.2+20040428-6 Severity: critical Justification: breaks the whole system hello pppeople, i've encountered serious problems with pppd and days of monitoring the issues lead me to the assumption that pppd is to blame. feel free to tell me otherwise and i'll happily forward the problem to the right people - because i am not guru enough to fix it. the problem in short: i am using pppd with pppoe (ADSL dialup) i get disconnected by my ISP once a day and pppd receives LCP terminated by peer then. normally pppd will reconnect and the box would be online again. this does not happen anymore, instead 30min after disconnect all RAM is chewed up by *something* (still don't know by what, please read on) and OOM killer kicks in, killing almost every application. i have reported the issue to the linux-kernel mailinglist first, because i thought it was a kernel issue: http://lkml.org/lkml/2005/3/8/173 http://lkml.org/lkml/2005/3/10/109 maybe the replies are noteworthy too. yesterday i hit the issue again, with more (hopefully helpful) details: http://www.ussg.iu.edu/hypermail/linux/kernel/0503.2/0316.html again, the time between LCP terminated by peer to the first OOM messages are (alsmost exactly) 30min. when this happens, i am usually not around, so when i get to the machine, OOM has already happened. i can recover the machine (no reboot, but SYSRQ-E) and i have to kill -9 pppd, anything else does not work: http://nerdbynature.de/bits/sheep/2.6.11/oom/pppd.log (Terminating on signal 15 does not work, +++ killed by SIGKILL +++ is the only choice.) i first noticed this behaviour with a new kernel, namely 2.6.11, but downgrading to a former (stable for me) kernel (2.6.11-rc5-bk2) did not help. instead, i downgraded from pppd-2.4.3-20041231+2 to ppp-2.4.2+20040428-6 and got rid of the problem! 2.4.3 was uploaded on 26.02.2005, and a couple of days later i switched to a new kernel and did a apt-get upgrade. that could be the cause, i suspected kernel issues first. have a look at http://nerdbynature.de/bits/sheep/2.6.11/oom/ for further details, feel free to ask. i still did not manage to reproduce the problem by hand, e.g. by setting up a pppoe server on the same machine, connecting with pppd call... and killing the pppoe server (in the hope that it might send LCP terminated by peer to the pppd). thank you for your time, Christian. -- System Information: Debian Release: 3.1 APT prefers unstable APT policy: (990, 'unstable'), (1, 'experimental') Architecture: i386 (i686) Kernel: Linux 2.6.11.3 Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968) Versions of packages ppp depends on: ii libc6 2.3.2.ds1-20 GNU C Library: Shared libraries an ii libpam-modules 0.76-22 Pluggable Authentication Modules f ii libpam-runtime 0.76-22 Runtime support for the PAM librar ii libpam0g0.76-22 Pluggable Authentication Modules l ii libpcap0.7 0.7.2-7 System interface for user-level pa ii libssl0.9.7 0.9.7e-3 SSL shared libraries ii makedev 2.3.1-76 creates device files in /dev ii netbase 4.20 Basic TCP/IP networking system ii procps 1:3.2.5-1/proc file system utilities -- no debconf information -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED] -- Justin Pryzby whois jgalt References [0] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#299875: ppp: out-of-memory 30min after LCP terminated by peer
Including a patch from kernel.org. Given the size of the patch, you may not trust it, but that's what I got .. http://www.kernel.org/diff/diffview.cgi?file=%2Fpub%2Flinux%2Fkernel%2Fv2.6%2Fpatch-2.6.11.4.bz2;z=15 If I understand the situation correctly, this needs to be reassigned and cloned to a couple kernels, but I'll that that up to someone else to confirm. Justin On Thu, Mar 17, 2005 at 02:09:19AM -0500, pryzbyj wrote: I assume that you have seen this: http://kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.11.4 If not .. tag patch:) In less than 100 lines, even! Justin On Thu, Mar 17, 2005 at 02:54:56AM +0100, Christian wrote: Package: ppp Version: 2.4.2+20040428-6 Severity: critical Justification: breaks the whole system hello pppeople, i've encountered serious problems with pppd and days of monitoring the issues lead me to the assumption that pppd is to blame. feel free to tell me otherwise and i'll happily forward the problem to the right people - because i am not guru enough to fix it. the problem in short: i am using pppd with pppoe (ADSL dialup) i get disconnected by my ISP once a day and pppd receives LCP terminated by peer then. normally pppd will reconnect and the box would be online again. this does not happen anymore, instead 30min after disconnect all RAM is chewed up by *something* (still don't know by what, please read on) and OOM killer kicks in, killing almost every application. i have reported the issue to the linux-kernel mailinglist first, because i thought it was a kernel issue: http://lkml.org/lkml/2005/3/8/173 http://lkml.org/lkml/2005/3/10/109 maybe the replies are noteworthy too. yesterday i hit the issue again, with more (hopefully helpful) details: http://www.ussg.iu.edu/hypermail/linux/kernel/0503.2/0316.html again, the time between LCP terminated by peer to the first OOM messages are (alsmost exactly) 30min. when this happens, i am usually not around, so when i get to the machine, OOM has already happened. i can recover the machine (no reboot, but SYSRQ-E) and i have to kill -9 pppd, anything else does not work: http://nerdbynature.de/bits/sheep/2.6.11/oom/pppd.log (Terminating on signal 15 does not work, +++ killed by SIGKILL +++ is the only choice.) i first noticed this behaviour with a new kernel, namely 2.6.11, but downgrading to a former (stable for me) kernel (2.6.11-rc5-bk2) did not help. instead, i downgraded from pppd-2.4.3-20041231+2 to ppp-2.4.2+20040428-6 and got rid of the problem! 2.4.3 was uploaded on 26.02.2005, and a couple of days later i switched to a new kernel and did a apt-get upgrade. that could be the cause, i suspected kernel issues first. have a look at http://nerdbynature.de/bits/sheep/2.6.11/oom/ for further details, feel free to ask. i still did not manage to reproduce the problem by hand, e.g. by setting up a pppoe server on the same machine, connecting with pppd call... and killing the pppoe server (in the hope that it might send LCP terminated by peer to the pppd). thank you for your time, Christian. -- System Information: Debian Release: 3.1 APT prefers unstable APT policy: (990, 'unstable'), (1, 'experimental') Architecture: i386 (i686) Kernel: Linux 2.6.11.3 Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968) Versions of packages ppp depends on: ii libc6 2.3.2.ds1-20 GNU C Library: Shared libraries an ii libpam-modules 0.76-22 Pluggable Authentication Modules f ii libpam-runtime 0.76-22 Runtime support for the PAM librar ii libpam0g0.76-22 Pluggable Authentication Modules l ii libpcap0.7 0.7.2-7 System interface for user-level pa ii libssl0.9.7 0.9.7e-3 SSL shared libraries ii makedev 2.3.1-76 creates device files in /dev ii netbase 4.20 Basic TCP/IP networking system ii procps 1:3.2.5-1/proc file system utilities -- no debconf information -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED] -- Justin Pryzby whois jgalt References [0] -- Justin Pryzby whois jgalt References [0] !DOCTYPE HTML PUBLIC -//W3C//DTD HTML 4.01//EN html head title/pub/linux/kernel/v2.6/patch-2.6.11.4.bz2/title meta http-equiv=Content-Type content=text/html; charset=ISO-8859-1 link href=/diff/diff.css rel=stylesheet type=text/css /head body !-- $Id: diffview.cgi,v 1.53 2005/02/26 02:18:44 hpa Exp $ -- !-- /pub/linux/kernel/v2.6/patch-2.6.11.4.bz2 15 -- h1a href=/pub/linux/kernel/v2.6/patch-2.6.11.4.bz2 class=lsrc/pub/linux/kernel/v2.6/patch-2.6.11.4.bz2/a/h1 pre div class=subha name=15--- a/drivers/net/ppp_async.c 2005-03-15 16:09:56 -08:00/a
