Bug#953829: regression/crash: mtr: Probes exhausted
On 3/17/20 4:02 AM, Marc Lehmann wrote: > On Sun, Mar 15, 2020 at 03:43:09PM -0700, Robert Woodcock > wrote: >> If you *do* need to differentiate, I would adjust the limit for the >> number of outstanding probes in packet/probe.h, and then recompile: >> >> #define MAX_PROBES 1024 >> >> However, that's not enough - I believe the intent of this limit is to >> match the default Linux limit of 1024 open files per process, which you >> will need to override in /etc/security/limits.conf to make effective use >> of the new limit in your copy of mtr. >> >> Probably the best long-term fix would be to automatically free the >> oldest probe when the newest one is requested, but I don't know how >> practical that would be. > Thanks a lot for this detailed analysis and tips. The strange thing, > though, is that this is clearly a regression, i.e. these limits (other > than the kernel limit) seem to either be new, or mtr silently handled this > in the past. > > Anyway, if that's going to be it, I'll simply have to adapt, and this > apparently works as designed and is not a bug. > MTR was pretty substantially redesigned about 3 years ago - the UI was separated from the backend (which has to have special privileges to open a raw socket - it used to need to be setuid, but now it uses setcap to provide only the RAW socket privilege needed). That redesign made it into the mtr 0.92 package in the current stable version of Debian - Buster - released in 2019. This was done for security reasons, to limit the possible attack surface for privilege escalation. I have not done a similar analysis on the older versions (0.87 and below) but I suspect that's what you were using before.
Bug#953829: regression/crash: mtr: Probes exhausted
On Sun, Mar 15, 2020 at 03:43:09PM -0700, Robert Woodcock wrote: > If you *do* need to differentiate, I would adjust the limit for the > number of outstanding probes in packet/probe.h, and then recompile: > > #define MAX_PROBES 1024 > > However, that's not enough - I believe the intent of this limit is to > match the default Linux limit of 1024 open files per process, which you > will need to override in /etc/security/limits.conf to make effective use > of the new limit in your copy of mtr. > > Probably the best long-term fix would be to automatically free the > oldest probe when the newest one is requested, but I don't know how > practical that would be. Thanks a lot for this detailed analysis and tips. The strange thing, though, is that this is clearly a regression, i.e. these limits (other than the kernel limit) seem to either be new, or mtr silently handled this in the past. Anyway, if that's going to be it, I'll simply have to adapt, and this apparently works as designed and is not a bug. -- The choice of a Deliantra, the free code+content MORPG -==- _GNU_ http://www.deliantra.net ==-- _ generation ---==---(_)__ __ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / schm...@schmorp.de -=/_/_//_/\_,_/ /_/\_\
Bug#953829: regression/crash: mtr: Probes exhausted
This would be caused by the use of very low intervals, as you mentioned (causing ping probes to be consumed at a fast rate) in combination with lost packets at one or more remote hops (causing the probes to not be freed soon enough). 10 seconds at 100 probes per second does not leave sufficient margin for the default limit of 1024 probes. Assuming you don't need to differentiate a lost ping packet from a reply that was received much later than expected, use the -Z option to shorten the timeout so that the probes are freed up faster. The default is 10 seconds (defined in ui/mtr.c as ctl->probe_timeout). If you *do* need to differentiate, I would adjust the limit for the number of outstanding probes in packet/probe.h, and then recompile: #define MAX_PROBES 1024 However, that's not enough - I believe the intent of this limit is to match the default Linux limit of 1024 open files per process, which you will need to override in /etc/security/limits.conf to make effective use of the new limit in your copy of mtr. Probably the best long-term fix would be to automatically free the oldest probe when the newest one is requested, but I don't know how practical that would be. On 3/13/20 2:37 PM, Marc Lehmann wrote: > Package: mtr-tiny > Version: 0.92-2 > Severity: normal > > Dear Maintainer, > > recent versions of mtr started to crash seemingly randomly with the > following message: > >mtr: Probes exhausted > > and an exit status of 1. It's not consistently reproducible, but I found > that when specifying a low interval, e.g. > >mtr -i 0.01 target > > it happens much more often than with the default interval. > > This might not be new. However, I use mtr daily for many years, and also > with low intervals, and have never seen this message until recently, so I > assume this is new behaviour. > > -- System Information: > Debian Release: 10.3 > APT prefers stable > APT policy: (990, 'stable'), (500, 'unstable-debug'), (500, > 'testing-debug'), (500, 'stable-updates'), (500, 'stable-debug'), (500, > 'unstable'), (500, 'testing'), (1, 'experimental-debug'), (1, 'experimental') > Architecture: amd64 (x86_64) > Foreign Architectures: i386, x32 > > Kernel: Linux 5.4.24-050424-generic (SMP w/8 CPU cores) > Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, > TAINT_UNSIGNED_MODULE > Locale: LANG=en_DK.UTF-8, LC_CTYPE=en_DK.UTF-8 (charmap=UTF-8), > LANGUAGE=en_DK.UTF-8 (charmap=UTF-8) > Shell: /bin/sh linked to /usr/bin/dash > Init: systemd (via /run/systemd/system) > LSM: AppArmor: enabled > > Versions of packages mtr-tiny depends on: > ii libc62.28-10 > ii libncurses6 6.1+20181013-2+deb10u2 > ii libtinfo66.1+20181013-2+deb10u2 > > mtr-tiny recommends no packages. > > mtr-tiny suggests no packages. > > -- no debconf information
Bug#953829: regression/crash: mtr: Probes exhausted
Package: mtr-tiny Version: 0.92-2 Severity: normal Dear Maintainer, recent versions of mtr started to crash seemingly randomly with the following message: mtr: Probes exhausted and an exit status of 1. It's not consistently reproducible, but I found that when specifying a low interval, e.g. mtr -i 0.01 target it happens much more often than with the default interval. This might not be new. However, I use mtr daily for many years, and also with low intervals, and have never seen this message until recently, so I assume this is new behaviour. -- System Information: Debian Release: 10.3 APT prefers stable APT policy: (990, 'stable'), (500, 'unstable-debug'), (500, 'testing-debug'), (500, 'stable-updates'), (500, 'stable-debug'), (500, 'unstable'), (500, 'testing'), (1, 'experimental-debug'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386, x32 Kernel: Linux 5.4.24-050424-generic (SMP w/8 CPU cores) Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE Locale: LANG=en_DK.UTF-8, LC_CTYPE=en_DK.UTF-8 (charmap=UTF-8), LANGUAGE=en_DK.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages mtr-tiny depends on: ii libc62.28-10 ii libncurses6 6.1+20181013-2+deb10u2 ii libtinfo66.1+20181013-2+deb10u2 mtr-tiny recommends no packages. mtr-tiny suggests no packages. -- no debconf information
