Re: Bug#638322: nfs-common: rpc.statd binds to udp port 631 preventing cups startup
On Sun, 21 Aug 2011, Henrique de Moraes Holschuh h...@debian.org wrote: On Sat, 20 Aug 2011, Andreas Barth wrote: * Henrique de Moraes Holschuh (h...@debian.org) [110820 14:39]: Yes. And we can easily maintain a current one for Debian-packaged software, although the initial build of such a blacklist will take some work. Actually, the existing interface net.ipv4.ip_local_port_range seems to work quite well. And there are so many ports that for most servers it # cat /proc/sys/net/ipv4/ip_local_port_range 32768 61000 The above is from one of my systems. This isn't used for RPC, presumably because they want the special 1024 port numbers that imply root ownership. No, it doesn't. And we have at least one extremely important protocol that needs as many ports as we can give it (DNS). Aug 21 11:42:48 ns named[2382]: using default UDP/IPv4 port range: [1024, 65535] Aug 21 11:42:48 ns named[2382]: using default UDP/IPv6 port range: [1024, 65535] BIND seems to use ports 1024 as well, again this is different from the typical RPC issues but does have the potential to cause problems (there are more than a few UDP ports 1024 in /etc/services). Maybe BIND should be patched to use the same port reservation procedure as RPC. A blacklist is the way to go, and we already have it. We just need to fill it, make it easier to extend (.d directory), tell people about it, and teach stuff other than SunRPC to use it when necessary. Yes. -- My Main Blog http://etbe.coker.com.au/ My Documents Bloghttp://doc.coker.com.au/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201108212238.05409.russ...@coker.com.au
Re: Bug#638322: nfs-common: rpc.statd binds to udp port 631 preventing cups startup
On Sat, 20 Aug 2011, Adam Borowski kilob...@angband.pl wrote: It seems to me that the only problem is if you run multiple instances of a daemon on different ports and don't use /etc/bindresvport.blacklist, SE Linux, or some other method of telling bindresvport() to leave your port alone. That wouldn't be an issue of sysadmin freedom but sysadmin ignorance (and I am one of the people who was ignorant of bindresvport.blacklist). You can't blame sysadmin ignorance. I've just grepped through every single man page in Debian (ok, amd64 main), and there is not a single Ignorance means not knowing. Sure there are probably some bug reports about man pages due, but it's still something you or I could have found out. apt-get source libc6 No other daemon I know has this problem. If I install daemon foo, I can expect it to not touch any ports it hasn't been configured to use. It's just portmap/SunRPC that uses random scatter-shot that can trample on something else. Yes, SunRPC and anything that opens a port for callback. So what about this: let's reserve a number of ports for portmap's exclusive usage[1]. There's like 900 unused assignments, so there's plenty of space than could be parcelled off. SunRPC has long since degenerated from something with a general purpose to a peculiarity of NFS, so not many ports are needed. Only under a pathological configuration one could exceed any reasonable static limit, and in that case bindresvport() would revert to the blacklist+scattershot. The problem with this theory is the fact that the problem that was reported with CUPS only occurred after bindresvport() had used every port from 1023 down to 631. A casual scan of /etc/services reveals that there are no long contiguous ranges available without reserved ports. If you start at the top the common ports pop3s and imaps could be reached quite quickly. So it seems that some sort of blacklist is the only way to go. The idea of a .d directory for blacklist files such that every package installation that is likely to use some ports will automatically have a reservation is a good one. Of course there's still the corner case of trying to install CUPS (or some other daemon) after a long-running RPC service has grabbed the port. Maybe we should default to having ports such as 631, 993, 995, 873, 587, 636, 546, and 547 reserved at all times. From a quick scan of /etc/services they seem to be the most likely ports to be used in the 500-1024 range. -- My Main Blog http://etbe.coker.com.au/ My Documents Bloghttp://doc.coker.com.au/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201108202219.58742.russ...@coker.com.au
Re: Bug#638322: nfs-common: rpc.statd binds to udp port 631 preventing cups startup
On Sat, 20 Aug 2011, Russell Coker wrote: On Sat, 20 Aug 2011, Adam Borowski kilob...@angband.pl wrote: It seems to me that the only problem is if you run multiple instances of a daemon on different ports and don't use /etc/bindresvport.blacklist, SE Linux, or some other method of telling bindresvport() to leave your port alone. That wouldn't be an issue of sysadmin freedom but sysadmin ignorance (and I am one of the people who was ignorant of bindresvport.blacklist). You can't blame sysadmin ignorance. I've just grepped through every single man page in Debian (ok, amd64 main), and there is not a single Ignorance means not knowing. Sure there are probably some bug reports about man pages due, but it's still something you or I could have found out. ... No other daemon I know has this problem. If I install daemon foo, I can expect it to not touch any ports it hasn't been configured to use. It's just portmap/SunRPC that uses random scatter-shot that can trample on something else. Yes, SunRPC and anything that opens a port for callback. Firewall port blocking can also cause such problems (denial of service). While it is a different problem, it has the same roots as SunRPC binding to undesired sockets: applications that use random sockets do not know whether they're going to get a socket they're supposed to use. Intelligent use of conntrack can help on single hosts (reducing the problem to incoming callback connections), but most sites have border policies that forbid any traffic to flow for some ports. It causes minor issues for DNS traffic (timeouts on a small fraction of the queries), for example. So it seems that some sort of blacklist is the only way to go. Yes. And we can easily maintain a current one for Debian-packaged software, although the initial build of such a blacklist will take some work. The idea of a .d directory for blacklist files such that every package installation that is likely to use some ports will automatically have a reservation is a good one. Of course there's still the corner case of trying to install CUPS (or some other daemon) after a long-running RPC service has grabbed the port. That's not such a big problem, as it will be noticed immediately and causes no surprise downtime of a service. Maybe we should default to having ports such as 631, 993, 995, 873, 587, 636, 546, and 547 reserved at all times. From a quick scan of /etc/services they seem to be the most likely ports to be used in the 500-1024 range. Looks good, and we can take extra ports as bug reports. A mail to d-d-a and a short article to planet.d.o and LWN may help to raise awareness of such issues: although this _is_ a longstanding and _well known_ issue, the ways to avoid the worst problems it can cause are _not_ well known. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110820123852.ga22...@khazad-dum.debian.net
Re: Bug#638322: nfs-common: rpc.statd binds to udp port 631 preventing cups startup
* Henrique de Moraes Holschuh (h...@debian.org) [110820 14:39]: Yes. And we can easily maintain a current one for Debian-packaged software, although the initial build of such a blacklist will take some work. Actually, the existing interface net.ipv4.ip_local_port_range seems to work quite well. And there are so many ports that for most servers it seems acceptable to limit the outgoing ports to only a tiny portion of port numbers (like 1/4th or so). Andi -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110820141725.gf15...@mails.so.argh.org
Re: Bug#638322: nfs-common: rpc.statd binds to udp port 631 preventing cups startup
On Sat, 2011-08-20 at 16:17 +0200, Andreas Barth wrote: * Henrique de Moraes Holschuh (h...@debian.org) [110820 14:39]: Yes. And we can easily maintain a current one for Debian-packaged software, although the initial build of such a blacklist will take some work. Actually, the existing interface net.ipv4.ip_local_port_range seems to work quite well. And there are so many ports that for most servers it seems acceptable to limit the outgoing ports to only a tiny portion of port numbers (like 1/4th or so). This has nothing to do with bindresvport(). Ben. signature.asc Description: This is a digitally signed message part
Re: Bug#638322: nfs-common: rpc.statd binds to udp port 631 preventing cups startup
On Sat, 20 Aug 2011, Andreas Barth wrote: * Henrique de Moraes Holschuh (h...@debian.org) [110820 14:39]: Yes. And we can easily maintain a current one for Debian-packaged software, although the initial build of such a blacklist will take some work. Actually, the existing interface net.ipv4.ip_local_port_range seems to work quite well. And there are so many ports that for most servers it No, it doesn't. And we have at least one extremely important protocol that needs as many ports as we can give it (DNS). A blacklist is the way to go, and we already have it. We just need to fill it, make it easier to extend (.d directory), tell people about it, and teach stuff other than SunRPC to use it when necessary. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110821035156.ga32...@khazad-dum.debian.net
Re: Bug#638322: nfs-common: rpc.statd binds to udp port 631 preventing cups startup
On Fri, Aug 19, 2011 at 10:13:17AM +1000, Russell Coker wrote: Systems running SE Linux tend not to have this problem. In most cases the daemons which use RPC services are not permitted to bind to any of the ports that are reserved for services and therefore such a bind attempt fails with EPERM, glibc will just decrement the port number and try again when this happens. http://etbe.coker.com.au/2007/11/06/squid-and-se-linux/ I mentioned this in the above blog post, I think it was in about 2002 that I wrote the policy to do this. We could also patch bindresvport() to skip all ports mentioned in /etc/services, to get similar behaviour as with SE Linux. Or patch the programs using it to first try to bind to a static port that does not conflict with those in /etc/services, and if that fails fall back to bindresvport(). -- Met vriendelijke groet / with kind regards, Guus Sliepen g...@debian.org signature.asc Description: Digital signature
Re: Bug#638322: nfs-common: rpc.statd binds to udp port 631 preventing cups startup
On Fri, 19 Aug 2011, Guus Sliepen g...@debian.org wrote: We could also patch bindresvport() to skip all ports mentioned in /etc/services, to get similar behaviour as with SE Linux. Or patch the programs using it to first try to bind to a static port that does not conflict with those in /etc/services, and if that fails fall back to bindresvport(). That would be a viable option. On my system there are 124 TCP ports listed with numbers 1024 (which seems to be the main problem area). Losing 12% of the address space seems viable. One thing to note when comparing this to SE Linux is that the SE Linux policy labels some ports that aren't in /etc/services but which are in relatively common use. One example is port 24 for LMTP. Also with SE Linux there is an easy way of adding new port labels and as the typical daemon won't be permitted to bind to an unlabeled port the sysadmin is compelled to do the correct thing. Now one could patch bindresvport() to also check /etc/services.local or some other source of configuration information about which ports are likely to be used. But getting the users to accept that will take some effort. Of course most users just don't have enough RPC traffic to generate the problem. -- My Main Blog http://etbe.coker.com.au/ My Documents Bloghttp://doc.coker.com.au/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201108191920.49612.russ...@coker.com.au
Re: Bug#638322: nfs-common: rpc.statd binds to udp port 631 preventing cups startup
On Fri, Aug 19, 2011 at 10:49:41AM +0200, Guus Sliepen wrote: On Fri, Aug 19, 2011 at 10:13:17AM +1000, Russell Coker wrote: Systems running SE Linux tend not to have this problem. In most cases the daemons which use RPC services are not permitted to bind to any of the ports that are reserved for services and therefore such a bind attempt fails with EPERM, glibc will just decrement the port number and try again when this happens. We could also patch bindresvport() to skip all ports mentioned in /etc/services, to get similar behaviour as with SE Linux. Or patch the programs using it to first try to bind to a static port that does not conflict with those in /etc/services, and if that fails fall back to bindresvport(). Or use a whitelist rather than pretending that /etc/services was complete anywhere within the last 20 years. Not to mention bindresvport() removes the freedom of the sysadmin to bind services to whatever ports she wishes. Or, say, run multiple instances of a service. -- 1KB // Yo momma uses IPv4! -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110819094712.ga24...@angband.pl
Re: Bug#638322: nfs-common: rpc.statd binds to udp port 631 preventing cups startup
On Fri, 19 Aug 2011, Adam Borowski kilob...@angband.pl wrote: Or use a whitelist rather than pretending that /etc/services was complete anywhere within the last 20 years. AFAIK /etc/services has always been a complete list of ports assigned by IANA. If someone makes a port commonly used without getting IANA approval that's their problem/mistake. Not to mention bindresvport() removes the freedom of the sysadmin to bind services to whatever ports she wishes. Or, say, run multiple instances of a service. If you make your program use bindresvport() then it means that you don't care what the port number is as long as it's in the reserved range. This generally means that it's a RPC service and the Portmapper will tell everyone which port to use or that there is some other channel to tell the clients which port to connect to (maybe a bit like the FTP two-port setup). If you run multiple instances of a service using RPC then I guess you could use different names with the Portmapper. It seems to me that the only problem is if you run multiple instances of a daemon on different ports and don't use /etc/bindresvport.blacklist, SE Linux, or some other method of telling bindresvport() to leave your port alone. That wouldn't be an issue of sysadmin freedom but sysadmin ignorance (and I am one of the people who was ignorant of bindresvport.blacklist). -- My Main Blog http://etbe.coker.com.au/ My Documents Bloghttp://doc.coker.com.au/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110822.12738.russ...@coker.com.au
Re: Bug#638322: nfs-common: rpc.statd binds to udp port 631 preventing cups startup
On Sat, Aug 20, 2011 at 12:02:12AM +1000, Russell Coker wrote: On Fri, 19 Aug 2011, Adam Borowski kilob...@angband.pl wrote: Not to mention bindresvport() removes the freedom of the sysadmin to bind services to whatever ports she wishes. Or, say, run multiple instances of a service. If you make your program use bindresvport() then it means that you don't care what the port number is as long as it's in the reserved range. Except that it should not get into ports used by something else. It seems to me that the only problem is if you run multiple instances of a daemon on different ports and don't use /etc/bindresvport.blacklist, SE Linux, or some other method of telling bindresvport() to leave your port alone. That wouldn't be an issue of sysadmin freedom but sysadmin ignorance (and I am one of the people who was ignorant of bindresvport.blacklist). You can't blame sysadmin ignorance. I've just grepped through every single man page in Debian (ok, amd64 main), and there is not a single reference to /etc/bindresvport.blacklist. In fact, even bindresvport() is referenced only from its own manpage and from portreserve which is another hack to work around this bug. portreserve is neither recommended/suggested, nor has any data that would allow it to work. No other daemon I know has this problem. If I install daemon foo, I can expect it to not touch any ports it hasn't been configured to use. It's just portmap/SunRPC that uses random scatter-shot that can trample on something else. So what about this: let's reserve a number of ports for portmap's exclusive usage[1]. There's like 900 unused assignments, so there's plenty of space than could be parcelled off. SunRPC has long since degenerated from something with a general purpose to a peculiarity of NFS, so not many ports are needed. Only under a pathological configuration one could exceed any reasonable static limit, and in that case bindresvport() would revert to the blacklist+scattershot. [1]. Unless the sysadmin knowingly takes them for some other purpose; no different from, say, having sshd listen on port 443. -- 1KB // Yo momma uses IPv4! -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110819175346.ga26...@angband.pl
Re: Bug#638322: nfs-common: rpc.statd binds to udp port 631 preventing cups startup
Systems running SE Linux tend not to have this problem. In most cases the daemons which use RPC services are not permitted to bind to any of the ports that are reserved for services and therefore such a bind attempt fails with EPERM, glibc will just decrement the port number and try again when this happens. http://etbe.coker.com.au/2007/11/06/squid-and-se-linux/ I mentioned this in the above blog post, I think it was in about 2002 that I wrote the policy to do this. -- My Main Blog http://etbe.coker.com.au/ My Documents Bloghttp://doc.coker.com.au/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201108191013.17845.russ...@coker.com.au
