On Tue, Feb 17, 2015 at 06:10:37PM -0500, Nick Holland wrote: > it's using the ftp(1) FTP client, which (in OpenBSD) does a wonderful job of > fetching things via the HTTP protocol as well as the FTP protocol. > > now, he says it is blowing up after around 100 states. Sounds like his > firewall/proxy/whatever is limiting the state count per station. > Goodness knows this works very well usually, so it's something different > between his system and mine...and I'm putting my money on his firewall or > proxy.
Now, pkg_add has a whole lot of magic to limit the number of active connections to a given site down to ONE single active connection at any given time. *however* it *does* close connections abruptly, just closing the fd connected to ftp(1), and letting it die. *If* those connections are not dropped properly (having to do with the machine setup), *then* you will end up with 100s of unterminated connections... which at some point is going to overflow the machine, of course. This has nothing to do with ftp:// , which is another can of trouble entirely. Aggressive NATs tend to break ftp, as any big package will have a DATA connection active for long, and will tend to terminate the CTRL connection early, unless the NAT knows about ftp (which is why ftp proxies are a good idea, and which is why I implemented the ftp-level "keep-alive" hack, which actually sends NOP commands veeeeryyy slowly on the CTRL connection while the DATA connection is going on, to avoid this drop). Again, there CAN be an issue with closing ftp connections early, as those depend on telnet urgent signaling mechanisms, which tend to be bungled by a lot of proxies (hey, why read the RFC when we can do a FUCKED UP UNTESTED JOB OF writing TESTOSTERONE ladden shit ?) As for http, well, there was some hope in using HTTP 1.1 to not terminate the connection. Unfortunately, most http servers screwed the pooch by being vulnerable to Byte-Range attacks (yeah, no-one learnt from the TCP fragmentation attacks from 15 years ago. But you know, man, http is all shiny and new, and the new generation doesn't even care about the lower layers as long as they've got their shiny JSON and node.js, and go/rust shitz)... so direct http 1.1 usage from pkg_add never went beyond the planning stage, as most http servers out there will just terminate http 1.1 connections early in a fairly random way). TL;DR: you got to fix your network setup so you can abort partial fetches thru ftp(1) without any dangling network state remaining after the ^C. That's what's screwed in that specific situation.
