Re: how to set network io priority for a process?
On Wed, Aug 01, 2007 at 11:03:50PM -0400, Douglas Allan Tutty wrote: On Wed, Aug 01, 2007 at 09:49:41PM -0300, Henrique de Moraes Holschuh wrote: It certainly can do what you want, if you leave it running and use it as a shell and not as a single-command download tool. lftp can carry as many transfers in parallel as needed, to as many sites as needed, and bounce from one to another as needed. Pause them, queue them, stop them, etc. If you need to detach lftp from terminals and access it remotely, screen is your friend. It sounds like it will do everything I need except be persistant over reboots. I'm also going to look into curl. According to the description it is scriptable. You could write an initscript ... Regards, Andrei -- If you can't explain it simply, you don't understand it well enough. (Albert Einstein) signature.asc Description: Digital signature
Re: how to set network io priority for a process?
On Thu, Aug 02, 2007 at 06:02:34PM +0300, Andrei Popescu wrote: On Wed, Aug 01, 2007 at 11:03:50PM -0400, Douglas Allan Tutty wrote: On Wed, Aug 01, 2007 at 09:49:41PM -0300, Henrique de Moraes Holschuh wrote: It certainly can do what you want, if you leave it running and use it as a shell and not as a single-command download tool. lftp can carry as many transfers in parallel as needed, to as many sites as needed, and bounce from one to another as needed. Pause them, queue them, stop them, etc. If you need to detach lftp from terminals and access it remotely, screen is your friend. It sounds like it will do everything I need except be persistant over reboots. I'm also going to look into curl. According to the description it is scriptable. You could write an initscript ... I'm considering creating some kind of user interface to pick URLs to put into a system queue and have a queue runner start on ppp/ip_up.d and end on ip_down.d. To do properly, its a big project. Similar to writing a new print spooler. Doug. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?
On Sun, 29 Jul 2007, Douglas Allan Tutty wrote: What we need is a multi-protocol proxy server that does proper throttling of download requests. Squid delay pools? Will work for http and ftp. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?
On Sun, 29 Jul 2007, Mike Bird wrote: packets aren't lost. This doesn't work for UDP and ICMP and works poorly for varying loads. Correct. But it works wonderfully for long-lived TCP connections, and if you are using ftp/http (and not, say, bittorrent) to get your ISOs, it will help you. That said, there is a very undocumented way to *queue* incoming traffic. It used to require that hideous buggy IMQ device (but at least that one is copiously documented!), but nowadays you can do it differently (something to do with dummy devices?) I think. You might want to look that up. That allows one to do real shaping on incoming, without dropping packets. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?
On Wed, Aug 01, 2007 at 11:08:06AM -0300, Henrique de Moraes Holschuh wrote: On Sun, 29 Jul 2007, Douglas Allan Tutty wrote: What we need is a multi-protocol proxy server that does proper throttling of download requests. Squid delay pools? Will work for http and ftp. But not rsync, which I use whenever I can for large downloads due to errors creeping in for some reason over my noisy phone line and freqent line drops (and susequent redials by pppd). Such a multi-protocol proxy server doesn't need to do any caching, it just needs to put a variable delay on the outgoing ACKs that trigger the next download block on ftp and rsync protocols, variable on whether there's an active http connection. Doug. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?
On Wed, 01 Aug 2007, Douglas Allan Tutty wrote: But not rsync, which I use whenever I can for large downloads due to errors creeping in for some reason over my noisy phone line and freqent line drops (and susequent redials by pppd). Why do you allow for damaged packets at all? I used analog async ITU-T V42 modems for a *long* time (fortunately, I was able to move away before V9x hit the market). You really want an error-free channel without compression for regular Internet over PPP domestic use, and any modem still on the market should be able to deliver that to you just fine. Just configure it for error correction without compression, and have a proper error-free short and well-shielded serial connection to it if it is an external modem. Such a multi-protocol proxy server doesn't need to do any caching, it just needs to put a variable delay on the outgoing ACKs that trigger the next download block on ftp and rsync protocols, variable on whether there's an active http connection. At that point, it is easier to just configure the kernel to do what you want re. QoS, shaping and policing. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?
On Wed, Aug 01, 2007 at 01:03:52PM -0300, Henrique de Moraes Holschuh wrote: On Wed, 01 Aug 2007, Douglas Allan Tutty wrote: But not rsync, which I use whenever I can for large downloads due to errors creeping in for some reason over my noisy phone line and freqent line drops (and susequent redials by pppd). Why do you allow for damaged packets at all? I used analog async ITU-T V42 modems for a *long* time (fortunately, I was able to move away before V9x hit the market). You really want an error-free channel without compression for regular Internet over PPP domestic use, and any modem still on the market should be able to deliver that to you just fine. Just configure it for error correction without compression, and have a proper error-free short and well-shielded serial connection to it if it is an external modem. Its an external 3Com Courier modem, that's not the problem. I don't know _where_ the errors happen, but the more times the download is interrupted the more frequently there are issues somewhere in the iso. Not a problem if I can find an rsync server with the file, but a royal pain with plain ftp. I _once_ had to download three defective copies of an ISO (took a couple of weeks) and use that gnu tool that creates a single good file out of three bad ones. It has never been a problem when downloading packages via aptitude, even if I have to interrupt it. However, under sarge, I found that gftp would die out and not resume properly, corrupting the file, and that sometimes wget will also corrupt the file. For all I know, the errors are happening upstream of my ISP's modem. I'm using porchlight which gives me unlimited for 9.99 per month. This never happened when I lived outside of Parry Sound, farther away from the telephone exchange, but paid $24.95 per month. Doug. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?
On Wed, 01 Aug 2007, Douglas Allan Tutty wrote: I used analog async ITU-T V42 modems for a *long* time (fortunately, I was able to move away before V9x hit the market). You really want an error-free channel without compression for regular Internet over PPP domestic use, and any modem still on the market should be able to deliver that to you just fine. Just configure it for error correction without compression, and have a proper error-free short and well-shielded serial connection to it if it is an external modem. Its an external 3Com Courier modem, that's not the problem. I don't That's the very best async modem ever produced, AFAIK. Especially if you managed to get the latest firmware into it before 3com went to become another useless outlet for stuff designed by others. know _where_ the errors happen, but the more times the download is interrupted the more frequently there are issues somewhere in the iso. It is probably not being restarted properly, and some crap is left in the file. One way you can get rid of that problem is to truncate the file throwing away the last 1MB of it or so, for example (assuming this is not one of those multiple connections per file transfers, in which case the bogosity might be anywhere in the file). If you DO get data errors with a Courier in error-correcting mode, it means your serial cabling or serial port is bad, or that the other side is sending you bad data. And TCP protects data end-to-end against damage, so it can't be corrupted that anyway, come to think of it. It has never been a problem when downloading packages via aptitude, even if I have to interrupt it. However, under sarge, I found that gftp would die out and not resume properly, corrupting the file, and that sometimes wget will also corrupt the file. Try lftp. I know of no better ftp client. But it is command-line, which is just as well: the transfer engine is well cared for, and not a secondary thing to the GUI. For all I know, the errors are happening upstream of my ISP's modem. Not on TCP connections. If it is ftp transfers that are being damaged when the line is teared down, it is because your ftp client is broken. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?
On Wed, Aug 01, 2007 at 01:41:56PM -0300, Henrique de Moraes Holschuh wrote: Try lftp. I know of no better ftp client. But it is command-line, which is just as well: the transfer engine is well cared for, and not a secondary thing to the GUI. I've got lftp installed but haven't tried it. I will. However, I would have thought that wget would be the cat's meow since its what debootstrap uses. Perhaps they only use it because its small. Doug. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?
On Wed, 01 Aug 2007, Douglas Allan Tutty wrote: On Wed, Aug 01, 2007 at 01:41:56PM -0300, Henrique de Moraes Holschuh wrote: Try lftp. I know of no better ftp client. But it is command-line, which is just as well: the transfer engine is well cared for, and not a secondary thing to the GUI. I've got lftp installed but haven't tried it. I will. However, I would have thought that wget would be the cat's meow since its what debootstrap uses. Perhaps they only use it because its small. wget is for script use. lftp is for interactive use :-) And AFAIK curl is supposed to be better than wget for script use, but I won't go into that topic. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?
On Wed, Aug 01, 2007 at 03:33:14PM -0300, Henrique de Moraes Holschuh wrote: On Wed, 01 Aug 2007, Douglas Allan Tutty wrote: On Wed, Aug 01, 2007 at 01:41:56PM -0300, Henrique de Moraes Holschuh wrote: Try lftp. I know of no better ftp client. But it is command-line, which is just as well: the transfer engine is well cared for, and not a secondary thing to the GUI. I've got lftp installed but haven't tried it. I will. However, I would have thought that wget would be the cat's meow since its what debootstrap uses. Perhaps they only use it because its small. wget is for script use. lftp is for interactive use :-) Except that for a download that I have to restart 5 or 10 times, its easier to put the url in a file and use wget, or for rsync I put the whole command line in a file, pound-hack it, chmod +x and away it goes. If lftp had a download queue that was persistant between invocations, it would be useful. And AFAIK curl is supposed to be better than wget for script use, but I won't go into that topic. I'll look into it. Doug. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?
On Wed, 01 Aug 2007, Douglas Allan Tutty wrote: Except that for a download that I have to restart 5 or 10 times, its easier to put the url in a file and use wget, or for rsync I put the whole command line in a file, pound-hack it, chmod +x and away it goes. If lftp had a download queue that was persistant between invocations, it would be useful. I think it does, but I never used it like that. Check the manpage. It certainly can do what you want, if you leave it running and use it as a shell and not as a single-command download tool. lftp can carry as many transfers in parallel as needed, to as many sites as needed, and bounce from one to another as needed. Pause them, queue them, stop them, etc. If you need to detach lftp from terminals and access it remotely, screen is your friend. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?
On Wed, Aug 01, 2007 at 09:49:41PM -0300, Henrique de Moraes Holschuh wrote: It certainly can do what you want, if you leave it running and use it as a shell and not as a single-command download tool. lftp can carry as many transfers in parallel as needed, to as many sites as needed, and bounce from one to another as needed. Pause them, queue them, stop them, etc. If you need to detach lftp from terminals and access it remotely, screen is your friend. It sounds like it will do everything I need except be persistant over reboots. I'm also going to look into curl. Thanks, Doug. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?
On Sun, Jul 29, 2007 at 06:14:46PM -, Paul Johnson wrote: On Jul 29, 11:10 am, Douglas Allan Tutty [EMAIL PROTECTED] wrote: I'm on slow dialup. Downloads of iso's take days. Yet, I still want to be able to browse the internet. I would like to set up something like trickle that will run something but limit its bandwidth so that it lower's its priority. [...] What I want is something like ionice but for network bandwidth. I'm not sure if that's workable, but if you have your own home network, you might look into setting up your own squid cache. Squid is a web proxy that maintains it's own cache of items people on your network have already visited. If you use offline mode, you may have to force a refresh on sites you know have new content (shift-refresh in most browsers), but items you've already downloaded for viewing before will load much more quickly on subsequent visits. I already have polipo cache set up and working. Perhaps this is something else I'll have to write from scratch. I'll have to see how, at any given time, one sees what is using the bandwidth. Thanks, Doug. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?
On 7/29/07, Douglas Allan Tutty [EMAIL PROTECTED] wrote: I'm on slow dialup. Downloads of iso's take days. Yet, I still want to be able to browse the internet. I would like to set up something like trickle that will run something but limit its bandwidth so that it lower's its priority. I think the QoS or traffic shaping features of the kernel may be able to do what you want. -- Andrew Barr We matter more than pounds and pence, your economic theory makes no sense... -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?
On Sun, 29 Jul 2007 15:28:17 -0400 Andrew J. Barr [EMAIL PROTECTED] wrote: On 7/29/07, Douglas Allan Tutty [EMAIL PROTECTED] wrote: I'm on slow dialup. Downloads of iso's take days. Yet, I still want to be able to browse the internet. I would like to set up something like trickle that will run something but limit its bandwidth so that it lower's its priority. I think the QoS or traffic shaping features of the kernel may be able to do what you want. I have issues similar to Doug's, and I have also wondered whether kernel based traffic shaping is what I need. Since we both use shorewall, which has an interface to the kernel's shaping capabilities, I suppose we ought to read shorewall-doc/html/traffic_shaping.htm Andrew Barr Celejar -- mailmin.sourceforge.net - remote access via secure (OpenPGP) email ssuds.sourceforge.net - A Simple Sudoku Solver and Generator -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?
On Sun, Jul 29, 2007 at 03:32:44PM -0400, Celejar wrote: I have issues similar to Doug's, and I have also wondered whether kernel based traffic shaping is what I need. Since we both use shorewall, which has an interface to the kernel's shaping capabilities, I suppose we ought to read shorewall-doc/html/traffic_shaping.htm That's what I'm about to do. I've only recently (within the past hour or two) determined that traffic shaping is what I'm trying to do. Ideally, I would just set ftp and rync to second priority and let everything else be normal. I'll keep the list posted. Doug. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?
On Sunday 29 July 2007 13:11, Douglas Allan Tutty wrote: On Sun, Jul 29, 2007 at 03:32:44PM -0400, Celejar wrote: I have issues similar to Doug's, and I have also wondered whether kernel based traffic shaping is what I need. Since we both use shorewall, which has an interface to the kernel's shaping capabilities, I suppose we ought to read shorewall-doc/html/traffic_shaping.htm That's what I'm about to do. I've only recently (within the past hour or two) determined that traffic shaping is what I'm trying to do. This is not a solution, just a tip to guide your research. Traffic shaping usually applies to output. Policing[0] usually applies to input. Since we often can't shape on the router transmitting data to us, in such cases we instead have to police on the receiving end and rely on the sender's TCP flow control to reduce rates to the point where packets aren't lost. This doesn't work for UDP and ICMP and works poorly for varying loads. --Mike Bird [0] http://www.cisco.com/warp/public/105/policevsshape.html#policingvsshaping -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?\
On Sun, Jul 29, 2007 at 01:54:36PM -0700, Mike Bird wrote: Traffic shaping usually applies to output. Policing[0] usually applies to input. Since we often can't shape on the router transmitting data to us, in such cases we instead have to police on the receiving end and rely on the sender's TCP flow control to reduce rates to the point where packets aren't lost. This doesn't work for UDP and ICMP and works poorly for varying loads. Right, but downloads happen by blocks. Couldn't something sit somewhere on the data stream and delay the requests for the net ftp or rsync block if there's an active http transfer in progress? In other words, controll the download bandwidth by throttling the rate of upbound requests. Doug. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?
On Sun, Jul 29, 2007 at 03:32:44PM -0400, Celejar wrote: I have issues similar to Doug's, and I have also wondered whether kernel based traffic shaping is what I need. Since we both use shorewall, which has an interface to the kernel's shaping capabilities, I suppose we ought to read shorewall-doc/html/traffic_shaping.htm As expected, shorewall can't help in this regard. Doug. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?
On Sun, 29 Jul 2007 20:11:43 -0400 Douglas Allan Tutty [EMAIL PROTECTED] wrote: On Sun, Jul 29, 2007 at 03:32:44PM -0400, Celejar wrote: I have issues similar to Doug's, and I have also wondered whether kernel based traffic shaping is what I need. Since we both use shorewall, which has an interface to the kernel's shaping capabilities, I suppose we ought to read shorewall-doc/html/traffic_shaping.htm As expected, shorewall can't help in this regard. Why not? Doug. Celejar -- mailmin.sourceforge.net - remote access via secure (OpenPGP) email ssuds.sourceforge.net - A Simple Sudoku Solver and Generator -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: how to set network io priority for a process?
On Sun, Jul 29, 2007 at 10:42:29PM -0400, Celejar wrote: On Sun, 29 Jul 2007 20:11:43 -0400 Douglas Allan Tutty [EMAIL PROTECTED] wrote: On Sun, Jul 29, 2007 at 03:32:44PM -0400, Celejar wrote: I have issues similar to Doug's, and I have also wondered whether kernel based traffic shaping is what I need. Since we both use shorewall, which has an interface to the kernel's shaping capabilities, I suppose we ought to read shorewall-doc/html/traffic_shaping.htm As expected, shorewall can't help in this regard. Why not? At best, it can duplicate wondershaper. However, its designed for high-speed internet access. Its hysteresis is larger than my dialup bandwidth. The big issue is that none of the available shaping methods work by throttling outgoing requests for a chunck of incoming data. They are intended to limit the rate of outbound data. What we need is a multi-protocol proxy server that does proper throttling of download requests. Doug. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
