[Fink-devel] parallel downloads...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 How hard would it be to add code to perform x number of downloads at once, where x is set in the config field? just wondering, for people who have fast connections. (would it be too hard to do for a perl beginner?) - -chris zubrzycki - - -- PGP public key: http://homepage.mac.com/beren/publickey.txt ID: 0xA2ABC070 Fingerprint: 26B0 BA6B A409 FA83 42B3 1688 FBF9 8232 A2AB C070 Twice blessed is help unlooked for. --Tolkien -BEGIN PGP SIGNATURE- Version: GnuPG v1.0.6 (Darwin) Comment: For info see http://www.gnupg.org iD8DBQE8xWUY+/mCMqKrwHARAjYmAJ4wIwTaTckP/DAhmwUwQFAGzviRJwCaAhVP DID+ht981MWR4W1B9wyyREA= =4Jko -END PGP SIGNATURE- ___ Fink-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/fink-devel
Re: [Fink-devel] parallel downloads...
At 9:43 Uhr -0400 23.04.2002, Chris Zubrzycki wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 How hard would it be to add code to perform x number of downloads at once, where x is set in the config field? just wondering, for people who have fast connections. First, you would have to do multiple process (forks). Then you have to manage those somehow. Now what do you do if one of the downloads fails - ok the process has to interact with the user. Now another fails. Doh. OK, maybe you can add a manager process which will first handle the one then the other. Now, what if the user aborts one of the downloads. Do the others continue or are they aborted also? You have to differentiate between being called as part of fink fetch-all or fink fetch-missing, and the case where you are called as part of fink build etc. Then, what if two packages try to download the same files (this does actually happen for some packages). So of course you have to handle that as well. Etc. etc. What I want to say with this is not that it's impossible, just not that trivial as you might think at the first glance. I didn't even think about this long (I just started writing the reply by writing what came to my head, so I am certain there are other issues left I didn't even think about). So you first would need to determine what it is actually you want to do (i.e. find answers for my questions above). Once you did this (the hard part), you can think about how to implement it. (would it be too hard to do for a perl beginner?) Depends on what the beginner knows? I.e. I don't view it as language problem, rather a design issue. Max -- --- Max Horn Software Developer email: mailto:[EMAIL PROTECTED] phone: (+49) 6151-494890 ___ Fink-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/fink-devel
Re: [Fink-devel] parallel downloads...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I see what you mean. on a similer note, would it be difficult to make a command- fink downloadinfo packages, or fink install --downloadinfo packages, which would go through the beginning of the install process, calculated missing dependencies, and output the correct wget/curl/insert method here/ to stdout? then it could be routed to a text file and used on a computer that has high bandwith. On Tuesday, April 23, 2002, at 11:45 AM, Max Horn wrote: At 9:43 Uhr -0400 23.04.2002, Chris Zubrzycki wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 How hard would it be to add code to perform x number of downloads at once, where x is set in the config field? just wondering, for people who have fast connections. First, you would have to do multiple process (forks). Then you have to manage those somehow. Now what do you do if one of the downloads fails - ok the process has to interact with the user. Now another fails. Doh. OK, maybe you can add a manager process which will first handle the one then the other. Now, what if the user aborts one of the downloads. Do the others continue or are they aborted also? You have to differentiate between being called as part of fink fetch-all or fink fetch-missing, and the case where you are called as part of fink build etc. Then, what if two packages try to download the same files (this does actually happen for some packages). So of course you have to handle that as well. Etc. etc. What I want to say with this is not that it's impossible, just not that trivial as you might think at the first glance. I didn't even think about this long (I just started writing the reply by writing what came to my head, so I am certain there are other issues left I didn't even think about). So you first would need to determine what it is actually you want to do (i.e. find answers for my questions above). Once you did this (the hard part), you can think about how to implement it. (would it be too hard to do for a perl beginner?) Depends on what the beginner knows? I.e. I don't view it as language problem, rather a design issue. Max -- --- Max Horn Software Developer email: mailto:[EMAIL PROTECTED] phone: (+49) 6151-494890 ___ Fink-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/fink-devel - -chris zubrzycki - - -- PGP public key: http://homepage.mac.com/beren/publickey.txt ID: 0xA2ABC070 Fingerprint: 26B0 BA6B A409 FA83 42B3 1688 FBF9 8232 A2AB C070 Security Is A Series Of Well-Defined Steps... chmod -R 0 / ; and smile :) -BEGIN PGP SIGNATURE- Version: GnuPG v1.0.6 (Darwin) Comment: For info see http://www.gnupg.org iD8DBQE8xYVw+/mCMqKrwHARAkI7AJ9xf6Eosj5IpO/FTSLY2YWnPDx99wCgiE7e KriSvk6/i+rg1Ed548vNu98= =J2XQ -END PGP SIGNATURE- ___ Fink-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/fink-devel
Re: [Fink-devel] parallel downloads...
On Tue, 23 Apr 2002, Max Horn wrote: At 9:43 Uhr -0400 23.04.2002, Chris Zubrzycki wrote: How hard would it be to add code to perform x number of downloads at once, where x is set in the config field? just wondering, for people who have fast connections. First, you would have to do multiple process (forks). Then you have to manage those somehow. Basically, re-implement a kinda backwards Apache, except instead of serving multiple parallel URLs, you're grabbing them. Max's points about the complexity of implementing this are all valid. I'll just add that, in addition to that complexity/overhead/debugging that this would involve, it's also not clear that it would save much time. Even given that the design issues are thought through properly implemented, I think the best case scenario (assuming that computational time of running all this is effectively zero we're bound instead by bandwidth) is that it takes exactly the same amount of time to download everything. Think about it: instead of four consequitive downloads that take (making up figures here) ten seconds each, you have four simultaneous downloads that take forty seconds each, because they're still sharing the same constrained bandwidth. You only stand to gain if this scheme can take advantage of different access paths (a second NIC or modem or something) or if the bottleneck is the remote server, and not your connection. Sometimes the latter is the case -- I think we all seem to be having a slow time getting downloads from Sourceforge's site, for example. But in most cases I don't think there's going to be enough gain from parallelizing to justify all the work it'll take to get it to work reliably. Too bad though. It's a cool idea, and I'd like to be proven wrong about my guesses about how the download times will work :) -- Chris Devers[EMAIL PROTECTED] Apache / mod_perl / http://homepage.mac.com/chdevers/resume/ More war soon. You know how it is.-- mnftiu.cc ___ Fink-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/fink-devel
Re: [Fink-devel] parallel downloads...
At 12:03 PM -0400 2002/04/23, Chris Devers wrote: On Tue, 23 Apr 2002, Max Horn wrote: At 9:43 Uhr -0400 23.04.2002, Chris Zubrzycki wrote: How hard would it be to add code to perform x number of downloads at once, where x is set in the config field? just wondering, for people who have fast connections. First, you would have to do multiple process (forks). Then you have to manage those somehow. Basically, re-implement a kinda backwards Apache, except instead of serving multiple parallel URLs, you're grabbing them. Max's points about the complexity of implementing this are all valid. I'll just add that, in addition to that complexity/overhead/debugging that this would involve, it's also not clear that it would save much time. Even given that the design issues are thought through properly implemented, I think the best case scenario (assuming that computational time of running all this is effectively zero we're bound instead by bandwidth) is that it takes exactly the same amount of time to download everything. That's not true. As you say below, the best case is if your local pipe can handle all downloads simultaneously, at the max speed the far-end servers can provide them. You could get 5 quick files, all simultaneous with getting a single 'slow' file. I know my download speeds are rarely the speed of my DSL line, but I can often run 4 simultaneously without visibly affecting download speed. The ideal implementation would be to use multidownload capabilities in curl or wget. You'd feed curl a bunch of URLs, and it would return when they were all fetched, or failed. Unfortunately, fink's needs are more complicated than they're likely to provide, since we'd need to provide alternates for each target. A simple alternative would be for fink to start 6 child threads, each of which starts curl download. The parent would just sleep until the children all returned, then restarted at the beginning of the download process, and discover that all or most of the packages aren't present. After the first attempt, fink could run in the current single-download try/fail/retry-alternate mode, getting human input for which alternatives to try. It's not very elegant, but this is deliberately a much simpler approach than building all the intelligence we could take advantage of, in terms of alternatives and user interaction. Think about it: instead of four consequitive downloads that take (making up figures here) ten seconds each, you have four simultaneous downloads that take forty seconds each, because they're still sharing the same constrained bandwidth. You only stand to gain if this scheme can take advantage of different access paths (a second NIC or modem or something) or if the bottleneck is the remote server, and not your connection. Sometimes the latter is the case -- I think we all seem to be having a slow time getting downloads from Sourceforge's site, for example. But in most cases I don't think there's going to be enough gain from parallelizing to justify all the work it'll take to get it to work reliably. Chris Pepper -- Chris Pepper: http://www.reppep.com/~pepper/ Rockefeller University: http://www.rockefeller.edu/ ___ Fink-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/fink-devel
Re: [Fink-devel] parallel downloads
Max Horn wrote: How hard would it be to add code to perform x number of downloads at once, where x is set in the config field? just wondering, for people who have fast connections. First, you would have to do multiple process (forks). Then you have to manage those somehow. Now what do you do if one of the downloads fails - ok the process has to interact with the user. Now another fails. Doh. OK, maybe you can add a manager process which will first handle the one then the other. Now, what if the user aborts one of the downloads. Do the others continue or are they aborted also? You have to differentiate between being called as part of fink fetch-all or fink fetch-missing, and the case where you are called as part of fink build etc. Then, what if two packages try to download the same files (this does actually happen for some packages). So of course you have to handle that as well. I believe that apt-get (or at least dselect) has code to deal with parallel downloads. At least, I noticed parallel downloads happening when I installed Debian Linux. So if you understand C/C++, it might be a good idea to see how they deal with these issues. I don't know much perl, but I could help you figure out what apt-get is doing if you're not familiar with C/C++. Toodle pip, Dave Vasilevsky ___ Fink-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/fink-devel
