On 21/10/20 11:54 am, lesto fante wrote: > Hi, > The general idea is to make it possible to have multiple XferCommand > running in parallel. > Rather than trying to keep track of multiple XferCommand, I thought it > would be much easier to let XferCommand to fork/send request to a > daemon and die; then let pacman call a final script `XferLockCommand` > that will block until all download are completed, it will return the > classic 0 on success and -1 on error. > > After the introduction of the parallel download, it has been given an > informal greenlight to submit a patch for XferCommand, so here I am. > > As I choose simplicity, there is currently no way for pacman to know > how many downloads are happening in the background, their status, > which one did fail, just the final result success/error.
So, you are just passing the full list of files to download to a download script. Downloads are not managed by pacman at all? Just add three more lines to your script: pacman -Sy pacman -Sup --noconfirm <downloads here> pacman -Su I don't see the point of implementing parallel XferCommand like that within pacman at all. > I see 2 major slowdown to my downloads; > - small file overhead > - mirror bandwidth > > Currently I have a script that will pick all uncommented servers in > pacman list, divide them in groups of 3, and for each group download > one package. This make sure there is only one connection for server (i > assume server will not artificially limit the bandwidth, and if they > do i don't want to bypass their limit) while having multiple file in > download at the same time (good for small file overhead) and full > speed (multiple mirror for each file) > > Also from your presentation it seems like ParallelDownloads will hit > only one server; it says sync issue, not really sure what you meant > there, afaik each package is downloaded with full versioning in the > name. It currently does. In the future that may change. At the moment our download error output is not great, and servers out of sync would result in a lot of download errors. We need to add logic to catch bad servers and exclude them for future downloads, but fixing the output needs to happen first. >> So if you have an update with 150 package, every single one starts >> downloading at the same time? >> [...] >> Any implementation of this needs to respect the ParallelDownloads >> configuration option. > > in this patch it is left to the XferCommand/XferLockCommand > implementation. Also the idea is that XferLockCommand may print on > stdout the information relative to the status of the download, and > those relaid back to the user (I may be wrong but this is the current > behaviour); this way the user will not be left wondering what is going > on. > > -- Alternative 1 -- > Add to XferLockCommands argument the number of maximum parallel > downloads; if the number is reached, the command block. > so the pseudocode became > > for each file > XferCommand //start one download > XferLockCommand 10 // block if all 10 download slot are used > XferLockCommand 0 // special case, block until all download are completed > > -- Alternative 2 -- > build an array of PID, each one refer to a XferCommand, but I am not > sure how much this would be portable and if there may be issues with > PID reuse; but would give pacman a bit more control on the running > process. > Pacman currently monitors a single download in a portable way. I see no reason it could not monitor more than one. Then it could use ParallelDownloads and provided some consistency across download methods. >> Why even need this? A user either has ParallelDownloads set to be >> greater than 1, or does not. > > As far as I understand from the code in dload.c, ParallelDownloads > does not affect XferCommand, only one XferCommand is running and > expected to complete before the next is run. It does not... I'd expect it would after an addition to XferCommand to support parallel downloads.
