On Mon, Aug 03, 2009 at 04:30:43PM -0700, Brock Pytlik wrote: > [email protected] wrote: >> [snip] >> >> Ok, but using the range header is somewhat orthogonal to keeping the >> progress tracker from rolling backwards. The progress tracker gets >> updated by the libcurl framework as the download occurs, so if a >> download fails we removed the progress accrued by that transaction. >> There's a tradeoff between having your progress updated as the download >> progresses and the previous way where we only updated the progress once >> the download finished. Now we're coupled to the framework but on a >> per-request instead of per file basis. >> > Well, except that since you're going to restart the download @ 40MB (for > example), you wouldn't (have to) rollback the progress tracker for that > 40MB just b/c the download timed out.
In the abstract this is true. However, due to the way libcurl's progress implementation and our progress implementation work together it's rather involved to take this approach. I listed two approaches that seemed feasibile in my first response to your question. In particular, the transport views most of the world as requests to URLs, with some glue thrown in to make sure actions point at the right file. The higher levels of the transport are able to figure out what files didn't come down successfully, and only retry for the failed requests. However, that layer isn't involved in progress tracking, and the framework facing bits are url/curlhandle-centric. I actually like the fact that we subtract the bytes that didn't come down successfully. It lets me see that some kind of transport error has occurred. Otherwise, we might just sit reporting nothing until we caught back up with the size we expected, or overreport the total download. IMO the ideal approach is to make the progress tracker keep track of status on a per-file basis instead of just keeping running counts like it does today. We could write functions that turn URLs back into files/actions and report progress individually, if desired. It wouldn't make much of a difference today, since downloads occur sequentially, but if we were to ever go multi-threaded, we would want a tracker that shows progress of the various download streams simultaneously. Given that most of this just works for now, I'm inclined to spend a time on areas of the code that need more help. However, if you have extra time feel free. And if not, file a rfe and I'll get to it as time permits. -j _______________________________________________ pkg-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
