On Tue, May 18, 2010 at 06:26:11PM -0500, Shawn Walker wrote: > On 05/18/10 04:33 PM, [email protected] wrote: > >I have to confess that there's not much of a use case for this type of > >transport access. Remote selective extraction is essentially what we > >have now through the file/0 interface. In situations where we want to > >reduce the transport latency, we either need to make fewer requests, or > >more requests in parallel. The first part of this work will be to > >multi-thread the transport. The second part is to download entire > >archives if enough files have changed between the N and N+1 version of > >the package. > > > >You can allow this type of access to be possible, but I don't think > >we'll actually want to enable it in the client. > > I believe that the selective, remote retrieval operations that can > be performed using the .p5p archive format can avoid the unnecessary > retrieval of variant data not applicable to the current image in the > entire archive retrieval scenario you mention above. > > In particular, once the byte offsets of the desired content in an > archive file are known, a *single* request can be made to the remote > server to retrieve all of the desired files in the archive. Think > of it as selective archive recreation using a remote data source. > > Selective, remote retrieval from the archive file works roughly like this: > > .--------------------------------------------------------. > | ustar header for package archive index file | 512 bytes > .--------------------------------------------------------. > | file data for package archive index file | X bytes > .--------------------------------------------------------. > | file1 header(s) and data | Y bytes > .--------------------------------------------------------. > | file2 header(s) and data | Z bytes > .________________________________________________________. > > > 1) client retrieves first 512 bytes of archive > > 2) client verifies that first 512 bytes is the index file and > retrieves the Y bytes for it > > 3) client parses index data retrieved to build a list of offsets > based on files requested for package > > 4) client makes a *single* request naming all of the desired byte ranges > in the archive file collapsing successive byte ranges to optimise > request (e.g. if two offset ranges are successive they can be > collapsed to a single range) and appends data > > 5) client stores data in download cache as it is retrieved > > I believe that might also be possible (in some scenarios) the > process above will provide better performance than the pipelined > request mechanism we use for /file/0/ since far less actual requests > are being made to the server.
I expect that this will perform similarly to filelist/0, which is why I'm not too keen to use it. I'm currently running experiments that compare multi-threaded download to a filelist download. This isn't an exact comparison, because we generate the tar headers on the fly, but it is pretty close. At small request sizes, and few threads, filelist is faster than parallel pipleined file/0, but as we add more threads and latency increases, the file case scales better. Keep in mind that the behavior where we download an entire archive would only be enabled for customers who have unlimited bandwidth. Those who have metered bandwidth and high-latency connections should see a big improvement once we parallelize the file/0 operations. Here's a list of reasons, in no particular order, of why I don't like the filelist and byte-range transfer approaches. - Caching. Filelist is a POST and isn't cacheable by design. The byte range approach might be cacheable, but according to RFC 2616, we need strong validators attached to the entity (etags) in order for caches to cope with this correctly. Without this support, the caches can't coalese various byte range requests that they've seen over time. I also don't know whether any code out there is actually smart enough to do this tody. (We'd need to check) Even so, the rules for validating byte range requests are pretty arcane. Assuming we find a cache that does this, my guess is that any problems would be hard to debug, assuming that it works to spec in the first place. - Header size. While there's no formal limit on HTTP request header sizes, in most implementations there's a limit to how much data you can cram into the headers. If we have a package with lots of tiny files, it's concievable that you could end up generating a lot of ranges if the files are non-contiguous. We hit a similar problem with POST-ing large filelist requests over SSL, due to the initial limits on buffer sizing. - Load balancing. Sending multiple GET requests through a load balancer allows the LB to spread the work across the backend servers. This means that in situations where you might be IOP-limited, distributing the work across a group of servers will help. - I/O performance. Seeking the disk is slow, so reading a bunch of byte range requests at different offsets within the file is slower than just reading the file in one go. That's the rationale for the one large archive download. If your latency is high, but you have high and unmetered bandwidth, it works best if we can stream that big file to you. The filesystem detects that we're performing a sequential read, and can get a lot of data prefetched. In the case where we're performing parallel file requests, we can spread the I/O out amongst servers behind the load-balancer, which helps. But any operation that gets a bunch of small files is going to be slower than something streaming, whether that's file/0, a byte range request, or filelist. -j _______________________________________________ pkg-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
