On 05/18/10 07:30 PM, [email protected] wrote:
On Tue, May 18, 2010 at 06:26:11PM -0500, Shawn Walker wrote:
On 05/18/10 04:33 PM, [email protected] wrote:
I have to confess that there's not much of a use case for this type of
transport access. Remote selective extraction is essentially what we
have now through the file/0 interface. In situations where we want to
reduce the transport latency, we either need to make fewer requests, or
more requests in parallel. The first part of this work will be to
multi-thread the transport. The second part is to download entire
archives if enough files have changed between the N and N+1 version of
the package.
You can allow this type of access to be possible, but I don't think
we'll actually want to enable it in the client.
I believe that the selective, remote retrieval operations that can
be performed using the .p5p archive format can avoid the unnecessary
retrieval of variant data not applicable to the current image in the
entire archive retrieval scenario you mention above.
In particular, once the byte offsets of the desired content in an
archive file are known, a *single* request can be made to the remote
server to retrieve all of the desired files in the archive. Think
of it as selective archive recreation using a remote data source.
Selective, remote retrieval from the archive file works roughly like this:
.--------------------------------------------------------.
| ustar header for package archive index file | 512 bytes
.--------------------------------------------------------.
| file data for package archive index file | X bytes
.--------------------------------------------------------.
| file1 header(s) and data | Y bytes
.--------------------------------------------------------.
| file2 header(s) and data | Z bytes
.________________________________________________________.
1) client retrieves first 512 bytes of archive
2) client verifies that first 512 bytes is the index file and
retrieves the Y bytes for it
3) client parses index data retrieved to build a list of offsets
based on files requested for package
4) client makes a *single* request naming all of the desired byte ranges
in the archive file collapsing successive byte ranges to optimise
request (e.g. if two offset ranges are successive they can be
collapsed to a single range) and appends data
5) client stores data in download cache as it is retrieved
I believe that might also be possible (in some scenarios) the
process above will provide better performance than the pipelined
request mechanism we use for /file/0/ since far less actual requests
are being made to the server.
I expect that this will perform similarly to filelist/0, which is why
I'm not too keen to use it. I'm currently running experiments that
compare multi-threaded download to a filelist download. This isn't an
exact comparison, because we generate the tar headers on the fly, but it
is pretty close. At small request sizes, and few threads, filelist is
faster than parallel pipleined file/0, but as we add more threads and
latency increases, the file case scales better.
Keep in mind that the behavior where we download an entire archive would
only be enabled for customers who have unlimited bandwidth. Those who
have metered bandwidth and high-latency connections should see a big
improvement once we parallelize the file/0 operations.
Here's a list of reasons, in no particular order, of why I don't like
the filelist and byte-range transfer approaches.
...great things snipped...
I would note that Apache's mod_cache and mod_disk_cache appear to
properly handle byte-range requests.
In the case where we're performing parallel file requests, we can
spread the I/O out amongst servers behind the load-balancer, which
helps. But any operation that gets a bunch of small files is going to
be slower than something streaming, whether that's file/0, a byte
range request, or filelist.
While I agree with the things you've pointed out to a certain extent, my
limited understanding of other protocols (such as bittorrent, etc.)
indicates they essentially completely (or can) operate on a set of byte
range requests and obviously can provide substantial aggregate transfer
performance. So with that in mind, even if we can't necessarily use
byte-range requests for http, perhaps we can use it for other protocols
such as ftp, torrent, etc. ?
I've also toyed with the idea of allowing the client to use an archive
file as an origin in a network scenario. While dynamic operations
wouldn't be possible, there's a certain attractiveness of putting up a
single giant archive file instead of extracting a few million files
somewhere.
Cheers,
-Shawn
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
