On 05/18/10 04:33 PM, [email protected] wrote:
On Tue, May 18, 2010 at 04:20:12PM -0500, Shawn Walker wrote:
Realistically, I don't expect this to be used that often. For
optimal access to the archive, there should really only be one index
file (especially if you were performing remote, selective
extraction).
I have to confess that there's not much of a use case for this type of
transport access. Remote selective extraction is essentially what we
have now through the file/0 interface. In situations where we want to
reduce the transport latency, we either need to make fewer requests, or
more requests in parallel. The first part of this work will be to
multi-thread the transport. The second part is to download entire
archives if enough files have changed between the N and N+1 version of
the package.
You can allow this type of access to be possible, but I don't think
we'll actually want to enable it in the client.
I believe that the selective, remote retrieval operations that can be
performed using the .p5p archive format can avoid the unnecessary
retrieval of variant data not applicable to the current image in the
entire archive retrieval scenario you mention above.
In particular, once the byte offsets of the desired content in an
archive file are known, a *single* request can be made to the remote
server to retrieve all of the desired files in the archive. Think of it
as selective archive recreation using a remote data source.
Selective, remote retrieval from the archive file works roughly like this:
.--------------------------------------------------------.
| ustar header for package archive index file | 512 bytes
.--------------------------------------------------------.
| file data for package archive index file | X bytes
.--------------------------------------------------------.
| file1 header(s) and data | Y bytes
.--------------------------------------------------------.
| file2 header(s) and data | Z bytes
.________________________________________________________.
1) client retrieves first 512 bytes of archive
2) client verifies that first 512 bytes is the index file and
retrieves the Y bytes for it
3) client parses index data retrieved to build a list of offsets
based on files requested for package
4) client makes a *single* request naming all of the desired byte ranges
in the archive file collapsing successive byte ranges to optimise
request (e.g. if two offset ranges are successive they can be
collapsed to a single range) and appends data
5) client stores data in download cache as it is retrieved
I believe that might also be possible (in some scenarios) the process
above will provide better performance than the pipelined request
mechanism we use for /file/0/ since far less actual requests are being
made to the server.
5.4. Client and Depot Support for On-Disk Archive Format:
The pkg.server.repository module will be updated to support
the serving of a repository in readonly mode using a pkg(5)
archive file.
The pkg.client.api transport system will be updated to support
the usage of a pkg(5) archive file as an origin for package
data. When multiple origins are present for a publisher, they
will be checked in this order of preference for package data:
- local client cache
- pkg(5) archive files
- remaining origins and mirrors
I'm okay with specifying that the local client cache is accessed first;
however, I don't think it's appropriate to define the order in which any
mirrors and origins will be accessed. This should be handled by the
quality measurement system in the transport.
This is really talking about what happens with _action_cached() so,
at the moment, is actually no different from what's in the gate
(except the archive files bit).
That seems like an implementation detal that should be omitted. I don't
have a problem with specifying that we'll look in the local cache first.
However, what exactly constitutes a local cache, and the order of access of
transport endpoints is too specific for this proposal.
That's agreeable; I'll simply omit this particular bit.
Cheers,
-Shawn
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
