Re: [pkg-discuss] Enabling Local Installs (on-demand depot?)
On Mon, May 12, 2008 at 8:42 PM, Bart Smaalders <[EMAIL PROTECTED]> wrote:
>
> Shawn Walker wrote:
>
> > On Mon, May 12, 2008 at 7:08 PM, Jordan Brown (Sun)
> > <[EMAIL PROTECTED]> wrote:
> >
> > > Would this process involve listening on a network or local port? In
> other
> > > words, is the depot server started a general purpose server, or is it in
> > > some fashion tied tightly to the requester (like talking via a pipe
> instead
> > > of a TCP connection)?
> > >
> >
> > For my prototype hack, it would have to be a tcp connection since
> > that's how the depot server communicates.
> >
> >
> > > Opening up a listening socket, even a local one, is security-impolite.
> > >
> >
> > This is really just a prototype, and yes, I agree it is security-impolite.
> >
> >
> > > It also sounds kind of inefficient, moving all those bytes through an
> > > "extra" process and some kind of IPC.
> > >
> >
> > Yes, but it does allow re-use of all of the existing depot code.
> >
> > I was discussing the "no on-disk package format" with Stephen and Bart
> > at Community One.
> >
> > I'm just trying to get a better feel for what some of the ideas have been.
> >
> >
>
> One of the approaches I've been musing about is to abstract the depot
> <> client communications more completely the goals would be as
> follows:
>
> 0) facilitate mirroring by allowing catalogs, manifests and files to come
> from different sources. By signing the catalogs and manifests, content
> can come from anywhere.
>
> 1) make downloaded files usable between pkg invocations; when your wireless
> craps out 450 MB into a large update, it's kind of frustrating that you
> have
> to start from the beginning. Also, installing multiple zones means a
> local
> cache is a good idea...
>
> 2) enable the use of alternate transports, such as multicast, local mirror,
> BitTorrent, DVDs, USB sticks, etc.
As part of the cherrypy depot work I've been doing, it has occurred to
me that one depot server, with many repositories is an interesting
option. As a result, I've been wanting to create depot and repository
objects and have our existing pkg.depotd just instantiate one of them.
We've already done this partially with depotcontroller.py in my
opinion.
The "napkin sketch in my head" looks something like this:
pkg.depot
pkg.depot.http (http://, https://)
pkg.depot.ftp (ftp://)
pkg.depot.smb (smb://))
pkg.depot.ssh (ssh://)
pkg.repository
pkg.repository.http
pkg.repository.file
pkg.repository.smb
pkg.repository.ssh
class depot(object):
def __init__(uri):
# uri is how the depot should be accessible (e.g. http://depot:1/)
self.repos = {}
def add_repository(self, uri, mountpoint):
# use mountpoint as name and check for duplicate
# if no duplicate found then:
# parse uri to determine type of pkg.repo object, then
self.repos[mountpoint] = pkg.repository.type(uri)
def start():
for repo in self.repos.values():
repo.start() # Allow repository to initialise, prepare itself, etc.
def stop():
for repo in self.repos.values():
repo.stop() # Allow repository to cleanup, etc.
...
class http(depot):
def search(self, mountpoint, args):
repo = self.repos[mountpoint]
results = repo.search(args)
# return results to client
...
def versions(self, mountpoint, args):
...
def catalog(self, mountpoint, args):
...
def filelist(self, mountpoint, args):
...
def manifest(self, mountpoint, args):
...
def add(self, mountpoint, args):
...
def file(self, mountpoint, args):
...
def abandon(self, mountpoint, args):
...
def close(self, mountpoint, args):
...
def open(self, mountpoint, args):
...
...
class repository(object):
def __init__(uri):
# uri is where to get the data from (e.g. http://depot:1/depot1/)
...
class http(repository):
def start():
# do initialisation
def stop():
# do cleanup
def search(args):
...
def versions(args):
...
def catalog(args):
...
def filelist(args):
...
def manifest(args):
...
def add(args):
...
def file(args):
...
def abandon(args):
...
def close(args):
...
def open(args):
...
Basic idea is that the depot server would:
depot = pkg.depot.httpd('http://localhost:10001/')
depot.add_repository('file:///var/depot1/', 'depot1')
depot.start()
The client would:
# parse uri to determine repository type then
repo = pkg.repository.type(uri)
repo.start()
repo.operation(args)
repo.stop()
Beg pardon if you find crumbles on my "napkin", I'm pretty tired now :-)
> 3) improve the asynchronous nature of pkg install by queuing download/
> uncompression and verification of received content.
Definitely. I think using the above "napkin sketch" we could make that
a reality.
--
Shawn Walker
"To err is human -- and to blame it on a computer is even more so." -
Robert Orben
___
pkg-discuss mailing list
pkg-discuss@opensolaris.org
http://mail.op
Re: [pkg-discuss] Enabling Local Installs (on-demand depot?)
On Mon, May 12, 2008 at 8:49 PM, Jordan Brown <[EMAIL PROTECTED]> wrote: > Bart Smaalders wrote: > > > 1) make downloaded files usable between pkg invocations; when your > wireless > > craps out 450 MB into a large update, it's kind of frustrating that you > have > > to start from the beginning. > > > > This is another reason for using static-file semantics whenever possible... > HTTP GET has standard mechanisms for continuing interrupted transfers. ...and one of the reasons why I'm keen on getting rid of the tar stream we do now and changing the client to do get pipelined requests. cherrypy supports pipelining. -- Shawn Walker "To err is human -- and to blame it on a computer is even more so." - Robert Orben ___ pkg-discuss mailing list pkg-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
Re: [pkg-discuss] Enabling Local Installs (on-demand depot?)
On Mon, 2008-05-12 at 18:42 -0700, Bart Smaalders wrote: > 0) facilitate mirroring by allowing catalogs, manifests and files to come > from different sources. By signing the catalogs and manifests, content > can come from anywhere. > 2) enable the use of alternate transports, such as multicast, local mirror, > BitTorrent, DVDs, USB sticks, etc. Would this allow networkless install too? E.g. would installing a network driver from a USB stick work, or do I need a working network connection to verify the signature? Laca ___ pkg-discuss mailing list pkg-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
Re: [pkg-discuss] Enabling Local Installs (on-demand depot?)
Bart Smaalders wrote: > 1) make downloaded files usable between pkg invocations; when your wireless > craps out 450 MB into a large update, it's kind of frustrating that you have > to start from the beginning. This is another reason for using static-file semantics whenever possible... HTTP GET has standard mechanisms for continuing interrupted transfers. ___ pkg-discuss mailing list pkg-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
Re: [pkg-discuss] Enabling Local Installs (on-demand depot?)
Shawn Walker wrote: > On Mon, May 12, 2008 at 7:08 PM, Jordan Brown (Sun) > <[EMAIL PROTECTED]> wrote: >> Would this process involve listening on a network or local port? In other >> words, is the depot server started a general purpose server, or is it in >> some fashion tied tightly to the requester (like talking via a pipe instead >> of a TCP connection)? > > For my prototype hack, it would have to be a tcp connection since > that's how the depot server communicates. > >> Opening up a listening socket, even a local one, is security-impolite. > > This is really just a prototype, and yes, I agree it is security-impolite. > >> It also sounds kind of inefficient, moving all those bytes through an >> "extra" process and some kind of IPC. > > Yes, but it does allow re-use of all of the existing depot code. > > I was discussing the "no on-disk package format" with Stephen and Bart > at Community One. > > I'm just trying to get a better feel for what some of the ideas have been. > One of the approaches I've been musing about is to abstract the depot <> client communications more completely the goals would be as follows: 0) facilitate mirroring by allowing catalogs, manifests and files to come from different sources. By signing the catalogs and manifests, content can come from anywhere. 1) make downloaded files usable between pkg invocations; when your wireless craps out 450 MB into a large update, it's kind of frustrating that you have to start from the beginning. Also, installing multiple zones means a local cache is a good idea... 2) enable the use of alternate transports, such as multicast, local mirror, BitTorrent, DVDs, USB sticks, etc. 3) improve the asynchronous nature of pkg install by queuing download/ uncompression and verification of received content. - Bart -- Bart Smaalders Solaris Kernel Performance [EMAIL PROTECTED] http://blogs.sun.com/barts "You will contribute more with mercurial than with thunderbird." ___ pkg-discuss mailing list pkg-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
Re: [pkg-discuss] Enabling Local Installs (on-demand depot?)
On Mon, May 12, 2008 at 7:08 PM, Jordan Brown (Sun) <[EMAIL PROTECTED]> wrote: > Would this process involve listening on a network or local port? In other > words, is the depot server started a general purpose server, or is it in > some fashion tied tightly to the requester (like talking via a pipe instead > of a TCP connection)? For my prototype hack, it would have to be a tcp connection since that's how the depot server communicates. > Opening up a listening socket, even a local one, is security-impolite. This is really just a prototype, and yes, I agree it is security-impolite. > It also sounds kind of inefficient, moving all those bytes through an > "extra" process and some kind of IPC. Yes, but it does allow re-use of all of the existing depot code. I was discussing the "no on-disk package format" with Stephen and Bart at Community One. I'm just trying to get a better feel for what some of the ideas have been. -- Shawn Walker "To err is human -- and to blame it on a computer is even more so." - Robert Orben ___ pkg-discuss mailing list pkg-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
Re: [pkg-discuss] Enabling Local Installs (on-demand depot?)
Would this process involve listening on a network or local port? In other words, is the depot server started a general purpose server, or is it in some fashion tied tightly to the requester (like talking via a pipe instead of a TCP connection)? Opening up a listening socket, even a local one, is security-impolite. It also sounds kind of inefficient, moving all those bytes through an "extra" process and some kind of IPC. Shawn Walker wrote: > Greetings, > > After working on the depot code lately, one of the things that I've > been thinking about is enabling easier local installation. > > Specifically, if a user can point us at a directory that contains a > valid repository, or at a tarball that contains a valid repository, we > could let the pkg client start a depot server child process and > perform the authority magic and everything else needed to install the > package from it. > > I know you have all been discussing this for some time already, but I > thought I would re-start this discussion as its something I'd like to > work on soon. > > I've been hacking on a prototype of this that modifies client.py by: > > * adding a global -d repo_dir parameter > > * doesn't allow it to be specified in combination with the authority > subcommands > > * Grabs the current authority for the image > > * Uses depotcontroller to start a local depot server using the value > of -d as the depot root > > * Sets the new authority to the URL of the newly started depot server > > * Proceeds with the actual subcommand they specified > > * Regardless of success resets the authority to what it was originally > > I was wondering if this was the approach that some of you were > thinking about using to deal with "repository on disk as a package". > > Cheers, ___ pkg-discuss mailing list pkg-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
