> > Then perhaps there should be an image install concept similar to flash > > archives. This could greatly improve performance. > > If such functionality is provided, it won't be by the packaging system. > > > Or maybe pkg/depo should be able to pre-compute images of defined > > packages so then instead of transferring file-by-file entire image > > would be transferred. > > It cannot; the client has to determine what is wants to retrieve since only > the > client has a picture of all of the possible packages involved in the > operation. > Remember that the server is also relatively dumb, it could actually be an > Apache process; not pkg.depotd(1M).
I know, but since you need to put into AI profile what you want to install instead of specifying individual packages one should be able to specify an image instead. Of course such an image would have to be pre-computed based on a list of packages and facets to be included. > > Something like: create a meta package called server-core and then 'pkg > > compute-image server-core' which would create a tar-like archive on > > the server. Then if a client needs a full copy of the core-server > > meta-package it could negotiate with server and transfer pre-computed > > images instead of a file-by-file for each package. For ad-hoc package > > install/upgrade it could transfer it using the current method. When an > > image is created one should be able to specify what should be included > > in it (what architectures/facets, etc.). > > This is actually what the package server used to do, but it turned out it was > the wrong answer for a number of reasons I really don't want to take the > time to get into. I guess it did it dynamically. I'm talking here about automated installs and such images should be *pre-built* in advance and then AI should unpack them if that's what specified in the configuration. Some sites each time they install an OS they pick latest version + all patches/updates at the time of installation. But then other sites they actually internally certify a specific, frozen in time internal release and then each time you install you are only allowed to install one of the internal certified releases. > With that said, long-term we'll be looking into the possibility of using pre- > generated package archives for installs, but given that with the right > configuration you already have very good performance, it hasn't been a > priority. And there's more performance work to be done yet. > > In enterprise installation it is usually more important to get quickly > > an initial OS install rather than a single package later on. And if > > Actually, I'd say the opposite. Its been my personal finding that most users > care far more about the time required for updates on already deployed > systems than initial provisioning. While I agree initial provisioning should > be > "reasonably fast", updates are actually more important. > Well, the performance is definitely acceptable for a relatively small number of packages or a minor update. Updates are actually different because of BEs - even if it takes an hour it doesn't really matter that much as it doesn't cause any downtime. Before BEs, depending on how a given company deploy updates the time it takes might or might not be critical. However they almost always caused downtime... BEs change most of it so time to apply updates is usually less critical and single package update is already acceptable. However the initial installation or actually when you need to re-build a server quite often matters as it impacts availability sometimes. > ... > > > Another thing to consider is that if you are using pkg.depotd and > > want > > better > > > scalability or performance, you could export the repository via an > > NFS > > share > > > instead, or place an Apache reverse caching proxy in front of > > pkg.depotd. > > > > Why would it make things faster? Shouldn't a single depotd be at least > > as fast as exporting the same repository over nfs or putting apache in > > front of it? > > pkg.depotd(1M) is essentially a web service. Like all web services, it > requires > tuning and considering for deployment. > > To add that, the HTTP protocol has significant overhead compared to NFS, > etc. plus the performance characteristics are totally different (the OS > provides NFS, while pkg.depotd is an application). > > As for Apache being faster, yes, that's expected with almost any web > application. Apache was designed and architected for very specific purposes, > and serving files happens to be one it's very good at. pkg.depotd(1M) on the > other hand, was primarily intended as a publication server for packaging > processes, and as an easy way to share package data when HTTP was the > most appropriate method. > > Keep in mind that pkg.depotd(1M) does provide services such as remote > search, and a browser user interface for examining package data that aren't > suitable for a static web server. > > Over time, it's become clear that pkg.depotd(1M) is not the right service to > share package data if you need the sort of large-scale provisioning typically > done with AI. As a result, it's likely that example configuration for using > Apache to serve package data will be provided (we already have an example > in the gate for caching and web proxying). > > > And to be honest - I don't have to deploy proxy servers or go to other > > means to get decent (*much* better) performance when using Linux's > > kickstart to get an OS installed over a network. It just works. > > Frankly AI+pkg should be able to easily saturate GbE on modern x86 > > hardware out-of-the-box - if they can't then they are broken. > > I'm sorry, but I'm going to have to disagree with your metrics since they > fail to > take into account all of the additional functionality that our package system > offers that they do not. > > Realistically, pkg(5) has a very different architecture that requires a > different > approach to deployment. > > Yes, the goal certainly is to have reasonable performance with little to no > configuration, but many deployments are going to require some thought. That's my point. With other platforms I can set-up network installs 10x faster in an existing environment (lets not pretend that entirely fresh deployment ara that common), additionally it would perform *considerably* faster out-of-the-box compared to AI. Again, we are talking about couple of MB/s with the default AI/pkg configuration on a GbE network where AI fetches its initial zlib archive at about 100MB/s rate - that's 40-50x difference. I agree that pkg is different and I actually really like it. Nevertheless for bigger installations AI+pkg performance is barely acceptable. Also I wouldn't underestimate how important it is to get the out-of-the-box settings relatively right. Solaris used to be known that it takes much more time to configure it to get similar performance as on Linux on the same hardware. > I had already offered you one easy option of just exporting your repository > via NFS and using that. And I'm grateful you did as instead of trying in the dark what might or might not improve performance you pointed me hopefuly in the right direction. I'm definitely going to try it. Thank you. _______________________________________________ caiman-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/caiman-discuss
