On 03/31/10 06:22 PM, Shawn Walker wrote:
On 03/31/10 11:10 AM, William Schumann wrote:
Given the package list currently defaulting for the Automated Installer,
from a local mirror here in Prague, it took almost 14 minutes to finish
the image_create(), plan_install() and to count the sizes at the cost of
186MiB all in <image directory>/var/pkg/. The reported size was 2.8GiB,
which sounds approximately correct. Ran 2 trials on a Ultra 20.
Is there any way to accelerate this process in order to inform the user
sooner about size requirements? Can we avoid the var/pkg/ downloads?
image_create()?
I'm sure it could be, there are a few open RFEs:
385 fast access to size of packages in repo
4134 pkg operations should report operation sizes
With that said, there are a few things that would be helpful to know:
* how long does a second planning operation take after the first
one has completed?
Modified test script to do a second identical planning operation with a
call to the reset() method between. (The reset() method is documented,
but not the requirement for the reset() to be done between planning
operations.) The second took about the same time as the first. The
following is the output from my test script, which is attached:
Thursday, April 1, 2010 01:12:02 PM CEST
pkglist ['entire', 'SUNWcsd', 'SUNWcs', 'babel_install']
img_dir /iscsipool/image
repo_uri http://ipkg.sfbay/dev
mirrors ['http://ipkg.czech:8000']
2010-04-01 13:12:05.426160 start
0:00:41.893355 image_create = <pkg.client.api.ImageInterface object at
0x862d30c>
0:12:14.623600 plan_install= True
0:00:00.000005 reset= None
0:12:29.152898 plan_install 2nd time= True
0:00:00.000009 describe = <pkg.client.api.PlanDescription object at
0x1c3d9f4c>
0:00:00.154494 get_changes()
0:00:47.679460 info() done
0:00:00.000931 summation: final size= 2792 MiB
done
Thursday, April 1, 2010 01:38:21 PM CEST
real 26m18.772s (2 plan_install() calls)
user 21m58.396s
sys 0m4.803s
The image_create took 42 seconds, the 1st plan_install took 12 1/4
minutes and the second plan_install took one 15 seconds longer than the
first. The info() method for the modified package list took 48 seconds.
* is this local mirror a pkg.depotd? does it have apache in front
of it?
It is a pkg.depotd without Apache in front of it.
One thing to keep in mind is that pkg.depotd doesn't currently
compress catalog or manifest data on the fly like a reverse Apache
proxy can. Using an Apache reverse proxy in front of your depot could
significantly reduce transfer time and the following RFE covers it:
9687 support for accept-encoding desired
Noted, thank you.
Also note that all of the manifests that were downloaded to perform
this operation would have been retrieved anyway to perform the actual
install.
The installing user wants to know the requirements to select an
appropriate installation disk. It is impractical for a user to initiate
an install and wait on the order of 10 minutes for this information,
downloading 150MB in the process.
A faster approximation would be more useful than an exact sum. Perhaps
a server-side calculation can be made and cached, taking advantage of
the fact that a small number of different package lists will be
statistically most common. A package depot might be primed to calculate
expected package lists periodically, or when the depot has a major update.
Thank you,
William
#!/usr/bin/python2.6
CLIENT_API_VERSION = 34
PKG_CLIENT_NAME = "install_test"
pkglist=["entire", "SUNWcsd", "SUNWcs", "babel_install"]
img_dir = "/iscsipool/image"
repo_uri = "http://ipkg.sfbay/dev";
mirrors = ["http://ipkg.czech:8000";]
print "pkglist",pkglist
print "img_dir",img_dir
print "repo_uri",repo_uri
print "mirrors",mirrors
import gettext
import locale
import pkg.client.api as api
import pkg.client.image as image
import pkg.client.progress as progress
import pkg.misc as misc
from pkg.client import global_settings
from pkg.client.api import IMG_TYPE_ENTIRE, IMG_TYPE_PARTIAL, IMG_TYPE_USER
from datetime import datetime
global_settings.client_name = PKG_CLIENT_NAME
misc.setlocale(locale.LC_ALL, "")
gettext.install("pkg", "/usr/share/locale")
progresstracker = progress.FancyUNIXProgressTracker()
#progresstracker = progress.QuietProgressTracker()
last = datetime.now()
print datetime.now(),"start"
#api_inst = api.ImageInterface(img_dir, CLIENT_API_VERSION, progresstracker, False, PKG_CLIENT_NAME)
api_inst = api.image_create(PKG_CLIENT_NAME, CLIENT_API_VERSION, img_dir, IMG_TYPE_ENTIRE, False, repo_uri=repo_uri, progtrack=progresstracker, mirrors = mirrors, force=True)
print datetime.now()-last,"image_create =",api_inst
last = datetime.now()
a = api_inst.plan_install(pkglist,refresh_catalogs=True,noexecute=False,verbose=False,update_index=True)
print datetime.now()-last,"plan_install=",a
last = datetime.now()
print datetime.now()-last,"reset=",api_inst.reset()
last = datetime.now()
a = api_inst.plan_install(pkglist,refresh_catalogs=True,noexecute=False,verbose=False,update_index=True)
print datetime.now()-last,"plan_install 2nd time=",a
last = datetime.now()
plan = api_inst.describe()
print datetime.now()-last,"describe =",plan
last = datetime.now()
new_pkg_list = [dest.fmri for src, dest in plan.get_changes()]
print datetime.now()-last,"get_changes()"
last = datetime.now()
ret = api_inst.info(new_pkg_list, False, frozenset([api.PackageInfo.SIZE]))
print datetime.now()-last,"info() done"
last = datetime.now()
pis = ret[api.ImageInterface.INFO_FOUND]
notfound = ret[api.ImageInterface.INFO_MISSING]
illegals = ret[api.ImageInterface.INFO_ILLEGALS]
multi_match = ret[api.ImageInterface.INFO_MULTI_MATCH]
if notfound:
print "notfound",notfound
if illegals:
print "illegals",illegals
if multi_match:
print "multi_match",multi_match
size = 0;
for i, pi in enumerate(pis):
#print pi.pkg_stem,"size",pi.size
size = size + pi.size
print datetime.now()-last,"summation: final size=",size/(1024*1024),"MiB"
#print "prepare"
#api_inst.prepare()
#print "execute"
#api_inst.execute_plan()
print "done"
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
