On Thu, 2012-05-10 at 18:13 -0500, Mark Hatle wrote: > In a pathological case, lots of files to process, the sstate_installpkg > performance was very poor. It interated over each file and ran 3 > individual sed commands per file. Changing this to keep iterating > but running only a single command took about 1/3 time time. > > However, when looking at the corresponding sstate_hardcode_path > function, it was clear we could optimize this further. > > Using the same encoding logic to specify only the minimumal sed > operation necessary, and using xargs to avoid the os.system call the > install step was able to be performed in 13% of the original time. > > Example timing numbers for perl: > > 3m7s original code > 1m20s single sed, but interating > 0m26s using xargs and limited sed > > Signed-off-by: Mark Hatle <[email protected]> > --- > meta/classes/sstate.bbclass | 28 +++++++++++++++++++++------- > 1 files changed, 21 insertions(+), 7 deletions(-) > > diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass > index a8c98e5..ad7d121 100644 > --- a/meta/classes/sstate.bbclass > +++ b/meta/classes/sstate.bbclass > @@ -174,18 +174,29 @@ def sstate_installpkg(ss, d): > bb.build.exec_func('sstate_unpack_package', d) > > # Fixup hardcoded paths > + # > + # Note: The logic below must match the reverse logic in > + # sstate_hardcode_path(d) > + > fixmefn = sstateinst + "fixmepath" > if os.path.isfile(fixmefn): > staging = d.getVar('STAGING_DIR', True) > staging_target = d.getVar('STAGING_DIR_TARGET', True) > staging_host = d.getVar('STAGING_DIR_HOST', True) > - fixmefd = open(fixmefn, "r") > - fixmefiles = fixmefd.readlines() > - fixmefd.close() > - for file in fixmefiles: > - os.system("sed -i -e s:FIXMESTAGINGDIRTARGET:%s:g %s" % > (staging_target, sstateinst + file)) > - os.system("sed -i -e s:FIXMESTAGINGDIRHOST:%s:g %s" % > (staging_host, sstateinst + file)) > - os.system("sed -i -e s:FIXMESTAGINGDIR:%s:g %s" % (staging, > sstateinst + file)) > + > + if bb.data.inherits_class('native', d) or > bb.data.inherits_class('nativesdk', d) or bb.data.inherits_class('crosssdk', > d) or bb.data.inherits_class('cross-canadian', d): > + sstate_sed_cmd = "sed -i -e 's:FIXMESTAGINGDIR:%s:g'" % > (staging) > + elif bb.data.inherits_class('cross', d): > + sstate_sed_cmd = "sed -i -e 's:FIXMESTAGINGDIRTARGET:%s:g; > s:FIXMESTAGINGDIR:%s:g'" % (staging_target, staging) > + else: > + sstate_sed_cmd = "sed -i -e 's:FIXMESTAGINGDIRHOST:%s:g'" % > (staging_host) > + > + # Add sstateinst to each filename in fixmepath, use xargs to > efficiently call sed > + sstate_hardcode_cmd = "sed -e 's:^:%s:g' %s | xargs %s" % (sstateinst, > fixmefn, sstate_sed_cmd) > + > + print "Replacing fixme paths in sstate package: %s" % > (sstate_hardcode_cmd) > + os.system(sstate_hardcode_cmd) > + > # Need to remove this or we'd copy it into the target directory and > may > # conflict with another writer > os.remove(fixmefn) > @@ -300,6 +311,9 @@ python sstate_cleanall() { > def sstate_hardcode_path(d): > # Need to remove hardcoded paths and fix these when we install the > # staging packages. > + # > + # Note: the logic in this function needs to match the reverse logic > + # in sstate_installpkg(ss, d) > > staging = d.getVar('STAGING_DIR', True) > staging_target = d.getVar('STAGING_DIR_TARGET', True)
I was thinking this looked familiar, it was the other half of the problem we originally solved: http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/sstate.bbclass?id=db94ad4cf32d8ce3e97a8287d1c89a58d008a142 :) Cheers, Richard _______________________________________________ Openembedded-core mailing list [email protected] http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
