On Wed, Jun 14, 2017 at 6:42 AM, Richard Purdie <[email protected]> wrote: > Allow the creation of ipks to happen in parallel, making best use of resources > on multiprocessor systems. > > Signed-off-by: Richard Purdie <[email protected]> > --- > meta/classes/package_ipk.bbclass | 19 +++++++++++++++++-- > 1 file changed, 17 insertions(+), 2 deletions(-) > > diff --git a/meta/classes/package_ipk.bbclass > b/meta/classes/package_ipk.bbclass > index d2ce3b3..282d212 100644 > --- a/meta/classes/package_ipk.bbclass > +++ b/meta/classes/package_ipk.bbclass > @@ -17,6 +17,8 @@ OPKG_ARGS += "${@['', '--add-exclude ' + ' --add-exclude > '.join((d.getVar('PACKA > OPKGLIBDIR = "${localstatedir}/lib" > > python do_package_ipk () { > + from multiprocessing import Process > + > oldcwd = os.getcwd() > > workdir = d.getVar('WORKDIR') > @@ -37,11 +39,24 @@ python do_package_ipk () { > if os.access(os.path.join(tmpdir, "stamps", "IPK_PACKAGE_INDEX_CLEAN"), > os.R_OK): > os.unlink(os.path.join(tmpdir, "stamps", "IPK_PACKAGE_INDEX_CLEAN")) > > - for pkg in packages.split(): > - ipk_write_pkg(pkg, d) > + max_process = int(d.getVar("BB_NUMBER_THREADS") or os.cpu_count() or 1) > + launched = [] > + pkgs = packages.split() > + while pkgs: > + if len(launched) < max_process: > + p = Process(target=ipk_write_pkg, args=(pkgs.pop(), d)) > + p.start() > + launched.append(p) > + for q in launched: > + # The finished processes are joined when calling is_alive() > + if not q.is_alive(): > + launched.remove(q) > + for p in launched: > + p.join() > > os.chdir(oldcwd) > } > +do_package_ipk[vardepsexclude] = "BB_NUMBER_THREADS" > > def ipk_write_pkg(pkg, d): > import re, copy
on a 44 core system, this patch bails out like below. Once I revert this it works all fine. ERROR: westeros-wpe-image-1.0-r0 do_rootfs: Unable to install packages. Command '/mnt/a/oe/build/tmp/work/raspberrypi3-bec-linux-musleabi/westeros-wpe-image/1.0-r0/recipe-sysroot-native/usr/bin/opkg --volatile-cache -f /mnt/a/oe/build/tmp /work/raspberrypi3-bec-linux-musleabi/westeros-wpe-image/1.0-r0/opkg.conf -t /mnt/a/oe/build/tmp/work/raspberrypi3-bec-linux-musleabi/westeros-wpe-image/1.0-r0/temp/ipktemp/ -o /mnt/a/oe/build/tmp/work/raspberrypi3-bec-linux-musleabi/west eros-wpe-image/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version install 96boards-tools kernel-modules opkg packagegroup-core-boot packagegroup-core-ssh-openssh packagegroup-ml-wpe packagegroup-westeros psplash-raspberrypi run -postinsts runit runit-serialgetty' returned 255: Collected errors: * opkg_prepare_url_for_install: Couldn't find anything to satisfy '96boards-tools'. * rm_r: Failed to open dir /mnt/a/oe/build/tmp/work/raspberrypi3-bec-linux-musleabi/westeros-wpe-image/1.0-r0/temp/ipktemp//opkg-J9PI7V: No such file or directory. ERROR: westeros-wpe-image-1.0-r0 do_rootfs: Function failed: do_rootfs -- _______________________________________________ Openembedded-core mailing list [email protected] http://lists.openembedded.org/mailman/listinfo/openembedded-core
