On Thu, Oct 1, 2015 at 9:49 AM, Gary Thomas <g...@mlbassoc.com> wrote:
> On 2015-10-01 10:38, Smith, Virgil wrote: > >> The following is roughly the procedure I follow and that works for me. >> Maybe someone could chime in with how some of this should be trimmed based >> on yocto/bitbake intent/design. >> Even so I'd probably stick with this level of extremism because without a >> known good backup of your downloads(sources) you may be incapable of >> (tweaking and) rebuilding your products if anything happens to your build >> server. >> >> The only reason I've seen that simply deleting the downloads folder >> causes problems is that external servers/content go away, violate their git >> history, or replace files with non-identical contents. >> >> >> Warning: The following does not maintain PR server information, so >> automatic upgrading of your own packages could break. If you rely on this >> work out how to extract that information (and back it up regularly). >> >> 1. rename/move your current downloads folder and create a new one. >> 2. for all of your product build configurations empty out the following >> folders >> 2.1 cache >> 2.2 state-cache >> 2.3 tmp >> 3. build (bitbake) all your product images with all appropriate >> configuration variances >> 4. run the following command to extract the unexpanded sources from >> downloads >> find -H downloads -maxdepth 1 \ >> -not -type d -and -not -name "*.done" \ >> -exec cp -L {} sources-tmp \; >> >> You now have everything you *currently* need for a sources mirror in the >> sources-tmp folder. >> >> 5. move sources-tmp to wherever/whatever backs your SOURCE_MIRROR_URL. >> 6. Check those contents into some form of revision control (even if that >> is just a manual set of backup folders/media). >> >> >> Yes this is costs time and space, you just have to decide how much your >> images and how much being able to reproduce them (with or without 'small' >> changes) is worth. >> > > I'm already doing more or less this same sequence. I use these commands to > stage the downloaded files to my mirror (/work/misc/Poky/sources for > historical reasons) > ( cd downloads; > find . -maxdepth 1 -type f | grep -v '.lock$' | grep -v '.done$' > >/tmp/files.$$; > rsync -auv --files-from=/tmp/files.$$ . /work/misc/Poky/sources > ) > > This works very well (I've been doing it for many years). The issue I'm > trying > to work on now is that my script leaves 'downloads' possibly full of > files, especially > if there are new versions that have just been downloaded. This is > especially noticeable > for the tarballs of GIT trees - there are a number that I need/use that > are measured in > gigabytes (e.g. the RaspberryPi board firmware is 4194568645 bytes as of > 2015-07-20!) > Once I've saved these to my mirror(s), I'd like to be able to purge them > from the local > download directory in my builds. As mentioned, I've found that just > wiping that in a > build tree tends to break things quite badly. Of course I can always > start over with > a new build tree, but that also defeats the purpose of incremental builds. I'd think something like this would get the job done: 1. Do a build of all your supported machines and configurations with BB_GENERATE_MIRROR_TARBALLS=1 to ensure you have current, not out of date scm tarballs. 2. Set up builds of all your supported machines and configurations, using a new DL_DIR, with PREMIRRORS pointing to the old DL_DIR. 3. Either clean up the old DL_DIR by access time before you kicked off the builds, or resolve the symlinks in the new DL_DIR and remove the old. -- Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
-- _______________________________________________ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto