Re: [yocto] How to maintain a global DL_DIR?
On Wed, 17 Aug 2016, Gary Thomas wrote: > On 2016-08-16 18:38, Robert P. J. Day wrote: > > On Tue, 16 Aug 2016, Khem Raj wrote: > > > > > > On Aug 16, 2016, at 8:37 AM, Robert P. J. Day > > > > wrote: > > > > > > > > On Tue, 16 Aug 2016, Patrick Doyle wrote: > > > > > > > > > As I am playing with several different Yocto/OE projects, I thought > > > > > it would be nice to set up a global downloads directory. > > > > > > > > > > Also, being a somewhat forgetful sort, I thought it would be nice if > > > > > I could set DL_DIR in my .bashrc and not have to remember to edit > > > > > conf/local.conf each time I follow the directions from a third party > > > > > site to set up their Yocto based distribution. > > > > > > > > > > Can I do this? If so, how? > > > > > > > > if you're talking about creating a local source mirror for tarballs > > > > to be shared across multiple projects, i taught an OE/YP course last > > > > week based on a digi ConnectCore 6 dev kit, and i wrote some wiki > > > > pages for it, including this one: > > > > > > > > http://www.crashcourse.ca/wiki/index.php/CC6_local_source_mirror > > > > > > a small nit, perhaps its better to use rsync instead of cp. > > > > good point, i will adjust that later. anything else i overlooked? > > I use the attached script to manage my download mirror. There are > two differences with the one Robert proposed: > > * It copies all files, regardless of name. This is important if you > want your download mirror to be useful in a no-network environment > (which seems to be very important to some of my larger customers) > I normally run my builds with BB_NO_NETWORK="1" which lets me know > immediately if my download mirror needs updating. I disable BB_NO_NETWORK > to download the new images, then update the mirror and turn it back > on again. i can appreciate that ... i'm always a little irked that the bash patches are downloaded from the 'Net. > * Once uploaded to the mirror, files are replaced by symbolic links > just as if the file had been present in the first place. This has > two benefits - the downloads directory stays as small as possible > and the next time the 'save_downloads' script runs, it won't have > to bother with files already uploaded. I use this process on a > number of build machines, some with Poky/Yocto build trees that have > been around for a long time (sometimes years) and keeping all those > old files in 'downloads' can really add up. true, but given that i keep my local mirror reasonably up to date, i'm not overly concerned about leaving the occasional downloaded tarball in a project directory. but here's something i *am* still curious about. after a while, if you just keep adding to your local mirror, it gets bigger and bigger and ... after a while, will contain versions of tarballs that are utterly obsolete in that no current recipes refer to them anymore, and it's useful to just clear out the rubbish once in a while. is there a clever way to do that? is there some way to list all versions of all sopurce that are, in any way, referenced by the current set of recipes in the layers you have? after all, just because you have a newer version of a tarball in no way means that the older version(s) are unnecessary. way back when, i came up with a hacky solution: i start a new project, point it at my local mirror, then run: $ bitbake -c fetchall world what i end up with is, of course, piles of symlinks in the downloads/ directory, plus any new content as actual tarballs. at that point, if i think what i've downloaded represents a decent cleaned representation, everything in the downloads/ directory is used as the basis for a new local source mirror, and the rest is thrown away. sure, i'm likely to toss a few things that will be referenced in the future but so what, they'll be downloaded again and life will go on. or is there a more clever way to do this? rday -- Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] How to maintain a global DL_DIR?
On Wed, 17 Aug 2016, Gary Thomas wrote: > On 2016-08-16 18:38, Robert P. J. Day wrote: > > On Tue, 16 Aug 2016, Khem Raj wrote: > > > > > > On Aug 16, 2016, at 8:37 AM, Robert P. J. Day > > > > wrote: > > > > > > > > On Tue, 16 Aug 2016, Patrick Doyle wrote: > > > > > > > > > As I am playing with several different Yocto/OE projects, I thought > > > > > it would be nice to set up a global downloads directory. > > > > > > > > > > Also, being a somewhat forgetful sort, I thought it would be nice if > > > > > I could set DL_DIR in my .bashrc and not have to remember to edit > > > > > conf/local.conf each time I follow the directions from a third party > > > > > site to set up their Yocto based distribution. > > > > > > > > > > Can I do this? If so, how? > > > > > > > > if you're talking about creating a local source mirror for tarballs > > > > to be shared across multiple projects, i taught an OE/YP course last > > > > week based on a digi ConnectCore 6 dev kit, and i wrote some wiki > > > > pages for it, including this one: > > > > > > > > http://www.crashcourse.ca/wiki/index.php/CC6_local_source_mirror > > > > > > a small nit, perhaps its better to use rsync instead of cp. > > > > good point, i will adjust that later. anything else i overlooked? > > I use the attached script to manage my download mirror... well, *that* escalated quickly. :-) rday -- Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] How to maintain a global DL_DIR?
On 2016-08-16 18:38, Robert P. J. Day wrote: On Tue, 16 Aug 2016, Khem Raj wrote: On Aug 16, 2016, at 8:37 AM, Robert P. J. Day wrote: On Tue, 16 Aug 2016, Patrick Doyle wrote: As I am playing with several different Yocto/OE projects, I thought it would be nice to set up a global downloads directory. Also, being a somewhat forgetful sort, I thought it would be nice if I could set DL_DIR in my .bashrc and not have to remember to edit conf/local.conf each time I follow the directions from a third party site to set up their Yocto based distribution. Can I do this? If so, how? if you're talking about creating a local source mirror for tarballs to be shared across multiple projects, i taught an OE/YP course last week based on a digi ConnectCore 6 dev kit, and i wrote some wiki pages for it, including this one: http://www.crashcourse.ca/wiki/index.php/CC6_local_source_mirror a small nit, perhaps its better to use rsync instead of cp. good point, i will adjust that later. anything else i overlooked? I use the attached script to manage my download mirror. There are two differences with the one Robert proposed: * It copies all files, regardless of name. This is important if you want your download mirror to be useful in a no-network environment (which seems to be very important to some of my larger customers) I normally run my builds with BB_NO_NETWORK="1" which lets me know immediately if my download mirror needs updating. I disable BB_NO_NETWORK to download the new images, then update the mirror and turn it back on again. * Once uploaded to the mirror, files are replaced by symbolic links just as if the file had been present in the first place. This has two benefits - the downloads directory stays as small as possible and the next time the 'save_downloads' script runs, it won't have to bother with files already uploaded. I use this process on a number of build machines, some with Poky/Yocto build trees that have been around for a long time (sometimes years) and keeping all those old files in 'downloads' can really add up. However you manage it, using 'own-mirrors' with a local mirror really improves the speed of builds - think about having to re-download the git tree for the RaspberryPi firmware (>3.5GB) for every new build! -- Gary Thomas | Consulting for the MLB Associates |Embedded world #! /bin/bash MIRROR=/work/misc/Poky/sources cd downloads find . -maxdepth 1 -type f | sed -e 's;^\./;;' | grep -v '.lock$' | grep -v '.done$' >/tmp/files.$$ LEN=`ls -l /tmp/files.$$ | awk '{print $5}'` if [ "${LEN}" == "0" ]; then exit 0 fi rsync -auv --files-from=/tmp/files.$$ . ${MIRROR} if [ $? == 0 ]; then read -p "Upload successful - purge files(y/n)? " PURGE if [ "${PURGE}" == "y" ]; then xargs -n1 -t rm -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] How to maintain a global DL_DIR?
On Tue, Aug 16, 2016 at 4:24 PM, Peter Kjellerstedt wrote: > You seem to be using an old version of Poky. I am using the version of poky distributed with the board. In this case, the board is the Edison board, and the manufacturer is Intel. I guess I was assuming that the Edison would have been distributed with the latest version of Yocto/OE. Oh well. For other boards with other distributions, who knows what I will find? > If you do not have the option of using Krogoth or newer, you could > use this trick in your .bashrc: > > alias bitbake="BB_ENV_EXTRAWHITE=\"$BB_ENV_EXTRAWHITE DL_DIR\" bitbake" > > It may not be kosher, but it works. ;) I like that solution a whole lot more than the solution I proposed. Learning that the Yocto shipped with the Edison is old, and that there is a newer/better way to address this problem is.. priceless. Thanks for the tip. --wpd -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] How to maintain a global DL_DIR?
> -Original Message- > From: yocto-boun...@yoctoproject.org [mailto:yocto- > boun...@yoctoproject.org] On Behalf Of Patrick Doyle > Sent: den 16 augusti 2016 17:13 > To: yocto@yoctoproject.org > Subject: Re: [yocto] How to maintain a global DL_DIR? > > Oh this is ugly... > I tried setting BB_ENV_EXTRAWHITE to DL_DIR in my .bashrc, but found > that BB_ENV_EXTRAWHITE gets overwritten by oe-buildenv-internal as I > mentioned previously. You seem to be using an old version of Poky. The code in oe-buildenv-internal was changed in Krogoth to add the variables that OE requires to BB_ENV_EXTRAWHITE. So if you are using Krogoth or later it should be possible to do export BB_ENV_EXTRAWHITE="DL_DIR" in your .bashrc and get it to work as expected... If you do not have the option of using Krogoth or newer, you could use this trick in your .bashrc: alias bitbake="BB_ENV_EXTRAWHITE=\"$BB_ENV_EXTRAWHITE DL_DIR\" bitbake" It may not be kosher, but it works. ;) > I tried setting BB_ENV_WHITELIST to DL_DIR in my .bashrc and found > that bitbake just stopped working. > > After rereading the manual > (https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake- > user-manual.html#var-BB_ENV_WHITELIST) > I tried setting BB_ENV_WHITELIST to "BBPATH BB_RESERVE_ENV > BB_ENV_WHITELIST BB_ENV_EXTRAWHITE DL_DIR", and bitbake still didn't > work. > > So I looked in poky/bitbake/lib/bb/utils.py and found the following: > > def preserved_envvars_exported(): > """Variables which are taken from the environment and placed in and > exported > from the metadata""" > return [ > 'BB_TASKHASH', > 'HOME', > 'LOGNAME', > 'PATH', > 'PWD', > 'SHELL', > 'TERM', > 'USER', > ] > > def preserved_envvars(): > """Variables which are taken from the environment and placed in > the metadata""" > v = [ > 'BBPATH', > 'BB_PRESERVE_ENV', > 'BB_ENV_WHITELIST', > 'BB_ENV_EXTRAWHITE', > ] > return v + preserved_envvars_exported() > > ... > if 'BB_ENV_WHITELIST' in os.environ: > approved = os.environ['BB_ENV_WHITELIST'].split() > approved.extend(['BB_ENV_WHITELIST']) > else: > approved = preserved_envvars() > > This looks like, if BB_ENV_WHITELIST is defined in the environment, > bitbake will use it (arbitrarily appending BB_ENV_WHITELIST to the end > of whatever was defined in the environment variable). If it is not > defined, then BB_ENV_WHITELIST is initialized to be: > > ['BBPATH', 'BB_PRESERVE_ENV', 'BB_ENV_WHITELIST', 'BB_ENV_EXTRAWHITE', > 'BB_TASKHASH', 'HOME', ...] > > So I finally tried setting BB_ENV_WHITELIST in .bashrc to be "BBPATH > BB_PRESERVE_ENV BB_ENV_EXTRAWHITE BB_TASKHASH HOME LOGNAME PATH PWD > SHELL TERM USER DL_DIR" > > But it is very sadly ugly... not terribly maintainable... and makes me > think I'm missing something important. > > Please tell me I'm missing something important. > > --wpd //Peter -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] How to maintain a global DL_DIR?
On Tue, 16 Aug 2016, Khem Raj wrote: > > On Aug 16, 2016, at 8:37 AM, Robert P. J. Day wrote: > > > > On Tue, 16 Aug 2016, Patrick Doyle wrote: > > > >> As I am playing with several different Yocto/OE projects, I thought > >> it would be nice to set up a global downloads directory. > >> > >> Also, being a somewhat forgetful sort, I thought it would be nice if > >> I could set DL_DIR in my .bashrc and not have to remember to edit > >> conf/local.conf each time I follow the directions from a third party > >> site to set up their Yocto based distribution. > >> > >> Can I do this? If so, how? > > > > if you're talking about creating a local source mirror for tarballs > > to be shared across multiple projects, i taught an OE/YP course last > > week based on a digi ConnectCore 6 dev kit, and i wrote some wiki > > pages for it, including this one: > > > > http://www.crashcourse.ca/wiki/index.php/CC6_local_source_mirror > > a small nit, perhaps its better to use rsync instead of cp. good point, i will adjust that later. anything else i overlooked? rday -- Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] How to maintain a global DL_DIR?
> On Aug 16, 2016, at 8:37 AM, Robert P. J. Day wrote: > > On Tue, 16 Aug 2016, Patrick Doyle wrote: > >> As I am playing with several different Yocto/OE projects, I thought >> it would be nice to set up a global downloads directory. >> >> Also, being a somewhat forgetful sort, I thought it would be nice if >> I could set DL_DIR in my .bashrc and not have to remember to edit >> conf/local.conf each time I follow the directions from a third party >> site to set up their Yocto based distribution. >> >> Can I do this? If so, how? > > if you're talking about creating a local source mirror for tarballs > to be shared across multiple projects, i taught an OE/YP course last > week based on a digi ConnectCore 6 dev kit, and i wrote some wiki > pages for it, including this one: > > http://www.crashcourse.ca/wiki/index.php/CC6_local_source_mirror > a small nit, perhaps its better to use rsync instead of cp. > that explains how to create a globally accessible directory full of > tarballs, how to configure each new project to look there first, and > there's even a script so that, if a new project *does* have to > download a bit of stuff that's not in your local source mirror, you > can run that script from the downloads/ directory, and it will copy > all the newly-downloaded stuff into your local source mirror. > > let me know if anything on that page is unclear. > > rday > > -- > > > Robert P. J. Day Ottawa, Ontario, CANADA >http://crashcourse.ca > > Twitter: http://twitter.com/rpjday > LinkedIn: http://ca.linkedin.com/in/rpjday > > > -- > ___ > yocto mailing list > yocto@yoctoproject.org > https://lists.yoctoproject.org/listinfo/yocto signature.asc Description: Message signed with OpenPGP using GPGMail -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] How to maintain a global DL_DIR?
On Tue, 16 Aug 2016, Patrick Doyle wrote: > As I am playing with several different Yocto/OE projects, I thought > it would be nice to set up a global downloads directory. > > Also, being a somewhat forgetful sort, I thought it would be nice if > I could set DL_DIR in my .bashrc and not have to remember to edit > conf/local.conf each time I follow the directions from a third party > site to set up their Yocto based distribution. > > Can I do this? If so, how? if you're talking about creating a local source mirror for tarballs to be shared across multiple projects, i taught an OE/YP course last week based on a digi ConnectCore 6 dev kit, and i wrote some wiki pages for it, including this one: http://www.crashcourse.ca/wiki/index.php/CC6_local_source_mirror that explains how to create a globally accessible directory full of tarballs, how to configure each new project to look there first, and there's even a script so that, if a new project *does* have to download a bit of stuff that's not in your local source mirror, you can run that script from the downloads/ directory, and it will copy all the newly-downloaded stuff into your local source mirror. let me know if anything on that page is unclear. rday -- Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] How to maintain a global DL_DIR?
Oh this is ugly... I tried setting BB_ENV_EXTRAWHITE to DL_DIR in my .bashrc, but found that BB_ENV_EXTRAWHITE gets overwritten by oe-buildenv-internal as I mentioned previously. I tried setting BB_ENV_WHITELIST to DL_DIR in my .bashrc and found that bitbake just stopped working. After rereading the manual (https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#var-BB_ENV_WHITELIST) I tried setting BB_ENV_WHITELIST to "BBPATH BB_RESERVE_ENV BB_ENV_WHITELIST BB_ENV_EXTRAWHITE DL_DIR", and bitbake still didn't work. So I looked in poky/bitbake/lib/bb/utils.py and found the following: def preserved_envvars_exported(): """Variables which are taken from the environment and placed in and exported from the metadata""" return [ 'BB_TASKHASH', 'HOME', 'LOGNAME', 'PATH', 'PWD', 'SHELL', 'TERM', 'USER', ] def preserved_envvars(): """Variables which are taken from the environment and placed in the metadata""" v = [ 'BBPATH', 'BB_PRESERVE_ENV', 'BB_ENV_WHITELIST', 'BB_ENV_EXTRAWHITE', ] return v + preserved_envvars_exported() ... if 'BB_ENV_WHITELIST' in os.environ: approved = os.environ['BB_ENV_WHITELIST'].split() approved.extend(['BB_ENV_WHITELIST']) else: approved = preserved_envvars() This looks like, if BB_ENV_WHITELIST is defined in the environment, bitbake will use it (arbitrarily appending BB_ENV_WHITELIST to the end of whatever was defined in the environment variable). If it is not defined, then BB_ENV_WHITELIST is initialized to be: ['BBPATH', 'BB_PRESERVE_ENV', 'BB_ENV_WHITELIST', 'BB_ENV_EXTRAWHITE', 'BB_TASKHASH', 'HOME', ...] So I finally tried setting BB_ENV_WHITELIST in .bashrc to be "BBPATH BB_PRESERVE_ENV BB_ENV_EXTRAWHITE BB_TASKHASH HOME LOGNAME PATH PWD SHELL TERM USER DL_DIR" But it is very sadly ugly... not terribly maintainable... and makes me think I'm missing something important. Please tell me I'm missing something important. --wpd -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto