Re: [yocto] How to maintain a global DL_DIR?

2016-08-17 Thread Robert P. J. Day 
On Wed, 17 Aug 2016, Gary Thomas wrote:

> On 2016-08-16 18:38, Robert P. J. Day wrote:
> > On Tue, 16 Aug 2016, Khem Raj wrote:
> >
> > > > On Aug 16, 2016, at 8:37 AM, Robert P. J. Day 
> > > > wrote:
> > > >
> > > > On Tue, 16 Aug 2016, Patrick Doyle wrote:
> > > >
> > > > > As I am playing with several different Yocto/OE projects, I thought
> > > > > it would be nice to set up a global downloads directory.
> > > > >
> > > > > Also, being a somewhat forgetful sort, I thought it would be nice if
> > > > > I could set DL_DIR in my .bashrc and not have to remember to edit
> > > > > conf/local.conf each time I follow the directions from a third party
> > > > > site to set up their Yocto based distribution.
> > > > >
> > > > > Can I do this?  If so, how?
> > > >
> > > > if you're talking about creating a local source mirror for tarballs
> > > > to be shared across multiple projects, i taught an OE/YP course last
> > > > week based on a digi ConnectCore 6 dev kit, and i wrote some wiki
> > > > pages for it, including this one:
> > > >
> > > > http://www.crashcourse.ca/wiki/index.php/CC6_local_source_mirror
> > >
> > > a small nit, perhaps its better to use rsync instead of cp.
> >
> >   good point, i will adjust that later. anything else i overlooked?
>
> I use the attached script to manage my download mirror.  There are
> two differences with the one Robert proposed:
>
> * It copies all files, regardless of name.  This is important if you
>   want your download mirror to be useful in a no-network environment
>   (which seems to be very important to some of my larger customers)
>   I normally run my builds with BB_NO_NETWORK="1" which lets me know
>   immediately if my download mirror needs updating.  I disable BB_NO_NETWORK
>   to download the new images, then update the mirror and turn it back
>   on again.

  i can appreciate that ... i'm always a little irked that the bash
patches are downloaded from the 'Net.

> * Once uploaded to the mirror, files are replaced by symbolic links
>   just as if the file had been present in the first place.  This has
>   two benefits - the downloads directory stays as small as possible
>   and the next time the 'save_downloads' script runs, it won't have
>   to bother with files already uploaded.  I use this process on a
>   number of build machines, some with Poky/Yocto build trees that have
>   been around for a long time (sometimes years) and keeping all those
>   old files in 'downloads' can really add up.

  true, but given that i keep my local mirror reasonably up to date,
i'm not overly concerned about leaving the occasional downloaded
tarball in a project directory. but here's something i *am* still
curious about.

  after a while, if you just keep adding to your local mirror, it gets
bigger and bigger and ... after a while, will contain versions of
tarballs that are utterly obsolete in that no current recipes refer to
them anymore, and it's useful to just clear out the rubbish once in a
while.

  is there a clever way to do that? is there some way to list all
versions of all sopurce that are, in any way, referenced by the
current set of recipes in the layers you have? after all, just because
you have a newer version of a tarball in no way means that the older
version(s) are unnecessary.

  way back when, i came up with a hacky solution: i start a new
project, point it at my local mirror, then run:

  $ bitbake -c fetchall world

what i end up with is, of course, piles of symlinks in the downloads/
directory, plus any new content as actual tarballs. at that point, if
i think what i've downloaded represents a decent cleaned
representation, everything in the downloads/ directory is used as the
basis for a new local source mirror, and the rest is thrown away.

  sure, i'm likely to toss a few things that will be referenced in the
future but so what, they'll be downloaded again and life will go on.
or is there a more clever way to do this?

rday

-- 


Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca

Twitter:   http://twitter.com/rpjday
LinkedIn:   http://ca.linkedin.com/in/rpjday



-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] How to maintain a global DL_DIR?

2016-08-17 Thread Robert P. J. Day 
On Wed, 17 Aug 2016, Gary Thomas wrote:

> On 2016-08-16 18:38, Robert P. J. Day wrote:
> > On Tue, 16 Aug 2016, Khem Raj wrote:
> >
> > > > On Aug 16, 2016, at 8:37 AM, Robert P. J. Day 
> > > > wrote:
> > > >
> > > > On Tue, 16 Aug 2016, Patrick Doyle wrote:
> > > >
> > > > > As I am playing with several different Yocto/OE projects, I thought
> > > > > it would be nice to set up a global downloads directory.
> > > > >
> > > > > Also, being a somewhat forgetful sort, I thought it would be nice if
> > > > > I could set DL_DIR in my .bashrc and not have to remember to edit
> > > > > conf/local.conf each time I follow the directions from a third party
> > > > > site to set up their Yocto based distribution.
> > > > >
> > > > > Can I do this?  If so, how?
> > > >
> > > > if you're talking about creating a local source mirror for tarballs
> > > > to be shared across multiple projects, i taught an OE/YP course last
> > > > week based on a digi ConnectCore 6 dev kit, and i wrote some wiki
> > > > pages for it, including this one:
> > > >
> > > > http://www.crashcourse.ca/wiki/index.php/CC6_local_source_mirror
> > >
> > > a small nit, perhaps its better to use rsync instead of cp.
> >
> >   good point, i will adjust that later. anything else i overlooked?
>
> I use the attached script to manage my download mirror...

  well, *that* escalated quickly. :-)

rday

-- 


Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca

Twitter:   http://twitter.com/rpjday
LinkedIn:   http://ca.linkedin.com/in/rpjday


-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] How to maintain a global DL_DIR?

2016-08-16 Thread Gary Thomas 

On 2016-08-16 18:38, Robert P. J. Day wrote:

On Tue, 16 Aug 2016, Khem Raj wrote:


On Aug 16, 2016, at 8:37 AM, Robert P. J. Day  wrote:

On Tue, 16 Aug 2016, Patrick Doyle wrote:


As I am playing with several different Yocto/OE projects, I thought
it would be nice to set up a global downloads directory.

Also, being a somewhat forgetful sort, I thought it would be nice if
I could set DL_DIR in my .bashrc and not have to remember to edit
conf/local.conf each time I follow the directions from a third party
site to set up their Yocto based distribution.

Can I do this?  If so, how?


 if you're talking about creating a local source mirror for tarballs
to be shared across multiple projects, i taught an OE/YP course last
week based on a digi ConnectCore 6 dev kit, and i wrote some wiki
pages for it, including this one:

http://www.crashcourse.ca/wiki/index.php/CC6_local_source_mirror


a small nit, perhaps its better to use rsync instead of cp.


  good point, i will adjust that later. anything else i overlooked?


I use the attached script to manage my download mirror.  There are
two differences with the one Robert proposed:

* It copies all files, regardless of name.  This is important if you
  want your download mirror to be useful in a no-network environment
  (which seems to be very important to some of my larger customers)
  I normally run my builds with BB_NO_NETWORK="1" which lets me know
  immediately if my download mirror needs updating.  I disable BB_NO_NETWORK
  to download the new images, then update the mirror and turn it back
  on again.
* Once uploaded to the mirror, files are replaced by symbolic links
  just as if the file had been present in the first place.  This has
  two benefits - the downloads directory stays as small as possible
  and the next time the 'save_downloads' script runs, it won't have
  to bother with files already uploaded.  I use this process on a
  number of build machines, some with Poky/Yocto build trees that have
  been around for a long time (sometimes years) and keeping all those
  old files in 'downloads' can really add up.

However you manage it, using 'own-mirrors' with a local mirror really
improves the speed of builds - think about having to re-download the
git tree for the RaspberryPi firmware (>3.5GB) for every new build!

--

Gary Thomas |  Consulting for the
MLB Associates  |Embedded world

#! /bin/bash

MIRROR=/work/misc/Poky/sources

cd downloads
find . -maxdepth 1 -type f | sed -e 's;^\./;;' | grep -v '.lock$' | grep -v 
'.done$' >/tmp/files.$$
LEN=`ls -l /tmp/files.$$ | awk '{print $5}'`
if [ "${LEN}" == "0" ]; then
exit 0
fi
rsync -auv --files-from=/tmp/files.$$ . ${MIRROR}
if [ $? == 0 ]; then
read -p "Upload successful - purge files(y/n)? " PURGE
if [ "${PURGE}" == "y" ]; then
xargs -n1 -t rm -- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] How to maintain a global DL_DIR?

2016-08-16 Thread Patrick Doyle 
On Tue, Aug 16, 2016 at 4:24 PM, Peter Kjellerstedt
 wrote:
> You seem to be using an old version of Poky.
I am using the version of poky distributed with the board.  In this
case, the board is the Edison board, and the manufacturer is Intel.  I
guess I was assuming that the Edison would have been distributed with
the latest version of Yocto/OE.  Oh well.

For other boards with other distributions, who knows what I will find?


> If you do not have the option of using Krogoth or newer, you could
> use this trick in your .bashrc:
>
> alias bitbake="BB_ENV_EXTRAWHITE=\"$BB_ENV_EXTRAWHITE DL_DIR\" bitbake"
>
> It may not be kosher, but it works. ;)

I like that solution a whole lot more than the solution I proposed.

Learning that the Yocto shipped with the Edison is old, and that there
is a newer/better way to address this problem is.. priceless.

Thanks for the tip.

--wpd
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] How to maintain a global DL_DIR?

2016-08-16 Thread Peter Kjellerstedt 
> -Original Message-
> From: yocto-boun...@yoctoproject.org [mailto:yocto-
> boun...@yoctoproject.org] On Behalf Of Patrick Doyle
> Sent: den 16 augusti 2016 17:13
> To: yocto@yoctoproject.org
> Subject: Re: [yocto] How to maintain a global DL_DIR?
> 
> Oh this is ugly...
> I tried setting BB_ENV_EXTRAWHITE to DL_DIR in my .bashrc, but found
> that BB_ENV_EXTRAWHITE gets overwritten by oe-buildenv-internal as I
> mentioned previously.

You seem to be using an old version of Poky. The code in 
oe-buildenv-internal was changed in Krogoth to add the variables 
that OE requires to BB_ENV_EXTRAWHITE. So if you are using Krogoth 
or later it should be possible to do 
export BB_ENV_EXTRAWHITE="DL_DIR"
in your .bashrc and get it to work as expected...

If you do not have the option of using Krogoth or newer, you could 
use this trick in your .bashrc:

alias bitbake="BB_ENV_EXTRAWHITE=\"$BB_ENV_EXTRAWHITE DL_DIR\" bitbake"

It may not be kosher, but it works. ;)

> I tried setting BB_ENV_WHITELIST to DL_DIR in my .bashrc and found
> that bitbake just stopped working.
> 
> After rereading the manual
> (https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-
> user-manual.html#var-BB_ENV_WHITELIST)
> I tried setting BB_ENV_WHITELIST to "BBPATH BB_RESERVE_ENV
> BB_ENV_WHITELIST BB_ENV_EXTRAWHITE DL_DIR", and bitbake still didn't
> work.
> 
> So I looked in poky/bitbake/lib/bb/utils.py and found the following:
> 
> def preserved_envvars_exported():
> """Variables which are taken from the environment and placed in and
> exported
> from the metadata"""
> return [
> 'BB_TASKHASH',
> 'HOME',
> 'LOGNAME',
> 'PATH',
> 'PWD',
> 'SHELL',
> 'TERM',
> 'USER',
> ]
> 
> def preserved_envvars():
> """Variables which are taken from the environment and placed in
> the metadata"""
> v = [
> 'BBPATH',
> 'BB_PRESERVE_ENV',
> 'BB_ENV_WHITELIST',
> 'BB_ENV_EXTRAWHITE',
> ]
> return v + preserved_envvars_exported()
> 
> ...
> if 'BB_ENV_WHITELIST' in os.environ:
> approved = os.environ['BB_ENV_WHITELIST'].split()
> approved.extend(['BB_ENV_WHITELIST'])
> else:
> approved = preserved_envvars()
> 
> This looks like, if BB_ENV_WHITELIST is defined in the environment,
> bitbake will use it (arbitrarily appending BB_ENV_WHITELIST to the end
> of whatever was defined in the environment variable).  If it is not
> defined, then BB_ENV_WHITELIST is initialized to be:
> 
> ['BBPATH', 'BB_PRESERVE_ENV', 'BB_ENV_WHITELIST', 'BB_ENV_EXTRAWHITE',
> 'BB_TASKHASH', 'HOME', ...]
> 
> So I finally tried setting BB_ENV_WHITELIST in .bashrc to be "BBPATH
> BB_PRESERVE_ENV BB_ENV_EXTRAWHITE BB_TASKHASH HOME LOGNAME PATH PWD
> SHELL TERM USER DL_DIR"
> 
> But it is very sadly ugly... not terribly maintainable... and makes me
> think I'm missing something important.
> 
> Please tell me I'm missing something important.
> 
> --wpd

//Peter

-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] How to maintain a global DL_DIR?

2016-08-16 Thread Robert P. J. Day 
On Tue, 16 Aug 2016, Khem Raj wrote:

> > On Aug 16, 2016, at 8:37 AM, Robert P. J. Day  wrote:
> >
> > On Tue, 16 Aug 2016, Patrick Doyle wrote:
> >
> >> As I am playing with several different Yocto/OE projects, I thought
> >> it would be nice to set up a global downloads directory.
> >>
> >> Also, being a somewhat forgetful sort, I thought it would be nice if
> >> I could set DL_DIR in my .bashrc and not have to remember to edit
> >> conf/local.conf each time I follow the directions from a third party
> >> site to set up their Yocto based distribution.
> >>
> >> Can I do this?  If so, how?
> >
> >  if you're talking about creating a local source mirror for tarballs
> > to be shared across multiple projects, i taught an OE/YP course last
> > week based on a digi ConnectCore 6 dev kit, and i wrote some wiki
> > pages for it, including this one:
> >
> > http://www.crashcourse.ca/wiki/index.php/CC6_local_source_mirror
>
> a small nit, perhaps its better to use rsync instead of cp.

  good point, i will adjust that later. anything else i overlooked?

rday

-- 


Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca

Twitter:   http://twitter.com/rpjday
LinkedIn:   http://ca.linkedin.com/in/rpjday


-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] How to maintain a global DL_DIR?

2016-08-16 Thread Khem Raj 

> On Aug 16, 2016, at 8:37 AM, Robert P. J. Day  wrote:
> 
> On Tue, 16 Aug 2016, Patrick Doyle wrote:
> 
>> As I am playing with several different Yocto/OE projects, I thought
>> it would be nice to set up a global downloads directory.
>> 
>> Also, being a somewhat forgetful sort, I thought it would be nice if
>> I could set DL_DIR in my .bashrc and not have to remember to edit
>> conf/local.conf each time I follow the directions from a third party
>> site to set up their Yocto based distribution.
>> 
>> Can I do this?  If so, how?
> 
>  if you're talking about creating a local source mirror for tarballs
> to be shared across multiple projects, i taught an OE/YP course last
> week based on a digi ConnectCore 6 dev kit, and i wrote some wiki
> pages for it, including this one:
> 
> http://www.crashcourse.ca/wiki/index.php/CC6_local_source_mirror
> 

a small nit, perhaps its better to use rsync instead of cp.

> that explains how to create a globally accessible directory full of
> tarballs, how to configure each new project to look there first, and
> there's even a script so that, if a new project *does* have to
> download a bit of stuff that's not in your local source mirror, you
> can run that script from the downloads/ directory, and it will copy
> all the newly-downloaded stuff into your local source mirror.
> 
>  let me know if anything on that page is unclear.
> 
> rday
> 
> --
> 
> 
> Robert P. J. Day Ottawa, Ontario, CANADA
>http://crashcourse.ca
> 
> Twitter:   http://twitter.com/rpjday
> LinkedIn:   http://ca.linkedin.com/in/rpjday
> 
> 
> --
> ___
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto



signature.asc
Description: Message signed with OpenPGP using GPGMail
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] How to maintain a global DL_DIR?

2016-08-16 Thread Robert P. J. Day 
On Tue, 16 Aug 2016, Patrick Doyle wrote:

> As I am playing with several different Yocto/OE projects, I thought
> it would be nice to set up a global downloads directory.
>
> Also, being a somewhat forgetful sort, I thought it would be nice if
> I could set DL_DIR in my .bashrc and not have to remember to edit
> conf/local.conf each time I follow the directions from a third party
> site to set up their Yocto based distribution.
>
> Can I do this?  If so, how?

  if you're talking about creating a local source mirror for tarballs
to be shared across multiple projects, i taught an OE/YP course last
week based on a digi ConnectCore 6 dev kit, and i wrote some wiki
pages for it, including this one:

http://www.crashcourse.ca/wiki/index.php/CC6_local_source_mirror

that explains how to create a globally accessible directory full of
tarballs, how to configure each new project to look there first, and
there's even a script so that, if a new project *does* have to
download a bit of stuff that's not in your local source mirror, you
can run that script from the downloads/ directory, and it will copy
all the newly-downloaded stuff into your local source mirror.

  let me know if anything on that page is unclear.

rday

-- 


Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca

Twitter:   http://twitter.com/rpjday
LinkedIn:   http://ca.linkedin.com/in/rpjday


-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] How to maintain a global DL_DIR?

2016-08-16 Thread Patrick Doyle 
Oh this is ugly...
I tried setting BB_ENV_EXTRAWHITE to DL_DIR in my .bashrc, but found
that BB_ENV_EXTRAWHITE gets overwritten by oe-buildenv-internal as I
mentioned previously.

I tried setting BB_ENV_WHITELIST to DL_DIR in my .bashrc and found
that bitbake just stopped working.

After rereading the manual
(https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#var-BB_ENV_WHITELIST)
I tried setting BB_ENV_WHITELIST to "BBPATH BB_RESERVE_ENV
BB_ENV_WHITELIST BB_ENV_EXTRAWHITE DL_DIR", and bitbake still didn't
work.

So I looked in poky/bitbake/lib/bb/utils.py and found the following:

def preserved_envvars_exported():
"""Variables which are taken from the environment and placed in and exported
from the metadata"""
return [
'BB_TASKHASH',
'HOME',
'LOGNAME',
'PATH',
'PWD',
'SHELL',
'TERM',
'USER',
]

def preserved_envvars():
"""Variables which are taken from the environment and placed in
the metadata"""
v = [
'BBPATH',
'BB_PRESERVE_ENV',
'BB_ENV_WHITELIST',
'BB_ENV_EXTRAWHITE',
]
return v + preserved_envvars_exported()

...
if 'BB_ENV_WHITELIST' in os.environ:
approved = os.environ['BB_ENV_WHITELIST'].split()
approved.extend(['BB_ENV_WHITELIST'])
else:
approved = preserved_envvars()

This looks like, if BB_ENV_WHITELIST is defined in the environment,
bitbake will use it (arbitrarily appending BB_ENV_WHITELIST to the end
of whatever was defined in the environment variable).  If it is not
defined, then BB_ENV_WHITELIST is initialized to be:

['BBPATH', 'BB_PRESERVE_ENV', 'BB_ENV_WHITELIST', 'BB_ENV_EXTRAWHITE',
'BB_TASKHASH', 'HOME', ...]

So I finally tried setting BB_ENV_WHITELIST in .bashrc to be "BBPATH
BB_PRESERVE_ENV BB_ENV_EXTRAWHITE BB_TASKHASH HOME LOGNAME PATH PWD
SHELL TERM USER DL_DIR"

But it is very sadly ugly... not terribly maintainable... and makes me
think I'm missing something important.

Please tell me I'm missing something important.

--wpd
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[yocto] How to maintain a global DL_DIR?

2016-08-16 Thread Patrick Doyle 
As I am playing with several different Yocto/OE projects, I thought it
would be nice to set up a global downloads directory.

Also, being a somewhat forgetful sort, I thought it would be nice if I
could set DL_DIR in my .bashrc and not have to remember to edit
conf/local.conf each time I follow the directions from a third party
site to set up their Yocto based distribution.

Can I do this?  If so, how?

Do I add DL_DIR to BB_ENV_WHITELIST?  Do I add it to
BB_ENV_EXTRAWHITE?  But I note that poky/scripts/oe-buildenv-internal
arbitrarily blasts its own values into BB_ENV_EXTRAWHITE, so setting
BB_ENV_EXTRAWHITE in my .bashrc won't work.

Perhaps I am misunderstanding the concept of the whitelist.  Perhaps I
should just ask for some help.

Help.

Can I maintain a single global download directory specified solely
through environment variables?  If so, how?

Thanks.

--wpd
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto