Re: [OE-core] Cache unihash ... doesn't match BB_UNIHASH ...
> One is occasionally I get a lot of undeterministic metadata errors when
> BB_CACHE_POLICY = "cache", multiconfig, and hash equiv are
> enabled. The errors are all on recipes for which SRCREV = "${AUTOREV}". It
> doesn't always happen. But it did just now when I rebased
> our "zeus-modified" branch onto the upstream "zeus" branch, to get the
> changes starting with
> 7dc72fde6edeb5d6ac6b3832530998afeea67cbc.
>
> Two is that, sometimes "Initializing tasks" stage appears stuck at 44% for a
> couple minutes. I traced it down to this code in
> runqueue.py (line 1168 on zeus):
>
> # Iterate over the task list and call into the siggen code
> dealtwith = set()
> todeal = set(self.runtaskentries)
> while len(todeal) > 0:
> for tid in todeal.copy():
> if len(self.runtaskentries[tid].depends - dealtwith) == 0:
> dealtwith.add(tid)
> todeal.remove(tid)
> self.prepare_task_hash(tid)
>
> When I instrument the loop to print out the size of "todeal", I see it
> decrease very slowly, sometimes only a couple at a time. I'm
> guessing this is because prepare_task_hash is contacting the hash equiv
> server, in a serial manner here. I'm over my work VPN which
> makes things extra slow. Is there an opportunity for batching here?
>
> Batching is hard because the hashes you request might depend on hashes
> previous received from the server. It might be possible to
> figure out these dependencies and submit multiple batch requests, but this
> would require some work in the code. Another option
> would be to have some multi-level caching scheme where you can have a local
> database that mirrors your centralized server.
>
> If you have any ideas on how to make it faster, we would love to hear your
> opinion :)
Gotcha, unfortunately I don't have any ideas right now :/. Regarding the first
issue, any ideas there?
Thanks,
Chris
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Cache unihash ... doesn't match BB_UNIHASH ...
Hi Richard,
> > > Anecdotally, we are running Zeus for nightly builds with three
> > > multiconfigs. I cherry-picked your "bitbake: fix2" and "bitbake:
> > > fixup" patches and haven't seen any of the BB_UNIHASH errors since.
> > > Granted it's only been a week. But before that, hash equiv +
> > > multiconfig was unusable due to the BB_UNIHASH errors.
> >
> > That is a really helpful data point, thanks. I should probably clean up
> > those bitbake patches and get them merged then, I couldn't decide if
> > they were right or not...
> >
>
> I just picked all your pending changes out of master-next into our
> local patch queue - will let you know how it looks when it's finished
> cooking!
There are two small issues I have observed.
One is occasionally I get a lot of undeterministic metadata errors when
BB_CACHE_POLICY = "cache", multiconfig, and hash equiv are enabled. The errors
are all on recipes for which SRCREV = "${AUTOREV}". It doesn't always happen.
But it did just now when I rebased our "zeus-modified" branch onto the upstream
"zeus" branch, to get the changes starting with
7dc72fde6edeb5d6ac6b3832530998afeea67cbc.
Two is that, sometimes "Initializing tasks" stage appears stuck at 44% for a
couple minutes. I traced it down to this code in runqueue.py (line 1168 on
zeus):
# Iterate over the task list and call into the siggen code
dealtwith = set()
todeal = set(self.runtaskentries)
while len(todeal) > 0:
for tid in todeal.copy():
if len(self.runtaskentries[tid].depends - dealtwith) == 0:
dealtwith.add(tid)
todeal.remove(tid)
self.prepare_task_hash(tid)
When I instrument the loop to print out the size of "todeal", I see it decrease
very slowly, sometimes only a couple at a time. I'm guessing this is because
prepare_task_hash is contacting the hash equiv server, in a serial manner here.
I'm over my work VPN which makes things extra slow. Is there an opportunity for
batching here?
Thanks,
Chris
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Cache unihash ... doesn't match BB_UNIHASH ...
> > Anecdotally, we are running Zeus for nightly builds with three > > multiconfigs. I cherry-picked your "bitbake: fix2" and "bitbake: > > fixup" patches and haven't seen any of the BB_UNIHASH errors since. > > Granted it's only been a week. But before that, hash equiv + > > multiconfig was unusable due to the BB_UNIHASH errors. > > That is a really helpful data point, thanks. I should probably clean up > those bitbake patches and get them merged then, I couldn't decide if > they were right or not... Sure thing! I should note, I did delete my hashserv.db (on the hash equiv server) out of caution after applying those patches. Chris -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Cache unihash ... doesn't match BB_UNIHASH ...
Hi Richard, > On Fri, 2020-02-07 at 08:16 +, Alex Kiernan wrote: > > Hi > > > > We've got a couple of multiconfig builds with different > > configurations in one bitbake run (whether it needs to be like this a > > different question - I suspect we had something that worked and > > stopped there!) > > > > However, for sometime we've been seeing errors like this on master > > (the recipes vary): > > > > ERROR: mc:forensic:ca-certificates-20190110-r0 > > do_deploy_source_date_epoch: Cache unihash > > 37d2546973a2b7c16fbcb1a8dd4939e4a1bba8c3d6d2dfb6e7204ca08d437ace > > doesn't match BB_UNIHASH > > 09baf78bd9316acdd308980c42ea4ed1eacea61bcc0d840c36293d1a5a22db69 > > This is an internal sanity test. It means the hash that it had in the > task/worker context didn't match what the server thought the unihash > should be. There have been changes in this area to try and fix various > things, I'm wondering if we introduced a bug with multiconfig handling. > > There are some patches in master-next related to this area so it would > be interesting to know if that made things better/worse. Anecdotally, we are running Zeus for nightly builds with three multiconfigs. I cherry-picked your "bitbake: fix2" and "bitbake: fixup" patches and haven't seen any of the BB_UNIHASH errors since. Granted it's only been a week. But before that, hash equiv + multiconfig was unusable due to the BB_UNIHASH errors. Thanks, Chris -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Custom CLASSOVERRIDE and a "sub-sysroot" for JavaScript
> Hmm, removing the stamp-extra-info lines leaves me with errors like these: > > ERROR: mc:mc-name:library-emscripten-1.0+gitAUTOINC+ab6d30c10c-r0 > do_packagedata_setscene: The recipe library-emscripten is > trying to install files into a shared area when those files already exist. > Those files and their manifest location are: > /home/laplante/yocto/build/tmp-mc-mc-name/pkgdata/mach/library-emscripten > (matched in manifest-cortexa9t2hf-neon-library-emscripten.packagedata) > > /home/laplante/yocto/build/tmp-mc-mc-name/pkgdata/mach/runtime/library-emscripten-staticdev > (matched in manifest-cortexa9t2hf-neon-library-emscripten.packagedata) > > /home/laplante/yocto/build/tmp-mc-mc-name/pkgdata/mach/runtime/library-emscripten-dev > (matched in manifest-cortexa9t2hf-neon-library-emscripten.packagedata) > ... > Actually never mind, I had deleted just tmp and retried, but I had forgotten to delete the tmp-mc-* directories. Deleting all of them and retrying worked. Thanks again. Chris -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Custom CLASSOVERRIDE and a "sub-sysroot" for JavaScript
> > To be honest, I'm not completely sure. I based this class off of
> > native.bbclass and nativesdk.bbclass and they both do it. I have
> > little idea what stamp-extra-info actually does.
>
> My suggestion is not to do that. native and to a lesser extent
> nativesdk are "special" in ways your class (being target related)
> isn't.
>
> Basically this code is how we were able to maintain a sysroot per
> machine and pkgdata per machine.
>
> The sysroot per machine is dead with recipe specific sysroots but today
> we still have pkgdata shared per machine:
>
> meta/classes/package.bbclass:do_packagedata[stamp-extra-info] =
> "${MACHINE_ARCH}"
>
> which means the packagedata task will rerun per machine.
>
> The nativesdk stamps only need to change per SDKMACHINE target (which
> changes SDK_ARCH so we're fine there).
>
> I'll try and remember to queue a cleanup for some bits of this as there
> is a tiny bit of dead but harmless code.
>
> Its in my queue to think about making pkgdata in WORKDIR then we could
> ditch this whole thing.
Hmm, removing the stamp-extra-info lines leaves me with errors like these:
ERROR: mc:mc-name:library-emscripten-1.0+gitAUTOINC+ab6d30c10c-r0
do_packagedata_setscene: The recipe library-emscripten is trying to install
files into a shared area when those files already exist. Those files and their
manifest location are:
/home/laplante/yocto/build/tmp-mc-mc-name/pkgdata/mach/library-emscripten
(matched in manifest-cortexa9t2hf-neon-library-emscripten.packagedata)
/home/laplante/yocto/build/tmp-mc-mc-name/pkgdata/mach/runtime/library-emscripten-staticdev
(matched in manifest-cortexa9t2hf-neon-library-emscripten.packagedata)
/home/laplante/yocto/build/tmp-mc-mc-name/pkgdata/mach/runtime/library-emscripten-dev
(matched in manifest-cortexa9t2hf-neon-library-emscripten.packagedata)
...
That might be why I added the stamp-extra-info lines in the first place - as a
bandaid to fix this :/.
Chris
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Looking for a way to build latest tagged releases in recipes
> From: LAPLANTE,CHRIS (Agilent USA)
> Sent: Thursday, January 16, 2020 3:21 PM
> To: Pascal Huerst ; Richard Purdie
> ; openembedded-
> c...@lists.openembedded.org
> Cc: Rich Persaud
> Subject: RE: [OE-core] Looking for a way to build latest tagged releases in
> recipes
>
> > TODO:
> >
> > * Right now, the class triggers a base environment change every time, which
> > means BitBake always reparses every recipe. I guess
> this
> > is because I'm modifying the datastore when I get bb.event.ConfigParsed and
> > bb.event.MultiConfigParsed, in order to ensure
> > REVRECORD_DATETIME is the same across all configurations when multiconfig
> > is active. Perhaps there is a more elegant way to do
> > this that won't trigger the environment change? To be fair I think in most
> > cases you won't care, since I expect this class to mainly be
> > used in a continuous integration environment where you'll be doing a clean
> > build and having to reparse everything anyway. But I
> > could also see this class a useful to thing to always have enabled, even
> > for personal builds, and in that case obviously I'd want to fix
> > this issue.
> > * When multiconfig is active, I would also like a "global" revs.inc to be
> > generated, which is the product of aggregated the "revs.inc"
> for
> > all the multiconfigs. Still need to think about how exactly this will work
> > in the face of conflicting SRCREVs.
> > * We have a use case for JSON format data as well ("revs.json") - I'll add
> > that too.
> > * Possibly a small tinfoil script that simply automates the task of
> > INHERIT'ing this class, parsing all the recipes, and then dumping
> > revs.inc.
>
I did some thinking about the second TODO item I had and I've decide that it is
unnecessary. One revs.inc for each multiconfig is fine. To use them to
reproduce a build, you simply dump the contents of each revs.inc into the
corresponding multiconfig conf (or local.conf for "default"). Any algorithm I
try to come up with to combine them all into a single file (for local.conf) is
bound to be wrong in some way.
The third TODO is also done. The only TODO remaining is #4, a tinfoil wrapper,
but so far I don't see a need.
Chris
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Custom CLASSOVERRIDE and a "sub-sysroot" for JavaScript
Hi Richard, > > I have a few concerns/questions: > > [a] Is the basic premise of a custom CLASSOVERRIDE supported by > > Yocto? Our situation seems a bit weird, because our emscripten > > recipes are still sort of “target” recipes, so I feel a bit funny > > completely wiping CLASSOVERRIDE. > > I assume this class extension only applies for target recipes? > > If so, why not append ":class-emscripten" to CLASSOVERRIDE rather than > overwrite it? We support multiple overrides. Makes sense, I'll give it a try. > > [b] At the very least, I know I did something wrong, because > > sometimes we get weird warnings like: > > > > WARNING: app-emscripten-1.0+gitAUTOINC+1d1073e7c4-r0 do_compile: > > Manifest /home/user/yocto/build/tmp/sstate-control/manifest- > > x86_64_x86_64-nativesdk-library-emscripten.populate_sysroot not found > > in my_mach cortexa9t2hf-neon cortexa9t2hf-vfp cortexa9hf-neon > > cortexa9hf-vfp armv7at2hf-neon armv7ahf-neon armv7at2hf-vfp armv7ahf- > > vfp armv6thf-vfp armv6hf-vfp armv5tehf-vfp armv5ehf-vfp armv5thf-vfp > > armv5hf-vfp allarch x86_64_x86_64-nativesdk (variant '')? > > > > ... usually followed by a build failure in “app-emscripten” because > > “library-emscripten:do_populate_sysroot” indeed didn’t run. Related > > to [a] or perhaps the stamp-extra-info lines? > > I'd say the stamp-extra-info lines. Why do you think you want those? To be honest, I'm not completely sure. I based this class off of native.bbclass and nativesdk.bbclass and they both do it. I have little idea what stamp-extra-info actually does. > > [c] Is there a better way to do the “sub-sysroot” (/usr/js)? Would > > multilib cover this whole use case? (2) > > No, it wouldn't since multilib assumes the headers are one set with two > sets of binaries/libs. You are doing something quite similar to the > multilib class extension though. OK, so the basic strategy seems OK then? Thanks again for your expertise, Chris -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] Custom CLASSOVERRIDE and a "sub-sysroot" for JavaScript
Hi all,
For about two years now, we have been using Emscripten together with Yocto, via
an emscripten.bbclass that I wrote. In a nutshell, it works by conducting the
actual build in a Docker container, whose image is where Emscripten is actually
installed. I mount all the necessary directories (e.g. ${B}, ${S}, etc.) into
the container, and I've got some Python scripting that ensures that containers
get cleaned up even if the build is aborted with Ctrl+C or otherwise dies. All
of our Emscripten-enabled recipes happen to be using CMake, so
emscripten.bbclass is CMake specific. It's basically a reimplementation of
cmake.bbclass (from Poky) that happens to run in a container.
For the time being the specifics of how the Emscripten part works isn't
relevant (1). I provide it as background for what I really want to ask about,
which is the Yocto-specific plumbing. Here's a snippet of it:
---
# Copyright (C) 2020 Agilent Technologies, Inc.
# SPDX-License-Identifier: GPL-2.0-only
CLASSOVERRIDE = "class-emscripten"
SPECIAL_PKGSUFFIX_class-emscripten += "-emscripten"
BASEDEPENDS_class-emscripten = ""
FILES_${PN} =+ "${libdir}/*.js"
export prefix = "/usr/js"
python emscripten_virtclass_handler() {
pn = e.data.getVar("PN")
if not pn.endswith("-emscripten"):
return
classextend = e.data.getVar('BBCLASSEXTEND') or ""
if "emscripten" not in classextend:
return
def map_dependencies(varname, d, suffix = ""):
if suffix:
varname = varname + "_" + suffix
deps = d.getVar(varname)
if not deps:
return
deps = bb.utils.explode_deps(deps)
newdeps = []
for dep in deps:
if dep == pn:
continue
elif dep in ["cmake-native"]:
continue
elif dep.endswith("-emscripten"):
newdeps.append(dep)
elif dep.endswith("-native"):
bb.warn("emscripten.bbclass: Native dependency {0} will not
be usable from within Emscripten Docker container".format(dep))
newdeps.append(dep)
d.setVar(varname, " ".join(newdeps))
e.data.setVar("OVERRIDES", e.data.getVar("OVERRIDES", False) +
":virtclass-emscripten")
map_dependencies("DEPENDS", e.data)
for pkg in [e.data.getVar("PN"), "", "${PN}"]:
map_dependencies("RDEPENDS", e.data, pkg)
map_dependencies("RRECOMMENDS", e.data, pkg)
map_dependencies("RSUGGESTS", e.data, pkg)
map_dependencies("RPROVIDES", e.data, pkg)
map_dependencies("RREPLACES", e.data, pkg)
provides = e.data.getVar("PROVIDES")
nprovides = []
for prov in provides.split():
if prov.find(pn) != -1:
nprovides.append(prov)
elif not prov.endswith("-emscripten"):
nprovides.append(prov.replace(prov, prov + "-emscripten"))
else:
nprovides.append(prov)
e.data.setVar("PROVIDES", ' '.join(nprovides))
}
addhandler emscripten_virtclass_handler
emscripten_virtclass_handler[eventmask] = "bb.event.RecipePreFinalise"
do_populate_sysroot[stamp-extra-info] = ""
do_packagedata[stamp-extra-info] = ""
---
There are two main goals of this code. First, it sets up dependencies between
-emscripten recipes. Second, it sets 'prefix' to be '/usr/js/' so that, for
example, a "foo" and "foo-emscripten" recipe that both install development
headers won't collide. I call this the "sub-sysroot".
I have a few concerns/questions:
[a] Is the basic premise of a custom CLASSOVERRIDE supported by Yocto? Our
situation seems a bit weird, because our emscripten recipes are still sort of
"target" recipes, so I feel a bit funny completely wiping CLASSOVERRIDE.
[b] At the very least, I know I did something wrong, because sometimes we get
weird warnings like:
WARNING: app-emscripten-1.0+gitAUTOINC+1d1073e7c4-r0 do_compile: Manifest
/home/user/yocto/build/tmp/sstate-control/manifest-x86_64_x86_64-nativesdk-library-emscripten.populate_sysroot
not found in my_mach cortexa9t2hf-neon cortexa9t2hf-vfp cortexa9hf-neon
cortexa9hf-vfp armv7at2hf-neon armv7ahf-neon armv7at2hf-vfp armv7ahf-vfp
armv6thf-vfp armv6hf-vfp armv5tehf-vfp armv5ehf-vfp armv5thf-vfp armv5hf-vfp
allarch x86_64_x86_64-nativesdk (variant '')?
... usually followed by a build failure in "app-emscripten" because
"library-emscripten:do_populate_sysroot" indeed didn't run. Related to [a] or
perhaps the stamp-extra-info lines?
[c] Is there a better way to do the "sub-sysroot" (/usr/js)? Would multilib
cover this whole use case? (2)
I appreciate any comments/help.
Chris
(1) I'm happy to cover it in a separate thread if people are interested. I hope
to rewrite the class using Clang's WebAssembly backend, rather
Re: [OE-core] Looking for a way to build latest tagged releases in recipes
> Cross the first TODO off the list: > https://gist.github.com/mostthingsweb/39e24b329de70c0212e5bf21a431e143/8d1a5f51b3c0bf260c730ed5716d9807c78335b8 > > (BTW eventually I will move this to poky-contrib; gist is just more > convenient for me for the short term. If there's a desire for it to > happen sooner let me know.) revrecord.bbclass has been moved to the cpl/revrecord branch of poky-contrib. I have added the ability to capture the branch for git repositories. Please check it out. Thanks, Chris -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Looking for a way to build latest tagged releases in recipes
> TODO:
>
> * Right now, the class triggers a base environment change every time, which
> means BitBake always reparses every recipe. I guess this
> is because I'm modifying the datastore when I get bb.event.ConfigParsed and
> bb.event.MultiConfigParsed, in order to ensure
> REVRECORD_DATETIME is the same across all configurations when multiconfig is
> active. Perhaps there is a more elegant way to do
> this that won't trigger the environment change? To be fair I think in most
> cases you won't care, since I expect this class to mainly be
> used in a continuous integration environment where you'll be doing a clean
> build and having to reparse everything anyway. But I
> could also see this class a useful to thing to always have enabled, even for
> personal builds, and in that case obviously I'd want to fix
> this issue.
> * When multiconfig is active, I would also like a "global" revs.inc to be
> generated, which is the product of aggregated the "revs.inc" for
> all the multiconfigs. Still need to think about how exactly this will work in
> the face of conflicting SRCREVs.
> * We have a use case for JSON format data as well ("revs.json") - I'll add
> that too.
> * Possibly a small tinfoil script that simply automates the task of
> INHERIT'ing this class, parsing all the recipes, and then dumping
> revs.inc.
Cross the first TODO off the list:
https://gist.github.com/mostthingsweb/39e24b329de70c0212e5bf21a431e143/8d1a5f51b3c0bf260c730ed5716d9807c78335b8
(BTW eventually I will move this to poky-contrib; gist is just more convenient
for me for the short term. If there's a desire for it to happen sooner let me
know.)
Chris
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Looking for a way to build latest tagged releases in recipes
> > As mentioned in IRC, as I get deeper into it, I'll be going with approach
> > #3 after all :/. Hope to post a tinfoil script soon.
>
> That would be very much appreciated!
Here's a prototype bbclass:
https://gist.github.com/mostthingsweb/39e24b329de70c0212e5bf21a431e143. Please
give it a try.
All you need to do is INHERIT it in local.conf:
INHERIT += "revrecord"
Then you can just do "bitbake --parse-only" and during parsing it will create a
directory structure in ${TOPDIR}/revs that looks something like this:
revs
├── 20200116170750
│ ├── recipe1
│ │ └── revs.inc
│ ├── recipe2
│ │ └── revs.inc
│ ├── recipe3
│ │ └── revs.inc
│ └── revs.inc
└── latest -> 20200116170750
It also supports multiconfig. When multiconfig is enabled, there will be an
additional layer of directories in the 'revs' directory, one per multiconfig
(plus "default"):
revs
├── 20200116170750
│ ├── default
│ │ ├── recipe1
│ │ │ └── revs.inc
│ │ ├── recipe2
│ │ │ └── revs.inc
│ │ └── revs.inc
│ └── mc1
│ ├── recipe1
│ │ └── revs.inc
│ ├── recipe2
│ │ └── revs.inc
│ └── revs.inc
└── latest -> 20200116170750
TODO:
* Right now, the class triggers a base environment change every time, which
means BitBake always reparses every recipe. I guess this is because I'm
modifying the datastore when I get bb.event.ConfigParsed and
bb.event.MultiConfigParsed, in order to ensure REVRECORD_DATETIME is the same
across all configurations when multiconfig is active. Perhaps there is a more
elegant way to do this that won't trigger the environment change? To be fair I
think in most cases you won't care, since I expect this class to mainly be used
in a continuous integration environment where you'll be doing a clean build and
having to reparse everything anyway. But I could also see this class a useful
to thing to always have enabled, even for personal builds, and in that case
obviously I'd want to fix this issue.
* When multiconfig is active, I would also like a "global" revs.inc to be
generated, which is the product of aggregated the "revs.inc" for all the
multiconfigs. Still need to think about how exactly this will work in the face
of conflicting SRCREVs.
* We have a use case for JSON format data as well ("revs.json") - I'll add that
too.
* Possibly a small tinfoil script that simply automates the task of INHERIT'ing
this class, parsing all the recipes, and then dumping revs.inc.
Thanks,
Chris
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Looking for a way to build latest tagged releases in recipes
> I've gone back and forth and re-implemented the code to do so a half dozen
> times. Chronologically, these are the fundamental
> strategies I've attempted:
>
> 1. Simply in terms of buildhistory-collect-srcrevs.
> Pros: dead simple
> Cons: requires that you actually build stuff first; requires
> buildhistory to be enabled
> 2. As a bbclass, injecting a task between do_fetch and do_unpack
> ("do_capture_srcrevs")
> Pros: can be invoked without actually building stuff, e.g. bitbake
> my-image --runall=do_capture_srcrevs
> Cons: adds yet another task to the recipe; a little ugly since you
> either need to INHERIT the class, or individually inherit it in
> recipes
> 3. As a separate tinfoil script (as you suggest) which I run before the
> actual bitbake.
> Pros: decoupled from the actual build
> Cons: since it's decoupled, it's slower - each recipe needs to be parsed
> 4. Hacking the srcrev cache (inside fetch2) to log all information necessary
> to generate the SRCREV overrides, paired with a script that
> reads the cache and does it
> Pros: Equally suitable for the "config-then-build" workflow ("bitbake
> --parse-only && generate_overrides.py >> local.conf &&
> bitbake image") or the "build-then-config" workflow ("bitbake image &&
> generate_overrides.py > overrides.conf")
> Cons: Invasive, required large changes to fetch2/__init__.py and some
> smaller changes to fetch2/git.py
>
> In the current incarnation (#4), I also go a step further and, in cases where
> a SRC_URI has a parameterized branch, I generate BRANCH
> override lines as well. For example, if I see:
As mentioned in IRC, as I get deeper into it, I'll be going with approach #3
after all :/. Hope to post a tinfoil script soon.
Chris
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Looking for a way to build latest tagged releases in recipes
Hi Richard,
> FWIW I do want to see the system used to do things like this. I've kind
> of envisaged you'd do it "up front" though. For example it should be
> trivial for a script to use tinfoil, iterate through and generate an
> include file of the revisions you want, using the fetcher calls.
>
> Yes, its slightly more annoying to generate the config, then build it
> but if it stops the core becoming unmaintainable with corner cases,
> that could be the right decision.
>
> If you do something like that it would be great to share it so others
> can see what you did and be able to build off it...
This is actually something I've been working on and thinking about a lot for
the past year and a half. Our application layer has probably 30 or so recipes
that use AUTOREV. For the purposes of reproducible builds, we need to generate
similar output to what buildhistory-collect-srcrevs generates, e.g.
SRCREV_pn-my_daemon = "githash"
I've gone back and forth and re-implemented the code to do so a half dozen
times. Chronologically, these are the fundamental strategies I've attempted:
1. Simply in terms of buildhistory-collect-srcrevs.
Pros: dead simple
Cons: requires that you actually build stuff first; requires
buildhistory to be enabled
2. As a bbclass, injecting a task between do_fetch and do_unpack
("do_capture_srcrevs")
Pros: can be invoked without actually building stuff, e.g. bitbake
my-image --runall=do_capture_srcrevs
Cons: adds yet another task to the recipe; a little ugly since you
either need to INHERIT the class, or individually inherit it in recipes
3. As a separate tinfoil script (as you suggest) which I run before the actual
bitbake.
Pros: decoupled from the actual build
Cons: since it's decoupled, it's slower - each recipe needs to be parsed
4. Hacking the srcrev cache (inside fetch2) to log all information necessary to
generate the SRCREV overrides, paired with a script that reads the cache and
does it
Pros: Equally suitable for the "config-then-build" workflow ("bitbake
--parse-only && generate_overrides.py >> local.conf && bitbake image") or the
"build-then-config" workflow ("bitbake image && generate_overrides.py >
overrides.conf")
Cons: Invasive, required large changes to fetch2/__init__.py and some
smaller changes to fetch2/git.py
In the current incarnation (#4), I also go a step further and, in cases where a
SRC_URI has a parameterized branch, I generate BRANCH override lines as well.
For example, if I see:
MY_BRANCH ??= "master"
SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1;
branch=${MY_BRANCH}"
I generate this line along with the corresponding SRCREV override:
MY_BRANCH_pn-recipe = "master"
The algorithm for detecting the branch variable in the SRC_URI entry is as a
little complicated, and not super relevant at this moment, but I want to get it
off my chest :). The code first tries to find a parameterized branch by using a
regex to find stuff like "branch=${VAR}". If that fails, it could be because
the branch is expressed as something more complicated like this:
BRANCH_RELEASE_NUMBER ??= "1.0"
SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1;
branch=release-${BRANCH_RELEASE_NUMBER}"
In that case, I fall back on a breadth-first search on all the variables that
SRC_URI references (recursively) to find the "deepest" one that controls the
branch. I wrote this in terms of d.expandWithRefs, and can detail it further if
people are interested. Basically, I didn't want to enforce a particular
convention on the name of the branch variable. You'd think BRANCH would cover
it, but the kernel recipes also have KMETA and stuff like that.
I realize the branch stuff as currently implemented is maybe too opinionated
(and a bit dangerous) for inclusion in bitbake. But I digress.
I'm very interested in discussing the SRCREV (and perhaps BRANCH) stuff further
with anyone interested. I would love to see some strategy for doing this make
its way into bitbake or poky, so I can stop reimplementing it and second
guessing myself :).
Thanks,
Chris
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [bitbake-devel] hashequiv, pseudo, and GIDs
> > On the contrary. For almost all practical use cases you care about what > > user/group owns a file, not what UID/GID they happen to be. E.g., in the > > recipe (or Makefile or whatever) you typically want to specify that a > > file shall be owned by user foo, not UID 1234. Then, either you do not > > care about the actual UIDs used, or if you do, you use statically assigned > > UIDs/GIDs for all users/groups. This is typically necessary if you want > > to be able to support upgrading your product, and it is very well > > supported by the useradd-staticids.bbclass. > > Interesting, thanks for the background. Out of curiosity, what is the > difference between using useradd-staticids.bbclass vs. using > normal useradd.bbclass but hardcoding the GIDs/UIDs anyway? I ask because > meta-skeleton/recipes-skeleton/useradd/useradd- > example.bb does the latter. Actually never mind, sorry. I see how it works in https://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#var-USERADDEXTENSION. Chris -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [bitbake-devel] hashequiv, pseudo, and GIDs
> > From a caching point of view, it seems incorrect to care about what the > > corresponding group name happened to be at the time when the cache was > > created. > > On the contrary. For almost all practical use cases you care about what > user/group owns a file, not what UID/GID they happen to be. E.g., in the > recipe (or Makefile or whatever) you typically want to specify that a > file shall be owned by user foo, not UID 1234. Then, either you do not > care about the actual UIDs used, or if you do, you use statically assigned > UIDs/GIDs for all users/groups. This is typically necessary if you want > to be able to support upgrading your product, and it is very well > supported by the useradd-staticids.bbclass. Interesting, thanks for the background. Out of curiosity, what is the difference between using useradd-staticids.bbclass vs. using normal useradd.bbclass but hardcoding the GIDs/UIDs anyway? I ask because meta-skeleton/recipes-skeleton/useradd/useradd-example.bb does the latter. Chris -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [bitbake-devel] hashequiv, pseudo, and GIDs
> > So why is the group name used in the hash, as opposed to the GID? (I also > > have the same question regarding user name vs UID). If > the GID was used we wouldn't have this issue. > > That was done in response to this mailing list thread: > http://lists.openembedded.org/pipermail/openembedded-core/2019-January/277755.html > > Basically, the GID and UID aren't stable values across builds so the > suggestion was to use the names. My understanding was that this > corresponded to how the user and groups are dealt with in general. That doesn't make sense (intuitively) to me. If I have a file with owner GID 1000, and copy it to some system, then the group name is defined by whatever groups happen to exist on the system. If I copy it to a different system, the owner group name might be different, depending on /etc/group. In both cases, it is the GID of 1000 that is stable. >From a caching point of view, it seems incorrect to care about what the >corresponding group name happened to be at the time when the cache was >created. Chris -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [bitbake-devel] hashequiv, pseudo, and GIDs
> > A workaround is to add: > > DEPENDS += "B" > > to recipe A, so that the group ends up in A's pseudo database. This > > works, but seems really really hacky, since it's only necessary when > > hashequiv is active. > > I suspect this isn't a hack, but necessary and you're finding a > determinism/missing dependency issue that hashequiv is highlighting? > > > So why is the group name used in the hash, as opposed to the GID? (I > > also have the same question regarding user name vs UID). If the GID > > was used we wouldn't have this issue. > > The only way I can think this might work is that you're statically > assigning GIDs and using the GID directly in do_install? That isn't the > way the defaults are setup. > > If that is the case we probably have to detect it and change the > behaviour of hashequiv's comparison function to use UID/GID but it does > depend on the wider system setup. Yes, you and Josh are correct. The recipe was doing "chgrp 3004". It definitely makes sense for me to add the dep on B, and change it "chgrp actual-group-name". But like you said I think it does expose a problem in hashequiv. Chris -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] hashequiv, pseudo, and GIDs
Hello all,
I'm encountering a failure like below when using hashequiv. What is happening
is recipe "A" is installing some files in do_install, and changing the group of
those files. The group is defined by a different recipe ("B"). There is no
dependency between A and B, so when A:do_package runs, the group is not present
in A's pseudo database. OEOuthashBasic tries to lookup the owning group name of
the file, and a KeyError is raised.
A workaround is to add:
DEPENDS += "B"
to recipe A, so that the group ends up in A's pseudo database. This works, but
seems really really hacky, since it's only necessary when hashequiv is active.
So why is the group name used in the hash, as opposed to the GID? (I also have
the same question regarding user name vs UID). If the GID was used we wouldn't
have this issue.
Thanks,
Chris
__
ERROR: A-1.0-r0 do_package: Error executing a python function in
exec_python_func() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function:
0001:
*** 0002:sstate_report_unihash(d)
0003:
File: '/home/laplante/yocto/sources/poky/meta/classes/sstate.bbclass', lineno:
802, function: sstate_report_unihash
0798:report_unihash = getattr(bb.parse.siggen, 'report_unihash', None)
0799:
0800:if report_unihash:
0801:ss = sstate_state_fromvars(d)
*** 0802:report_unihash(os.getcwd(), ss['task'], d)
0803:}
0804:
0805:#
0806:# Shell function to decompress and prepare a package for installation
File: '/home/laplante/yocto/sources/poky/bitbake/lib/bb/siggen.py', lineno:
485, function: report_unihash
0481:
0482:if "." in self.method:
0483:(module, method) = self.method.rsplit('.', 1)
0484:locs['method'] =
getattr(importlib.import_module(module), method)
*** 0485:outhash = bb.utils.better_eval('method(path, sigfile,
task, d)', locs)
0486:else:
0487:outhash = bb.utils.better_eval(self.method + '(path,
sigfile, task, d)', locs)
0488:
0489:try:
File: '/home/laplante/yocto/sources/poky/bitbake/lib/bb/utils.py', lineno: 421,
function: better_eval
0417:if extraglobals:
0418:ctx = copy.copy(ctx)
0419:for g in extraglobals:
0420:ctx[g] = extraglobals[g]
*** 0421:return eval(source, ctx, locals)
0422:
0423:@contextmanager
0424:def fileslocked(files):
0425:"""Context manager for locking and unlocking file locks."""
File: '', lineno: 1, function:
File "", line 1, in
File: '/home/laplante/yocto/sources/poky/meta/lib/oe/sstatesig.py', lineno:
553, function: OEOuthashBasic
0549:process(root)
0550:for f in files:
0551:if f == 'fixmepath':
0552:continue
*** 0553:process(os.path.join(root, f))
0554:finally:
0555:os.chdir(prev_dir)
0556:
0557:return h.hexdigest()
File: '/home/laplante/yocto/sources/poky/meta/lib/oe/sstatesig.py', lineno:
516, function: process
0512:add_perm(stat.S_IXOTH, 'x')
0513:
0514:if include_owners:
0515:update_hash(" %10s" %
pwd.getpwuid(s.st_uid).pw_name)
*** 0516:update_hash(" %10s" %
grp.getgrgid(s.st_gid).gr_name)
0517:
0518:update_hash(" ")
0519:if stat.S_ISBLK(s.st_mode) or stat.S_ISCHR(s.st_mode):
0520:update_hash("%9s" % ("%d.%d" %
(os.major(s.st_rdev), os.minor(s.st_rdev
Exception: KeyError: 'getgrgid(): gid not found: 3004'
__
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [bitbake-devel] Keeping multiconfig config files in sync
> From: Richard Purdie > Sent: Thursday, September 26, 2019 4:53 PM > > This has bugged me since we first created multiconfig. It was always > intended we'd have configs you could reference. It turned out to be > hard to code and we (well, I) decided to "come back to it". That hasn't > happened yet. > > I'd welcome proposals on how it could work (its harder than it first > appears). It'd be good to check we do have a bug open for it too. Hmm, I just tried moving my multiconfig directory from build/conf to the conf directory of an arbitrary layer, and it worked. I added 3 multiconfigs, each with a different MACHINE set. Then I added an anonymous Python func that just bb.warn'd out the machine. When I ran in parse-only mode, it printed out 4 warnings (including one for the default config I guess), with the machine different each time. The BB manual says "The reason for this required file hierarchy is because the BBPATH variable is not constructed until the layers are parsed. Consequently, using the configuration file as a pre-configuration file is not possible unless it is located in the current working directory." You actually fixed this limitation in 2017 when you reworked the server API (poky commits 62bd2e36 and 4602408c). So, happily, I think this is just a case of documentation that needs updating :) Chris -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] Keeping multiconfig config files in sync
Hello, Thanks to Richard and others recent hard work, multiconfig is poised to become much more practical in YP 3.0. One thing I'm not clear on, however, is how it will work in a team environment. If I have recipes with multiconfig dependencies, then I must ensure that anyone else using those recipes has the exact same set of multiconfig config files defined in their build/conf/multiconfig/ directory. Or at least, compatible multiconfigs with matching names. Since these files are under build/conf/, it's not immediately clear how one might version control these. Unless I'm mistaken and there is a better way, as a workaround I will probably put the multiconfigs in a layer and instruct developers to symlink build/conf/multiconfig to there. Any insight? Thanks, Chris -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [bitbake-devel] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)
> > I think this supports my point about being more interested in patches
> > allowing people to extend/customise buildhistory than just adding X.
> >
> > Whilst we want to have good defaults, there are always going to be
> > niche cases for people wanting to extend it...
>
>
> Agreed. Then we can implement our BRANCH scheme without polluting the core
> code with it.
Richard, et al. -
I have created the cpl/buildhistory branch on poky-contrib to share a draft of
my ideas for making buildhistory more modular and extensible. Would appreciate
any comments.
The first change was to break up buildhistory.bbclass. Previously it was ~1000
lines, one of the biggest bbclasses in meta/classes. Now, buildhistory just
contains common functionality and plumbing for the buildhistory features. Each
feature listed in BUILDHISTORY_FEATURES causes a class to be automatically
inherited, with a naming convention of "buildhistory-{feature}". I created
buildhistory-package, buildhistory-task, buildhistory-sdk, and
buildhistory-image.
Users could extend buildhistory by inheriting from buildhistory directly (e.g.
if they have a completely new type of buildhistory to implement), or they can
inherit from a feature class. For example,
buildhistory-my-custom-packaging.bbclass. Then, they would use something like
the following:
BUILDHISTORY_FEATURES_remove = "package"
BUILDHISTORY_FEATURES_append = " my-custom-packaging"
Alternatively, they could just add "buildhistory-my-custom-packaging" to
INHERIT.
Some features need to do additional things in the buildhistory event handler,
e.g. "task" writes task signatures during commit. For this, I created
BUILDHISTORY_COMMIT_PREFUNCS, which nicely decouples "task" from the base
buildhistory class.
The only other big change (for now) on this branch is to abstract the
buildhistory-packaging feature a bit. I do this by introducing several "hooks":
BUILDHISTORY_PACKAGE_HOOKS - Python functions or BB functions run at the
end of buildhistory_emit_pkghistory. Currently used to invoke
buildhistory_list_pkg_files.
BUILDHISTORY_PACKAGE_RECIPEHISTORY_HOOKS - Python functions invoked in
write_recipehistory, providing an opportunity to write additional lines to the
per-recipe "latest" file.
BUILDHISTORY_PACKAGE_PACKAGEHISTORY_HOOKS - Python functions invoked in
write_packagehistory providing an opportunity to write additional lines to the
per-package "latest" file.
BUILDHISTORY_PACKAGE_SRCREV_HOOKS - Python functions invoked in
write_latest_srcrev, called once per srcrev-supporting SRC_URI.
BUILDHISTORY_PACKAGE_TAGSRCREV_HOOKS - Python functions invoked in
write_latest_srcrev, called once per tag srcrev SRC_URI.
Additionally, I have re-implemented the latest_srcrev functionality in terms of
BUILDHISTORY_PACKAGE_SRCREV_HOOKS and BUILDHISTORY_PACKAGE_TAGSRCREV_HOOKS.
An example usage of BUILDHISTORY_PACKAGE_SRCREV_HOOKS could be the following:
def branch_hook(d, name, srcrev, ud, total_count, f):
if total_count == 1:
f.write('BRANCH = "{0}"\n'.format(ud.branches[name]))
else:
f.write('BRANCH_{0} = "{1}"\n'.format(name, ud.branches[name]))
BUILDHISTORY_PACKAGE_SRCREV_HOOKS_append = " branch_hook"
Things I'm currently unsure about:
1. Variable naming - I thought about calling them _PREFUNCS or _POSTFUNCS
instead of _HOOKS, or something like that. But the problem is that some of them
can only be Python functions, not BB functions. Which leads me to:
2. Most hooks are intended to be Python functions that are found via
looking in globals(). If someone has a better solution I'd be interested. But I
also don't think this one is necessarily bad.
3. I'm trying to err on the side of not adding too many hooks for the first
release. Hopefully the set I have chosen to add is seen as useful.
4. I wonder if I should create a "buildhistory" directory under "classes"
into which I would move the feature classes. To avoid polluting the "classes"
directory.
Additional stuff to explore:
1. Adding a buildhistory-logs feature class, which archives the
log.do_{TASK} for a user-configurable selection of tasks. Default would be
compile, configure, and install.
2. Adding other extension points where it makes sense.
Thanks,
Chris
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Automatic prebuilt management
> But I would like something smarter, e.g. first trying to check if the SRC_URI
> is available, if not switching on using the prebuilt package if available
> (e.g.
> in a DIR_PREBUILT)...
>
> Before going further is there already an existing solution for that? do you
> have any recommendation on the easier/best way to
> achieve this?
I don't have a "best practice" recommendation to offer. But coincidentally I
suggested something similar for upstreaming a few weeks ago. It was
(rightfully) rejected as too "hacky" for inclusion in core code. Regardless,
here's a bbclass we currently use:
addhandler skip_eventhandler
python skip_eventhandler() {
try:
bb.fetch2.Fetch(d.getVar("SRC_URI").split(), d)
except bb.fetch2.FetchError:
raise bb.parse.SkipRecipe("skip-inaccessible: could not access upstream
repo; check SRC_URI, access rights, and network availability")
}
skip_eventhandler[eventmask] = "bb.event.RecipePreFinalise"
As was pointed out to me, this is a pretty heavy-handed approach - it can fail
for any number of reasons, not just the URI actually being unavailable. So you
may consider a more surgical check, e.g. actually trying to fetch the SRC_URI
directly using raw GET requests.
To implement the handover, you'd have separate recipes for the
build-from-source and the prebuilt case which PROVIDE the same package. You
probably also want to set PREFERRED_PROVIDER in your layer conf to prefer the
build-from-source recipe.
Alternatively, you can do it all in a single recipe by rewriting the SRC_URI
from within the event handler. But I think separate recipes is nicer.
Chris
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)
> > I've patched our tree so that SRC_URI, LICENSE and CVE_PRODUCT are > > archived in buildhistory. SRC_URI has many uses and changes and > > patches can be easily identified. Same with LICENSE, any changes > > trigger a review. CVE_PRODUCT is exported so that we can do QA check > > to make sure mapping from CVE_PRODUCT for non CLOSED licenses exists > > to NVD database product names (maintaining a white list of recipes > > which don't have any CVEs yet). > > I think this supports my point about being more interested in patches > allowing people to extend/customise buildhistory than just adding X. > > Whilst we want to have good defaults, there are always going to be > niche cases for people wanting to extend it... Agreed. Then we can implement our BRANCH scheme without polluting the core code with it. Chris -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)
> I'd be curious to see the patches. > It's definitely something we could use here; we used to have nightly > build checking the build using AUTOREV as well. > > On Fri, Aug 2, 2019 at 3:43 AM wrote: > > > > Hi, > > > > On Thu, Aug 01, 2019 at 04:51:38PM +, chris.laplante--- via > > Openembedded-core wrote: > > > > > I'm interesting in adding SRC_URI support to buildhistory (or a similar > > > mechanism), and would like to get some input. > > > > Yes to this. > > > > Also would be nice if there was an easy way to add bitbake variables to > > buildhistory. > > > > I've patched our tree so that SRC_URI, LICENSE and CVE_PRODUCT are archived > > in buildhistory. SRC_URI has many uses and changes and patches can > > be easily identified. Same with LICENSE, any changes trigger a review. > > CVE_PRODUCT is exported so that we can do QA check to make sure mapping > > from CVE_PRODUCT for non CLOSED licenses exists to NVD database product > > names (maintaining a white list of recipes which don't have any CVEs yet). > > > > We've also changed the SDK name to be stable across builds and added > > DISTRO to the path. In our case IMAGE_NAME and SDK_NAME will include > > git tree tag and hash if tree is dirty, which changes buildhistory SDK paths > > for every build with different input. > > > > I could submit the patches if there is interest in them. I like the idea of including LICENSE and CVE_PRODUCT as well. I will look into to making it extensible via a variable, e.g. BUILDHISTORY_PACKAGE_EXTRA_VARIABLES or something. Chris -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] UNVERIFIED SENDER Re: UNVERIFIED SENDER Re: [bitbake-devel] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)
> > Wouldn't it be a bit overkill to add a new class just to handle SRC_URI? I > > would think it would just be a little extra code inside > write_latest_srcrev. > > Ordinarily I'd say yes. But RP had concerns that these changes add up and > unless everyone wants them it could prove to be a maintenance problem. At the > time he wanted a system for people to opt in to new functionality. (I'm not > trying to speak for him, just repeating what I was told when I asked him a > similar question.) In my case though I was suggesting much more > functionality > then just this one item. Makes sense. I definitely like the idea of the automatic changelog. For now, as a quick prototype of the basic functionality (capturing SRC_URI), I will do as the bbclass suggests and extend BUILDHISTORY_PRESERVE. Thanks, Chris -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] UNVERIFIED SENDER Re: [bitbake-devel] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)
> I've had some thoughts of capturing that same information, as well as > automatically generating a changelog (from short-commit entries) from the last > build to the current one use standard git commands... but I've never had time > to > implement anything. > > When I talked with RP about this a while back, he mentioned it should be > possible to extend the functionality of buildhistory with an additional class. > So the user could opt into extended functionality by including more then one > buildhistory class. > > So my recommendation, store the URL you pulled from along with the SRCREV (if > it's not already been stored). This can then be used later to investigate -- > or > extended [internally or externally] to create changelogs. Wouldn't it be a bit overkill to add a new class just to handle SRC_URI? I would think it would just be a little extra code inside write_latest_srcrev. Chris -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [bitbake-devel] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)
We’re well aware of the limitations and possibility of failed builds, as I said
we hardcode revisions using SRCREV overrides when necessary. It’s been working
well for us so far. So, respectfully, I think that’s not relevant to the
discussion.
Chris
From: Alexander Kanavin
Sent: Thursday, August 01, 2019 12:59 PM
To: LAPLANTE,CHRIS (Agilent USA)
Cc: Openembedded-core@lists.openembedded.org;
bitbake-de...@lists.openembedded.org
Subject: Re: [bitbake-devel] RFC: exposing information about the
SRC_URI(s)/branch via buildhistory (or similar mechanism)
Apologies, but I think you should drop the practice of using AUTOREV. This
completely destroys reproducibility of builds, and makes them susceptible to
global breakage when someone pushes a broken commit into one of the component
repositories.
Yes, bumping SRCREV is annoying, but a) it can be automated; b) you can catch
and reject any problems with the build at that point, so you always have a
nightly that works.
AUTOREV was never meant for the project level, it is a facility strictly for
local development.
Alex
On Thu, 1 Aug 2019 at 18:51, chris.laplante--- via bitbake-devel
mailto:bitbake-de...@lists.openembedded.org>>
wrote:
Hello all,
Most of my team’s closed source recipes use something like the following:
SRC_URI = "git://git@host/path;protocol=ssh;branch=${BRANCH}"
SRCREV = “${AUTOREV}”
BRANCH ??= “master”
(BRANCH is just a convention we use to make the SRC_URI branch easy to
override.)
This makes nightly builds convenient because we always build from the latest.
For release versions, we can use SRCREV_pn-recipe and/or BRANCH_pn-recipe
overrides in local.conf. We get the SRCREV overrides using
buildhistory-collect-srcrevs. But buildhistory has no notion or tracking of
SRC_URI or branches, so currently we use a script that generates the BRANCH
overrides.
I’m interesting in adding SRC_URI support to buildhistory (or a similar
mechanism), and would like to get some input.
Option 1) The easiest way, I think, is to just generate SRC_URI_pn- overrides
with variable expansion.
Option 2) I think it could be useful to introduce BRANCH as a convention.
Currently the “branch” SRC_URI parameter implicitly defaults to “master”. I
could foresee it implicitly defaulting to “${BRANCH}”, with BRANCH ??= “master”
to replicate existing functionality. To handle multiple source-controlled
SRC_URIs, we’d have to do something similar to how SRCREV has a “name”
override. With this option, I wouldn’t think it would be necessary to generate
SRC_URI overrides; just BRANCH overrides.
Option 3) A combination of 1 and 2?
Looking forward to input.
Thanks,
Chris
--
___
bitbake-devel mailing list
bitbake-de...@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/bitbake-devel
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)
Hello all,
Most of my team's closed source recipes use something like the following:
SRC_URI = "git://git@host/path;protocol=ssh;branch=${BRANCH}"
SRCREV = "${AUTOREV}"
BRANCH ??= "master"
(BRANCH is just a convention we use to make the SRC_URI branch easy to
override.)
This makes nightly builds convenient because we always build from the latest.
For release versions, we can use SRCREV_pn-recipe and/or BRANCH_pn-recipe
overrides in local.conf. We get the SRCREV overrides using
buildhistory-collect-srcrevs. But buildhistory has no notion or tracking of
SRC_URI or branches, so currently we use a script that generates the BRANCH
overrides.
I'm interesting in adding SRC_URI support to buildhistory (or a similar
mechanism), and would like to get some input.
Option 1) The easiest way, I think, is to just generate SRC_URI_pn- overrides
with variable expansion.
Option 2) I think it could be useful to introduce BRANCH as a convention.
Currently the "branch" SRC_URI parameter implicitly defaults to "master". I
could foresee it implicitly defaulting to "${BRANCH}", with BRANCH ??= "master"
to replicate existing functionality. To handle multiple source-controlled
SRC_URIs, we'd have to do something similar to how SRCREV has a "name"
override. With this option, I wouldn't think it would be necessary to generate
SRC_URI overrides; just BRANCH overrides.
Option 3) A combination of 1 and 2?
Looking forward to input.
Thanks,
Chris
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Problems with sstate-cache in Artifactory
> DEBUG: For url file://5c/sstate:u-boot-ti-staging:wsl-linux- > gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz > returning > https://artifactory.local.gallagher.io/artifactory/rnd-yocto-packages/sstate-cache/wsl-thud/5c/sstate%3Au-boot-ti- > staging%3Awsl-linux- > gnueabi%3A2019.01%2BgitAUTOINC%2B06e0b5babe%3Ar17%3Awsl%3A3%3A5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz; > downloadfilename=5c/sstate:u-boot-ti-staging:wsl-linux- > gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz > DEBUG: checkstatus: trying again > DEBUG: checkstatus: trying again > DEBUG: checkstatus: trying again > DEBUG: checkstatus: trying again > DEBUG: checkstatus() urlopen failed: HTTP Error 404: Not Found Is this mapping correct? Does using curl to download https://artifactory.local.gallagher.io/artifactory/rnd-yocto-packages/sstate-cache/wsl-thud/5c/sstate%3Au-boot-ti-staging%3Awsl-linux-gnueabi%3A2019.01%2BgitAUTOINC%2B06e0b5babe%3Ar17%3Awsl%3A3%3A5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz;downloadfilename=5c/sstate:u-boot-ti-staging:wsl-linux-gnueabi:2019.01+gitAUTOINC+06e0b5babe:r17:wsl:3:5cdf9380b9b3f8c94a706344cf19554a_packagedata.tgz work? Chris -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
