Re: [OE-core] [bitbake-devel] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)
> > I think this supports my point about being more interested in patches
> > allowing people to extend/customise buildhistory than just adding X.
> >
> > Whilst we want to have good defaults, there are always going to be
> > niche cases for people wanting to extend it...
>
>
> Agreed. Then we can implement our BRANCH scheme without polluting the core
> code with it.
Richard, et al. -
I have created the cpl/buildhistory branch on poky-contrib to share a draft of
my ideas for making buildhistory more modular and extensible. Would appreciate
any comments.
The first change was to break up buildhistory.bbclass. Previously it was ~1000
lines, one of the biggest bbclasses in meta/classes. Now, buildhistory just
contains common functionality and plumbing for the buildhistory features. Each
feature listed in BUILDHISTORY_FEATURES causes a class to be automatically
inherited, with a naming convention of "buildhistory-{feature}". I created
buildhistory-package, buildhistory-task, buildhistory-sdk, and
buildhistory-image.
Users could extend buildhistory by inheriting from buildhistory directly (e.g.
if they have a completely new type of buildhistory to implement), or they can
inherit from a feature class. For example,
buildhistory-my-custom-packaging.bbclass. Then, they would use something like
the following:
BUILDHISTORY_FEATURES_remove = "package"
BUILDHISTORY_FEATURES_append = " my-custom-packaging"
Alternatively, they could just add "buildhistory-my-custom-packaging" to
INHERIT.
Some features need to do additional things in the buildhistory event handler,
e.g. "task" writes task signatures during commit. For this, I created
BUILDHISTORY_COMMIT_PREFUNCS, which nicely decouples "task" from the base
buildhistory class.
The only other big change (for now) on this branch is to abstract the
buildhistory-packaging feature a bit. I do this by introducing several "hooks":
BUILDHISTORY_PACKAGE_HOOKS - Python functions or BB functions run at the
end of buildhistory_emit_pkghistory. Currently used to invoke
buildhistory_list_pkg_files.
BUILDHISTORY_PACKAGE_RECIPEHISTORY_HOOKS - Python functions invoked in
write_recipehistory, providing an opportunity to write additional lines to the
per-recipe "latest" file.
BUILDHISTORY_PACKAGE_PACKAGEHISTORY_HOOKS - Python functions invoked in
write_packagehistory providing an opportunity to write additional lines to the
per-package "latest" file.
BUILDHISTORY_PACKAGE_SRCREV_HOOKS - Python functions invoked in
write_latest_srcrev, called once per srcrev-supporting SRC_URI.
BUILDHISTORY_PACKAGE_TAGSRCREV_HOOKS - Python functions invoked in
write_latest_srcrev, called once per tag srcrev SRC_URI.
Additionally, I have re-implemented the latest_srcrev functionality in terms of
BUILDHISTORY_PACKAGE_SRCREV_HOOKS and BUILDHISTORY_PACKAGE_TAGSRCREV_HOOKS.
An example usage of BUILDHISTORY_PACKAGE_SRCREV_HOOKS could be the following:
def branch_hook(d, name, srcrev, ud, total_count, f):
if total_count == 1:
f.write('BRANCH = "{0}"\n'.format(ud.branches[name]))
else:
f.write('BRANCH_{0} = "{1}"\n'.format(name, ud.branches[name]))
BUILDHISTORY_PACKAGE_SRCREV_HOOKS_append = " branch_hook"
Things I'm currently unsure about:
1. Variable naming - I thought about calling them _PREFUNCS or _POSTFUNCS
instead of _HOOKS, or something like that. But the problem is that some of them
can only be Python functions, not BB functions. Which leads me to:
2. Most hooks are intended to be Python functions that are found via
looking in globals(). If someone has a better solution I'd be interested. But I
also don't think this one is necessarily bad.
3. I'm trying to err on the side of not adding too many hooks for the first
release. Hopefully the set I have chosen to add is seen as useful.
4. I wonder if I should create a "buildhistory" directory under "classes"
into which I would move the feature classes. To avoid polluting the "classes"
directory.
Additional stuff to explore:
1. Adding a buildhistory-logs feature class, which archives the
log.do_{TASK} for a user-configurable selection of tasks. Default would be
compile, configure, and install.
2. Adding other extension points where it makes sense.
Thanks,
Chris
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [bitbake-devel] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)
We’re well aware of the limitations and possibility of failed builds, as I said
we hardcode revisions using SRCREV overrides when necessary. It’s been working
well for us so far. So, respectfully, I think that’s not relevant to the
discussion.
Chris
From: Alexander Kanavin
Sent: Thursday, August 01, 2019 12:59 PM
To: LAPLANTE,CHRIS (Agilent USA)
Cc: Openembedded-core@lists.openembedded.org;
bitbake-de...@lists.openembedded.org
Subject: Re: [bitbake-devel] RFC: exposing information about the
SRC_URI(s)/branch via buildhistory (or similar mechanism)
Apologies, but I think you should drop the practice of using AUTOREV. This
completely destroys reproducibility of builds, and makes them susceptible to
global breakage when someone pushes a broken commit into one of the component
repositories.
Yes, bumping SRCREV is annoying, but a) it can be automated; b) you can catch
and reject any problems with the build at that point, so you always have a
nightly that works.
AUTOREV was never meant for the project level, it is a facility strictly for
local development.
Alex
On Thu, 1 Aug 2019 at 18:51, chris.laplante--- via bitbake-devel
mailto:bitbake-de...@lists.openembedded.org>>
wrote:
Hello all,
Most of my team’s closed source recipes use something like the following:
SRC_URI = "git://git@host/path;protocol=ssh;branch=${BRANCH}"
SRCREV = “${AUTOREV}”
BRANCH ??= “master”
(BRANCH is just a convention we use to make the SRC_URI branch easy to
override.)
This makes nightly builds convenient because we always build from the latest.
For release versions, we can use SRCREV_pn-recipe and/or BRANCH_pn-recipe
overrides in local.conf. We get the SRCREV overrides using
buildhistory-collect-srcrevs. But buildhistory has no notion or tracking of
SRC_URI or branches, so currently we use a script that generates the BRANCH
overrides.
I’m interesting in adding SRC_URI support to buildhistory (or a similar
mechanism), and would like to get some input.
Option 1) The easiest way, I think, is to just generate SRC_URI_pn- overrides
with variable expansion.
Option 2) I think it could be useful to introduce BRANCH as a convention.
Currently the “branch” SRC_URI parameter implicitly defaults to “master”. I
could foresee it implicitly defaulting to “${BRANCH}”, with BRANCH ??= “master”
to replicate existing functionality. To handle multiple source-controlled
SRC_URIs, we’d have to do something similar to how SRCREV has a “name”
override. With this option, I wouldn’t think it would be necessary to generate
SRC_URI overrides; just BRANCH overrides.
Option 3) A combination of 1 and 2?
Looking forward to input.
Thanks,
Chris
--
___
bitbake-devel mailing list
bitbake-de...@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/bitbake-devel
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [bitbake-devel] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)
On 8/1/19 11:51 AM, chris.laplante--- via bitbake-devel wrote:
> Hello all,
>
>
>
> Most of my team’s closed source recipes use something like the following:
>
>
>
> SRC_URI = "git://git@host/path;protocol=ssh;branch=${BRANCH}"
>
> SRCREV = “${AUTOREV}”
>
> BRANCH ??= “master”
>
>
>
> (BRANCH is just a convention we use to make the SRC_URI branch easy to
> override.)
>
>
>
> This makes nightly builds convenient because we always build from the latest.
> For release versions, we can use SRCREV_pn-recipe and/or BRANCH_pn-recipe
> overrides in local.conf. We get the SRCREV overrides using
> buildhistory-collect-srcrevs. But buildhistory has no notion or tracking of
> SRC_URI or branches, so currently we use a script that generates the BRANCH
> overrides.
>
>
>
> I’m interesting in adding SRC_URI support to buildhistory (or a similar
> mechanism), and would like to get some input.
>
>
>
> Option 1) The easiest way, I think, is to just generate SRC_URI_pn- overrides
> with variable expansion.
>
>
>
> Option 2) I think it could be useful to introduce BRANCH as a convention.
> Currently the “branch” SRC_URI parameter implicitly defaults to “master”. I
> could foresee it implicitly defaulting to “${BRANCH}”, with BRANCH ??=
> “master”
> to replicate existing functionality. To handle multiple source-controlled
> SRC_URIs, we’d have to do something similar to how SRCREV has a “name”
> override.
> With this option, I wouldn’t think it would be necessary to generate SRC_URI
> overrides; just BRANCH overrides.
>
>
>
> Option 3) A combination of 1 and 2?
>
I've had some thoughts of capturing that same information, as well as
automatically generating a changelog (from short-commit entries) from the last
build to the current one use standard git commands... but I've never had time to
implement anything.
When I talked with RP about this a while back, he mentioned it should be
possible to extend the functionality of buildhistory with an additional class.
So the user could opt into extended functionality by including more then one
buildhistory class.
So my recommendation, store the URL you pulled from along with the SRCREV (if
it's not already been stored). This can then be used later to investigate -- or
extended [internally or externally] to create changelogs.
--Mark
>
> Looking forward to input.
>
>
>
> Thanks,
>
> Chris
>
>
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [bitbake-devel] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)
Apologies, but I think you should drop the practice of using AUTOREV. This
completely destroys reproducibility of builds, and makes them susceptible
to global breakage when someone pushes a broken commit into one of the
component repositories.
Yes, bumping SRCREV is annoying, but a) it can be automated; b) you can
catch and reject any problems with the build at that point, so you always
have a nightly that works.
AUTOREV was never meant for the project level, it is a facility strictly
for local development.
Alex
On Thu, 1 Aug 2019 at 18:51, chris.laplante--- via bitbake-devel <
bitbake-de...@lists.openembedded.org> wrote:
> Hello all,
>
>
>
> Most of my team’s closed source recipes use something like the following:
>
>
>
> SRC_URI = "git://git@host/path;protocol=ssh;branch=${BRANCH}"
>
> SRCREV = “${AUTOREV}”
>
> BRANCH ??= “master”
>
>
>
> (BRANCH is just a convention we use to make the SRC_URI branch easy to
> override.)
>
>
>
> This makes nightly builds convenient because we always build from the
> latest. For release versions, we can use SRCREV_pn-recipe and/or
> BRANCH_pn-recipe overrides in local.conf. We get the SRCREV overrides using
> buildhistory-collect-srcrevs. But buildhistory has no notion or tracking of
> SRC_URI or branches, so currently we use a script that generates the BRANCH
> overrides.
>
>
>
> I’m interesting in adding SRC_URI support to buildhistory (or a similar
> mechanism), and would like to get some input.
>
>
>
> Option 1) The easiest way, I think, is to just generate SRC_URI_pn-
> overrides with variable expansion.
>
>
>
> Option 2) I think it could be useful to introduce BRANCH as a convention.
> Currently the “branch” SRC_URI parameter implicitly defaults to “master”. I
> could foresee it implicitly defaulting to “${BRANCH}”, with BRANCH ??=
> “master” to replicate existing functionality. To handle multiple
> source-controlled SRC_URIs, we’d have to do something similar to how SRCREV
> has a “name” override. With this option, I wouldn’t think it would be
> necessary to generate SRC_URI overrides; just BRANCH overrides.
>
>
>
> Option 3) A combination of 1 and 2?
>
>
>
> Looking forward to input.
>
>
>
> Thanks,
>
> Chris
> --
> ___
> bitbake-devel mailing list
> bitbake-de...@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
