Re: [yocto] RFC: Backwards compatibility checking sstate-cache

[Michael Ho]

Hi Richard (and all),

@Richard: Thanks for the input regarding the topic and apologies for not 
replying until now, I was a bit buried under workload and simultaneously 
completing my Masters studies. I also wanted to take some time to digest your 
comments amid working on this CDEPENDS concept, and I just want to say that I'm 
very open to finding another way to implement this concept of rebuild avoidance 
in Yocto.

I think the idea of a non-invasive approach can be a bit tricky, but your idea 
of a secondary sstate signature and mapping table sounds workable to me. (In 
fact, as I slowly developed out this concept, I think it's almost required). 
One point though with the removal of the metadata changes is that the view of 
"what is compatible" will be concrete and cannot differ among children. 
Sometimes you have only specific children of a recipe that can handle 
compatibility changes (and at different levels of compatibility) while others 
cannot handle any change at all, and that is why I developed with the metadata 
changes as a basis. But definitely, yes, the change to metadata makes things 
more complex and does in a sense impose policy, which I understand would be 
undesirable. Maybe we can figure out an approach that avoids metadata changes 
but keeps this ability to differ compatibility handling among children.

Currently I have a working proof of concept (I've developed out the earlier 
shared patches further) that I'd like to share soon and demonstrate. It still 
needs some further refinement but I think it's showing some interesting results 
already. I'll think further about the topic and see what I can come up with 
that matches your line of thoughts.

@Martin Jansa: Thank you for the comments and information, it's good to know 
that there's some interest from the community to look into this idea.

@Mike Looijmans: Yes, that's essentially the problem I want to resolve with 
this proof of concept, breaking long build chains at points where a clear 
precision cut could've been made depending on whether a rebuild is actually 
required or not (i.e. the bitstream tool not changing). In that case I would 
even use the tool version and assume even if the tool binary has changed, that 
the output result of tool would stay the same so it is compatible in essence 
(and avoid rebuilding 31 hours of stuff).

@Trevor Woerner: Thanks for the suggestion, I was planning to cross-post when I 
had a more stable proof of concept code to share.

Note: I'll be following my colleague, Mario Domenech Goulart (mario-goulart), 
to OEDEM 2017 where I hope to discuss the current concept code and show some 
potential use cases.

Kind regards,
Michael Ho

--
BMW Car IT GmbH
Michael Ho
Spezialist Entwicklung - Linux Software Integration
Lise-Meitner-Str. 14
89081 Ulm

Tel.: +49 731 3780 4071
Mobil: +49 152 5498 0471
Fax: +49-731-37804-001
Mail: michael...@bmw-carit.de
Web: http://www.bmw-carit.de

BMW Car IT GmbH
Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin
Sitz und Registergericht: München HRB 134810



From: Richard Purdie <richard.pur...@linuxfoundation.org>
Sent: 21 September 2017 18:11
To: Michael Ho; yocto@yoctoproject.org
Subject: Re: [yocto] RFC: Backwards compatibility checking sstate-cache

On Fri, 2017-06-30 at 09:48 +, Michael Ho wrote:
> Hi, at BMW Car IT we are working on an experimental feature to
> improve sstate cache hits and we are looking for comments on the
> approach who might have some insights to the problem and seeing if
> anyone is interested in the feature for mainline.

Sorry I didn't see this before now but it was brought to my attention.

I have given this problem quite some thought off and on. I do worry
that the interface you propose is complex and requires changes
throughout the metadata and those changes impose "policy". One of our
strengths is that we're managed to keep "policy" out of the metadata.

In this context by policy, I mean when a recipe is equivalent and when
it is not.

I do have a counter-proposal of how we could solve this in a less
invasive way. This isn't implemented but wouldn't in theory be hard to
do.

The idea would be to have some kind of equivalence list. The first
built object's sstate checksum would become the definitive checksum and
the object added to the sstate cache.

If a new object is built, it would be compared with the one in sstate.
If deemed equivalent (by whatever policy), a mapping would be added to
the list. If not, there is no mapping and it becomes a new definitive
checksum.

All runqueue would then have to do is present its list of sstate
checksums to the list and convert them (in dependency order) into
canonical checksums.

This list could be a local equivalence file, or an equivalence server
which could resolve list of checksums.

D

Re: [yocto] RFC: Backwards compatibility checking sstate-cache

[Mike Looijmans]

On 23-09-17 00:51, Joshua Lock wrote:

On 22/09/17 15:00, Mike Looijmans wrote:
I think this remark in the referenced link is the best summary of "what 
could be improved":


"""the biggest weakness of the sstate signature bits, in my opinion, is that 
it only tracks inputs, not outputs. If task A depends on B, and the metadata 
input to B changes, then A will be rebuilt, even if the *output* of B didn't 
change as a result of the change to its metadata."""


For example, if someone fixes a bug in gcc that only applies to MIPS, then 
builds that only target an ARM machine shouldn't be affected. Much worse 
than that, I have a dependency like:

gcc -> ... -> python -> bitstream tool -> FPGA image

so this means that a change in gcc causes Python to rebuild, which causes 
the bitstream tool to be rebuilt and produce idential output, and that 
triggers a roughly 31-hour build of lots of FPGA images. These are the ones 
we really want to break. A binary output compare would help a lot, even if 
80% of the libraries fail to create reproducible output, I may still be 
spared those 31 hours...


I think tracking digital output is technically feasible, and won't require 
any change to existing recipes.


Also think about "feed" setups. When I see my settop reporting "331 packages 
need updating"... It'd be great if that could be avoided.




That's what packagefeed-stability.bbclass is for. Work on binary 
reproducibility will improve things here too.


Interesting, thanks for the hint.

That might cut down a bit on the over 4TB/month "update my box" traffic.


Kind regards,

Mike Looijmans
System Expert

TOPIC Products
Materiaalweg 4, NL-5681 RJ Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
E-mail: mike.looijm...@topicproducts.com
Website: www.topicproducts.com

Please consider the environment before printing this e-mail


Visit us at the Space Tech Expo Europe (Stand E-71)
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] RFC: Backwards compatibility checking sstate-cache

[Joshua Lock]

On 22/09/17 15:00, Mike Looijmans wrote:
I think this remark in the referenced link is the best summary of "what 
could be improved":


"""the biggest weakness of the sstate signature bits, in my opinion, is 
that it only tracks inputs, not outputs. If task A depends on B, and the 
metadata input to B changes, then A will be rebuilt, even if the 
*output* of B didn't change as a result of the change to its metadata."""


For example, if someone fixes a bug in gcc that only applies to MIPS, 
then builds that only target an ARM machine shouldn't be affected. Much 
worse than that, I have a dependency like:

gcc -> ... -> python -> bitstream tool -> FPGA image

so this means that a change in gcc causes Python to rebuild, which 
causes the bitstream tool to be rebuilt and produce idential output, and 
that triggers a roughly 31-hour build of lots of FPGA images. These are 
the ones we really want to break. A binary output compare would help a 
lot, even if 80% of the libraries fail to create reproducible output, I 
may still be spared those 31 hours...


I think tracking digital output is technically feasible, and won't 
require any change to existing recipes.


Also think about "feed" setups. When I see my settop reporting "331 
packages need updating"... It'd be great if that could be avoided.




That's what packagefeed-stability.bbclass is for. Work on binary 
reproducibility will improve things here too.


Joshua
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] RFC: Backwards compatibility checking sstate-cache

[Mike Looijmans]

I think this remark in the referenced link is the best summary of "what could 
be improved":


"""the biggest weakness of the sstate signature bits, in my opinion, is that 
it only tracks inputs, not outputs. If task A depends on B, and the metadata 
input to B changes, then A will be rebuilt, even if the *output* of B didn't 
change as a result of the change to its metadata."""


For example, if someone fixes a bug in gcc that only applies to MIPS, then 
builds that only target an ARM machine shouldn't be affected. Much worse than 
that, I have a dependency like:

gcc -> ... -> python -> bitstream tool -> FPGA image

so this means that a change in gcc causes Python to rebuild, which causes the 
bitstream tool to be rebuilt and produce idential output, and that triggers a 
roughly 31-hour build of lots of FPGA images. These are the ones we really 
want to break. A binary output compare would help a lot, even if 80% of the 
libraries fail to create reproducible output, I may still be spared those 31 
hours...


I think tracking digital output is technically feasible, and won't require any 
change to existing recipes.


Also think about "feed" setups. When I see my settop reporting "331 packages 
need updating"... It'd be great if that could be avoided.



On 01-07-17 09:48, Martin Jansa wrote:
I haven't tried the patches, but I really like this idea (I was suggesting 
something like that since 2011 
http://permalink.gmane.org/gmane.comp.handhelds.openembedded.core/10327) and 
I'm glad you weren't discouraged attempting to do this.


It also implements 3) b) idea from 
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970 , you might be 
interested read that ticket.


Thanks

On Fri, Jun 30, 2017 at 11:48 AM, Michael Ho > wrote:


Hi, at BMW Car IT we are working on an experimental feature to improve 
sstate
cache hits and we are looking for comments on the approach who might have 
some
insights to the problem and seeing if anyone is interested in the feature 
for
mainline.

The sstate-cache of a recipe is tied closely to its build dependencies, as 
the
signature generated for a task includes all parent task's signatures as
part of
the signature information. This means that when changes occur in the parent
recipes, even if insignificant, all children recipes that have valid sstate
cache must invalidate their sstate cache objects.

What this patchset does is propose to add another optional variable to
recipes,
CDEPENDS, which acts like DEPENDS for all RunQueue purposes but for 
signature
generation it excludes any parent tasks that come from dependencies listed 
in
it. This is to break the signature change domino effect.

This patchset also proposes modifying RunQueue to then be able to run a
compatibility checker during task execution phase for recipes and tasks
that use
CDEPENDS and allow for deciding to re-execute a task despite being covered 
by
sstate-cache.

The patchset is based on the jethro branch for the poky repository, as this 
is
the branch that we are using.  If the general idea sounds good, we can
consider
porting it to master.

Included is an patch that adds an example recipe and compatibility checker,
where compatibility is based on the file checksums of the parent recipes
packages. An example recipe, cdepends-test1, generates a compatibility 
report
containing the file checksums of all files that it packages and which file
paths
they are at. Another recipe, cdepends-test2, can then strip this 
compatibility
report to the minimal files it needs to be consistent and can compare the
latest
checksums it used to configure/compile/install with and if they have 
changed,
trigger a rebuild. If not, the previous version restored from sstate-cache 
is
used.

We are still experimenting with the usages of this, including the use of
having
abi-compliance-checker to compare the ABI of shared libraries as a
compatibility
checker during RunQueue and using the results to avoid rebuilding child
recipes
when the .so files they depend on are still compatible. Example use cases of
this are allowing recipes which provide multiple shared libraries to change 
a
single .so file without rebuilding all its children that depend on the other
shared libraries but not the one that changed.

We're aware of the SIGGEN_EXCLUDERECIPES_ABISAFE feature but feel it
didn't meet
the feature requirements of what this compatibility checker callback is 
doing,
although maybe when porting to master we could refactor to make better use 
of
the work already done there. The current implementation is a bit hacky but
comments would be appreciated in regards to if the concept is feasible and 
if
people are interested in making use of it and their use cases.

Kind regards,
 

Re: [yocto] RFC: Backwards compatibility checking sstate-cache

[Richard Purdie]

On Fri, 2017-06-30 at 09:48 +, Michael Ho wrote:
> Hi, at BMW Car IT we are working on an experimental feature to
> improve sstate cache hits and we are looking for comments on the
> approach who might have some insights to the problem and seeing if
> anyone is interested in the feature for mainline.

Sorry I didn't see this before now but it was brought to my attention.

I have given this problem quite some thought off and on. I do worry
that the interface you propose is complex and requires changes
throughout the metadata and those changes impose "policy". One of our
strengths is that we're managed to keep "policy" out of the metadata.

In this context by policy, I mean when a recipe is equivalent and when
it is not.

I do have a counter-proposal of how we could solve this in a less
invasive way. This isn't implemented but wouldn't in theory be hard to
do.

The idea would be to have some kind of equivalence list. The first
built object's sstate checksum would become the definitive checksum and
the object added to the sstate cache.

If a new object is built, it would be compared with the one in sstate.
If deemed equivalent (by whatever policy), a mapping would be added to
the list. If not, there is no mapping and it becomes a new definitive
checksum.

All runqueue would then have to do is present its list of sstate
checksums to the list and convert them (in dependency order) into
canonical checksums.

This list could be a local equivalence file, or an equivalence server
which could resolve list of checksums.

Doing it this way totally isolates the comparison from the metadata
itself and in my view protects us from future changes in definition of
equivalence.

How does that sound?

Cheers,

Richard



-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] RFC: Backwards compatibility checking sstate-cache

[Martin Jansa]

I haven't tried the patches, but I really like this idea (I was suggesting
something like that since 2011
http://permalink.gmane.org/gmane.comp.handhelds.openembedded.core/10327)
and I'm glad you weren't discouraged attempting to do this.

It also implements 3) b) idea from
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970 , you might be
interested read that ticket.

Thanks

On Fri, Jun 30, 2017 at 11:48 AM, Michael Ho 
wrote:

> Hi, at BMW Car IT we are working on an experimental feature to improve
> sstate
> cache hits and we are looking for comments on the approach who might have
> some
> insights to the problem and seeing if anyone is interested in the feature
> for
> mainline.
>
> The sstate-cache of a recipe is tied closely to its build dependencies, as
> the
> signature generated for a task includes all parent task's signatures as
> part of
> the signature information. This means that when changes occur in the parent
> recipes, even if insignificant, all children recipes that have valid sstate
> cache must invalidate their sstate cache objects.
>
> What this patchset does is propose to add another optional variable to
> recipes,
> CDEPENDS, which acts like DEPENDS for all RunQueue purposes but for
> signature
> generation it excludes any parent tasks that come from dependencies listed
> in
> it. This is to break the signature change domino effect.
>
> This patchset also proposes modifying RunQueue to then be able to run a
> compatibility checker during task execution phase for recipes and tasks
> that use
> CDEPENDS and allow for deciding to re-execute a task despite being covered
> by
> sstate-cache.
>
> The patchset is based on the jethro branch for the poky repository, as
> this is
> the branch that we are using.  If the general idea sounds good, we can
> consider
> porting it to master.
>
> Included is an patch that adds an example recipe and compatibility checker,
> where compatibility is based on the file checksums of the parent recipes
> packages. An example recipe, cdepends-test1, generates a compatibility
> report
> containing the file checksums of all files that it packages and which file
> paths
> they are at. Another recipe, cdepends-test2, can then strip this
> compatibility
> report to the minimal files it needs to be consistent and can compare the
> latest
> checksums it used to configure/compile/install with and if they have
> changed,
> trigger a rebuild. If not, the previous version restored from sstate-cache
> is
> used.
>
> We are still experimenting with the usages of this, including the use of
> having
> abi-compliance-checker to compare the ABI of shared libraries as a
> compatibility
> checker during RunQueue and using the results to avoid rebuilding child
> recipes
> when the .so files they depend on are still compatible. Example use cases
> of
> this are allowing recipes which provide multiple shared libraries to
> change a
> single .so file without rebuilding all its children that depend on the
> other
> shared libraries but not the one that changed.
>
> We're aware of the SIGGEN_EXCLUDERECIPES_ABISAFE feature but feel it
> didn't meet
> the feature requirements of what this compatibility checker callback is
> doing,
> although maybe when porting to master we could refactor to make better use
> of
> the work already done there. The current implementation is a bit hacky but
> comments would be appreciated in regards to if the concept is feasible and
> if
> people are interested in making use of it and their use cases.
>
> Kind regards,
> Michael Ho
>
> --
> BMW Car IT GmbH
> Michael Ho
> Spezialist Entwicklung - Linux Software Integration
> Lise-Meitner-Str. 14
> 89081 Ulm
>
> Tel.: +49 731 3780 4071
> Mobil: +49 152 5498 0471
> Fax: +49-731-37804-001
> Mail: michael...@bmw-carit.de
> Web: http://www.bmw-carit.de
> 
> BMW Car IT GmbH
> Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin
> Sitz und Registergericht: München HRB 134810
> 
>
> --
> ___
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>
>
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] RFC: Backwards compatibility checking sstate-cache

[Michael Ho]

Hi Darcy,

Thank you for the feedback. Actually, depending on what exactly you mean by user
supplied script, this patch set for the concept already supports that.

I'm not sure if my attachments got skewered by the mailing list or not but if
you can see my patch 
"0009-recipes-cdepends-test-example-recipe-framework-for-u.patch",
you can see an example implementation and what needs to be done to introduce a
custom compatibility checker.

This patch set provides the hooks for Bitbake and recipes to communicate about
the compatible dependency checker it needs to use. In essence, the recipes point
Bitbake to a python module that it can load from any meta-layer lib folder and
run a predefined function, passing the directories of the compatibility reports
(the latest from the freshly built recipe, and the older from the sstate cache
of the child recipe) which are also defined by the recipes. This python function
can then do whatever it wants with the compatibility data it has access to (i.e.
call a custom user script to look at the report information) and return
"incompatible" or "compatible" to control the RunQueue. Bitbake will rebuild if
incompatible is returned, and not rebuild if compatible is returned. It can also
return a different value which Bitbake will interpret as a compatibility error
(e.g. binary delivery recipe that will be incompatible so you want to crash the
build instead of detecting the error at runtime).

Therefore, the recipes themselves define all the compatibility report
information and how to do the comparison, allowing for "user supplied hook
scripts" which can be defined at any meta-layer level.

In the case you're hinting towards, you would need to implement the following
which can already hook into these patches:
1. a report generation function for the kernel recipe to output the data you
   need to compare - i.e. a compatibility report (a directory of files) that
   consists of the file checksums of the kernel headers stored in a text file
   and the PV of the recipe stored in another text file - to do so the kernel
   recipe merely needs to inherit compatible-depends-report and add a function
   that outputs these report files and add this function to the prefuncs of the
   task do_compatibility_report.
2. A python function to compare the different versions of the report format
   suggested in (1).
3. Modifying the child recipes to add the kernel to CDEPENDS and point Bitbake
   to the python function by setting the variable CDEPENDS_TOOL_xxx for the
   kernel provider.

If you look at the patch 0009, you'll see the example is fairly simple but
similar to what I explained above. The python function added to the lib folder
is the actual compatibility comparison function and more can be easily
introduced in any meta-layer to match the compatibility checking situation. (If
interested, I can share another example compatibility checker that uses the PV
of a recipe to do rebuild decisions).

Happy to answer in more detail if you require, just let me know.

Thank you.

Kind regards,
Michael Ho

--
BMW Car IT GmbH
Michael Ho
Spezialist Entwicklung - Linux Software Integration
Lise-Meitner-Str. 14
89081 Ulm

Tel.: +49 731 3780 4071
Mobil: +49 152 5498 0471
Fax: +49-731-37804-001
Mail: michael...@bmw-carit.de
Web: http://www.bmw-carit.de

BMW Car IT GmbH
Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin
Sitz und Registergericht: München HRB 134810


From: Darcy Watkins <dwatk...@sierrawireless.com>
Sent: 30 June 2017 17:01
To: Michael Ho
Cc: yocto@yoctoproject.org
Subject: Re: [yocto] RFC: Backwards compatibility checking sstate-cache
  

Hi,


It would also be nice to add some sort of hook script support where user 
supplied hook script performs a check and returns either a "yes it's OK" or a 
"no have to rebuild it" status.
 And then tie that into the dependency logic.


I have unusual use case where I need to have a dependency on changes to the 
kernel version, but not necessarily something triggered after every kernel 
rebuild.   The package I'm building will trigger OpenSSL and a whole whack of  
stuff each time it rebuilds.


It depends on the kernel version and the headers, but shouldn't need to depend 
on every patch or kconfig change that happens.



___



Regards,
 
Darcy
 
Darcy Watkins ::  Senior Staff Engineer, Firmware
 
SIERRA WIRELESS
Direct  +1 604 233 7989   ::  Fax  +1 604 231 1109  ::  Main  +1  604 231 1100
13811 Wireless Way  :: Richmond, BC Canada V6V 3A4
[M4]
dwatk...@sierrawireless.com :: www.sierrawireless.com :: 
www.inmotiontechnology.com
 
  

On Jun 30, 2017, at 2:53 AM, Michael Ho <michael...@bmw-carit.de> wrote:

 
Hi, at BMW Car IT we are working on an experimental feature to improve sstate
cache hits and we are looking for comments on the approach who might have some
i

Re: [yocto] RFC: Backwards compatibility checking sstate-cache

[Darcy Watkins]

Hi,

It would also be nice to add some sort of hook script support where user 
supplied hook script performs a check and returns either a "yes it's OK" or a 
"no have to rebuild it" status.
 And then tie that into the dependency logic.

I have unusual use case where I need to have a dependency on changes to the 
kernel version, but not necessarily something triggered after every kernel 
rebuild.   The package I'm building will trigger OpenSSL and a whole whack of 
stuff each time it rebuilds.

It depends on the kernel version and the headers, but shouldn't need to depend 
on every patch or kconfig change that happens.


___

Regards,

Darcy

Darcy Watkins ::  Senior Staff Engineer, Firmware

SIERRA WIRELESS
Direct  +1 604 233 7989   ::  Fax  +1 604 231 
1109  ::  Main  +1 604 231 
1100
13811 Wireless Way  :: Richmond, BC Canada V6V 3A4
[M4]
dwatk...@sierrawireless.com :: 
www.sierrawireless.com :: 
www.inmotiontechnology.com


On Jun 30, 2017, at 2:53 AM, Michael Ho 
> wrote:

Hi, at BMW Car IT we are working on an experimental feature to improve sstate
cache hits and we are looking for comments on the approach who might have some
insights to the problem and seeing if anyone is interested in the feature for
mainline.

The sstate-cache of a recipe is tied closely to its build dependencies, as the
signature generated for a task includes all parent task's signatures as part of
the signature information. This means that when changes occur in the parent
recipes, even if insignificant, all children recipes that have valid sstate
cache must invalidate their sstate cache objects.

What this patchset does is propose to add another optional variable to recipes,
CDEPENDS, which acts like DEPENDS for all RunQueue purposes but for signature
generation it excludes any parent tasks that come from dependencies listed in
it. This is to break the signature change domino effect.

This patchset also proposes modifying RunQueue to then be able to run a
compatibility checker during task execution phase for recipes and tasks that use
CDEPENDS and allow for deciding to re-execute a task despite being covered by
sstate-cache.

The patchset is based on the jethro branch for the poky repository, as this is
the branch that we are using.  If the general idea sounds good, we can consider
porting it to master.

Included is an patch that adds an example recipe and compatibility checker,
where compatibility is based on the file checksums of the parent recipes
packages. An example recipe, cdepends-test1, generates a compatibility report
containing the file checksums of all files that it packages and which file paths
they are at. Another recipe, cdepends-test2, can then strip this compatibility
report to the minimal files it needs to be consistent and can compare the latest
checksums it used to configure/compile/install with and if they have changed,
trigger a rebuild. If not, the previous version restored from sstate-cache is
used.

We are still experimenting with the usages of this, including the use of having
abi-compliance-checker to compare the ABI of shared libraries as a compatibility
checker during RunQueue and using the results to avoid rebuilding child recipes
when the .so files they depend on are still compatible. Example use cases of
this are allowing recipes which provide multiple shared libraries to change a
single .so file without rebuilding all its children that depend on the other
shared libraries but not the one that changed.

We're aware of the SIGGEN_EXCLUDERECIPES_ABISAFE feature but feel it didn't meet
the feature requirements of what this compatibility checker callback is doing,
although maybe when porting to master we could refactor to make better use of
the work already done there. The current implementation is a bit hacky but
comments would be appreciated in regards to if the concept is feasible and if
people are interested in making use of it and their use cases.

Kind regards,
Michael Ho

--
BMW Car IT GmbH
Michael Ho
Spezialist Entwicklung - Linux Software Integration
Lise-Meitner-Str. 14
89081 Ulm

Tel.: +49 731 3780 4071
Mobil: +49 152 5498 0471
Fax: +49-731-37804-001
Mail: michael...@bmw-carit.de
Web: http://www.bmw-carit.de

BMW Car IT GmbH
Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin
Sitz und Registergericht: München HRB 134810

<0001-cache.py-add-support-for-CDEPENDS.patch>
<0002-siggen.py-add-support-for-CDEPENDS.patch>
<0003-runqueue.py-add-support-for-CDEPENDS.patch>
<0004-taskdata.py-add-support-for-CDEPENDS.patch>
<0005-rm_work.bbclass-add-exception-for-do_cache_compatibi.patch>

[yocto] RFC: Backwards compatibility checking sstate-cache

[Michael Ho]

Hi, at BMW Car IT we are working on an experimental feature to improve sstate
cache hits and we are looking for comments on the approach who might have some
insights to the problem and seeing if anyone is interested in the feature for
mainline.

The sstate-cache of a recipe is tied closely to its build dependencies, as the
signature generated for a task includes all parent task's signatures as part of
the signature information. This means that when changes occur in the parent
recipes, even if insignificant, all children recipes that have valid sstate
cache must invalidate their sstate cache objects.

What this patchset does is propose to add another optional variable to recipes,
CDEPENDS, which acts like DEPENDS for all RunQueue purposes but for signature
generation it excludes any parent tasks that come from dependencies listed in
it. This is to break the signature change domino effect.

This patchset also proposes modifying RunQueue to then be able to run a
compatibility checker during task execution phase for recipes and tasks that use
CDEPENDS and allow for deciding to re-execute a task despite being covered by
sstate-cache.

The patchset is based on the jethro branch for the poky repository, as this is
the branch that we are using.  If the general idea sounds good, we can consider
porting it to master.

Included is an patch that adds an example recipe and compatibility checker,
where compatibility is based on the file checksums of the parent recipes
packages. An example recipe, cdepends-test1, generates a compatibility report
containing the file checksums of all files that it packages and which file paths
they are at. Another recipe, cdepends-test2, can then strip this compatibility
report to the minimal files it needs to be consistent and can compare the latest
checksums it used to configure/compile/install with and if they have changed,
trigger a rebuild. If not, the previous version restored from sstate-cache is
used.

We are still experimenting with the usages of this, including the use of having
abi-compliance-checker to compare the ABI of shared libraries as a compatibility
checker during RunQueue and using the results to avoid rebuilding child recipes
when the .so files they depend on are still compatible. Example use cases of
this are allowing recipes which provide multiple shared libraries to change a
single .so file without rebuilding all its children that depend on the other
shared libraries but not the one that changed.

We're aware of the SIGGEN_EXCLUDERECIPES_ABISAFE feature but feel it didn't meet
the feature requirements of what this compatibility checker callback is doing,
although maybe when porting to master we could refactor to make better use of
the work already done there. The current implementation is a bit hacky but
comments would be appreciated in regards to if the concept is feasible and if
people are interested in making use of it and their use cases.

Kind regards,
Michael Ho

--
BMW Car IT GmbH
Michael Ho
Spezialist Entwicklung - Linux Software Integration
Lise-Meitner-Str. 14
89081 Ulm

Tel.: +49 731 3780 4071
Mobil: +49 152 5498 0471
Fax: +49-731-37804-001
Mail: michael...@bmw-carit.de
Web: http://www.bmw-carit.de

BMW Car IT GmbH
Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin
Sitz und Registergericht: München HRB 134810

From 0afcfc5bde251e96a434c345b4e0e4db895feeae Mon Sep 17 00:00:00 2001
From: Michael Ho 
Date: Fri, 30 Jun 2017 06:52:12 +0200
Subject: [PATCH 1/9] cache.py: add support for CDEPENDS
To: yocto@yoctoproject.org

Modifies the bitbake recipe cache handling to now parse also the CDEPENDS
variable, the cdepends flag that can be set for tasks, and also the machine
architecture for the recipe target (since compatibility will be tied to the
machine type also).
---
 bitbake/lib/bb/cache.py | 29 +
 1 file changed, 29 insertions(+)

diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index ab09b08..3596242 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -146,6 +146,9 @@ class CoreRecipeInfo(RecipeInfoCommon):
 self.fakerootenv  = self.getvar('FAKEROOTENV', metadata)
 self.fakerootdirs = self.getvar('FAKEROOTDIRS', metadata)
 self.fakerootnoenv= self.getvar('FAKEROOTNOENV', metadata)
+self.cdepends = self.depvar('CDEPENDS', metadata)
+self.task_cdepends= self.flaglist('cdepends', self.tasks, metadata, True)
+self.target_sys   = self.getvar('MULTIMACH_TARGET_SYS', metadata)
 
 @classmethod
 def init_cacheData(cls, cachedata):
@@ -172,6 +175,12 @@ class CoreRecipeInfo(RecipeInfoCommon):
 cachedata.rproviders = defaultdict(list)
 cachedata.packages_dynamic = defaultdict(list)
 
+cachedata.all_cdepends = []
+cachedata.compatible_deps = defaultdict(list)
+