Re: [OE-core] [PATCH] sstate: Add sstate usage summary to the build
On Mon, 2018-08-27 at 23:47 +0100, richard.pur...@linuxfoundation.org wrote: > On Fri, 2018-08-24 at 23:24 +0200, Martin Jansa wrote: > > This is nice addition, but maybe it would be worth explaining > > somewhere what these numbers actually mean. > > > > e.g. my current build shows: > > Sstate summary: Wanted 181 Found 39 Missed 142 Current 279 (21% > > match, 69% complete) I should add that with a bit of experience you can learnt to interpret these. A build with a ~20% match often means the do_populate_lic sstate matched and nothing else did. That means there is a fair amount of work to be done. On the other hand, the 69% is fairly positive as a large chunk of the build is already complete. I'm basing this just on having watched a few builds on the autobuilder... Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] sstate: Add sstate usage summary to the build
On Fri, 2018-08-24 at 23:24 +0200, Martin Jansa wrote: > This is nice addition, but maybe it would be worth explaining > somewhere what these numbers actually mean. > > e.g. my current build shows: > Sstate summary: Wanted 181 Found 39 Missed 142 Current 279 (21% > match, 69% complete) > > From quick look at the code my understanding is that: > 1) Wanted = Found + Missed > 2) Wanted is number of sstate covered tasks needed for current build, > Found is how many of them will be covered by existing sstate (fetched > from SSTATE_MIRROR or local) and match is just Found/Wanted > 3) Current is most confusing for me as it is a sum of tasks covered > either by normal stamps or setscene and complete is then percentage > of the tasks which either doesn't need to be executed at all or will > be covered by sstate Wanted is the number of artefacts the build ideally wants from sstate Found is the number of artefacts present in sstate locally or it thinks are fetchable from any mirrors Missed = Wanted - Found Current is the number of stamps which bitbake viewed as "current", i.e. it needed to do nothing. Since they're current, it wouldn't bother querying sstate or considering them any further. "% match" tells you how much of the sstate queried, matched "% complete" tells you how much of the build is already current or completed The reason the current/complete numbers are there are because a build directory where a build has already partly completed could show a "0% match" against sstate but its not immediately clear that things which have already run are never checked for in sstate. A build which was 90% complete would likely have a high "% match" for a fresh tmpdir so this was added to try and give people a real idea of how much work remains in a given build "up front" when you start the build. Does that help explain it? I'm open to better terminology but its hard to find something short and simple which conveys this... > 4) Neither of these is about not-sstate-covered tasks which for this > build was 1983 and 1574 were already covered by stamp files (as shown > at the end of the build shown bellow) The % current is close. 1574/1983 = 79% which is approx equal (but greater than) 69%. The difference is that 69% is before any sstate was fetched, there was a 21% sstate match which meant X tasks were then covered. Therefore after setscene ran, the number of completed tasks would be higher and at 1574. We couldn't easily compute 1574 in advance. >Am I close? > > I'm used to meta/classes/buildstats-summary.bbclass output > (unfortunately after the build) which just shows 3 numbers per sstate > covered task and this new summary is a bit confusing for me. People wanted to know about coverage in advance of the build running and we have a different set of numbers to work with unfortunately. Statistics afterwards are much easier! :) I'm open to better ways of presenting it to make it clearer what it means but we don't really want to print too much either. It really is meant so you can tell at a glance roughly how much work a build will need to do. Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] sstate: Add sstate usage summary to the build
This is nice addition, but maybe it would be worth explaining somewhere
what these numbers actually mean.
e.g. my current build shows:
Sstate summary: Wanted 181 Found 39 Missed 142 Current 279 (21% match, 69%
complete)
>From quick look at the code my understanding is that:
1) Wanted = Found + Missed
2) Wanted is number of sstate covered tasks needed for current build, Found
is how many of them will be covered by existing sstate (fetched from
SSTATE_MIRROR or local) and match is just Found/Wanted
3) Current is most confusing for me as it is a sum of tasks covered either
by normal stamps or setscene and complete is then percentage of the tasks
which either doesn't need to be executed at all or will be covered by sstate
4) Neither of these is about not-sstate-covered tasks which for this build
was 1983 and 1574 were already covered by stamp files (as shown at the end
of the build shown bellow)
Am I close?
I'm used to meta/classes/buildstats-summary.bbclass output (unfortunately
after the build) which just shows 3 numbers per sstate covered task and
this new summary is a bit confusing for me.
NOTE: Tasks Summary: Attempted 1983 tasks of which 1574 didn't need to be
rerun and 1 failed.
NOTE: Writing buildhistory
NOTE: Uploading the report
/OE/build/oe-core/tmp-glibc/log/error-report/error_report_20180824202134.txt
to errors.yoctoproject.org:80
NOTE: Report submitted: 200 Your entry can be found here:
http://errors.yoctoproject.org/Errors/Build/66557/
NOTE: Build completion summary:
NOTE: do_populate_sysroot: 8.1% sstate reuse(3 setscene, 34 scratch)
NOTE: do_package: 0.0% sstate reuse(0 setscene, 33 scratch)
NOTE: do_packagedata: 0.0% sstate reuse(0 setscene, 32 scratch)
NOTE: do_package_write_ipk: 0.0% sstate reuse(0 setscene, 32 scratch)
NOTE: do_populate_lic: 0.0% sstate reuse(0 setscene, 1 scratch)
On Tue, Aug 7, 2018 at 4:49 PM Richard Purdie <
richard.pur...@linuxfoundation.org> wrote:
> Currently the user has no indication of how much sstate was already present
> or that would be used by the build. This change adds some summary
> information
> so that the user can see how much reuse is occurring. To fully work it
> needs some extra information from a recent bitbake commit but this is
> optional.
>
> When combined with bitbake --dry-run this feature can be used to check
> if sstate would be reused in a build.
>
> [YOCTO #12749]
>
> Signed-off-by: Richard Purdie
> ---
> meta/classes/sstate.bbclass | 12
> 1 file changed, 12 insertions(+)
>
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index 350d3107f59..6743becf071 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -892,6 +892,18 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash,
> sq_hashfn, d, siginfo=False):
> evdata['found'].append( (sq_fn[task], sq_task[task],
> sq_hash[task], sstatefile ) )
> bb.event.fire(bb.event.MetadataEvent("MissedSstate", evdata), d)
>
> +# Print some summary statistics about the current task completion and
> how much sstate
> +# reuse there was. Avoid divide by zero errors.
> +total = len(sq_fn)
> +currentcount = d.getVar("BB_SETSCENE_STAMPCURRENT_COUNT") or 0
> +complete = 0
> +if currentcount:
> +complete = (len(ret) + currentcount) / (total + currentcount) *
> 100
> +match = 0
> +if total:
> +match = len(ret) / total * 100
> +bb.plain("Sstate summary: Wanted %d Found %d Missed %d Current %d
> (%d%% match, %d%% complete)" % (total, len(ret), len(missed), currentcount,
> match, complete))
> +
> if hasattr(bb.parse.siggen, "checkhashes"):
> bb.parse.siggen.checkhashes(missed, ret, sq_fn, sq_task, sq_hash,
> sq_hashfn, d)
>
> --
> 2.17.1
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH] sstate: Add sstate usage summary to the build
Currently the user has no indication of how much sstate was already present
or that would be used by the build. This change adds some summary information
so that the user can see how much reuse is occurring. To fully work it
needs some extra information from a recent bitbake commit but this is
optional.
When combined with bitbake --dry-run this feature can be used to check
if sstate would be reused in a build.
[YOCTO #12749]
Signed-off-by: Richard Purdie
---
meta/classes/sstate.bbclass | 12
1 file changed, 12 insertions(+)
diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 350d3107f59..6743becf071 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -892,6 +892,18 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn,
d, siginfo=False):
evdata['found'].append( (sq_fn[task], sq_task[task],
sq_hash[task], sstatefile ) )
bb.event.fire(bb.event.MetadataEvent("MissedSstate", evdata), d)
+# Print some summary statistics about the current task completion and how
much sstate
+# reuse there was. Avoid divide by zero errors.
+total = len(sq_fn)
+currentcount = d.getVar("BB_SETSCENE_STAMPCURRENT_COUNT") or 0
+complete = 0
+if currentcount:
+complete = (len(ret) + currentcount) / (total + currentcount) * 100
+match = 0
+if total:
+match = len(ret) / total * 100
+bb.plain("Sstate summary: Wanted %d Found %d Missed %d Current %d (%d%%
match, %d%% complete)" % (total, len(ret), len(missed), currentcount, match,
complete))
+
if hasattr(bb.parse.siggen, "checkhashes"):
bb.parse.siggen.checkhashes(missed, ret, sq_fn, sq_task, sq_hash,
sq_hashfn, d)
--
2.17.1
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
