Re: [OE-core] How to find out why shared sstate is not being used for a recipe
On 05/27/2014 08:58 AM, Mike Looijmans wrote:
I have a deja-vu feeling about this question.
I have this recipe:
https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image-miami.bb
Which includes this one:
https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image.inc
I have a build server that exports its sstate-cache directory through HTTP,
and a local host that attempts to use that sstate-cache. This works fine,
except for the recipe above. Building this recipe takes about 1 hour, so i
really really really want to share that state at any cost. As you can see,
I've done a big shotgun blast of vardepdsexclude to get the recipe to be as
common as possible. Still any host wants to build its own version.
I recently figured out the cause of this:
LIC_FILES_CHKSUM =
file://${META_ZYNQ_BASE}/COPYING;md5=751419260aa954499f7abaabaa882bbe
The META_ZYNQ_BASE variable is set to the overlay path in the layer.conf file.
This path varies on different machines.
I added the following line to layer.conf to prevent this from happening:
META_ZYNQ_BASE[vardepvalue] = meta-zynq
This triggered a rebuild of all pacakges that I've seen to have this sstate
problem, thereby confirming that this really was the cause.
It DID surprise me that the LIC_FILES_CHKSUM full path filename somehow ended
up in the hash for the package's contents. It would have been better to only
include the license contents itself (or its checksum).
Mike.
Met vriendelijke groet / kind regards,
Mike Looijmans
TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax: (+31) (0) 499 33 69 70
E-mail: mike.looijm...@topic.nl
Website: www.topic.nl
Please consider the environment before printing this e-mail
Bezoek ons op 9 en 10 september tijdens Technology for Health Den Bosch (stand
53)
http://www.technologyforhealth.nl
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] How to find out why shared sstate is not being used for a recipe
On 06/03/2014 04:10 PM, Richard Purdie wrote:
On Tue, 2014-06-03 at 15:54 +0200, Mike Looijmans wrote:
On 06/03/2014 10:45 AM, Richard Purdie wrote:
the next steps depend upon how clear the differences are. Basically
there should be some degree of commonality between the two builds and at
some point there will be divergence. We need to pinpoint the point of
divergence. The divergence will be in fpga-image-miami itself or in one
of its dependencies.
ALL stamps for ALL tasks for fpga-image-miami are different. There's a few
thousand lines in these files, with very few similarities.
Ok, this gives a very strong pointer then, see below.
I'll just go for the do_fetch task then, since that's about the first to
execute.
This is a good move.
So far, on one project (that is, NOT the fpga-image-miami) the SRC_URI is
slightly different. The machines are on various sides of a router/firewall, so
SRC_URI=git://${MY_SERVER}/blabla evaluates differently.
How can I tell the system that the value of MY_SERVER is irrelevant to each
and every build step in each and every recipe?
The SRC_URI changing would indeed cause the tasks build separately since
bitbake thinks they're different things.
You have two ways of addressing this. Firstly, you could setup PREMIRROR
entries for these git urls which remaps them to the correct thing in
each case. There would then be one canonical url in the recipe and
you'd not need to change the hash generation.
The other way would be to either exclude the variable from the checksum
generation or give it a specific value. This would be something like:
do_fetch[vardepsexclude] += SRC_URI
you may also have to do this with other tasks that use SRC_URI such as
do_unpack and do_patch. The other option is:
SRC_URI[vardepvalue] = git://canonical/url/for/recipe
which gives it a specific value to use for checksum purposes.
In this case, a very useful syntax. The machines have different MY_SERVER
values in their local.conf, so I've changed one into:
MY_SERVER = git://some/local/path
MY_SERVER[vardepvalue] = git://common.server/path
This generates the same SRC_URI signatures for the locations that look
different but actually refer to the same location.
The PREMIRROR sounds like a generic solution to the underlying problem of
accessing servers from different locations. I'll keep that in mind for the
next project.
Next, I'll need to investigate why the fpga-image-miami generates different
signatures, as both machines fetch it from the same URI.
Mike.
Met vriendelijke groet / kind regards,
Mike Looijmans
TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax: (+31) (0) 499 33 69 70
E-mail: mike.looijm...@topic.nl
Website: www.topic.nl
Please consider the environment before printing this e-mail
Visit us at MEDTEC Europe 3-5 June, Messe Stuttgart, stand number 5D20
http://medteceurope.com/index.php?page=home-en
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] How to find out why shared sstate is not being used for a recipe
On 06/03/2014 10:45 AM, Richard Purdie wrote: On Tue, 2014-06-03 at 07:35 +0200, Mike Looijmans wrote: On 06/03/2014 07:25 AM, Mike Looijmans wrote: Worst case, you can pull the siginfo files from one build and the siginfo files from the sstate mirror and then see which ones are different, then run bitbake-diffsigs X Y to compare the two files. How do I find what to pull? I have (ssh) access to both machines. The sstate-cache dir contains a bunch of two-digit directories and a gazillion files. I could just copy the whole thing to one machine, there's gigabit between them, but then what do I do with these files? mike:~/zynq_platform/build$ find sstate-cache -name '*fpga-image-miami*.siginfo' | wc -l 480 Suppose I copy them. Where do I copy them to, and what do I do with these 480 files to tell me why the system insists on rebuilding this package? Mike. PS: I really really want to find out. Several of these FPGA recipes take 3.5 hours to build on the fastest i5 we could buy. So you can imagine we really want to prevent having to build it more than once. Let me explain the manual process you can follow for this. Its a pain to walk through and its what -S attempts to automate but you should be able to get an answer manually this way. You have BUILDA and BUILDB, the two builds which should be reusing sstate but are not. The fact they're on different machines is irrelevant to this procedure. It would help if these two builds are just the result of bitbake fpga-image-miami but that isn't essential, it will just introduce more noise. It will also help if they are either both built from an existing sstate cache or both not. The first thing I'd do in each build: find tmp/stamps/ -type f | sort stamplistA.txt I'd then so something like: meld stamplistA stamplistB the next steps depend upon how clear the differences are. Basically there should be some degree of commonality between the two builds and at some point there will be divergence. We need to pinpoint the point of divergence. The divergence will be in fpga-image-miami itself or in one of its dependencies. The one thing which can confuse this view is if some things were reused from sstate. You can tell since a task which runs looks like: tmp/stamps/i586-poky-linux/x11-common/0.1-r47.do_populate_sysroot.d90d4404368125acd109a2ac64ca688f.qemux86 and a task which is from sstate looks like: tmp/stamps/all-poky-linux/gstreamer1.0-meta-base/1.0-r0.do_populate_sysroot_setscene.a49fa811727c557c14ab6ce6f2973587.qemux86 The _setscene part tells you this. Machine specific tasks have the machine name appended to them .qemux86 in this case. The hash representing the task is clear in the filename (a49fa811727c557c14ab6ce6f2973587 in this case). You'll have to filter out any noise from these changes you're not interested in. If a task is _setscene its dependencies may be missing from the list of files entirely (no install/compile/configure etc.). So you take a guess at the divergence point and take note of the two different hashes. You can then find the corresponding siginfo files in sstate: find sstate-cache/ -name *d90d4404368125acd109a2ac64ca688f* sstate-cache/d9/sstate:x11-common:i586-poky-linux:0.1:r47:i586:3:d90d4404368125acd109a2ac64ca688f_populate_sysroot.tgz sstate-cache/d9/sstate:x11-common:i586-poky-linux:0.1:r47:i586:3:d90d4404368125acd109a2ac64ca688f_populate_sysroot.tgz.siginfo which are the sstate files corresponding to my above x11-common task. You will note that the first two letters of the hash are used as a directory prefix. You can also find sigdata files in the stamp directory: $ find tmp/stamps/ -type f | grep d90d440 tmp/stamps/i586-poky-linux/x11-common/0.1-r47.do_populate_sysroot.d90d4404368125acd109a2ac64ca688f.qemux86 tmp/stamps/i586-poky-linux/x11-common/0.1-r47.do_populate_sysroot.sigdata.d90d4404368125acd109a2ac64ca688f The sigdata and siginfo files are identical and equivalent. Once you have the two sstate files, you can run: bitbake-diffsigs A.siginfo B.siginfo and this will tell you why their hashes are different. If you need help deciphering that output, link to it and I can help. Dependency on task fpga-image-medical-demo.bb.do_patch was added with hash 8fbb54a572a402e28d05daae0a238f8b Dependency on task fpga-image-medical-demo.bb.do_patch was removed with hash 44a252a24b20321e8dbec6d9d8b54c67 Now this means that I have to look at the unpack task now, right? The weird thing is, both machines have identical do_patch hashes: tmp-eglibc/stamps/zynq_miami_7030-oe-linux-gnueabi/fpga-image-medical-demo/0.AUTOINC-06349d0386-r0.do_patch.sigdata.44a252a24b20321e8dbec6d9d8b54c67 I cannot find that 8fbb54a572a402e28d05daae0a238f8b hash anywhere. Met vriendelijke groet / kind regards, Mike Looijmans TOPIC Embedded Systems Eindhovenseweg 32-C, NL-5683 KH Best Postbus 440, NL-5680 AK Best Telefoon: (+31) (0) 499 33 69 79 Telefax: (+31) (0) 499 33 69 70 E-mail: mike.looijm...@topic.nl Website:
Re: [OE-core] How to find out why shared sstate is not being used for a recipe
On Tue, Jun 03, 2014 at 07:25:08AM +0200, Mike Looijmans wrote: On 05/29/2014 01:12 AM, Richard Purdie wrote: On Wed, 2014-05-28 at 13:46 -0700, Christopher Larson wrote: On Wed, May 28, 2014 at 1:42 PM, Khem Raj raj.k...@gmail.com wrote: On Mon, May 26, 2014 at 11:58 PM, Mike Looijmans mike.looijm...@topic.nl wrote: I have a deja-vu feeling about this question. I have this recipe: https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image-miami.bb Which includes this one: https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image.inc I have a build server that exports its sstate-cache directory through HTTP, and a local host that attempts to use that sstate-cache. This works fine, except for the recipe above. Building this recipe takes about 1 hour, so i really really really want to share that state at any cost. As you can see, I've done a big shotgun blast of vardepdsexclude to get the recipe to be as common as possible. Still any host wants to build its own version. How can I diagnose the REASON that my machine thinks it isn't building the exact same thing as the build server? see https://wiki.yoctoproject.org/wiki/Enable_sstate_cache towards the end it talks about verifying sstate sigs If the sstate is local at least, you can use bitbake -S printdiff target. There's also bitbake-whatchanged, but the bitbake one is superior. It's on two different machines, so I think that does not qualify as local. Worst case, you can pull the siginfo files from one build and the siginfo files from the sstate mirror and then see which ones are different, then run bitbake-diffsigs X Y to compare the two files. How do I find what to pull? I have (ssh) access to both machines. The sstate-cache dir contains a bunch of two-digit directories and a gazillion files. I could just copy the whole thing to one machine, there's gigabit between them, but then what do I do with these files? You can also use openembedded-core/scripts/sstate-diff-machines.sh script to create just .sigdata files on both machines and then copy just this sstate-diff directory, see header of that script for very short readme. bitbake -S just tries to automate that process if it can. bitbake -S usually crashes here. Are you using latest bitbake? There was fix for that in newer bitbake already/ Met vriendelijke groet / kind regards, Mike Looijmans TOPIC Embedded Systems Eindhovenseweg 32-C, NL-5683 KH Best Postbus 440, NL-5680 AK Best Telefoon: (+31) (0) 499 33 69 79 Telefax: (+31) (0) 499 33 69 70 E-mail: mike.looijm...@topic.nl Website: www.topic.nl Please consider the environment before printing this e-mail Visit us at MEDTEC Europe 3-5 June, Messe Stuttgart, stand number 5D20 http://medteceurope.com/index.php?page=home-en -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core -- Martin 'JaMa' Jansa jabber: martin.ja...@gmail.com signature.asc Description: Digital signature -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] How to find out why shared sstate is not being used for a recipe
On Tue, 2014-06-03 at 07:35 +0200, Mike Looijmans wrote: On 06/03/2014 07:25 AM, Mike Looijmans wrote: Worst case, you can pull the siginfo files from one build and the siginfo files from the sstate mirror and then see which ones are different, then run bitbake-diffsigs X Y to compare the two files. How do I find what to pull? I have (ssh) access to both machines. The sstate-cache dir contains a bunch of two-digit directories and a gazillion files. I could just copy the whole thing to one machine, there's gigabit between them, but then what do I do with these files? mike:~/zynq_platform/build$ find sstate-cache -name '*fpga-image-miami*.siginfo' | wc -l 480 Suppose I copy them. Where do I copy them to, and what do I do with these 480 files to tell me why the system insists on rebuilding this package? Mike. PS: I really really want to find out. Several of these FPGA recipes take 3.5 hours to build on the fastest i5 we could buy. So you can imagine we really want to prevent having to build it more than once. Let me explain the manual process you can follow for this. Its a pain to walk through and its what -S attempts to automate but you should be able to get an answer manually this way. You have BUILDA and BUILDB, the two builds which should be reusing sstate but are not. The fact they're on different machines is irrelevant to this procedure. It would help if these two builds are just the result of bitbake fpga-image-miami but that isn't essential, it will just introduce more noise. It will also help if they are either both built from an existing sstate cache or both not. The first thing I'd do in each build: find tmp/stamps/ -type f | sort stamplistA.txt I'd then so something like: meld stamplistA stamplistB the next steps depend upon how clear the differences are. Basically there should be some degree of commonality between the two builds and at some point there will be divergence. We need to pinpoint the point of divergence. The divergence will be in fpga-image-miami itself or in one of its dependencies. The one thing which can confuse this view is if some things were reused from sstate. You can tell since a task which runs looks like: tmp/stamps/i586-poky-linux/x11-common/0.1-r47.do_populate_sysroot.d90d4404368125acd109a2ac64ca688f.qemux86 and a task which is from sstate looks like: tmp/stamps/all-poky-linux/gstreamer1.0-meta-base/1.0-r0.do_populate_sysroot_setscene.a49fa811727c557c14ab6ce6f2973587.qemux86 The _setscene part tells you this. Machine specific tasks have the machine name appended to them .qemux86 in this case. The hash representing the task is clear in the filename (a49fa811727c557c14ab6ce6f2973587 in this case). You'll have to filter out any noise from these changes you're not interested in. If a task is _setscene its dependencies may be missing from the list of files entirely (no install/compile/configure etc.). So you take a guess at the divergence point and take note of the two different hashes. You can then find the corresponding siginfo files in sstate: find sstate-cache/ -name *d90d4404368125acd109a2ac64ca688f* sstate-cache/d9/sstate:x11-common:i586-poky-linux:0.1:r47:i586:3:d90d4404368125acd109a2ac64ca688f_populate_sysroot.tgz sstate-cache/d9/sstate:x11-common:i586-poky-linux:0.1:r47:i586:3:d90d4404368125acd109a2ac64ca688f_populate_sysroot.tgz.siginfo which are the sstate files corresponding to my above x11-common task. You will note that the first two letters of the hash are used as a directory prefix. You can also find sigdata files in the stamp directory: $ find tmp/stamps/ -type f | grep d90d440 tmp/stamps/i586-poky-linux/x11-common/0.1-r47.do_populate_sysroot.d90d4404368125acd109a2ac64ca688f.qemux86 tmp/stamps/i586-poky-linux/x11-common/0.1-r47.do_populate_sysroot.sigdata.d90d4404368125acd109a2ac64ca688f The sigdata and siginfo files are identical and equivalent. Once you have the two sstate files, you can run: bitbake-diffsigs A.siginfo B.siginfo and this will tell you why their hashes are different. If you need help deciphering that output, link to it and I can help. If you didn't guess the divergence point correctly, it will tell you that some prior task is actually different. In that case you would then go and fetch the siginfo tasks for the previous task and compare those. Either you find the difference or again, you have to trace it further back. Its a pain of a process to go through but it is deterministic and you will eventually find the difference. Have a go and if you have any issues let me know and I'll do what I can to help. Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] How to find out why shared sstate is not being used for a recipe
On Tue, 2014-06-03 at 07:25 +0200, Mike Looijmans wrote: On 05/29/2014 01:12 AM, Richard Purdie wrote: bitbake -S just tries to automate that process if it can. bitbake -S usually crashes here. Is there a bug report for that? It shouldn't crash... Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] How to find out why shared sstate is not being used for a recipe
On 06/03/2014 10:45 AM, Richard Purdie wrote: On Tue, 2014-06-03 at 07:35 +0200, Mike Looijmans wrote: On 06/03/2014 07:25 AM, Mike Looijmans wrote: Worst case, you can pull the siginfo files from one build and the siginfo files from the sstate mirror and then see which ones are different, then run bitbake-diffsigs X Y to compare the two files. How do I find what to pull? I have (ssh) access to both machines. The sstate-cache dir contains a bunch of two-digit directories and a gazillion files. I could just copy the whole thing to one machine, there's gigabit between them, but then what do I do with these files? mike:~/zynq_platform/build$ find sstate-cache -name '*fpga-image-miami*.siginfo' | wc -l 480 Suppose I copy them. Where do I copy them to, and what do I do with these 480 files to tell me why the system insists on rebuilding this package? Mike. PS: I really really want to find out. Several of these FPGA recipes take 3.5 hours to build on the fastest i5 we could buy. So you can imagine we really want to prevent having to build it more than once. Let me explain the manual process you can follow for this. Its a pain to walk through and its what -S attempts to automate but you should be able to get an answer manually this way. You have BUILDA and BUILDB, the two builds which should be reusing sstate but are not. The fact they're on different machines is irrelevant to this procedure. It would help if these two builds are just the result of bitbake fpga-image-miami but that isn't essential, it will just introduce more noise. It will also help if they are either both built from an existing sstate cache or both not. The first thing I'd do in each build: find tmp/stamps/ -type f | sort stamplistA.txt I'd then so something like: meld stamplistA stamplistB the next steps depend upon how clear the differences are. Basically there should be some degree of commonality between the two builds and at some point there will be divergence. We need to pinpoint the point of divergence. The divergence will be in fpga-image-miami itself or in one of its dependencies. ALL stamps for ALL tasks for fpga-image-miami are different. There's a few thousand lines in these files, with very few similarities. The one thing which can confuse this view is if some things were reused from sstate. You can tell since a task which runs looks like: tmp/stamps/i586-poky-linux/x11-common/0.1-r47.do_populate_sysroot.d90d4404368125acd109a2ac64ca688f.qemux86 and a task which is from sstate looks like: tmp/stamps/all-poky-linux/gstreamer1.0-meta-base/1.0-r0.do_populate_sysroot_setscene.a49fa811727c557c14ab6ce6f2973587.qemux86 The _setscene part tells you this. Machine specific tasks have the machine name appended to them .qemux86 in this case. The hash representing the task is clear in the filename (a49fa811727c557c14ab6ce6f2973587 in this case). You'll have to filter out any noise from these changes you're not interested in. If a task is _setscene its dependencies may be missing from the list of files entirely (no install/compile/configure etc.). So you take a guess at the divergence point and take note of the two different hashes. You can then find the corresponding siginfo files in sstate: find sstate-cache/ -name *d90d4404368125acd109a2ac64ca688f* sstate-cache/d9/sstate:x11-common:i586-poky-linux:0.1:r47:i586:3:d90d4404368125acd109a2ac64ca688f_populate_sysroot.tgz sstate-cache/d9/sstate:x11-common:i586-poky-linux:0.1:r47:i586:3:d90d4404368125acd109a2ac64ca688f_populate_sysroot.tgz.siginfo I'll just go for the do_fetch task then, since that's about the first to execute. which are the sstate files corresponding to my above x11-common task. You will note that the first two letters of the hash are used as a directory prefix. You can also find sigdata files in the stamp directory: $ find tmp/stamps/ -type f | grep d90d440 tmp/stamps/i586-poky-linux/x11-common/0.1-r47.do_populate_sysroot.d90d4404368125acd109a2ac64ca688f.qemux86 tmp/stamps/i586-poky-linux/x11-common/0.1-r47.do_populate_sysroot.sigdata.d90d4404368125acd109a2ac64ca688f The sigdata and siginfo files are identical and equivalent. Once you have the two sstate files, you can run: bitbake-diffsigs A.siginfo B.siginfo and this will tell you why their hashes are different. If you need help deciphering that output, link to it and I can help. If you didn't guess the divergence point correctly, it will tell you that some prior task is actually different. In that case you would then go and fetch the siginfo tasks for the previous task and compare those. Either you find the difference or again, you have to trace it further back. Its a pain of a process to go through but it is deterministic and you will eventually find the difference. Have a go and if you have any issues let me know and I'll do what I can to help. Thank you very much! So far, on one project (that is, NOT the fpga-image-miami) the SRC_URI is slightly different. The machines are
Re: [OE-core] How to find out why shared sstate is not being used for a recipe
On Tue, 2014-06-03 at 15:54 +0200, Mike Looijmans wrote:
On 06/03/2014 10:45 AM, Richard Purdie wrote:
the next steps depend upon how clear the differences are. Basically
there should be some degree of commonality between the two builds and at
some point there will be divergence. We need to pinpoint the point of
divergence. The divergence will be in fpga-image-miami itself or in one
of its dependencies.
ALL stamps for ALL tasks for fpga-image-miami are different. There's a few
thousand lines in these files, with very few similarities.
Ok, this gives a very strong pointer then, see below.
I'll just go for the do_fetch task then, since that's about the first to
execute.
This is a good move.
So far, on one project (that is, NOT the fpga-image-miami) the SRC_URI is
slightly different. The machines are on various sides of a router/firewall,
so
SRC_URI=git://${MY_SERVER}/blabla evaluates differently.
How can I tell the system that the value of MY_SERVER is irrelevant to each
and every build step in each and every recipe?
The SRC_URI changing would indeed cause the tasks build separately since
bitbake thinks they're different things.
You have two ways of addressing this. Firstly, you could setup PREMIRROR
entries for these git urls which remaps them to the correct thing in
each case. There would then be one canonical url in the recipe and
you'd not need to change the hash generation.
The other way would be to either exclude the variable from the checksum
generation or give it a specific value. This would be something like:
do_fetch[vardepsexclude] += SRC_URI
you may also have to do this with other tasks that use SRC_URI such as
do_unpack and do_patch. The other option is:
SRC_URI[vardepvalue] = git://canonical/url/for/recipe
which gives it a specific value to use for checksum purposes.
Cheers,
Richard
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] How to find out why shared sstate is not being used for a recipe
On 05/29/2014 01:12 AM, Richard Purdie wrote: On Wed, 2014-05-28 at 13:46 -0700, Christopher Larson wrote: On Wed, May 28, 2014 at 1:42 PM, Khem Raj raj.k...@gmail.com wrote: On Mon, May 26, 2014 at 11:58 PM, Mike Looijmans mike.looijm...@topic.nl wrote: I have a deja-vu feeling about this question. I have this recipe: https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image-miami.bb Which includes this one: https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image.inc I have a build server that exports its sstate-cache directory through HTTP, and a local host that attempts to use that sstate-cache. This works fine, except for the recipe above. Building this recipe takes about 1 hour, so i really really really want to share that state at any cost. As you can see, I've done a big shotgun blast of vardepdsexclude to get the recipe to be as common as possible. Still any host wants to build its own version. How can I diagnose the REASON that my machine thinks it isn't building the exact same thing as the build server? see https://wiki.yoctoproject.org/wiki/Enable_sstate_cache towards the end it talks about verifying sstate sigs If the sstate is local at least, you can use bitbake -S printdiff target. There's also bitbake-whatchanged, but the bitbake one is superior. It's on two different machines, so I think that does not qualify as local. Worst case, you can pull the siginfo files from one build and the siginfo files from the sstate mirror and then see which ones are different, then run bitbake-diffsigs X Y to compare the two files. How do I find what to pull? I have (ssh) access to both machines. The sstate-cache dir contains a bunch of two-digit directories and a gazillion files. I could just copy the whole thing to one machine, there's gigabit between them, but then what do I do with these files? bitbake -S just tries to automate that process if it can. bitbake -S usually crashes here. Met vriendelijke groet / kind regards, Mike Looijmans TOPIC Embedded Systems Eindhovenseweg 32-C, NL-5683 KH Best Postbus 440, NL-5680 AK Best Telefoon: (+31) (0) 499 33 69 79 Telefax: (+31) (0) 499 33 69 70 E-mail: mike.looijm...@topic.nl Website: www.topic.nl Please consider the environment before printing this e-mail Visit us at MEDTEC Europe 3-5 June, Messe Stuttgart, stand number 5D20 http://medteceurope.com/index.php?page=home-en -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] How to find out why shared sstate is not being used for a recipe
On 06/03/2014 07:25 AM, Mike Looijmans wrote: Worst case, you can pull the siginfo files from one build and the siginfo files from the sstate mirror and then see which ones are different, then run bitbake-diffsigs X Y to compare the two files. How do I find what to pull? I have (ssh) access to both machines. The sstate-cache dir contains a bunch of two-digit directories and a gazillion files. I could just copy the whole thing to one machine, there's gigabit between them, but then what do I do with these files? mike:~/zynq_platform/build$ find sstate-cache -name '*fpga-image-miami*.siginfo' | wc -l 480 Suppose I copy them. Where do I copy them to, and what do I do with these 480 files to tell me why the system insists on rebuilding this package? Mike. PS: I really really want to find out. Several of these FPGA recipes take 3.5 hours to build on the fastest i5 we could buy. So you can imagine we really want to prevent having to build it more than once. Met vriendelijke groet / kind regards, Mike Looijmans TOPIC Embedded Systems Eindhovenseweg 32-C, NL-5683 KH Best Postbus 440, NL-5680 AK Best Telefoon: (+31) (0) 499 33 69 79 Telefax: (+31) (0) 499 33 69 70 E-mail: mike.looijm...@topic.nl Website: www.topic.nl Please consider the environment before printing this e-mail Visit us at MEDTEC Europe 3-5 June, Messe Stuttgart, stand number 5D20 http://medteceurope.com/index.php?page=home-en -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] How to find out why shared sstate is not being used for a recipe
On Mon, May 26, 2014 at 11:58 PM, Mike Looijmans mike.looijm...@topic.nl wrote: I have a deja-vu feeling about this question. I have this recipe: https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image-miami.bb Which includes this one: https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image.inc I have a build server that exports its sstate-cache directory through HTTP, and a local host that attempts to use that sstate-cache. This works fine, except for the recipe above. Building this recipe takes about 1 hour, so i really really really want to share that state at any cost. As you can see, I've done a big shotgun blast of vardepdsexclude to get the recipe to be as common as possible. Still any host wants to build its own version. How can I diagnose the REASON that my machine thinks it isn't building the exact same thing as the build server? see https://wiki.yoctoproject.org/wiki/Enable_sstate_cache towards the end it talks about verifying sstate sigs Met vriendelijke groet / kind regards, Mike Looijmans TOPIC Embedded Systems Eindhovenseweg 32-C, NL-5683 KH Best Postbus 440, NL-5680 AK Best Telefoon: (+31) (0) 499 33 69 79 Telefax: (+31) (0) 499 33 69 70 E-mail: mike.looijm...@topic.nl Website: www.topic.nl Please consider the environment before printing this e-mail Visit us at MEDTEC Europe 3-5 June, Messe Stuttgart, stand number 5D20 http://medteceurope.com/index.php?page=home-en -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] How to find out why shared sstate is not being used for a recipe
On Wed, May 28, 2014 at 1:42 PM, Khem Raj raj.k...@gmail.com wrote: On Mon, May 26, 2014 at 11:58 PM, Mike Looijmans mike.looijm...@topic.nl wrote: I have a deja-vu feeling about this question. I have this recipe: https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image-miami.bb Which includes this one: https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image.inc I have a build server that exports its sstate-cache directory through HTTP, and a local host that attempts to use that sstate-cache. This works fine, except for the recipe above. Building this recipe takes about 1 hour, so i really really really want to share that state at any cost. As you can see, I've done a big shotgun blast of vardepdsexclude to get the recipe to be as common as possible. Still any host wants to build its own version. How can I diagnose the REASON that my machine thinks it isn't building the exact same thing as the build server? see https://wiki.yoctoproject.org/wiki/Enable_sstate_cache towards the end it talks about verifying sstate sigs If the sstate is local at least, you can use bitbake -S printdiff target. There's also bitbake-whatchanged, but the bitbake one is superior. -- Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] How to find out why shared sstate is not being used for a recipe
On Wed, 2014-05-28 at 13:46 -0700, Christopher Larson wrote: On Wed, May 28, 2014 at 1:42 PM, Khem Raj raj.k...@gmail.com wrote: On Mon, May 26, 2014 at 11:58 PM, Mike Looijmans mike.looijm...@topic.nl wrote: I have a deja-vu feeling about this question. I have this recipe: https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image-miami.bb Which includes this one: https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image.inc I have a build server that exports its sstate-cache directory through HTTP, and a local host that attempts to use that sstate-cache. This works fine, except for the recipe above. Building this recipe takes about 1 hour, so i really really really want to share that state at any cost. As you can see, I've done a big shotgun blast of vardepdsexclude to get the recipe to be as common as possible. Still any host wants to build its own version. How can I diagnose the REASON that my machine thinks it isn't building the exact same thing as the build server? see https://wiki.yoctoproject.org/wiki/Enable_sstate_cache towards the end it talks about verifying sstate sigs If the sstate is local at least, you can use bitbake -S printdiff target. There's also bitbake-whatchanged, but the bitbake one is superior. Worst case, you can pull the siginfo files from one build and the siginfo files from the sstate mirror and then see which ones are different, then run bitbake-diffsigs X Y to compare the two files. bitbake -S just tries to automate that process if it can. Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
