Re: [yocto] Building restricted source code
Mark, Option 1 isn't a very big concern since it's likely only developers working on the restricted code will cause its hashes to permute. Option 2 is precisely what we were thinking about doing. The concern, though, is what happens when a developer who doesn't have access to the code changes a dependency of the restricted code in such a way that it actually affects the restricted code. That developer is unable to test the effects of his changes without pushing them to a back-end build, which would involve pushing untested changes somewhere. I don't love that thought. Another option would be to make the recipe smart enough to build restricted source on a different machine from the user's using an SDK (which is what I think you were suggesting), but that involves transferring the user's uncommitted code and/or built objects to the separate machine, adding a fairly significant layer of complexity (and likely a significant amount of time) to what is otherwise a relatively straightforward build process. I'm not the biggest fan of any of these options presently, but if we have to choose one of these routes, we will. (We were likely going to use what you presented as your second option, but that's still going to involve someone updating recipes for restricted source every time new binaries become available.) I do very much appreciate the input and I'm glad to know the ideas we came up with aren't completely crazy, since someone else has thought of them too ;-) Any other thoughts on this issue? Kind regards, Jerrod On Tue, Mar 26, 2013 at 3:21 PM, Mark Hatle mark.ha...@windriver.comwrote: On 3/26/13 7:53 AM, Jerrod Peach wrote: As part of my company's firmware builds, we have to build some code that only a handful of developers are allowed to view. We call this restricted source code. Getting our official system builds to build this code isn't a problem. What is a problem is a regular developer's build of this code. Imagine this scenario: The restricted source depends upon eglibc. Our low-level team, who doesn't have access to the restricted source, updates the recipe for eglibc. The hash for the restricted package is permuted, and so they can't get an sstate hit and are required to rebuild the source. But, since they can't check out the source, they can't build it. This would cause a build failure. Is anyone dealing with this scenario while using Yocto currently? If so, how? I know it may be unlikely that many people are hitting this since Yocto is primarily used to build open-source code, but I thought I'd take a shot in the dark and hope for the best. :-) Two ways I know of doing this. Slightly different way of doing things: 1) If the code does -not- rely on outside influences, such as eglibc. Configure the recipe and pretty much ignore everything else with a vardepsexp. Then ship the sstate-cache files that cover the compiled code. (Along with the original recipe...) 2) For code that DOES rely on outside influences.. create a custom recipe that either downloads the source and builds it [if the user has access to the source], or will pull the binaries from a specific location and simply install/package them. This is actually the more common approach. (To seed that location, you can extract the items from your restricted build -- or build it outside the tree using an SDK or similar.) --Mark __**_ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.**org/listinfo/yoctohttps://lists.yoctoproject.org/listinfo/yocto __**_ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.**org/listinfo/yoctohttps://lists.yoctoproject.org/listinfo/yocto ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Building restricted source code
While this may not resolve your problem, these are my 0.2 cents. I assume you already know this: The hash does not need to be permuted between dependency updates, as (e.g.) eglibc promises ABI and API stability between major releases (like most Linux libraries). This means that the hash has to change _only_ if the major version of the dependency changes (you need to recompile your restricted code anyway in this case), or if the configuration of the dependency changes in an incompatible way (ouch!). Of course this is only true for library dependencies. I am not sure if it is possible to just configure the hashing algorithm of bitbake to consider only ABI / API breakage instead of always rehashing every change recursively. Some other thoughts: It may be possible to inject packages during the build via binary package feed... Both ideas would require expanding the bitbake code though. 2013/3/27 Jerrod Peach pea...@lexmark.com: Mark, Option 1 isn't a very big concern since it's likely only developers working on the restricted code will cause its hashes to permute. Option 2 is precisely what we were thinking about doing. The concern, though, is what happens when a developer who doesn't have access to the code changes a dependency of the restricted code in such a way that it actually affects the restricted code. That developer is unable to test the effects of his changes without pushing them to a back-end build, which would involve pushing untested changes somewhere. I don't love that thought. Another option would be to make the recipe smart enough to build restricted source on a different machine from the user's using an SDK (which is what I think you were suggesting), but that involves transferring the user's uncommitted code and/or built objects to the separate machine, adding a fairly significant layer of complexity (and likely a significant amount of time) to what is otherwise a relatively straightforward build process. I'm not the biggest fan of any of these options presently, but if we have to choose one of these routes, we will. (We were likely going to use what you presented as your second option, but that's still going to involve someone updating recipes for restricted source every time new binaries become available.) I do very much appreciate the input and I'm glad to know the ideas we came up with aren't completely crazy, since someone else has thought of them too ;-) Any other thoughts on this issue? Kind regards, Jerrod On Tue, Mar 26, 2013 at 3:21 PM, Mark Hatle mark.ha...@windriver.com wrote: On 3/26/13 7:53 AM, Jerrod Peach wrote: As part of my company's firmware builds, we have to build some code that only a handful of developers are allowed to view. We call this restricted source code. Getting our official system builds to build this code isn't a problem. What is a problem is a regular developer's build of this code. Imagine this scenario: The restricted source depends upon eglibc. Our low-level team, who doesn't have access to the restricted source, updates the recipe for eglibc. The hash for the restricted package is permuted, and so they can't get an sstate hit and are required to rebuild the source. But, since they can't check out the source, they can't build it. This would cause a build failure. Is anyone dealing with this scenario while using Yocto currently? If so, how? I know it may be unlikely that many people are hitting this since Yocto is primarily used to build open-source code, but I thought I'd take a shot in the dark and hope for the best. :-) Two ways I know of doing this. Slightly different way of doing things: 1) If the code does -not- rely on outside influences, such as eglibc. Configure the recipe and pretty much ignore everything else with a vardepsexp. Then ship the sstate-cache files that cover the compiled code. (Along with the original recipe...) 2) For code that DOES rely on outside influences.. create a custom recipe that either downloads the source and builds it [if the user has access to the source], or will pull the binaries from a specific location and simply install/package them. This is actually the more common approach. (To seed that location, you can extract the items from your restricted build -- or build it outside the tree using an SDK or similar.) --Mark ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto -- Regards Samuel ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Building restricted source code
On 3/26/13 7:53 AM, Jerrod Peach wrote: As part of my company's firmware builds, we have to build some code that only a handful of developers are allowed to view. We call this restricted source code. Getting our official system builds to build this code isn't a problem. What is a problem is a regular developer's build of this code. Imagine this scenario: The restricted source depends upon eglibc. Our low-level team, who doesn't have access to the restricted source, updates the recipe for eglibc. The hash for the restricted package is permuted, and so they can't get an sstate hit and are required to rebuild the source. But, since they can't check out the source, they can't build it. This would cause a build failure. Is anyone dealing with this scenario while using Yocto currently? If so, how? I know it may be unlikely that many people are hitting this since Yocto is primarily used to build open-source code, but I thought I'd take a shot in the dark and hope for the best. :-) Two ways I know of doing this. Slightly different way of doing things: 1) If the code does -not- rely on outside influences, such as eglibc. Configure the recipe and pretty much ignore everything else with a vardepsexp. Then ship the sstate-cache files that cover the compiled code. (Along with the original recipe...) 2) For code that DOES rely on outside influences.. create a custom recipe that either downloads the source and builds it [if the user has access to the source], or will pull the binaries from a specific location and simply install/package them. This is actually the more common approach. (To seed that location, you can extract the items from your restricted build -- or build it outside the tree using an SDK or similar.) --Mark ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
