Re: [yocto] [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools

[Patrick Ohly]

On Thu, 2017-03-16 at 10:48 +, Jack Mitchell wrote:
> An issue with not having separate recipes is going to cause problems 
> with OTA package updates. I currently have a node package that I 
> maintain for a client and it's so big (excess of 30,000 files with all 
> deps) that we have to have a special update procedure of uninstalling 
> the package, re-installing the new version and then performing a system 
> update. A package update with 30,000 files on a quad core fast arm64 
> chip takes upwards of 40 minutes. It would be fantastic, if only the 
> core node application and changed dependencies could be updated.

If only some files changed, then going from a package-based update
mechanism (where a package gets reinstalled when any file gets changed)
to a file-based one (like swupd or OSTree) should also address the
problem, because then the update detects exactly the modified files and
only needs to replace those.

Just a thought, in case that auto-packaging turns out to be too hard.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.



-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools

[Mark Hatle]

On 3/16/17 5:35 AM, Alexander Kanavin wrote:
> On 03/16/2017 11:30 AM, Paul Barker wrote:
>> Staying with NPM as the example, say we have a tree of 1000
>> dependencies for a top-level application recipe. I understand that
>> generating and then parsing 1000 recipes will bloat things and slow
>> down the build. However, we do need some way to apply patches to those
>> dependencies or fix metadata. For example, what if the license is
>> stated wrong on npmjs? What if the dependency list itself is wrong? What
>> I've always liked about OE is that you can fix this sort of problem in a
>> recipe when you need to.
> 
> Uhm, no. Separate recipes are simply not gonna work for npm, or Go, or 
> Rust without ugly hacking. Those 1000 deps are all pulled by 'npm 
> install' into the source tree of the thing you want to build and then 
> installed together with it into a private directory, having them as 
> separate recipes would mean that you need to fetch and install them 
> separately, which is simply not how npm operates.
> 
> You can still patch them dependencies just fine if there's just one 
> recipe; once they're pulled in, apply the patches.

There really shouldn't be a reason why we can't have 'one fetch' for multiple
'recipes'.  The concept might be a bit odd, but it also might work reasonably
well if architected to work with the individual schemes.

--Mark

> 
> Alex
> 
> ___
> Openembedded-architecture mailing list
> openembedded-architect...@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
> 

-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools

[Mark Hatle]

On 3/16/17 4:30 AM, Paul Barker wrote:
> On Thu, 16 Mar 2017 09:17:57 +0100
> Josef Holzmayr  wrote:
> 
>> I'll give it a shot and try to sum up the current state of affairs in 
>> this discussion. In accordance to the "Package managers all the way 
>> down"-presentation, I'm gonna use the term LPM (for language package 
>> manager) for now on.
>>
>> *Requirements*
>> - repeatable and verifyable licensing of all dependencies that a LPM 
>> pulls in.
>> - locking down specific versions of packages and their dependencies for 
>> reproductible builds.
> 
> Agreed with these.
> 
>>
>> *Optionals*
>> - strict separation of fetch, compile, install stages. If a specifc LPM 
>> requires it, those might be intermingled or left out [Mark]
> 
> I think what we need is strict separation of fetch (allowed to access
> network) from the later tasks in a build (not allowed to access
> networks). We need to continue to support users behind corporate
> firewalls via mirroring and offline builds.

I agree here.  The 'network fetch', 'do something with it', 'deploy it'  Are
really the three logical steps we have.  The network fetch (as other pieces of
this thread mentioned) could include some specific dependency fetching or
whatever, but in the end this must be able to be mirrored and disable actual
network connections.  (network dependency fetching is possibly very tricky as
dependency scanning in bitbake happens very early, and there is no way to say
"please re-evaluate the dependencies on this branch".  [Maybe this is something
that needs to be considered for this type of work?)

--Mark

>> - opaque packaging: similar to static linking, we should at least have a 
>> way to bundle up a complete application into a single package. Maybe it 
>> might even be the default (like rust does it at the moment).
>> - leverage as much as possible of the functionality the LPMs provide 
>> instead of reimplementing it.
> 
> We should also aim for automatic generation of recipes from pypi,
> npmjs, crates.io, etc using devtool. I'm not sure how much of this has
> already been implemented, I'm not entirely up-to-date on devtool
> functionality.
> 
> Staying with NPM as the example, say we have a tree of 1000
> dependencies for a top-level application recipe. I understand that
> generating and then parsing 1000 recipes will bloat things and slow
> down the build. However, we do need some way to apply patches to those
> dependencies or fix metadata. For example, what if the license is
> stated wrong on npmjs? What if the dependency list itself is wrong? What
> I've always liked about OE is that you can fix this sort of problem in a
> recipe when you need to.
> 
>>
>> *Wish List*
>> - separating out the LPM infrastructure into one or more distinct 
>> layers, not treating it as OE/bitbake core functionality. [Paul]
>> - support for the use of multiple languages/LPMs inside a single recipe, 
>> hopefully even package. [myself]
>>
>> *Proposed Solutions*
>> - having lockdown files shipped with the recipes (in whatever form to be 
>> defined)
>> - leveraging the recipe system to resolve licensing.
>>
>> If we can boil things down to the common set that we all expect, it will 
>> in my opnion serve as a blueprint for the actual implementation to follow.
>>
>> Greetz,
> 
> Thanks,
> Paul
> 

-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools

[Alexander Kanavin]

By the way, Javascript developers do recognize that they have a 
framework problem.


(source: http://thiswas.notinventedhe.re/on/2015-05-21 )


Alex




-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools

[Alexander Kanavin]

On 03/16/2017 12:48 PM, Jack Mitchell wrote:


An issue with not having separate recipes is going to cause problems
with OTA package updates. I currently have a node package that I
maintain for a client and it's so big (excess of 30,000 files with all
deps) that we have to have a special update procedure of uninstalling
the package, re-installing the new version and then performing a system
update. A package update with 30,000 files on a quad core fast arm64
chip takes upwards of 40 minutes. It would be fantastic, if only the
core node application and changed dependencies could be updated.

Would it be possible to use something similar to the 'auto-package'
stuff we already have to split out the inner package dependencies and
have specific packages for each dependency but not a separate recipe?


Yes, certainly. In fact, I think npm.bbclass already does it.

Alex

--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools

[Jack Mitchell]

On 16/03/17 10:35, Alexander Kanavin wrote:

On 03/16/2017 11:30 AM, Paul Barker wrote:

Staying with NPM as the example, say we have a tree of 1000
dependencies for a top-level application recipe. I understand that
generating and then parsing 1000 recipes will bloat things and slow
down the build. However, we do need some way to apply patches to those
dependencies or fix metadata. For example, what if the license is
stated wrong on npmjs? What if the dependency list itself is wrong? What
I've always liked about OE is that you can fix this sort of problem in a
recipe when you need to.


Uhm, no. Separate recipes are simply not gonna work for npm, or Go, or
Rust without ugly hacking. Those 1000 deps are all pulled by 'npm
install' into the source tree of the thing you want to build and then
installed together with it into a private directory, having them as
separate recipes would mean that you need to fetch and install them
separately, which is simply not how npm operates.

You can still patch them dependencies just fine if there's just one
recipe; once they're pulled in, apply the patches.


Alex



An issue with not having separate recipes is going to cause problems 
with OTA package updates. I currently have a node package that I 
maintain for a client and it's so big (excess of 30,000 files with all 
deps) that we have to have a special update procedure of uninstalling 
the package, re-installing the new version and then performing a system 
update. A package update with 30,000 files on a quad core fast arm64 
chip takes upwards of 40 minutes. It would be fantastic, if only the 
core node application and changed dependencies could be updated.


Would it be possible to use something similar to the 'auto-package' 
stuff we already have to split out the inner package dependencies and 
have specific packages for each dependency but not a separate recipe?


Cheers,
Jack.
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools

[Alexander Kanavin]

On 03/16/2017 11:30 AM, Paul Barker wrote:

Staying with NPM as the example, say we have a tree of 1000
dependencies for a top-level application recipe. I understand that
generating and then parsing 1000 recipes will bloat things and slow
down the build. However, we do need some way to apply patches to those
dependencies or fix metadata. For example, what if the license is
stated wrong on npmjs? What if the dependency list itself is wrong? What
I've always liked about OE is that you can fix this sort of problem in a
recipe when you need to.


Uhm, no. Separate recipes are simply not gonna work for npm, or Go, or 
Rust without ugly hacking. Those 1000 deps are all pulled by 'npm 
install' into the source tree of the thing you want to build and then 
installed together with it into a private directory, having them as 
separate recipes would mean that you need to fetch and install them 
separately, which is simply not how npm operates.


You can still patch them dependencies just fine if there's just one 
recipe; once they're pulled in, apply the patches.



Alex

--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools

[Alexander Kanavin]

On 03/16/2017 10:17 AM, Josef Holzmayr wrote:


- locking down specific versions of packages and their dependencies for
reproductible builds.


Thanks for summarizing! I have just one comment: at least initially, 
locking down specific versions has to be optional. We certainly can try 
our best, but some environments offer no support for it, and even 
explicitly encourage the opposite approach (Go for instance); Yocto 
Project has no resources to fight against that. If someone wants to 
package an app, and lock it down in a reproducible way, they should take 
their concern directly to upstream.


Alex
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools

[Paul Barker]

On Thu, 16 Mar 2017 09:17:57 +0100
Josef Holzmayr  wrote:

> I'll give it a shot and try to sum up the current state of affairs in 
> this discussion. In accordance to the "Package managers all the way 
> down"-presentation, I'm gonna use the term LPM (for language package 
> manager) for now on.
> 
> *Requirements*
> - repeatable and verifyable licensing of all dependencies that a LPM 
> pulls in.
> - locking down specific versions of packages and their dependencies for 
> reproductible builds.

Agreed with these.

> 
> *Optionals*
> - strict separation of fetch, compile, install stages. If a specifc LPM 
> requires it, those might be intermingled or left out [Mark]

I think what we need is strict separation of fetch (allowed to access
network) from the later tasks in a build (not allowed to access
networks). We need to continue to support users behind corporate
firewalls via mirroring and offline builds.

> - opaque packaging: similar to static linking, we should at least have a 
> way to bundle up a complete application into a single package. Maybe it 
> might even be the default (like rust does it at the moment).
> - leverage as much as possible of the functionality the LPMs provide 
> instead of reimplementing it.

We should also aim for automatic generation of recipes from pypi,
npmjs, crates.io, etc using devtool. I'm not sure how much of this has
already been implemented, I'm not entirely up-to-date on devtool
functionality.

Staying with NPM as the example, say we have a tree of 1000
dependencies for a top-level application recipe. I understand that
generating and then parsing 1000 recipes will bloat things and slow
down the build. However, we do need some way to apply patches to those
dependencies or fix metadata. For example, what if the license is
stated wrong on npmjs? What if the dependency list itself is wrong? What
I've always liked about OE is that you can fix this sort of problem in a
recipe when you need to.

> 
> *Wish List*
> - separating out the LPM infrastructure into one or more distinct 
> layers, not treating it as OE/bitbake core functionality. [Paul]
> - support for the use of multiple languages/LPMs inside a single recipe, 
> hopefully even package. [myself]
> 
> *Proposed Solutions*
> - having lockdown files shipped with the recipes (in whatever form to be 
> defined)
> - leveraging the recipe system to resolve licensing.
> 
> If we can boil things down to the common set that we all expect, it will 
> in my opnion serve as a blueprint for the actual implementation to follow.
> 
> Greetz,

Thanks,
Paul
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools

[Josef Holzmayr]

I'll give it a shot and try to sum up the current state of affairs in 
this discussion. In accordance to the "Package managers all the way 
down"-presentation, I'm gonna use the term LPM (for language package 
manager) for now on.


*Requirements*
- repeatable and verifyable licensing of all dependencies that a LPM 
pulls in.
- locking down specific versions of packages and their dependencies for 
reproductible builds.


*Optionals*
- strict separation of fetch, compile, install stages. If a specifc LPM 
requires it, those might be intermingled or left out [Mark]
- opaque packaging: similar to static linking, we should at least have a 
way to bundle up a complete application into a single package. Maybe it 
might even be the default (like rust does it at the moment).
- leverage as much as possible of the functionality the LPMs provide 
instead of reimplementing it.


*Wish List*
- separating out the LPM infrastructure into one or more distinct 
layers, not treating it as OE/bitbake core functionality. [Paul]
- support for the use of multiple languages/LPMs inside a single recipe, 
hopefully even package. [myself]


*Proposed Solutions*
- having lockdown files shipped with the recipes (in whatever form to be 
defined)

- leveraging the recipe system to resolve licensing.

If we can boil things down to the common set that we all expect, it will 
in my opnion serve as a blueprint for the actual implementation to follow.


Greetz,
--
Josef Holzmayr
Software Developer Embedded Systems

Tel: +49 8444 9204-48
Fax: +49 8444 9204-50

R-S-I Elektrotechnik GmbH & Co. KG
Woelkestrasse 11
D-85301 Schweitenkirchen
www.rsi-elektrotechnik.de
———
Amtsgericht Ingolstadt – GmbH: HRB 191328 – KG: HRA 170393
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
Ust-IdNr: DE 128592548

_
Amtsgericht Ingolstadt - GmbH: HRB 191328 - KG: HRA 170363
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
USt-IdNr.: DE 128592548

--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto