Even SRC_URI need not be long and ugly. Consider the git submodules fetcher: we simpy list the single top level revision in gitsm:// and trust that the git executable, having just that one revision, will both verify source integrity for all submodules and produce a tree suitable for archiving and offline builds.
I don't see why npm can't behave similarly: first produce a shrinkwrap with checksums separately (if not already provided by upstream), then trust that npm (that we can build ourselves) will utilize the shrinkwrap to provide the same guarantees as git does with submodules. Alex On Mon, 17 Jan 2022 at 23:46, Richard Purdie < [email protected]> wrote: > On Mon, 2022-01-17 at 13:50 +0100, Stefan Herbrechtsmeier wrote: > > Hi Mark, > > > > Am 14.01.2022 um 21:09 schrieb Mark Asselstine: > > > > > > > > > On 2022-01-14 14:38, Stefan Herbrechtsmeier wrote: > > > > Hi Mark, > > > > > > > > Am 14.01.2022 um 17:58 schrieb Mark Asselstine: > > > > > On 2022-01-14 11:35, Stefan Herbrechtsmeier wrote: > > > > > > Am 14.01.2022 um 16:22 schrieb Mark Asselstine: > > > > > > > On 2022-01-14 10:05, Stefan Herbrechtsmeier wrote: > > > > > > > > Am 14.01.2022 um 15:15 schrieb Mark Asselstine via > > > > > > > > lists.openembedded.org: > > > > > > > > > > > > > > > > > > > > > > > > > > > On 2022-01-14 07:18, Alexander Kanavin wrote: > > > > > > > > > > If we do seriously embark on making npm/go better, the > first > > > > > > > > > > step could be to make npm/go write out a reproducible > manifest > > > > > > > > > > for the licenses and sub-packages that can be verified > against > > > > > > > > > > two recipe checksums during fetch, and ensures no > further > > > > > > > > > > network access is necessary. That alone would make it a > viable > > > > > > > > > > fetcher. Those manifests could contain all needed > information > > > > > > > > > > for further processing (e.g. versions, and what depends > on what > > > > > > > > > > etc.) And yes, it's a bundled self-contained approach, > but that > > > > > > > > > > matches how the rest of the world is using npm. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I can't speak to npm but for go this was where I wanted to > see > > > > > > > > > things go. Just as work was done to avoid unexpected > downloads of > > > > > > > > > Python eggs I always felt the key to improving go > integration was > > > > > > > > > some for of automated SRC_URI generation. Once this would > be > > > > > > > > > available it could be leveraged for licensing and such. > > > > > > > > > > > > > > > > > > Stefan, by the way the reason (a) is not possible is that > > > > > > > > > multiple go applications can use a shared 'library' but > different > > > > > > > > > versions (or even different git commit ids). > > > > > > > > > > > > > > > > Why is this simpler? The recipes need to list every > information > > > > > > > > about its dependencies. That means you repeat a lot of code > and > > > > > > > > need to change a lot of files if you update a single > dependency. > > > > > > > > > > > > > > We went through this with go recipes in meta-virt. It didn't > work. > > > > > > > You end up producing a lot of Yocto Project specific files > > > > > > > containing information which is already available in other > forms. > > > > > > > Throw in the multiple versions issue I described before and > you get > > > > > > > a mess. > > > > > > > > > > > > I assume you want to use the version the project recommend and > not a > > > > > > single major version. What makes Go so special that the reasons > for > > > > > > a single major version are irrelevant? Why don't we use multiple > > > > > > version for C/C++ projects? > > > > > > > > > > Sure, go projects can opt to only use released versions of > > > > > dependencies, but this doesn't always happen. For now, and > possibly > > > > > into the future, we have to accept this is not the case. > > > > > > > > > > Not using the versions recommended in a go project will result in > a > > > > > level in invalidation of their testing and validation as well as > CVE > > > > > tracking. > > > > > > > > Who does this work and how long he do it? > > > > > > This work is done at the project level when we use the upstream > > > configuration verbatim. Only if we decide to deviate in the YP would > > > there be additional work. Which is why I propose we stick to what we > > > know the project is using, testing with and shipping in their releases. > > > > [snip] > > > > But why OE and distributions like Debian use a single project version > > instead of individual dependency versions? I think we have good reasons > > to use a single version and this reasons are independent of the language > > or an existing package manager. > > > > I really miss a comment from a npm user and a TSC member because Alex > > and you propose fundamental changes in OE. > > The TSC had a meeting today and talked about this a little. I'm going to > give my > opinion and the others on the TSC can chime in if they want to add or > differ. > > Firstly, the c) option of highlighting issues with upstreams and working > with > them is important and we need to do that. I'm taking it as a given we will > talk > and try and work with them. > > In parallel, we need to make solutions which work today. In many ways there > aren't perfect answers here and a) or b) may be appropriate in different > cases. > What we as in OE care about in particular is that: > > * recipes precisely define the thing they're building such that it is > uniquely > identified > * builds from mirrors/archives work, you don't need an online upstream to > make a > build (and hence it can be reproduced from archive) > > From that perspective I don't really care if the SRC_URI is long and ugly > in a > recipe as long as it precisely defines what is being built so it is > reproducible > and offline builds/caches work. > > Obviously individual recipes are nice to have in some cases but in the npm > case > where that results in 5000 recipes, it simply won't work with bitbake the > way > bitbake works today. We have no plans that will let us scale bitbake to > 5000 > recipes so we will need to look at the other solutions. > > Using language specific tools and language specific fetchers is ok and we > are > seeing that with the npm shrinkrap and cargo plugins and this is likely the > direction we'll have to move going forward. > > I appreciate there are challenges both ways but does that give an idea of > the > direction the TSC envisages? > > Cheers, > > Richard > > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#1427): https://lists.openembedded.org/g/openembedded-architecture/message/1427 Mute This Topic: https://lists.openembedded.org/mt/88417908/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
