Hi Richard,
Am 17.01.2022 um 23:46 schrieb Richard Purdie:
On Mon, 2022-01-17 at 13:50 +0100, Stefan Herbrechtsmeier wrote:
Hi Mark,
Am 14.01.2022 um 21:09 schrieb Mark Asselstine:
On 2022-01-14 14:38, Stefan Herbrechtsmeier wrote:
Hi Mark,
Am 14.01.2022 um 17:58 schrieb Mark Asselstine:
On 2022-01-14 11:35, Stefan Herbrechtsmeier wrote:
Am 14.01.2022 um 16:22 schrieb Mark Asselstine:
On 2022-01-14 10:05, Stefan Herbrechtsmeier wrote:
Am 14.01.2022 um 15:15 schrieb Mark Asselstine via
lists.openembedded.org:
On 2022-01-14 07:18, Alexander Kanavin wrote:
If we do seriously embark on making npm/go better, the first
step could be to make npm/go write out a reproducible manifest
for the licenses and sub-packages that can be verified against
two recipe checksums during fetch, and ensures no further
network access is necessary. That alone would make it a viable
fetcher. Those manifests could contain all needed information
for further processing (e.g. versions, and what depends on what
etc.) And yes, it's a bundled self-contained approach, but that
matches how the rest of the world is using npm.
I can't speak to npm but for go this was where I wanted to see
things go. Just as work was done to avoid unexpected downloads of
Python eggs I always felt the key to improving go integration was
some for of automated SRC_URI generation. Once this would be
available it could be leveraged for licensing and such.
Stefan, by the way the reason (a) is not possible is that
multiple go applications can use a shared 'library' but different
versions (or even different git commit ids).
Why is this simpler? The recipes need to list every information
about its dependencies. That means you repeat a lot of code and
need to change a lot of files if you update a single dependency.
We went through this with go recipes in meta-virt. It didn't work.
You end up producing a lot of Yocto Project specific files
containing information which is already available in other forms.
Throw in the multiple versions issue I described before and you get
a mess.
I assume you want to use the version the project recommend and not a
single major version. What makes Go so special that the reasons for
a single major version are irrelevant? Why don't we use multiple
version for C/C++ projects?
Sure, go projects can opt to only use released versions of
dependencies, but this doesn't always happen. For now, and possibly
into the future, we have to accept this is not the case.
Not using the versions recommended in a go project will result in a
level in invalidation of their testing and validation as well as CVE
tracking.
Who does this work and how long he do it?
This work is done at the project level when we use the upstream
configuration verbatim. Only if we decide to deviate in the YP would
there be additional work. Which is why I propose we stick to what we
know the project is using, testing with and shipping in their releases.
[snip]
But why OE and distributions like Debian use a single project version
instead of individual dependency versions? I think we have good reasons
to use a single version and this reasons are independent of the language
or an existing package manager.
I really miss a comment from a npm user and a TSC member because Alex
and you propose fundamental changes in OE.
The TSC had a meeting today and talked about this a little. I'm going to give my
opinion and the others on the TSC can chime in if they want to add or differ.
Thanks for your opinions.
Firstly, the c) option of highlighting issues with upstreams and working with
them is important and we need to do that. I'm taking it as a given we will talk
and try and work with them.
None of the solutions works again upstream but in some cases the
opinions (ex. stability vs security) are different and we need a way to
overcome this. The question is how much of OE features we want to reuse
and if we want to support features which are missing in npm (ex. update
indirect dependencies or replace deprecated packages).
In parallel, we need to make solutions which work today. In many ways there
aren't perfect answers here and a) or b) may be appropriate in different cases.
What we as in OE care about in particular is that:
* recipes precisely define the thing they're building such that it is uniquely
identified
Does this include the licenses and license checksums of dependencies?
Does this include the CVE product names of all dependencies?
* builds from mirrors/archives work, you don't need an online upstream to make a
build (and hence it can be reproduced from archive)
Is this the only OE feature the npm integration should support?
From that perspective I don't really care if the SRC_URI is long and ugly in a
recipe as long as it precisely defines what is being built so it is reproducible
and offline builds/caches work.
Which approach do you prefer?
1) All dependency urls inside the SRC_URI.
2) Language specific lock file beside the recipe in a meta layer.
3) Additional do_fetch_dependency and do_patch_dependency task after
do_patch to reuse the language specific lock file from the recipe source.
Obviously individual recipes are nice to have in some cases but in the npm case
where that results in 5000 recipes, it simply won't work with bitbake the way
bitbake works today. We have no plans that will let us scale bitbake to 5000
recipes so we will need to look at the other solutions.
NPM per design use a lot of duplicates, multiple versions and deprecated
packages. Furthermore the packages ship dead code or have missing
license. Without individual recipes it is impossible to fix this issues
or we have to maintain fixes in multiple recipes or fix it outside of OE.
Using language specific tools and language specific fetchers is ok and we are
seeing that with the npm shrinkrap and cargo plugins and this is likely the
direction we'll have to move going forward.
The npm shrinkrap fetcher doesn't use a language specific tool. It
extracts the URLs from the lock file and use the normal fetcher to
generate the dependency package tree.
I appreciate there are challenges both ways but does that give an idea of the
direction the TSC envisages?
If I understand you correct you say that the solution of individual
recipes won't work for npm because of the high number of individual
packages and it isn't planed to fix this. The npm integration need to
support the download proxy. What about all the other features of OE I
mention in my first mail?
Should OE only be a build runner for other package manager? This means
you have to manage the dependencies outside of OE. This leads to the
following questions:
1) Should it be possible to use a lock file from the source?
2) Should it be possible to patch the lock file?
3) Should it be possible to patch a dependency?
4) Must a patch be applied to all packages with the same version or
should it be applied to an individual package inside the dependency tree?
4) Should the recipe detect license changes of dependencies?
5) Should the recipetool or build process generate the licenses and
license checksums?
6) Should the recipetool or build process extract the CVE product name
and version?
It would be nice if you could help me to get a clear vision about the
integration of language specific package managers so that I can adapt my
work.
Regards
Stefan
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1428):
https://lists.openembedded.org/g/openembedded-architecture/message/1428
Mute This Topic: https://lists.openembedded.org/mt/88417908/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
