On Fri, May 6, 2022 at 3:39 PM Mark Asselstine
<[email protected]> wrote:
>
>
>
> On 2022-05-06 14:41, Bruce Ashfield wrote:
> > On Fri, May 6, 2022 at 1:10 PM <[email protected]> wrote:
> >>
> >> Hello all,
> >>
> >> I just posted a series of patches for the purpose of generating recipies
> >> for a go offline build:
> >>
> >> [PATCH 1/5] recipetool-create: add ensure_native_cmd function
> >> (openembedded.org)
> >> [PATCH 2/5] create_npm: reuse ensure_native_cmd from create.py
> >> (openembedded.org)
> >> [PATCH 3/5] poky-meta: add go vendor class for offline builds
> >> (openembedded.org)
> >> [PATCH 4/5] recipetool: add go recipe generator (openembedded.org)
> >> [PATCH 5/5] oe-selftest: add go recipe create selftest (openembedded.org)
>
> Hi Lukas,
>
> As you might have figured out from last week's thread on this subject I
> am interested in this work, as well as Bruce's scripts. We will be sure
> to try out your work over the coming days and provide feedback.
At this point, I probably won't bother releasing my scripts. They suit
my needs, but the amount of effort (and time) to get them in a more
usable state (i.e. for more upstreams) isn't worth the effort.
What's here doesn't impact what I'm doing and vice versa, so there's
no need to consolidate.
Bruce
>
> MarkA
>
>
> >>
> >> The series generates recipies of the following form (grpc-web as an
> >> example):
> >>
> >> SRC_URI = "git://${GO_IMPORT};nobranch=1;name=${PN};protocol=https \
> >>
> >> file://go.mod.patch;patchdir=src/${GO_IMPORT} \
> >>
> >>
> >> ${@go_src_uri('github.com/cenkalti/backoff',path='github.com/cenkalti/backoff/v4')}
> >> \
> >>
> >> ${@go_src_uri('github.com/desertbit/timer')} \
> >>
> >> ...
> >>
> >> #TODO: Subdirectories are heuristically derived from the import path and
> >> might be incorrect.
> >>
> >> # github.com/cenkalti/backoff/[email protected] =>
> >> 6b0e4ad0cd65431b217393bff47d1dff727b264b
> >>
> >> SRCREV_github.com.cenkalti.backoff.v4 =
> >> "6b0e4ad0cd65431b217393bff47d1dff727b264b"
> >>
> >> GO_MODULE_PATH[github.com.cenkalti.backoff.v4] =
> >> "github.com/cenkalti/backoff/v4"
> >>
> >> GO_MODULE_VERSION[github.com.cenkalti.backoff.v4] = "v4.1.3"
> >>
> >> # github.com/desertbit/[email protected] =>
> >> c41aec40b27f0eeb2b94300fffcd624c69b02990
> >>
> >> SRCREV_github.com.desertbit.timer =
> >> "c41aec40b27f0eeb2b94300fffcd624c69b02990"
> >>
> >> GO_MODULE_PATH[github.com.desertbit.timer] = "github.com/desertbit/timer"
> >>
> >> GO_MODULE_VERSION[github.com.desertbit.timer] =
> >> "v0.0.0-20180107155436-c41aec40b27f"
> >>
> >>
> >> How it works:
> >>
> >> the recipetool resolves the go-module-path for each dependency in the
> >> go.mod file to a corresponding src-url (mostly git repository urls, but
> >> other vcs would work as well). The recipetools also resolves the
> >> (pseudo)-semver to the corresponding git commit. git-url and commit hash
> >> are added to the generated go-recipe and processed later during build by
> >> the go-vendor.bbclass. During 'unpack' the bbclass copies the fetched
> >> dependency sources to the 'vendor' subdirectory and generates a
> >> 'modules.txt' manifest.
> >> Most of my test projects worked right away. However, I'm sure that I
> >> missed some corner cases, as I'm not a go expert. Thus, I would be
> >> grateful for your opinion on my approach.
> >>
> >
> > I saw the patches on the list, the two approaches are very similar to
> > what I've been using and generating for meta-virtualization. I even
> > see some similar hoops you had to jump through that I did as well
> > (indirect dependencies, etc).
> >
> > The recipe format is different, but the approaches of resolving the
> > upstreams are very similar.
> >
> > Some folks will like the generated information from the python, others
> > won't (I fall into the latter), but that is more of a preference than
> > a technical issue.
> >
> > I unfortunately won't have any time to review or otherwise look at
> > this much until June, but luckily, no matter which direction this
> > heads, what I've been using continues to work .. and that's what I
> > like about the approaches.
> >
> > Bruce
> >
> >> Best regards
> >> Lukas
> >>
> >> On Sat, Apr 30, 2022 at 12:06 AM, Bruce Ashfield wrote:
> >>
> >> On Fri, Apr 29, 2022 at 4:21 PM Khem Raj <[email protected]> wrote:
> >>
> >>
> >>
> >>
> >> On 4/29/22 12:15 PM, Bruce Ashfield wrote:
> >>
> >> On Fri, Apr 29, 2022 at 2:59 PM Mark Asselstine
> >> <[email protected]> wrote:
> >>
> >>
> >> Folks,
> >>
> >> For those of us who have had to work on GO recipes I think there has
> >> been an open item that would be nice to get resolved in this new release
> >> cycle.
> >>
> >> Specifically, as with many of the modern languages GO includes its own
> >> fetcher for 'dependencies', in GO these are referred to as 'modules'. As
> >> it stands there is no unified or easy way to handle these and it would
> >> be good to have this addressed in bitbake.
> >>
> >> For example I can use two GO projects which we wanted to write recipes
> >> for recently.
> >>
> >> #1 buildah - https://github.com/containers/buildah - as you can see in
> >> this project they actively maintain a 'vendor' directory and regularly
> >> make commits to the vendor directory to keep it up-to-date. If you do
> >> something like the following you can verify that no additional downloads
> >> are needed during a build
> >>
> >> $> export GOPATH=/tmp/go
> >> $> git clone https://github.com/containers/buildah
> >> $> cd buildah
> >> $> go build -mod vendor ./cmd/buildah
> >> $> ls /tmp/go/pkg/mod/
> >>
> >> The /tmp/go/pkg/mod directory remains empty and no download msgs are
> >> seen during the build to confirm that no downloads take place.
> >>
> >> #2 Minio Client (MC) - https://github.com/minio/mc - as you can see in
> >> this project there is no maintained 'vendor' directory. Similarly if you
> >> clone and build this project you will experience additional downloads
> >>
> >> $> export GOPATH=/tmp/go
> >> $> git clone https://github.com/minio/mc
> >> $> cd mc
> >> $> go build -trimpath -tags kqueue -o ./mc
> >> $> ls /tmp/go/pkg/mod/
> >>
> >> This time you will find /tmp/go/pkg/mod is filled with mods and during
> >> the build download msgs will have been displayed.
> >>
> >> For folks that follow the mailing lists you might have seen discussions
> >> about writing recipes for each of the "dependencies" but as you will
> >> have noted in these threads GO projects typically have their own
> >> versions and even specific commits of dependencies which they are
> >> building against. In most cases you want to use what is dictated in the
> >> project's go.mod and such to not invalidate the projects build and
> >> runtime testing.
> >>
> >> Solutions for writing recipes for projects like Minio Client:
> >>
> >> Potential (actually currently in use) solution #1 is what has been done
> >> for a recipe like k3s
> >> (https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/k3s/k3s_git.bb).
> >> Where each dependency has a SRC_URI and related information, which is
> >> used to populate the vendor directory.
> >>
> >> Potential solution #2 would be to run 'go mod vendor' to populate the
> >> 'vendor' directory, after which a patch can be generated which would be
> >> added to the SRC_URI and as long as we capture the license information
> >> in the recipe for all "dependencies" we can build without worries about
> >> additional download attempts.
> >>
> >> This was looked at (and discarded) as a solution for meta-virtualization.
> >>
> >> It can work, but has big maintenance drawbacks around patching, CVEs.
> >> It also doesn't allow for download caching, etc, as it has no fetcher
> >> integration. If you want to do something like this, you need to do a
> >> fetcher implementation, setup module proxies, etc, rather than
> >> generating a giant patch to restore 'vendor'
> >>
> >> I think a recipe tool to write recipes for go projects is going to serve
> >> the community well, like what cargo-bitbake is in rust ecosystem. And
> >> then bitbake fetcher could aide a bit in fetching part.
> >>
> >> That's what I'm describing with the tool/script, it generates the
> >> recipe elements. It could be dynamic, or as it currently stands, is
> >> manually run to create a recipe.
> >>
> >>
> >>
> >> Plus it makes us reliant on the golang module infrastructure, which
> >> has shown to not always be reliable.
> >>
> >> I would agree this was in flux before go modules were accepted as
> >> standard way of doing dependencies, there were many solutions for
> >> dependency management but I think we should now very much settle that it
> >> is go modules now, since its part of core tooling in go community now a
> >> days. So if we build tooling to use go modules underneath we wont be
> >> painting ourselves to a corner.
> >>
> >> There's no real painting into a corner in what I'm describing, it
> >> isn't going around the modules and modules specification, it just
> >> isn't using go itself to fetch. I've watched enough changes in golang,
> >> that maybe this is the last .. maybe it isn't :) The go module format
> >> hasn't been in flux for quite some time. It is the infrastructure that
> >> goes through by default, has a few choke points and can have issues.
> >> Of course the go mod infrastructure is flexible and can setup
> >> mirrors/proxy, caching, etc, and that is what a fetcher implementation
> >> could absolutely do.
> >>
> >> It is the ongoing license management, maintenance/security, developer
> >> flow, etc, that I found as a challenge.
> >>
> >> Someone can absolutely implement a fetcher for it, set up the
> >> variables to point to downloads for the proxy/mirrors (and avoid the
> >> network), lock down go.mod/go.sum as the dependency declaration and
> >> build without vendoring (and network). We talked about that model in
> >> some of the earlier threads as well. That someone who writes it, just
> >> isn't me at the moment :) I have something working that is independent
> >> of those entanglements and can work even if something else is
> >> developed.
> >>
> >> Cheers,
> >>
> >> Bruce
> >>
> >>
> >>
> >>
> >>
> >>
> >> Potential solution #3 would be to add a 'go get' to the fetcher. This
> >> would allow transposing go.mod information into the SRC_URI, drastically
> >> reducing the tedious work done for solution #1.
> >>
> >> This is difficult to implement, and leaves us with challenges for
> >> locking down versions, licensing, etc. Definitely doable, but also
> >> makes us again reliant on the go module infrastructure, unless we
> >> setup mirrors, etc.
> >>
> >>
> >> Potential solution #4 extend the tooling to facilitate the SRC_URI
> >> generation for #1.
> >>
> >> Not necessary. This is how #1 is actually done :)
> >>
> >>
> >> Each of these have pros and cons, for instance #2 is the least amount of
> >> work but also does nothing for saving download space when multiple GO
> >> recipes do share dependencies. #3 and #2 are less secure compared to #1.
> >> None of these address licensing very well...
> >>
> >> I know this subject came up a few months ago but I don't recall there
> >> being a resolution and since we are starting a new dev cycle I thought
> >> now would be a good time to get thoughts from folks who have written GO
> >> project recipes and take steps to making updates to moving this forward,
> >> even if just by a little bit.
> >>
> >> For go, we did come to a conclusion amongst the interested parties.
> >>
> >> The "solution" that I'm using, is what you see in meta-virt. Use the
> >> well established fetchers to go to the source repositories. That
> >> allows us all the re-use and well tested functionality of git, etc,
> >> and no need to re-invent the wheel.
> >>
> >> I've written a tool that generates everything required for those
> >> SRC_URIs, so for the most part, it isn't a manual process.
> >>
> >> My presentation on the topic was accepted to the yocto summit, and
> >> before that, I'll make the script available .. I just haven't had time
> >> with all the release chaos and kernel issues at the moment.
> >>
> >> Bruce
> >>
> >>
> >> MarkA
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> --
> >> - Thou shalt not follow the NULL pointer, for chaos and madness await
> >> thee at its end
> >> - "Use the force Harry" - Gandalf, Star Trek II
> >>
> >>
> >>
> >>
> >
> >
> >
> >
> >
> >
--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1542):
https://lists.openembedded.org/g/openembedded-architecture/message/1542
Mute This Topic: https://lists.openembedded.org/mt/90782883/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
