On Fri, May 6, 2022 at 1:10 PM <[email protected]> wrote: > > Hello all, > > I just posted a series of patches for the purpose of generating recipies for > a go offline build: > > [PATCH 1/5] recipetool-create: add ensure_native_cmd function > (openembedded.org) > [PATCH 2/5] create_npm: reuse ensure_native_cmd from create.py > (openembedded.org) > [PATCH 3/5] poky-meta: add go vendor class for offline builds > (openembedded.org) > [PATCH 4/5] recipetool: add go recipe generator (openembedded.org) > [PATCH 5/5] oe-selftest: add go recipe create selftest (openembedded.org) > > The series generates recipies of the following form (grpc-web as an example): > > SRC_URI = "git://${GO_IMPORT};nobranch=1;name=${PN};protocol=https \ > > file://go.mod.patch;patchdir=src/${GO_IMPORT} \ > > > ${@go_src_uri('github.com/cenkalti/backoff',path='github.com/cenkalti/backoff/v4')} > \ > > ${@go_src_uri('github.com/desertbit/timer')} \ > > ... > > #TODO: Subdirectories are heuristically derived from the import path and > might be incorrect. > > # github.com/cenkalti/backoff/[email protected] => > 6b0e4ad0cd65431b217393bff47d1dff727b264b > > SRCREV_github.com.cenkalti.backoff.v4 = > "6b0e4ad0cd65431b217393bff47d1dff727b264b" > > GO_MODULE_PATH[github.com.cenkalti.backoff.v4] = > "github.com/cenkalti/backoff/v4" > > GO_MODULE_VERSION[github.com.cenkalti.backoff.v4] = "v4.1.3" > > # github.com/desertbit/[email protected] => > c41aec40b27f0eeb2b94300fffcd624c69b02990 > > SRCREV_github.com.desertbit.timer = "c41aec40b27f0eeb2b94300fffcd624c69b02990" > > GO_MODULE_PATH[github.com.desertbit.timer] = "github.com/desertbit/timer" > > GO_MODULE_VERSION[github.com.desertbit.timer] = > "v0.0.0-20180107155436-c41aec40b27f" > > > How it works: > > the recipetool resolves the go-module-path for each dependency in the go.mod > file to a corresponding src-url (mostly git repository urls, but other vcs > would work as well). The recipetools also resolves the (pseudo)-semver to the > corresponding git commit. git-url and commit hash are added to the generated > go-recipe and processed later during build by the go-vendor.bbclass. During > 'unpack' the bbclass copies the fetched dependency sources to the 'vendor' > subdirectory and generates a 'modules.txt' manifest. > Most of my test projects worked right away. However, I'm sure that I missed > some corner cases, as I'm not a go expert. Thus, I would be grateful for your > opinion on my approach. >
I saw the patches on the list, the two approaches are very similar to what I've been using and generating for meta-virtualization. I even see some similar hoops you had to jump through that I did as well (indirect dependencies, etc). The recipe format is different, but the approaches of resolving the upstreams are very similar. Some folks will like the generated information from the python, others won't (I fall into the latter), but that is more of a preference than a technical issue. I unfortunately won't have any time to review or otherwise look at this much until June, but luckily, no matter which direction this heads, what I've been using continues to work .. and that's what I like about the approaches. Bruce > Best regards > Lukas > > On Sat, Apr 30, 2022 at 12:06 AM, Bruce Ashfield wrote: > > On Fri, Apr 29, 2022 at 4:21 PM Khem Raj <[email protected]> wrote: > > > > > On 4/29/22 12:15 PM, Bruce Ashfield wrote: > > On Fri, Apr 29, 2022 at 2:59 PM Mark Asselstine > <[email protected]> wrote: > > > Folks, > > For those of us who have had to work on GO recipes I think there has > been an open item that would be nice to get resolved in this new release > cycle. > > Specifically, as with many of the modern languages GO includes its own > fetcher for 'dependencies', in GO these are referred to as 'modules'. As > it stands there is no unified or easy way to handle these and it would > be good to have this addressed in bitbake. > > For example I can use two GO projects which we wanted to write recipes > for recently. > > #1 buildah - https://github.com/containers/buildah - as you can see in > this project they actively maintain a 'vendor' directory and regularly > make commits to the vendor directory to keep it up-to-date. If you do > something like the following you can verify that no additional downloads > are needed during a build > > $> export GOPATH=/tmp/go > $> git clone https://github.com/containers/buildah > $> cd buildah > $> go build -mod vendor ./cmd/buildah > $> ls /tmp/go/pkg/mod/ > > The /tmp/go/pkg/mod directory remains empty and no download msgs are > seen during the build to confirm that no downloads take place. > > #2 Minio Client (MC) - https://github.com/minio/mc - as you can see in > this project there is no maintained 'vendor' directory. Similarly if you > clone and build this project you will experience additional downloads > > $> export GOPATH=/tmp/go > $> git clone https://github.com/minio/mc > $> cd mc > $> go build -trimpath -tags kqueue -o ./mc > $> ls /tmp/go/pkg/mod/ > > This time you will find /tmp/go/pkg/mod is filled with mods and during > the build download msgs will have been displayed. > > For folks that follow the mailing lists you might have seen discussions > about writing recipes for each of the "dependencies" but as you will > have noted in these threads GO projects typically have their own > versions and even specific commits of dependencies which they are > building against. In most cases you want to use what is dictated in the > project's go.mod and such to not invalidate the projects build and > runtime testing. > > Solutions for writing recipes for projects like Minio Client: > > Potential (actually currently in use) solution #1 is what has been done > for a recipe like k3s > (https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/k3s/k3s_git.bb). > Where each dependency has a SRC_URI and related information, which is > used to populate the vendor directory. > > Potential solution #2 would be to run 'go mod vendor' to populate the > 'vendor' directory, after which a patch can be generated which would be > added to the SRC_URI and as long as we capture the license information > in the recipe for all "dependencies" we can build without worries about > additional download attempts. > > This was looked at (and discarded) as a solution for meta-virtualization. > > It can work, but has big maintenance drawbacks around patching, CVEs. > It also doesn't allow for download caching, etc, as it has no fetcher > integration. If you want to do something like this, you need to do a > fetcher implementation, setup module proxies, etc, rather than > generating a giant patch to restore 'vendor' > > I think a recipe tool to write recipes for go projects is going to serve > the community well, like what cargo-bitbake is in rust ecosystem. And > then bitbake fetcher could aide a bit in fetching part. > > That's what I'm describing with the tool/script, it generates the > recipe elements. It could be dynamic, or as it currently stands, is > manually run to create a recipe. > > > > Plus it makes us reliant on the golang module infrastructure, which > has shown to not always be reliable. > > I would agree this was in flux before go modules were accepted as > standard way of doing dependencies, there were many solutions for > dependency management but I think we should now very much settle that it > is go modules now, since its part of core tooling in go community now a > days. So if we build tooling to use go modules underneath we wont be > painting ourselves to a corner. > > There's no real painting into a corner in what I'm describing, it > isn't going around the modules and modules specification, it just > isn't using go itself to fetch. I've watched enough changes in golang, > that maybe this is the last .. maybe it isn't :) The go module format > hasn't been in flux for quite some time. It is the infrastructure that > goes through by default, has a few choke points and can have issues. > Of course the go mod infrastructure is flexible and can setup > mirrors/proxy, caching, etc, and that is what a fetcher implementation > could absolutely do. > > It is the ongoing license management, maintenance/security, developer > flow, etc, that I found as a challenge. > > Someone can absolutely implement a fetcher for it, set up the > variables to point to downloads for the proxy/mirrors (and avoid the > network), lock down go.mod/go.sum as the dependency declaration and > build without vendoring (and network). We talked about that model in > some of the earlier threads as well. That someone who writes it, just > isn't me at the moment :) I have something working that is independent > of those entanglements and can work even if something else is > developed. > > Cheers, > > Bruce > > > > > > > Potential solution #3 would be to add a 'go get' to the fetcher. This > would allow transposing go.mod information into the SRC_URI, drastically > reducing the tedious work done for solution #1. > > This is difficult to implement, and leaves us with challenges for > locking down versions, licensing, etc. Definitely doable, but also > makes us again reliant on the go module infrastructure, unless we > setup mirrors, etc. > > > Potential solution #4 extend the tooling to facilitate the SRC_URI > generation for #1. > > Not necessary. This is how #1 is actually done :) > > > Each of these have pros and cons, for instance #2 is the least amount of > work but also does nothing for saving download space when multiple GO > recipes do share dependencies. #3 and #2 are less secure compared to #1. > None of these address licensing very well... > > I know this subject came up a few months ago but I don't recall there > being a resolution and since we are starting a new dev cycle I thought > now would be a good time to get thoughts from folks who have written GO > project recipes and take steps to making updates to moving this forward, > even if just by a little bit. > > For go, we did come to a conclusion amongst the interested parties. > > The "solution" that I'm using, is what you see in meta-virt. Use the > well established fetchers to go to the source repositories. That > allows us all the re-use and well tested functionality of git, etc, > and no need to re-invent the wheel. > > I've written a tool that generates everything required for those > SRC_URIs, so for the most part, it isn't a manual process. > > My presentation on the topic was accepted to the yocto summit, and > before that, I'll make the script available .. I just haven't had time > with all the release chaos and kernel issues at the moment. > > Bruce > > > MarkA > > > > > > > > > > > -- > - Thou shalt not follow the NULL pointer, for chaos and madness await > thee at its end > - "Use the force Harry" - Gandalf, Star Trek II > > > > -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#1540): https://lists.openembedded.org/g/openembedded-architecture/message/1540 Mute This Topic: https://lists.openembedded.org/mt/90782883/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
