On Fri, Apr 29, 2022 at 2:59 PM Mark Asselstine <[email protected]> wrote: > > Folks, > > For those of us who have had to work on GO recipes I think there has > been an open item that would be nice to get resolved in this new release > cycle. > > Specifically, as with many of the modern languages GO includes its own > fetcher for 'dependencies', in GO these are referred to as 'modules'. As > it stands there is no unified or easy way to handle these and it would > be good to have this addressed in bitbake. > > For example I can use two GO projects which we wanted to write recipes > for recently. > > #1 buildah - https://github.com/containers/buildah - as you can see in > this project they actively maintain a 'vendor' directory and regularly > make commits to the vendor directory to keep it up-to-date. If you do > something like the following you can verify that no additional downloads > are needed during a build > > $> export GOPATH=/tmp/go > $> git clone https://github.com/containers/buildah > $> cd buildah > $> go build -mod vendor ./cmd/buildah > $> ls /tmp/go/pkg/mod/ > > The /tmp/go/pkg/mod directory remains empty and no download msgs are > seen during the build to confirm that no downloads take place. > > #2 Minio Client (MC) - https://github.com/minio/mc - as you can see in > this project there is no maintained 'vendor' directory. Similarly if you > clone and build this project you will experience additional downloads > > $> export GOPATH=/tmp/go > $> git clone https://github.com/minio/mc > $> cd mc > $> go build -trimpath -tags kqueue -o ./mc > $> ls /tmp/go/pkg/mod/ > > This time you will find /tmp/go/pkg/mod is filled with mods and during > the build download msgs will have been displayed. > > For folks that follow the mailing lists you might have seen discussions > about writing recipes for each of the "dependencies" but as you will > have noted in these threads GO projects typically have their own > versions and even specific commits of dependencies which they are > building against. In most cases you want to use what is dictated in the > project's go.mod and such to not invalidate the projects build and > runtime testing. > > Solutions for writing recipes for projects like Minio Client: > > Potential (actually currently in use) solution #1 is what has been done > for a recipe like k3s > (https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/k3s/k3s_git.bb). > Where each dependency has a SRC_URI and related information, which is > used to populate the vendor directory. > > Potential solution #2 would be to run 'go mod vendor' to populate the > 'vendor' directory, after which a patch can be generated which would be > added to the SRC_URI and as long as we capture the license information > in the recipe for all "dependencies" we can build without worries about > additional download attempts.
This was looked at (and discarded) as a solution for meta-virtualization. It can work, but has big maintenance drawbacks around patching, CVEs. It also doesn't allow for download caching, etc, as it has no fetcher integration. If you want to do something like this, you need to do a fetcher implementation, setup module proxies, etc, rather than generating a giant patch to restore 'vendor' Plus it makes us reliant on the golang module infrastructure, which has shown to not always be reliable. > > Potential solution #3 would be to add a 'go get' to the fetcher. This > would allow transposing go.mod information into the SRC_URI, drastically > reducing the tedious work done for solution #1. This is difficult to implement, and leaves us with challenges for locking down versions, licensing, etc. Definitely doable, but also makes us again reliant on the go module infrastructure, unless we setup mirrors, etc. > > Potential solution #4 extend the tooling to facilitate the SRC_URI > generation for #1. Not necessary. This is how #1 is actually done :) > > Each of these have pros and cons, for instance #2 is the least amount of > work but also does nothing for saving download space when multiple GO > recipes do share dependencies. #3 and #2 are less secure compared to #1. > None of these address licensing very well... > > I know this subject came up a few months ago but I don't recall there > being a resolution and since we are starting a new dev cycle I thought > now would be a good time to get thoughts from folks who have written GO > project recipes and take steps to making updates to moving this forward, > even if just by a little bit. For go, we did come to a conclusion amongst the interested parties. The "solution" that I'm using, is what you see in meta-virt. Use the well established fetchers to go to the source repositories. That allows us all the re-use and well tested functionality of git, etc, and no need to re-invent the wheel. I've written a tool that generates everything required for those SRC_URIs, so for the most part, it isn't a manual process. My presentation on the topic was accepted to the yocto summit, and before that, I'll make the script available .. I just haven't had time with all the release chaos and kernel issues at the moment. Bruce > > MarkA > > > > > -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#1534): https://lists.openembedded.org/g/openembedded-architecture/message/1534 Mute This Topic: https://lists.openembedded.org/mt/90782883/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
