Hello all, I just posted a series of patches for the purpose of generating recipies for a go offline build:
[PATCH 1/5] recipetool-create: add ensure_native_cmd function (openembedded.org) ( https://lists.openembedded.org/g/openembedded-core/message/165323 ) [PATCH 2/5] create_npm: reuse ensure_native_cmd from create.py (openembedded.org) ( https://lists.openembedded.org/g/openembedded-core/message/165325 ) [PATCH 3/5] poky-meta: add go vendor class for offline builds (openembedded.org) ( https://lists.openembedded.org/g/openembedded-core/message/165324 ) [PATCH 4/5] recipetool: add go recipe generator (openembedded.org) ( https://lists.openembedded.org/g/openembedded-core/message/165326 ) [PATCH 5/5] oe-selftest: add go recipe create selftest (openembedded.org) ( https://lists.openembedded.org/g/openembedded-core/message/165327 ) The series generates recipies of the following form (grpc-web as an example): > > > > SRC_URI = "git://${GO_IMPORT};nobranch=1;name=${PN};protocol=https \ > > > > file://go.mod.patch;patchdir=src/${GO_IMPORT} \ > > > > ${@go_src_uri('github.com/cenkalti/backoff',path='github.com/cenkalti/backoff/v4')} > \ > > > > ${@go_src_uri('github.com/desertbit/timer')} \ > > > > ... > > > > #TODO: Subdirectories are heuristically derived from the import path and > might be incorrect. > > > > # github.com/cenkalti/backoff/[email protected] => > 6b0e4ad0cd65431b217393bff47d1dff727b264b > > > > SRCREV_github.com.cenkalti.backoff.v4 = > "6b0e4ad0cd65431b217393bff47d1dff727b264b" > > > > GO_MODULE_PATH[github.com.cenkalti.backoff.v4] = > "github.com/cenkalti/backoff/v4" > > > > GO_MODULE_VERSION[github.com.cenkalti.backoff.v4] = "v4.1.3" > > > > # github.com/desertbit/[email protected] => > c41aec40b27f0eeb2b94300fffcd624c69b02990 > > > > SRCREV_github.com.desertbit.timer = > "c41aec40b27f0eeb2b94300fffcd624c69b02990" > > > > GO_MODULE_PATH[github.com.desertbit.timer] = "github.com/desertbit/timer" > > > > GO_MODULE_VERSION[github.com.desertbit.timer] = > "v0.0.0-20180107155436-c41aec40b27f" > > How it works: the recipetool resolves the go-module-path for each dependency in the go.mod file to a corresponding src-url (mostly git repository urls, but other vcs would work as well). The recipetools also resolves the (pseudo)-semver to the corresponding git commit. git-url and commit hash are added to the generated go-recipe and processed later during build by the go-vendor.bbclass. During 'unpack' the bbclass copies the fetched dependency sources to the 'vendor' subdirectory and generates a 'modules.txt' manifest. Most of my test projects worked right away. However, I'm sure that I missed some corner cases, as I'm not a go expert. Thus, I would be grateful for your opinion on my approach. Best regards Lukas On Sat, Apr 30, 2022 at 12:06 AM, Bruce Ashfield wrote: > > On Fri, Apr 29, 2022 at 4:21 PM Khem Raj <[email protected]> wrote: > >> >> >> >> On 4/29/22 12:15 PM, Bruce Ashfield wrote: >> >>> On Fri, Apr 29, 2022 at 2:59 PM Mark Asselstine >>> <[email protected]> wrote: >>> >>>> >>>> Folks, >>>> >>>> For those of us who have had to work on GO recipes I think there has >>>> been an open item that would be nice to get resolved in this new release >>>> cycle. >>>> >>>> Specifically, as with many of the modern languages GO includes its own >>>> fetcher for 'dependencies', in GO these are referred to as 'modules'. As >>>> it stands there is no unified or easy way to handle these and it would >>>> be good to have this addressed in bitbake. >>>> >>>> For example I can use two GO projects which we wanted to write recipes >>>> for recently. >>>> >>>> #1 buildah - https://github.com/containers/buildah - as you can see in >>>> this project they actively maintain a 'vendor' directory and regularly >>>> make commits to the vendor directory to keep it up-to-date. If you do >>>> something like the following you can verify that no additional downloads >>>> are needed during a build >>>> >>>> $> export GOPATH=/tmp/go >>>> $> git clone https://github.com/containers/buildah >>>> $> cd buildah >>>> $> go build -mod vendor ./cmd/buildah >>>> $> ls /tmp/go/pkg/mod/ >>>> >>>> The /tmp/go/pkg/mod directory remains empty and no download msgs are >>>> seen during the build to confirm that no downloads take place. >>>> >>>> #2 Minio Client (MC) - https://github.com/minio/mc - as you can see in >>>> this project there is no maintained 'vendor' directory. Similarly if you >>>> clone and build this project you will experience additional downloads >>>> >>>> $> export GOPATH=/tmp/go >>>> $> git clone https://github.com/minio/mc >>>> $> cd mc >>>> $> go build -trimpath -tags kqueue -o ./mc >>>> $> ls /tmp/go/pkg/mod/ >>>> >>>> This time you will find /tmp/go/pkg/mod is filled with mods and during >>>> the build download msgs will have been displayed. >>>> >>>> For folks that follow the mailing lists you might have seen discussions >>>> about writing recipes for each of the "dependencies" but as you will >>>> have noted in these threads GO projects typically have their own >>>> versions and even specific commits of dependencies which they are >>>> building against. In most cases you want to use what is dictated in the >>>> project's go.mod and such to not invalidate the projects build and >>>> runtime testing. >>>> >>>> Solutions for writing recipes for projects like Minio Client: >>>> >>>> Potential (actually currently in use) solution #1 is what has been done >>>> for a recipe like k3s >>>> ( >>>> https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/k3s/k3s_git.bb >>>> ). >>>> Where each dependency has a SRC_URI and related information, which is >>>> used to populate the vendor directory. >>>> >>>> Potential solution #2 would be to run 'go mod vendor' to populate the >>>> 'vendor' directory, after which a patch can be generated which would be >>>> added to the SRC_URI and as long as we capture the license information >>>> in the recipe for all "dependencies" we can build without worries about >>>> additional download attempts. >>> >>> This was looked at (and discarded) as a solution for meta-virtualization. >>> >>> It can work, but has big maintenance drawbacks around patching, CVEs. >>> It also doesn't allow for download caching, etc, as it has no fetcher >>> integration. If you want to do something like this, you need to do a >>> fetcher implementation, setup module proxies, etc, rather than >>> generating a giant patch to restore 'vendor' >> >> I think a recipe tool to write recipes for go projects is going to serve >> the community well, like what cargo-bitbake is in rust ecosystem. And >> then bitbake fetcher could aide a bit in fetching part. > > That's what I'm describing with the tool/script, it generates the > recipe elements. It could be dynamic, or as it currently stands, is > manually run to create a recipe. > > >> >> >>> >>> Plus it makes us reliant on the golang module infrastructure, which >>> has shown to not always be reliable. >> >> I would agree this was in flux before go modules were accepted as >> standard way of doing dependencies, there were many solutions for >> dependency management but I think we should now very much settle that it >> is go modules now, since its part of core tooling in go community now a >> days. So if we build tooling to use go modules underneath we wont be >> painting ourselves to a corner. > > There's no real painting into a corner in what I'm describing, it > isn't going around the modules and modules specification, it just > isn't using go itself to fetch. I've watched enough changes in golang, > that maybe this is the last .. maybe it isn't :) The go module format > hasn't been in flux for quite some time. It is the infrastructure that > goes through by default, has a few choke points and can have issues. > Of course the go mod infrastructure is flexible and can setup > mirrors/proxy, caching, etc, and that is what a fetcher implementation > could absolutely do. > > It is the ongoing license management, maintenance/security, developer > flow, etc, that I found as a challenge. > > Someone can absolutely implement a fetcher for it, set up the > variables to point to downloads for the proxy/mirrors (and avoid the > network), lock down go.mod/go.sum as the dependency declaration and > build without vendoring (and network). We talked about that model in > some of the earlier threads as well. That someone who writes it, just > isn't me at the moment :) I have something working that is independent > of those entanglements and can work even if something else is > developed. > > Cheers, > > Bruce > > > > >> >> >>> >>> >>>> >>>> Potential solution #3 would be to add a 'go get' to the fetcher. This >>>> would allow transposing go.mod information into the SRC_URI, drastically >>>> reducing the tedious work done for solution #1. >>> >>> This is difficult to implement, and leaves us with challenges for >>> locking down versions, licensing, etc. Definitely doable, but also >>> makes us again reliant on the go module infrastructure, unless we >>> setup mirrors, etc. >>> >>> >>>> >>>> Potential solution #4 extend the tooling to facilitate the SRC_URI >>>> generation for #1. >>> >>> Not necessary. This is how #1 is actually done :) >>> >>> >>>> >>>> Each of these have pros and cons, for instance #2 is the least amount of >>>> work but also does nothing for saving download space when multiple GO >>>> recipes do share dependencies. #3 and #2 are less secure compared to #1. >>>> None of these address licensing very well... >>>> >>>> I know this subject came up a few months ago but I don't recall there >>>> being a resolution and since we are starting a new dev cycle I thought >>>> now would be a good time to get thoughts from folks who have written GO >>>> project recipes and take steps to making updates to moving this forward, >>>> even if just by a little bit. >>> >>> For go, we did come to a conclusion amongst the interested parties. >>> >>> The "solution" that I'm using, is what you see in meta-virt. Use the >>> well established fetchers to go to the source repositories. That >>> allows us all the re-use and well tested functionality of git, etc, >>> and no need to re-invent the wheel. >>> >>> I've written a tool that generates everything required for those >>> SRC_URIs, so for the most part, it isn't a manual process. >>> >>> My presentation on the topic was accepted to the yocto summit, and >>> before that, I'll make the script available .. I just haven't had time >>> with all the release chaos and kernel issues at the moment. >>> >>> Bruce >>> >>> >>>> >>>> MarkA >>>> >>>> >>>> >>>> >>>> >>> >>> >>> >>> >>> >>> >> >> > > > > -- > - Thou shalt not follow the NULL pointer, for chaos and madness await > thee at its end > - "Use the force Harry" - Gandalf, Star Trek II
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#1539): https://lists.openembedded.org/g/openembedded-architecture/message/1539 Mute This Topic: https://lists.openembedded.org/mt/90782883/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
