Best regards
Lukas
On Sat, Apr 30, 2022 at 12:06 AM, Bruce Ashfield wrote:
On Fri, Apr 29, 2022 at 4:21 PM Khem Raj <[email protected]> wrote:
On 4/29/22 12:15 PM, Bruce Ashfield wrote:
On Fri, Apr 29, 2022 at 2:59 PM Mark Asselstine
<[email protected]> wrote:
Folks,
For those of us who have had to work on GO recipes I think there has
been an open item that would be nice to get resolved in this new release
cycle.
Specifically, as with many of the modern languages GO includes its own
fetcher for 'dependencies', in GO these are referred to as 'modules'. As
it stands there is no unified or easy way to handle these and it would
be good to have this addressed in bitbake.
For example I can use two GO projects which we wanted to write recipes
for recently.
#1 buildah - https://github.com/containers/buildah - as you can see in
this project they actively maintain a 'vendor' directory and regularly
make commits to the vendor directory to keep it up-to-date. If you do
something like the following you can verify that no additional downloads
are needed during a build
$> export GOPATH=/tmp/go
$> git clone https://github.com/containers/buildah
$> cd buildah
$> go build -mod vendor ./cmd/buildah
$> ls /tmp/go/pkg/mod/
The /tmp/go/pkg/mod directory remains empty and no download msgs are
seen during the build to confirm that no downloads take place.
#2 Minio Client (MC) - https://github.com/minio/mc - as you can see in
this project there is no maintained 'vendor' directory. Similarly if you
clone and build this project you will experience additional downloads
$> export GOPATH=/tmp/go
$> git clone https://github.com/minio/mc
$> cd mc
$> go build -trimpath -tags kqueue -o ./mc
$> ls /tmp/go/pkg/mod/
This time you will find /tmp/go/pkg/mod is filled with mods and during
the build download msgs will have been displayed.
For folks that follow the mailing lists you might have seen discussions
about writing recipes for each of the "dependencies" but as you will
have noted in these threads GO projects typically have their own
versions and even specific commits of dependencies which they are
building against. In most cases you want to use what is dictated in the
project's go.mod and such to not invalidate the projects build and
runtime testing.
Solutions for writing recipes for projects like Minio Client:
Potential (actually currently in use) solution #1 is what has been done
for a recipe like k3s
(https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/k3s/k3s_git.bb).
Where each dependency has a SRC_URI and related information, which is
used to populate the vendor directory.
Potential solution #2 would be to run 'go mod vendor' to populate the
'vendor' directory, after which a patch can be generated which would be
added to the SRC_URI and as long as we capture the license information
in the recipe for all "dependencies" we can build without worries about
additional download attempts.
This was looked at (and discarded) as a solution for meta-virtualization.
It can work, but has big maintenance drawbacks around patching, CVEs.
It also doesn't allow for download caching, etc, as it has no fetcher
integration. If you want to do something like this, you need to do a
fetcher implementation, setup module proxies, etc, rather than
generating a giant patch to restore 'vendor'
I think a recipe tool to write recipes for go projects is going to serve
the community well, like what cargo-bitbake is in rust ecosystem. And
then bitbake fetcher could aide a bit in fetching part.
That's what I'm describing with the tool/script, it generates the
recipe elements. It could be dynamic, or as it currently stands, is
manually run to create a recipe.
Plus it makes us reliant on the golang module infrastructure, which
has shown to not always be reliable.
I would agree this was in flux before go modules were accepted as
standard way of doing dependencies, there were many solutions for
dependency management but I think we should now very much settle that it
is go modules now, since its part of core tooling in go community now a
days. So if we build tooling to use go modules underneath we wont be
painting ourselves to a corner.
There's no real painting into a corner in what I'm describing, it
isn't going around the modules and modules specification, it just
isn't using go itself to fetch. I've watched enough changes in golang,
that maybe this is the last .. maybe it isn't :) The go module format
hasn't been in flux for quite some time. It is the infrastructure that
goes through by default, has a few choke points and can have issues.
Of course the go mod infrastructure is flexible and can setup
mirrors/proxy, caching, etc, and that is what a fetcher implementation
could absolutely do.
It is the ongoing license management, maintenance/security, developer
flow, etc, that I found as a challenge.
Someone can absolutely implement a fetcher for it, set up the
variables to point to downloads for the proxy/mirrors (and avoid the
network), lock down go.mod/go.sum as the dependency declaration and
build without vendoring (and network). We talked about that model in
some of the earlier threads as well. That someone who writes it, just
isn't me at the moment :) I have something working that is independent
of those entanglements and can work even if something else is
developed.
Cheers,
Bruce
Potential solution #3 would be to add a 'go get' to the fetcher. This
would allow transposing go.mod information into the SRC_URI, drastically
reducing the tedious work done for solution #1.
This is difficult to implement, and leaves us with challenges for
locking down versions, licensing, etc. Definitely doable, but also
makes us again reliant on the go module infrastructure, unless we
setup mirrors, etc.
Potential solution #4 extend the tooling to facilitate the SRC_URI
generation for #1.
Not necessary. This is how #1 is actually done :)
Each of these have pros and cons, for instance #2 is the least amount of
work but also does nothing for saving download space when multiple GO
recipes do share dependencies. #3 and #2 are less secure compared to #1.
None of these address licensing very well...
I know this subject came up a few months ago but I don't recall there
being a resolution and since we are starting a new dev cycle I thought
now would be a good time to get thoughts from folks who have written GO
project recipes and take steps to making updates to moving this forward,
even if just by a little bit.
For go, we did come to a conclusion amongst the interested parties.
The "solution" that I'm using, is what you see in meta-virt. Use the
well established fetchers to go to the source repositories. That
allows us all the re-use and well tested functionality of git, etc,
and no need to re-invent the wheel.
I've written a tool that generates everything required for those
SRC_URIs, so for the most part, it isn't a manual process.
My presentation on the topic was accepted to the yocto summit, and
before that, I'll make the script available .. I just haven't had time
with all the release chaos and kernel issues at the moment.
Bruce
MarkA
--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
