Thanks for replying Bruce. I wasn't expecting to see anything until next
week but you were someone I did expect to hear from.
On 2022-04-29 15:15, Bruce Ashfield wrote:
On Fri, Apr 29, 2022 at 2:59 PM Mark Asselstine
<[email protected]> wrote:
Folks,
For those of us who have had to work on GO recipes I think there has
been an open item that would be nice to get resolved in this new release
cycle.
Specifically, as with many of the modern languages GO includes its own
fetcher for 'dependencies', in GO these are referred to as 'modules'. As
it stands there is no unified or easy way to handle these and it would
be good to have this addressed in bitbake.
For example I can use two GO projects which we wanted to write recipes
for recently.
#1 buildah - https://github.com/containers/buildah - as you can see in
this project they actively maintain a 'vendor' directory and regularly
make commits to the vendor directory to keep it up-to-date. If you do
something like the following you can verify that no additional downloads
are needed during a build
$> export GOPATH=/tmp/go
$> git clone https://github.com/containers/buildah
$> cd buildah
$> go build -mod vendor ./cmd/buildah
$> ls /tmp/go/pkg/mod/
The /tmp/go/pkg/mod directory remains empty and no download msgs are
seen during the build to confirm that no downloads take place.
#2 Minio Client (MC) - https://github.com/minio/mc - as you can see in
this project there is no maintained 'vendor' directory. Similarly if you
clone and build this project you will experience additional downloads
$> export GOPATH=/tmp/go
$> git clone https://github.com/minio/mc
$> cd mc
$> go build -trimpath -tags kqueue -o ./mc
$> ls /tmp/go/pkg/mod/
This time you will find /tmp/go/pkg/mod is filled with mods and during
the build download msgs will have been displayed.
For folks that follow the mailing lists you might have seen discussions
about writing recipes for each of the "dependencies" but as you will
have noted in these threads GO projects typically have their own
versions and even specific commits of dependencies which they are
building against. In most cases you want to use what is dictated in the
project's go.mod and such to not invalidate the projects build and
runtime testing.
Solutions for writing recipes for projects like Minio Client:
Potential (actually currently in use) solution #1 is what has been done
for a recipe like k3s
(https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/k3s/k3s_git.bb).
Where each dependency has a SRC_URI and related information, which is
used to populate the vendor directory.
Potential solution #2 would be to run 'go mod vendor' to populate the
'vendor' directory, after which a patch can be generated which would be
added to the SRC_URI and as long as we capture the license information
in the recipe for all "dependencies" we can build without worries about
additional download attempts.
This was looked at (and discarded) as a solution for meta-virtualization.
It can work, but has big maintenance drawbacks around patching, CVEs.
It also doesn't allow for download caching, etc, as it has no fetcher
integration. If you want to do something like this, you need to do a
fetcher implementation, setup module proxies, etc, rather than
generating a giant patch to restore 'vendor'
Plus it makes us reliant on the golang module infrastructure, which
has shown to not always be reliable.
Agreed. In a pinch it can come in handy but long term I also believe it
has many drawbacks. I just wanted to include it as it has come up and
wanted to have it covered here.
Potential solution #3 would be to add a 'go get' to the fetcher. This
would allow transposing go.mod information into the SRC_URI, drastically
reducing the tedious work done for solution #1.
This is difficult to implement, and leaves us with challenges for
locking down versions, licensing, etc. Definitely doable, but also
makes us again reliant on the go module infrastructure, unless we
setup mirrors, etc.
Potential solution #4 extend the tooling to facilitate the SRC_URI
generation for #1.
Not necessary. This is how #1 is actually done :)
Well don't leave me hanging. Did I miss something that has been made
available that I am blissfully/ignorantly unaware of, or is this
something you have access to but hasn't been shared? Ah, more below.
Each of these have pros and cons, for instance #2 is the least amount of
work but also does nothing for saving download space when multiple GO
recipes do share dependencies. #3 and #2 are less secure compared to #1.
None of these address licensing very well...
I know this subject came up a few months ago but I don't recall there
being a resolution and since we are starting a new dev cycle I thought
now would be a good time to get thoughts from folks who have written GO
project recipes and take steps to making updates to moving this forward,
even if just by a little bit.
For go, we did come to a conclusion amongst the interested parties.
The "solution" that I'm using, is what you see in meta-virt. Use the
well established fetchers to go to the source repositories. That
allows us all the re-use and well tested functionality of git, etc,
and no need to re-invent the wheel.
I've written a tool that generates everything required for those
SRC_URIs, so for the most part, it isn't a manual process.
My presentation on the topic was accepted to the yocto summit, and
before that, I'll make the script available .. I just haven't had time
with all the release chaos and kernel issues at the moment.
Phew, so I am not ignorant, at least not in relation to the tooling.
That would be great if you can share your work and I think it goes
without saying that I appreciate your work on this.
I opted for this email instead of just pinging you on IRC as I thought
it would be fair for others that might have followed the discussion from
a few months back. Again I wasn't expecting a Friday afternoon response,
but thanks for taking the time to get back so quickly. I look forward to
seeing your scripts and talk.
Have a great weekend,
MarkA
Bruce
MarkA
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1535):
https://lists.openembedded.org/g/openembedded-architecture/message/1535
Mute This Topic: https://lists.openembedded.org/mt/90782883/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
