On Thu, Mar 3, 2022 at 04:34 AM, Bruce Ashfield wrote: > > On Wed, Mar 2, 2022 at 4:57 PM Andrei Gherzan <[email protected]> wrote: > >> >> Mar 1, 2022 20:15:52 Bruce Ashfield <[email protected]>: >> >> >>> On Tue, Mar 1, 2022 at 10:54 AM <[email protected]> wrote: >>> >>>> On Tue, Mar 1, 2022 at 02:14 PM, Bruce Ashfield wrote: >>>> >>>> On Tue, Mar 1, 2022 at 6:42 AM Andrei Gherzan <[email protected]> wrote: >>>> >>>> >>>> On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote: >>>> >>>> On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via >>>> lists.openembedded.org >>>> <[email protected]> wrote: >>>> >>>> >>>> On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan <[email protected]> wrote: >>>> >>>> >>>> >>>> From: Andrei Gherzan <[email protected]> >>>> >>>> Compile pulls in the go.mod list requiring network. Without this, do >>>> compile would fail with a similar error to the following: >>>> >>>> dial tcp: lookup proxy.golang.org: Temporary failure in name resolution >>>> >>>> This is something that needs to be carried in your own layers, IMHO it >>>> isn't appropriate for core. >>>> >>>> It isn't about the fetching, it is the entire gap in functionality >>>> that we are missing if go starts fetching dependencies during compile. >>>> >>>> A further thought is that if this is for go.mod issues, there is the >>>> go-mod.bbclass. >>>> >>>> Perhaps enabling it in that class and doing a bbwarn about go fetching >>>> dependencies would be appropriate ? >>>> >>>> Otherwise, someone may not know that this is happening and that a no >>>> network configuration has no chance of working. >>>> >>>> I reckon that is reasonable. I'll personally go down the recipe level to >>>> workaround this change but understanding and agreeing with the reasoning >>>> behind this change, I want to invest a bit into trying to find a proper >>>> solution in the core. Bruce, I know you invested a fair amount of time >>>> into this already. Would you be willing to sync up and see how we can work >>>> together in tackling this? >>>> >>>> Definitely, more ideas are good. In fact, I think there are probably >>>> several approaches that can co-exist, depending on what a >>>> recipe/developer needs. >>>> >>>> I'm in the Eastern time zone here, and will try and grab folks on IRC >>>> to have a level set >>>> >>>> Bruce >>>> >>>> Added Zyga to CC as he is also interested in this as part of his go >>>> development activities. >>>> >>>> Thanks, >>>> Andrei >>>> >>>> >>>> >>>> -- >>>> - Thou shalt not follow the NULL pointer, for chaos and madness await >>>> thee at its end >>>> - "Use the force Harry" - Gandalf, Star Trek II >>>> >>>> The problem in allowing downloads during compile (e.g. by go) is, that it >>>> leads to non-reproducable builds. I'm currently facing the same issue and >>>> would like to have a reproducable go *offline* build. >>>> I would like to propose two ideas to workaround the go-compile fetching >>>> issue: >>>> >>>> First: >>>> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by >>>> writing a seperate go fetcher or a wget-fetcher) and unpack the >>>> dependencies into go projects 'vendor' folder. This forces go to compile >>>> offline. However, one have to generate the 'modules.txt' file in the >>>> vendor folder 'manually' during unpack. This is error prone, as there is >>>> no official documentation how this format should look like. Anyway, I've >>>> tried this approach and it works for me. >>>> >>>> Second: >>>> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by >>>> writing a seperate go fetcher) and unpack the dependencies into a local >>>> (workdir) go-path. This seemed a good solution for me as the go-path is >>>> well defined. But for some reason 'go' fetches the zip-files during >>>> compile into it's download-cache AGAIN, even if the source is already >>>> unpacked in the go-path. I'll assume this is required to verify the source >>>> files integrity?! With this approach one have to adapt 'go' to suppress >>>> this download behaviour. >>> >>> I've been doing offline builds using a constructed vendor/ directory >>> and generated modules.txt. >>> >>> The only difference between what I have working and what you are >>> suggesting (type 1), is that I've gone directly to the sources and >>> constructed the vendor directory using the OE git fetcher. That allows >>> all functionality to continue to work that is part of OEcore, and the >>> build to continue. Switching out the git fetches for tarballs would >>> be possible, I just wasn't sure how to use the proxied modules (and I >>> wanted the history for debug). >>> >>> I've never had any issues with the modules.txt, as I generate it at >>> the same time as the git fetch lines for the SRC_URI. I've also not >>> been using information from the go.mod directly from go.proxy.org, it >>> is information I've generated from a clone of the project and dumped >>> via go mod. There's likely improvements I can do there, but with what >>> I'm doing, I'm going directly to the source of the projects and doing >>> clones, which keeps everything clear of the go infrastructure. >>> >>> I have a utility that I'm still cleaning up that generates the SRC_URI >>> lines, as well as the modules.txt, when I resolve a few nagging >>> issues, I'll make the WIP scripts available. >>> >>> Other projects (BSD, etc), have been doing different sorts of >>> constructed vendor directories, but they are similar in approach. >>> >>> For the short term (i.e. the upcoming release), that is pretty much >>> all we can do. There isn't enough time to implement a new go fetcher >>> backend for bitbake. >>> >>> In the end, how we fetch and place the dependencies is a transport, so >>> whether or not we fetch them ourselves, or let go do it, that part is >>> largely the same. >>> >>> For now (short term), I favour vendor/, as it is workable, but not >>> perfect. It isn't exactly efficient or pretty, but at least it seems >>> to produce correct output, and allows all of the project capabilities >>> to work. And of course, the approach will continue to work regardless >>> of development on other go.mod elements. >> >> After reflecting on this for a while I reckon this is the fastest way >> forward while addressing the reproducibility issue. I'm wondering what we >> can do in terms of compliance? Maybe we can turn the script you were >> talking about into a recipe generator that also deals with this by >> querying the licenses of all the dependencies (direct and indirect). > > That was my rough plan, generate a recipe or have it generate an > include that recipes pull in, there are some repeating patterns go > modules, so there is some re-use to be found. > > I roughed out a process for it to work with k3s, and have a working > updated recipe that creates a vendor/ directory and doesn't touch the > network during the actual build. > > There's definitely efficiencies to be found, as the first fetch is > quite long, and there's some i/o required as the fetches secondarily > shuffled into place that go expects in a vendor directory. > > I'm trying to complete a second recipe with the generated SRC_URI > entries now (nerdctl) and I ran into an issue with the script where > some repeated fetches were breaking the vendor directory creation. I > need to spend time with that on Thursday, but after I sort that out, > I can remove the curse words from the script and do a bit of cleanup. > There's plenty of bugs, and alternate ways things can operate (maybe > some of the packaged go modules versus git, etc, etc), but since those > choices don't required bitbake/fetcher or other core changes, we have > a bit of time to iterate on a workable approach. > > Bruce
Bruce, I'm looking forward to review your approach. My biggest concern in fetching the imports from source via git is, that an 'import' my not necessarily relate to a git repository. 'go' supports multiple backends (e.g. hg, svn, etc.). That said, an import-path cannot be transformed to a git SRC_URI in a 1:1 manner. That's why I ended up downloading the modules from golang-proxy. Lukas > > > >> That being said, how can I help? It seems that there is an existing WIP >> state on this. Can I take something from it? Maybe help with cleaning up >> the script? >> -- >> Andrei Gherzan >> gpg: rsa4096/D4D94F67AD0E9640 > > > > -- > - Thou shalt not follow the NULL pointer, for chaos and madness await > thee at its end > - "Use the force Harry" - Gandalf, Star Trek II
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#162658): https://lists.openembedded.org/g/openembedded-core/message/162658 Mute This Topic: https://lists.openembedded.org/mt/89464905/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
