On Tue, Mar 1, 2022 at 10:54 AM <[email protected]> wrote: > > On Tue, Mar 1, 2022 at 02:14 PM, Bruce Ashfield wrote: > > On Tue, Mar 1, 2022 at 6:42 AM Andrei Gherzan <[email protected]> wrote: > > > On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote: > > On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via > lists.openembedded.org > <[email protected]> wrote: > > > On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan <[email protected]> wrote: > > > From: Andrei Gherzan <[email protected]> > > Compile pulls in the go.mod list requiring network. Without this, do > compile would fail with a similar error to the following: > > dial tcp: lookup proxy.golang.org: Temporary failure in name resolution > > This is something that needs to be carried in your own layers, IMHO it > isn't appropriate for core. > > It isn't about the fetching, it is the entire gap in functionality > that we are missing if go starts fetching dependencies during compile. > > A further thought is that if this is for go.mod issues, there is the > go-mod.bbclass. > > Perhaps enabling it in that class and doing a bbwarn about go fetching > dependencies would be appropriate ? > > Otherwise, someone may not know that this is happening and that a no > network configuration has no chance of working. > > I reckon that is reasonable. I'll personally go down the recipe level to > workaround this change but understanding and agreeing with the reasoning > behind this change, I want to invest a bit into trying to find a proper > solution in the core. Bruce, I know you invested a fair amount of time into > this already. Would you be willing to sync up and see how we can work > together in tackling this? > > Definitely, more ideas are good. In fact, I think there are probably > several approaches that can co-exist, depending on what a > recipe/developer needs. > > I'm in the Eastern time zone here, and will try and grab folks on IRC > to have a level set > > Bruce > > Added Zyga to CC as he is also interested in this as part of his go > development activities. > > Thanks, > Andrei > > > > -- > - Thou shalt not follow the NULL pointer, for chaos and madness await > thee at its end > - "Use the force Harry" - Gandalf, Star Trek II > > The problem in allowing downloads during compile (e.g. by go) is, that it > leads to non-reproducable builds. I'm currently facing the same issue and > would like to have a reproducable go *offline* build. > I would like to propose two ideas to workaround the go-compile fetching issue: > > First: > - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by > writing a seperate go fetcher or a wget-fetcher) and unpack the dependencies > into go projects 'vendor' folder. This forces go to compile offline. However, > one have to generate the 'modules.txt' file in the vendor folder 'manually' > during unpack. This is error prone, as there is no official documentation how > this format should look like. Anyway, I've tried this approach and it works > for me. > > Second: > - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by > writing a seperate go fetcher) and unpack the dependencies into a local > (workdir) go-path. This seemed a good solution for me as the go-path is well > defined. But for some reason 'go' fetches the zip-files during compile into > it's download-cache AGAIN, even if the source is already unpacked in the > go-path. I'll assume this is required to verify the source files integrity?! > With this approach one have to adapt 'go' to suppress this download behaviour. >
I've been doing offline builds using a constructed vendor/ directory and generated modules.txt. The only difference between what I have working and what you are suggesting (type 1), is that I've gone directly to the sources and constructed the vendor directory using the OE git fetcher. That allows all functionality to continue to work that is part of OEcore, and the build to continue. Switching out the git fetches for tarballs would be possible, I just wasn't sure how to use the proxied modules (and I wanted the history for debug). I've never had any issues with the modules.txt, as I generate it at the same time as the git fetch lines for the SRC_URI. I've also not been using information from the go.mod directly from go.proxy.org, it is information I've generated from a clone of the project and dumped via go mod. There's likely improvements I can do there, but with what I'm doing, I'm going directly to the source of the projects and doing clones, which keeps everything clear of the go infrastructure. I have a utility that I'm still cleaning up that generates the SRC_URI lines, as well as the modules.txt, when I resolve a few nagging issues, I'll make the WIP scripts available. Other projects (BSD, etc), have been doing different sorts of constructed vendor directories, but they are similar in approach. For the short term (i.e. the upcoming release), that is pretty much all we can do. There isn't enough time to implement a new go fetcher backend for bitbake. In the end, how we fetch and place the dependencies is a transport, so whether or not we fetch them ourselves, or let go do it, that part is largely the same. For now (short term), I favour vendor/, as it is workable, but not perfect. It isn't exactly efficient or pretty, but at least it seems to produce correct output, and allows all of the project capabilities to work. And of course, the approach will continue to work regardless of development on other go.mod elements. Cheers, Bruce > Please let me know your opinion on these two approaches. > > > -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#162573): https://lists.openembedded.org/g/openembedded-core/message/162573 Mute This Topic: https://lists.openembedded.org/mt/89464905/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
