On Fri, Jan 15, 2021 at 2:17 PM Jay Conrod <jaycon...@google.com> wrote: > > I was initially going to suggest adding the module subdirectories as > requirements to the main go.mod, then replacing those with the subdirectories. > > module example.com/m > > go 1.15 > > require ( > example.com/other1 v0.0.0 > example.com/other2 v0.0.0 > example.com/m/submod v0.0.0 > ) > > replace ( > example.com/other1 => ./staging/src/example.com/other1 > example.com/other2 => ./staging/src/example.com/other2 > example.com/m/submod v0.0.0 => ./submod > ) > > > I think you might have tried this already. It gives the same "main module ... > does not contain package" error. I believe that's a bug. I've opened #43733 > to track it.
Interesting. If that's a bug, then maybe I'll be able to do what I need once fixed. > In general, it should be possible to give 'go list' an absolute or relative > path (starting with ./ or ../) to any directory containing a package which is > part of any module in the build list. For example, some tools list > directories in the module cache to find out what package a .go file belongs > to. > > As a workaround, you could put a go.mod in an otherwise empty directory (in > /tmp or something), then require the relevant modules from the repo and > replace them with absolute paths. Then you can run 'go list' in that > directory with absolute paths of package directories. Interesting - is the difference the absolute paths vs relative? I hoped maybe `-modfile` would do the same trick, but alas not: ``` $ (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/) example.com/m/submod/used $ go list --modfile /tmp/gomodhack/go.mod /tmp/go-list-modules/submod/used/ main module (tmp) does not contain package tmp/submod/used ``` It also fails some cases: ``` (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/) example.com/m/submod/used thockin@thockin-glaptop4 go-list-modules main /$ (cd /tmp/gomodhack/; go list /tmp/go-list-modules/staging/src/example.com/other1/used/) go: finding module for package example.com/m/staging/src/example.com/other1/used cannot find module providing package example.com/m/staging/src/example.com/other1/used: unrecognized import path "example.com/m/staging/src/example.com/other1/used": reading https://example.com/m/staging/src/example.com/other1/used?go-get=1: 404 Not Found ``` It seems that is because the "main" (top-level dir) go.mod has `replace` directives with relative paths, which kubernetes really does. > Incidentally, golang.org/x/tools/go/packages will call 'go list' under the > hood in module mode. go/build might do the same, depending on how it's > invoked. 'go list' may be the best thing to use if it gives the information > you need. Yeah, I noticed. When GO111MODULE=off, everything I am doing is much faster. I'm wary of depending on that forever, though. Stepping back, I fear I am pushing the square peg into a round hole. Let me restate what I am trying to do. I want to run a slow codegen process only if the packages it depends on have ACTUALLY changed (mtime is a good enough proxy) and I don't know a priori which packages need codegen. I want to scan the file tree, find the files that need codegen, check their deps, and only then run the codegen. We do this today with `go list` and GO111MODULE=off, but I was advised at some point that x/tools/go/packages was the future-safe approach. If there's a better way, I am all ears. Tim GO111MODULE=off > On Fri, Jan 15, 2021 at 11:59 AM 'Tim Hockin' via golang-nuts > <golang-nuts@googlegroups.com> wrote: >> >> Hi. This isn't exactly burning urgent, but it is a long-term issue >> for Kubernetes. If there's anything I can do to catalyze the >> discussion - tests to run, info to dig up, etc - please let me know. >> >> On Wed, Dec 23, 2020 at 10:48 AM Tim Hockin <thoc...@google.com> wrote: >> > >> > Hi Paul! >> > >> > On Wed, Dec 23, 2020 at 4:23 AM Paul Jolly <p...@myitcv.io> wrote: >> > > >> > > > I just can't figure out how to do this. Maybe it can't be done in `go >> > > > list` ? Or maybe we're just missing some detail of go modules.. >> > > >> > > go list operates in the context of a single module (in the mode you >> > > are interested in), so you cannot do this with a single command across >> > > multiple modules. >> > >> > This might be a real problem for us. For this post I am reducing it >> > to `go list`, but in actuality we have a small program that we wrote >> > which does what we need in terms of `go/build`. It works great when >> > `GO111MODULE=off` but is more than 100x slower normally. I thought it >> > was finally time to rewrite it in terms of `go/packages` and get rid >> > of GO111MODULE=off. That didn't pan out, hence this post. >> > >> > More inline and below >> > >> > > > First I do a `find` for any file that has a specific comment tag, >> > > > indicating that the package needs codegen. The results span several >> > > > of the in-repo submodules. >> > > >> > > Just to check, I'm assuming the results of this find command are being >> > > translated to a list of packages? Because the transitive dependencies >> > > of a list of packages within a module can be done via a single go list >> > > command. >> > >> > The trick is "within a module". I'll update >> > https://github.com/thockin/go-list-modules to reflect the process >> > more. I've added a >> > get_codegen_deps.sh that models the behavior. Note that I really want >> > files, not packages, so I can express the dep-graph. >> > >> > What do you mean by "translated to a list of packages" - which specific >> > syntax? >> > >> > What I end up with is something like `go list ./path/to/dir1 >> > ./path/to/dir2 ./path/to/dir3`. Any of those dirs might be in >> > different modules. So `go list` tells me "main module (example.com/m) >> > does not contain package example.com/m/path/to/dir1" and so on. >> > Setting `GO111MODULE=off` does work, but I fear the future of that. >> > >> > > > For each target package, I want to get the list of all deps and >> > > > extract the GoFiles. Then I can use that to determine if the codegen >> > > > needs to run. >> > > >> > > FWIW I wrote a tool to do just this: >> > > https://pkg.go.dev/myitcv.io@v0.0.0-20201125173645-a7167afc9e13/cmd/gogenerate >> > > which might work in your situation. >> > >> > I will take a look - it seems I will need to restructure a bunch of >> > tooling to prove it works for us or doesn't :) >> > >> > > > Where it breaks down is that I can't seem to `go list` all at once: >> > > > >> > > > ``` >> > > > # This works within the "root" module >> > > > $ go list -f '{{.GoFiles}}' ./subdir >> > > > [file.go] >> > > >> > > This will work. >> > > >> > > > # This does not work across modules >> > > > $ go list -f '{{.GoFiles}}' ./submod/used ./submod/unused >> > > > main module (example.com/m) does not contain package >> > > > example.com/m/submod/used >> > > > main module (example.com/m) does not contain package >> > > > example.com/m/submod/unused >> > > >> > > Per above, this will not work across module boundaries. >> > >> > It works with `GO111MODULE=off` which means that introducing modules >> > is a breaking change. Can I depend on GO111MODULE=off to work the >> > same way forever? >> > >> > > > # Nor does this work, even with module replacements >> > > > $ go list -f '{{.GoFiles}}' ./staging/src/example.com/other1/used >> > > > ./staging/src/example.com/other1/unused >> > > > main module (example.com/m) does not contain package >> > > > example.com/m/staging/src/example.com/other1/used >> > > > main module (example.com/m) does not contain package >> > > > example.com/m/staging/src/example.com/other1/unused >> > > > ``` >> > > >> > > With replace directives in place this should work, but you won't be >> > > able to use the relative path to the modules (which is in fact >> > > interpreted as a directory): it will need to be the full >> > > module/package path. >> > >> > Given a "./path/to/pkg" - how do I convert that to a module/package >> > path? I can run `(cd $dir && go list -m)` but that is super slow. >> > Running JUST that for each directory that needs codegen in kubernetes >> > takes 20+ seconds. Is there a better way, short of writing my own >> > directory-climb and parsing go.mod? >> > >> > > > I can run `go list` multiple times, but that's INCREDIBLY slow - most >> > > > of these submodules have common deps that are large. This re-parses >> > > > everything over and over. It takes almost 60 seconds just to do `cd >> > > > $dir; go list` (on the real kubernetes repo). >> > > >> > > Do you have a repro of this taking 60 seconds? Because that really >> > > shouldn't be the case with a populated local module cache. >> > >> > github.com/kubernetes/kubernetes >> > >> > ``` >> > $ time \ >> > find . -type f -name \*.go \ >> > | xargs grep -l "^// *+k8s:" \ >> > | xargs -n 1 dirname \ >> > | sort \ >> > | uniq \ >> > | while read X; do \ >> > (cd $X; go list -f '{{.Deps}}'); \ >> > done \ >> > > /dev/null >> > >> > real 0m50.488s >> > user 0m46.686s >> > sys 0m18.416s >> > ``` >> > >> > Just running that inner `go list` with GO111MODULE=off cuts the run >> > time in half. >> > >> > Compare to: >> > >> > ``` >> > time \ >> > ( \ >> > export GO111MODULE=off; \ >> > find . -type f -name \*.go \ >> > | xargs grep -l "^// *+k8s:" \ >> > | xargs -n 1 dirname \ >> > | sort \ >> > | uniq \ >> > | xargs go list -e -f '{{.Deps}}' \ >> > ) \ >> > > /dev/null >> > >> > real 0m1.323s >> > user 0m1.174s >> > sys 0m0.567s >> > ``` >> > >> > The model repo doesn't show so significantly because it is small. >> > Kubernetes is not small. >> > >> > I'm happy to hear better approaches - I really don't like relying on >> > GO111MODULE=off forever - it seems like the sort of thing that will >> > eventually get removed. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/CAO_Rewa6rMW79iBHj2Jz6HfJ-tCFLFhNAhYiwDh%3DNy6M35Y91Q%40mail.gmail.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAO_RewZwtA92fg5w%2BS2xxoJfGJT_03NczfTB1Z%2BZx-onG5-4RQ%40mail.gmail.com.
