Hi Paul! On Wed, Dec 23, 2020 at 4:23 AM Paul Jolly <p...@myitcv.io> wrote: > > > I just can't figure out how to do this. Maybe it can't be done in `go > > list` ? Or maybe we're just missing some detail of go modules.. > > go list operates in the context of a single module (in the mode you > are interested in), so you cannot do this with a single command across > multiple modules.
This might be a real problem for us. For this post I am reducing it to `go list`, but in actuality we have a small program that we wrote which does what we need in terms of `go/build`. It works great when `GO111MODULE=off` but is more than 100x slower normally. I thought it was finally time to rewrite it in terms of `go/packages` and get rid of GO111MODULE=off. That didn't pan out, hence this post. More inline and below > > First I do a `find` for any file that has a specific comment tag, > > indicating that the package needs codegen. The results span several > > of the in-repo submodules. > > Just to check, I'm assuming the results of this find command are being > translated to a list of packages? Because the transitive dependencies > of a list of packages within a module can be done via a single go list > command. The trick is "within a module". I'll update https://github.com/thockin/go-list-modules to reflect the process more. I've added a get_codegen_deps.sh that models the behavior. Note that I really want files, not packages, so I can express the dep-graph. What do you mean by "translated to a list of packages" - which specific syntax? What I end up with is something like `go list ./path/to/dir1 ./path/to/dir2 ./path/to/dir3`. Any of those dirs might be in different modules. So `go list` tells me "main module (example.com/m) does not contain package example.com/m/path/to/dir1" and so on. Setting `GO111MODULE=off` does work, but I fear the future of that. > > For each target package, I want to get the list of all deps and > > extract the GoFiles. Then I can use that to determine if the codegen > > needs to run. > > FWIW I wrote a tool to do just this: > https://pkg.go.dev/myitcv.io@v0.0.0-20201125173645-a7167afc9e13/cmd/gogenerate > which might work in your situation. I will take a look - it seems I will need to restructure a bunch of tooling to prove it works for us or doesn't :) > > Where it breaks down is that I can't seem to `go list` all at once: > > > > ``` > > # This works within the "root" module > > $ go list -f '{{.GoFiles}}' ./subdir > > [file.go] > > This will work. > > > # This does not work across modules > > $ go list -f '{{.GoFiles}}' ./submod/used ./submod/unused > > main module (example.com/m) does not contain package > > example.com/m/submod/used > > main module (example.com/m) does not contain package > > example.com/m/submod/unused > > Per above, this will not work across module boundaries. It works with `GO111MODULE=off` which means that introducing modules is a breaking change. Can I depend on GO111MODULE=off to work the same way forever? > > # Nor does this work, even with module replacements > > $ go list -f '{{.GoFiles}}' ./staging/src/example.com/other1/used > > ./staging/src/example.com/other1/unused > > main module (example.com/m) does not contain package > > example.com/m/staging/src/example.com/other1/used > > main module (example.com/m) does not contain package > > example.com/m/staging/src/example.com/other1/unused > > ``` > > With replace directives in place this should work, but you won't be > able to use the relative path to the modules (which is in fact > interpreted as a directory): it will need to be the full > module/package path. Given a "./path/to/pkg" - how do I convert that to a module/package path? I can run `(cd $dir && go list -m)` but that is super slow. Running JUST that for each directory that needs codegen in kubernetes takes 20+ seconds. Is there a better way, short of writing my own directory-climb and parsing go.mod? > > I can run `go list` multiple times, but that's INCREDIBLY slow - most > > of these submodules have common deps that are large. This re-parses > > everything over and over. It takes almost 60 seconds just to do `cd > > $dir; go list` (on the real kubernetes repo). > > Do you have a repro of this taking 60 seconds? Because that really > shouldn't be the case with a populated local module cache. github.com/kubernetes/kubernetes ``` $ time \ find . -type f -name \*.go \ | xargs grep -l "^// *+k8s:" \ | xargs -n 1 dirname \ | sort \ | uniq \ | while read X; do \ (cd $X; go list -f '{{.Deps}}'); \ done \ > /dev/null real 0m50.488s user 0m46.686s sys 0m18.416s ``` Just running that inner `go list` with GO111MODULE=off cuts the run time in half. Compare to: ``` time \ ( \ export GO111MODULE=off; \ find . -type f -name \*.go \ | xargs grep -l "^// *+k8s:" \ | xargs -n 1 dirname \ | sort \ | uniq \ | xargs go list -e -f '{{.Deps}}' \ ) \ > /dev/null real 0m1.323s user 0m1.174s sys 0m0.567s ``` The model repo doesn't show so significantly because it is small. Kubernetes is not small. I'm happy to hear better approaches - I really don't like relying on GO111MODULE=off forever - it seems like the sort of thing that will eventually get removed. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAO_RewYxGCeZanWg_DCOPm8%2B_okO0CtSziCsng11GG0YC-G5Kw%40mail.gmail.com.
