Re: [go-nuts] How to automatically embed module name from go.mod into executables?

['Jay Conrod' via golang-nuts]

Good news! The main package path, the containing module, and all of the
dependencies and their versions are already embedded in Go executables. You
can read that info with `go version -m path/to/exe`.

$ go install golang.org/x/tools/cmd/stringer@latest
$ go version -m ~/go/bin/stringer

/Users/jayconrod/go/bin/stringer: go1.17.2

path golang.org/x/tools/cmd/stringer

mod golang.org/x/tools v0.1.7
h1:6j8CgantCy3yc8JGBqkDLMKWqZ0RDU2g1HVgacojGWQ=

dep golang.org/x/mod v0.4.2 h1:Gz96sIWK3OalVv/I/qNygP42zyoKp3xptRVCWRFEBvo=

dep golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e
h1:WUoyKPm6nCo1BnNUvPGnFG3T5DUVem42yDJZZ4CNxMA=

dep golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=




On Fri, Oct 8, 2021 at 10:05 AM Zigster Programmer <
zigsterprogram...@gmail.com> wrote:

> One of my pet peeves with all executables, not just those created with go,
> is that determining their origin can easily be lost in the mists of time.
>
> This is a problem because I might "go get" some source code and install
> it, then not update it for many years. Then when I have some need to do the
> update, I have completely lost track of where the code came from and I
> might have long ago blown away my ~/go src directory.
>
> Then I have to do a bunch of web searches and run strings/grep on the
> executable in the hope that some URL will be exposed. Most often tho I fail.
>
> For example, consider the 'q' command in
> https://github.com/miekg/exdns/blob/master/q/q.go. 'q' is a fantastic
> utility that I've used for years yet it takes me ages to re-discover the
> origin and do an update because the 'q' executable gives no clue as to
> where it came from. It's also an awful term to use when you make a web
> search.
>
> Sorry for the long-winded prelude.
>
> What I'm leading up to is this, is there a way of automatically importing
> the module name from the go.mod file into all go generated executables such
> that we can automatically discover the origin URL without authors needing
> to write any code?
>
> If not, how do we make it so?
>
> The obvious wrong answer is to use the "embed" package.
>
> It's the wrong answer because embed doesn't allow you to look above the
> current directory, so if you have
> github.com/me/somepackage/cmd/somecommand then somecommand cannot embed
> ../../go.mod
>
> For those of you who have been around Unix for a while, what I effectively
> want to replicate is the "ident" command for go executables. Ideally all
> that such executables need to do is import an "ident" package and they get
> it for free.
>
> I think go is uniquely place because of the existence of go.mod. IOWs, it
> should be a no-brainer to get the module path embedded into the executable,
> but for the life of me I cannot work out how such an "ident" package can do
> that.
>
> So I need help.
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/f6224457-a9b8-4920-a1dd-a4197f94eba1n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGCADbbz%2BDgPrfG1A9-e-0EfpGs3LKakFD3g_ySOPTCmRsaQQg%40mail.gmail.com.

Re: [go-nuts] Does the module name have to include the repo name when you publish a Go module?

['Jay Conrod' via golang-nuts]

The module path needs to be part of a valid URL, but it doesn't have to be
the same as the repository URL. That's what my golang.org/x/mod example was
meant to demonstrate. The repository is hosted on a different domain.

On Tue, Sep 7, 2021 at 1:05 PM Dean Schulze 
wrote:

> Your first paragraph seems to say "no" to my question.  But then this
> seems to say "yes" to my question
>
> "In short, the go command needs to be able to form a valid URL from the
> module path,"
>
> In my experience it is a definite yes.
>
> On Tuesday, September 7, 2021 at 1:51:16 PM UTC-6 jayc...@google.com
> wrote:
>
>> The module path doesn't need to match the repo URL, but if other modules
>> depend on your module, then the go command needs to be able to find your
>> repository given your module path.
>>
>> The lexical constraints for module paths are documented at
>> https://golang.org/ref/mod#go-mod-file-ident. In short, the go command
>> needs to be able to form a valid URL from the module path, so only certain
>> characters may be used, there must be a dot in the first path element and
>> so on.
>>
>> The go command looks up the repository location by sending a GET request
>> to https://?go-get=1. It looks for an HTML response with a
>> meta tag that says where the module is located. For example, for the module
>> golang.org/x/mod, the go command will request
>> https://golang.org/x/mod?go-get=1 and will get a response with:
>>
>> https://go.googlesource.com/mod;>
>>
>>
>> That says the module is in a Git repository at
>> https://go.googlesource.com/mod.
>>
>> That process is explained at https://golang.org/ref/mod#vcs-find.
>> There's a special case for GitHub module paths, so if your module is hosted
>> there, you don't need to stand up a server for that URL. Also, if a path
>> component ends with ".git", ".hg", etc, the go command will use everything
>> up to there as the repository URL without performing the lookup.
>>
>>
>> On Tue, Sep 7, 2021 at 12:34 PM Dean Schulze 
>> wrote:
>>
>>> If you are going to publish a Go module does the module name have to
>>> include the repo name?  I couldn't find this documented anywhere but
>>> through trial and error I've discovered that it is required.
>>>
>>> I named my module like this and published it to github:
>>>
>>> *module key-value-mod*
>>>
>>> In another module I declared it to be a dependency
>>>
>>> *require github.com/dwschulze/key-value-mod
>>>  v0.1.0*
>>>
>>> When I ran *go get* I got this error:
>>>
>>> *$ go get github.com/dwschulze/key-va...@v0.1.0
>>> *
>>> *go: github.com/dwschulze/key-va...@v0.1.0
>>> : parsing go.mod:*
>>> *module declares its path as: key-value-mod*
>>> *but was required as: github.com/dwschulze/key-value-mod
>>> *
>>>
>>> I changed the module's name to include the repo (and retagged that
>>> commit and pushed both)
>>>
>>> *module github.com/dwschulze/key-value-mod
>>> *
>>>
>>> and then it worked.
>>>
>>> This is a private repo if that makes any difference.
>>>
>>> I also discovered that when you have module problems like this you have
>>> to clear the mod cache or you end up getting the same error over and over
>>> again.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golang-nuts...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/d7c5cc18-6c54-4379-92fe-f313a2c5bc01n%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/64f091c8-21be-4106-b7c7-07a0a55b01fan%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGCADbb3MB4bq9ON9jXb1_MWvEFj2eaZPTUMSv24v_RMmOVWhA%40mail.gmail.com.

Re: [go-nuts] Does the module name have to include the repo name when you publish a Go module?

['Jay Conrod' via golang-nuts]

The module path doesn't need to match the repo URL, but if other modules
depend on your module, then the go command needs to be able to find your
repository given your module path.

The lexical constraints for module paths are documented at
https://golang.org/ref/mod#go-mod-file-ident. In short, the go command
needs to be able to form a valid URL from the module path, so only certain
characters may be used, there must be a dot in the first path element and
so on.

The go command looks up the repository location by sending a GET request to
https://?go-get=1. It looks for an HTML response with a meta
tag that says where the module is located. For example, for the module
golang.org/x/mod, the go command will request
https://golang.org/x/mod?go-get=1 and will get a response with:

https://go.googlesource.com/mod;>


That says the module is in a Git repository at
https://go.googlesource.com/mod.

That process is explained at https://golang.org/ref/mod#vcs-find. There's a
special case for GitHub module paths, so if your module is hosted there,
you don't need to stand up a server for that URL. Also, if a path component
ends with ".git", ".hg", etc, the go command will use everything up to
there as the repository URL without performing the lookup.


On Tue, Sep 7, 2021 at 12:34 PM Dean Schulze 
wrote:

> If you are going to publish a Go module does the module name have to
> include the repo name?  I couldn't find this documented anywhere but
> through trial and error I've discovered that it is required.
>
> I named my module like this and published it to github:
>
> *module key-value-mod*
>
> In another module I declared it to be a dependency
>
> *require github.com/dwschulze/key-value-mod
>  v0.1.0*
>
> When I ran *go get* I got this error:
>
> *$ go get github.com/dwschulze/key-value-mod@v0.1.0
> *
> *go: github.com/dwschulze/key-value-mod@v0.1.0
> : parsing go.mod:*
> *module declares its path as: key-value-mod*
> *but was required as: github.com/dwschulze/key-value-mod
> *
>
> I changed the module's name to include the repo (and retagged that commit
> and pushed both)
>
> *module github.com/dwschulze/key-value-mod
> *
>
> and then it worked.
>
> This is a private repo if that makes any difference.
>
> I also discovered that when you have module problems like this you have to
> clear the mod cache or you end up getting the same error over and over
> again.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/d7c5cc18-6c54-4379-92fe-f313a2c5bc01n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGCADbbrusUHqfM-qTkWXmLrOrEZZiuwq2D28binZzdKYynCeg%40mail.gmail.com.

Re: [go-nuts] Leading dash invalid in test filename inside module

['Jay Conrod' via golang-nuts]

You're exactly right, the proxy is still in the process of migrating to Go
1.17. It should be done in the next day or two, at which point it should be
able to fetch this version.

In the mean time, you can work around it by setting GOPRIVATE=
github.com/benhoyt/goawk , which
tells the go command to fetch that module directly from vcs and not to
contact GOSUMDB for it.

On Wed, Aug 18, 2021 at 2:57 PM Ben Hoyt  wrote:

> (I wasn't quite sure whether golang-nuts or golang-dev was the best place
> for this -- sending to golang-nuts for now.)
>
> I just released a new version of GoAWK (v1.8.0) using Go version 1.17, and
> then tested installing it using "go install". However, go install complains
> with the following:
>
> $ go install github.com/benhoyt/goawk@latest
> go: downloading github.com/benhoyt/goawk v1.8.0
> go install: github.com/benhoyt/goawk@latest:
> github.com/benhoyt/goawk@v1.8.0: verifying module:
> github.com/benhoyt/goawk@v1.8.0: reading
> https://sum.golang.org/lookup/github.com/benhoyt/goawk@v1.8.0: 410 Gone
> server response: not found: create zip: -ftest: malformed file path
> "-ftest": leading dash
>
> Sure enough, I had a file in the repo's top-level directory called
> "-ftest" (the "-" is a regular ASCII hyphen or "dash"). This file was used
> in tests, to test whether the program's command line handling distinguished
> flags from file names correctly.
>
> It looks like this error is coming from Go's module code, in the checkPath
> function:
> https://cs.opensource.google/go/x/mod/+/refs/tags/v0.5.0:module/module.go;l=397
>
> However, the module docs on file naming constraints don't seem to mention
> this constraint: https://golang.org/ref/mod#zip-path-size-constraints
>
> "File and directory names within a module may consist of Unicode letters,
> ASCII digits, the ASCII space character (U+0020), and the ASCII punctuation
> characters !#$%&()+,-.=@[]^_{}~. Note that package paths may not contain
> all these characters. See module.CheckFilePath and module.CheckImportPath
> for the differences."
>
> I checked the module.CheckFilePath and module.CheckImportPath doc
> comments, and they don't seem to mention a leading dash being a problem,
> either. The /ref/mod doc *does* say that for module paths, "The leading
> path element ... must contain at least one dot and cannot start with a
> dash." However, this is stated only for module paths, not for general file
> paths.
>
> I noticed a related issue (https://github.com/golang/go/issues/45447) and
> change (https://go-review.googlesource.com/c/mod/+/316629/) for "embed"
> that looks like it would fix this issue -- the code now only does this
> "leading dash" check for file paths, not import paths.
>
> However, the "leading dash" error is a server response, so I suspect the
> code running on Google's servers is not running Go 1.17 and doesn't have
> this fix.
>
> Will this just fix itself after Google compiles the server with Go 1.17?
> Or is another fix required, even if it's just mentioning that constraint in
> the docs?
>
> -Ben
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CAL9jXCFB0Ww427mx2pbB0MJf2c_3mTSw%2BQhwbRt%3D13ufF7Q-Nw%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGCADbZio42hMnvb0knx6z2iZsKCydy-UZyo3zrqMLP5OnfJnQ%40mail.gmail.com.

Re: [go-nuts] go.sum security error

['Jay Conrod' via golang-nuts]

Ideally `go mod verify` would help in this situation, but it only compares
the contents of go.sum against the module cache, and if they're consistent
with each other but not the outside world, it won't report an error. I've
opened #47752  for this.

On Tue, Aug 17, 2021 at 9:05 AM Jay Conrod  wrote:

> I think the problem is in go.sum. If it already contains an incorrect sum
> for a module version, the go command will report a security error when
> downloading that version (if the download has a different sum) or when
> using that version (if the cached version had a different sum that appeared
> to be valid at the time).
>
> I'd suggest the following:
>
>- Make sure GOPRIVATE is set correctly (if you depend on any private
>modules). For example, 'go env -w GOPRIVATE=github.com/my-private-org'.
>- Make sure GOSUMDB and GONOSUMDB are not set.
>- Clear the module cache with 'go clean -modcache' or temporarily set
>it to an empty directory with 'export GOMODCACHE=$HOME/tmpmodcache'.
>- Manually delete lines from go.sum for publicly available modules
>(not matched by GOPRIVATE).
>- Run 'go mod tidy' to re-fetch modules, re-validate sums, and
>re-populate go.sum.
>
> When the go command downloads a module, by default it fetches a .mod and
> .zip file from proxy.golang.org. It computes a hash for each of those. If
> that hash is present in go.sum and it doesn't match, the go command will
> report a security error and delete the downloaded file. If the hash is not
> present in go.sum, the go command checks that against sum.golang.org,
> which you can think of as one big go.sum file for all public modules.
> sum.golang.org hashes the contents of proxy.golang.org, so this should
> work. The go command will then add the hash to go.sum.
>
> That procedure is pretty much unchanged since Go 1.13, when the checksum
> database was introduced.
>
> On Tue, Aug 17, 2021 at 8:08 AM Sean Liao  wrote:
>
>> Where did you install `go` from and what's the output of `go env` for
>> both versions?
>>
>> On Tuesday, August 17, 2021 at 8:25:06 AM UTC+2 Igor Chubin wrote:
>>
>>> Thank you for your answers!
>>>
>>> This is definitely not in the cache, because the problem exists
>>> everywhere,
>>> including new containers and new cloud instances.
>>>
>>> I can test it with 1.14 and 1.15 too; I don't think that the problem is
>>> specific
>>> for 1.13 only.
>>>
>>> You say, that the security error is correct: but how can it be then it
>>> is detected
>>> by only one of the Go versions and is ignored by the other?
>>> On Monday, August 16, 2021 at 7:57:49 PM UTC+2 jayc...@google.com wrote:
>>>
 This doesn't seem like a problem with Go versions. The security error
 is correct. It looks like the module author tagged v1.1.1 with this
 go.mod file
  then
 changed the tag to point to a different commit with this file
 .

 The file on proxy.golang.org is hashed and included in the checksum
 database. It looks like the hash
  there
 is h1:fx79htI3WZA9Ep4jphLFq06l3iRDimfOWTrkKOz+OAA=. That's the correct
 one to put in go.sum.

 The incorrect version may still be in your module cache. You can remove
 it with `go clean -modcache` (though this will remove everything else
 there, too).

 On Mon, Aug 16, 2021 at 9:19 AM Ian Lance Taylor 
 wrote:

> On Mon, Aug 16, 2021 at 9:11 AM Igor Chubin  wrote:
> >
> > When I generate `go.sum` with go 1.16, and try to build it with go
> of a different version (1.13 in my case), I get `SECURITY ERROR`:
> >
> > ```
> > verifying github.com/tredoe/osu...@v1.1.1/go.mod
> : checksum mismatch
> > downloaded: h1:fx79htI3WZA9Ep4jphLFq06l3iRDimfOWTrkKOz+OAA=
> > go.sum: h1:wHEjPMepmXQXkZhf9H4sQcCtmC45KuFo5VR97zG9/dY=
> >
> > SECURITY ERROR
> > This download does NOT match an earlier download recorded in go.sum.
> > The bits may have been replaced on the origin server, or an attacker
> may
> > have intercepted the download attempt.
> >
> > For more information, see 'go help module-auth'.
> > ```
> >
> > Then I fix (remove the entry and run `go mod tidy`) `go.sum` and try
> to build it again. It works with 1.13, but the problem appears then with
> 1.16.
> >
> > So there should be some incompatibility between Go 1.13 and 1.16
> (not sure exactly when it was introduced, so don't know about 1.14 and
> 1.15).
> >
> > Currently, as a workaround, I added this to my build scripts:
> >
> > ```
> > sed -i /osutil/d go.sum \
> > && go mod download github.com/tredoe/osutil
> > ```
> >
> > but it is not a 

Re: [go-nuts] go.sum security error

['Jay Conrod' via golang-nuts]

I think the problem is in go.sum. If it already contains an incorrect sum
for a module version, the go command will report a security error when
downloading that version (if the download has a different sum) or when
using that version (if the cached version had a different sum that appeared
to be valid at the time).

I'd suggest the following:

   - Make sure GOPRIVATE is set correctly (if you depend on any private
   modules). For example, 'go env -w GOPRIVATE=github.com/my-private-org'.
   - Make sure GOSUMDB and GONOSUMDB are not set.
   - Clear the module cache with 'go clean -modcache' or temporarily set it
   to an empty directory with 'export GOMODCACHE=$HOME/tmpmodcache'.
   - Manually delete lines from go.sum for publicly available modules (not
   matched by GOPRIVATE).
   - Run 'go mod tidy' to re-fetch modules, re-validate sums, and
   re-populate go.sum.

When the go command downloads a module, by default it fetches a .mod and
.zip file from proxy.golang.org. It computes a hash for each of those. If
that hash is present in go.sum and it doesn't match, the go command will
report a security error and delete the downloaded file. If the hash is not
present in go.sum, the go command checks that against sum.golang.org, which
you can think of as one big go.sum file for all public modules.
sum.golang.org hashes the contents of proxy.golang.org, so this should
work. The go command will then add the hash to go.sum.

That procedure is pretty much unchanged since Go 1.13, when the checksum
database was introduced.

On Tue, Aug 17, 2021 at 8:08 AM Sean Liao  wrote:

> Where did you install `go` from and what's the output of `go env` for both
> versions?
>
> On Tuesday, August 17, 2021 at 8:25:06 AM UTC+2 Igor Chubin wrote:
>
>> Thank you for your answers!
>>
>> This is definitely not in the cache, because the problem exists
>> everywhere,
>> including new containers and new cloud instances.
>>
>> I can test it with 1.14 and 1.15 too; I don't think that the problem is
>> specific
>> for 1.13 only.
>>
>> You say, that the security error is correct: but how can it be then it is
>> detected
>> by only one of the Go versions and is ignored by the other?
>> On Monday, August 16, 2021 at 7:57:49 PM UTC+2 jayc...@google.com wrote:
>>
>>> This doesn't seem like a problem with Go versions. The security error is
>>> correct. It looks like the module author tagged v1.1.1 with this go.mod
>>> file  then
>>> changed the tag to point to a different commit with this file
>>> .
>>>
>>> The file on proxy.golang.org is hashed and included in the checksum
>>> database. It looks like the hash
>>>  there
>>> is h1:fx79htI3WZA9Ep4jphLFq06l3iRDimfOWTrkKOz+OAA=. That's the correct
>>> one to put in go.sum.
>>>
>>> The incorrect version may still be in your module cache. You can remove
>>> it with `go clean -modcache` (though this will remove everything else
>>> there, too).
>>>
>>> On Mon, Aug 16, 2021 at 9:19 AM Ian Lance Taylor 
>>> wrote:
>>>
 On Mon, Aug 16, 2021 at 9:11 AM Igor Chubin  wrote:
 >
 > When I generate `go.sum` with go 1.16, and try to build it with go of
 a different version (1.13 in my case), I get `SECURITY ERROR`:
 >
 > ```
 > verifying github.com/tredoe/osu...@v1.1.1/go.mod
 : checksum mismatch
 > downloaded: h1:fx79htI3WZA9Ep4jphLFq06l3iRDimfOWTrkKOz+OAA=
 > go.sum: h1:wHEjPMepmXQXkZhf9H4sQcCtmC45KuFo5VR97zG9/dY=
 >
 > SECURITY ERROR
 > This download does NOT match an earlier download recorded in go.sum.
 > The bits may have been replaced on the origin server, or an attacker
 may
 > have intercepted the download attempt.
 >
 > For more information, see 'go help module-auth'.
 > ```
 >
 > Then I fix (remove the entry and run `go mod tidy`) `go.sum` and try
 to build it again. It works with 1.13, but the problem appears then with
 1.16.
 >
 > So there should be some incompatibility between Go 1.13 and 1.16 (not
 sure exactly when it was introduced, so don't know about 1.14 and 1.15).
 >
 > Currently, as a workaround, I added this to my build scripts:
 >
 > ```
 > sed -i /osutil/d go.sum \
 > && go mod download github.com/tredoe/osutil
 > ```
 >
 > but it is not a real solution, of course.
 >
 > How am I supposed to fix this problem?

 We no longer support Go 1.13.

 You can probably work around this problem temporarily and insecurely
 by setting the GONOSUMDB environment variable.  See the mentions of
 GONOSUMDB at https://pkg.go.dev/cmd/go.

 Ian

 --
 You received this message because you are subscribed to the Google
 Groups "golang-nuts" group.
 To unsubscribe from this group and 

Re: [go-nuts] Go mod ignoring project sub-modules

['Jay Conrod' via golang-nuts]

If you are using submodules, the best way to do that is to treat each
modules as a separately versioned, releasable unit. So each module has a
"require" directive for other modules it needs at the minimum usable
version (managed with 'go get' or 'go mod tidy'). You can tag versions for
each submodule by adding a directory prefix in front of the tag. For
example, if you have a module in subdirectory "a/b", the tag would look
like "a/b/v1.0.0". You don't have to tag of course; pseudo-versions
referring to commits still work fine.

This makes coordinating changes across modules a bit cumbersome though. For
example, if you have a module A that depends on module B, and A needs a
bugfix in B, you'll probably want to 1) add local replace directives in A
and B, 2) make the change in both and confirm it works, 3) remove the
replace directives, 4) commit the change in B, 5) tag a release in B, and
6) update A's dependency on B.

On Mon, Aug 16, 2021 at 11:25 PM Nicholas Bunn 
wrote:

> Thanks for the prompt response, Jay!
>
> And thanks for clearing that up, it's good to know that it's intended to
> run as such. For the purpose and scale of this project, it's a bit easier
> to keep everything together and use sub-modules where I can (this hasn't
> been an issue thus far) so I'd like to keep it this way if possible. If I
> do stick with this approach, how could I work around this? Intuitively, I'd
> just install the sub-modules individually - but this is what produced my
> issue in the first place. Is there a way I can force the modules to exist
> on their own instead of as modules in a subdirectory?
> On Monday, August 16, 2021 at 8:35:15 PM UTC+2 jayc...@google.com wrote:
>
>> Hi Nic,
>>
>> This is actually working as intended, though I don't think we've
>> adequately documented it. I've sent CL 342531
>>  to mention this
>> in the reference documentation.
>>
>> In short though, files in a subdirectory that contains a go.mod file are
>> considered part of a different module, so when you download the module in
>> the parent directory, the files in that subdirectory are not included.
>>
>> In many cases, putting multiple Go modules in the same repository can be
>> counter-intuitive. I'd only recommend it in situations where you have
>> multiple projects that need to be released and versioned independently. If
>> the projects are tightly couple and need to be released together it may be
>> better to have them in one module.
>>
>> On Mon, Aug 16, 2021 at 9:11 AM Nicholas Bunn 
>> wrote:
>>
>>> Hi all,
>>>
>>> Hoping someone can give me a bit of assistance here. I'm busy
>>> refactoring a gRPC project of mine so that all proto files are generated
>>> into a single, "protoFile" directory, instead of having the generated files
>>> living under the service's directory. My idea is to have the generated
>>> files implemented as modules, allowing any client to access them from this
>>> central location.
>>>
>>> The issue I've run into is the "module found ... but does not contain
>>> ..." one. I've hit this a couple of times before (as anyone new to Go
>>> modules does, I'm sure) and can usually solve it by cleaning modcache
>>> and/or re-initialising the modules. However this time the usual solutions
>>> haven't helped much. I've attempted to implement semantic versioning on my
>>> Github too, which didn't resolve anything either. Go get fetches the
>>> correct version of my project directory (
>>> github.com/NicholasBunn/mastersCaseStudy), but when I check my pkg
>>> directory, it's omitted all subfolders that contain Go modules (including
>>> all go services with a go.mod file).
>>>
>>> The project is hosted at github.com/NicholasBunn/mastersCaseStudy, with
>>> the generated protos living in protoFiles/go. If you'd like to try and
>>> re-create the issue, cd into
>>> mastersCaseStudy/services/authenticationService, and run "go mod tidy".
>>> This service makes use of the generated protos living in
>>> mastersCaseStudy/protoFiles/go/authenticationService/v1, which has its
>>> go.mod and go.sum files, but the "go mod tidy" fails on my side with the
>>> error mentioned above.
>>>
>>> Hoping I've just misunderstood something with Go modules and someone
>>> here can throw me onto the right path again!
>>>
>>> Thanks in advance,
>>> Nic
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golang-nuts...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/aa1c413c-f9ab-4be0-822d-4886a37bbdb9n%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To 

[go-nuts] Share your thoughts about Go’s built-in fuzzing

['Jay Conrod' via golang-nuts]

Hi, earlier this year the Go Fuzzing Team released a public preview of
built-in fuzzing support  in Go. Now
that folks have had some time to experiment with it, we’d love to learn
more about what’s working well and what could be improved. If you’ve tried
out Go’s built-in fuzzing support, please let us know how it’s worked for
you by taking this short survey:
https://google.qualtrics.com/jfe/form/SV_eOGi9cS2kEPSOjQ?s=g

Thanks,

The Go Fuzzing Team

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGCADbYvmdHmxxPnM1_kcBm3rdEy8QndoSWY_0JJugOtECUzhw%40mail.gmail.com.

Re: [go-nuts] Go mod ignoring project sub-modules

['Jay Conrod' via golang-nuts]

Hi Nic,

This is actually working as intended, though I don't think we've adequately
documented it. I've sent CL 342531
 to mention this in
the reference documentation.

In short though, files in a subdirectory that contains a go.mod file are
considered part of a different module, so when you download the module in
the parent directory, the files in that subdirectory are not included.

In many cases, putting multiple Go modules in the same repository can be
counter-intuitive. I'd only recommend it in situations where you have
multiple projects that need to be released and versioned independently. If
the projects are tightly couple and need to be released together it may be
better to have them in one module.

On Mon, Aug 16, 2021 at 9:11 AM Nicholas Bunn 
wrote:

> Hi all,
>
> Hoping someone can give me a bit of assistance here. I'm busy refactoring
> a gRPC project of mine so that all proto files are generated into a single,
> "protoFile" directory, instead of having the generated files living under
> the service's directory. My idea is to have the generated files implemented
> as modules, allowing any client to access them from this central location.
>
> The issue I've run into is the "module found ... but does not contain ..."
> one. I've hit this a couple of times before (as anyone new to Go modules
> does, I'm sure) and can usually solve it by cleaning modcache and/or
> re-initialising the modules. However this time the usual solutions haven't
> helped much. I've attempted to implement semantic versioning on my Github
> too, which didn't resolve anything either. Go get fetches the correct
> version of my project directory (github.com/NicholasBunn/mastersCaseStudy),
> but when I check my pkg directory, it's omitted all subfolders that contain
> Go modules (including all go services with a go.mod file).
>
> The project is hosted at github.com/NicholasBunn/mastersCaseStudy, with
> the generated protos living in protoFiles/go. If you'd like to try and
> re-create the issue, cd into
> mastersCaseStudy/services/authenticationService, and run "go mod tidy".
> This service makes use of the generated protos living in
> mastersCaseStudy/protoFiles/go/authenticationService/v1, which has its
> go.mod and go.sum files, but the "go mod tidy" fails on my side with the
> error mentioned above.
>
> Hoping I've just misunderstood something with Go modules and someone here
> can throw me onto the right path again!
>
> Thanks in advance,
> Nic
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/aa1c413c-f9ab-4be0-822d-4886a37bbdb9n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGCADbZWkk%3DMz1G_vgU57hnaUd3ONbG4XbzS4p2HO3Gcqe7nqQ%40mail.gmail.com.

Re: [go-nuts] go.sum security error

['Jay Conrod' via golang-nuts]

This doesn't seem like a problem with Go versions. The security error is
correct. It looks like the module author tagged v1.1.1 with this go.mod file
 then
changed the tag to point to a different commit with this file
.

The file on proxy.golang.org is hashed and included in the checksum
database. It looks like the hash
 there
is h1:fx79htI3WZA9Ep4jphLFq06l3iRDimfOWTrkKOz+OAA=.
That's the correct one to put in go.sum.

The incorrect version may still be in your module cache. You can remove it
with `go clean -modcache` (though this will remove everything else there,
too).

On Mon, Aug 16, 2021 at 9:19 AM Ian Lance Taylor  wrote:

> On Mon, Aug 16, 2021 at 9:11 AM Igor Chubin  wrote:
> >
> > When I generate `go.sum` with go 1.16, and try to build it with go of a
> different version (1.13 in my case), I get `SECURITY ERROR`:
> >
> > ```
> > verifying github.com/tredoe/osutil@v1.1.1/go.mod: checksum mismatch
> > downloaded: h1:fx79htI3WZA9Ep4jphLFq06l3iRDimfOWTrkKOz+OAA=
> > go.sum: h1:wHEjPMepmXQXkZhf9H4sQcCtmC45KuFo5VR97zG9/dY=
> >
> > SECURITY ERROR
> > This download does NOT match an earlier download recorded in go.sum.
> > The bits may have been replaced on the origin server, or an attacker may
> > have intercepted the download attempt.
> >
> > For more information, see 'go help module-auth'.
> > ```
> >
> > Then I fix (remove the entry and run `go mod tidy`) `go.sum` and try to
> build it again. It works with 1.13, but the problem appears then with 1.16.
> >
> > So there should be some incompatibility between Go 1.13 and 1.16 (not
> sure exactly when it was introduced, so don't know about 1.14 and 1.15).
> >
> > Currently, as a workaround, I added this to my build scripts:
> >
> > ```
> > sed -i /osutil/d go.sum \
> > && go mod download github.com/tredoe/osutil
> > ```
> >
> > but it is not a real solution, of course.
> >
> > How am I supposed to fix this problem?
>
> We no longer support Go 1.13.
>
> You can probably work around this problem temporarily and insecurely
> by setting the GONOSUMDB environment variable.  See the mentions of
> GONOSUMDB at https://pkg.go.dev/cmd/go.
>
> Ian
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcV56QDp1TXTaNsr%2B1UezWmoMbYRhk8iN58bDRzJq83xkA%40mail.gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGCADbZu-XbqPfj81u0Nh36vSYAP6CC31XF_vr%3DuvfZ1-s8jDQ%40mail.gmail.com.

Re: [go-nuts] `go list` across multiple modules?

['Jay Conrod' via golang-nuts]

Despite what the blog post says, Go 1.17 won't drop support for GO111MODULE
or GOPATH mode. It will be more or less the same as GOPATH mode.

We'll continue to work on these issues and others that are blocking people
from migrating. I apologize for the slow progress–we have a lot going on.

On Wed, Aug 11, 2021 at 12:34 PM Tim Hockin  wrote:

> Hi all.  I realized that https://blog.golang.org/go116-module-changes
> says "We plan to drop support for GOPATH mode in Go 1.17. In other words,
> Go 1.17 will ignore GO111MODULE".  This will  break Kubernetes - our builds
> will be orders of magnitude slower.
>
> AFAICT, https://github.com/golang/go/issues/43806 has had no progress and
> https://github.com/golang/go/issues/43733 is still open, too.  I have
> zero confidence that these are the ONLY issues, and until I can get past
> them, I can't really find out.
>
> Please advise - how should I proceed?
>
> Tim
>
>
> On Wed, Jan 20, 2021 at 12:57 PM Tim Hockin  wrote:
>
>> As long as these things are bugs and not "that's not how it works or will
>> ever work", I have some hope.  This is a back-burner exploration for me, so
>> I am not in a panic.  Every now and again I will circle back and see if
>> anything new happens :)
>>
>> On Wed, Jan 20, 2021 at 10:50 AM Jay Conrod  wrote:
>>
>>> You appear to have discovered another new bug. Sorry for that. I've
>>> opened #43806  to track it.
>>>
>>> With that bug, it doesn't look like `go list` with local paths (relative
>>> or absolute) in a repository with nested replacement directories will work
>>> (at least not yet).
>>>
>>> `go list` should still work on full packages paths, but I'm not sure
>>> there's an easy way to resolve directory paths to package paths, other than
>>> by walking up the directory tree to the nearest go.mod and reading the
>>> module path from there.
>>>
>>> On Tue, Jan 19, 2021 at 8:06 PM Tim Hockin  wrote:
>>>


 On Tue, Jan 19, 2021 at 7:54 AM Jay Conrod 
 wrote:

> By the way, I'm not sure if you're already doing this, but if you can
> batch all of the `go list` runs (or go/packages
> .Load) together per module (passing
> an argument per package), that will be much faster than loading individual
> packages with separate `go list` calls. It will save `go list` from having
> to load the module graph and common sets of dependencies multiple times.
>
> If you need to load all the packages in a module, you can also use an
> argument like ./... from the root directory of the module to load
> everything there (excluding modules in subdirectories).
>

 Yeah, I am trying that but tripping on the "does not contain package"
 error.


>
> On Tue, Jan 19, 2021 at 10:02 AM Jay Conrod 
> wrote:
>
>> > Interesting - is the difference the absolute paths vs relative?
>>
>> It looks like the bug has to do with whether the directory is below
>> the main module root directory or not. If it is, the go command takes a
>> path that assumes it's part of the main module, which it's not.
>>
>> > I hoped maybe `-modfile` would do the same trick, but alas not:
>>
>> -modfile lets you change the effective content of go.mod but not the
>> module root directory. Unfortunately it doesn't look like that can be 
>> used
>> to work around the issue.
>>
>> > It seems that is because the "main" (top-level dir) go.mod has
>> > `replace` directives with relative paths, which kubernetes really
>> > does.
>>
>> You may need to copy those over to the tmp go.mod and adjust the
>> paths. Sorry this has gotten pretty involved.
>>
>> > Yeah, I noticed.  When GO111MODULE=off, everything I am doing is
>> much
>> > faster.  I'm wary of depending on that forever, though.
>>
>> Module-aware mode is quite a bit more complicated than GOPATH mode,
>> so to some degree it's not surprising it's slower... it's surprising that
>> it's a LOT slower though. I expect there's some optimization work for us 
>> to
>> do in the next development cycle.
>>
>> We would eventually like to deprecate GOPATH mode though, so it's a
>> good idea not to depend on it in new tooling today. 'go list' should be
>> fine to get package dependency info in either module mode or GOPATH mode.
>> go/packages  is useful if you need
>> additional information on top of that (parsed syntax trees, type info).
>>
>> > I want to run a slow codegen process only if the packages it depends
>> > on have ACTUALLY changed (mtime is a good enough proxy) and I don't
>> > know a priori which packages need codegen.  I want to scan the file
>> > tree, find the files that need codegen, check their deps, and only
>> > then run the codegen.
>>
>> How much dependency 

Re: [go-nuts] Re: go install an/import/path@revision and /vendor in Go 1.16

['Jay Conrod' via golang-nuts]

Indeed, this should be better documented. I've mailed CL 340509
 to mention in a
couple places in /ref/mod that vendor directories are omitted from zip
files.

On Fri, Aug 6, 2021 at 3:55 AM Brian Candler  wrote:

> On Friday, 6 August 2021 at 11:46:24 UTC+1 Konstantin Khomoutov wrote:
>
>> On Fri, Aug 06, 2021 at 12:35:16AM -0700, Brian Candler wrote:
>>
>> > The key point I just learned from #40276:
>> > "Vendor directories are not included in module zip files."
>> >
>> > I found this surprising because -mod=vendor is now the default (here
>> > ) - therefore, I expected the
>> vendor
>> > directory, if present in the source, always to be used unless
>> explicitly
>> > disabled.
>>
>> An interesting point regarding our particular case - which I sadly failed
>> to
>> mention - is that we do not use any module proxies for own internal
>> projects,
>> and the hostname of our GitLab instance is listed in the GONOPROXY
>> environment
>> variable, so technically `go install` did not download a "classic"
>> module,
>> which is a ZIP archive file, but has rather performed the usual sequence
>> of
>> issuing a HTTP GET request with the "?go-get=1" query parameter followed
>> by a
>> shallow clone of the target Git repository.
>> Hence technically the "vendor" directory with its full contents was
>> available.
>
>
> However according to #40276:
>
> "Even when go connects directly to a repository instead of a
> proxy, it still generates zip files so that builds work consistently no
> matter
> how modules are fetched. Those zip files don't contain nested modules or
> vendor
> directories."
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/9f4d48bd-fecf-46bf-aa85-5115f0513f53n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGCADbbdnWR-WU-1YeyrsWwX2sreBkAmqrs-bzYwgFwrQL8FhA%40mail.gmail.com.

Re: [go-nuts] Showing effective replace in go modules

['Jay Conrod' via golang-nuts]

The go command only applies replace directives from the main module's
go.mod file. In this example, replace directives in your module B would be
applied, but replace directives in A (or any other dependency) would be
ignored. This is important for avoiding conflicts that can't easily be
resolved (for example, you depend on X and Y, and they both depend on
incompatible forked versions of Z).

The command 'go list -m -json $modulepath' shows information about any
module you depend on. In the output, the Replace field will be set if the
module is replaced.

On Thu, Jul 29, 2021 at 9:54 PM Amit Saha  wrote:

> Say, i have a module A (github.com/username/foo/v1)  with a replace of
> a third-party package in its go.mod:
>
> replace github.com/dgrijalva/jwt-go => github.com/golang-jwt/jwt
> v3.2.1+incompatible
>
> Now, i  require module A/v1 package from module A in my module B
> containing the main package:
>
>  require (
> github.com/username/foo/v1
>  )
>
> I want to make sure that, in my module B, "github.com/golang-jwt/jwt
> v3.2.1" is pulled in.
>
> I could verify that using the https://github.com/Helcaraxan/gomod tool:
>
> $ ~/go/bin/gomod reveal | grep jwt
> 'github.com/dgrijalva/jwt-go' is replaced:
>github.com/launchdarkly/ld-relay/v6 -> github.com/golang-jwt/jwt @
> v3.2.1+incompatible
>
> Is there a way to verify the above using one of the go mod or go
> build/list commands?
>
> Thanks,
> Amit.
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CANODV3kGLUnA0cz_NTBZnYL5%2BrO7rec4WD6FvBqz9ohfNLMMRw%40mail.gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGCADbYE9mqCyBqGcC5AFF0xKZektLTM6Nw%3D6uOF%2BN4nHk_BSw%40mail.gmail.com.

Re: [go-nuts] how to purge invalid tags from GOPROXY?

['Jay Conrod' via golang-nuts]

As Dan mentioned, the version containing the retractions must be the
highest version, so you'd need to tag v1.4.3 or higher. If that version
retracts itself, it won't appear in the version list either.

On Thu, Jul 1, 2021 at 1:58 AM Sebastien Binet  wrote:

> Hi Jay,
>
> On Thu Jul 1, 2021 at 00:45 CET, Jay Conrod wrote:
> > Hi Sebastien, once a version is in proxy.golang.org, it usually can't be
> > removed. This is important to ensure that builds continue working when a
> > repository disappears upstream.
> >
> > You may want to publish a new version with a retract directive in
> > go.mod,
> > marking the earlier versions as invalid. In Go 1.16 and higher, the go
> > command won't automatically upgrade to a retracted version. The version
> > containing the retractions may retract itself. Retract Module Versions
> >  is an
> > interactive tutorial explaining how to use this feature.
> >
> > Even with retractions in place, you won't be able to reuse the old
> > version
> > numbers. They'll be hidden, but still available to any users that depend
> > on
> > them.
>
> thanks for the pointer to the tutorial.
> I think I did apply all the expected operations on my repo:
>
> - tagged a v0.4.1 version with the `retract v1.4.2` stanza:
>
> https://github.com/go-pdf/fpdf/pull/10/files#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6
> - waited for a minute or so and issued:
>
> $> go list -versions -m
> github.com/go-pdf/fpdf v0.1.0 v0.2.0 v0.3.0 v0.3.1 v0.4.0 v0.4.1 v1.4.2
>
> I also requested v0.4.1 both on the pkg.go.dev site and explicitly in a
> dummy module with `go get github.com/go-pdf/fpdf@v0.4.1`
> :
>
> $> mkdir foo && cd ./foo
> $> go mod init foo
> $> go get -u -v github.com/go-pdf/fpdf@v0.4.1
> github.com/go-pdf/fpdf
>
> $> go get -u -v github.com/go-pdf/fpdf
> go get: github.com/go-pdf/fpdf@v1.4.2: parsing go.mod:
> module declares its path as: github.com/phpdave11/gofpdf
> but was required as: github.com/go-pdf/fpdf
>
>
> I am probably handing it the wrong way.
> (I am using go-tip at 9d65578b83)
>
> thanks again,
> -s
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGCADbbx6eBK9o2jZ8AbcYVFKvseR3%2B%2Bg%3Dat5Q3zxEijgPWUdQ%40mail.gmail.com.

Re: [go-nuts] how to purge invalid tags from GOPROXY?

['Jay Conrod' via golang-nuts]

Hi Sebastien, once a version is in proxy.golang.org, it usually can't be
removed. This is important to ensure that builds continue working when a
repository disappears upstream.

You may want to publish a new version with a retract directive in go.mod,
marking the earlier versions as invalid. In Go 1.16 and higher, the go
command won't automatically upgrade to a retracted version. The version
containing the retractions may retract itself. Retract Module Versions
 is an
interactive tutorial explaining how to use this feature.

Even with retractions in place, you won't be able to reuse the old version
numbers. They'll be hidden, but still available to any users that depend on
them.

On Fri, Jun 25, 2021 at 1:32 AM Sebastien Binet  wrote:

> hi there,
>
> I've forked a fork of a Go package (github.com/jung-kurt/gofpdf).
> But as the modus operandi of such a thing is rather unwieldy in github
> (PRs would always be created by default against the original package
> instead of my fork), I've deleted the forked repo, created a new repo
> with the same name and pushed the original git tree there.
>
> so now, my PRs are, by default against my fork, which is great.
>
> unfortunately, I've also decided that my fork would start at v0.1.0
> (what would be the last commit from the old gofpdf package) while the
> original package was at v1.4.2
>
> and it seems GOPROXY cached it somehow:
>
> $> docker run --rm -it golang
> root@clrinfopc42:/go# cd src/
> root@clrinfopc42:/go/src# mkdir foo
> root@clrinfopc42:/go/src# cd foo/
> root@clrinfopc42:/go/src/foo# go mod init foo
> go: creating new go.mod: module foo
> root@clrinfopc42:/go/src/foo# go version
> go version go1.16.5 linux/amd64
> root@clrinfopc42:/go/src/foo# cat >> go.mod
>
> require github.com/go-pdf/fpdf v0.3.1
> ^C
> root@clrinfopc42:/go/src/foo# cat >> main.go
> package main
>
> import _ "github.com/go-pdf/fpdf"
>
> func main() {}
> ^C
>
> root@clrinfopc42:/go/src/foo# go mod tidy
> go: downloading github.com/go-pdf/fpdf v0.3.1
>
> root@clrinfopc42:/go/src/foo# go build -v
> github.com/go-pdf/fpdf
> foo
>
> root@clrinfopc42:/go/src/foo# go get -u -v github.com/go-pdf/fpdf@latest
> go: downloading github.com/go-pdf/fpdf v1.4.2
> go get: github.com/go-pdf/fpdf@v0.3.1 updating to
> github.com/go-pdf/fpdf@v1.4.2: parsing go.mod:
> module declares its path as: github.com/phpdave11/gofpdf
> but was required as: github.com/go-pdf/fpdf
>
> root@clrinfopc42:/go/src/foo# go get -x -u -v
> github.com/go-pdf/fpdf@latest
> # get https://proxy.golang.org/github.com/@v/list
> # get https://proxy.golang.org/github.com/go-pdf/fpdf/@v/list
> # get https://proxy.golang.org/github.com/go-pdf/@v/list
> # get https://proxy.golang.org/github.com/@v/list: 410 Gone (0.107s)
> # get https://proxy.golang.org/github.com/go-pdf/fpdf/@v/list: 200 OK
> (0.107s)
> # get https://proxy.golang.org/github.com/go-pdf/@v/list: 410 Gone
> (0.107s)
> go get: github.com/go-pdf/fpdf@v0.3.1 updating to
> github.com/go-pdf/fpdf@v1.4.2: parsing go.mod:
> module declares its path as: github.com/phpdave11/gofpdf
> but was required as: github.com/go-pdf/fpdf
>
> of course, it works if I give the explicit new tag of go-pdf/fpdf:
>
> root@clrinfopc42:/go/src/foo# go get -u -v github.com/go-pdf/fpdf@v0.4.0
> # get https://proxy.golang.org/github.com/@v/v0.4.0.info
> # get https://proxy.golang.org/github.com/go-pdf/fpdf/@v/v0.4.0.info
> # get https://proxy.golang.org/github.com/go-pdf/@v/v0.4.0.info
> # get https://proxy.golang.org/github.com/@v/v0.4.0.info: 410 Gone
> (0.223s)
> # get https://proxy.golang.org/github.com/go-pdf/@v/v0.4.0.info: 410 Gone
> (0.223s)
> # get https://proxy.golang.org/github.com/go-pdf/fpdf/@v/v0.4.0.info: 200
> OK (0.226s)
> go: downloading github.com/go-pdf/fpdf v0.4.0
> # get https://proxy.golang.org/github.com/go-pdf/fpdf/@v/v0.4.0.zip
> # get https://proxy.golang.org/github.com/go-pdf/fpdf/@v/v0.4.0.zip: 200
> OK (0.370s)
> # get https://proxy.golang.org/github.com/go-pdf/fpdf/@v/list
> # get https://proxy.golang.org/github.com/go-pdf/fpdf/@v/list: 200 OK
> (0.029s)
> [...]
> go get: upgraded github.com/go-pdf/fpdf v0.3.1 => v0.4.0
>
>
> any idea how one could purge the tags (coming from
> {jung-kurt,phpdave11}/gofpdf?
> I understand that comes a bit at odds with the way the GOPROXY ledger is
> supposed to work though...
>
> any idea on how to work around that?
> thanks.
>
> cheers,
> -s
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CCCK5I3B9536.1RAC82P987RAU%40zoidberg
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To 

Re: [go-nuts] proposal: infer module name via version control when running `go mod init`

['Jay Conrod' via golang-nuts]

The go command used to infer the module path from .git/config for github
origins specifically. However, that was removed in 1.13, based on
discussion in #27951 . It wasn't
a strong enough signal, and since it's important to get the module path
right early on, it seemed better to require that the path be set explicitly
when it can't be inferred from the location within $GOPATH/src.

On Wed, May 12, 2021 at 11:01 AM Jon Calhoun  wrote:

> I suspect that the VCS path gives nearly the same level of confidence in
> practice. My personal experience shows that it is pretty rare for the two
> not to line up, and when it does the developer made a conscious decision to
> do so and knows to work around it.
>
> I'm also not suggesting that we remove existing behavior. go mod init
>  is still valid so we don't lose robustness there.
>
>
> On Wednesday, May 12, 2021 at 10:42:29 AM UTC-4 Bryan C. Mills wrote:
>
>> go mod init should only infer a module path when it can be certain that
>> path is correct.
>>
>> For packages already stored in GOPATH/src, it knows the correct path with
>> high confidence: the path relative to GOPATH/src is the path by which the
>> packages in the module are imported when in GOPATH mode, so it is safe to
>> assume the same path in module mode.
>>
>> On the other hand, a module hosted at a particular VCS path can be served
>> through an arbitrarily different path using go-import
>>  metadata
>> , and it could also be a private
>> fork intended for use only with a replace directive
>>  (or intended to
>> temporarily host a change while an upstream PR is outstanding). So we
>> cannot assume that the VCS path necessarily corresponds to the module path
>> — it seems more robust to ask the user to supply the intended path
>> explicitly.
>>
>> On Wednesday, May 12, 2021 at 9:25:48 AM UTC-4 jonca...@gmail.com wrote:
>>
>>> After more thought, the proposal should be altered to only support git
>>> in the first pass. It would cover a majority of users, wouldn't affect the
>>> experience in other VCS, and would simplify the initial implementation.
>>>
>>> Doing every VCS at once would require someone learning how every VCS
>>> defines the default remote repository (if it even does), and building off
>>> that. It would be better to do each individually and let someone familiar
>>> with each VCS take charge of that specific PR using existing VCS
>>> implementations as a pattern to follow.
>>> On Wednesday, May 12, 2021 at 7:28:11 AM UTC-4 ohir wrote:
>>>
 Dnia 2021-05-11, o godz. 11:14:24
 Jon Calhoun  napisał(a):

 > it saves me a handful of keystrokes at best - so I understand if it
 doesn't seem
 > worth the effort.

 Right.

 > It feels close enough to the current behavior to be a good addition

 Gopath is retiring soon. Anyway, Someone needs to do many more
 keystrokes
 to int repository anyway.

 > , but I admittedly haven't explored how easy/hard it would be to
 implement.

 Look at the list of currently supported VCSes to get a feel of the
 task.
 For the starter: infer module name from a Fossil's checkout file (it is
 a sqlite db).

 Hoe this helps

 > Best,
 > Jon

 --
 Wojciech S. Czarnecki
 << ^oo^ >> OHIR-RIPE

>>> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/7beb2ab6-940c-451c-bee6-45ccf73ad8edn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGCADba09WLXZ%3D9oYK9OiZFNQ7%2BU_PUaJtNra0mK6KzK%2BDfOtw%40mail.gmail.com.

Re: [go-nuts] Trying to use a tool in my build (a friction log)

['Jay Conrod' via golang-nuts]

> Is it by design that these two commands ['go mod tidy' and 'go mod
vendor'] are unstable like that?

That is more or less by design, but the behavior has evolved over time and
it may be worth revisiting.

This was discussed in #29058 .
The rationale was that because 'go mod vendor' deletes and rebuilds the
entire vendor directory, it would be dangerous to do that implicitly inside
another command like 'go mod tidy', since people frequently make local
changes to vendor, like when preparing a patch for upstream.

That discussion was a couple years ago, before vendoring was enabled
automatically. At the time, you had to specify -mod=vendor with build
commands.

A better user experience might be 1) support for incremental changes to the
vendor directory via 'go get', 'go mod tidy', 2) detection of local changes
to the vendor directory so we can avoid overwriting them.

I've opened #45161  to restart
that discussion.

> This tool in particular seems to try to read a DB file from a subdir of
the directory its source code came from (as per `runtime.Caller(0)`) which
...doesn't work at all with vendoring, since the dir doesn't contain Go
code.

> AFAIK there's not a way to ask `go mod vendor` to include the whole repo
is there?  I can't find a spec for modules.txt - it would be super cool if
I could add some tag in there...  As it stands, all this was for naught.

Is there another way to include and locate that file? Even if it were
vendored, runtime.Caller(0) may be unreliable when -trimpath is used, and
the files will be read-only in the module cache when vendoring is not used.

'go mod vendor' only pulls in directories for packages needed to build
packages in the main module and their tests ('go list -test -deps ./...',
essentially). I don't think we even include testdata or packages imported
by those packages' tests.

We don't have a spec for vendor/modules.txt. it basically contains enough
information for the go command to know which packages are vendored and
which modules they came from.

On Sat, Mar 20, 2021 at 12:47 PM Manlio Perillo 
wrote:

> Il giorno venerdì 19 marzo 2021 alle 23:20:49 UTC+1 Tim Hockin ha scritto:
>
>> Thanks for feedback, comments below
>>
>> On Thu, Mar 18, 2021 at 3:35 PM Jay Conrod  wrote:
>>
>>>
>>> > [...]
>
>>
>>
>>> *2. "writing go.mod cache" error messages*
>>>
>>> This error message should be a lot better. Sorry about that.
>>>
>>
>> I mean, "mkdir /home/thockin/go: not a directory" is pretty concise -
>> it's not a directory (on purpose) because I really don't want Go randomly
>> deciding to write stuff to my homedir (which I suspected was happening, and
>> that's why I made it a non-directory :)
>>
>>
>
> If you don't want the go tool to write files to your home directory, you
> can just define a custom GOPATH or GOMODCACHE.  As an example, on my
> sysytem I have set GOPATH to ~/.local/lib/go, and this directory is
> excluded from the backup.
>
> Manlio
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/8002edf8-1a9e-44a6-a427-3fbc2b4e1dc1n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGCADbYLzKJi7AYdgzyJmm4WHHe4GWgQ%3DseFfxLPOLmpL5Bg3w%40mail.gmail.com.

Re: [go-nuts] Trying to use a tool in my build (a friction log)

['Jay Conrod' via golang-nuts]

Hey Tim, thanks for writing this up. These kinds of reports are helpful.
I'll try to respond to a few problems individually.

*1. Command installation with "go install"*

There are basically two ways to do this, depending on whether you want to
install the command within the context of your current module or "globally"
ignoring go.mod, go.sum, and vendor.

To install a command globally (you can replace @latest with a specific
version like @v1.2.3):

go install github.com/google/go-licenses@latest


This form (with the @version suffix) is new in Go 1.16
.

Installing a command within the context of the current module is more
complicated. I'd suggest this:

# Add a dependency on the tool (only need to do this once)
go get -d github.com/google/go-licenses

# Import the command from tools.go

# See below


# Sync vendor directory (only if vendoring; only after changing go.mod with
'go get' above)
go mod vendor

# Install the command
go install github.com/google/go-licenses

# Alternatively, build and write to some other directory
go build -o bin/go-licenses github.com/google/go-licenses


The problem with this workflow is that 'go mod tidy' removes the
requirement on the command's module because nothing imports it. 'go mod
vendor' won't vendor the package for the same reason. There is a workaround
documented on the Wiki : import
the command package from a file (usually named tools.go) that is excluded
by a build constraint (usually "tools").

// +build tools

package tools

import _ "github.com/google/go-licenses"


This works, but personally, I don't find this to be an intuitive solution.
At the moment, we don't have a concrete proposal to improve it though.
There's some discussion at #25922
.

*2. "writing go.mod cache" error messages*

This error message should be a lot better. Sorry about that.

Most module-aware commands need to download files into the module cache.
The location of the cache can be set by setting the GOMODCACHE environment
variable (or 'go env -w GOMODCACHE=/some/path'). That defaults to
$GOPATH/pkg/mod, and GOPATH defaults to $HOME/go. So by default, the module
cache is at $HOME/go/pkg/mod. It can't create that directory though.

The error message should explain what went wrong and how to fix it, and it
really shouldn't be repeated. I've opened #45113
 for this.

*3. 'go get' updating go.mod and go.sum*

This is an unfortunate consequence of 'go get' being overloaded for
downloading, installing, and managing package versions. Ideally, we want to
separate these roles into different commands: 'go get' should be used to
manage dependencies (changing requirements in go.mod and go.sum), and 'go
install' should be used to install commands.

We've taken a couple steps toward this in 1.16. The 'go install cmd@version'
form is new. Also, 'go install', 'go build', and other commands no longer
change go.mod or go.sum automatically when something is missing.

We plan to deprecate 'go get' for installing commands. In Go 1.17, 'go get'
will print a warning when the -d flag is not used and the arguments are
main packages. In Go 1.18, we plan to remove that functionality entirely
(the -d flag will always be on). #40276
 is where that deprecation was
proposed.

On Thu, Mar 18, 2021 at 3:36 PM 'Tim Hockin' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> First: go version go1.16 linux/amd64
>
> In one of my side-projects, I build a container image.  It turns out
> to be useful to people, but I got a request to put the license files
> for all code used in the binary into the container.  Lawyers, what can
> you do?
>
> So these people sent me a PR to gather the licenses with `go install
> github.com/google/go-licenses`  in
> the `make container` rule.  Here's
> the friction.
>
> 1) Their PR includes dozens of new lines in go.sum, which makes me
> grumpy.  I set out to see if I can do better.
>
> 2) When I run that command on my workstation I get:
>
> ```
> $ go install github.com/google/go-licenses
> cannot find package "." in:
> /home/thockin/src/go/src/
> k8s.io/git-sync/vendor/github.com/google/go-licenses
> ```
>
> I don't know what to make of that message.  It's not in vendor/ yet -
> I want to put it there (I think?).
>
> So I am left wondering - what is the normal flow for "I want to use
> this Go tool in my build" ?
>
> I tried `-mod=mod`:
>
> ```
> $ (unset GOPATH; go install -mod=mod github.com/google/go-licenses)
> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
> go: writing go.mod cache: mkdir /home/thockin/go: not a 

Re: [go-nuts] `go list` across multiple modules?

['Jay Conrod' via golang-nuts]

You appear to have discovered another new bug. Sorry for that. I've opened
#43806  to track it.

With that bug, it doesn't look like `go list` with local paths (relative or
absolute) in a repository with nested replacement directories will work (at
least not yet).

`go list` should still work on full packages paths, but I'm not sure
there's an easy way to resolve directory paths to package paths, other than
by walking up the directory tree to the nearest go.mod and reading the
module path from there.

On Tue, Jan 19, 2021 at 8:06 PM Tim Hockin  wrote:

>
>
> On Tue, Jan 19, 2021 at 7:54 AM Jay Conrod  wrote:
>
>> By the way, I'm not sure if you're already doing this, but if you can
>> batch all of the `go list` runs (or go/packages
>> .Load) together per module (passing an
>> argument per package), that will be much faster than loading individual
>> packages with separate `go list` calls. It will save `go list` from having
>> to load the module graph and common sets of dependencies multiple times.
>>
>> If you need to load all the packages in a module, you can also use an
>> argument like ./... from the root directory of the module to load
>> everything there (excluding modules in subdirectories).
>>
>
> Yeah, I am trying that but tripping on the "does not contain package"
> error.
>
>
>>
>> On Tue, Jan 19, 2021 at 10:02 AM Jay Conrod  wrote:
>>
>>> > Interesting - is the difference the absolute paths vs relative?
>>>
>>> It looks like the bug has to do with whether the directory is below the
>>> main module root directory or not. If it is, the go command takes a path
>>> that assumes it's part of the main module, which it's not.
>>>
>>> > I hoped maybe `-modfile` would do the same trick, but alas not:
>>>
>>> -modfile lets you change the effective content of go.mod but not the
>>> module root directory. Unfortunately it doesn't look like that can be used
>>> to work around the issue.
>>>
>>> > It seems that is because the "main" (top-level dir) go.mod has
>>> > `replace` directives with relative paths, which kubernetes really
>>> > does.
>>>
>>> You may need to copy those over to the tmp go.mod and adjust the paths.
>>> Sorry this has gotten pretty involved.
>>>
>>> > Yeah, I noticed.  When GO111MODULE=off, everything I am doing is much
>>> > faster.  I'm wary of depending on that forever, though.
>>>
>>> Module-aware mode is quite a bit more complicated than GOPATH mode, so
>>> to some degree it's not surprising it's slower... it's surprising that it's
>>> a LOT slower though. I expect there's some optimization work for us to do
>>> in the next development cycle.
>>>
>>> We would eventually like to deprecate GOPATH mode though, so it's a good
>>> idea not to depend on it in new tooling today. 'go list' should be fine to
>>> get package dependency info in either module mode or GOPATH mode.
>>> go/packages  is useful if you need
>>> additional information on top of that (parsed syntax trees, type info).
>>>
>>> > I want to run a slow codegen process only if the packages it depends
>>> > on have ACTUALLY changed (mtime is a good enough proxy) and I don't
>>> > know a priori which packages need codegen.  I want to scan the file
>>> > tree, find the files that need codegen, check their deps, and only
>>> > then run the codegen.
>>>
>>> How much dependency info do you need? If the codegen is only within
>>> packages with files that have changed, 'go list' might be overkill (it
>>> always loads dependencies, even if they aren't printed). If you need
>>> dependencies or reverse dependencies, 'go list' or go/packages
>>>  are probably the right tools.
>>>
>>> On Fri, Jan 15, 2021 at 6:43 PM Tim Hockin  wrote:
>>>
 On Fri, Jan 15, 2021 at 2:17 PM Jay Conrod 
 wrote:
 >
 > I was initially going to suggest adding the module subdirectories as
 requirements to the main go.mod, then replacing those with the
 subdirectories.
 >
 > module example.com/m
 >
 > go 1.15
 >
 > require (
 >   example.com/other1 v0.0.0
 >   example.com/other2 v0.0.0
 >   example.com/m/submod v0.0.0
 > )
 >
 > replace (
 >   example.com/other1 => ./staging/src/example.com/other1
 >   example.com/other2 => ./staging/src/example.com/other2
 >   example.com/m/submod v0.0.0 => ./submod
 > )
 >
 >
 > I think you might have tried this already. It gives the same "main
 module ... does not contain package" error. I believe that's a bug. I've
 opened #43733 to track it.

 Interesting.  If that's a bug, then maybe I'll be able to do what I
 need once fixed.

 > In general, it should be possible to give 'go list' an absolute or
 relative path (starting with ./ or ../) to any directory containing a
 package which is part of any module in the build list. For example, some
 

Re: [go-nuts] `go list` across multiple modules?

['Jay Conrod' via golang-nuts]

By the way, I'm not sure if you're already doing this, but if you can batch
all of the `go list` runs (or go/packages.Load) together per module
(passing an argument per package), that will be much faster than loading
individual packages with separate `go list` calls. It will save `go list`
from having to load the module graph and common sets of dependencies
multiple times.

If you need to load all the packages in a module, you can also use an
argument like ./... from the root directory of the module to load
everything there (excluding modules in subdirectories).

On Tue, Jan 19, 2021 at 10:02 AM Jay Conrod  wrote:

> > Interesting - is the difference the absolute paths vs relative?
>
> It looks like the bug has to do with whether the directory is below the
> main module root directory or not. If it is, the go command takes a path
> that assumes it's part of the main module, which it's not.
>
> > I hoped maybe `-modfile` would do the same trick, but alas not:
>
> -modfile lets you change the effective content of go.mod but not the
> module root directory. Unfortunately it doesn't look like that can be used
> to work around the issue.
>
> > It seems that is because the "main" (top-level dir) go.mod has
> > `replace` directives with relative paths, which kubernetes really
> > does.
>
> You may need to copy those over to the tmp go.mod and adjust the paths.
> Sorry this has gotten pretty involved.
>
> > Yeah, I noticed.  When GO111MODULE=off, everything I am doing is much
> > faster.  I'm wary of depending on that forever, though.
>
> Module-aware mode is quite a bit more complicated than GOPATH mode, so to
> some degree it's not surprising it's slower... it's surprising that it's a
> LOT slower though. I expect there's some optimization work for us to do in
> the next development cycle.
>
> We would eventually like to deprecate GOPATH mode though, so it's a good
> idea not to depend on it in new tooling today. 'go list' should be fine to
> get package dependency info in either module mode or GOPATH mode.
> go/packages  is useful if you need
> additional information on top of that (parsed syntax trees, type info).
>
> > I want to run a slow codegen process only if the packages it depends
> > on have ACTUALLY changed (mtime is a good enough proxy) and I don't
> > know a priori which packages need codegen.  I want to scan the file
> > tree, find the files that need codegen, check their deps, and only
> > then run the codegen.
>
> How much dependency info do you need? If the codegen is only within
> packages with files that have changed, 'go list' might be overkill (it
> always loads dependencies, even if they aren't printed). If you need
> dependencies or reverse dependencies, 'go list' or go/packages
>  are probably the right tools.
>
> On Fri, Jan 15, 2021 at 6:43 PM Tim Hockin  wrote:
>
>> On Fri, Jan 15, 2021 at 2:17 PM Jay Conrod  wrote:
>> >
>> > I was initially going to suggest adding the module subdirectories as
>> requirements to the main go.mod, then replacing those with the
>> subdirectories.
>> >
>> > module example.com/m
>> >
>> > go 1.15
>> >
>> > require (
>> >   example.com/other1 v0.0.0
>> >   example.com/other2 v0.0.0
>> >   example.com/m/submod v0.0.0
>> > )
>> >
>> > replace (
>> >   example.com/other1 => ./staging/src/example.com/other1
>> >   example.com/other2 => ./staging/src/example.com/other2
>> >   example.com/m/submod v0.0.0 => ./submod
>> > )
>> >
>> >
>> > I think you might have tried this already. It gives the same "main
>> module ... does not contain package" error. I believe that's a bug. I've
>> opened #43733 to track it.
>>
>> Interesting.  If that's a bug, then maybe I'll be able to do what I
>> need once fixed.
>>
>> > In general, it should be possible to give 'go list' an absolute or
>> relative path (starting with ./ or ../) to any directory containing a
>> package which is part of any module in the build list. For example, some
>> tools list directories in the module cache to find out what package a .go
>> file belongs to.
>> >
>> > As a workaround, you could put a go.mod in an otherwise empty directory
>> (in /tmp or something), then require the relevant modules from the repo and
>> replace them with absolute paths. Then you can run 'go list' in that
>> directory with absolute paths of package directories.
>>
>> Interesting - is the difference the absolute paths vs relative?
>>
>> I hoped maybe `-modfile` would do the same trick, but alas not:
>>
>> ```
>> $ (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/)
>> example.com/m/submod/used
>>
>> $ go list --modfile /tmp/gomodhack/go.mod
>> /tmp/go-list-modules/submod/used/
>> main module (tmp) does not contain package tmp/submod/used
>> ```
>>
>> It also fails some cases:
>>
>> ```
>>  (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/)
>> example.com/m/submod/used
>> thockin@thockin-glaptop4 go-list-modules main /$ (cd 

Re: [go-nuts] `go list` across multiple modules?

['Jay Conrod' via golang-nuts]

> Interesting - is the difference the absolute paths vs relative?

It looks like the bug has to do with whether the directory is below the
main module root directory or not. If it is, the go command takes a path
that assumes it's part of the main module, which it's not.

> I hoped maybe `-modfile` would do the same trick, but alas not:

-modfile lets you change the effective content of go.mod but not the module
root directory. Unfortunately it doesn't look like that can be used to work
around the issue.

> It seems that is because the "main" (top-level dir) go.mod has
> `replace` directives with relative paths, which kubernetes really
> does.

You may need to copy those over to the tmp go.mod and adjust the paths.
Sorry this has gotten pretty involved.

> Yeah, I noticed.  When GO111MODULE=off, everything I am doing is much
> faster.  I'm wary of depending on that forever, though.

Module-aware mode is quite a bit more complicated than GOPATH mode, so to
some degree it's not surprising it's slower... it's surprising that it's a
LOT slower though. I expect there's some optimization work for us to do in
the next development cycle.

We would eventually like to deprecate GOPATH mode though, so it's a good
idea not to depend on it in new tooling today. 'go list' should be fine to
get package dependency info in either module mode or GOPATH mode.
go/packages is useful if you need additional information on top of that
(parsed syntax trees, type info).

> I want to run a slow codegen process only if the packages it depends
> on have ACTUALLY changed (mtime is a good enough proxy) and I don't
> know a priori which packages need codegen.  I want to scan the file
> tree, find the files that need codegen, check their deps, and only
> then run the codegen.

How much dependency info do you need? If the codegen is only within
packages with files that have changed, 'go list' might be overkill (it
always loads dependencies, even if they aren't printed). If you need
dependencies or reverse dependencies, 'go list' or go/packages are probably
the right tools.

On Fri, Jan 15, 2021 at 6:43 PM Tim Hockin  wrote:

> On Fri, Jan 15, 2021 at 2:17 PM Jay Conrod  wrote:
> >
> > I was initially going to suggest adding the module subdirectories as
> requirements to the main go.mod, then replacing those with the
> subdirectories.
> >
> > module example.com/m
> >
> > go 1.15
> >
> > require (
> >   example.com/other1 v0.0.0
> >   example.com/other2 v0.0.0
> >   example.com/m/submod v0.0.0
> > )
> >
> > replace (
> >   example.com/other1 => ./staging/src/example.com/other1
> >   example.com/other2 => ./staging/src/example.com/other2
> >   example.com/m/submod v0.0.0 => ./submod
> > )
> >
> >
> > I think you might have tried this already. It gives the same "main
> module ... does not contain package" error. I believe that's a bug. I've
> opened #43733 to track it.
>
> Interesting.  If that's a bug, then maybe I'll be able to do what I
> need once fixed.
>
> > In general, it should be possible to give 'go list' an absolute or
> relative path (starting with ./ or ../) to any directory containing a
> package which is part of any module in the build list. For example, some
> tools list directories in the module cache to find out what package a .go
> file belongs to.
> >
> > As a workaround, you could put a go.mod in an otherwise empty directory
> (in /tmp or something), then require the relevant modules from the repo and
> replace them with absolute paths. Then you can run 'go list' in that
> directory with absolute paths of package directories.
>
> Interesting - is the difference the absolute paths vs relative?
>
> I hoped maybe `-modfile` would do the same trick, but alas not:
>
> ```
> $ (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/)
> example.com/m/submod/used
>
> $ go list --modfile /tmp/gomodhack/go.mod /tmp/go-list-modules/submod/used/
> main module (tmp) does not contain package tmp/submod/used
> ```
>
> It also fails some cases:
>
> ```
>  (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/)
> example.com/m/submod/used
> thockin@thockin-glaptop4 go-list-modules main /$ (cd /tmp/gomodhack/;
> go list /tmp/go-list-modules/staging/src/example.com/other1/used/)
> go: finding module for package
> example.com/m/staging/src/example.com/other1/used
> cannot find module providing package
> example.com/m/staging/src/example.com/other1/used: unrecognized import
> path "example.com/m/staging/src/example.com/other1/used": reading
> https://example.com/m/staging/src/example.com/other1/used?go-get=1:
> 404 Not Found
> ```
>
> It seems that is because the "main" (top-level dir) go.mod has
> `replace` directives with relative paths, which kubernetes really
> does.
>
> > Incidentally, golang.org/x/tools/go/packages will call 'go list' under
> the hood in module mode. go/build  might
> do the same, depending on how it's invoked. 'go list' may be the best thing
> to use if it gives the 

Re: [go-nuts] `go list` across multiple modules?

['Jay Conrod' via golang-nuts]

I was initially going to suggest adding the module subdirectories as
requirements to the main go.mod, then replacing those with the
subdirectories.

module example.com/m

go 1.15

require (
  example.com/other1 v0.0.0
  example.com/other2 v0.0.0
  example.com/m/submod v0.0.0
)

replace (
  example.com/other1 => ./staging/src/example.com/other1
  example.com/other2 => ./staging/src/example.com/other2
  example.com/m/submod v0.0.0 => ./submod
)


I think you might have tried this already. It gives the same "main module
... does not contain package" error. I believe that's a bug. I've opened
#43733  to track it.

In general, it should be possible to give 'go list' an absolute or relative
path (starting with ./ or ../) to any directory containing a package which
is part of any module in the build list. For example, some tools list
directories in the module cache to find out what package a .go file belongs
to.

As a workaround, you could put a go.mod in an otherwise empty directory (in
/tmp or something), then require the relevant modules from the repo and
replace them with absolute paths. Then you can run 'go list' in that
directory with absolute paths of package directories.

module tmp

go 1.15

require (
  example.com/other1 v0.0.0
  example.com/other2 v0.0.0
  example.com/m v0.0.0
  example.com/m/submod v0.0.0
)

replace (
  example.com/other1 => /home/gopher/go/src/
example.com/m/staging/src/example.com/other1
  example.com/other2 => /home/gopher/go/src/
example.com/m/staging/src/example.com/other2
  example.com/m v0.0.0 => /home/gopher/go/src/example.com/m
  example.com/m/submod v0.0.0 => /home/gopher/go/src/example.com/m/submod
)


Incidentally, golang.org/x/tools/go/packages will call 'go list' under the
hood in module mode. go/build might do the same, depending on how it's
invoked. 'go list' may be the best thing to use if it gives the information
you need.

On Fri, Jan 15, 2021 at 11:59 AM 'Tim Hockin' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> Hi.  This isn't exactly burning urgent, but it is a long-term issue
> for Kubernetes.  If there's anything I can do to catalyze the
> discussion - tests to run, info to dig up, etc - please let me know.
>
> On Wed, Dec 23, 2020 at 10:48 AM Tim Hockin  wrote:
> >
> > Hi Paul!
> >
> > On Wed, Dec 23, 2020 at 4:23 AM Paul Jolly  wrote:
> > >
> > > > I just can't figure out how to do this.  Maybe it can't be done in
> `go
> > > > list` ?  Or maybe we're just missing some detail of go modules..
> > >
> > > go list operates in the context of a single module (in the mode you
> > > are interested in), so you cannot do this with a single command across
> > > multiple modules.
> >
> > This might be a real problem for us.  For this post I am reducing it
> > to `go list`, but in actuality we have a small program that we wrote
> > which does what we need in terms of `go/build`.  It works great when
> > `GO111MODULE=off` but is more than 100x slower normally.  I thought it
> > was finally time to rewrite it in terms of `go/packages` and get rid
> > of GO111MODULE=off.  That didn't pan out, hence this post.
> >
> > More inline and below
> >
> > > > First I do a `find` for any file that has a specific comment tag,
> > > > indicating that the package needs codegen.  The results span several
> > > > of the in-repo submodules.
> > >
> > > Just to check, I'm assuming the results of this find command are being
> > > translated to a list of packages? Because the transitive dependencies
> > > of a list of packages within a module can be done via a single go list
> > > command.
> >
> > The trick is "within a module".  I'll update
> > https://github.com/thockin/go-list-modules to reflect the process
> > more.   I've added a
> > get_codegen_deps.sh that models the behavior.  Note that I really want
> > files, not packages, so I can express the dep-graph.
> >
> > What do you mean by "translated to a list of packages" - which specific
> syntax?
> >
> > What I end up with is something like `go list ./path/to/dir1
> > ./path/to/dir2 ./path/to/dir3`.  Any of those dirs might be in
> > different modules.  So `go list` tells me "main module (example.com/m)
> > does not contain package example.com/m/path/to/dir1" and so on.
> > Setting `GO111MODULE=off` does work, but I fear the future of that.
> >
> > > > For each target package, I want to get the list of all deps and
> > > > extract the GoFiles.  Then I can use that to determine if the codegen
> > > > needs to run.
> > >
> > > FWIW I wrote a tool to do just this:
> > >
> https://pkg.go.dev/myitcv.io@v0.0.0-20201125173645-a7167afc9e13/cmd/gogenerate
> > > which might work in your situation.
> >
> > I will take a look - it seems I will need to restructure a bunch of
> > tooling to prove it works for us or doesn't :)
> >
> > > > Where it breaks down is that I can't seem to `go list` all at once:
> > > >
> > > > ```
> > > > # This works within the "root" module
> > > > $ go list -f 

Re: [go-nuts] Re: Go 1.14 is released

['Jay Conrod' via golang-nuts]

Both pages were moved to the x/website repository

so
they could be kept up to date outside the 6-month release cycle.

On Wed, Feb 26, 2020 at 7:33 AM HaWe  wrote:

> Firts of all, many thanks for a great new release.
>
> Now my question:
> The (new) document "How to Write Go Code" is available at
> golang.org/doc/code.html, the older version, now named "How to Write Go
> Code (with GOPATH)" at golang.org/doc/gopath_code.html.
> But they seem to be no longer part of the Git repository.
>
> Is this deliberate?
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/70d888f4-ea86-49ef-9815-6b02dc26a883%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGCADbZqZLvTmg%2BYNLdy-LK61zn41vqiNyuCvUoL7NSHqrw%2B%2BQ%40mail.gmail.com.