Re: [go-nuts] Trying to use a tool in my build (a friction log)

['Jay Conrod' via golang-nuts]

> Is it by design that these two commands ['go mod tidy' and 'go mod
vendor'] are unstable like that?

That is more or less by design, but the behavior has evolved over time and
it may be worth revisiting.

This was discussed in #29058 .
The rationale was that because 'go mod vendor' deletes and rebuilds the
entire vendor directory, it would be dangerous to do that implicitly inside
another command like 'go mod tidy', since people frequently make local
changes to vendor, like when preparing a patch for upstream.

That discussion was a couple years ago, before vendoring was enabled
automatically. At the time, you had to specify -mod=vendor with build
commands.

A better user experience might be 1) support for incremental changes to the
vendor directory via 'go get', 'go mod tidy', 2) detection of local changes
to the vendor directory so we can avoid overwriting them.

I've opened #45161  to restart
that discussion.

> This tool in particular seems to try to read a DB file from a subdir of
the directory its source code came from (as per `runtime.Caller(0)`) which
...doesn't work at all with vendoring, since the dir doesn't contain Go
code.

> AFAIK there's not a way to ask `go mod vendor` to include the whole repo
is there?  I can't find a spec for modules.txt - it would be super cool if
I could add some tag in there...  As it stands, all this was for naught.

Is there another way to include and locate that file? Even if it were
vendored, runtime.Caller(0) may be unreliable when -trimpath is used, and
the files will be read-only in the module cache when vendoring is not used.

'go mod vendor' only pulls in directories for packages needed to build
packages in the main module and their tests ('go list -test -deps ./...',
essentially). I don't think we even include testdata or packages imported
by those packages' tests.

We don't have a spec for vendor/modules.txt. it basically contains enough
information for the go command to know which packages are vendored and
which modules they came from.

On Sat, Mar 20, 2021 at 12:47 PM Manlio Perillo 
wrote:

> Il giorno venerdì 19 marzo 2021 alle 23:20:49 UTC+1 Tim Hockin ha scritto:
>
>> Thanks for feedback, comments below
>>
>> On Thu, Mar 18, 2021 at 3:35 PM Jay Conrod  wrote:
>>
>>>
>>> > [...]
>
>>
>>
>>> *2. "writing go.mod cache" error messages*
>>>
>>> This error message should be a lot better. Sorry about that.
>>>
>>
>> I mean, "mkdir /home/thockin/go: not a directory" is pretty concise -
>> it's not a directory (on purpose) because I really don't want Go randomly
>> deciding to write stuff to my homedir (which I suspected was happening, and
>> that's why I made it a non-directory :)
>>
>>
>
> If you don't want the go tool to write files to your home directory, you
> can just define a custom GOPATH or GOMODCACHE.  As an example, on my
> sysytem I have set GOPATH to ~/.local/lib/go, and this directory is
> excluded from the backup.
>
> Manlio
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/8002edf8-1a9e-44a6-a427-3fbc2b4e1dc1n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGCADbYLzKJi7AYdgzyJmm4WHHe4GWgQ%3DseFfxLPOLmpL5Bg3w%40mail.gmail.com.

Re: [go-nuts] Trying to use a tool in my build (a friction log)

[Manlio Perillo]

Il giorno venerdì 19 marzo 2021 alle 23:20:49 UTC+1 Tim Hockin ha scritto:

> Thanks for feedback, comments below
>
> On Thu, Mar 18, 2021 at 3:35 PM Jay Conrod  wrote:
>
>>
>> > [...] 

>  
>
>> *2. "writing go.mod cache" error messages*
>>
>> This error message should be a lot better. Sorry about that.
>>
>
> I mean, "mkdir /home/thockin/go: not a directory" is pretty concise - it's 
> not a directory (on purpose) because I really don't want Go randomly 
> deciding to write stuff to my homedir (which I suspected was happening, and 
> that's why I made it a non-directory :)
>  
>

If you don't want the go tool to write files to your home directory, you 
can just define a custom GOPATH or GOMODCACHE.  As an example, on my 
sysytem I have set GOPATH to ~/.local/lib/go, and this directory is 
excluded from the backup.

Manlio 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/8002edf8-1a9e-44a6-a427-3fbc2b4e1dc1n%40googlegroups.com.

Re: [go-nuts] Trying to use a tool in my build (a friction log)

['Tim Hockin' via golang-nuts]

Thanks for feedback, comments below

On Thu, Mar 18, 2021 at 3:35 PM Jay Conrod  wrote:

>
> Installing a command within the context of the current module is more
> complicated. I'd suggest this:
>
> # Add a dependency on the tool (only need to do this once)
> go get -d github.com/google/go-licenses
>
> # Import the command from tools.go
>
> # See below
>
>
> # Sync vendor directory (only if vendoring; only after changing go.mod
> with 'go get' above)
> go mod vendor
>
> # Install the command
> go install github.com/google/go-licenses
>
> # Alternatively, build and write to some other directory
> go build -o bin/go-licenses github.com/google/go-licenses
>
>
I bent this a bit, and it seems to work:

1) add tools.go and import the tool
2) go mod vendor
3) go build

IOW, I skipped `go get -d` and it all seems OK.

Now it builds.  If I run `go mod tidy` it removes a line from my go.mod (`
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b // indirect`) and
then refuses to build:

```
go: inconsistent vendoring in /src:
github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b: is marked as
explicit in vendor/modules.txt, but not explicitly required in go.mod

run 'go mod vendor' to sync, or use -mod=mod or -mod=readonly to ignore the
vendor directory
```

Now I run `go mod tidy` again and it adds a line to `vendor/modules.txt`
and happily compiles again.

Subsequent runs of either command seem to be no-op at this point.

Is it by design that these two commands are unstable like that?

Now that it builds, extra fun.  This tool in particular seems to try to
read a DB file from a subdir of the directory its source code came from (as
per `runtime.Caller(0)`) which ...doesn't work at all with vendoring, since
the dir doesn't contain Go code.

AFAIK there's not a way to ask `go mod vendor` to include the whole repo is
there?  I can't find a spec for modules.txt - it would be super cool if I
could add some tag in there...  As it stands, all this was for naught.


> *2. "writing go.mod cache" error messages*
>
> This error message should be a lot better. Sorry about that.
>

I mean, "mkdir /home/thockin/go: not a directory" is pretty concise - it's
not a directory (on purpose) because I really don't want Go randomly
deciding to write stuff to my homedir (which I suspected was happening, and
that's why I made it a non-directory :)


> *3. 'go get' updating go.mod and go.sum*
>
> This is an unfortunate consequence of 'go get' being overloaded for
> downloading, installing, and managing package versions. Ideally, we want to
> separate these roles into different commands: 'go get' should be used to
> manage dependencies (changing requirements in go.mod and go.sum), and 'go
> install' should be used to install commands.
>
> We've taken a couple steps toward this in 1.16. The 'go install cmd@version'
> form is new. Also, 'go install', 'go build', and other commands no longer
> change go.mod or go.sum automatically when something is missing.
>

>From where I site, I like the distinction, and I might even argue for
explicitness.  E.g.  if I said `go get -mod=vendor` I feel like I am
unambiguously asking the tool to fetch the code and put it in ./vendor and
update the module files.  If I said just `go get` I (not so unambiguously)
mean "global" (for me).  I find the automatic-detection of module mode very
confusing, personally.

Tim

On Thu, Mar 18, 2021 at 3:36 PM 'Tim Hockin' via golang-nuts <
> golang-nuts@googlegroups.com> wrote:
>
>> First: go version go1.16 linux/amd64
>>
>> In one of my side-projects, I build a container image.  It turns out
>> to be useful to people, but I got a request to put the license files
>> for all code used in the binary into the container.  Lawyers, what can
>> you do?
>>
>> So these people sent me a PR to gather the licenses with `go install
>> github.com/google/go-licenses`  in
>> the `make container` rule.  Here's
>> the friction.
>>
>> 1) Their PR includes dozens of new lines in go.sum, which makes me
>> grumpy.  I set out to see if I can do better.
>>
>> 2) When I run that command on my workstation I get:
>>
>> ```
>> $ go install github.com/google/go-licenses
>> cannot find package "." in:
>> /home/thockin/src/go/src/
>> k8s.io/git-sync/vendor/github.com/google/go-licenses
>> ```
>>
>> I don't know what to make of that message.  It's not in vendor/ yet -
>> I want to put it there (I think?).
>>
>> So I am left wondering - what is the normal flow for "I want to use
>> this Go tool in my build" ?
>>
>> I tried `-mod=mod`:
>>
>> ```
>> $ (unset GOPATH; go install -mod=mod github.com/google/go-licenses)
>> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
>> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
>> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
>> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
>> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
>> 

Re: [go-nuts] Trying to use a tool in my build (a friction log)

['Jay Conrod' via golang-nuts]

Hey Tim, thanks for writing this up. These kinds of reports are helpful.
I'll try to respond to a few problems individually.

*1. Command installation with "go install"*

There are basically two ways to do this, depending on whether you want to
install the command within the context of your current module or "globally"
ignoring go.mod, go.sum, and vendor.

To install a command globally (you can replace @latest with a specific
version like @v1.2.3):

go install github.com/google/go-licenses@latest


This form (with the @version suffix) is new in Go 1.16
.

Installing a command within the context of the current module is more
complicated. I'd suggest this:

# Add a dependency on the tool (only need to do this once)
go get -d github.com/google/go-licenses

# Import the command from tools.go

# See below


# Sync vendor directory (only if vendoring; only after changing go.mod with
'go get' above)
go mod vendor

# Install the command
go install github.com/google/go-licenses

# Alternatively, build and write to some other directory
go build -o bin/go-licenses github.com/google/go-licenses


The problem with this workflow is that 'go mod tidy' removes the
requirement on the command's module because nothing imports it. 'go mod
vendor' won't vendor the package for the same reason. There is a workaround
documented on the Wiki : import
the command package from a file (usually named tools.go) that is excluded
by a build constraint (usually "tools").

// +build tools

package tools

import _ "github.com/google/go-licenses"


This works, but personally, I don't find this to be an intuitive solution.
At the moment, we don't have a concrete proposal to improve it though.
There's some discussion at #25922
.

*2. "writing go.mod cache" error messages*

This error message should be a lot better. Sorry about that.

Most module-aware commands need to download files into the module cache.
The location of the cache can be set by setting the GOMODCACHE environment
variable (or 'go env -w GOMODCACHE=/some/path'). That defaults to
$GOPATH/pkg/mod, and GOPATH defaults to $HOME/go. So by default, the module
cache is at $HOME/go/pkg/mod. It can't create that directory though.

The error message should explain what went wrong and how to fix it, and it
really shouldn't be repeated. I've opened #45113
 for this.

*3. 'go get' updating go.mod and go.sum*

This is an unfortunate consequence of 'go get' being overloaded for
downloading, installing, and managing package versions. Ideally, we want to
separate these roles into different commands: 'go get' should be used to
manage dependencies (changing requirements in go.mod and go.sum), and 'go
install' should be used to install commands.

We've taken a couple steps toward this in 1.16. The 'go install cmd@version'
form is new. Also, 'go install', 'go build', and other commands no longer
change go.mod or go.sum automatically when something is missing.

We plan to deprecate 'go get' for installing commands. In Go 1.17, 'go get'
will print a warning when the -d flag is not used and the arguments are
main packages. In Go 1.18, we plan to remove that functionality entirely
(the -d flag will always be on). #40276
 is where that deprecation was
proposed.

On Thu, Mar 18, 2021 at 3:36 PM 'Tim Hockin' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> First: go version go1.16 linux/amd64
>
> In one of my side-projects, I build a container image.  It turns out
> to be useful to people, but I got a request to put the license files
> for all code used in the binary into the container.  Lawyers, what can
> you do?
>
> So these people sent me a PR to gather the licenses with `go install
> github.com/google/go-licenses`  in
> the `make container` rule.  Here's
> the friction.
>
> 1) Their PR includes dozens of new lines in go.sum, which makes me
> grumpy.  I set out to see if I can do better.
>
> 2) When I run that command on my workstation I get:
>
> ```
> $ go install github.com/google/go-licenses
> cannot find package "." in:
> /home/thockin/src/go/src/
> k8s.io/git-sync/vendor/github.com/google/go-licenses
> ```
>
> I don't know what to make of that message.  It's not in vendor/ yet -
> I want to put it there (I think?).
>
> So I am left wondering - what is the normal flow for "I want to use
> this Go tool in my build" ?
>
> I tried `-mod=mod`:
>
> ```
> $ (unset GOPATH; go install -mod=mod github.com/google/go-licenses)
> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
> go: writing go.mod cache: mkdir /home/thockin/go: not a