Re: [go-nuts] Trying to use a tool in my build (a friction log)

['Jay Conrod' via golang-nuts]

> Is it by design that these two commands ['go mod tidy' and 'go mod
vendor'] are unstable like that?

That is more or less by design, but the behavior has evolved over time and
it may be worth revisiting.

This was discussed in #29058 .
The rationale was that because 'go mod vendor' deletes and rebuilds the
entire vendor directory, it would be dangerous to do that implicitly inside
another command like 'go mod tidy', since people frequently make local
changes to vendor, like when preparing a patch for upstream.

That discussion was a couple years ago, before vendoring was enabled
automatically. At the time, you had to specify -mod=vendor with build
commands.

A better user experience might be 1) support for incremental changes to the
vendor directory via 'go get', 'go mod tidy', 2) detection of local changes
to the vendor directory so we can avoid overwriting them.

I've opened #45161  to restart
that discussion.

> This tool in particular seems to try to read a DB file from a subdir of
the directory its source code came from (as per `runtime.Caller(0)`) which
...doesn't work at all with vendoring, since the dir doesn't contain Go
code.

> AFAIK there's not a way to ask `go mod vendor` to include the whole repo
is there?  I can't find a spec for modules.txt - it would be super cool if
I could add some tag in there...  As it stands, all this was for naught.

Is there another way to include and locate that file? Even if it were
vendored, runtime.Caller(0) may be unreliable when -trimpath is used, and
the files will be read-only in the module cache when vendoring is not used.

'go mod vendor' only pulls in directories for packages needed to build
packages in the main module and their tests ('go list -test -deps ./...',
essentially). I don't think we even include testdata or packages imported
by those packages' tests.

We don't have a spec for vendor/modules.txt. it basically contains enough
information for the go command to know which packages are vendored and
which modules they came from.

On Sat, Mar 20, 2021 at 12:47 PM Manlio Perillo 
wrote:

> Il giorno venerdì 19 marzo 2021 alle 23:20:49 UTC+1 Tim Hockin ha scritto:
>
>> Thanks for feedback, comments below
>>
>> On Thu, Mar 18, 2021 at 3:35 PM Jay Conrod  wrote:
>>
>>>
>>> > [...]
>
>>
>>
>>> *2. "writing go.mod cache" error messages*
>>>
>>> This error message should be a lot better. Sorry about that.
>>>
>>
>> I mean, "mkdir /home/thockin/go: not a directory" is pretty concise -
>> it's not a directory (on purpose) because I really don't want Go randomly
>> deciding to write stuff to my homedir (which I suspected was happening, and
>> that's why I made it a non-directory :)
>>
>>
>
> If you don't want the go tool to write files to your home directory, you
> can just define a custom GOPATH or GOMODCACHE.  As an example, on my
> sysytem I have set GOPATH to ~/.local/lib/go, and this directory is
> excluded from the backup.
>
> Manlio
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/8002edf8-1a9e-44a6-a427-3fbc2b4e1dc1n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGCADbYLzKJi7AYdgzyJmm4WHHe4GWgQ%3DseFfxLPOLmpL5Bg3w%40mail.gmail.com.

Re: [go-nuts] Trying to use a tool in my build (a friction log)

[Manlio Perillo]

Il giorno venerdì 19 marzo 2021 alle 23:20:49 UTC+1 Tim Hockin ha scritto:

> Thanks for feedback, comments below
>
> On Thu, Mar 18, 2021 at 3:35 PM Jay Conrod  wrote:
>
>>
>> > [...] 

>  
>
>> *2. "writing go.mod cache" error messages*
>>
>> This error message should be a lot better. Sorry about that.
>>
>
> I mean, "mkdir /home/thockin/go: not a directory" is pretty concise - it's 
> not a directory (on purpose) because I really don't want Go randomly 
> deciding to write stuff to my homedir (which I suspected was happening, and 
> that's why I made it a non-directory :)
>  
>

If you don't want the go tool to write files to your home directory, you 
can just define a custom GOPATH or GOMODCACHE.  As an example, on my 
sysytem I have set GOPATH to ~/.local/lib/go, and this directory is 
excluded from the backup.

Manlio 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/8002edf8-1a9e-44a6-a427-3fbc2b4e1dc1n%40googlegroups.com.

Re: [go-nuts] Trying to use a tool in my build (a friction log)

['Tim Hockin' via golang-nuts]

Thanks for feedback, comments below

On Thu, Mar 18, 2021 at 3:35 PM Jay Conrod  wrote:

>
> Installing a command within the context of the current module is more
> complicated. I'd suggest this:
>
> # Add a dependency on the tool (only need to do this once)
> go get -d github.com/google/go-licenses
>
> # Import the command from tools.go
>
> # See below
>
>
> # Sync vendor directory (only if vendoring; only after changing go.mod
> with 'go get' above)
> go mod vendor
>
> # Install the command
> go install github.com/google/go-licenses
>
> # Alternatively, build and write to some other directory
> go build -o bin/go-licenses github.com/google/go-licenses
>
>
I bent this a bit, and it seems to work:

1) add tools.go and import the tool
2) go mod vendor
3) go build

IOW, I skipped `go get -d` and it all seems OK.

Now it builds.  If I run `go mod tidy` it removes a line from my go.mod (`
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b // indirect`) and
then refuses to build:

```
go: inconsistent vendoring in /src:
github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b: is marked as
explicit in vendor/modules.txt, but not explicitly required in go.mod

run 'go mod vendor' to sync, or use -mod=mod or -mod=readonly to ignore the
vendor directory
```

Now I run `go mod tidy` again and it adds a line to `vendor/modules.txt`
and happily compiles again.

Subsequent runs of either command seem to be no-op at this point.

Is it by design that these two commands are unstable like that?

Now that it builds, extra fun.  This tool in particular seems to try to
read a DB file from a subdir of the directory its source code came from (as
per `runtime.Caller(0)`) which ...doesn't work at all with vendoring, since
the dir doesn't contain Go code.

AFAIK there's not a way to ask `go mod vendor` to include the whole repo is
there?  I can't find a spec for modules.txt - it would be super cool if I
could add some tag in there...  As it stands, all this was for naught.


> *2. "writing go.mod cache" error messages*
>
> This error message should be a lot better. Sorry about that.
>

I mean, "mkdir /home/thockin/go: not a directory" is pretty concise - it's
not a directory (on purpose) because I really don't want Go randomly
deciding to write stuff to my homedir (which I suspected was happening, and
that's why I made it a non-directory :)


> *3. 'go get' updating go.mod and go.sum*
>
> This is an unfortunate consequence of 'go get' being overloaded for
> downloading, installing, and managing package versions. Ideally, we want to
> separate these roles into different commands: 'go get' should be used to
> manage dependencies (changing requirements in go.mod and go.sum), and 'go
> install' should be used to install commands.
>
> We've taken a couple steps toward this in 1.16. The 'go install cmd@version'
> form is new. Also, 'go install', 'go build', and other commands no longer
> change go.mod or go.sum automatically when something is missing.
>

>From where I site, I like the distinction, and I might even argue for
explicitness.  E.g.  if I said `go get -mod=vendor` I feel like I am
unambiguously asking the tool to fetch the code and put it in ./vendor and
update the module files.  If I said just `go get` I (not so unambiguously)
mean "global" (for me).  I find the automatic-detection of module mode very
confusing, personally.

Tim

On Thu, Mar 18, 2021 at 3:36 PM 'Tim Hockin' via golang-nuts <
> golang-nuts@googlegroups.com> wrote:
>
>> First: go version go1.16 linux/amd64
>>
>> In one of my side-projects, I build a container image.  It turns out
>> to be useful to people, but I got a request to put the license files
>> for all code used in the binary into the container.  Lawyers, what can
>> you do?
>>
>> So these people sent me a PR to gather the licenses with `go install
>> github.com/google/go-licenses`  in
>> the `make container` rule.  Here's
>> the friction.
>>
>> 1) Their PR includes dozens of new lines in go.sum, which makes me
>> grumpy.  I set out to see if I can do better.
>>
>> 2) When I run that command on my workstation I get:
>>
>> ```
>> $ go install github.com/google/go-licenses
>> cannot find package "." in:
>> /home/thockin/src/go/src/
>> k8s.io/git-sync/vendor/github.com/google/go-licenses
>> ```
>>
>> I don't know what to make of that message.  It's not in vendor/ yet -
>> I want to put it there (I think?).
>>
>> So I am left wondering - what is the normal flow for "I want to use
>> this Go tool in my build" ?
>>
>> I tried `-mod=mod`:
>>
>> ```
>> $ (unset GOPATH; go install -mod=mod github.com/google/go-licenses)
>> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
>> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
>> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
>> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
>> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
>> 

Re: [go-nuts] Trying to use a tool in my build (a friction log)

['Jay Conrod' via golang-nuts]

Hey Tim, thanks for writing this up. These kinds of reports are helpful.
I'll try to respond to a few problems individually.

*1. Command installation with "go install"*

There are basically two ways to do this, depending on whether you want to
install the command within the context of your current module or "globally"
ignoring go.mod, go.sum, and vendor.

To install a command globally (you can replace @latest with a specific
version like @v1.2.3):

go install github.com/google/go-licenses@latest


This form (with the @version suffix) is new in Go 1.16
.

Installing a command within the context of the current module is more
complicated. I'd suggest this:

# Add a dependency on the tool (only need to do this once)
go get -d github.com/google/go-licenses

# Import the command from tools.go

# See below


# Sync vendor directory (only if vendoring; only after changing go.mod with
'go get' above)
go mod vendor

# Install the command
go install github.com/google/go-licenses

# Alternatively, build and write to some other directory
go build -o bin/go-licenses github.com/google/go-licenses


The problem with this workflow is that 'go mod tidy' removes the
requirement on the command's module because nothing imports it. 'go mod
vendor' won't vendor the package for the same reason. There is a workaround
documented on the Wiki : import
the command package from a file (usually named tools.go) that is excluded
by a build constraint (usually "tools").

// +build tools

package tools

import _ "github.com/google/go-licenses"


This works, but personally, I don't find this to be an intuitive solution.
At the moment, we don't have a concrete proposal to improve it though.
There's some discussion at #25922
.

*2. "writing go.mod cache" error messages*

This error message should be a lot better. Sorry about that.

Most module-aware commands need to download files into the module cache.
The location of the cache can be set by setting the GOMODCACHE environment
variable (or 'go env -w GOMODCACHE=/some/path'). That defaults to
$GOPATH/pkg/mod, and GOPATH defaults to $HOME/go. So by default, the module
cache is at $HOME/go/pkg/mod. It can't create that directory though.

The error message should explain what went wrong and how to fix it, and it
really shouldn't be repeated. I've opened #45113
 for this.

*3. 'go get' updating go.mod and go.sum*

This is an unfortunate consequence of 'go get' being overloaded for
downloading, installing, and managing package versions. Ideally, we want to
separate these roles into different commands: 'go get' should be used to
manage dependencies (changing requirements in go.mod and go.sum), and 'go
install' should be used to install commands.

We've taken a couple steps toward this in 1.16. The 'go install cmd@version'
form is new. Also, 'go install', 'go build', and other commands no longer
change go.mod or go.sum automatically when something is missing.

We plan to deprecate 'go get' for installing commands. In Go 1.17, 'go get'
will print a warning when the -d flag is not used and the arguments are
main packages. In Go 1.18, we plan to remove that functionality entirely
(the -d flag will always be on). #40276
 is where that deprecation was
proposed.

On Thu, Mar 18, 2021 at 3:36 PM 'Tim Hockin' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> First: go version go1.16 linux/amd64
>
> In one of my side-projects, I build a container image.  It turns out
> to be useful to people, but I got a request to put the license files
> for all code used in the binary into the container.  Lawyers, what can
> you do?
>
> So these people sent me a PR to gather the licenses with `go install
> github.com/google/go-licenses`  in
> the `make container` rule.  Here's
> the friction.
>
> 1) Their PR includes dozens of new lines in go.sum, which makes me
> grumpy.  I set out to see if I can do better.
>
> 2) When I run that command on my workstation I get:
>
> ```
> $ go install github.com/google/go-licenses
> cannot find package "." in:
> /home/thockin/src/go/src/
> k8s.io/git-sync/vendor/github.com/google/go-licenses
> ```
>
> I don't know what to make of that message.  It's not in vendor/ yet -
> I want to put it there (I think?).
>
> So I am left wondering - what is the normal flow for "I want to use
> this Go tool in my build" ?
>
> I tried `-mod=mod`:
>
> ```
> $ (unset GOPATH; go install -mod=mod github.com/google/go-licenses)
> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
> go: writing go.mod cache: mkdir /home/thockin/go: not a directory
> go: writing go.mod cache: mkdir /home/thockin/go: not a 

[go-nuts] Trying to use a tool in my build (a friction log)

['Tim Hockin' via golang-nuts]

First: go version go1.16 linux/amd64

In one of my side-projects, I build a container image.  It turns out
to be useful to people, but I got a request to put the license files
for all code used in the binary into the container.  Lawyers, what can
you do?

So these people sent me a PR to gather the licenses with `go install
github.com/google/go-licenses` in the `make container` rule.  Here's
the friction.

1) Their PR includes dozens of new lines in go.sum, which makes me
grumpy.  I set out to see if I can do better.

2) When I run that command on my workstation I get:

```
$ go install github.com/google/go-licenses
cannot find package "." in:
/home/thockin/src/go/src/k8s.io/git-sync/vendor/github.com/google/go-licenses
```

I don't know what to make of that message.  It's not in vendor/ yet -
I want to put it there (I think?).

So I am left wondering - what is the normal flow for "I want to use
this Go tool in my build" ?

I tried `-mod=mod`:

```
$ (unset GOPATH; go install -mod=mod github.com/google/go-licenses)
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: finding module for package github.com/google/go-licenses
go: downloading github.com/google/go-licenses v0.0.0-20201026145851-73411c8fa237
mkdir /home/thockin/go: not a directory
```

I keep hearing that GOPATH is dead (and frankly good riddance) but
this tries to write to GOPATH or ~/go (which is also annoying).  I
don't want this project's build spilling over into my homedir.

I tried `go get` but it's the same.

```
$ (unset GOPATH; go get github.com/google/go-licenses)
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: writing go.mod cache: mkdir /home/thockin/go: not a directory
go: downloading github.com/google/go-licenses v0.0.0-20201026145851-73411c8fa237
go get github.com/google/go-licenses: mkdir /home/thockin/go: not a directory
```

What do I *want*?   I want to download a tool repo, build it, drop the
binary into ./bin/tools (or something) and then either vendor it or
have no artifacts.

Since GOPATH seems to be a requirement, I thought maybe I can fake it:

```
$ (export GOPATH="`pwd`/.gopath"; export GOBIN=`pwd`/bin/tools; mkdir
-p "$GOBIN"; go get github.com/google/go-licenses)
go: downloading github.com/google/go-licenses v0.0.0-20201026145851-73411c8fa237
go: downloading github.com/otiai10/copy v1.2.0
go: downloading github.com/spf13/cobra v0.0.5
go: downloading github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
go: downloading github.com/google/licenseclassifier
v0.0.0-20190926221455-842c0d70d702
go: downloading golang.org/x/tools v0.0.0-20191118222007-07fc4c7f2b98
go: downloading gopkg.in/src-d/go-git.v4 v4.13.1
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/inconshreveable/mousetrap v1.0.0
go: downloading github.com/sergi/go-diff v1.0.0
go: downloading gopkg.in/src-d/go-billy.v4 v4.3.2
go: downloading golang.org/x/crypto v0.0.0-20191117063200-497ca9f6d64f
go: downloading golang.org/x/sys v0.0.0-20191119060738-e882bf8e40c2
go: downloading github.com/emirpasic/gods v1.12.0
go: downloading github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99
go: downloading github.com/src-d/gcfg v1.4.0
go: downloading github.com/kevinburke/ssh_config
v0.0.0-20190725054713-01f96b0aa0cd
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/xanzy/ssh-agent v0.2.1
go: downloading golang.org/x/net v0.0.0-20191119073136-fc4aabc6c914
go: downloading gopkg.in/warnings.v0 v0.1.2
go get: added github.com/google/go-licenses v0.0.0-20201026145851-73411c8fa237
```

This does actually build