Re: [go-nuts] Build problems with modules in Docker
Hello,
On Wed, Apr 3, 2019 at 3:49 PM David Riley wrote:
> A few things. Responses inline.
>
Will answer inline aswell
>
> > I have a Makefile does some simple tasks for building, it creates a
> tarball of my code directory and starts a docker build -t job.
>
> Why make a tarball? You'll get a lot more mileage out of just copying the
> files from the Docker context, which Docker is already tarring up to pass
> around anyway. You'll get fewer surprises that way.
>
To be honest I failed to test COPY and realize it does a recursive copy of
all files and subdirectories to the image layer.
> In our Dockerfiles, we generally have the following as preamble (we also
> add some non-root permissioning, which is more complex than you really need
> right now):
>
> # Provide arguments for the module name (required) and the
> # optional module proxy for hermetic builds
> ARG MOD_NAME=modname
> ARG GOPROXY
> ENV GOPROXY ${GOPROXY}
>
> # Set a variable for our working directory (make sure it matches
> # the module name)
> ENV D $HOME/build/$MOD_NAME
> RUN mkdir -p $D
> WORKDIR $D
>
> # Copy go.sum/go.mod and warm up the module cache (so that this
> # rather long step can be cached if go.mod/go.sum don't change)
> COPY go.* $D/
> CMD go mod download
>
I think this is a quite clever solution, to put the dependency download in
its own layer. Added it directly to my Dockerfile.
Just the CMD for the `go mod download` must be a RUN, otherwise it does not
downlad but prepares that other kind of entrypoint.
> RUN cd cmd/asm
>
> This doesn't doo what you think it does; it does not change the working
> directory. For that, you want the WORKDIR directive. However...
>
> >
> > Why does the go tool try to kind of resolve the import path of my
> project itself? I thought this would be defined by the module directive in
> my go.mod file, at the source root of my project directory?
>
> It is, the Go tool generally uses that to specify where the package
> "lives" if it's working properly. This looks to me almost like it's not
> seeing go.mod correctly.
>
The error is, as stated above and by Tamás, that my `RUN cd .. ` did not
enter the directory with my main.go file.
In fact, the compiler finds my go.mod and go.sum files and starts the
download and then something strange happens. It does not find any *.go
files as there are none in the top level directory and the message I get
with "can't load package .." is the error message for not finding anything
it could compile. I get this error on my local box as well as soon as I try
to `go build` inside the same root directory.
Now, looking back at yesterday, I understand what confused me the most.
Inside my $GOPATH the error message is quite similar but a bit more
distinct:
can't load package: package github.com/Comradin/go-experiments: no Go files
in /Users/marcus/go/src/github.com/Comradin/go-experiments
`no Go files`, but in the module directory this changes to `unknown import
path`
Many thanks, I raised a level of experience and everything builds just fine
now,
Marcus
--
pedo mellon a minno
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Build problems with modules in Docker
A few things. Responses inline.
> On Apr 3, 2019, at 7:05 AM, Marcus Franke wrote:
>
> Hello,
>
> I have a small project here at work, that does not compile using modules
> inside the golang docker image.
>
> The software resides inside a rather monorepo like repository inside the
> organizations private repository at github. So far, a great place for the
> modules, as I can develop the software outside my GOPATH and building it on
> my machine works great.
>
> My code resides inside this private repository inside an arbitrary path,
> which is not fully part of the name I initiated the module with. Which does
> not impose a problem when building on my laptop.
>
> My go.mod file looks like this:
> ```
> module github.com/org/repo/asm
>
> go 1.12
>
> require (
> github.com/aws/aws-sdk-go v1.19.5
> github.com/kr/pretty v0.1.0
> github.com/stretchr/testify v1.3.0 // indirect
> golang.org/x/net v0.0.0-20190327091125-710a502c58a2 // indirect
> gopkg.in/yaml.v2 v2.2.2
> )
> ```
This generally looks OK.
>
> I have a Makefile does some simple tasks for building, it creates a tarball
> of my code directory and starts a docker build -t job.
Why make a tarball? You'll get a lot more mileage out of just copying the files
from the Docker context, which Docker is already tarring up to pass around
anyway. You'll get fewer surprises that way.
In our Dockerfiles, we generally have the following as preamble (we also add
some non-root permissioning, which is more complex than you really need right
now):
# Provide arguments for the module name (required) and the
# optional module proxy for hermetic builds
ARG MOD_NAME=modname
ARG GOPROXY
ENV GOPROXY ${GOPROXY}
# Set a variable for our working directory (make sure it matches
# the module name)
ENV D $HOME/build/$MOD_NAME
RUN mkdir -p $D
WORKDIR $D
# Copy go.sum/go.mod and warm up the module cache (so that this
# rather long step can be cached if go.mod/go.sum don't change)
COPY go.* $D/
CMD go mod download
# Now copy the rest.
COPY . $D
# Run the build script, which in the end generally calls
# "go install ./cmd/$app1 ./cmd/$app2 ...etc"
RUN ./scripts/build.sh install-all -v
>
> My simplified Dockerfile:
> ```
> FROM golang:1.12
> ENV GO111MODULE=on
> CMD mkdir asm
> WORKDIR /go/asm
> ADD code.tar .
> CMD tar xvf code.tar
Careful with ADD, it automatically untars things when they're compressed
(though yes, this is not currently compressed). Again, though, there's usually
not much reason to do that; you should probably just do a COPY of the files
from your context instead.
> RUN cd cmd/asm
This doesn't doo what you think it does; it does not change the working
directory. For that, you want the WORKDIR directive. However...
> RUN go build -o asm
Instead of trying to change directories, you might want to just say go build
./cmd/asm. You'll do yourself a lot of favors if you do your builds from the
base of your repo
> ```
>
> When I execute the build, I get the following error output:
> ```
> Step 10/10 : RUN go build -o asm
> ---> Running in 243e73e7ed25
> go: finding github.com/stretchr/testify v1.3.0
> go: finding github.com/kr/pretty v0.1.0
> go: finding github.com/aws/aws-sdk-go v1.19.5
> go: finding gopkg.in/yaml.v2 v2.2.2
> go: finding golang.org/x/net v0.0.0-20190327091125-710a502c58a2
> go: finding github.com/kr/text v0.1.0
> go: finding github.com/davecgh/go-spew v1.1.0
> go: finding github.com/pmezard/go-difflib v1.0.0
> go: finding github.com/stretchr/objx v0.1.0
> go: finding gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
> go: finding golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2
> go: finding golang.org/x/text v0.3.0
> go: finding github.com/kr/pty v1.1.1
> go: finding golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a
> go: finding github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af
> can't load package: package github.com/org/repo/asm: unknown import path
> "github.com/org/repo/asm": cannot find module providing package
> github.com/org/repo/asm
> The command '/bin/sh -c go build -o asm' returned a non-zero code: 1
> ```
>
> Why does the go tool try to kind of resolve the import path of my project
> itself? I thought this would be defined by the module directive in my go.mod
> file, at the source root of my project directory?
It is, the Go tool generally uses that to specify where the package "lives" if
it's working properly. This looks to me almost like it's not seeing go.mod
correctly.
>
> My repository contains two internal packages below a pkg/ directory and these
> are being imported just fine with "github.com/org/repo/asm/pkg/foo" and
> "github.com/org/repo/asm/pkg/bar" in my code. On my laptop the compiler can,
> as written above, compile the project just fine. Here it seems it does not
> fumble with finding that particular and rather virtual module name.
>
> Am I doing something wrong or did I just misunderstand the
[go-nuts] Build problems with modules in Docker
Hello, I have a small project here at work, that does not compile using modules inside the golang docker image. The software resides inside a rather monorepo like repository inside the organizations private repository at github. So far, a great place for the modules, as I can develop the software outside my GOPATH and building it on my machine works great. My code resides inside this private repository inside an arbitrary path, which is not fully part of the name I initiated the module with. Which does not impose a problem when building on my laptop. My go.mod file looks like this: ``` module github.com/org/repo/asm go 1.12 require ( github.com/aws/aws-sdk-go v1.19.5 github.com/kr/pretty v0.1.0 github.com/stretchr/testify v1.3.0 // indirect golang.org/x/net v0.0.0-20190327091125-710a502c58a2 // indirect gopkg.in/yaml.v2 v2.2.2 ) ``` I have a Makefile does some simple tasks for building, it creates a tarball of my code directory and starts a docker build -t job. My simplified Dockerfile: ``` FROM golang:1.12 ENV GO111MODULE=on CMD mkdir asm WORKDIR /go/asm ADD code.tar . CMD tar xvf code.tar RUN cd cmd/asm RUN go build -o asm ``` When I execute the build, I get the following error output: ``` Step 10/10 : RUN go build -o asm ---> Running in 243e73e7ed25 go: finding github.com/stretchr/testify v1.3.0 go: finding github.com/kr/pretty v0.1.0 go: finding github.com/aws/aws-sdk-go v1.19.5 go: finding gopkg.in/yaml.v2 v2.2.2 go: finding golang.org/x/net v0.0.0-20190327091125-710a502c58a2 go: finding github.com/kr/text v0.1.0 go: finding github.com/davecgh/go-spew v1.1.0 go: finding github.com/pmezard/go-difflib v1.0.0 go: finding github.com/stretchr/objx v0.1.0 go: finding gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 go: finding golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2 go: finding golang.org/x/text v0.3.0 go: finding github.com/kr/pty v1.1.1 go: finding golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a go: finding github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af can't load package: package github.com/org/repo/asm: unknown import path " github.com/org/repo/asm": cannot find module providing package github.com/org/repo/asm The command '/bin/sh -c go build -o asm' returned a non-zero code: 1 ``` Why does the go tool try to kind of resolve the import path of my project itself? I thought this would be defined by the module directive in my go.mod file, at the source root of my project directory? My repository contains two internal packages below a pkg/ directory and these are being imported just fine with "github.com/org/repo/asm/pkg/foo" and "github.com/org/repo/asm/pkg/bar" in my code. On my laptop the compiler can, as written above, compile the project just fine. Here it seems it does not fumble with finding that particular and rather virtual module name. Am I doing something wrong or did I just misunderstand the way modules work? Kind and puzzled regards, Marcus -- pedo mellon a minno -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
