Re: [gentoo-dev] [PATCH 1/3] eclass/go-module: add support for building based on go.sum
Stay tuned for v2 with major improvements based on talking to upstream.
- Dropped an entire distfile per golang module (down to 1-2 files per
module in go.sum)
- Better Go 1.13 support (some semantics changed slightly from 1.12)
- Easier way to track & include licenses of all the modules
On Thu, Feb 13, 2020 at 05:57:57PM +0100, Michał Górny wrote:
> > +# EGO_SUM=(
> > +# "github.com/BurntSushi/toml v0.3.1
> > h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ="
> > +# "github.com/BurntSushi/toml v0.3.1/go.mod
> > h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ="
> Is it expected that the two entries would have the same hash?
In this case, they SHOULD have been different, but it does happen in
reality for the /go.mod entries, here's an example:
github.com/stretchr/testify v1.2.1/go.mod
h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.2.2/go.mod
h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.3.0/go.mod
h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod
h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
1.2.1->1.2.2 there was no change in the dependencies, so the file didn't change.
> > -EXPORT_FUNCTIONS src_unpack pkg_postinst
> > +EXPORT_FUNCTIONS src_unpack src_prepare pkg_postinst
> Exporting a new phase looks potentially dangerous. Are you sure no
> ebuilds are broken by this?
I'm not sure, so I've rolled it into src_unpack for now.
>
> > +
> > +# @ECLASS-VARIABLE: EGO_SUM
> > +# @DESCRIPTION:
> > +# This variable duplicates the go.sum content from inside the target
> > package.
> > +# Entries of the form /go.mod should be excluded.
> ...but you've included one of them in the example on top of the eclass.
There was an ongoing discussion with upstream, where I was trying to
trim down the number of distfiles involved. Sadly generating the .mod
files turns out to have some non-trivial corner cases
I did manage to drop the .info files at least, so it's 1-2 distfiles per
dependency now.
>
> > +#
> > +#
>
> Now I'm confused. Unless my eyes betray me, PATCH 2 has entries without
> hash.
>
> Also, the description fails to mention that you're supposed to quote
> each line.
Improved in documentation, and covered why the hash is optional right
now.
> > +# The format is described upstream here:
> > +# https://tip.golang.org/cmd/go/#hdr-Module_authentication_using_go_sum
...
> I think it would be valuable to include an example here as well.
Done.
> > +# proxy generally verifies modules via the Hash1 code.
> > +#
> > +# Note: Users in China may find some mirrors in the list blocked, and may
> > wish
> > +# to an explicit entry to /etc/portage/mirrors pointing mirror://goproxy/
> > to
> > +# https://goproxy.cn/, or change this variable.
> > +# See https://arslan.io/2019/08/02/why-you-should-use-a-go-module-proxy/
> > for further details
> > +: "${GOMODULE_GOPROXY_BASEURI:=mirror://goproxy/}"
> 'Changing this variable' sounds like violating metadata immutability
> rule and running in trouble with the caches.
Covered who & why this should be set, esp wrt to immutability.
> > +
> > +# @FUNCTION: go-module_set_globals
> > +# @DESCRIPTION:
> > +# Convert the information in EGO_SUM for other usage in the ebuild.
> > +# - Populates EGO_SUM_SRC_URI that can be added to SRC_URI
> > +# - Exports _EGO_SUM_MAPPING which provides reverse mapping from distfile
> > back
> > +# to the relative part of SRC_URI, as needed for GOPROXY=file:///...
> > +go-module_set_globals() {
> > + local line error_in_gosum errorlines errormsg exts
> > + local newline=$'\n'
> > + error_in_gosum=0
> > + errorlines=( )
> > + for line in "${EGO_SUM[@]}"; do
> > + local module version modfile version_modfile hash1 x
> > + read -r module version_modfile hash1 x <<< "${line}"
> > + # Validate input
> > + if [[ -n $hash1 ]] && [[ ${hash1:0:3} != "h1:" ]] ; then
>
> Please use ${foo} everywhere consistently, and put && inside [[ ]].
> Also, I dare say wildcard match is more readable than hardcoding string
> length, i.e.:
>
> [[ -n ${hash1} && ${hash1} != h1:* ]]
...
> > + # Split 'v0.3.0/go.mod' into 'v0.3.0' and '/go.mod'
> > + version=${version_modfile%%/*}
> > + modfile=${version_modfile#*/}
> > + [[ "$modfile" == "${version_modfile}" ]] && modfile=
> Check the initial string, not the result of arbitrary manipulations
> on it. This would wrongly evaluate true for 'v0.3.0/v0.3.0'.
Reworked this
> > +go-module_src_unpack() {
> > + if [[ "${#EGO_VENDOR[@]}" -gt 0 ]]; then
> > + _go-module_src_unpack_vendor
> > + elif [[ "${#EGO_SUM[@]}" -gt 0 ]]; then
> > + _go-module_src_unpack_gosum
> Does that mean those two are mutually exclusive?
Yes.
> > +# @FUNCTION: go-module_src_prepare
...
> Wouldn't it be better to append this to src_unpack? Overriding
> src_prepare is generally problematic, and as I've said above,
Re: [gentoo-dev] [PATCH 1/3] eclass/go-module: add support for building based on go.sum
On Sun, 2020-02-09 at 12:31 -0800, Robin H. Johnson wrote:
> EGO_SUM mode now supplements the existing EGO_VENDOR mode.
>
> EGO_SUM should be populated by the maintainer, directly from the go.sum
> file of the root package. See eclass and conversion example
> (dev-go/go-tour & app-admin/kube-bench) for further details.
>
> The go-module_set_globals function performs validation of
> inputs and does die on fatal errors.
>
> Signed-off-by: Robin H. Johnson
> ---
> eclass/go-module.eclass| 328 +++--
> profiles/thirdpartymirrors | 1 +
> 2 files changed, 311 insertions(+), 18 deletions(-)
>
> diff --git eclass/go-module.eclass eclass/go-module.eclass
> index d5de5f60ccdf..b8a635d52de7 100644
> --- eclass/go-module.eclass
> +++ eclass/go-module.eclass
> @@ -4,22 +4,46 @@
> # @ECLASS: go-module.eclass
> # @MAINTAINER:
> # William Hubbs
> +# @AUTHOR:
> +# William Hubbs
> +# Robin H. Johnson
> # @SUPPORTED_EAPIS: 7
> # @BLURB: basic eclass for building software written as go modules
> # @DESCRIPTION:
> -# This eclass provides basic settings and functions
> -# needed by all software written in the go programming language that uses
> -# go modules.
> +# This eclass provides basic settings and functions needed by all software
> +# written in the go programming language that uses go modules.
> +#
> +# You might know the software you are packaging uses modules because
> +# it has files named go.sum and go.mod in its top-level source directory.
> +# If it does not have these files, try use the golang-* eclasses FIRST!
> +# There ARE legacy Golang packages that use external modules with none of
> +# go.mod, go.sum, vendor/ that can use this eclass regardless.
> +#
> +# Guidelines for usage:
> +# "go.mod" && "go.sum" && "vendor/":
> +# - pre-vendored package. Do NOT set EGO_SUM or EGO_VENDOR.
> +#
> +# "go.mod" && "go.sum":
> +# - Populate EGO_SUM with entries from go.sum
> +# - Do NOT include any lines that contain /go.mod
> +#
> +# "go.mod" only:
> +# - Populate EGO_VENDOR
> #
> -# You will know the software you are packaging uses modules because
> -# it will have files named go.sum and go.mod in its top-level source
> -# directory. If it does not have these files, use the golang-* eclasses.
> +# None of the above:
> +# - Did you try golang-* eclasses first? Upstream has undeclared dependencies
> +# (perhaps really old source). You can use either EGO_SUM or EGO_VENDOR.
> +
> #
> -# If it has these files and a directory named vendor in its top-level
> -# source directory, you only need to inherit the eclass since upstream
> -# is vendoring the dependencies.
> +# If it has these files AND a directory named "vendor" in its top-level
> source
> +# directory, you only need to inherit the eclass since upstream has already
> +# vendored the dependencies.
> +
> +# If it does not have a vendor directory, you should use the EGO_SUM
> +# variable and the go-module_gosum_uris function as shown in the
> +# example below to handle dependencies.
> #
> -# If it does not have a vendor directory, you should use the EGO_VENDOR
> +# Alternatively, older versions of this eclass used the EGO_VENDOR
> # variable and the go-module_vendor_uris function as shown in the
> # example below to handle dependencies.
> #
> @@ -28,6 +52,21 @@
> # dependencies. So please make sure it is accurate.
> #
> # @EXAMPLE:
> +# @CODE
> +#
> +# inherit go-module
> +#
> +# EGO_SUM=(
> +#"github.com/BurntSushi/toml v0.3.1
> h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ="
> +#"github.com/BurntSushi/toml v0.3.1/go.mod
> h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ="
Is it expected that the two entries would have the same hash?
> +# )
> +# S="${WORKDIR}/${MY_P}"
> +# go-module_set_globals
> +#
> +# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz ->
> ${P}.tar.gz
> +# ${EGO_SUM_SRC_URI}"
> +#
> +# @CODE
> #
> # @CODE
> #
> @@ -35,7 +74,7 @@
> #
> # EGO_VENDOR=(
> #"github.com/xenolf/lego 6cac0ea7d8b28c889f709ec7fa92e92b82f490dd"
> -# "golang.org/x/crypto 453249f01cfeb54c3d549ddb75ff152ca243f9d8
> github.com/golang/crypto"
> +#"golang.org/x/crypto 453249f01cfeb54c3d549ddb75ff152ca243f9d8
> github.com/golang/crypto"
> # )
> #
> # SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz ->
> ${P}.tar.gz
> @@ -64,10 +103,12 @@ export GO111MODULE=on
> export GOCACHE="${T}/go-build"
>
> # The following go flags should be used for all builds.
> -# -mod=vendor stopps downloading of dependencies from the internet.
> # -v prints the names of packages as they are compiled
> # -x prints commands as they are executed
> -export GOFLAGS="-mod=vendor -v -x"
> +# -mod=vendor use the vendor directory instead of downloading dependencies
> +# -mod=readonly do not update go.mod/go.sum but fail if updates are needed
> +export GOFLAGS="-v -x -mod=readonly"
> +[[ ${#EGO_VENDOR[@]} -gt 0 ]] && GOFLAGS+=" -mod=vendor"
>
> # Do not complain about CFLAGS etc since
Re: [gentoo-dev] [PATCH 1/3] eclass/go-module: add support for building based on go.sum
On Sun, Feb 09, 2020 at 11:35:25PM +, Robin H. Johnson wrote: > On Sun, Feb 09, 2020 at 04:11:28PM -0600, William Hubbs wrote: > > On Sun, Feb 09, 2020 at 12:31:19PM -0800, Robin H. Johnson wrote: > > > +# "go.mod" only: > > > +# - Populate EGO_VENDOR > > go.mod without go.sum can mean that there are no external dependencies, so > > there > > shouldn't be a reason to populate EGO_VENDOR in this case. > ... > > The way I see this going is to transition to EGO_SUM and > > drop EGO_VENDOR. unless I'm missing something. > I know another corner case for legacy stuff, but let's drop entirely and > encourage migration to EGO_SUM. I'm curious, what is the corner case I'm missing? Thanks, William signature.asc Description: Digital signature
Re: [gentoo-dev] [PATCH 1/3] eclass/go-module: add support for building based on go.sum
On Sun, Feb 09, 2020 at 04:11:28PM -0600, William Hubbs wrote: > On Sun, Feb 09, 2020 at 12:31:19PM -0800, Robin H. Johnson wrote: > > +# "go.mod" only: > > +# - Populate EGO_VENDOR > go.mod without go.sum can mean that there are no external dependencies, so > there > shouldn't be a reason to populate EGO_VENDOR in this case. ... > The way I see this going is to transition to EGO_SUM and > drop EGO_VENDOR. unless I'm missing something. I know another corner case for legacy stuff, but let's drop entirely and encourage migration to EGO_SUM. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: PGP signature
Re: [gentoo-dev] [PATCH 1/3] eclass/go-module: add support for building based on go.sum
On Sun, Feb 09, 2020 at 12:31:19PM -0800, Robin H. Johnson wrote: > EGO_SUM mode now supplements the existing EGO_VENDOR mode. > > EGO_SUM should be populated by the maintainer, directly from the go.sum > file of the root package. See eclass and conversion example > (dev-go/go-tour & app-admin/kube-bench) for further details. > > The go-module_set_globals function performs validation of > inputs and does die on fatal errors. > > Signed-off-by: Robin H. Johnson > --- > eclass/go-module.eclass| 328 +++-- > profiles/thirdpartymirrors | 1 + > 2 files changed, 311 insertions(+), 18 deletions(-) > > diff --git eclass/go-module.eclass eclass/go-module.eclass > index d5de5f60ccdf..b8a635d52de7 100644 > --- eclass/go-module.eclass > +++ eclass/go-module.eclass > @@ -4,22 +4,46 @@ > # @ECLASS: go-module.eclass > # @MAINTAINER: > # William Hubbs > +# @AUTHOR: > +# William Hubbs > +# Robin H. Johnson > # @SUPPORTED_EAPIS: 7 > # @BLURB: basic eclass for building software written as go modules > # @DESCRIPTION: > -# This eclass provides basic settings and functions > -# needed by all software written in the go programming language that uses > -# go modules. > +# This eclass provides basic settings and functions needed by all software > +# written in the go programming language that uses go modules. > +# > +# You might know the software you are packaging uses modules because > +# it has files named go.sum and go.mod in its top-level source directory. > +# If it does not have these files, try use the golang-* eclasses FIRST! > +# There ARE legacy Golang packages that use external modules with none of > +# go.mod, go.sum, vendor/ that can use this eclass regardless. > +# > +# Guidelines for usage: > +# "go.mod" && "go.sum" && "vendor/": > +# - pre-vendored package. Do NOT set EGO_SUM or EGO_VENDOR. > +# > +# "go.mod" && "go.sum": > +# - Populate EGO_SUM with entries from go.sum > +# - Do NOT include any lines that contain /go.mod > +# > +# "go.mod" only: > +# - Populate EGO_VENDOR go.mod without go.sum can mean that there are no external dependencies, so there shouldn't be a reason to populate EGO_VENDOR in this case. Here is a valid go.mod: --- cut here --- module github.com/williamh/get-ego-vendor go 1.12 --- cut here --- If go.mod has require lines in it and go.sum doesn't exist, this is an issue to address upstream. The way I see this going is to transition to EGO_SUM and drop EGO_VENDOR. unless I'm missing something. Thanks, William signature.asc Description: Digital signature
[gentoo-dev] [PATCH 1/3] eclass/go-module: add support for building based on go.sum
EGO_SUM mode now supplements the existing EGO_VENDOR mode.
EGO_SUM should be populated by the maintainer, directly from the go.sum
file of the root package. See eclass and conversion example
(dev-go/go-tour & app-admin/kube-bench) for further details.
The go-module_set_globals function performs validation of
inputs and does die on fatal errors.
Signed-off-by: Robin H. Johnson
---
eclass/go-module.eclass| 328 +++--
profiles/thirdpartymirrors | 1 +
2 files changed, 311 insertions(+), 18 deletions(-)
diff --git eclass/go-module.eclass eclass/go-module.eclass
index d5de5f60ccdf..b8a635d52de7 100644
--- eclass/go-module.eclass
+++ eclass/go-module.eclass
@@ -4,22 +4,46 @@
# @ECLASS: go-module.eclass
# @MAINTAINER:
# William Hubbs
+# @AUTHOR:
+# William Hubbs
+# Robin H. Johnson
# @SUPPORTED_EAPIS: 7
# @BLURB: basic eclass for building software written as go modules
# @DESCRIPTION:
-# This eclass provides basic settings and functions
-# needed by all software written in the go programming language that uses
-# go modules.
+# This eclass provides basic settings and functions needed by all software
+# written in the go programming language that uses go modules.
+#
+# You might know the software you are packaging uses modules because
+# it has files named go.sum and go.mod in its top-level source directory.
+# If it does not have these files, try use the golang-* eclasses FIRST!
+# There ARE legacy Golang packages that use external modules with none of
+# go.mod, go.sum, vendor/ that can use this eclass regardless.
+#
+# Guidelines for usage:
+# "go.mod" && "go.sum" && "vendor/":
+# - pre-vendored package. Do NOT set EGO_SUM or EGO_VENDOR.
+#
+# "go.mod" && "go.sum":
+# - Populate EGO_SUM with entries from go.sum
+# - Do NOT include any lines that contain /go.mod
+#
+# "go.mod" only:
+# - Populate EGO_VENDOR
#
-# You will know the software you are packaging uses modules because
-# it will have files named go.sum and go.mod in its top-level source
-# directory. If it does not have these files, use the golang-* eclasses.
+# None of the above:
+# - Did you try golang-* eclasses first? Upstream has undeclared dependencies
+# (perhaps really old source). You can use either EGO_SUM or EGO_VENDOR.
+
#
-# If it has these files and a directory named vendor in its top-level
-# source directory, you only need to inherit the eclass since upstream
-# is vendoring the dependencies.
+# If it has these files AND a directory named "vendor" in its top-level source
+# directory, you only need to inherit the eclass since upstream has already
+# vendored the dependencies.
+
+# If it does not have a vendor directory, you should use the EGO_SUM
+# variable and the go-module_gosum_uris function as shown in the
+# example below to handle dependencies.
#
-# If it does not have a vendor directory, you should use the EGO_VENDOR
+# Alternatively, older versions of this eclass used the EGO_VENDOR
# variable and the go-module_vendor_uris function as shown in the
# example below to handle dependencies.
#
@@ -28,6 +52,21 @@
# dependencies. So please make sure it is accurate.
#
# @EXAMPLE:
+# @CODE
+#
+# inherit go-module
+#
+# EGO_SUM=(
+# "github.com/BurntSushi/toml v0.3.1
h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ="
+# "github.com/BurntSushi/toml v0.3.1/go.mod
h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ="
+# )
+# S="${WORKDIR}/${MY_P}"
+# go-module_set_globals
+#
+# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz ->
${P}.tar.gz
+# ${EGO_SUM_SRC_URI}"
+#
+# @CODE
#
# @CODE
#
@@ -35,7 +74,7 @@
#
# EGO_VENDOR=(
# "github.com/xenolf/lego 6cac0ea7d8b28c889f709ec7fa92e92b82f490dd"
-# "golang.org/x/crypto 453249f01cfeb54c3d549ddb75ff152ca243f9d8
github.com/golang/crypto"
+# "golang.org/x/crypto 453249f01cfeb54c3d549ddb75ff152ca243f9d8
github.com/golang/crypto"
# )
#
# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz ->
${P}.tar.gz
@@ -64,10 +103,12 @@ export GO111MODULE=on
export GOCACHE="${T}/go-build"
# The following go flags should be used for all builds.
-# -mod=vendor stopps downloading of dependencies from the internet.
# -v prints the names of packages as they are compiled
# -x prints commands as they are executed
-export GOFLAGS="-mod=vendor -v -x"
+# -mod=vendor use the vendor directory instead of downloading dependencies
+# -mod=readonly do not update go.mod/go.sum but fail if updates are needed
+export GOFLAGS="-v -x -mod=readonly"
+[[ ${#EGO_VENDOR[@]} -gt 0 ]] && GOFLAGS+=" -mod=vendor"
# Do not complain about CFLAGS etc since go projects do not use them.
QA_FLAGS_IGNORED='.*'
@@ -75,7 +116,23 @@ QA_FLAGS_IGNORED='.*'
# Go packages should not be stripped with strip(1).
RESTRICT="strip"
-EXPORT_FUNCTIONS src_unpack pkg_postinst
+EXPORT_FUNCTIONS src_unpack src_prepare pkg_postinst
+
+# @ECLASS-VARIABLE: EGO_SUM
+# @DESCRIPTION:
+# This variable duplicates the go.sum content from inside the
