EGO_SUM mode now supplements the existing EGO_VENDOR mode. EGO_SUM should be populated by the maintainer, directly from the go.sum file of the root package. See eclass and conversion example (dev-go/go-tour & app-admin/kube-bench) for further details.
The go-module_set_globals function performs validation of inputs and does die on fatal errors. Signed-off-by: Robin H. Johnson <robb...@gentoo.org> --- eclass/go-module.eclass | 328 +++++++++++++++++++++++++++++++++++-- profiles/thirdpartymirrors | 1 + 2 files changed, 311 insertions(+), 18 deletions(-) diff --git eclass/go-module.eclass eclass/go-module.eclass index d5de5f60ccdf..b8a635d52de7 100644 --- eclass/go-module.eclass +++ eclass/go-module.eclass @@ -4,22 +4,46 @@ # @ECLASS: go-module.eclass # @MAINTAINER: # William Hubbs <willi...@gentoo.org> +# @AUTHOR: +# William Hubbs <willi...@gentoo.org> +# Robin H. Johnson <robb...@gentoo.org> # @SUPPORTED_EAPIS: 7 # @BLURB: basic eclass for building software written as go modules # @DESCRIPTION: -# This eclass provides basic settings and functions -# needed by all software written in the go programming language that uses -# go modules. +# This eclass provides basic settings and functions needed by all software +# written in the go programming language that uses go modules. +# +# You might know the software you are packaging uses modules because +# it has files named go.sum and go.mod in its top-level source directory. +# If it does not have these files, try use the golang-* eclasses FIRST! +# There ARE legacy Golang packages that use external modules with none of +# go.mod, go.sum, vendor/ that can use this eclass regardless. +# +# Guidelines for usage: +# "go.mod" && "go.sum" && "vendor/": +# - pre-vendored package. Do NOT set EGO_SUM or EGO_VENDOR. +# +# "go.mod" && "go.sum": +# - Populate EGO_SUM with entries from go.sum +# - Do NOT include any lines that contain <version>/go.mod +# +# "go.mod" only: +# - Populate EGO_VENDOR # -# You will know the software you are packaging uses modules because -# it will have files named go.sum and go.mod in its top-level source -# directory. If it does not have these files, use the golang-* eclasses. +# None of the above: +# - Did you try golang-* eclasses first? Upstream has undeclared dependencies +# (perhaps really old source). You can use either EGO_SUM or EGO_VENDOR. + # -# If it has these files and a directory named vendor in its top-level -# source directory, you only need to inherit the eclass since upstream -# is vendoring the dependencies. +# If it has these files AND a directory named "vendor" in its top-level source +# directory, you only need to inherit the eclass since upstream has already +# vendored the dependencies. + +# If it does not have a vendor directory, you should use the EGO_SUM +# variable and the go-module_gosum_uris function as shown in the +# example below to handle dependencies. # -# If it does not have a vendor directory, you should use the EGO_VENDOR +# Alternatively, older versions of this eclass used the EGO_VENDOR # variable and the go-module_vendor_uris function as shown in the # example below to handle dependencies. # @@ -28,6 +52,21 @@ # dependencies. So please make sure it is accurate. # # @EXAMPLE: +# @CODE +# +# inherit go-module +# +# EGO_SUM=( +# "github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=" +# "github.com/BurntSushi/toml v0.3.1/go.mod h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=" +# ) +# S="${WORKDIR}/${MY_P}" +# go-module_set_globals +# +# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz +# ${EGO_SUM_SRC_URI}" +# +# @CODE # # @CODE # @@ -35,7 +74,7 @@ # # EGO_VENDOR=( # "github.com/xenolf/lego 6cac0ea7d8b28c889f709ec7fa92e92b82f490dd" -# "golang.org/x/crypto 453249f01cfeb54c3d549ddb75ff152ca243f9d8 github.com/golang/crypto" +# "golang.org/x/crypto 453249f01cfeb54c3d549ddb75ff152ca243f9d8 github.com/golang/crypto" # ) # # SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz @@ -64,10 +103,12 @@ export GO111MODULE=on export GOCACHE="${T}/go-build" # The following go flags should be used for all builds. -# -mod=vendor stopps downloading of dependencies from the internet. # -v prints the names of packages as they are compiled # -x prints commands as they are executed -export GOFLAGS="-mod=vendor -v -x" +# -mod=vendor use the vendor directory instead of downloading dependencies +# -mod=readonly do not update go.mod/go.sum but fail if updates are needed +export GOFLAGS="-v -x -mod=readonly" +[[ ${#EGO_VENDOR[@]} -gt 0 ]] && GOFLAGS+=" -mod=vendor" # Do not complain about CFLAGS etc since go projects do not use them. QA_FLAGS_IGNORED='.*' @@ -75,7 +116,23 @@ QA_FLAGS_IGNORED='.*' # Go packages should not be stripped with strip(1). RESTRICT="strip" -EXPORT_FUNCTIONS src_unpack pkg_postinst +EXPORT_FUNCTIONS src_unpack src_prepare pkg_postinst + +# @ECLASS-VARIABLE: EGO_SUM +# @DESCRIPTION: +# This variable duplicates the go.sum content from inside the target package. +# Entries of the form <version>/go.mod should be excluded. +# +# <module> <version> <hash> +# +# The format is described upstream here: +# https://tip.golang.org/cmd/go/#hdr-Module_authentication_using_go_sum +# +# <hash> is the Hash1 structure used by upstream Go +# Note that Hash1 is MORE stable than Gentoo distfile hashing, and upstream +# warns that it's conceptually possible for the Hash1 value to remain stable +# while the upstream zipfiles change. E.g. it does NOT capture mtime changes in +# files within a zipfile. # @ECLASS-VARIABLE: EGO_VENDOR # @DESCRIPTION: @@ -106,13 +163,202 @@ go-module_vendor_uris() { done } +# @ECLASS-VARIABLE: GOMODULE_GOPROXY_BASEURI +# @DESCRIPTION: +# Golagg module proxy service to fetch module files from. Note that the module +# proxy generally verifies modules via the Hash1 code. +# +# Note: Users in China may find some mirrors in the list blocked, and may wish +# to an explicit entry to /etc/portage/mirrors pointing mirror://goproxy/ to +# https://goproxy.cn/, or change this variable. +# See https://arslan.io/2019/08/02/why-you-should-use-a-go-module-proxy/ for further details +: "${GOMODULE_GOPROXY_BASEURI:=mirror://goproxy/}" + +# @FUNCTION: go-module_set_globals +# @DESCRIPTION: +# Convert the information in EGO_SUM for other usage in the ebuild. +# - Populates EGO_SUM_SRC_URI that can be added to SRC_URI +# - Exports _EGO_SUM_MAPPING which provides reverse mapping from distfile back +# to the relative part of SRC_URI, as needed for GOPROXY=file:///... +go-module_set_globals() { + local line error_in_gosum errorlines errormsg exts + local newline=$'\n' + error_in_gosum=0 + errorlines=( ) + for line in "${EGO_SUM[@]}"; do + local module version modfile version_modfile hash1 x + read -r module version_modfile hash1 x <<< "${line}" + # Validate input + if [[ -n $hash1 ]] && [[ ${hash1:0:3} != "h1:" ]] ; then + error_in_gosum=1 + errorlines+=( "Unknown hash: ${line}" ) + elif [[ -n $x ]]; then + error_in_gosum=1 + errorlines+=( "Trailing data: ${line}" ) + fi + + # Split 'v0.3.0/go.mod' into 'v0.3.0' and '/go.mod' + version=${version_modfile%%/*} + modfile=${version_modfile#*/} + [[ "$modfile" == "${version_modfile}" ]] && modfile= + + # The trailing part should be either empty or '/go.mod' + # There is a chance that upstream Go might add something else here in + # future, and we should be prepared to capture it. + exts=() + errormsg='' + case "$modfile" in + '') exts=( mod info zip ) ;; + 'go.mod'|'/go.mod') exts=( mod info ) ;; + #'go.mod'|'/go.mod') errormsg="Prohibited file: You must exclude /go.mod lines from EGO_SUM! " ;; + *) errormsg="Unknown modfile: line='${line}', modfile='${modfile}'" ;; + esac + + # If it was a bad entry, restart the loop + if [[ -n $errormsg ]]; then + error_in_gosum=1 + errorlines+=( "${errormsg} line='${line}', modfile='${modfile}'" ) + continue + fi + + # Directory structure for Go proxy hosts: + # - def encode(s): + # return re.sub('([A-Z]{1})', r'!\1', s).lower() + # + # Sed variant: + # This uses GNU Sed extension \l to downcase the match + #_dir=$(echo "${module}" |sed 's,[A-Z],!\l&,g') + # + # Bash variant: + re='(.*)([A-Z])(.*)' + input=${module} + while [[ $input =~ $re ]]; do + lower='!'"${BASH_REMATCH[2],}" + input="${BASH_REMATCH[1]}${lower}${BASH_REMATCH[3]}" + done + _dir=$input + unset lower input re + + for _ext in "${exts[@]}" ; do + # Relative URI within a GOPROXY for a file + _reluri="${_dir}/@v/${version}.${_ext}" + # SRC_URI: LHS entry + _uri="${GOMODULE_GOPROXY_BASEURI}/${_reluri}" + # SRC_URI: RHS entry, encode any slash in the path as %2F in the filename + _distfile="${_reluri//\//%2F}" + + EGO_SUM_SRC_URI+=" ${_uri} -> ${_distfile}${newline}" + _EGO_SUM_MAPPING+=" ${_distfile}:${_reluri}${newline}" + done + done + + if [[ $error_in_gosum != 0 ]]; then + eerror "Trailing information in EGO_SUM in ${P}.ebuild" + for line in "${errorlines[@]}" ; do + eerror "${line}" + done + die "Invalid EGO_SUM format" + fi + + # Ensure these variables not not changed past this point + readonly EGO_SUM + readonly EGO_SUM_SRC_URI + readonly _EGO_SUM_MAPPING + + # Set the guard that we are safe + _GO_MODULE_SET_GLOBALS_CALLED=1 +} + + # @FUNCTION: go-module_src_unpack # @DESCRIPTION: +# Extract & configure Go modules for consumpations. +# - Modules listed in EGO_SUM are configured as a local GOPROXY via symlinks (fast!) +# - Modules listed in EGO_VENDOR are extracted to "${S}/vendor" (slow) +# +# This function does NOT unpack the base distfile of a Go-based package. +# While the entries in EGO_SUM will be listed in ${A}, they should NOT be +# unpacked, Go will directly consume the files, including zips. +go-module_src_unpack() { + if [[ "${#EGO_VENDOR[@]}" -gt 0 ]]; then + _go-module_src_unpack_vendor + elif [[ "${#EGO_SUM[@]}" -gt 0 ]]; then + _go-module_src_unpack_gosum + else + die "Neither EGO_SUM nor EGO_VENDOR are set!" + fi +} + +# @FUNCTION: go-module_src_prepare +# @DESCRIPTION: +# Prepare for building. Presently only needed for EGO_SUM variant. +go-module_src_prepare() { + # shellcheck disable=SC2120 + debug-print-function "${FUNCNAME}" "$@" + + if [[ "${#EGO_SUM[@]}" -gt 0 ]]; then + _go-module_src_prepare_gosum + fi + + default +} + +# @ECLASS-VARIABLE: GOMODULE_GOSUM_PATH +# @DESCRIPTION: +# Path to root go.sum of package. If your ebuild modifies S after inheriting +# the eclass, you may need to update this variable. +: "${GO_MODULE_GOSUM_PATH:=${S}/go.sum}" + +# @FUNCTION: _go-module_src_unpack_gosum +# @DESCRIPTION: +# Populate a GOPROXY directory hierarchy with distfiles from EGO_SUM +# +# Exports GOPROXY environment variable so that Go calls will source the +# directory correctly. +_go-module_src_unpack_gosum() { + # shellcheck disable=SC2120 + debug-print-function "${FUNCNAME}" "$@" + + if [[ ! ${_GO_MODULE_SET_GLOBALS_CALLED} ]]; then + die "go-module_set_globals must be called in global scope" + fi + + local goproxy_dir="${T}/goproxy" + local goproxy_mod_dir + mkdir -p "${goproxy_dir}" + # Convert the list format to an associative array to avoid O(N*M) + # performance when populating the GOPROXY directory structure. + declare -A _EGO_SUM_MAPPING_ASSOC + for s in ${_EGO_SUM_MAPPING}; do + a=${s//:*} + b=${s//*:} + _EGO_SUM_MAPPING_ASSOC["$a"]=$b + done + + # For each Golang module distfile, look up where it's supposed to go, and + # symlink into place. + for _A in ${A}; do + goproxy_mod_path="${_EGO_SUM_MAPPING_ASSOC["${_A}"]}" + if [[ -n "${goproxy_mod_path}" ]]; then + einfo "Populating goproxy for $goproxy_mod_path" + # Build symlink hierarchy + goproxy_mod_dir=$( dirname "${goproxy_dir}"/"${goproxy_mod_path}" ) + mkdir -p "${goproxy_mod_dir}" + ln -sf "${DISTDIR}"/"${_A}" "${goproxy_dir}/${goproxy_mod_path}" || die "Failed to ln" + fi + done + export GOPROXY="file://${goproxy_dir}" + unset _EGO_SUM_MAPPING_ASSOC +} + +# @FUNCTION: _go-module_src_unpack_vendor +# @DESCRIPTION: # Extract all archives in ${a} which are not nentioned in ${EGO_VENDOR} # to their usual locations then extract all archives mentioned in # ${EGO_VENDOR} to ${S}/vendor. -go-module_src_unpack() { - debug-print-function ${FUNCNAME} "$@" +_go-module_src_unpack_vendor() { + # shellcheck disable=SC2120 + debug-print-function "${FUNCNAME}" "$@" local f hash import line repo tarball vendor_tarballs x vendor_tarballs=() for line in "${EGO_VENDOR[@]}"; do @@ -145,13 +391,59 @@ go-module_src_unpack() { done } +# @FUNCTION: _go-module_src_prepare_gosum +# @DESCRIPTION: +# Validate the Go modules declared by EGO_SUM are sufficent to cover building +# the package, without actually building it yet. +_go-module_src_prepare_gosum() { + # shellcheck disable=SC2120 + debug-print-function "${FUNCNAME}" "$@" + + if [[ ! ${_GO_MODULE_SET_GLOBALS_CALLED} ]]; then + die "go-module_set_globals must be called in global scope" + fi + + # go.sum entries ending in /go.mod aren't strictly needed at this phase + if [[ ! -e "${GO_MODULE_GOSUM_PATH}" ]]; then + die "Could not find package root go.sum, please update GO_MODULE_GOSUM_PATH" + fi + go-module_minimize_gosum "${GO_MODULE_GOSUM_PATH}" + + # Verify that all needed modules are present. + GO111MODULE=on \ + go get -v -d -mod readonly || die "Some module is missing, update EGO_SUM" + + # Need to re-minimize because go-get expands it again + go-module_minimize_gosum "${GO_MODULE_GOSUM_PATH}" +} + +# @FUNCTION: go-module_minimize_gosum +# @DESCRIPTION: +# Remove all /go.mod entries from go.sum files +# In most cases, if go.sum only has a /go.mod entry without a corresponding +# direct entry, this is a sign of a weak dependency that is NOT required for +# building the package. +go-module_minimize_gosum() { + local gosumfile=${1} + if test ! -e "${gosumfile}".orig; then + cp -f "${gosumfile}"{,.orig} || die + fi + awk -e '$2 ~ /\/go.mod$/{next} {print}' \ + <"${gosumfile}".orig \ + >"${gosumfile}" || die + if grep -sq /go.mod "${gosumfile}"; then + die "sed failed to remove all module go.mod entries from go.sum" + fi +} + # @FUNCTION: go-module_live_vendor # @DESCRIPTION: # This function is used in live ebuilds to vendor the dependencies when # upstream doesn't vendor them. go-module_live_vendor() { - debug-print-function ${FUNCNAME} "$@" + debug-print-function "${FUNCNAME}" "$@" + # shellcheck disable=SC2086 has live ${PROPERTIES} || die "${FUNCNAME} only allowed in live ebuilds" [[ "${EBUILD_PHASE}" == unpack ]] || @@ -168,7 +460,7 @@ go-module_live_vendor() { # @DESCRIPTION: # Display a warning about security updates for Go programs. go-module_pkg_postinst() { - debug-print-function ${FUNCNAME} "$@" + debug-print-function "${FUNCNAME}" "$@" [[ -n ${REPLACING_VERSIONS} ]] && return 0 ewarn "${PN} is written in the Go programming language." ewarn "Since this language is statically linked, security" diff --git profiles/thirdpartymirrors profiles/thirdpartymirrors index ad4c4b972146..d60f166e07c9 100644 --- profiles/thirdpartymirrors +++ profiles/thirdpartymirrors @@ -25,3 +25,4 @@ sourceforge https://download.sourceforge.net sourceforge.jp http://iij.dl.sourceforge.jp https://osdn.dl.sourceforge.jp https://jaist.dl.sourceforge.jp ubuntu http://mirror.internode.on.net/pub/ubuntu/ubuntu/ https://mirror.tcc.wa.edu.au/ubuntu/ http://ubuntu.uni-klu.ac.at/ubuntu/ http://mirror.dhakacom.com/ubuntu-archive/ http://ubuntu.c3sl.ufpr.br/ubuntu/ http://ubuntu.uni-sofia.bg/ubuntu/ http://hr.archive.ubuntu.com/ubuntu/ http://cz.archive.ubuntu.com/ubuntu/ https://mirror.dkm.cz/ubuntu http://ftp.cvut.cz/ubuntu/ http://ftp.stw-bonn.de/ubuntu/ https://ftp-stud.hs-esslingen.de/ubuntu/ https://mirror.netcologne.de/ubuntu/ https://mirror.unej.ac.id/ubuntu/ http://kr.archive.ubuntu.com/ubuntu/ https://mirror.nforce.com/pub/linux/ubuntu/ http://mirror.amsiohosting.net/archive.ubuntu.com/ http://nl3.archive.ubuntu.com/ubuntu/ https://mirror.timeweb.ru/ubuntu/ http://ubuntu.mirror.su.se/ubuntu/ https://ftp.yzu.edu.tw/ubuntu/ https://mirror.aptus.co.tz/pub/ubuntuarchive/ https://ubuntu.volia.net/ubuntu-archive/ https://mirror.sax.uk.as61049.net/ubuntu/ https://mirror.pnl.gov/ubuntu/ http://mirror.cc.columbia.edu/pub/linux/ubuntu/archive/ https://mirrors.namecheap.com/ubuntu/ vdr-developerorg http://projects.vdr-developer.org/attachments/download +goproxy https://proxy.golang.org/ https://goproxy.io/ https://gocenter.io/ -- 2.25.0