EGO_SUM mode now supplements the existing EGO_VENDOR mode. EGO_SUM should be populated by the maintainer, directly from the go.sum file of the root package. See eclass and conversion examples for further details: dev-go/go-tour, app-admin/kube-bench, dev-vcs/cli
The go-module_set_globals function performs validation of inputs and dies on fatal errors. Signed-off-by: Robin H. Johnson <[email protected]> --- eclass/go-module.eclass | 419 +++++++++++++++++++++++++++++++++++-- profiles/thirdpartymirrors | 1 + 2 files changed, 397 insertions(+), 23 deletions(-) diff --git eclass/go-module.eclass eclass/go-module.eclass index 80ff2902b3ad..50aff92735af 100644 --- eclass/go-module.eclass +++ eclass/go-module.eclass @@ -4,22 +4,45 @@ # @ECLASS: go-module.eclass # @MAINTAINER: # William Hubbs <[email protected]> +# @AUTHOR: +# William Hubbs <[email protected]> +# Robin H. Johnson <[email protected]> # @SUPPORTED_EAPIS: 7 # @BLURB: basic eclass for building software written as go modules # @DESCRIPTION: -# This eclass provides basic settings and functions -# needed by all software written in the go programming language that uses -# go modules. +# This eclass provides basic settings and functions needed by all software +# written in the go programming language that uses go modules. +# +# You might know the software you are packaging uses modules because +# it has files named go.sum and go.mod in its top-level source directory. +# If it does not have these files, try use the golang-* eclasses FIRST! +# There ARE legacy Golang packages that use external modules with none of +# go.mod, go.sum, vendor/ that can use this eclass regardless. +# +# Guidelines for usage: +# "vendor/": +# - pre-vendored package. Do NOT set EGO_SUM or EGO_VENDOR. # -# You will know the software you are packaging uses modules because -# it will have files named go.sum and go.mod in its top-level source -# directory. If it does not have these files, use the golang-* eclasses. +# "go.mod" && "go.sum": +# - Populate EGO_SUM with entries from go.sum +# - Append license:${GENTOO_LICENSE} to any modules needed at runtime. +# dev-go/golicense can tell you which modules in a Golang binary are used at +# runtime (requires network connectivity). # -# If it has these files and a directory named vendor in its top-level -# source directory, you only need to inherit the eclass since upstream -# is vendoring the dependencies. +# None of the above: +# - Did you try golang-* eclasses first? Upstream has undeclared dependencies +# (perhaps really old source). You can use either EGO_SUM or EGO_VENDOR. + +# +# If it has these files AND a directory named "vendor" in its top-level source +# directory, you only need to inherit the eclass since upstream has already +# vendored the dependencies. + +# If it does not have a vendor directory, you should use the EGO_SUM +# variable and the go-module_gosum_uris function as shown in the +# example below to handle dependencies. # -# If it does not have a vendor directory, you should use the EGO_VENDOR +# Alternatively, older versions of this eclass used the EGO_VENDOR # variable and the go-module_vendor_uris function as shown in the # example below to handle dependencies. # @@ -28,6 +51,28 @@ # dependencies. So please make sure it is accurate. # # @EXAMPLE: +# @CODE +# +# inherit go-module +# +# EGO_SUM=( +# "github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59/go.mod h1:q/89r3U2H7sSsE2t6Kca0lfwTK8JdoNGS/yzM/4iH5I= license:BSD-2,MIT" +# "github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59 h1:WWB576BN5zNSZc/M9d/10pqEx5VHNhaQ/yOVAkmj5Yo=" +# ) +# S="${WORKDIR}/${MY_P}" +# go-module_set_globals +# +# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz +# ${EGO_SUM_SRC_URI}" +# +# LICENSE="some-license ${EGO_SUM_LICENSES}" +# +# src_unpack() { +# unpack ${P}.tar.gz +# go-module_src_unpack +# } +# +# @CODE # # @CODE # @@ -35,7 +80,7 @@ # # EGO_VENDOR=( # "github.com/xenolf/lego 6cac0ea7d8b28c889f709ec7fa92e92b82f490dd" -# "golang.org/x/crypto 453249f01cfeb54c3d549ddb75ff152ca243f9d8 github.com/golang/crypto" +# "golang.org/x/crypto 453249f01cfeb54c3d549ddb75ff152ca243f9d8 github.com/golang/crypto" # ) # # SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz @@ -64,10 +109,12 @@ export GO111MODULE=on export GOCACHE="${T}/go-build" # The following go flags should be used for all builds. -# -mod=vendor stopps downloading of dependencies from the internet. # -v prints the names of packages as they are compiled # -x prints commands as they are executed -export GOFLAGS="-mod=vendor -v -x" +# -mod=vendor use the vendor directory instead of downloading dependencies +# -mod=readonly do not update go.mod/go.sum but fail if updates are needed +export GOFLAGS="-v -x -mod=readonly" +[[ ${#EGO_VENDOR[@]} -gt 0 ]] && GOFLAGS+=" -mod=vendor" # Do not complain about CFLAGS etc since go projects do not use them. QA_FLAGS_IGNORED='.*' @@ -77,6 +124,42 @@ RESTRICT="strip" EXPORT_FUNCTIONS src_unpack pkg_postinst +# @ECLASS-VARIABLE: EGO_SUM +# @DESCRIPTION: +# This is an array based on the go.sum content from inside the target package. +# Each array entry must be quoted and contain: +# +# "<module> <version> [h1:<hash>] [<key>:<value>]" +# +# To ease conversion from EGO_VENDOR, the presence of the h1:hash is NOT +# enforced in ebuilds at this time. +# +# The format is described upstream here: +# https://tip.golang.org/cmd/go/#hdr-Module_authentication_using_go_sum +# +# h1:<hash> is the Hash1 structure used by upstream Go +# Note that Hash1 is MORE stable than Gentoo distfile hashing, and upstream +# warns that it's conceptually possible for the Hash1 value to remain stable +# while the upstream zipfiles change. E.g. it does NOT capture mtime changes in +# files within a zipfile. +# +# <key>:<value> is an additional set of key-value tuples, to amend Gentoo +# metadata. +# +# The extra metadata keys accepted at this time are: +# - license: for dependencies built into the final runtime, the value field is +# a comma seperated list of Gentoo licenses to apply to the LICENSE variable, +# +# @EXAMPLE: +# # github.com/BurntSushi/toml is a build-time only dep +# # github.com/aybabtme/rgbterm is a runtime dep, annotated with licenses +# EGO_SUM=( +# "github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=" +# "github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=" +# "github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59/go.mod h1:q/89r3U2H7sSsE2t6Kca0lfwTK8JdoNGS/yzM/4iH5I= license:BSD-2,MIT" +# "github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59 h1:WWB576BN5zNSZc/M9d/10pqEx5VHNhaQ/yOVAkmj5Yo=" +# ) + # @ECLASS-VARIABLE: EGO_VENDOR # @DESCRIPTION: # This variable contains a list of vendored packages. @@ -95,10 +178,10 @@ go-module_vendor_uris() { local hash import line repo x for line in "${EGO_VENDOR[@]}"; do read -r import hash repo x <<< "${line}" - if [[ -n $x ]]; then + if [[ -n ${x} ]]; then eerror "Trailing information in EGO_VENDOR in ${P}.ebuild" eerror "${line}" - eerror "Trailing information is: \"$x\"" + eerror "Trailing information is: \"${x}\"" die "Invalid EGO_VENDOR format" fi : "${repo:=${import}}" @@ -106,18 +189,242 @@ go-module_vendor_uris() { done } +# @ECLASS-VARIABLE: _GOMODULE_GOPROXY_BASEURI +# @DESCRIPTION: +# Golang module proxy service to fetch module files from. Note that the module +# proxy generally verifies modules via the Hash1 code. +# +# Users in China may find some mirrors in the default list blocked, and should +# explicitly set an entry in /etc/portage/mirrors for goproxy to +# https://goproxy.cn/ or another mirror that is not blocked in China. +# See https://arslan.io/2019/08/02/why-you-should-use-a-go-module-proxy/ for further details +# +# This variable is NOT intended for user-level configuration of mirrors, but +# rather to cover Golang modules that might exist only on specific Goproxy +# servers for non-technical reasons. +# +# This variable should NOT be present in user-level configuration e.g. +# /etc/portage/make.conf, as it will violate metadata immutability! +: "${_GOMODULE_GOPROXY_BASEURI:=mirror://goproxy/}" + + +# @ECLASS-VARIABLE: _GOMODULE_GOSUM_REVERSE_MAP +# @DESCRIPTION: +# Mapping back from Gentoo distfile name to upstream distfile path. +# Associative array to avoid O(N*M) performance when populating the GOPROXY +# directory structure. +declare -A -g _GOMODULE_GOSUM_REVERSE_MAP + +# @FUNCTION: go-module_set_globals +# @DESCRIPTION: +# Convert the information in EGO_SUM for other usage in the ebuild. +# - Populates EGO_SUM_SRC_URI that can be added to SRC_URI +# - Exports _GOMODULE_GOSUM_REVERSE_MAP which provides reverse mapping from +# distfile back to the relative part of SRC_URI, as needed for +# GOPROXY=file:///... +go-module_set_globals() { + local line exts + # for tracking go.sum errors + local error_in_gosum=0 + local -a gosum_errorlines + # used make SRC_URI easier to read + local newline=$'\n' + # capture unique licenses + local -A unique_licenses + + # Now parse EGO_SUM + for line in "${EGO_SUM[@]}"; do + local module version modfile version_modfile kvs x + local hash1='' license='' + read -r module version_modfile kvs <<< "${line}" + # Parse the key:value set on the end of the line + for _kv in ${kvs}; do + local _v=${_kv#*:} + local _k=${_kv%$_v} + case "$_k" in + 'h1:') hash1+=" $_kv" ;; + 'license:') license+=" ${_v//,/ }" ;; + *) error_in_gosum=1 ; gosum_errorlines+=( "Unknown EGO_SUM key:value: k=$_k v=$_v" ) ;; + esac + done + + # Split 'v0.3.0/go.mod' into 'v0.3.0' and '/go.mod' + # It might NOT have the trailing /go.mod + IFS=/ read -r version modfile x <<<"${version_modfile}" + # Reject multiple slashes + if [[ -n ${x} ]]; then + error_in_gosum=1 + gosum_errorlines+=( "Bad version: ${version_modfile}" ) + continue + fi + + # The modfile variable should be either empty or '/go.mod' + # There is a chance that upstream Go might add something else here in + # future, and we should be prepared to capture it. + # The .info files do not need to be downloaded, they will be created + # based on the .mod file. + # See https://github.com/golang/go/issues/35922#issuecomment-584824275 + exts=() + local errormsg='' + case "${modfile}" in + '') exts=( zip ) ;; + 'go.mod'|'/go.mod') exts=( mod ) ;; + *) errormsg="Unknown modfile: line='${line}', modfile='${modfile}'" ;; + esac + + # If it was a bad entry, restart the loop + if [[ -n ${errormsg} ]]; then + error_in_gosum=1 + gosum_errorlines+=( "${errormsg} line='${line}', modfile='${modfile}'" ) + continue + fi + + # Capture unique licenses + for _l in ${license} ; do + unique_licenses["${_l}"]=1 + done + + _dir=$(_go-module_gomod_encode "${module}") + + for _ext in "${exts[@]}" ; do + # Relative URI within a GOPROXY for a file + _reluri="${_dir}/@v/${version}.${_ext}" + # SRC_URI: LHS entry + _uri="${_GOMODULE_GOPROXY_BASEURI}/${_reluri}" + # SRC_URI: RHS entry, encode any slash in the path as %2F in the filename + _distfile="${_reluri//\//%2F}" + + EGO_SUM_SRC_URI+=" ${_uri} -> ${_distfile}${newline}" + _GOMODULE_GOSUM_REVERSE_MAP["${_distfile}"]="${_reluri}" + done + done + + # Dedupe licenses + for license in "${!unique_licenses[@]}" ; do + EGO_SUM_LICENSES+=" ${license}" + done + + + if [[ ${error_in_gosum} != 0 ]]; then + eerror "Trailing information in EGO_SUM in ${P}.ebuild" + for line in "${gosum_errorlines[@]}" ; do + eerror "${line}" + done + die "Invalid EGO_SUM format" + fi + + # Ensure these variables not not changed past this point + readonly EGO_SUM + readonly EGO_SUM_SRC_URI + readonly EGO_SUM_LICENSES + readonly _GOMODULE_GOSUM_REVERSE_MAP + + # Set the guard that we are safe + _GO_MODULE_SET_GLOBALS_CALLED=1 +} + + # @FUNCTION: go-module_src_unpack # @DESCRIPTION: +# Extract & configure Go modules for consumpations. +# - Modules listed in EGO_SUM are configured as a local GOPROXY via symlinks (fast!) +# - Modules listed in EGO_VENDOR are extracted to "${S}/vendor" (slow) +# +# This function does NOT unpack the base distfile of a Go-based package. +# While the entries in EGO_SUM will be listed in ${A}, they should NOT be +# unpacked, Go will directly consume the files, including zips. +go-module_src_unpack() { + if [[ "${#EGO_VENDOR[@]}" -gt 0 ]]; then + _go-module_src_unpack_vendor + elif [[ "${#EGO_SUM[@]}" -gt 0 ]]; then + _go-module_src_unpack_gosum + else + die "Neither EGO_SUM nor EGO_VENDOR are set!" + fi +} + +# @FUNCTION: _go-module_src_unpack_gosum +# @DESCRIPTION: +# Populate a GOPROXY directory hierarchy with distfiles from EGO_SUM +# +# Exports GOPROXY environment variable so that Go calls will source the +# directory correctly. +_go-module_src_unpack_gosum() { + # shellcheck disable=SC2120 + debug-print-function "${FUNCNAME}" "$@" + + if [[ ! ${_GO_MODULE_SET_GLOBALS_CALLED} ]]; then + die "go-module_set_globals must be called in global scope" + fi + + local goproxy_dir="${T}/goproxy" + mkdir -p "${goproxy_dir}" || die + + # For each Golang module distfile, look up where it's supposed to go, and + # symlink into place. + local _A + local goproxy_mod_dir + for _A in ${A}; do + goproxy_mod_path="${_GOMODULE_GOSUM_REVERSE_MAP["${_A}"]}" + if [[ -n "${goproxy_mod_path}" ]]; then + debug-print-function "Populating goproxy for ${goproxy_mod_path}" + # Build symlink hierarchy + goproxy_mod_dir=$( dirname "${goproxy_dir}"/"${goproxy_mod_path}" ) + mkdir -p "${goproxy_mod_dir}" || die + ln -sf "${DISTDIR}"/"${_A}" "${goproxy_dir}/${goproxy_mod_path}" || die "Failed to ln" + local v=${goproxy_mod_path} + v="${v%.mod}" + v="${v%.zip}" + v="${v//*\/}" + _go-module_gosum_synthesize_files "${goproxy_mod_dir}" "${v}" + fi + done + export GOPROXY="file://${goproxy_dir}" + + # Validate the gosum now + _go-module_src_unpack_verify_gosum +} + +# @FUNCTION: _go-module_gosum_synthesize_files +# @DESCRIPTION: +# Given a path & version, populate all Goproxy metadata files which aren't +# needed to be downloaded directly. +# - .../@v/${version}.info +# - .../@v/list +_go-module_gosum_synthesize_files() { + local target=$1 + local version=$2 + # 'go get' doesn't care about the hash of the .info files, they + # just need a 'version' element! + # This saves a download of a tiny file + # The .time key is omitted, because that is the time a module was added to the + # upstream goproxy, and not metadata about the module itself. + cat >"${target}/${version}.info" <<-EOF + { + "Version": "${version}", + "shortName": "${version}", + "Name": "${version}" + } + EOF + listfile="${target}"/list + if ! grep -sq -x -e "${version}" "${listfile}" 2>/dev/null; then + echo "${version}" >>"${listfile}" + fi +} + +# @FUNCTION: _go-module_src_unpack_vendor +# @DESCRIPTION: # Extract all archives in ${a} which are not nentioned in ${EGO_VENDOR} # to their usual locations then extract all archives mentioned in # ${EGO_VENDOR} to ${S}/vendor. -go-module_src_unpack() { - debug-print-function ${FUNCNAME} "$@" +_go-module_src_unpack_vendor() { + # shellcheck disable=SC2120 + debug-print-function "${FUNCNAME}" "$@" local f hash import line repo tarball vendor_tarballs x vendor_tarballs=() for line in "${EGO_VENDOR[@]}"; do read -r import hash repo x <<< "${line}" - if [[ -n $x ]]; then + if [[ -n ${x} ]]; then eerror "Trailing information in EGO_VENDOR in ${P}.ebuild" eerror "${line}" die "Invalid EGO_VENDOR format" @@ -125,10 +432,10 @@ go-module_src_unpack() { : "${repo:=${import}}" vendor_tarballs+=("${repo//\//-}-${hash}.tar.gz") done - for f in $A; do - [[ -n ${vendor_tarballs[*]} ]] && has "$f" "${vendor_tarballs[@]}" && + for f in ${A}; do + [[ -n ${vendor_tarballs[*]} ]] && has "${f}" "${vendor_tarballs[@]}" && continue - unpack "$f" + unpack "${f}" done [[ -z ${vendor_tarballs[*]} ]] && return @@ -145,13 +452,53 @@ go-module_src_unpack() { done } +# @FUNCTION: _go-module_src_unpack_verify_gosum +# @DESCRIPTION: +# Validate the Go modules declared by EGO_SUM are sufficent to cover building +# the package, without actually building it yet. +# +# This might be a good candidate for src_prepare in later versions of the +# eclass, but all consumers will need updating! +_go-module_src_unpack_verify_gosum() { + # shellcheck disable=SC2120 + debug-print-function "${FUNCNAME}" "$@" + + if [[ ! ${_GO_MODULE_SET_GLOBALS_CALLED} ]]; then + die "go-module_set_globals must be called in global scope" + fi + + cd "${S}" + + # Cleanup the modules before starting anything else + # This will print 'downloading' messages, but it's accessing content from + # the $GOPROXY file:/// URL! + einfo "Tidying go.mod/go.sum" + go mod tidy >/dev/null + + # Verify that all needed modules are really present, by fetching everything + # in the package's main go.mod. If the EGO_SUM was missing an entry then + # 'go mod tidy' && 'go get' will flag it. + # -v = verbose + # -d = download only, don't install + # -mod readonly = treat modules as readonly source + einfo "Verifying linked Golang modules" + GO111MODULE=on \ + go get \ + -v \ + -d \ + -mod readonly \ + all \ + || die "Some module is missing, update EGO_SUM" +} + # @FUNCTION: go-module_live_vendor # @DESCRIPTION: # This function is used in live ebuilds to vendor the dependencies when # upstream doesn't vendor them. go-module_live_vendor() { - debug-print-function ${FUNCNAME} "$@" + debug-print-function "${FUNCNAME}" "$@" + # shellcheck disable=SC2086 has live ${PROPERTIES} || die "${FUNCNAME} only allowed in live ebuilds" [[ "${EBUILD_PHASE}" == unpack ]] || @@ -168,7 +515,7 @@ go-module_live_vendor() { # @DESCRIPTION: # Display a warning about security updates for Go programs. go-module_pkg_postinst() { - debug-print-function ${FUNCNAME} "$@" + debug-print-function "${FUNCNAME}" "$@" [[ -n ${REPLACING_VERSIONS} ]] && return 0 ewarn "${PN} is written in the Go programming language." ewarn "Since this language is statically linked, security" @@ -179,4 +526,30 @@ go-module_pkg_postinst() { ewarn "stable tree." } +# @FUNCTION: _go-module_gomod_encode +# @DESCRIPTION: +# Encode the name(path) of a Golang module in the format expected by Goproxy. +# +# Upper letters are replaced by their lowercase version with a '!' prefix. +# +_go-module_gomod_encode() { + ## Python: + # return re.sub('([A-Z]{1})', r'!\1', s).lower() + + ## Sed: + ## This uses GNU Sed extension \l to downcase the match + #echo "${module}" |sed 's,[A-Z],!\l&,g' + # + # Bash variant: + debug-print-function "${FUNCNAME}" "$@" + #local re input lower + re='(.*)([A-Z])(.*)' + input="${1}" + while [[ ${input} =~ ${re} ]]; do + lower='!'"${BASH_REMATCH[2],}" + input="${BASH_REMATCH[1]}${lower}${BASH_REMATCH[3]}" + done + echo "${input}" +} + fi diff --git profiles/thirdpartymirrors profiles/thirdpartymirrors index ad4c4b972146..d60f166e07c9 100644 --- profiles/thirdpartymirrors +++ profiles/thirdpartymirrors @@ -25,3 +25,4 @@ sourceforge https://download.sourceforge.net sourceforge.jp http://iij.dl.sourceforge.jp https://osdn.dl.sourceforge.jp https://jaist.dl.sourceforge.jp ubuntu http://mirror.internode.on.net/pub/ubuntu/ubuntu/ https://mirror.tcc.wa.edu.au/ubuntu/ http://ubuntu.uni-klu.ac.at/ubuntu/ http://mirror.dhakacom.com/ubuntu-archive/ http://ubuntu.c3sl.ufpr.br/ubuntu/ http://ubuntu.uni-sofia.bg/ubuntu/ http://hr.archive.ubuntu.com/ubuntu/ http://cz.archive.ubuntu.com/ubuntu/ https://mirror.dkm.cz/ubuntu http://ftp.cvut.cz/ubuntu/ http://ftp.stw-bonn.de/ubuntu/ https://ftp-stud.hs-esslingen.de/ubuntu/ https://mirror.netcologne.de/ubuntu/ https://mirror.unej.ac.id/ubuntu/ http://kr.archive.ubuntu.com/ubuntu/ https://mirror.nforce.com/pub/linux/ubuntu/ http://mirror.amsiohosting.net/archive.ubuntu.com/ http://nl3.archive.ubuntu.com/ubuntu/ https://mirror.timeweb.ru/ubuntu/ http://ubuntu.mirror.su.se/ubuntu/ https://ftp.yzu.edu.tw/ubuntu/ https://mirror.aptus.co.tz/pub/ubuntuarchive/ https://ubuntu.volia.net/ubuntu-archive/ https://mirror.sax.uk.as61049.net/ubuntu/ https://mirror.pnl.gov/ubuntu/ http://mirror.cc.columbia.edu/pub/linux/ubuntu/archive/ https://mirrors.namecheap.com/ubuntu/ vdr-developerorg http://projects.vdr-developer.org/attachments/download +goproxy https://proxy.golang.org/ https://goproxy.io/ https://gocenter.io/ -- 2.25.0
