Re: [arch-projects] [devtools] [PATCH 0/3] New tool,

[email]

On 17.11.2019 21:32, Eli Schwartz via arch-projects wrote:

I've finally finished up this thing for testing if a package can be
rebuilt reproducibly. It uses makechrootpkg/mkarchroot under the hood,
and requires arch-install-scripts >= 23.


I have been testing this locally along with others in #archlinux-reproducible 
on Freenode and it works well for me; we have a bunch of packages now

building reproducibly.

I've also pulled in this patch on my local box and reviewed the code briefly,
LGTM.

Cheers

---
Daniel Edgecumbe | esotericnonsense
SE +46 10 195 6918 (no SMS) | UK +44 7723 432110
em...@esotericnonsense.com | https://esotericnonsense.com

[arch-projects] [devtools] [PATCH 2/3] makerepropkg: add new program to try to reproducibly build a package

[Eli Schwartz via arch-projects]

This attempts to recreate a package that was probably created using
makechrootpkg, and see if it conforms to the
https://reproducible-builds.org/ specification.

Signed-off-by: Eli Schwartz 
---
 Makefile|   1 +
 makerepropkg.in | 186 
 2 files changed, 187 insertions(+)
 create mode 100755 makerepropkg.in

diff --git a/Makefile b/Makefile
index 0eb7a88..090063d 100644
--- a/Makefile
+++ b/Makefile
@@ -14,6 +14,7 @@ IN_PROGS = \
finddeps \
find-libdeps \
lddd \
+   makerepropkg \
mkarchroot \
makechrootpkg \
rebuildpkgs \
diff --git a/makerepropkg.in b/makerepropkg.in
new file mode 100755
index 000..d1414d8
--- /dev/null
+++ b/makerepropkg.in
@@ -0,0 +1,186 @@
+#!/bin/bash
+#   makerepropkg - rebuild a package to see if it is reproducible
+#
+#   Copyright (c) 2019 by Eli Schwartz 
+#
+#   This program is free software; you can redistribute it and/or modify
+#   it under the terms of the GNU General Public License as published by
+#   the Free Software Foundation; either version 2 of the License, or
+#   (at your option) any later version.
+#
+#   This program is distributed in the hope that it will be useful,
+#   but WITHOUT ANY WARRANTY; without even the implied warranty of
+#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#   GNU General Public License for more details.
+#
+#   You should have received a copy of the GNU General Public License
+#   along with this program.  If not, see .
+#
+
+m4_include(lib/common.sh)
+m4_include(lib/archroot.sh)
+
+source /usr/share/makepkg/util/config.sh
+source /usr/share/makepkg/util/message.sh
+
+declare -A buildinfo
+declare -a buildenv buildopts installed installpkgs
+
+archiveurl='https://archive.archlinux.org/packages'
+buildroot=/var/lib/archbuild/reproducible
+chroot=testenv
+
+parse_buildinfo() {
+local line var val
+
+while read -r line; do
+var="${line%% = *}"
+val="${line#* = }"
+case ${var} in
+buildenv)
+buildenv+=("${val}")
+;;
+options)
+buildopts+=("${val}")
+;;
+installed)
+installed+=("${val}")
+;;
+*)
+buildinfo["${var}"]="${val}"
+;;
+esac
+done
+}
+
+get_pkgfile() {
+local cdir=${cache_dirs[0]}
+local pkgfilebase=${1}
+local pkgname=${pkgfilebase%-*-*-*}
+local pkgfile ext
+
+for ext in .xz .zstd ''; do
+pkgfile=${pkgfilebase}.pkg.tar${ext}
+
+for c in "${cache_dirs[@]}"; do
+if [[ -f ${c}/${pkgfile} ]]; then
+cdir=${c}
+break
+fi
+done
+
+for f in "${pkgfile}" "${pkgfile}.sig"; do
+if [[ ! -f "${cdir}/${f}" ]]; then
+msg2 "retrieving '%s'..." "${f}" >&2
+curl -Llf -# -o "${cdir}/${f}" 
"${archiveurl}/${pkgname:0:1}/${pkgname}/${f}" || continue 2
+fi
+done
+printf '%s\n' "file://${cdir}/${pkgfile}"
+return 0
+done
+
+return 1
+}
+
+usage() {
+cat << __EOF__
+usage: ${BASH_SOURCE[0]##*/} [options] 
+
+Run this script in a PKGBUILD dir to build a package inside a
+clean chroot while attempting to reproduce it. The package file
+will be used to derive metadata needed for reproducing the
+package, including the .PKGINFO as well as the buildinfo.
+
+For more details see https://reproducible-builds.org/
+
+OPTIONS
+-c   Set pacman cache
+-M  Location of a makepkg config file
+-hShow this usage message
+__EOF__
+}
+
+while getopts 'M:c:h' arg; do
+case "$arg" in
+M) archroot_args+=(-M "$OPTARG") ;;
+c) cache_dirs+=("$OPTARG") ;;
+h) usage; exit 0 ;;
+*|?) usage; exit 1 ;;
+esac
+done
+shift $((OPTIND - 1))
+
+check_root
+
+if [[ -n $1 ]]; then
+pkgfile="$1"
+else
+error "no package file specified. Try '${BASH_SOURCE[0]##*/} -h' for more 
information. "
+exit 1
+fi
+
+if (( ${#cache_dirs[@]} == 0 )); then
+   mapfile -t cache_dirs < <(pacman-conf CacheDir)
+fi
+
+ORIG_HOME=${HOME}
+IFS=: read -r _ _ _ _ _ HOME _ < <(getent passwd "${SUDO_USER:-$USER}")
+load_makepkg_config
+HOME=${ORIG_HOME}
+[[ -d ${SRCDEST} ]] || SRCDEST=${PWD}
+
+parse_buildinfo < <(bsdtar -xOqf "${pkgfile}" .BUILDINFO)
+export SOURCE_DATE_EPOCH="${buildinfo[builddate]}"
+PACKAGER="${buildinfo[packager]}"
+BUILDDIR="${buildinfo[builddir]}"
+
+# nuke and restore reproducible testenv
+for copy in "${buildroot}"/*/; do
+[[ -d ${copy} ]] || continue
+subvolume_delete_recursive "${copy}"
+done
+rm -rf --one-file-system "${buildroot}"
+(umask 0022; mkdir -p "${buildroot}")
+
+for fname in "${installed[@]}"; do
+if ! allpkgfiles+=("$(get_pkgfile "${fname}")"); then
+error 

[arch-projects] [devtools] [PATCH 0/3] New tool,

[Eli Schwartz via arch-projects]

I've finally finished up this thing for testing if a package can be
rebuilt reproducibly. It uses makechrootpkg/mkarchroot under the hood,
and requires arch-install-scripts >= 23.

This, and other commits recently submitted to the list, can be pulled
directly from my repository at 
https://github.com/eli-schwartz/devtools/commits/arch-projects-review

Eli Schwartz (3):
  mkarchroot: support wrapping pacstrap -U
  makerepropkg: add new program to try to reproducibly build a package
  doc: add manpage for the new makerepropkg tool

 Makefile|   2 +
 doc/makerepropkg.1.asciidoc |  38 
 doc/mkarchroot.1.asciidoc   |   3 +
 makerepropkg.in | 186 
 mkarchroot.in   |  13 ++-
 5 files changed, 238 insertions(+), 4 deletions(-)
 create mode 100644 doc/makerepropkg.1.asciidoc
 create mode 100755 makerepropkg.in

-- 
2.24.0

[arch-projects] [devtools] [PATCH 3/3] doc: add manpage for the new makerepropkg tool

[Eli Schwartz via arch-projects]

Signed-off-by: Eli Schwartz 
---
 Makefile|  1 +
 doc/makerepropkg.1.asciidoc | 38 +
 2 files changed, 39 insertions(+)
 create mode 100644 doc/makerepropkg.1.asciidoc

diff --git a/Makefile b/Makefile
index 090063d..4ca7155 100644
--- a/Makefile
+++ b/Makefile
@@ -72,6 +72,7 @@ MANS = \
doc/checkpkg.1 \
doc/offload-build.1 \
doc/sogrep.1 \
+   doc/makerepropkg.1 \
doc/mkarchroot.1 \
doc/find-libdeps.1 \
doc/find-libprovides.1
diff --git a/doc/makerepropkg.1.asciidoc b/doc/makerepropkg.1.asciidoc
new file mode 100644
index 000..7d68e5e
--- /dev/null
+++ b/doc/makerepropkg.1.asciidoc
@@ -0,0 +1,38 @@
+makerepropkg(1)
+
+
+Name
+
+makerepropkg - Rebuild a package to see if it is reproducible
+
+Synopsis
+
+makerepropkg [OPTIONS] 
+
+Description
+---
+
+Given the path to a built pacman package, attempt to rebuild it using the
+PKGBUILD in the current directory. The package will be built in an environment
+as closely matching the environment of the initial package as possible, by
+building up a chroot to match the information exposed in the package's
+linkman:BUILDINFO[5] manifest. On success, the resulting package will be
+compared to the input package, and makerepropkg will report whether the
+artifacts are identical.
+
+This implements a verifier for pacman/libalpm packages in accordance with the
+link:https://reproducible-builds.org/[Reproducible Builds] project.
+
+Options
+---
+
+*-c*::
+   Set the pacman cache directory.
+
+*-M* ::
+   Location of a makepkg config file.
+
+*-h*::
+   Show this usage message
+
+include::footer.asciidoc[]
-- 
2.24.0

[arch-projects] [devtools] [PATCH 1/3] mkarchroot: support wrapping pacstrap -U

[Eli Schwartz via arch-projects]

Needed to support reproducible builds.

Signed-off-by: Eli Schwartz 
---
 doc/mkarchroot.1.asciidoc |  3 +++
 mkarchroot.in | 13 +
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/doc/mkarchroot.1.asciidoc b/doc/mkarchroot.1.asciidoc
index 03d032c..eb0670a 100644
--- a/doc/mkarchroot.1.asciidoc
+++ b/doc/mkarchroot.1.asciidoc
@@ -20,6 +20,9 @@ en_US.UTF-8 and de_DE.UTF-8 locale and a generated machine-id.
 Options
 ---
 
+*-U*::
+   Use 'pacman -U' to install packages.
+
 *-C* ::
Location of a pacman config file.
 
diff --git a/mkarchroot.in b/mkarchroot.in
index 6c42d3b..37f6802 100644
--- a/mkarchroot.in
+++ b/mkarchroot.in
@@ -18,6 +18,7 @@ m4_include(lib/archroot.sh)
 umask 0022
 
 working_dir=''
+umode=''
 
 files=()
 nspawn_args=()
@@ -25,6 +26,7 @@ nspawn_args=()
 usage() {
echo "Usage: ${0##*/} [options] working-dir package-list..."
echo ' options:'
+   echo '-UUse pacman -U to install packages'
echo '-C  Location of a pacman config file'
echo '-M  Location of a makepkg config file'
echo '-c   Set pacman cache'
@@ -34,8 +36,9 @@ usage() {
exit 1
 }
 
-while getopts 'hC:M:c:f:s' arg; do
+while getopts 'hUC:M:c:f:s' arg; do
case "$arg" in
+   U) umode=U ;;
C) pac_conf="$OPTARG" ;;
M) makepkg_conf="$OPTARG" ;;
c) cache_dirs+=("$OPTARG") ;;
@@ -44,8 +47,10 @@ while getopts 'hC:M:c:f:s' arg; do
h|?) usage ;;
*) error "invalid argument '%s'" "$arg"; usage ;;
esac
-nspawn_args+=("-$arg")
-[[ -v OPTARG ]] && nspawn_args+=("$OPTARG")
+   if [[ $arg != U ]]; then
+   nspawn_args+=("-$arg")
+   [[ -v OPTARG ]] && nspawn_args+=("$OPTARG")
+   fi
 done
 shift $((OPTIND - 1))
 
@@ -85,7 +90,7 @@ for file in "${files[@]}"; do
cp "$file" "$working_dir$file"
 done
 
-pacstrap -Mcd ${pac_conf:+-C "$pac_conf"} "$working_dir" \
+pacstrap -${umode}Mcd ${pac_conf:+-C "$pac_conf"} "$working_dir" \
"${cache_dirs[@]/#/--cachedir=}" "$@" || die 'Failed to install all 
packages'
 
 printf '%s.UTF-8 UTF-8\n' en_US de_DE > "$working_dir/etc/locale.gen"
-- 
2.24.0