Re: [gentoo-releng] Re: No New Image 12th February 2023

[Ben Kohler]

On 2/19/23 11:43, Luna Jernberg wrote:

Failed images today too

On 2/12/23, Luna Jernberg  wrote:

Broken builds for amd64 today too :(

On 1/16/23, Luna Jernberg  wrote:

Hey!

Did the ISO builds not go as planned last week?

Please join us in IRC (Libera network) in #gentoo-releng to help us 
worth through these issues, I believe we are close to back on track but 
feedback & testing are very helpful.


-Ben

[gentoo-releng] [PATCH] catalyst-auto-{amd64,x86}*.conf: change cmd array to upload function

[Ben Kohler]

Use an easier-to-read "upload" function in place of "${cmd[@]}" array
syntax.  Added a helper var UPLOAD_DEST to be set in advance so that
upload function takes only a list of filenames to be uploaded.

Signed-off-by: Ben Kohler 
---
 tools/catalyst-auto-amd64-experimental.conf | 48 +++--
 tools/catalyst-auto-amd64.conf  | 48 +++--
 tools/catalyst-auto-x86-experimental.conf   | 29 +++--
 tools/catalyst-auto-x86.conf| 29 +++--
 4 files changed, 86 insertions(+), 68 deletions(-)

diff --git a/tools/catalyst-auto-amd64-experimental.conf 
b/tools/catalyst-auto-amd64-experimental.conf
index 2aa91a03..f09cdb3b 100644
--- a/tools/catalyst-auto-amd64-experimental.conf
+++ b/tools/catalyst-auto-amd64-experimental.conf
@@ -26,7 +26,6 @@ 
SET_minimal_nomultilib_OPTIONAL_SPECS="stage4-nomultilib-minimal.spec"
 
 SET_x32_SPECS="stage1-x32.spec stage2-x32.spec stage3-x32.spec"
 
-
 SET_hardened_multilib_SPECS="hardened/stage1.spec hardened/stage2.spec 
hardened/stage3.spec"
 SET_hardened_multilib_OPTIONAL_SPECS="hardened/admincd-stage1.spec 
hardened/admincd-stage2.spec"
 SET_minimal_hardened_multilib_OPTIONAL_SPECS="hardened/stage4-minimal.spec"
@@ -62,16 +61,19 @@ update_symlinks() {
done
 }
 
+upload() {
+   rsync \
+   -e 'ssh -i /root/.ssh/id_rsa -o UserKnownHostsFile=/dev/null -o 
VerifyHostKeyDNS=yes -o StrictHostKeyChecking=no' \
+   -a \
+   --omit-dir-times \
+   --delay-updates \
+   "$@" \
+   "${UPLOAD_DEST}"
+}
+
 post_build() {
local set=$1 spec=$2
 
-   cmd=(
-   rsync
-   -e 'ssh -i /root/.ssh/id_rsa -o UserKnownHostsFile=/dev/null -o 
VerifyHostKeyDNS=yes -o StrictHostKeyChecking=no'
-   -a
-   --omit-dir-times
-   --delay-updates
-   )
case $HOSTNAME in
# If you ever need to ensure it's copied somewhere local, drop 
a hostname in here!
LOCAL)
@@ -85,53 +87,55 @@ post_build() {
esac
 
pushd "${BUILD_SRCDIR_BASE}"/default >/dev/null || exit
+   UPLOAD_DEST=${DEST_DEFAULT}
case ${spec} in
stage3.spec)
-   "${cmd[@]}" stage3-amd64-${TIMESTAMP}.tar.xz* "${DEST_DEFAULT}"
+   upload stage3-amd64-${TIMESTAMP}.tar.xz*
;;
installcd-stage2-minimal.spec)
-   "${cmd[@]}" install-amd64-minimal-${TIMESTAMP}.iso* 
"${DEST_DEFAULT}"
+   upload install-amd64-minimal-${TIMESTAMP}.iso*
;;
stage4-minimal.spec)
-   "${cmd[@]}" stage4-amd64-minimal-${TIMESTAMP}.tar.xz* 
"${DEST_DEFAULT}"
+   upload stage4-amd64-minimal-${TIMESTAMP}.tar.xz*
;;
stage3-nomultilib.spec)
-   "${cmd[@]}" stage3-amd64-nomultilib-${TIMESTAMP}.tar.xz* 
"${DEST_DEFAULT}"
+   upload stage3-amd64-nomultilib-${TIMESTAMP}.tar.xz*
;;
stage4-nomultilib-minimal.spec)
-   "${cmd[@]}" 
stage4-amd64-minimal-nomultilib-${TIMESTAMP}.tar.xz* "${DEST_DEFAULT}"
+   upload stage4-amd64-minimal-nomultilib-${TIMESTAMP}.tar.xz*
;;
stage3-x32.spec)
-   "${cmd[@]}" stage3-x32-${TIMESTAMP}.tar.xz* "${DEST_DEFAULT}"
+   upload stage3-x32-${TIMESTAMP}.tar.xz*
;;
esac
popd >/dev/null || exit
 
pushd "${BUILD_SRCDIR_BASE}"/hardened >/dev/null || exit
+   UPLOAD_DEST=${DEST_HARDENED}
case ${spec} in
hardened/stage3.spec)
-   "${cmd[@]}" stage3-amd64-hardened-${TIMESTAMP}.tar.xz* 
"${DEST_HARDENED}"
+   upload stage3-amd64-hardened-${TIMESTAMP}.tar.xz*
;;
hardened/admincd-stage2.spec)
-   "${cmd[@]}" admincd-amd64-${TIMESTAMP}.iso* "${DEST_HARDENED}"
+   upload admincd-amd64-${TIMESTAMP}.iso*
;;
hardened/stage4-minimal.spec)
-   "${cmd[@]}" stage4-amd64-hardened+minimal-${TIMESTAMP}.tar.xz* 
"${DEST_HARDENED}"
+   upload stage4-amd64-hardened+minimal-${TIMESTAMP}.tar.xz*
;;
hardened/stage3-nomultilib.spec)
-   "${cmd[@]}" 
stage3-amd64-hardened+nomultilib-${TIMESTAMP}.tar.xz* "${DEST_HARDENED}"
+   upload stage3-amd64-hardened+nomultilib-${TIMESTAMP}.tar.xz*
;;
hardened/stage4-nomultilib-minimal.spec)
-   "${cmd[@]}" 
stage4-amd64-hardened+minimal-nomultilib-${TIMESTAMP}.tar.xz* "${DEST_HARDENED}"
+   upload 
stage4-amd64-hardened+minimal-nomultilib

[gentoo-releng] [PATCH 2/2] catalyst-auto-x86*.conf: explicitly list files to be transferred

[Ben Kohler]

Previously when we were copying locally there was no (major) problem
with wildcarding a long list of files to be transferred after every
single spec finishes.  Now that we are transferring to a remote host,
this results in a lot of wasted transfer time & bandwidth.

I have adjusted the post_build function to only transfer the new files
known to be produced by each spec (which has files meant to be
published).

Signed-off-by: Ben Kohler 
---
 tools/catalyst-auto-x86-experimental.conf | 71 +--
 tools/catalyst-auto-x86.conf  | 57 +-
 2 files changed, 67 insertions(+), 61 deletions(-)

diff --git a/tools/catalyst-auto-x86-experimental.conf 
b/tools/catalyst-auto-x86-experimental.conf
index 3fd60287..061cab92 100644
--- a/tools/catalyst-auto-x86-experimental.conf
+++ b/tools/catalyst-auto-x86-experimental.conf
@@ -30,17 +30,27 @@ EXTENSIONS="[.tar.xz,.tar.bz2,.tar.gz,.tar,.sfs]"
 update_symlinks() {
# Symlink the latest stages3 to build from
for d in ${BUILD_SRCDIR_BASE}/{default,hardened} ; do
-   pushd $d >/dev/null
+   pushd "${d}" >/dev/null || exit
for f in $(ls stage3*${EXTENSIONS} | grep -v latest | 
give_latest_from_dates ) ; do
-   of=${f/20[0-9][0-9].[0-9]/latest} # for 20XX.Y stuff
-   of=${of/20[0-9][0-9][0-1][0-9][0-9][0-9]/latest} # for 
20YYMMDD stuff
-   ln -sf $f $of
+   # 20yymmddThhmmssZ
+   # 20yymmddhhmmss
+   # 20yymmdd
+   # 20yy.n
+   of=$(perl -p \
+   -e 's/20\d{6}T\d{6}Z/latest/g;' \
+   -e 's/20\d{6}\d{6}/latest/g;' \
+   -e 's/20\d{6}/latest/g;' \
+   -e 's/20\d{2}\.\d{2}/latest/g;' \
+   <<<"$f")
+   ln -svf "$f" "$of"
done
-   popd >/dev/null
+   popd >/dev/null || exit
done
 }
 
 post_build() {
+   local set=$1 spec=$2
+
cmd=(
rsync
-e 'ssh -i /root/.ssh/id_rsa -o UserKnownHostsFile=/dev/null -o 
VerifyHostKeyDNS=yes -o StrictHostKeyChecking=no'
@@ -59,34 +69,31 @@ post_build() {

DEST_HARDENED=${ARCH}@releng-incoming.gentoo.org:${BUILD_DESTDIR_BASE}/hardened
;;
esac
-   pushd ${BUILD_SRCDIR_BASE}/default >/dev/null
-   mkdir -p ${TMPDIR}/empty
-   "${cmd[@]}" ${TMPDIR}/empty ${DEST_DEFAULT}
-   for file in $(ls stage{3,4}*{${DATESTAMP},${TIMESTAMP}}*${EXTENSIONS} 
); do
-   if [ -f $file ]; then
-   "${cmd[@]}" ${file}* ${DEST_DEFAULT}
-   fi
-   done
-   if [ -f *${DATESTAMP}*.iso ]; then
-   "${cmd[@]}" *${DATESTAMP}*.iso* ${DEST_DEFAULT}
-   elif [ -f *${TIMESTAMP}*.iso ]; then
-   "${cmd[@]}" *${TIMESTAMP}*.iso* ${DEST_DEFAULT}
-   fi
-   popd >/dev/null
 
-   pushd ${BUILD_SRCDIR_BASE}/hardened >/dev/null
-   "${cmd[@]}" ${TMPDIR}/empty ${DEST_HARDENED}
-   for file in $(ls stage{3,4}*{${DATESTAMP},${TIMESTAMP}}*${EXTENSIONS} 
); do
-   if [ -f $file ]; then
-   "${cmd[@]}" $file* ${DEST_HARDENED}
-   fi
-   done
-   if [ -f *${DATESTAMP}*.iso ]; then
-   "${cmd[@]}" *${DATESTAMP}*.iso* ${DEST_HARDENED}
-   elif [ -f *${TIMESTAMP}*.iso ]; then
-   "${cmd[@]}" *${TIMESTAMP}*.iso* ${DEST_HARDENED}
-   fi
-   popd >/dev/null
+   pushd "${BUILD_SRCDIR_BASE}"/default >/dev/null || exit
+   case ${spec} in
+   stage3.spec)
+   "${cmd[@]}" stage3-i486-${TIMESTAMP}.tar.xz* "${DEST_DEFAULT}"
+   ;;
+   installcd-stage2-minimal.spec)
+   "${cmd[@]}" install-x86-minimal-${TIMESTAMP}.iso* 
"${DEST_DEFAULT}"
+   ;;
+   i686/stage3.spec)
+   "${cmd[@]}" stage3-i686-${TIMESTAMP}.tar.xz* "${DEST_DEFAULT}"
+   ;;
+   esac
+   popd >/dev/null || exit
+
+   pushd "${BUILD_SRCDIR_BASE}"/hardened >/dev/null || exit
+   case ${spec} in
+   hardened/stage3.spec)
+   "${cmd[@]}" stage3-i686-hardened-${TIMESTAMP}.tar.xz* 
"${DEST_HARDENED}"
+   ;;
+   hardened/admincd-stage2.spec)
+   "${cmd[@]}" admincd-x86-${TIMESTAMP}.iso* "${DEST_HARDENED}"
+   ;;
+   esac
+   popd >/dev/null || exit
 }
 
 # vim:ft=sh:
diff --git a/tools/catalyst-auto-x86.conf b/tools/catalyst-aut

[gentoo-releng] [PATCH] specs: add usbutils (& pciutils explicitly) to amd64/x86 installcds

[Ben Kohler]

We have needed usbutils on our installcds for a long time.  This is very 
useful to get usb networking & other hardware up early in the install 
procedure.


I have added that and also pciutils, which we currently get via a 
dependency of another package.  New installers use pciutils very often 
so we wouldn't want to accidentally lose this package.


I have only staged the change for the main x86/amd64 installcds but it 
could be applied to admincds as well.


-Ben
>From afe9c1ccb5901ed09afdb3690f26445601c5c3ae Mon Sep 17 00:00:00 2001
From: Ben Kohler 
Date: Fri, 29 Jun 2018 11:00:43 -0500
Subject: [PATCH] specs: add usbutils (& pciutils explicitly) to amd64/x86
 installcds

Add usbutils, this is an important tool during installation to bring up
usb networking and other harware, before installation can start.

Also add pciutils explicitly, we get it as a dep already but this is an
important tool during install as well, we use it directly.
---
 releases/weekly/specs/amd64/installcd-stage1.spec | 2 ++
 releases/weekly/specs/x86/installcd-stage1.spec   | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/releases/weekly/specs/amd64/installcd-stage1.spec b/releases/weekly/specs/amd64/installcd-stage1.spec
index acdf93ae..59cf0d85 100644
--- a/releases/weekly/specs/amd64/installcd-stage1.spec
+++ b/releases/weekly/specs/amd64/installcd-stage1.spec
@@ -76,7 +76,9 @@ livecd/packages:
 	sys-apps/iproute2
 	sys-apps/memtester
 	sys-apps/netplug
+	sys-apps/pciutils
 	sys-apps/sdparm
+	sys-apps/usbutils
 	sys-block/parted
 	sys-block/partimage
 	sys-fs/btrfs-progs
diff --git a/releases/weekly/specs/x86/installcd-stage1.spec b/releases/weekly/specs/x86/installcd-stage1.spec
index 14e27af7..b5a4807e 100644
--- a/releases/weekly/specs/x86/installcd-stage1.spec
+++ b/releases/weekly/specs/x86/installcd-stage1.spec
@@ -74,7 +74,9 @@ livecd/packages:
 	sys-apps/iproute2
 	sys-apps/memtester
 	sys-apps/netplug
+	sys-apps/pciutils
 	sys-apps/sdparm
+	sys-apps/usbutils
 	sys-block/parted
 	sys-block/partimage
 	sys-fs/btrfs-progs
-- 
2.18.0

Re: [gentoo-releng] Subject: [PATCH] releases/weekly/specs/*/*cd-stage1.spec: remove dead ntlmaps pkg

[Ben Kohler]

On Mon, Apr 9, 2018 at 9:20 AM, Jorge Manuel B. S. Vicetto
<jmbsvice...@gentoo.org> wrote:
> On Mon, Apr 9, 2018 at 1:57 PM, Ben Kohler <bkoh...@gmail.com> wrote:
>> Hello,
>
> Hi.
>
>> Attached is a very simple patch for your review.
>
> Commit it.
>
>> Thanks,
>>
>> Ben
>
> Thanks,
> Jorge
>

Pushed, thanks!

-Ben

[gentoo-releng] Subject: [PATCH] releases/weekly/specs/*/*cd-stage1.spec: remove dead ntlmaps pkg

[Ben Kohler]

Hello,

Attached is a very simple patch for your review.

Thanks,

Ben
From 6286776992e1eb2ee9254f41656f6f3f8c00a524 Mon Sep 17 00:00:00 2001
From: Ben Kohler <bkoh...@gmail.com>
Date: Mon, 9 Apr 2018 08:51:10 -0500
Subject: [PATCH] releases/weekly/specs/*/*cd-stage1.spec: remove dead ntlmaps
 pkg

net-proxy/ntlmaps was removed from ::gentoo in 2016 [1], time to remove
it from installcd package list.

[1] https://gitweb.gentoo.org/repo/gentoo.git/commit/net-proxy/ntlmaps?id=031ceb4bc4a53dca5c20906459ff08b8c48f7c5e
---
 releases/weekly/specs/alpha/installcd-stage1.spec| 1 -
 releases/weekly/specs/amd64/hardened/admincd-stage1-selinux.spec | 1 -
 releases/weekly/specs/amd64/hardened/admincd-stage1.spec | 1 -
 releases/weekly/specs/amd64/installcd-stage1.spec| 1 -
 releases/weekly/specs/amd64/livecd-stage1.spec   | 1 -
 releases/weekly/specs/amd64/livedvd-stage1.spec  | 1 -
 releases/weekly/specs/hppa/installcd-stage1.spec | 1 -
 releases/weekly/specs/hppa/installcd-stage2-minimal.spec | 1 -
 releases/weekly/specs/ia64/installcd-stage1.spec | 1 -
 releases/weekly/specs/ppc/ppc64/installcd-stage1.spec| 1 -
 releases/weekly/specs/sparc/sparc64/installcd-stage1.spec| 1 -
 releases/weekly/specs/x86/hardened/admincd-stage1.spec   | 1 -
 releases/weekly/specs/x86/hardened/installcd-stage1.spec | 1 -
 releases/weekly/specs/x86/i686/livecd-stage1.spec| 1 -
 releases/weekly/specs/x86/i686/livedvd-stage1.spec   | 1 -
 releases/weekly/specs/x86/installcd-stage1.spec  | 1 -
 16 files changed, 16 deletions(-)

diff --git a/releases/weekly/specs/alpha/installcd-stage1.spec b/releases/weekly/specs/alpha/installcd-stage1.spec
index 300e7eaf..b2a2f250 100644
--- a/releases/weekly/specs/alpha/installcd-stage1.spec
+++ b/releases/weekly/specs/alpha/installcd-stage1.spec
@@ -50,7 +50,6 @@ livecd/packages:
 	net-misc/rdate
 #	net-misc/vconfig
 	net-proxy/dante
-#	net-proxy/ntlmaps
 	net-proxy/tsocks
 #	net-wireless/ipw2100-firmware
 #	net-wireless/ipw2200-firmware
diff --git a/releases/weekly/specs/amd64/hardened/admincd-stage1-selinux.spec b/releases/weekly/specs/amd64/hardened/admincd-stage1-selinux.spec
index c24c5d3e..70365c39 100644
--- a/releases/weekly/specs/amd64/hardened/admincd-stage1-selinux.spec
+++ b/releases/weekly/specs/amd64/hardened/admincd-stage1-selinux.spec
@@ -115,7 +115,6 @@ livecd/packages:
 	net-misc/wget
 	net-misc/whois
 	net-proxy/dante
-#	net-proxy/ntlmaps
 	net-proxy/tsocks
 	net-vpn/openvpn
 	net-wireless/b43-fwcutter
diff --git a/releases/weekly/specs/amd64/hardened/admincd-stage1.spec b/releases/weekly/specs/amd64/hardened/admincd-stage1.spec
index 8cc0b8aa..c07f82b7 100644
--- a/releases/weekly/specs/amd64/hardened/admincd-stage1.spec
+++ b/releases/weekly/specs/amd64/hardened/admincd-stage1.spec
@@ -118,7 +118,6 @@ livecd/packages:
 	net-misc/wget
 	net-misc/whois
 	net-proxy/dante
-#	net-proxy/ntlmaps
 	net-proxy/tsocks
 	net-vpn/openvpn
 	net-wireless/b43-fwcutter
diff --git a/releases/weekly/specs/amd64/installcd-stage1.spec b/releases/weekly/specs/amd64/installcd-stage1.spec
index f0efe78c..acdf93ae 100644
--- a/releases/weekly/specs/amd64/installcd-stage1.spec
+++ b/releases/weekly/specs/amd64/installcd-stage1.spec
@@ -58,7 +58,6 @@ livecd/packages:
 	net-misc/rsync
 	net-misc/vconfig
 	net-proxy/dante
-#	net-proxy/ntlmaps
 	net-proxy/tsocks
 	net-wireless/b43-fwcutter
 ### Masked (~keywords)
diff --git a/releases/weekly/specs/amd64/livecd-stage1.spec b/releases/weekly/specs/amd64/livecd-stage1.spec
index d3d187a6..31e773ee 100644
--- a/releases/weekly/specs/amd64/livecd-stage1.spec
+++ b/releases/weekly/specs/amd64/livecd-stage1.spec
@@ -72,7 +72,6 @@ livecd/packages:
 	net-misc/whois
 	net-p2p/bittorrent
 	net-proxy/dante
-#	net-proxy/ntlmaps
 	net-proxy/tsocks
 	net-wireless/b43-fwcutter
 ### Masked (~amd64)
diff --git a/releases/weekly/specs/amd64/livedvd-stage1.spec b/releases/weekly/specs/amd64/livedvd-stage1.spec
index f23255ab..2d5dd9b6 100644
--- a/releases/weekly/specs/amd64/livedvd-stage1.spec
+++ b/releases/weekly/specs/amd64/livedvd-stage1.spec
@@ -167,7 +167,6 @@ livecd/packages:
 	net-p2p/limewire
 	net-print/cups
 	net-proxy/dante
-#	net-proxy/ntlmaps
 	net-proxy/tsocks
 ### Masked (~amd64)
 #	net-wireless/aircrack-ng
diff --git a/releases/weekly/specs/hppa/installcd-stage1.spec b/releases/weekly/specs/hppa/installcd-stage1.spec
index bfa9b9e3..b027e997 100644
--- a/releases/weekly/specs/hppa/installcd-stage1.spec
+++ b/releases/weekly/specs/hppa/installcd-stage1.spec
@@ -47,7 +47,6 @@ livecd/packages:
 	net-misc/rdate
 	net-misc/vconfig
 	net-proxy/dante
-#	net-proxy/ntlmaps
 	sys-apps/ethtool
 	sys-apps/hdparm
 	sys-apps/hwsetup
diff --git a/releases/weekly/specs/hppa/installcd-stage2-minimal.spec b/releases/weekly/specs/hppa/installcd-stage2-minimal.spec
index ceab76cb..d0

Re: [gentoo-releng] Workaround for stage1 failures introduced with portage-2.3.19-r1

[Ben Kohler]

On Tue, Jan 30, 2018 at 12:54 PM, Jorge Manuel B. S. Vicetto
 wrote:
> I believe there was some misunderstanding about my comment.
> I meant I prefer to add to our /etc/portage/package.keywords an entry
> for a portage version with this issue fixed.
> Per Zac's comment above, I'll do that for portage2.3.21.
>
> Thanks,
> Jorge.
>

You're going to ship unstable portage in stage3 just to avoid adding a
temporary portage config that users won't even see?

That seems questionable, but I guess if it gets autobuilds back on
track, it's something.

-Ben

[gentoo-releng] Workaround for stage1 failures introduced with portage-2.3.19-r1

[Ben Kohler]

Due to bug 645002 (and possibly others), our stage1 builds are making
very odd choices on several || ( ) deps, and failing to resolve deps.

One example failure log:
https://archives.gentoo.org/gentoo-releng-autobuilds/message/19e4086ee421a504c445edafc2e83d88

This has been fixed in portage-2.3.20 but there is no indication yet
of if or when this version is going to be stabilized.  If the next
portage stabilization is more than a couple of days out, I propose
that we add a temporary mask to force portage to make the right
decisions.  Since catalyst now cleans our releng portage configs,
there would be no visible signs of this workaround in the resulting
stages.

Proposed masks:

releng/releases/weekly/portage/stages/package.mask/releng/portage-workarounds:

dev-util/pkgconf
sys-apps/paludis
sys-fs/static-dev


Thoughts?

-Ben