Complete diff visible in unapproved queue https://launchpadlibrarian.net/882134526/cloud- init_26.2-0ubuntu1~24.04.1_26.2-0ubuntu1~24.04.2.diff.gz
Attaching verification script. I'll exercise both this script and cloud-init full integration-tests against the new deb package to ensure we don't risk further regressions in this targetted revert. ** Description changed: [ Impact ] - * Daily launches of arm64 platform with cloud-init 26.2 on noble run + * Daily launches of arm64 platform with cloud-init 26.2 on noble run into issues attempting to contact archive.ubuntu.com due to invalid default apt mirror configuration being set. 2026-10-22T09:33:28.0139640Z E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/noble-backports/main/binary- arm64/Packages 404 Not Found [IP: 91.189.92.23 80] - [ Test Plan ] - * launch daily arm64 image and confirm incorrect archive.ubuntu.com is configured in /etc/apt/sources.list.d/*, confirm apt update errors + * launch daily arm64 image check cloud-init version and confirm whether incorrect archive.ubuntu.com is configured in /etc/apt/sources.list.d/*, confirm apt update errors ports.ubuntu.com is configured as the apt mirror instead of archive.ubuntu.com - * upgrade cloud-init to cloud-init_26.2-0ubuntu1~24.04.2 from noble- + * upgrade cloud-init to cloud-init_26.2-0ubuntu1~24.04.2 from noble- proposed - * Clean all cloud-init artifacts and reboot: - - sudo cloud-init clean --logs -c all --reboot - - * Verify ports.ubuntu.com in /etc/apt/sources.list.d/* instead of + * Clean all cloud-init artifacts and reboot: + + sudo cloud-init clean --logs -c all --reboot + + * Verify ports.ubuntu.com in /etc/apt/sources.list.d/* instead of archive.ubuntu.com. Confirm no apt update error - - test script: - TODO - + test script will look something like this: + ``` + + #!/usr/bin/env bash + # + # repro-lp-2171959.sh -- Reproduce LP #2171959 "cloud-init breaks noble arm64 + # targets" (regression from the noble backport of GH PR #6826). + # + # What this does + # 1. Boots an Ubuntu noble arm64 cloud image under qemu-system-aarch64 + # (TCG emulation; KVM is used automatically on arm64 hosts). + # 2. Snapshots the image's default /etc/apt/sources.list.d/ubuntu.sources + # before cloud-init gets a chance to rewrite it (via bootcmd). + # 3. Reports the apt sources cloud-init generated on first boot. + # 4. Optionally installs a pinned cloud-init version (default: + # 26.1-0ubuntu1~24.04.1, the affected noble package), then re-runs a + # fresh first boot with `cloud-init clean --reboot`. + # 5. Asserts: + # - URIs in /etc/apt/sources.list.d/ubuntu.sources are still + # ports.ubuntu.com (noble arm64 correct) and NOT + # archive.ubuntu.com (wrong; 404s for arm64), and + # - `apt-get update` succeeds. + # + # Exit codes + # 0 bug NOT reproduced (apt sources stayed on ports.ubuntu.com; healthy) + # 1 bug REPRODUCED (arm64 sources pointed at archive.ubuntu.com and + # apt-get update failed) + # 2 environment/setup problem + # + # Requirements (host) + # qemu-system-aarch64, qemu-img, wget or curl, ssh/scp/ssh-keygen, and one + # of: cloud-localds | genisoimage | mkisofs | xorriso (NoCloud seed ISO), + # plus UEFI firmware for aarch64 (Ubuntu/Debian: qemu-efi-aarch64 or + # edk2 packages; the qemu-provided edk2-aarch64-code.fd also works). + # + # On an x86_64 host the arm64 guest runs under TCG emulation: expect + # roughly 30-90 minutes total and a few GB of downloads/tmp files. + # + # Usage + # ./repro-lp-2171959.sh [--preflight] [--keep] + # Env overrides: + # WORKDIR=/tmp/lp-2171959 scratch dir (image, seed, evidence) + # IMAGE_URL=https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-arm64.img + # SSH_PORT=2222 host port forwarded to guest :22 + # SSH_TIMEOUT=2400 max seconds to wait for ssh per boot + # CLOUD_INIT_VERSION=26.1-0ubuntu1~24.04.1 + # version to pin before the second boot; + # "image" = keep whatever the image ships + # CLOUD_INIT_DEB_DIR=/path/to/debs + # install local .deb(s) instead of the apt + # pin (e.g. a fixed build from + # ppa:cloud-init-dev/proposed) to test the + # fix and expect exit code 0 + # SKIP_BASELINE_APT_UPDATE=1 skip the control `apt-get update` on + # boot 1 (it also runs when a pin install + # is needed) + # + + set -euo pipefail + + WORKDIR="${WORKDIR:-/tmp/lp-2171959}" + IMAGE_URL="${IMAGE_URL:-https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-arm64.img}"; + SSH_PORT="${SSH_PORT:-2222}" + SSH_TIMEOUT="${SSH_TIMEOUT:-2400}" + CLOUD_INIT_VERSION="${CLOUD_INIT_VERSION:-26.1-0ubuntu1~24.04.1}" + CLOUD_INIT_DEB_DIR="${CLOUD_INIT_DEB_DIR:-}" + SKIP_BASELINE_APT_UPDATE="${SKIP_BASELINE_APT_UPDATE:-0}" + MEM="${MEM:-2048}" + SMP="${SMP:-2}" + + KEEP=0 + PREFLIGHT=0 + while [ $# -gt 0 ]; do + case "$1" in + --keep) KEEP=1 ;; + --preflight) PREFLIGHT=1 ;; + -h|--help) sed -n '2,60p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + *) echo "unknown option: $1" >&2; exit 2 ;; + esac + shift + done + + log() { printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*" >&2; } + warn() { printf '[%s] WARN: %s\n' "$(date +%H:%M:%S)" "$*" >&2; } + die() { log "ERROR: $*"; exit 2; } + + EVIDENCE="$WORKDIR/evidence" + KEY="$WORKDIR/id_ed25519" + DISK="$WORKDIR/disk.qcow2" + SEED="$WORKDIR/seed.iso" + PIDFILE="$WORKDIR/qemu.pid" + + QEMU_PID="" + cleanup() { + set +e + if [ -n "${QEMU_PID:-}" ]; then kill "$QEMU_PID" 2>/dev/null; fi + if [ -s "$PIDFILE" ]; then kill "$(cat "$PIDFILE")" 2>/dev/null; fi + if [ "$KEEP" = 0 ] && [ -f "$DISK" ]; then rm -f "$DISK"; fi + return 0 + } + trap cleanup EXIT + + # ---------------------------------------------------------------- preflight + need() { command -v "$1" >/dev/null 2>&1 || die "missing required tool: $1 (install it on the host)"; } + need qemu-system-aarch64 + need qemu-img + need ssh-keygen + need ssh + need scp + { command -v wget >/dev/null 2>&1 || command -v curl >/dev/null 2>&1; } || die "missing wget or curl" + + SEED_TOOL="" + for t in cloud-localds genisoimage mkisofs xorriso; do + if command -v "$t" >/dev/null 2>&1; then SEED_TOOL="$t"; break; fi + done + [ -n "$SEED_TOOL" ] || die "no ISO tool for the NoCloud seed found; install cloud-image-utils or genisoimage" + + UEFI="${QEMU_UEFI:-}" + [ -z "$UEFI" ] && for f in \ + /usr/share/qemu/edk2-aarch64-code.fd \ + /usr/share/AAVMF/AAVMF_CODE.fd \ + /usr/share/qemu-efi-aarch64/QEMU_EFI.fd \ + /usr/share/edk2/aarch64/QEMU_EFI.fd ; do + [ -f "$f" ] && UEFI="$f" && break + done + [ -n "$UEFI" ] || die "no aarch64 UEFI firmware found; install qemu-efi-aarch64 (Debian/Ubuntu) or edk2-aarch64 (Fedora), or set QEMU_UEFI" + + case "$(uname -m)" in + aarch64|arm*) + if [ -e /dev/kvm ] && [ -w /dev/kvm ]; then + QEMU_ACCEL="-accel kvm"; QEMU_CPU="-cpu host" + else + QEMU_ACCEL="-accel tcg"; QEMU_CPU="-cpu cortex-a57" + fi ;; + *) + QEMU_ACCEL="-accel tcg"; QEMU_CPU="-cpu cortex-a57" + warn "host is $(uname -m): the arm64 guest runs under TCG emulation, this will be slow" ;; + esac + + ( exec 3<>"/dev/tcp/127.0.0.1/$SSH_PORT" ) 2>/dev/null && die "port $SSH_PORT is already in use (set SSH_PORT)" + true + + log "preflight OK: qemu-system-aarch64, UEFI=$UEFI, seed tool=$SEED_TOOL, accel=${QEMU_ACCEL}" + if [ "$PREFLIGHT" = 1 ]; then + log "--preflight given; environment is ready, stopping here" + exit 0 + fi + + # ------------------------------------------------------------------ setup + mkdir -p "$WORKDIR" "$EVIDENCE" + + IMAGE="$WORKDIR/$(basename "$IMAGE_URL")" + if [ ! -f "$IMAGE" ]; then + log "downloading $IMAGE_URL" + if command -v wget >/dev/null 2>&1; then + wget -c -O "$IMAGE" "$IMAGE_URL" + else + curl -fL -C - -o "$IMAGE" "$IMAGE_URL" + fi + fi + + if [ ! -f "$DISK" ]; then + log "creating qcow2 overlay of $IMAGE (root fs grown on boot by growpart)" + qemu-img create -f qcow2 -F qcow2 -b "$IMAGE" "$DISK" 10G + fi + + [ -f "$KEY" ] || ssh-keygen -q -t ed25519 -N '' -f "$KEY" >/dev/null + PUBKEY="$(cat "$KEY.pub")" + + if [ ! -f "$SEED" ]; then + log "building NoCloud seed (ds=nocloud; no apt config on purpose: we want cloud-init's defaults)" + cat > "$WORKDIR/user-data" <<EOF + #cloud-config + hostname: lp2171959 + ssh_authorized_keys: + - $PUBKEY + bootcmd: + - [ sh, -c, "test -e /var/tmp/lp2171959.pristine || cp /etc/apt/sources.list.d/ubuntu.sources /var/tmp/lp2171959.pristine || true" ] + EOF + printf 'instance-id: iid-lp2171959\nlocal-hostname: lp2171959\n' > "$WORKDIR/meta-data" + case "$SEED_TOOL" in + cloud-localds) + cloud-localds "$SEED" "$WORKDIR/user-data" "$WORKDIR/meta-data" ;; + genisoimage|mkisofs) + "$SEED_TOOL" -output "$SEED" -volid cidata -joliet -rock \ + "$WORKDIR/user-data" "$WORKDIR/meta-data" ;; + xorriso) + xorriso -as mkisofs -output "$SEED" -volid cidata -joliet -rock \ + "$WORKDIR/user-data" "$WORKDIR/meta-data" ;; + esac + fi + + # ------------------------------------------------------------------ launch + if [ -s "$PIDFILE" ] && kill -0 "$(cat "$PIDFILE")" 2>/dev/null; then + die "a previous run's qemu is still alive (pid $(cat "$PIDFILE")); kill it or remove $PIDFILE" + fi + + log "starting arm64 VM (noble, ${MEM}M, ${SMP} vCPU, ${QEMU_ACCEL}); log: $WORKDIR/console.log" + set -- qemu-system-aarch64 \ + -machine virt $QEMU_ACCEL $QEMU_CPU \ + -smp "$SMP" -m "$MEM" \ + -bios "$UEFI" \ + -drive "if=none,id=hd0,file=$DISK,format=qcow2" \ + -device virtio-blk-device,drive=hd0 \ + -drive "if=none,id=cid,file=$SEED,format=raw" \ + -device virtio-blk-device,drive=cid \ + -netdev "user,id=net0,hostfwd=tcp:127.0.0.1:${SSH_PORT}-:22" \ + -device virtio-net-device,netdev=net0 \ + -device virtio-rng-pci \ + -display none -monitor none \ + -serial "file:$WORKDIR/console.log" \ + -pidfile "$PIDFILE" -daemonize + printf '%s\n' "$*" > "$WORKDIR/qemu.cmd" + "$@" || { cat "$WORKDIR/console.log" 2>/dev/null; die "qemu failed to start"; } + QEMU_PID="$(cat "$PIDFILE")" + + SSHOPTS=(-i "$KEY" -p "$SSH_PORT" -o StrictHostKeyChecking=no -o + UserKnownHostsFile=/dev/null -o ConnectTimeout=10 -o + ConnectionAttempts=1 -o LogLevel=ERROR) + + wait_ssh() { # $1 = label for logs + local deadline=$(( $(date +%s) + SSH_TIMEOUT )) + log "waiting for ssh (up to $SSH_TIMEOUT s) [$1]" + while :; do + if ssh "${SSHOPTS[@]}" [email protected] true 2>/dev/null; then + log "ssh is up [$1]" + return 0 + fi + kill -0 "$QEMU_PID" 2>/dev/null || die "qemu exited during boot [$1]; see $WORKDIR/console.log" + [ "$(date +%s)" -ge "$deadline" ] && die "timed out waiting for ssh [$1]" + sleep 5 + done + } + + guest() { ssh "${SSHOPTS[@]}" [email protected] "$@"; } + + collect() { # $1 = evidence filename, rest = guest command + local name="$1"; shift + guest "$@" | tee "$EVIDENCE/$name" + } + + wait_cloud_init() { + collect "cloud-init-status-$1.txt" \ + 'sudo cloud-init status --wait --format json 2>&1 || cloud-init status --wait 2>&1 || true' + } + + # ------------------------------------------------------- boot 1: baseline + wait_ssh "boot1" + wait_cloud_init boot1 + + log "collecting first-boot evidence" + collect guest-info.txt 'dpkg --print-architecture; grep -E "^(VERSION_CODENAME|VERSION_ID)=" /etc/os-release; cloud-init --version 2>&1 || dpkg-query -W cloud-init' + collect sources-image-default.txt 'cat /var/tmp/lp2171959.pristine 2>/dev/null || echo "pristine snapshot missing"' + collect sources-boot1.txt "grep -E '^(Types|URIs|Suites|Components|Signed-By):' /etc/apt/sources.list.d/ubuntu.sources || true" + + apt_update() { # $1 = boot label; runs `apt-get update` in the guest and records rc + local logp="/var/tmp/lp2171959-apt-update-$1.log" + collect "apt-update-$1.log" \ + "sudo mkdir -p /var/tmp/lp2171959; sudo sh -c 'apt-get update > $logp 2>&1'; rc=\$?; cat $logp; echo apt-get-update-rc=\$rc" + } + + bad_uris() { # $1 = evidence file; exits 0 if arm64 sources point at archive.ubuntu.com + grep -E '^URIs:.*archive\.ubuntu\.com' "$EVIDENCE/$1" >/dev/null 2>&1 + } + + if [ "$SKIP_BASELINE_APT_UPDATE" != 1 ]; then + log "control check: apt-get update with the image's current apt sources" + apt_update boot1 + fi + + BAD_BOOT1=0 + if bad_uris sources-boot1.txt; then + BAD_BOOT1=1 + log "image's cloud-init already rewrote arm64 sources to archive.ubuntu.com on first boot" + fi + + if [ "$BAD_BOOT1" = 1 ]; then + if [ ! -f "$EVIDENCE/apt-update-boot1.log" ]; then + apt_update boot1 + fi + log "image already carries the regression; skipping pin install and second boot" + FINAL_SOURCES=sources-boot1.txt + FINAL_APT=apt-update-boot1.log + else + # ------------------------------------------------- install affected pkg + if [ -n "$CLOUD_INIT_DEB_DIR" ]; then + log "installing local cloud-init debs from $CLOUD_INIT_DEB_DIR" + scp -i "$KEY" -P "$SSH_PORT" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \ + "$CLOUD_INIT_DEB_DIR"/*.deb [email protected]:/tmp/ || die "scp of local debs failed" + guest "sudo apt-get install -y /tmp/cloud-init*.deb" \ + || die "install of local cloud-init debs failed" + elif [ "$CLOUD_INIT_VERSION" != "image" ]; then + log "pinning cloud-init=$CLOUD_INIT_VERSION (the affected noble package)" + if ! guest "sudo apt-get install -y --allow-downgrades cloud-init=$CLOUD_INIT_VERSION"; then + warn "apt pin of cloud-init=$CLOUD_INIT_VERSION failed (superseded in noble-updates)." + warn "Provide the affected/fixed arm64 deb(s) and re-run with CLOUD_INIT_DEB_DIR=/path/to/debs" + warn "Continuing with the image's installed cloud-init." + fi + collect cloud-init-pinned.txt 'cloud-init --version 2>&1 || dpkg-query -W cloud-init' + fi + + # ------------------------------------------- boot 2: fresh first boot + log "resetting cloud-init state and rebooting into a fresh first boot" + guest 'sudo cloud-init clean --logs --reboot || { sudo rm -rf /var/lib/cloud/*; sudo reboot; }' >/dev/null 2>&1 || true + QEMU_PID="$(cat "$PIDFILE")" # pid unchanged across guest reboots + sleep 20 + wait_ssh "boot2" + wait_cloud_init boot2 + + log "collecting fresh-first-boot evidence (this is the actual regression check)" + collect sources-boot2.txt "grep -E '^(Types|URIs|Suites|Components|Signed-By):' /etc/apt/sources.list.d/ubuntu.sources || true" + apt_update boot2 + FINAL_SOURCES=sources-boot2.txt + FINAL_APT=apt-update-boot2.log + fi + + # ---------------------------------------------------------------- verdict + log "evaluating results" + + if [ ! -f "$EVIDENCE/$FINAL_SOURCES" ] || [ ! -s "$EVIDENCE/$FINAL_SOURCES" ]; then + die "no apt sources evidence collected ($EVIDENCE/$FINAL_SOURCES)" + fi + if ! grep -q 'apt-get-update-rc=' "$EVIDENCE/$FINAL_APT"; then + die "no apt-get update evidence collected ($EVIDENCE/$FINAL_APT)" + fi + + URI_BAD=0; URI_GOOD=0; APT_RC_FAIL=0; APT_ARCHIVE_404=0 + if bad_uris "$FINAL_SOURCES"; then URI_BAD=1; fi + if grep -E '^URIs:.*ports\.ubuntu\.com' "$EVIDENCE/$FINAL_SOURCES" >/dev/null 2>&1; then URI_GOOD=1; fi + apt_rc_line="$(grep -o 'apt-get-update-rc=[0-9]*' "$EVIDENCE/$FINAL_APT" | tail -1)" + [ "${apt_rc_line#apt-get-update-rc=}" != 0 ] && APT_RC_FAIL=1 + grep -E "(archive|apt\.archive)\.ubuntu\.com.*(404|Failed to fetch)" "$EVIDENCE/$FINAL_APT" >/dev/null 2>&1 && APT_ARCHIVE_404=1 + + echo + echo "===============================================================" + echo " LP #2171959 reproduction results ($(date -u +%FT%TZ)) " + echo "---------------------------------------------------------------" + echo " evidence dir : $EVIDENCE" + echo " guest sources file: $( [ -n "${FINAL_SOURCES:-}" ] && echo "$EVIDENCE/$FINAL_SOURCES" )" + echo " image defaults : $(grep '^URIs:' "$EVIDENCE/sources-image-default.txt" 2>/dev/null | tr '\n' ' ' || echo n/a)" + echo " final URIs : $(grep '^URIs:' "$EVIDENCE/$FINAL_SOURCES" | tr '\n' ' ')" + echo " arm64 uses archive.ubuntu.com (BUG) : $URI_BAD" + echo " arm64 uses ports.ubuntu.com (OK) : $URI_GOOD" + echo " apt-get update failed : $APT_RC_FAIL (archive.ubuntu.com 404s: $APT_ARCHIVE_404)" + echo "===============================================================" + + if [ "$URI_BAD" = 1 ] && [ "$APT_RC_FAIL" = 1 ]; then + echo " VERDICT: REPRODUCED -- LP #2171959" + echo " cloud-init rewrote /etc/apt/sources.list.d/ubuntu.sources on arm64" + echo " noble to archive.ubuntu.com (noble arm64 packages only exist on" + echo " ports.ubuntu.com), so apt-get update fails. See:" + echo " $EVIDENCE/$FINAL_SOURCES" + echo " $EVIDENCE/$FINAL_APT" + exit 1 + fi + if [ "$URI_GOOD" = 1 ] && [ "$APT_RC_FAIL" = 0 ]; then + echo " VERDICT: NOT REPRODUCED -- apt sources stayed on ports.ubuntu.com" + echo " and apt-get update succeeded (fixed / unaffected cloud-init)." + exit 0 + fi + die "ambiguous result: URIs and apt-get update outcome disagree; inspect $EVIDENCE (is guest networking OK?)" + + ``` [ Where problems could occur ] - * this is a highly specific fix that reverts a small single commit + * this is a highly specific fix that reverts a small single commit that landed in upstream cloud-init to retain original behavior seen in 26.1. Prior to 26.2 - [ Other Info ] [ Original Description ] It looks like the PR https://github.com/canonical/cloud-init/pull/6826 was backported to noble, breaking arm64 targets, as noble stil uses ports.ubuntu.com. # Steps to reproduce the problem Install a noble arm64 image with could-init enabled. # Environment details * noble * arm64 * 26.1-0ubuntu1~24.04.1 # cloud-init logs Nothing relevant in logs, it just overrides ubuntu.sources with wrong addresses. ** Attachment added: "repro-lp-2171959.sh" https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/2171959/+attachment/6011748/+files/repro-lp-2171959.sh -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2171959 Title: cloud-init breaks noble arm64 targets To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/2171959/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
