** Description changed:
[ Impact ]
- * reprepro keeps all repository state in Berkeley DB files under db/
- (checksums.db, packages.db, references.db, packagenames.db). It opens them
- without a transactional environment, so they are only consistent if
- reprepro closes them cleanly before it exits.
+ * reprepro keeps all repository state in Berkeley DB files under db/
+ (checksums.db, packages.db, references.db, packagenames.db). It opens them
+ without a transactional environment, so they are only consistent if
+ reprepro closes them cleanly before it exits.
- * reprepro handles SIGTERM, SIGINT, SIGQUIT and SIGABRT by stopping through
- its graceful interruption path, which closes the databases and removes
- db/lockfile. It does not handle SIGHUP. When the terminal reprepro runs on
- goes away, the kernel sends it SIGHUP and it dies at once with its
- databases open. That happens when an ssh session drops, when the process
- that started reprepro on a pty exits, or a reboot.
+ * reprepro handles SIGTERM, SIGINT, SIGQUIT and SIGABRT by stopping through
+ its graceful interruption path, which closes the databases and removes
+ db/lockfile. It does not handle SIGHUP. When the terminal reprepro runs on
+ goes away, the kernel sends it SIGHUP and it dies at once with its
+ databases open. That happens when an ssh session drops, when the process
+ that started reprepro on a pty exits, or a reboot.
- * If the hangup arrives while reprepro is writing (from "Installing (and
- possibly deleting) packages..." until it closes its databases), the
- databases are left corrupt and db/lockfile is left behind.
+ * If the hangup arrives while reprepro is writing (from "Installing (and
+ possibly deleting) packages..." until it closes its databases), the
+ databases are left corrupt and db/lockfile is left behind.
- * Landscape Server mirrors Ubuntu archives with reprepro. Its
- job-handler service runs "reprepro update" on a pty (Twisted
- spawnProcess with usePTY=True) so it can read the line-buffered
- "--show-percent" progress output. Stopping job-handler (e.g. a Landscape
- restart or a reboot) closes the pty, and reprepro
- receives SIGHUP. If a mirror sync is in its install phase at that moment,
- the databases of that distribution are corrupted. Every series and pocket
- of a distribution shares one db/, so the damage is not limited to the
- pocket that was syncing. Landscape takes sync progress only from
- reprepro's "Got N%" lines, so a later sync where reprepro loops on the
- damaged btree shows 0% indefinitely, with no error.
+ * Landscape Server mirrors Ubuntu archives with reprepro. Its
+ job-handler service runs "reprepro update" on a pty (Twisted
+ spawnProcess with usePTY=True) so it can read the line-buffered
+ "--show-percent" progress output. Stopping job-handler (e.g. a Landscape
+ restart or a reboot) closes the pty, and reprepro
+ receives SIGHUP. If a mirror sync is in its install phase at that moment,
+ the databases of that distribution are corrupted. Every series and pocket
+ of a distribution shares one db/, so the damage is not limited to the
+ pocket that was syncing. Landscape takes sync progress only from
+ reprepro's "Got N%" lines, so a later sync where reprepro loops on the
+ damaged btree shows 0% indefinitely, with no error.
[ Test Plan ]
- todo
+ Use a disposable machine or container for the Ubuntu release being tested.
+ Run the commands in Bash, keeping the same shell open throughout. No existing
+ repository or signing key is needed. Do not use production repository data.
+
+ 1. Install the release package and record its version and origin:
+
+ sudo apt update
+ sudo apt install reprepro
+ dpkg-query -W reprepro
+ apt-cache policy reprepro
+
+ 2. Create 2,000 tiny test packages. This gives us enough time to interrupt an
+ import. Keep these files for both runs.
+
+ WORK="$(mktemp -d)"
+ mkdir -p "$WORK/build/DEBIAN" "$WORK/pkgs"
+ for i in $(seq -w 1 2000); do
+ cat > "$WORK/build/DEBIAN/control" <<EOF
+ Package: sru-test-$i
+ Version: 1.0
+ Architecture: all
+ Section: misc
+ Priority: optional
+ Maintainer: Test <[email protected]>
+ Description: test package for LP 2169601
+ EOF
+ dpkg-deb --build "$WORK/build" "$WORK/pkgs/$i.deb" >/dev/null
+ done
+
+ 3. Create an empty repository. Repeat this step whenever the plan asks for a
+ fresh repository; it leaves earlier results untouched.
+
+ REPO="$(mktemp -d "$WORK/repo.XXXXXX")"
+ mkdir -p "$REPO/conf"
+ cat > "$REPO/conf/distributions" <<EOF
+ Codename: test
+ Components: main
+ Architectures: $(dpkg --print-architecture)
+ EOF
+
+ 4. Start an import and send one SIGHUP after the twentieth package reaches
+ the repository. This is the signal reprepro receives on a terminal hangup.
+ The loop waits about 20 seconds for that package, rather than relying on
+ a fixed sleep or just the lock file appearing.
+
+ reprepro -b "$REPO" includedeb test "$WORK"/pkgs/*.deb >"$REPO/import.log"
2>&1 &
+ pid=$!
+ ready="$REPO/pool/main/s/sru-test-0020/sru-test-0020_1.0_all.deb"
+ for attempt in $(seq 1 400); do
+ test -f "$ready" && break
+ kill -0 "$pid" 2>/dev/null || break
+ sleep 0.05
+ done
+ if test -f "$ready" && kill -HUP "$pid"; then
+ status=0
+ wait "$pid" || status=$?
+ echo "exit=$status"
+ else
+ echo "INCONCLUSIVE: missed the import; repeat steps 3-4."
+ fi
+ cat "$REPO/import.log"
+ if test -e "$REPO/db/lockfile"; then echo "lock remains"; else echo "lock
removed"; fi
+
+ Before the fix: expect exit=129 (terminated by SIGHUP) and "lock remains".
+ Bash may also print "Hangup". Database corruption is timing-dependent and
+ is not required to reproduce the bug; the abrupt exit and stale lock are
+ sufficient. Do not delete this lock or reuse this repository.
+
+ If the command has not exited within a minute after the signal, record a
+ failure and retain the log; do not send another SIGHUP to make it pass.
+
+ 5. Rebuild and install reprepro with cherry picked upstream commit
+ e4dfe4f2857a3566fa944527156594024009595a
+
(https://salsa.debian.org/debian/reprepro/-/commit/e4dfe4f2857a3566fa944527156594024009595a)
+
+ Repeat steps 3-4 with the fixed package, using a fresh repository.
+
+ After the fix: expect "Interruption in progress" in the log and
+ "lock removed". A nonzero exit is normal (247 in the tested package), but
+ it must not be the default SIGHUP termination seen before the fix.
+ A warning about unexported indices is also normal: interruption stops the
+ import early; it does not undo packages already imported.
+
+ 6. Check that the interrupted repository is usable, then finish the
+ import:
+
+ timeout 30s reprepro -b "$REPO" check && echo "check passed"
+ timeout 30s reprepro -b "$REPO" list test >"$REPO/list.txt" && echo "list
passed"
+ timeout 60s reprepro -b "$REPO" includedeb test "$WORK"/pkgs/*.deb
>"$REPO/resume.log" 2>&1 && echo "import passed"
+ timeout 30s reprepro -b "$REPO" export test && echo "export passed"
+ timeout 30s reprepro -b "$REPO" check && echo "check passed"
+ timeout 30s reprepro -b "$REPO" list test >"$REPO/list.txt" && wc -l
<"$REPO/list.txt"
+
+ Expected: every "passed" message appears, no command times out, and the
+ final count is 2000. A timeout, error, or missing package is a failure.
+
+ 7. Check the nohup regression: repeat step 3, then repeat step 4 with only its
+ first command changed to the following (keep the rest of step 4 unchanged):
+
+ nohup reprepro -b "$REPO" includedeb test "$WORK"/pkgs/*.deb
+ >"$REPO/import.log" 2>&1 &
+
+ Expected: despite receiving SIGHUP, the import completes with exit=0 and
+ "lock removed". Confirm all packages were imported and the repository is
+ consistent:
+
+ timeout 30s reprepro -b "$REPO" list test >"$REPO/list.txt" && wc -l
<"$REPO/list.txt"
+ timeout 30s reprepro -b "$REPO" check && echo "check passed"
+
+ Expected: 2000 and "check passed". This also works before the fix; it
checks
+ that adding the SIGHUP handler has not broken inherited ignored SIGHUP.
[ Where problems could occur ]
- * The change is in signal setup at the start of main(). If it were wrong,
- the visible effect would be in how reprepro reacts to a terminal hangup.
- Repository contents, the database format and the command line are not
- touched.
+ * The change is in signal setup at the start of main(). If it were wrong,
+ the visible effect would be in how reprepro reacts to a terminal hangup.
+ Repository contents, the database format and the command line are not
+ touched.
- todo
+ * A regression would most likely show up as incorrect process behavior when
+ the controlling terminal hangs up:
+ - reprepro could ignore a SIGHUP that should interrupt it.
+ - reprepro could terminate immediately rather than running cleanup.
+ - nohup-style runs could stop being immune to SIGHUP.
+ - cleanup could fail to remove db/lockfile, blocking later repository work.
+
[ Other Info ]
- todo
+ * Debian merge request, merged into master:
+ https://salsa.debian.org/debian/reprepro/-/merge_requests/22
+
+ * Noble test PPA:
+ https://launchpad.net/~quanchobi/+archive/ubuntu/reprepro-lp2169601
** Changed in: reprepro (Ubuntu)
Assignee: (unassigned) => Jacob Hanks (quanchobi)
** Also affects: reprepro (Debian)
Importance: Undecided
Status: New
** Changed in: reprepro (Debian)
Status: New => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2169601
Title:
SIGHUP during db write corrupts the repository
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/reprepro/+bug/2169601/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
