Script 'mail_helper' called by obssrc Hello community, here is the log from the commit of package partclone for openSUSE:Factory checked in at 2026-10-30 18:17:24 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/partclone (Old) and /work/SRC/openSUSE:Factory/.partclone.new.1255 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "partclone" Fri Oct 30 18:17:24 2026 rev:33 rq:1393850 version:0.3.52 Changes: -------- --- /work/SRC/openSUSE:Factory/partclone/partclone.changes 2026-10-27 17:00:12.309730776 +0100 +++ /work/SRC/openSUSE:Factory/.partclone.new.1255/partclone.changes 2026-10-30 18:23:29.659383554 +0100 @@ -1,0 +2,11 @@ +Fri Oct 30 07:58:50 UTC 2026 - Martin Pluskal <[email protected]> + +- Update to 0.3.52: + * btrfs: deduplicate extents scanned more than once when + building the bitmap - bitmap calculation on snapshot-heavy + filesystems (e.g. snapper) no longer burns CPU looking like + a hang + * btrfs: new bitmap benchmark/verifier plus metadata and + file-checksum regression tests + +------------------------------------------------------------------- Old: ---- partclone-0.3.51.tar.gz New: ---- partclone-0.3.52.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ partclone.spec ++++++ --- /var/tmp/diff_new_pack.LrZIGr/_old 2026-10-30 18:23:30.662425543 +0100 +++ /var/tmp/diff_new_pack.LrZIGr/_new 2026-10-30 18:23:30.665425669 +0100 @@ -18,7 +18,7 @@ Name: partclone -Version: 0.3.51 +Version: 0.3.52 Release: 0 Summary: File System Clone Utilities License: GPL-2.0-or-later ++++++ partclone-0.3.51.tar.gz -> partclone-0.3.52.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/partclone-0.3.51/CONTRIBUTORS new/partclone-0.3.52/CONTRIBUTORS --- old/partclone-0.3.51/CONTRIBUTORS 2026-10-27 03:22:33.000000000 +0100 +++ new/partclone-0.3.52/CONTRIBUTORS 2026-10-28 08:13:32.000000000 +0100 @@ -1,4 +1,4 @@ - 997 Thomas Tsai + 1003 Thomas Tsai 257 thomas 59 Patryck Rouleau 54 Kenji Okimoto diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/partclone-0.3.51/ChangeLog new/partclone-0.3.52/ChangeLog --- old/partclone-0.3.51/ChangeLog 2026-10-27 03:22:33.000000000 +0100 +++ new/partclone-0.3.52/ChangeLog 2026-10-28 08:13:32.000000000 +0100 @@ -1,3 +1,83 @@ +2026-10-28 Thomas Tsai <[email protected]> + + Merge pull request #319 from Thomas-Tsai/fix/btrfs-bitmap-dedup + fix(btrfs): deduplicate extents scanned more than once when building the bitmap + +2026-10-27 Thomas Tsai <[email protected]> + + test(btrfs): metadata preservation check (flags, xattrs, links, qgroups) + Builds a small image with rich metadata fixtures (modes, owners, + symlinks, hardlinks, xattrs incl. empty/1.2K values, read-only+rw + snapshots, quota qgroups), then verifies that clone -> restore yields a + device whose full metadata dump (per-path lstat/xattr records, subvolume + ro flags, qgroup state) is byte-identical to the source. + + Two test-infra quirks are handled explicitly: + + * fixture creation must 'sync; umount' before exit, otherwise every + later mount replays the log tree and stamps wall-clock times into + recovered metadata; + * nested snapshot roots (a subvolume under another subvolume) report + mount-synthesized mtime/ctime per mount even though the on-disk bytes + are stable (proven by image sha256 being unchanged across mounts), so + those two fields are filtered for that path class only. + + test(btrfs): file-level sha256 check across snapshots after restore + Mounts each device read-only via guestmount (no root) and compares the + full file tree, the sha256 of every file in the root subvolume and the + sha256 of every file in one complete snapshot against the reference + device. Catches failures that block-range comparison cannot: broken + paths, unreadable files, mangled directory trees. + + test(btrfs): snapshot-heavy bitmap benchmark and ground-truth bitmap verifier + btrfs-bitmap-bench.sh builds a worst-case btrfs image (many snapshots + sharing the same extents, snapper-style) without root using libguestfs, + clones it with each given partclone.btrfs, compares clone times and + image hashes, and optionally gates on the verifier with a full + clone -> restore -> byte-compare cycle. + + btrfs-bitmap-verify.py extracts every EXTENT_ITEM/METADATA_ITEM (the + filesystem's allocation truth), maps all mirrors through the chunk + tree, asserts each covered block is set in the image bitmap, and + optionally byte-compares a restored device against the source over all + mapped physical ranges. + + fix(btrfs): deduplicate extents scanned more than once when building the bitmap + Snapshots re-enumerate every shared extent once per snapshot, and the + extent tree ROOT_ITEM walk re-enumerates all extents again on top of the + fs trees, so check_extent_bitmap() mapped and marked the same extent + many times over. On snapshot-heavy systems (e.g. snapper on openSUSE) + bitmap calculation burned tens of minutes of CPU for a few minutes of + real work, looking like a hang (Clonezilla live 3.3.3-37/3.3.4-10). + + Keep an rbtree of fully-processed extents and skip exact repeats; the + mapping table is static for the duration of the scan, so a skipped + repeat would produce the identical set of marks. + + Care was needed with the cache key and the lookup semantics: + + * cache_tree_comp_range() matches on ANY overlap, not containment, so a + partial overlap would wrongly count as visited and the unvisited tail + of an extent would be dropped from the bitmap. Only a strict + whole-range containment check may trigger the skip. + + * The key must include the check_extent_bitmap() "type" argument. + btrfs_map_block() expands the requested length to the rest of a + single-profile chunk, and the type flag decides whether that + expansion is clamped back to the request (type 1) or kept (type 0), + so identical (bytenr, length) calls with different type do NOT mark + the same physical range (observed: 16 MiB requested, 16 MiB vs 64 MiB + marked). insert_cache_extent2()/lookup_cache_extent2() let the rbtree + order by (type, start, size) while still allowing overlapping ranges + across types. + + Verified on a snapshot-full btrfs image: the generated clone image is + bit-identical to the image produced by the unpatched code, while the + CPU time spent in bitmap calculation no longer scales with the number + of snapshots. + + Bump version to 0.3.51 + 2026-10-26 Thomas Tsai <[email protected]> chore(btrfs): sync with btrfs-progs v7.3 diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/partclone-0.3.51/configure.ac new/partclone-0.3.52/configure.ac --- old/partclone-0.3.51/configure.ac 2026-10-27 03:22:33.000000000 +0100 +++ new/partclone-0.3.52/configure.ac 2026-10-28 08:13:32.000000000 +0100 @@ -1,4 +1,4 @@ -AC_INIT([Partclone],[0.3.51],[[email protected]]) +AC_INIT([Partclone],[0.3.52],[[email protected]]) #condition for CPU target AC_CANONICAL_TARGET diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/partclone-0.3.51/src/btrfsclone.c new/partclone-0.3.52/src/btrfsclone.c --- old/partclone-0.3.51/src/btrfsclone.c 2026-10-27 03:22:33.000000000 +0100 +++ new/partclone-0.3.52/src/btrfsclone.c 2026-10-28 08:13:32.000000000 +0100 @@ -68,6 +68,65 @@ void *thread_update_bitmap_pui(void *arg); +/// Cache of logical extents already pushed through check_extent_bitmap(). +/// Snapshots re-enumerate the same shared extents once per snapshot, and the +/// extent tree walk enumerates every extent again on top of the fs trees, so +/// the same extent is mapped and marked many times over. Skipping fully +/// visited extents turns that O(snapshots x extents) rework into O(1) +/// lookups; marking is idempotent, so the final bitmap is unchanged. +/// +/// The cache key is (type, bytenr, length): the "type" argument of +/// check_extent_bitmap() selects whether the length expanded by +/// btrfs_map_block() (to the rest of a single-profile chunk) is clamped +/// back to the request, so two calls with identical (bytenr, length) but +/// different type do NOT mark the same physical range. Keying only on +/// (bytenr, length) was observed to drop those extra marks: on a +/// single-device FS, a 16 MiB file extent visited first with type 1 then +/// skipped with type 0 lost the remaining 48 MiB of chunk marking. +static struct cache_tree visited_extents; + +static int extent_visited(u64 bytenr, u64 len, int type) +{ + struct cache_extent *e; + + if (len == 0) + return 1; + /* lookup_cache_extent2() matches any overlap within the same objectid, + * not just containment (see cache_tree_comp_range2()); a partial + * overlap must NOT count as visited or the unvisited tail of the + * extent would be dropped from the bitmap. Only skip when an existing + * entry strictly contains the whole [bytenr, bytenr+len) range. */ + e = lookup_cache_extent2(&visited_extents, (u64)type, bytenr, len); + if (!e) + return 0; + return e->start <= bytenr && bytenr + len <= e->start + e->size; +} + +static void extent_mark_visited(u64 bytenr, u64 len, int type) +{ + struct cache_extent *pe; + + if (len == 0) + return; + pe = malloc(sizeof(*pe)); + if (!pe) + return; + pe->objectid = (u64)type; + pe->start = bytenr; + pe->size = len; + /* insert fails on overlap with an entry of the same objectid + * (-EEXIST); the overlapping part was already marked, dropping the + * insert only means the unique tail may be re-processed later, which + * is safe. */ + if (insert_cache_extent2(&visited_extents, pe)) + free(pe); +} + +static void visited_extents_free(struct cache_extent *pe) +{ + free(pe); +} + /// number of tree blocks that could not be read while building the bitmap; /// any non-zero value means the resulting image would silently miss data, /// so read_bitmap() aborts at the end instead of producing a corrupt image. @@ -151,6 +210,11 @@ if (*num_bytes % root->fs_info->sectorsize) return -EINVAL; + if (extent_visited(bytenr, *num_bytes, type)) { + log_mesg(3, 0, 0, fs_opt.debug, "%s: skip already-marked extent bytenr %llu size %llu\n", __FILE__, bytenr, *num_bytes); + return 0; + } + if (type == 1){ maxlen = *num_bytes; } @@ -176,6 +240,7 @@ free(multi); multi = NULL; } + extent_mark_visited(bytenr, *num_bytes, type); return 0; } @@ -472,6 +537,7 @@ pthread_t prog_bitmap_thread; total_block = fs_info.totalblock; + cache_tree_init(&visited_extents); /// init progress; the tree walk total is unknown, so a background /// thread maps structural phases, pre-scanned root count and a @@ -665,6 +731,7 @@ //csum_bitmap(bitmap, root); btrfs_release_path(&path); bitmap_done = 1; + cache_tree_free_extents(&visited_extents, visited_extents_free); update_pui(&prog, 1, 1, 1); log_mesg(1, 0, 0, fs_opt.debug, "%s: bitmap roots = %lu, est items = %.0f, walked items = %llu\n", __FILE__, roots_total, est_total, walk_items); if (unreadable_tree_blocks) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/partclone-0.3.51/src/version.h new/partclone-0.3.52/src/version.h --- old/partclone-0.3.51/src/version.h 2026-10-27 03:22:33.000000000 +0100 +++ new/partclone-0.3.52/src/version.h 2026-10-28 08:13:32.000000000 +0100 @@ -3,5 +3,5 @@ * WHETHER THEY ARE BUILT BY OTHERS OR DURING DEVELOPMENT OR FOR THE * OFFICIAL PARTCLONE RELEASES. */ -#define git_version "1de3957f" +#define git_version "60c1a1f8" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/partclone-0.3.51/tests/btrfs-bitmap-bench.sh new/partclone-0.3.52/tests/btrfs-bitmap-bench.sh --- old/partclone-0.3.51/tests/btrfs-bitmap-bench.sh 1970-01-01 01:00:00.000000000 +0100 +++ new/partclone-0.3.52/tests/btrfs-bitmap-bench.sh 2026-10-28 08:13:32.000000000 +0100 @@ -0,0 +1,148 @@ +#!/bin/bash +# btrfs-bitmap-bench.sh - worst-case bitmap benchmark for partclone.btrfs +# +# Builds a snapshot-heavy btrfs image (snapper-like worst case: many +# snapshots sharing the same extents) using libguestfs (no root needed), +# then runs each given partclone.btrfs binary in clone mode and: +# 1. reports wall/user/sys time per run, +# 2. compares every produced image bit-for-bit against the FIRST +# binary's image (the reference), to prove no bitmap behavior change. +# +# Usage: +# btrfs-bitmap-bench.sh [-i IMAGE] [-s SNAPSHOTS] [-m SIZE_MB] [-d DATA_MB] \ +# [-k] [-V] [-R] [-n] BINARY [BINARY...] +# -k : keep existing IMAGE if present (default: rebuild) +# -V : verify each image against the fs's own extent tree (add when mixing +# binaries whose images legitimately differ, e.g. extent-tree builds) +# -R : with -V, also restore each image and byte-compare used extents +# -n : no bit-compare gate against the first binary's image +# Example: +# tests/btrfs-bitmap-bench.sh /path/master/partclone.btrfs src/partclone.btrfs +set -euo pipefail + +IMG=/tmp/opencode/btrfs-worst.img +SNAPS=64 +SIZE_MB=2048 +DATA_MB=384 +KEEP=0 +VERIFY=0 +RESTORE=0 +COMPARE=1 +SCRIPTDIR="$(cd "$(dirname "$0")" && pwd)" +WORKDIR="$(mktemp -d /tmp/opencode/btbench.XXXXXX)" + +while getopts "i:s:m:d:kVRn" o; do + case "$o" in + i) IMG="$OPTARG";; + s) SNAPS="$OPTARG";; + m) SIZE_MB="$OPTARG";; + d) DATA_MB="$OPTARG";; + k) KEEP=1;; + V) VERIFY=1;; + R) RESTORE=1; VERIFY=1;; + n) COMPARE=0;; + *) echo "unknown option" >&2; exit 2;; + esac +done +shift $((OPTIND-1)) +[ $# -ge 1 ] || { echo "need at least one partclone.btrfs binary" >&2; exit 2; } + +command -v guestfish >/dev/null || { echo "guestfish required" >&2; exit 1; } +command -v mkfs.btrfs >/dev/null || { echo "mkfs.btrfs required" >&2; exit 1; } + +if [ "$KEEP" = 0 ] || [ ! -f "$IMG" ]; then + echo "== building worst-case image: $IMG (${SIZE_MB}MiB, ${DATA_MB}MiB shared data, ${SNAPS} snapshots)" + rm -f "$IMG" + dd if=/dev/zero of="$IMG" bs=1M count="$SIZE_MB" status=none + mkfs.btrfs -q -f "$IMG" + # guestfish script: files of 16 MiB each in /data, snapshot after each + # small churn step so snapshots share almost everything. + { + echo "run" + echo "mount /dev/sda /" + echo "mkdir /data" + echo "mkdir /.snapshots" + nfiles=$(( (DATA_MB + 15) / 16 )) + for i in $(seq 1 "$nfiles"); do + echo "fill $((64 + i % 32)) 16777216 /data/blob$i" + done + # many small one-sector files: each costs one EXTENT_DATA item that + # every snapshot's fs tree re-enumerates; this is what makes the + # snapshot-heavy walk quadratic-ish in practice. + # snapshot-create speed in the appliance makes 4096 too slow, so keep + # it bounded; increase via -n if the box is fast. + smallfiles=${SMALLFILES:-1024} + for i in $(seq 1 "$smallfiles"); do + echo "fill $((33 + i % 16)) 4096 /data/tiny$i" + done + for s in $(seq 1 "$SNAPS"); do + # churn: overwrite 512 KiB in one file and add a tiny file per snap + echo "fill $((96 + s % 16)) 524288 /data/churn" + echo "btrfs-subvolume-snapshot / /.snapshots/snap$s" + done + echo "umount /" + echo "exit" + } > "$WORKDIR/build.gf" + guestfish --rw -a "$IMG" -f "$WORKDIR/build.gf" +fi + +echo "== source fs summary" +btrfs inspect-internal dump-super "$IMG" 2>/dev/null | grep -E "bytes_used|generation" || true + +REF="" +REFIMG="" +RESULTS=() +for bin in "$@"; do + name="$(basename "$(dirname "$(dirname "$bin")")").$(basename "$bin")" + name="${name//\//_}.$(echo "$bin" | md5sum | cut -c1-6)" + out="$WORKDIR/$name.pcl" + tlog="$WORKDIR/$name.time" + echo "== cloning with $bin" + /usr/bin/time -f "%e %U %S" -o "$tlog" "$bin" -c -a0 -s "$IMG" -O "$out" -L /dev/null >/dev/null 2>&1 + read -r wall user sys < "$tlog" + sum="$(sha256sum "$out" | awk '{print $1}')" + if [ -z "$REFIMG" ]; then + REFIMG="$out"; REF="$sum"; refname="$name"; verdict="(reference)" + elif [ "$COMPARE" = 1 ]; then + if [ "$sum" = "$REF" ]; then verdict="OK bit-identical to $refname"; else verdict="FAIL differs from $refname"; fi + else + verdict="(no bit-compare)" + fi + echo " wall=${wall}s user=${user}s sys=${sys}s sha=$sum $verdict" + RESULTS+=("$verdict") + + if [ "$VERIFY" = 1 ]; then + raw="$WORKDIR/$name.raw" + if [ "$RESTORE" = 1 ]; then + dd if=/dev/zero of="$raw" bs=1M count="$SIZE_MB" status=none + "$bin" -r -C -s "$out" -O "$raw" -L /dev/null >/dev/null 2>&1 + if python3 "$SCRIPTDIR/btrfs-bitmap-verify.py" "$out" "$IMG" "$raw" \ + > "$WORKDIR/$name.verify" 2>&1; then + RESULTS+=("verify+restore OK $name") + else + RESULTS+=("verify FAIL $name (see $WORKDIR/$name.verify)") + fi + else + if python3 "$SCRIPTDIR/btrfs-bitmap-verify.py" "$out" "$IMG" \ + > "$WORKDIR/$name.verify" 2>&1; then + RESULTS+=("verify OK $name") + else + RESULTS+=("verify FAIL $name (see $WORKDIR/$name.verify)") + fi + fi + tail -1 "$WORKDIR/$name.verify" | sed 's/^/ /' + fi +done + +echo +for v in "${RESULTS[@]}"; do + case "$v" in + *FAIL*) echo "RESULT: $v"; exit 1;; + esac +done +if [ "$COMPARE" = 1 ]; then + echo "RESULT: all images bit-identical" +else + echo "RESULT: all checks passed" +fi +echo "workdir: $WORKDIR" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/partclone-0.3.51/tests/btrfs-bitmap-verify.py new/partclone-0.3.52/tests/btrfs-bitmap-verify.py --- old/partclone-0.3.51/tests/btrfs-bitmap-verify.py 1970-01-01 01:00:00.000000000 +0100 +++ new/partclone-0.3.52/tests/btrfs-bitmap-verify.py 2026-10-28 08:13:32.000000000 +0100 @@ -0,0 +1,192 @@ +#!/usr/bin/env python3 +"""btrfs-bitmap-verify.py - ground-truth verification of a partclone image bitmap. + +Extracts every EXTENT_ITEM/METADATA_ITEM (the filesystem's allocation truth) +from a btrfs device, maps them to physical ranges through the chunk tree, and +asserts that every mirror of every used extent is marked in the bitmap stored +inside the given partclone image (format 0002, BM_BIT, no per-block checksum). + +Usage: btrfs-bitmap-verify.py IMAGE.pcl SOURCE_DEVICE [RESTORED_RAW] +Exit 0 when every used extent is covered; exit 1 with details otherwise. + +With RESTORED_RAW (a device restored from the same image), additionally byte +-compares SOURCE_DEVICE vs RESTORED_RAW over every mapped physical range of +the used extents, proving the restore reproduces the real data exactly. + +Intentionally supports the layouts used by the benches (single device, +single/DUP/RAID1{,C3,C4} profiles); anything RAID0/5/6/10 aborts as +unsupported instead of guessing. +""" +import re +import struct +import subprocess +import sys + +def run(cmd): + return subprocess.run(cmd, capture_output=True, text=True, check=True).stdout + +def parse_super(raw): + tot = used = node = sect = None + for line in raw.splitlines(): + m = re.match(r"\s*(total_bytes|bytes_used|nodesize|sectorsize)\s+(\d+)", line) + if m: + v = int(m.group(2)) + if m.group(1) == "total_bytes": tot = v + elif m.group(1) == "bytes_used": used = v + elif m.group(1) == "nodesize": node = v + elif m.group(1) == "sectorsize": sect = v + return tot, used, node, sect + +CHUNK_TYPES = { + "SINGLE": 0, "DUP": 0x10, + "RAID0": 1, "RAID1": 2, "RAID1C3": 0x40, "RAID1C4": 0x80, + "RAID10": 4, "RAID5": 8, "RAID6": 0x20, +} + +def parse_chunks(text): + chunks = [] + cur = None + for line in text.splitlines(): + m = re.search(r"key \(\S+ CHUNK_ITEM (\d+)\)", line) + if m: + cur = {"logical": int(m.group(1)), "stripes": [], "type": None, "len": None} + chunks.append(cur) + continue + m = re.match(r"\s*length (\d+) owner \d+ stripe_len \d+ type (\S+)", line) + if m and cur is not None: + cur["len"] = int(m.group(1)) + cur["type"] = m.group(2) + continue + m = re.match(r"\s*num_stripes (\d+)", line) + if m and cur is not None: + cur["num_stripes"] = int(m.group(1)) + continue + m = re.match(r"\s*stripe \d+ devid \d+ offset (\d+)", line) + if m and cur is not None: + cur["stripes"].append(int(m.group(1))) + return chunks + +def parse_extents(text): + exts = [] + for line in text.splitlines(): + m = re.search(r"key \((\d+) EXTENT_ITEM (\d+)\)", line) + if m: + exts.append((int(m.group(1)), int(m.group(2)), "data")) + continue + m = re.search(r"key \((\d+) METADATA_ITEM (\d+)\)", line) + if m: + exts.append((int(m.group(1)), None, "meta")) # len = nodesize + return exts + +def map_ranges(chunks, logical, length): + for c in chunks: + if c["logical"] <= logical < c["logical"] + c["len"]: + t = c["type"] + if "RAID0" in t or "RAID5" in t or "RAID6" in t or "RAID10" in t: + sys.exit(f"unsupported chunk profile {t}") + off = logical - c["logical"] + if off + length > c["len"]: + sys.exit(f"extent {logical}+{length} crosses chunk boundary (unsupported)") + return [(s + off, length) for s in c["stripes"]] + sys.exit(f"no chunk covers logical {logical}") + +def main(): + img_path, dev = sys.argv[1], sys.argv[2] + restored_path = sys.argv[3] if len(sys.argv) > 3 else None + # image_desc_v2 is packed: head(36) + fs_info(52) + options(18) + crc(4) + HEAD_V2 = 36 + DESC_V2 = 110 + with open(img_path, "rb") as f: + head = f.read(DESC_V2) + if head[0:15] != b"partclone-image": + sys.exit(f"{img_path}: not a partclone image") + if head[30:34] != b"0002": + sys.exit("only format 0002 supported") + size, total, used_sb, used_bm = struct.unpack("<QQQQ", head[52:84]) + bsize = struct.unpack("<I", head[84:88])[0] + csum_mode, = struct.unpack("<H", head[96:98]) + bitmap_mode = head[105] + if csum_mode != 0: + sys.exit("verification only supports -a0 images (no block checksums)") + if bitmap_mode != 1: # BM_BIT + sys.exit(f"unsupported bitmap_mode {bitmap_mode} (only BM_BIT)") + if struct.unpack("<I", head[88:92])[0] != 18: + sys.exit("unexpected image_options_v2 layout") + if bsize == 0 or total == 0: + sys.exit("invalid header fields") + nbitmap = (total + 7) // 8 + f.seek(DESC_V2) + bm = f.read(nbitmap) + if len(bm) != nbitmap: + sys.exit("image shorter than expected (bitmap truncated?)") + + def block_set(b): + if b >= total: + return False + return bool(bm[b // 8] & (1 << (b % 8))) + + sup = run(["btrfs", "inspect-internal", "dump-super", dev]) + dev_total, dev_used, nodesize, sectorsize = parse_super(sup) + if bsize != nodesize: + sys.exit(f"image block_size {bsize} != fs nodesize {nodesize}") + + chunks = parse_chunks(run(["btrfs", "inspect-internal", "dump-tree", "-t", "chunk", dev])) + exts = parse_extents(run(["btrfs", "inspect-internal", "dump-tree", "-t", "extent", dev])) + if not chunks or not exts: + sys.exit("failed to parse chunk or extent trees") + + missing_blocks = set() + covered = 0 + mapped_ranges = [] + for logical, length, kind in exts: + if kind == "meta": + length = nodesize + for phys, plen in map_ranges(chunks, logical, length): + mapped_ranges.append((phys, plen)) + # same coverage rule as partclone's set_bitmap() + b0 = phys // bsize + b1 = (phys + plen + bsize - 1) // bsize + for b in range(b0, b1): + if block_set(b): + covered += 1 + else: + missing_blocks.add(b) + + print(f"device: totalblocks={total} bsize={bsize} used_sb={used_sb} image_used_bm={used_bm}") + print(f"extents: {len(exts)} items, covered blocks={covered}, missing blocks={len(missing_blocks)}") + if missing_blocks: + mb = sorted(missing_blocks) + print("first missing ranges:") + s = p = mb[0] + shown = 0 + for x in mb[1:]: + if x != p + 1: + print(f" blocks {s}-{p} (0x{s*bsize:x}-0x{(p+1)*bsize:x})") + shown += 1 + if shown >= 10: + break + s = x + p = x + print(f" blocks {s}-{p} (0x{s*bsize:x}-0x{(p+1)*bsize:x})") + sys.exit(1) + print("OK: every used extent (all mirrors) is marked in the image bitmap") + + if restored_path: + bad = 0 + with open(dev, "rb") as fs, open(restored_path, "rb") as rr: + for phys, plen in mapped_ranges: + fs.seek(phys) + rr.seek(phys) + a, b = fs.read(plen), rr.read(plen) + if a != b: + off = next((i for i in range(len(a)) if a[i] != b[i]), 0) + print(f"MISMATCH at 0x{phys+off:x} (extent phys 0x{phys:x} len {plen})") + bad += 1 + if bad >= 10: + break + if bad: + sys.exit(1) + print(f"OK: restored data matches source over {len(mapped_ranges)} physical ranges") + +if __name__ == "__main__": + main() diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/partclone-0.3.51/tests/btrfs-file-cksum-check.sh new/partclone-0.3.52/tests/btrfs-file-cksum-check.sh --- old/partclone-0.3.51/tests/btrfs-file-cksum-check.sh 1970-01-01 01:00:00.000000000 +0100 +++ new/partclone-0.3.52/tests/btrfs-file-cksum-check.sh 2026-10-28 08:13:32.000000000 +0100 @@ -0,0 +1,56 @@ +#!/bin/bash +# btrfs-file-cksum-check.sh - file-level checksum verification after restore. +# +# Mounts each given btrfs device read-only via guestmount (libguestfs, no +# root needed) and compares, against the FIRST device (the reference): +# 1. the complete file list of the whole tree (all snapshots included), +# 2. sha256 of every file outside /.snapshots (the root subvolume), +# 3. sha256 of every file in one full snapshot (.snapshots/snap50 by +# default, SNAPSHOT= to override). +# +# Usage: btrfs-file-cksum-check.sh SOURCE_DEV RESTORED_DEV [RESTORED_DEV...] +set -euo pipefail + +SNAPSUB="${SNAPSHOT:-.snapshots/snap50}" +[ $# -ge 2 ] || { echo "usage: $0 SOURCE_DEV RESTORED_DEV..." >&2; exit 2; } + +W="$(mktemp -d /tmp/opencode/ckchk.XXXXXX)" +mkdir -p "$W/mnt" + +collect() { + local dev="$1" tag="$2" + guestmount --ro -a "$dev" -m /dev/sda "$W/mnt" + ( cd "$W/mnt" && { + find . -type f | sort > "$W/files.$tag" + find . -path ./.snapshots -prune -o -type f -print0 | sort -z \ + | xargs -0 sha256sum > "$W/cksums.$tag" + if [ -d "$SNAPSUB" ]; then + find "./$SNAPSUB" -type f -print0 | sort -z \ + | xargs -0 sha256sum | sed "s|\./$SNAPSUB/|@snap/|" > "$W/cksumsnap.$tag" + fi + } ) + guestunmount "$W/mnt" +} + +fail=0 +collect "$1" ref +shift +for dev in "$@"; do + tag="$(basename "$dev")" + echo "== $dev" + collect "$dev" "$tag" + cmp -s "$W/files.ref" "$W/files.$tag" \ + && echo " file list: identical ($(wc -l < "$W/files.$tag") files)" \ + || { echo " file list: DIFFERS"; fail=1; } + cmp -s "$W/cksums.ref" "$W/cksums.$tag" \ + && echo " root-subvol sha256 ($(wc -l < "$W/cksums.$tag") files): identical" \ + || { echo " root-subvol sha256: DIFFERS"; fail=1; } + if [ -s "$W/cksumsnap.ref" ]; then + cmp -s "$W/cksumsnap.ref" "$W/cksumsnap.$tag" \ + && echo " $SNAPSUB sha256 ($(wc -l < "$W/cksumsnap.$tag") files): identical" \ + || { echo " $SNAPSUB sha256: DIFFERS"; fail=1; } + fi +done +echo "workdir: $W" +[ "$fail" = 0 ] && echo "RESULT: all file checksums identical" || echo "RESULT: FAILURES above" +exit "$fail" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/partclone-0.3.51/tests/btrfs-metadata-check.sh new/partclone-0.3.52/tests/btrfs-metadata-check.sh --- old/partclone-0.3.51/tests/btrfs-metadata-check.sh 1970-01-01 01:00:00.000000000 +0100 +++ new/partclone-0.3.52/tests/btrfs-metadata-check.sh 2026-10-28 08:13:32.000000000 +0100 @@ -0,0 +1,146 @@ +#!/bin/bash +# btrfs-metadata-check.sh - metadata preservation check for partclone.btrfs. +# +# Builds a small btrfs image packed with metadata fixtures (modes, owners, +# symlinks, hardlinks, xattrs incl. empty/multiline/binary-ish values, +# read-only+rw snapshots, quota qgroups), then for each given partclone.btrfs: +# clone -> restore -> dump every file's metadata (mode/uid/gid/size/type/ +# nlink/mtime/target/xattrs) plus subvolume ro flags and qgroup state from +# the restored device, and diff it against the source device's dump. +# +# Usage: btrfs-metadata-check.sh [-i IMG] [-k] BINARY [BINARY...] +# -k : keep/build-once (default: rebuild image each run) +set -euo pipefail + +IMG=/tmp/opencode/btrfs-meta.img +KEEP=0 +while getopts "i:k" o; do + case "$o" in + i) IMG="$OPTARG";; + k) KEEP=1;; + *) exit 2;; + esac +done +shift $((OPTIND-1)) +[ $# -ge 1 ] || { echo "need at least one partclone.btrfs binary" >&2; exit 2; } + +W="$(mktemp -d /tmp/opencode/metachk.XXXXXX)" +SIZE_MB=256 + +if [ "$KEEP" = 0 ] || [ ! -f "$IMG" ]; then + rm -f "$IMG" + truncate -s ${SIZE_MB}M "$IMG" + mkfs.btrfs -q -f "$IMG" + BIG=$(printf 'X%.0s' $(seq 1 1200)) + guestfish --rw -a "$IMG" <<EOF +run +mount /dev/sda / +mkdir /d700 +chmod 0700 /d700 +chown 1234 4321 /d700 +mkdir /d750 +chmod 0750 /d750 +write /d700/a.txt "alpha" +write /d700/b.bin "0 1 2 3" +chmod 0640 /d700/a.txt +chmod 0600 /d700/b.bin +chown 100 200 /d700/a.txt +chown 0 0 /d700/b.bin +setxattr user.author "alice; the builder" 18 /d700/a.txt +setxattr user.empty "" 0 /d700/a.txt +setxattr user.big "$BIG" 1200 /d700/a.txt +setxattr user.role "build\none" 10 /d750 +setxattr user.dir "yes" 3 /d700 +ln-s /d700/a.txt /link.rel +ln-s ../d700 /d750/link.up +ln /d700/a.txt /d700/hard.txt +mkdir /.snapshots +btrfs-subvolume-snapshot / /.snapshots/rw1 +btrfs-subvolume-snapshot / /.snapshots/ro1 ro:true +btrfs-subvolume-snapshot / /.snapshots/ro2 ro:true +btrfs-quota-enable / true +btrfs-quota-rescan / +btrfs-qgroup-create 1/1000 / +btrfs-qgroup-limit / 1073741824 +sync +umount / +EOF + # best-effort: xattr on symlink itself (kernel-dependent; ignore failure) + guestfish --rw -a "$IMG" run : mount /dev/sda / : lsetxattr user.lnk "on-symlink" 10 /link.rel : sync : umount / >/dev/null 2>&1 || true +fi + +# dump_metadata DEVICE OUTPUT +# one deterministic text stream with all metadata of the filesystem; +# uses two appliance sessions: one to enumerate, one for the full dump. +dump_metadata() { + local dev="$1" out="$2" + guestfish --rw -a "$dev" run : mount-ro /dev/sda / : btrfs-subvolume-list / : find / > "$W/enum" 2>&1 + sed 's|^/||' "$W/enum" | grep -vE '^(\[|\}| *btrfssubvolume|$)' | sort | uniq > "$W/files" + grep -oE "btrfssubvolume_path: .*" "$W/enum" | cut -d' ' -f2 | sort -u > "$W/subvols" + # "nested snapshot roots" (a subvolume whose path has a subvolume + # ancestor): the kernel synthesizes their root-dir mtime/ctime at mount + # time -- the on-disk bytes are stable (verified by image sha256), but + # every mount reports a different value, so they must be filtered out + # of the comparison or it can never pass. + : > "$W/nested" + # a path P is a nested snapshot root when P = Q + "/" + S where Q is a + # listed subvolume ancestor and S is itself a listed subvolume path + # (such nested roots are hidden from `subvolume list`) + while read -r p; do + while read -r q; do + case "$p" in + "$q"/*) + rest="${p#$q/}" + while read -r s; do + [ "$rest" = "$s" ] && { echo "$p" >> "$W/nested"; break 2; } + done < "$W/subvols" + ;; + esac + done < "$W/subvols" + done < "$W/files" + sort -u "$W/nested" -o "$W/nested" 2>/dev/null || true + { + echo "run"; echo "mount-ro /dev/sda /" + echo "btrfs-subvolume-list /" + while read -r sv; do + printf 'echo "SUBVOL-FLAGS %s"\n' "$sv" + printf 'btrfs-subvolume-show "/%s"\n' "$sv" + done < "$W/subvols" + echo "btrfs-qgroup-show /" + while read -r p; do + printf 'echo "PATH /%s"\n' "$p" + printf 'lstatns "/%s"\n' "$p" + printf 'lgetxattrs "/%s"\n' "$p" + done < "$W/files" + echo "exit" + } > "$W/dump.gf" + guestfish --rw -a "$dev" -f "$W/dump.gf" 2>&1 | awk -v nestedfile="$W/nested" ' + BEGIN { while ((getline l < nestedfile) > 0) nested["/" l] = 1; skip = 0 } + /^PATH / { skip = ($2 in nested) ? 1 : 0 } + skip && /^st_(mtime|ctime)_(sec|nsec):/ { next } + { print } + ' > "$out" || true +} + +echo "== dumping source: $IMG" +dump_metadata "$IMG" "$W/dump.ref" +fail=0 +for bin in "$@"; do + name="$(basename "$(dirname "$(dirname "$bin")")").$(echo "$bin" | md5sum | cut -c1-6)" + img="$W/$name.pcl"; raw="$W/$name.raw" + echo "== $bin" + "$bin" -c -a0 -s "$IMG" -O "$img" -L /dev/null >/dev/null 2>&1 + dd if=/dev/zero of="$raw" bs=1M count=$SIZE_MB status=none + "$bin" -r -C -s "$img" -O "$raw" -L /dev/null >/dev/null 2>&1 + dump_metadata "$raw" "$W/dump.$name" + if cmp -s "$W/dump.ref" "$W/dump.$name"; then + echo " metadata dump ($W/dump.$name): IDENTICAL to source" + else + echo " metadata dump: DIFFERS from source:" + diff "$W/dump.ref" "$W/dump.$name" | head -30 || true + fail=1 + fi +done +echo "workdir: $W" +[ "$fail" = 0 ] && echo "RESULT: all metadata preserved" || echo "RESULT: FAILURES above" +exit "$fail"
