https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=259796
Bug ID: 259796
Summary: [zfs] datasets corrupted when moving between openzfs
on 12.3-BETA* and 13.0-RELEASE zfs
Product: Base System
Version: 13.0-RELEASE
Hardware: amd64
OS: Any
Status: New
Severity: Affects Only Me
Priority: ---
Component: kern
Assignee: [email protected]
Reporter: [email protected]
It's not entirely clear to me what's happened to get to this
point, so first I report the observed issue, and then try to
recall the steps.
some file contents are different after transferring zstd-9
datasets between zpools, between 13.0-RELEASE & 12.3-BETA1
+ openzfs-kmod, using `zfs send -LevcR` & `zfs recv -Fuvs`
flags.
sha comparisons and git repo diffs confirm this, as does hd
output. file size appears correct, but files are not in fact
same length according to hd(1).
-rw-r--r-- 1 dch wheel 461B Oct 30 22:32 base.yml
-rw-r--r-- 1 dch wheel 392B Oct 30 22:32 bootstrap.yml
-rw-r--r-- 1 dch wheel 280B Oct 30 22:32 certs.yml
-rw-r--r-- 1 dch wheel 75B Oct 30 22:32 common.yml
-rw-r--r-- 1 dch wheel 797B Oct 30 22:32 dns.yml
-rw-r--r-- 1 dch wheel 741B Oct 30 22:32 jails.yml
-rw-r--r-- 1 dch wheel 68B Oct 30 22:32 lb.yml
-rw-r--r-- 1 dch wheel 401B Oct 30 22:32 mgmt.yml
-rw-r--r-- 1 dch wheel 80B Oct 30 22:32 provision.yml
-rw-r--r-- 1 dch wheel 266B Oct 30 22:32 site.yml
-rw-r--r-- 1 dch wheel 452B Oct 30 22:32 toggle-backends.yml
-rw-r--r-- 1 dch wheel 200B Oct 30 22:32 update.yml
-rw-r--r-- 1 dch wheel 221B Oct 30 22:32 upgrade.yml
-rw-r--r-- 1 dch wheel 93B Oct 30 22:32 zen.yml
## sha256 of yml files from pre & post send filesystems
uniq -c |egrep -v '\s+2'
1 SHA256 (dns.yml) =
18681a16ee52ec4f57aef197a28a442157485db38529e2c818a5f2f841a330a0
1 SHA256 (dns.yml) =
7dfdbec88da403910b89aa41601650c2eb1b5e2a580e34d043c39e91cf138849
1 SHA256 (jails.yml) =
4387866be6ce796d321f3738a4a7a2a310bd256f902dd11eca87db700bf2f8c0
1 SHA256 (jails.yml) =
6b6d50f5cd9e52fe3da39a44d0560e5f9b80b9778f8f52d72b7b7c0bb34f97c9
...
# sample file
## corrupt jails.yml
00000000 00 00 00 f5 f0 19 2d 2d 2d 0a 2d 20 68 6f 73 74 |......---.- host|
00000010 73 3a 20 73 65 72 65 6e 69 74 79 5f 6a 61 69 6c |s: serenity_jail|
00000020 73 0a 20 20 62 65 63 6f 6d 65 3a 20 79 65 0e 00 |s. become: ye..|
00000030 a0 67 61 74 68 65 72 5f 66 61 63 2d 00 f5 02 6e |.gather_fac-...n|
00000040 6f 0a 20 20 72 6f 6c 65 73 3a 0a 20 20 20 20 2d |o. roles:. -|
00000050 3f 00 16 0a 52 00 4f 65 64 65 6e 4e 00 24 73 64 |?...R.OedenN.$sd|
00000060 6e 73 64 69 73 74 5c 00 00 49 00 03 0b 00 97 7a |nsdist\..I.....z|
00000070 6f 6e 65 32 6a 73 6f 6e 68 00 7f 63 6f 75 63 68 |one2jsonh..couch|
00000080 64 62 6b 00 24 03 3e 00 07 50 00 3f 77 77 77 4c |dbk.$.>..P.?wwwL|
00000090 00 24 20 68 32 18 01 00 0f 01 37 77 77 77 52 00 |.$ h2.....7wwwR.|
000000a0 2f 6d 75 51 00 24 27 6d 75 46 00 1f 66 e4 00 26 |/muQ.$'muF..f..&|
000000b0 19 66 e0 00 6f 6d 61 74 72 69 78 92 00 24 87 64 |.f..omatrix..$.d|
000000c0 65 6e 64 72 69 74 65 98 00 9f 61 76 61 6c 61 6e |endrite...avalan|
000000d0 63 68 65 53 00 24 05 40 00 07 54 00 9f 73 65 61 |[email protected]|
000000e0 77 65 65 64 66 73 54 00 24 05 40 00 2f 0a 00 01 |weedfsT.$.@./...|
000000f0 00 ff 03 50 00 00 00 00 00 00 00 00 00 00 00 00 |...P............|
00000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000002e0
too short!!
## valid jails.yml
00000000 2d 2d 2d 0a 2d 20 68 6f 73 74 73 3a 20 73 65 72 |---.- hosts: ser|
00000010 65 6e 69 74 79 5f 6a 61 69 6c 73 0a 20 20 62 65 |enity_jails. be|
00000020 63 6f 6d 65 3a 20 79 65 73 0a 20 20 67 61 74 68 |come: yes. gath|
00000030 65 72 5f 66 61 63 74 73 3a 20 6e 6f 0a 20 20 72 |er_facts: no. r|
00000040 6f 6c 65 73 3a 0a 20 20 20 20 2d 20 73 65 72 65 |oles:. - sere|
00000050 6e 69 74 79 0a 0a 2d 20 68 6f 73 74 73 3a 20 65 |nity..- hosts: e|
00000060 64 65 6e 5f 6a 61 69 6c 73 0a 20 20 62 65 63 6f |den_jails. beco|
00000070 6d 65 3a 20 79 65 73 0a 20 20 67 61 74 68 65 72 |me: yes. gather|
00000080 5f 66 61 63 74 73 3a 20 6e 6f 0a 20 20 72 6f 6c |_facts: no. rol|
00000090 65 73 3a 0a 20 20 20 20 2d 20 64 6e 73 64 69 73 |es:. - dnsdis|
000000a0 74 0a 20 20 20 20 2d 20 65 64 65 6e 0a 20 20 20 |t. - eden. |
000000b0 20 2d 20 7a 6f 6e 65 32 6a 73 6f 6e 0a 0a 2d 20 | - zone2json..- |
000000c0 68 6f 73 74 73 3a 20 63 6f 75 63 68 64 62 5f 6a |hosts: couchdb_j|
000000d0 61 69 6c 73 0a 20 20 62 65 63 6f 6d 65 3a 20 79 |ails. become: y|
000000e0 65 73 0a 20 20 67 61 74 68 65 72 5f 66 61 63 74 |es. gather_fact|
000000f0 73 3a 20 6e 6f 0a 20 20 72 6f 6c 65 73 3a 0a 20 |s: no. roles:. |
00000100 20 20 20 2d 20 63 6f 75 63 68 64 62 0a 0a 2d 20 | - couchdb..- |
00000110 68 6f 73 74 73 3a 20 77 77 77 5f 6a 61 69 6c 73 |hosts: www_jails|
00000120 0a 20 20 62 65 63 6f 6d 65 3a 20 79 65 73 0a 20 |. become: yes. |
00000130 20 67 61 74 68 65 72 5f 66 61 63 74 73 3a 20 6e | gather_facts: n|
00000140 6f 0a 20 20 72 6f 6c 65 73 3a 0a 20 20 20 20 2d |o. roles:. -|
00000150 20 68 32 6f 0a 20 20 20 20 2d 20 77 77 77 0a 0a | h2o. - www..|
00000160 2d 20 68 6f 73 74 73 3a 20 6d 75 5f 6a 61 69 6c |- hosts: mu_jail|
00000170 73 0a 20 20 62 65 63 6f 6d 65 3a 20 79 65 73 0a |s. become: yes.|
00000180 20 20 67 61 74 68 65 72 5f 66 61 63 74 73 3a 20 | gather_facts: |
00000190 6e 6f 0a 20 20 72 6f 6c 65 73 3a 0a 20 20 20 20 |no. roles:. |
000001a0 2d 20 6d 75 0a 0a 2d 20 68 6f 73 74 73 3a 20 66 |- mu..- hosts: f|
000001b0 64 62 5f 6a 61 69 6c 73 0a 20 20 62 65 63 6f 6d |db_jails. becom|
000001c0 65 3a 20 79 65 73 0a 20 20 67 61 74 68 65 72 5f |e: yes. gather_|
000001d0 66 61 63 74 73 3a 20 6e 6f 0a 20 20 72 6f 6c 65 |facts: no. role|
000001e0 73 3a 0a 20 20 20 20 2d 20 66 64 62 0a 0a 2d 20 |s:. - fdb..- |
000001f0 68 6f 73 74 73 3a 20 6d 61 74 72 69 78 5f 6a 61 |hosts: matrix_ja|
00000200 69 6c 73 0a 20 20 62 65 63 6f 6d 65 3a 20 79 65 |ils. become: ye|
00000210 73 0a 20 20 67 61 74 68 65 72 5f 66 61 63 74 73 |s. gather_facts|
00000220 3a 20 6e 6f 0a 20 20 72 6f 6c 65 73 3a 0a 20 20 |: no. roles:. |
00000230 20 20 2d 20 64 65 6e 64 72 69 74 65 0a 0a 2d 20 | - dendrite..- |
00000240 68 6f 73 74 73 3a 20 61 76 61 6c 61 6e 63 68 65 |hosts: avalanche|
00000250 5f 6a 61 69 6c 73 0a 20 20 62 65 63 6f 6d 65 3a |_jails. become:|
00000260 20 79 65 73 0a 20 20 67 61 74 68 65 72 5f 66 61 | yes. gather_fa|
00000270 63 74 73 3a 20 6e 6f 0a 20 20 72 6f 6c 65 73 3a |cts: no. roles:|
00000280 0a 20 20 20 20 2d 20 61 76 61 6c 61 6e 63 68 65 |. - avalanche|
00000290 0a 0a 2d 20 68 6f 73 74 73 3a 20 73 65 61 77 65 |..- hosts: seawe|
000002a0 65 64 66 73 5f 6a 61 69 6c 73 0a 20 20 62 65 63 |edfs_jails. bec|
000002b0 6f 6d 65 3a 20 79 65 73 0a 20 20 67 61 74 68 65 |ome: yes. gathe|
000002c0 72 5f 66 61 63 74 73 3a 20 6e 6f 0a 20 20 72 6f |r_facts: no. ro|
000002d0 6c 65 73 3a 0a 20 20 20 20 2d 20 73 65 61 77 65 |les:. - seawe|
000002e0 65 64 66 73 0a |edfs.|
000002e5
## original issue, with "approximate facts"
- 13.0-RELEASE zpool is incompatible with 12.3-BETA because of upgraded pool,
also wrong ashift
- lets just add a new 12.3-BETA1 zpool and zfs send the bits across
- re-create the 13.0-RELEASE pool with corrected ashift
- zfs send the stuff back again
- on booting, we see that the loader is very angry and wont boot
- loader won't boot from this new pool zstd-9 compressed bootfs (since
confirmed by kevans as by design)
- when moving bootfs back to lz4 again we can boot
- all other datasets suffer from the same corruption issues, as seen from
13.0-RELEASE
## zpool history | sort
2021-10-31.00:24:03 zpool create -o failmode=continue -O atime=off -O
compression=zstd-9 -O checksum=skein -R /mnt puffy /dev/gpt/puffy
2021-10-31.00:38:25 zfs recv -Fuvs -o compression=zstd-9 -o checksum=skein -o
atime=off puffy
2021-10-31.00:41:28 zpool set bootfs=puffy/ROOT/13.0-RELEASE puffy
2021-10-31.00:46:21 zpool create -o failmode=continue -O atime=off -O
compression=zstd-9 -O checksum=skein -R /mnt zroot /dev/gpt/zfs
2021-10-31.00:56:10 zfs send -LevcPR puffy@20211031-0024
2021-10-31.00:56:16 zfs recv -Fuvs zroot
2021-10-31.00:58:53 zpool set bootfs=zroot/ROOT/13.0-RELEASE zroot
2021-10-31.00:59:02 zpool set bootfs= puffy
2021-10-31.01:00:19 zfs set mountpoint=/ zroot/ROOT/12.3
2021-10-31.01:00:25 zfs set mountpoint=/ zroot/ROOT/13.0-RELEASE
2021-10-31.01:09:19 zpool set bootfs=zroot/ROOT/12.3 zroot
2021-10-31.02:05:03 zfs set compression=lz4 zroot/ROOT
2021-10-31.02:12:58 zpool import -R /mnt -N puffy
2021-10-31.02:13:03 zpool set bootfs=puffy/ROOT/13.0-RELEASE puffy
2021-10-31.02:13:13 zpool set bootfs= zroot
2021-10-31.02:17:33 zpool import -f -R /mnt -N zroot
2021-10-31.02:19:03 zfs destroy -vrf zroot/ROOT
2021-10-31.02:24:00 zfs send -LvPR puffy/ROOT@20211031-0024
2021-10-31.02:24:03 zfs recv -Fuvs -o compression=lz4 zroot/ROOT
2021-10-31.02:24:18 zpool set bootfs= puffy
2021-10-31.02:24:23 zpool set bootfs=zroot/ROOT/13.0-RELEASE zroot
2021-10-31.08:48:14 zpool import -fR /mnt -N puffy
2021-10-31.08:48:42 zpool set bootfs=puffy/ROOT/12.3 puffy
2021-10-31.08:48:51 zpool set bootfs= zroot
2021-10-31.08:49:05 zpool set bootfs=puffy/ROOT/13.0-RELEASE puffy
2021-10-31.16:31:48 zpool import -f -R /mnt -N zroot
2021-10-31.21:20:24 zfs send -LvcPR puffy/ROOT/12.3@20211031-0024
2021-10-31.21:20:28 zfs recv -Fuvs zroot/ROOT/12.3-lvcpr
2021-10-31.21:22:44 zfs create -o canmount=off -o mountpoint=none -o
compression=zstd-9 zroot/WOOT
2021-10-31.21:23:08 zfs destroy -vrRf zroot/ROOT/12.3-lvcpr
2021-10-31.21:26:21 zfs send -LevcR puffy/ROOT/12.3@20211031-0024
2021-10-31.21:26:23 zfs recv -Fuvs zroot/WOOT/12.3-levcr
2021-10-31.21:29:13 zfs send -cR puffy/ROOT/12.3@20211031-0024
2021-10-31.21:29:14 zfs recv -Fus zroot/WOOT/12.3-vcr
2021-10-31.21:32:23 zfs send -LvcR puffy/ROOT/12.3@20211031-0024
2021-10-31.21:32:25 zfs recv -Fus zroot/WOOT/12.3-lvcr
2021-10-31.21:35:12 zfs send -evcR puffy/ROOT/12.3@20211031-0024
2021-10-31.21:35:14 zfs recv -Fus zroot/WOOT/12.3-evcr
2021-10-31.21:43:04 zfs send -vR puffy/ROOT/12.3@20211031-0024
2021-10-31.21:43:35 zfs recv -Fus zroot/WOOT/12.3-vr
2021-10-31.21:44:15 zfs rename zroot/WOOT zroot/thirteens
2021-10-31.22:02:54 zpool import -f -R /mnt -N zroot
2021-10-31.22:04:06 zfs create -o canmount=off -o mountpoint=none -o
compression=zstd-9 zroot/twelve
2021-10-31.22:07:24 zfs send -cR puffy/ROOT/12.3@20211031-0024
2021-10-31.22:07:29 zfs recv -Fus zroot/twelve/12.3-vcr
2021-10-31.22:10:04 zfs send -LevcR puffy/ROOT/12.3@20211031-0024
2021-10-31.22:10:09 zfs recv -Fuvs zroot/twelve/12.3-levcr
2021-10-31.22:12:43 zfs send -LvcR puffy/ROOT/12.3@20211031-0024
2021-10-31.22:12:49 zfs recv -Fuvs zroot/twelve/12.3-lvcr
2021-10-31.22:15:23 zfs send -evcR puffy/ROOT/12.3@20211031-0024
2021-10-31.22:15:28 zfs recv -Fus zroot/twelve/12.3-evcr
2021-10-31.22:37:18 zfs rename zroot/thirteens zroot/thirteen
2021-10-31.22:40:54 zfs send -vR puffy/ROOT/12.3@20211031-0024
2021-10-31.22:41:52 zfs recv -Fus zroot/twelve/12.3-vr
2021-11-08.08:14:57 zpool import -f -R /mnt -N zroot
2021-11-08.08:37:09 zfs destroy -vrRf zroot/ROOT/12.3
2021-11-08.08:37:28 zfs snapshot -r puffy/ROOT@fixit
2021-11-11.09:54:15 zpool import -f -R /mnt -N zroot
2021-11-11.09:54:47 zfs snapshot -r puffy@20211111-0954
2021-11-11.09:54:47 zfs snapshot -r zroot@20211111-0954
2021-11-11.09:59:42 zfs destroy -vrRf zroot/thirteen
2021-11-11.10:00:12 zfs recv -Fuvs zroot/12.3-RELEASE
2021-11-11.10:00:15 zfs send -LvcR puffy/ROOT/12.3@20211111-0954
2021-11-11.10:00:24 zfs destroy -vrRf zroot/twelve
2021-11-11.10:00:54 zfs rename zroot/12.3-RELEASE zroot/ROOT/12.3-RELEASE
2021-11-11.10:01:35 zpool set bootfs=zroot/ROOT/12.3-RELEASE zroot
2021-11-11.10:01:42 zpool set bootfs= puffy
2021-11-11.10:04:51 zfs destroy -vrf zroot/usr/home/dch
2021-11-11.10:05:35 zfs send -LvcR puffy/usr/home/dch@20211111-0954
2021-11-11.10:05:37 zfs recv -Fuvs zroot/usr/home/dch
2021-11-11.10:06:39 zfs set compression=lz4 zroot
2021-11-11.11:46:45 zpool import -R /mnt -N -f zroot
2021-11-11.11:47:32 zfs destroy -vr zroot@fixit
2021-11-11.11:47:35 zfs snapshot -r zroot@fixit
2021-11-11.12:07:49 zpool import -R /mnt2 -N -f puffy
2021-11-11.12:21:42 zfs rename zroot/ROOT zroot/WOOT
2021-11-11.12:22:01 zfs destroy -r puffy@fixit
2021-11-11.12:22:09 zfs snapshot -r puffy@fixit
2021-11-11.12:26:52 zfs send -LevR puffy/ROOT@fixit
2021-11-11.12:26:58 zfs recv -Fuvs -o compression=lz4 zroot/ROOT
2021-11-11.12:27:33 zpool set bootfs=zroot/ROOT/13.0-RELEASE zroot
2021-11-11.12:27:41 zpool set bootfs= puffy
2021-11-11.12:27:53 zpool export puffy
2021-11-11.12:27:56 zpool export zroot
2021-11-11.12:33:05 zfs destroy -vr zroot/WOOT
2021-11-11.12:38:45 zpool import -R /mnt -N puffy
2021-11-11.12:52:47 zfs destroy -vr puffy/usr/home@fixit
2021-11-11.12:53:00 zfs snapshot -r puffy/usr/home@fixit
2021-11-11.12:53:48 zfs rename zroot/usr/home/dch zroot/usr/home/borked
2021-11-11.12:54:41 zfs recv -o compression=lz4 -Fuvs zroot/usr/home/dch
2021-11-11.12:54:41 zfs send -vR puffy/usr/home/dch@fixit
...
--
You are receiving this mail because:
You are the assignee for the bug.
