I have a quite weird problem where at least one object ends up losing its data
after a replication stream is produced and received. I noticed the problem
because the object in question is a directory / ZAP object, dn_type =
DMU_OT_DIRECTORY_CONTENTS, and having no data for such an object utterly
confuses the ZAP code (=> crash).
There is a twist here: if I send a regular, non-replication stream for the
filesystem in question, then the object is received as an exact copy. It's only
when sending a replication stream that the problem occurs. I produced and
received both kinds of streams several times to make sure that the problem was
not caused by a glitch. The filesystem in question, as a ZFS dataset, has
several children, all of them filesystems. The filesystem and the children have
some snapshots and a few clones.
I am trying to analyze the object and the streams.
This is what I have so far.
The original object:
$ zdb -dddddd pond/ROOT/20131121 63
Dataset pond/ROOT/20131121 [ZPL], ID 1939, cr_txg 2411178437, 1.63G, 14163
objects, rootbp DVA[0]=<0:36935ba00:200> DVA[1]=<0:50062b4800:200> [L0 DMU
objset] fletcher4 lzjb LE contiguous unique double size=800L/200P
birth=2412628207L/2412628207P fill=14163
cksum=1ad2870487:81c3cde6df6:15e1bd6a2562c:2ac4d7d00e78d6
Object lvl iblk dblk dsize lsize %full type
63 1 16K 512 1K 512 100.00 ZFS directory (K=inherit)
(Z=inherit)
192 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
path /var/empty
uid 0
gid 0
atime Wed Jul 29 21:12:21 2009
mtime Tue May 26 15:29:41 2009
ctime Thu Nov 21 22:24:21 2013
crtime Tue May 26 15:29:41 2009
gen 2642
mode 40555
size 2
parent 4
links 2
pflags 40000000044
microzap: 512 bytes, 0 entries
Indirect blocks:
0 L0 0:f2800:200 0:16000f2c00:200 200L/200P F=1 B=2642/2642
segment [0000000000000000, 0000000000000200) size 512
$ zdb -R pond 0:f2800:200
Found vdev type: mirror
0:f2800:200
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
000000: 8000000000000003 000000003d096e2f ......../n.=....
000010: 0000000000000000 0000000000000000 ................
....
The simple stream generated by zfs send has the following for this object:
BEGIN record
hdrtype = 1
features = 4
magic = 2f5bacbac
creation_time = 52f9e387
type = 2
flags = 0x0
toguid = 45e035ffa3168cb0
fromguid = 0
toname = pond/ROOT/20131121@migration-1
....
OBJECT object = 63 type = 20 bonustype = 44 blksz = 512 bonuslen = 192
....
WRITE object = 63 type = 20 checksum type = 7
offset = 0 length = 512 props = 200000000
The replication stream generated with -R has a bit more entries (I reproduce
only the ones which seem relevant to me):
BEGIN record
hdrtype = 1
features = 4
magic = 2f5bacbac
creation_time = 5278fd05
type = 2
flags = 0x0
toguid = 7c29eee2603a466c
fromguid = 0
toname = pond/ROOT/20131121@20131105
OBJECT object = 63 type = 20 bonustype = 17 blksz = 512 bonuslen = 264
WRITE object = 63 type = 20 checksum type = 7
offset = 0 length = 512 props = 200000000
BEGIN record
hdrtype = 1
features = 4
magic = 2f5bacbac
creation_time = 528e65ce
type = 2
flags = 0x0
toguid = 25fe11138ae715f3
fromguid = 7c29eee2603a466c
toname = pond/ROOT/20131121@20131121
OBJECT object = 63 type = 20 bonustype = 17 blksz = 512 bonuslen = 264
FREE object = 63 offset = 512 length = -1
BEGIN record
hdrtype = 1
features = 4
magic = 2f5bacbac
creation_time = 5290daf5
type = 2
flags = 0x0
toguid = 7b35bb45616fd542
fromguid = 25fe11138ae715f3
toname = pond/ROOT/20131121@drop
OBJECT object = 63 type = 20 bonustype = 44 blksz = 512 bonuslen = 192
FREE object = 63 offset = 512 length = -1
BEGIN record
hdrtype = 1
features = 4
magic = 2f5bacbac
creation_time = 529347d5
type = 2
flags = 0x0
toguid = c84ec632845b12e2
fromguid = 7b35bb45616fd542
toname = pond/ROOT/20131121@kms
OBJECT object = 63 type = 20 bonustype = 44 blksz = 512 bonuslen = 192
FREE object = 63 offset = 512 length = -1
[skipping several sub-streams with the same pattern as the previous two]
BEGIN record
hdrtype = 1
features = 4
magic = 2f5bacbac
creation_time = 52f9e387
type = 2
flags = 0x0
toguid = 45e035ffa3168cb0
fromguid = 3a853e7894a0dd77
toname = pond/ROOT/20131121@migration-1
A few notable things:
- the first snapshot stream includes data for the object
- initial bonustype is 17 DMU_OT_ZNODE unlike the latest bonustype of 44
DMU_OT_SA
- all intermediate snapshot streams have only FREE records for the object
- the last snapshot stream has no record for the object at all
The filesystem in question has a very long history and I suspect that there
might be a bug related to DMU_OT_ZNODE -> DMU_OT_SA upgrade. On the other hand,
I do not preclude any sort of local problem.
I will appreciate any insights into the issue.
Many thanks in advance!
P.S.
Couple more zdb dumps to double-check the information from the stream dump:
$ zdb -dddddd pond/ROOT/20131121@20131105 63
Dataset pond/ROOT/20131121@20131105 [ZPL], ID 30163, cr_txg 2410888055, 1.07G,
14116 objects, rootbp DVA[0]=<0:6559910800:200> DVA[1]=<0:919e94e00:200> [L0 DMU
objset] fletcher4 lzjb LE contiguous unique double size=800L/200P
birth=2410888026L/2410888026P fill=14116
cksum=1a90c40182:8ae92339048:18a97b855161c:31d1b7f6d31b5a
Object lvl iblk dblk dsize lsize %full type
63 1 16K 512 1K 512 100.00 ZFS directory (K=inherit)
(Z=inherit)
264 bonus ZFS znode
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
path /var/empty
uid 0
gid 0
atime Wed Jul 29 21:12:21 2009
mtime Tue May 26 15:29:41 2009
ctime Sun Aug 23 20:09:30 2009
crtime Tue May 26 15:29:41 2009
gen 2642
mode 40555
size 2
parent 4
links 2
pflags 41800000044
xattr 0
rdev 0x0000000000000000
microzap: 512 bytes, 0 entries
Indirect blocks:
0 L0 0:f2800:200 0:16000f2c00:200 200L/200P F=1 B=2642/2642
segment [0000000000000000, 0000000000000200) size 512
$ zdb -dddddd pond/ROOT/20131121@migration-1 63
Dataset pond/ROOT/20131121@migration-1 [ZPL], ID 6169, cr_txg 2412590450, 1.41G,
13868 objects, rootbp DVA[0]=<0:371907c00:200> DVA[1]=<0:156f54b600:200> [L0 DMU
objset] fletcher4 lzjb LE contiguous unique double size=800L/200P
birth=2412590445L/2412590445P fill=13868
cksum=1b0a69d70c:86e99190f2c:1771ba6229459:2f212635c23126
Object lvl iblk dblk dsize lsize %full type
63 1 16K 512 1K 512 100.00 ZFS directory (K=inherit)
(Z=inherit)
192 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
path /var/empty
uid 0
gid 0
atime Wed Jul 29 21:12:21 2009
mtime Tue May 26 15:29:41 2009
ctime Thu Nov 21 22:24:21 2013
crtime Tue May 26 15:29:41 2009
gen 2642
mode 40555
size 2
parent 4
links 2
pflags 40000000044
microzap: 512 bytes, 0 entries
Indirect blocks:
0 L0 0:f2800:200 0:16000f2c00:200 200L/200P F=1 B=2642/2642
segment [0000000000000000, 0000000000000200) size 512
--
Andriy Gapon
_______________________________________________
developer mailing list
[email protected]
http://lists.open-zfs.org/mailman/listinfo/developer
