On Fri, Feb 14, 2014 at 2:20 PM, Andriy Gapon <[email protected]> wrote:
> > I have a quite weird problem where at least one object ends up losing its > data > after a replication stream is produced and received. I noticed the problem > because the object in question is a directory / ZAP object, dn_type = > DMU_OT_DIRECTORY_CONTENTS, and having no data for such an object utterly > confuses the ZAP code (=> crash). > > There is a twist here: if I send a regular, non-replication stream for the > filesystem in question, then the object is received as an exact copy. > It's only > when sending a replication stream that the problem occurs. I produced and > received both kinds of streams several times to make sure that the problem > was > not caused by a glitch. The filesystem in question, as a ZFS dataset, has > several children, all of them filesystems. The filesystem and the > children have > some snapshots and a few clones. > > I am trying to analyze the object and the streams. > > > ... > A few notable things: > - the first snapshot stream includes data for the object > - initial bonustype is 17 DMU_OT_ZNODE unlike the latest bonustype of 44 > DMU_OT_SA > - all intermediate snapshot streams have only FREE records for the object > - the last snapshot stream has no record for the object at all > > > The filesystem in question has a very long history and I suspect that there > might be a bug related to DMU_OT_ZNODE -> DMU_OT_SA upgrade. On the other > hand, > I do not preclude any sort of local problem. > I think the problem is in fact a local (HybridCluster-only) problem with ZNODE->SA upgrade (i.e. ZPL upgrade from <=v4 to v5). The patch we discussed in the thread "restore_object vs dnode_sync" starting April 1, 2013, would cause this problem. In stock OpenZFS, dmu_object_reclaim() only frees the object's contents if the block size or nblkptrs changed, but your patch frees the file contents if the bonustype changes, which is exactly the case you've identified here (bonustype change from ZNODE -> SA). --matt > I will appreciate any insights into the issue. > Many thanks in advance! > P.S. > Couple more zdb dumps to double-check the information from the stream dump: > > $ zdb -dddddd pond/ROOT/20131121@20131105 63 > Dataset pond/ROOT/20131121@20131105 [ZPL], ID 30163, cr_txg 2410888055, > 1.07G, > 14116 objects, rootbp DVA[0]=<0:6559910800:200> DVA[1]=<0:919e94e00:200> > [L0 DMU > objset] fletcher4 lzjb LE contiguous unique double size=800L/200P > birth=2410888026L/2410888026P fill=14116 > cksum=1a90c40182:8ae92339048:18a97b855161c:31d1b7f6d31b5a > > Object lvl iblk dblk dsize lsize %full type > 63 1 16K 512 1K 512 100.00 ZFS directory > (K=inherit) > (Z=inherit) > 264 bonus ZFS znode > dnode flags: USED_BYTES USERUSED_ACCOUNTED > dnode maxblkid: 0 > path /var/empty > uid 0 > gid 0 > atime Wed Jul 29 21:12:21 2009 > mtime Tue May 26 15:29:41 2009 > ctime Sun Aug 23 20:09:30 2009 > crtime Tue May 26 15:29:41 2009 > gen 2642 > mode 40555 > size 2 > parent 4 > links 2 > pflags 41800000044 > xattr 0 > rdev 0x0000000000000000 > microzap: 512 bytes, 0 entries > > Indirect blocks: > 0 L0 0:f2800:200 0:16000f2c00:200 200L/200P F=1 B=2642/2642 > > segment [0000000000000000, 0000000000000200) size 512 > > $ zdb -dddddd pond/ROOT/20131121@migration-1 63 > Dataset pond/ROOT/20131121@migration-1 [ZPL], ID 6169, cr_txg 2412590450, > 1.41G, > 13868 objects, rootbp DVA[0]=<0:371907c00:200> DVA[1]=<0:156f54b600:200> > [L0 DMU > objset] fletcher4 lzjb LE contiguous unique double size=800L/200P > birth=2412590445L/2412590445P fill=13868 > cksum=1b0a69d70c:86e99190f2c:1771ba6229459:2f212635c23126 > > Object lvl iblk dblk dsize lsize %full type > 63 1 16K 512 1K 512 100.00 ZFS directory > (K=inherit) > (Z=inherit) > 192 bonus System attributes > dnode flags: USED_BYTES USERUSED_ACCOUNTED > dnode maxblkid: 0 > path /var/empty > uid 0 > gid 0 > atime Wed Jul 29 21:12:21 2009 > mtime Tue May 26 15:29:41 2009 > ctime Thu Nov 21 22:24:21 2013 > crtime Tue May 26 15:29:41 2009 > gen 2642 > mode 40555 > size 2 > parent 4 > links 2 > pflags 40000000044 > microzap: 512 bytes, 0 entries > > Indirect blocks: > 0 L0 0:f2800:200 0:16000f2c00:200 200L/200P F=1 B=2642/2642 > > segment [0000000000000000, 0000000000000200) size 512 > -- > Andriy Gapon > > _______________________________________________ > developer mailing list > [email protected] > http://lists.open-zfs.org/mailman/listinfo/developer > >
_______________________________________________ developer mailing list [email protected] http://lists.open-zfs.org/mailman/listinfo/developer
