Hello Michael, Thank you kindly for your thorough reply, it helped me to make sense of what was going on with the snapshots, indeed, once I reduced the daily retention time to 7d for snapshots, I was able to get rid of those old ones with hammer cleanup !
I also learned about being able to have different retention time for the daily PFS slave snapshot which is neat. Apologies for the borked output regarding hammer info. As it stands now after running prune-everything on all PFS's, I still have a noticable size difference, I'll try to investigate some more and maybe leave some time to hammer mirror-stream to even things out. Cheers, Laurent -----Original message----- From:Michael Neumann <mneum...@ntecs.de> Sent:Mon 08-27-2018 10:19 pm Subject:Re: A few questions about HAMMER master/slave PFS To:Laurent Vivier <laur...@lamzi.com>; CC:users@dragonflybsd.org; On Mon, Aug 27, 2018 at 06:08:10PM +0200, Laurent Vivier wrote: > Hello DFlyers, > > I am running DragonFly 5.2.2 as an NFS server with 2x 2TB LUKS-backed HDD's > with HAMMER1 v7 as FS in a PFS master/slave mirror-stream setup and it's been > working great so far :) > > The setup looks like this : > > Disk1 -> LUKS -> HAMMER1_2TB -> PFS# 0 (root) > Disk2 -> LUKS -> HAMMER_SLAVE -> PFS# 0 (root) + PFS 1 (slave to HAMMER1_2TB > / PFS# 0) > > Now that I am using the system for a little while, I have a few questions > regarding its behavior : > > 1) I realized that the HAMMER slave PFS has several snapshots (not created by > me) that seems seemingly impossible to remove e.g HAMMER1 uses fine-grained snapshots, which means that it basically automatically creates an "unnamed" snapshot whenever it flushes something to disk (roughly every 30 seconds). Usually, you don't want to keep all these fine-grained snapshots and instead keep one snapshot per day (or one per week...). This is what "hammer cleanup" does. You can configure it's history retention policy by running "hammer viconfig". From the man page of "hammer cleanup": snapshots 1d 60d # 0d 0d for PFS /tmp, /var/tmp, /usr/obj prune 1d 5m rebalance 1d 5m #dedup 1d 5m # not enabled by default reblock 1d 5m recopy 30d 10m This means, when you run "hammer cleanup", it takes one snapshot every day, and retains the last 60 daily snapshots. hammer cleanup performs other tasks, for instance pruning (1d = every day, for 5 minutes). Pruning deletes all the intermediate fine-grained snapshots between the "named" daily snapshots. It also rebalances the B-tree, dedups, reblocks and recopies. These are all operations to optimize performance. Dedup is to save space. If you want to delete snapshots, just change "snapshots 1d 60d" to, for instance, "snapshots 1d 7d" and run "hammer cleanup". If you want to delete all historical data, you might use "hammer prune-everything", but be careful and read the man page!!! One nice feature of HAMMER1 is that the master PFS and slave PFS can have different history retention policies in place. > > Hikaeme# hammer info /HAMMER_SLAVE > Volume identification > ?????? Label???????????????????????????? hammer1_secure_slave > ?????? No. Volumes???????????????? 1 > ?????? HAMMER Volumes?????????? /dev/mapper/knox2 > ?????? Root Volume???????????????? /dev/mapper/knox2 > ?????? FSID?????????????????????????????? 0198767f-7139-11e8-9608-6d626d258b95 > ?????? HAMMER Version?????????? 7 > Big-block information > ?????? Total?????????????????? 238335 > ?????? Used???????????????????? 192009 (80.56%) > ?????? Reserved???????????????????? 32 (0.01%) > ?????? Free?????????????????????? 46294 (19.42%) > Space information > ?????? No. Inodes?????????? 35668 > ?????? Total size???????????? 1.8T (1999298887680 bytes) > ?????? Used???????????????????????? 1.5T (80.56%) > ?????? Reserved???????????????? 256M (0.01%) > ?????? Free???????????????????????? 362G (19.42%) > PFS information > ?????? ?? PFS#?? Mode?????? Snaps > ?????? ???????? 0?? MASTER?????????? 0 (root PFS) > ?????? ???????? 1?? SLAVE???????????? 3 > Hikaeme# hammer snapls /HAMMER_SLAVE/pfs/hanma > Snapshots on /HAMMER_SLAVE/pfs/hanma?????? PFS#1 > Transaction ID?????? ?????? Timestamp?????? ?????? Note > 0x00000001034045c0?????? 2018-07-04 18:19:42 CEST?????? - > 0x00000001034406c0?????? 2018-07-09 19:28:04 CEST?????? - > 0x000000010383bc30?????? 2018-08-12 10:51:07 CEST?????? - > Hikaeme# hammer snaprm 0x00000001034045c0 > hammer: hammer snaprm 0x00000001034045c0: Operation not supported Have you tried hammer snaprm /HAMMER_SLAVE/pfs/hanma@@0x00000001034045c0 > My question here is should I worry about it/is that an intended behavior ? > > 2) When executing hammer info and looking at the used space between master > and slave PFS, I have quite a big difference (22GB, even after running hammer > cleanup) > > Hikaeme# hammer info > Volume identification > ?????? Label???????????????????????????? HAMMER1_2TB > ?????? No. Volumes???????????????? 1 > ?????? HAMMER Volumes?????????? /dev/mapper/knox > ?????? Root Volume???????????????? /dev/mapper/knox > ?????? FSID?????????????????????????????? 81e9d5eb-6be7-11e8-802d-6d626d258b95 > ?????? HAMMER Version?????????? 7 > Big-block information > ?????? Total?????????????????? 238335 > ?????? Used???????????????????? 194636 (81.66%) > ?????? Reserved???????????????????? 32 (0.01%) > ?????? Free?????????????????????? 43667 (18.32%) > Space information > ?????? No. Inodes?????????? 35665 > ?????? Total size???????????? 1.8T (1999298887680 bytes) > ?????? Used???????????????????????? 1.5T (81.66%) > ?????? Reserved???????????????? 256M (0.01%) > ?????? Free???????????????????????? 341G (18.32%) > PFS information > ?????? ?? PFS#?? Mode?????? Snaps > ?????? ???????? 0?? MASTER?????????? 0 (root PFS) > > Volume identification > ?????? Label???????????????????????????? hammer1_secure_slave > ?????? No. Volumes???????????????? 1 > ?????? HAMMER Volumes?????????? /dev/mapper/knox2 > ?????? Root Volume???????????????? /dev/mapper/knox2 > ?????? FSID?????????????????????????????? 0198767f-7139-11e8-9608-6d626d258b95 > ?????? HAMMER Version?????????? 7 > Big-block information > ?????? Total?????????????????? 238335 > ?????? Used???????????????????? 191903 (80.52%) > ?????? Reserved???????????????????? 32 (0.01%) > ?????? Free?????????????????????? 46400 (19.47%) > Space information > ?????? No. Inodes?????????? 35668 > ?????? Total size???????????? 1.8T (1999298887680 bytes) > ?????? Used???????????????????????? 1.5T (80.52%) > ?????? Reserved???????????????? 256M (0.01%) > ?????? Free???????????????????????? 363G (19.47%) > PFS information > ?????? ?? PFS#?? Mode?????? Snaps > ?????? ???????? 0?? MASTER?????????? 0 (root PFS) > ?????? ???????? 1?? SLAVE???????????? 3 > > Is that something I should be worried about too ? As far as I can tell the > replication of new files from master and slave works great, I can see the new > files on the slave PFS fairly quickly. It's hard to read your above output, as there are too many "???". Note that HAMMER1 mirroring does not operate on the block level, but on the logical level. So both master and slave PFSs are separate file systems and their underlying disk blocks are differently organized. If you have different history retention policies on slave and master, this might lead to different file system sizes. This is not an issue. Just configure it using "hammer viconfig" and run "hammer cleanup". Best regards, Michael