Sigbjorn Stop! Don't do it... it's a waste of time. We tried exactly what you're thinking of... we bought two Sun/Oracle 7000 series storage units with 20TB of ZFS storage each planning to use them as a backup target for Networker. We ran into several issues eventually gave up the ZFS networker combo. We've used other storage devices in the past (virtual tape libraries) that had deduplication. We were used to seeing dedup ratios better than 20x on our backup data. The ZFS filesystem only gave us 1.03x, and it had regular issues because it couldn't do dedup for such large filesystems very easily. We didn't know it ahead of time, but VTL solutions use something called "variable length" block dedup, whereas ZFS uses "fixed block" length dedup. Like one of the other posters mentioned, things just don't line up right and the dedup ratio suffers. Yes, compression works to some degree -- I think we got 2 or 3x on that, but it was a far cry from the 20x that we were used to seeing on our old VTL.
We recently ditched the 7000 series boxes in favor of a much pricier competitor. It's about double the cost, but dedup ratios are better than 20x. Personally I love ZFS and I use it in many other places, but we were very disappointed with the dedup ability for that type of data. We went to Sun with our problems and they ran it up the food chain and word came back down from the developers that this was the way it was designed, and it's not going to change anytime soon. The type of files that Networker writes out just are not friendly at all with the dedup mechanism used in ZFS. They gave us a few ideas and things to tweak in Networker, but no measurable gains ever came from any of the tweaks. If are considering a home-grown ZFS solution for budget reasons, go for it.... just do yourself a favor and save yourself the overhead of "trying" to dedup. When we disabled dedup on our 7000 series boxes, everything worked great and compression was fine with next to no overhead. Unfortunately, we NEEDED at least a 10x ratio to keep the 3 week backups we were trying to do. We couldn't even keep a 1 week backup with the dedup performance of ZFS. If you need more details, I'm happy to help. We went through months of pain trying to make it work and it just doesn't for Networker data. best wishes Daniel 2010/8/18 Sigbjorn Lie <sigbj...@nixtra.com>: > Hi, > > We are considering using a ZFS based storage as a staging disk for Networker. > We're aiming at > providing enough storage to be able to keep 3 months worth of backups on > disk, before it's moved > to tape. > > To provide storage for 3 months of backups, we want to utilize the dedup > functionality in ZFS. > > I've searched around for these topics and found no success stories, however > those who has tried > did not mention if they had attempted to change the blocksize to any smaller > than the default of > 128k. > > Does anyone have any experience with this kind of setup? > > > Regards, > Sigbjorn > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
