On 2020-08-03 16:17, deloptes wrote:
any thoughts on using deduplication? For example I started using borg some
time ago. It saves a lot of space and makes it possible to have multiple
backups and longer retention.
ZFS supports de-duplication, but the documents warn about enabling it.
So, of course I enabled de-duplication on my ZFS SOHO file server. ;-)
Everything was groovy when utilization was ~30%, but performance for
bulk writes degraded precipitously as the pool filled. This includes
backup replication jobs. I am fairly certain de-duplication is a major
contributing factor. The only way to test this hypothesis is to create
a fresh pool using similar hardware, replicate the data without
de-duplication, and benchmark.
jdupes looks interesting, and should work on any file system that
supports hard links. I expect BorgBackup either calls jdupes or
implements similar functionality:
https://linuxcommandlibrary.com/man/jdupes.html
David