Hi there,
On Tue, 2 Apr 2019 Bzzzz wrote:
On Tue, 2 Apr 2019, Stefan Schumacher wrote:
I want to set up a new zfs volume as storage for Backuppc. I plan on
using the zfs features encryption, deduplication and compression.
Unless you have an _absolute_ need, do NOT use deduplication into ZFS
and if you persist, do not take carelessly the 5 GB/TB _lower_ limit
than is given in docs for dedup - if your files are plenty and small,
you hit the jackpot and can raise this limit to 10 GB/TB.
Not to mention the loss of performance, especially in writing.
You will need to test the performance yourself. Performance can be
improved by avoiding disc writes, which will take orders of magnitude
longer than reading RAM. ZFS checksums are in RAM, so you might need
a lot of it. ZFS deduplication takes place at disc block level, not
at file level, so if you have for example files which grow from backup
to backup where the first parts of files are identical, then you might
see performance improvements from _both_ kinds of deduplication. It
will obviously depend on your data profile, and it may also depend on
encryption; I have no idea what impact that might have for example on
deduplication of files which have identical blocks before encryption.
I'd expect any sensible encryption system to use something like salts,
so that blocks stored on disc would be different after encryption even
if they were identical before it. Otherwise, interesting attacks on
the encrypted data can become possible. There's a lot of literature.
In any event, in my view, the stability of the filesystem is a much
more important consideration. I should be reluctant to move any of my
backups from ext4 to ZFS simply because I have very little information
about ZFS to work with and (call it my disclaimer) I have no personal
experience of it at all. Certainly using the ZFS encryption feature
would for me be a risk too far.
On Tue, 2 Apr 2019, Ray Frush wrote:
For a backupPC pool, which is already significantly deduplicated
(via the hash pool), deduplication probably won?t buy you as much as
you?d hope. We recently moved to ZFS backed storage and rely on
using ZFS?s compression instead of BackupPC. You don?t want to do
both as you?ll take a small penalty on writes as LZ4 has to decide
not to compress your blocks.
All agreed, but nothing can beat _methodically_ testing it in your own
situation with your own data.
On Tue, 2 Apr 2019, Stefan Schumacher wrote:
1) Am I correct in assuming that I should disable pooling and
compression in Backuppc?
"Assume" makes an ass out of u and me. :)
Perhaps disable compression first on one, then the other, then both,
and test for performance. Report your results here. That would be a
very useful service to the community. As others have said, you cannot
disable pooling. You probably would not want to, even if you could.
Statistics on storage improvement using ZFS deduplication both for ZFS
encrypted and unencrypted files would also be useful. Any information
concerning (in)stability would be gratefully received I'm sure, but we
will need to be talking about years of operation, not days or weeks.
Many of us have been running ext[234] filesystems for several decades,
with no stability issues at all. I can remember trying ReiserFS about
a decade ago, they said it was a big step forward. It was a disaster.
--
73,
Ged.
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
