Thanks so much for the thoughtful response. The extra time to re-encrypt
everything locally is not a big problem for me, so I will keep in mind your
workaround should I find my fuse-based solution unworkable. I do agree
with your assessment of many of the backup solutions that perform snapshots
and only transfer those snapshots off site.
As for my needs, I started looking at encfs, and ended up with gocryptfs.
Both are FUSE based tools and both offer a "reverse" mode which makes it
possible to create an encrypted file-by-file view of my unencrypted backup
directory to feed into rsync to replicate offsite. Thus, I don't have any
issue with snapshots and infinite growth, it is a rather direct drop-in
replacement for rsyncrypto. The only disadvantage I see to gocryptfs is
that a change to any file will likely result in the whole file being copied
during rsync because of the encryption algorithm used. However, for files
where only a small part of them would change they tend to be small enough
that this should be rather minor. Larger files in my particular use are
almost always static.
Another potential advantage of gocryptfs is I can create an unencrypted
view of the backup to recover individual files: sshfs mount the offsite
backup drive, then gocryptfs (in non-reverse mode) to create a unencrypted
view of the encrypted backup. I would do this for a full recovery, but it
could come in handy to access a single file.
I say these things not to poo-poo on rsyncrypto, it is a great project and
I used it successfully for many years. But instead to suggest that the use
of FUSE to create encrypted / non-encrypted views of source data is, I
think, superior to taking a file and producing an encrypted/unencrypted
copy of the file and in-line with the Unix philosophy of single-purpose
tools. If I were to put work into a new release of rsyncrypto I would move
in the direction of a FUSE approach like encfs and gocryptfs but improve by
using the rsync-friendly encryption techniques that makes rsyncrypto shine.
Best,
David Diepenbrock
On Sat, Nov 9, 2019 at 8:00 AM Shachar Shemesh wrote:
>
> On 08/11/2019 19:00, David Diepenbrock wrote:
>
> I may have discovered another bug, and I apologize in advance for not
> digging into it more, and for polluting this thread. When I ran on top of
> my rather old encrypted copy, after compiling with the delete bug fixed, I
> noticed that the size was *significantly* larger than it should be, on the
> order of double or more what I was expecting. Now I had cleaned out the
> source dir quite a bit, so I suspect that not everything was properly
> deleted. A quick check at the file counts showed the encrypted directory
> had nearly 4x the file count of the source dir.
>
> That may be a bug, indeed. I'll try to have a look at it.
>
>
> With that said, please note that there is a very easy work around. Just
> delete all of the encrypted files, leaving only the key files (which are
> very small), and then re-encrypt your data. You will have just the relevant
> files, and they will still be rsyncable.
>
>
> Unfortunately, this made it a no-go for me to resume using rsyncrypto.
> For now I've migrated to using a fuse encrypted filesystem
> instead (gocryptfs in my case, since I needed something with reverse mode
> for feeding into rsync). With the fuse option it means I don't need to
> preserve an encrypted copy of the data on disk, which rather significantly
> outweighs any of the drawbacks. As such, I can't see myself moving back to
> rsyncrypto anytime soon, unless someone can point out something I might be
> missing?
>
> Here's what my understanding of how Fuse based solutions work. The file
> system keeps track over what changed, encrypts the delta, and sends it
> over. This keeps the changes small and gives you, in many cases, bandwidth
> efficiency similar to rsyncrypto[1].
>
>
> Here's this system's down side. Personally, I find this down side so bad
> as to make the system unworkable, but most people don't seem to care: You
> can never free space on the back up storage device. Every single
> intermediate backup is potentially crucial for correct restore. There are
> two ways, and only two ways, to keep your system up to date. Either resign
> yourself to your remote backup folder getting bigger and bigger as time
> moves on, or periodically re-sync the whole data set. For a system designed
> to keep bandwidth low, I find it unacceptable.
>
>
> When I say this to people, the common answer is that neither bandwidth nor
> storage are that expensive these days[2]. I find this answer near-sighted,
> as it only views the "backup" side of the equation. There is a very serious
> "restore" side to consider.
>
>
> Encrypted backup should, ideally, have just one "single point of failure"
> point, which is the encryption key (for rsyncrypto, even that's not true:
> you store the symmetric key locally in the key files, and you have your RSA
> master key). Fuse based solutions have
