Re: [BackupPC-users] Necro'd: Use S3 buckets for the pool

[Marcelo Ricardo Leitner]

Hi,

Maybe a solution composed by backuppc + rclone is the way forward.
I never used rclone myself, but AFAICT it handles all the issues
you raised, plus more:

>From their page: "Virtual backends wrap local and cloud file systems
to apply encryption, caching, chunking and joining."

https://rclone.org/

Been willing to try it, but didn't have the chance so far.

Best regards,
Marcelo

On Tue, Jul 21, 2020 at 10:42:30AM +0300, Johan Ehnberg wrote:
> Hi Kris,
> 
> Indeed the object storage transformation has not been a hot topic for some
> time. I explored this some years ago and did some proof of concept testing.
> It requires a fairly different architecture:
> 
> https://molnix.com/proposal-new-open-source-backup-solution/
> 
> Essentially, the object storage to fuse approach is not workable as-is from
> a performance standpoint, at least in almost any production scenario I can
> imagine.
> 
> The compute node needs to independently do at least the checksum matching,
> storage buffering or tiering, and object storage rate limiting.
> 
> With the fuse approach you can get pretty close to that by putting only pool
> or cpool on fuse, and keeping pc locally, added with some configuration
> tweaks. However, the object storage needs to be asynchronous to the backup
> for it to really make sense.
> 
> I believe the efforts to create FS-to-object layers to transform existing
> software to cloud concepts without actual code changes quieted down exactly
> because of these types of issues.
> 
> Best regards,
> 
> Johan
> 
> 
> On 21/07/2020 02.37, Kris Lou via BackupPC-users wrote:
> > This hasn't been addressed for a while, and I didn't find anything in
> > recent archives.
> > 
> > Anybody have any experience or hypothetical issues with writing the BPC4
> > Pool over s3fs-fuse to S3 or something similar?  Pros, Cons?
> > 
> > Thanks,
> > -Kris
> > 
> > 
> > 
> > Kris Lou
> > k...@themusiclink.net 
> > 
> > 
> > ___
> > BackupPC-users mailing list
> > BackupPC-users@lists.sourceforge.net
> > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> > Wiki:http://backuppc.wiki.sourceforge.net
> > Project: http://backuppc.sourceforge.net/
> -- 
> Signature
> *Johan Ehnberg*
> 
> Founder, CEO
> 
> Molnix Oy
> 
> 
> jo...@molnix.com 
> 
> +358 50 320 96 88
> 
> molnix.com 
> 
> 
> /The contents of this e-mail and its attachments are for the use of the
> intended recipient only, and are confidential and may contain legally
> privileged information. If you are not the intended recipient or have
> otherwise received the e-mail in error, please notify the sender by replying
> to this e-mail immediately and then delete it immediately from your system.
> Any dissemination, distribution, copying or use of this communication
> without prior and explicit permission of the sender is strictly prohibited./
> 
> /*Please consider the environment - do not print this e-mail unless you
> really need to.*/
> 


> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/



___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Large rsyncTmp files

[Marcelo Ricardo Leitner]

On Fri, May 01, 2020 at 10:17:19AM -0700, Craig Barratt wrote:
> It could also be a sparse file (eg, below /proc or /var/log/wtmp) that
> isn't being excluded.

Spot on!

[root@linux ~]# ls -lahs /var/log/lastlog
52K -rw-r--r--. 1 root root 409G Mai  1 15:11 /var/log/lastlog

Thanks folks.


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

[BackupPC-users] Large rsyncTmp files

[Marcelo Ricardo Leitner]

Hi,

Is it expected for rsync-bpc to be writting such large temporary files?
It seems they are as big as the full backup itself:
# ls -la */*/rsync*
-rw--- 1 112 122 302598406144 May  1 10:54 HOST/180/rsyncTmp.4971.0.29

That's a 300GB file, it filled the partition, and the full size for
this host is 337GB.

Thanks,
Marcelo


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/