Re: [gentoo-user] How to compress lots of tarballs

2021-10-01 Thread Frank Steinmetzger
Am Wed, Sep 29, 2021 at 03:04:41PM -0500 schrieb Dale: > Curious question here.  As you may recall, I backup to a external hard > drive.  Would it make sense to use that software for a external hard > drive? Since you are using LVM for everything IIRC, it would be a very efficient way for you to

Re: [gentoo-user] How to compress lots of tarballs

2021-09-30 Thread antlists
On 30/09/2021 00:17, Rich Freeman wrote: On Wed, Sep 29, 2021 at 5:48 PM Wols Lists wrote: An LVM snapshot creates a "copy on write" image. I'm just beginning to dig into it myself, but I agree it's a bit confusing. So, snapshots in general are a solution for making backups atomic. That is,

Re: [gentoo-user] How to compress lots of tarballs

2021-09-29 Thread Rich Freeman
On Wed, Sep 29, 2021 at 5:48 PM Wols Lists wrote: > > An LVM snapshot creates a "copy on write" image. I'm just beginning to > dig into it myself, but I agree it's a bit confusing. So, snapshots in general are a solution for making backups atomic. That is, they allow a backup to look as if the

Re: [gentoo-user] How to compress lots of tarballs

2021-09-29 Thread Wols Lists
On 29/09/2021 21:58, Dale wrote: Since the drive also uses LVM, someone mentioned using snapshots. Me? Still not real clear on those even tho I've read a bit about them.  Some of the backup technics are confusing to me.  I get plain files, even incremental to a extent but some of the new

Re: [gentoo-user] How to compress lots of tarballs

2021-09-29 Thread Dale
Laurence Perkins wrote: >> >> Curious question here. As you may recall, I backup to a external hard >> drive. Would it make sense to use that software for a external hard drive? >> Right now, I'm just doing file updates with rsync and the drive is >> encrypted. Thing is, I'm going to have

RE: [gentoo-user] How to compress lots of tarballs

2021-09-29 Thread Laurence Perkins
> > On Wed, Sep 29, 2021 at 4:27 AM Peter Humphrey > > wrote: > >> Thanks Laurence. I've looked at borg before, wondering whether I > >> needed a more sophisticated tool than just tar, but it looked like > >> too much work for little gain. I didn't know about duplicity, but I'm > >> used to

Re: [gentoo-user] How to compress lots of tarballs

2021-09-29 Thread Dale
Rich Freeman wrote: > On Wed, Sep 29, 2021 at 4:27 AM Peter Humphrey wrote: >> Thanks Laurence. I've looked at borg before, wondering whether I needed a >> more sophisticated tool than just tar, but it looked like too much work for >> little gain. I didn't know about duplicity, but I'm used to my

Re: [gentoo-user] How to compress lots of tarballs

2021-09-29 Thread Rich Freeman
On Wed, Sep 29, 2021 at 4:27 AM Peter Humphrey wrote: > > Thanks Laurence. I've looked at borg before, wondering whether I needed a > more sophisticated tool than just tar, but it looked like too much work for > little gain. I didn't know about duplicity, but I'm used to my weekly routine > and

Re: [gentoo-user] How to compress lots of tarballs

2021-09-29 Thread Peter Humphrey
On Tuesday, 28 September 2021 18:43:06 BST Laurence Perkins wrote: > There are also backup tools which will handle the compression step for you. > > app-backup/duplicity uses a similar tar file and index system with periodic > full and then incremental chains. Plus it keeps a condensed list of

RE: [gentoo-user] How to compress lots of tarballs

2021-09-28 Thread Laurence Perkins
Regular xzutils now does multiple threads with the -T option. > -Original Message- > From: Ramon Fischer > Sent: Sunday, September 26, 2021 5:23 AM > To: gentoo-user@lists.gentoo.org > Subject: Re: [gentoo-user] How to compress lots of tarballs > > In addition

RE: [gentoo-user] How to compress lots of tarballs

2021-09-28 Thread Laurence Perkins
>On Monday, 27 September 2021 14:30:36 BST Peter Humphrey wrote: >> On Monday, 27 September 2021 02:39:19 BST Adam Carter wrote: >> > On Sun, Sep 26, 2021 at 8:57 PM Peter Humphrey > >> > >> > wrote: >> > > Hello list, >> > > >> > > I have an external USB-3 drive with various system backups.

Re: [gentoo-user] How to compress lots of tarballs

2021-09-28 Thread Peter Humphrey
On Tuesday, 28 September 2021 12:38:42 BST Rich Freeman wrote: > You keep mentioning USB3, but I think the main factor here is that the > external drive is probably a spinning hard drive (I don't think you > explicitly mentioned this but it seems likely esp with the volume of > data). That math

Re: [gentoo-user] How to compress lots of tarballs

2021-09-28 Thread Rich Freeman
On Mon, Sep 27, 2021 at 9:30 AM Peter Humphrey wrote: > > Thanks to all who've helped. I can't avoid feeling, though, that the main > bottleneck has been missed: that I have to read and write on a USB-3 drive. > It's just taken 23 minutes to copy the current system backup from USB-3 to > SATA

Re: [gentoo-user] How to compress lots of tarballs

2021-09-28 Thread Peter Humphrey
On Sunday, 26 September 2021 11:57:43 BST Peter Humphrey wrote: > Hello list, > > I have an external USB-3 drive with various system backups. There are 350 > .tar files (not .tar.gz etc.), amounting to 2.5TB. I was sure I wouldn't > need to compress them, so I didn't, but now I think I'm going to

Re: [gentoo-user] How to compress lots of tarballs

2021-09-27 Thread Peter Humphrey
On Monday, 27 September 2021 14:30:36 BST Peter Humphrey wrote: > On Monday, 27 September 2021 02:39:19 BST Adam Carter wrote: > > On Sun, Sep 26, 2021 at 8:57 PM Peter Humphrey > > > > wrote: > > > Hello list, > > > > > > I have an external USB-3 drive with various system backups. There are >

Re: [gentoo-user] How to compress lots of tarballs

2021-09-27 Thread Peter Humphrey
On Monday, 27 September 2021 02:39:19 BST Adam Carter wrote: > On Sun, Sep 26, 2021 at 8:57 PM Peter Humphrey > > wrote: > > Hello list, > > > > I have an external USB-3 drive with various system backups. There are 350 > > .tar files (not .tar.gz etc.), amounting to 2.5TB. I was sure I wouldn't

Re: [gentoo-user] How to compress lots of tarballs

2021-09-26 Thread Adam Carter
On Sun, Sep 26, 2021 at 8:57 PM Peter Humphrey wrote: > Hello list, > > I have an external USB-3 drive with various system backups. There are 350 > .tar > files (not .tar.gz etc.), amounting to 2.5TB. I was sure I wouldn't need > to > compress them, so I didn't, but now I think I'm going to have

Re: [gentoo-user] How to compress lots of tarballs

2021-09-26 Thread antlists
On 26/09/2021 16:38, Peter Humphrey wrote: Or, I could connect a second USB-3 drive to a different interface, then read from one and write to the other, with or without the SATA between. If you've got a second drive, consider changing your strategy ... First of all, you want eSATA or USB3 for

Re: [gentoo-user] How to compress lots of tarballs

2021-09-26 Thread Peter Humphrey
On Sunday, 26 September 2021 13:25:24 BST Ramon Fischer wrote: > Addendum: > > To complete the list. Here the parallel implementation of "lzip": > > "plzip": https://www.nongnu.org/lzip/plzip.html > > -Ramon > > On 26/09/2021 14:23, Ramon Fischer wrote: > > In addition to this, you may

Re: [gentoo-user] How to compress lots of tarballs

2021-09-26 Thread Ramon Fischer
Addendum: To complete the list. Here the parallel implementation of "lzip": "plzip": https://www.nongnu.org/lzip/plzip.html -Ramon On 26/09/2021 14:23, Ramon Fischer wrote: In addition to this, you may want to use the parallel implementations of "gzip", "xz", "bzip2" or the new "zstd"

Re: [gentoo-user] How to compress lots of tarballs

2021-09-26 Thread Ramon Fischer
In addition to this, you may want to use the parallel implementations of "gzip", "xz", "bzip2" or the new "zstd" (zstandard), which are "pigz"[1], "pixz"[2], "pbzip2"[3], or "zstmt" (within package "app-arch/zstd")[4] in order to increase performance: $ cd $ for tar_archive in *.tar;

Re: [gentoo-user] How to compress lots of tarballs

2021-09-26 Thread Simon Thelen
[2021-09-26 11:57] Peter Humphrey > part text/plain 382 > Hello list, Hi, > I have an external USB-3 drive with various system backups. There are 350 .tar > files (not .tar.gz etc.), amounting to 2.5TB. I was sure I wouldn't need to > compress them, so I didn't, but now I