RE: [gentoo-user] How to compress lots of tarballs
Regular xzutils now does multiple threads with the -T option.
> -Original Message-
> From: Ramon Fischer
> Sent: Sunday, September 26, 2021 5:23 AM
> To: gentoo-user@lists.gentoo.org
> Subject: Re: [gentoo-user] How to compress lots of tarballs
>
> In addition to this, you may want to use the parallel implementations of
> "gzip", "xz", "bzip2" or the new "zstd" (zstandard), which are "pigz"[1],
> "pixz"[2], "pbzip2"[3], or "zstmt" (within package "app-arch/zstd")[4] in
> order to increase performance:
>
> $ cd
> $ for tar_archive in *.tar; do pixz "${tar_archive}"; done
>
> -Ramon
>
> [1]
> * https://www.zlib.net/pigz/
>
> [2]
> * https://github.com/vasi/pixz
>
> [3]
> * https://launchpad.net/pbzip2
> * http://compression.ca/pbzip2/
>
> [4]
> * https://facebook.github.io/zstd/
>
>
> On 26/09/2021 13:36, Simon Thelen wrote:
> > [2021-09-26 11:57] Peter Humphrey
> >> part text/plain 382
> >> Hello list,
> > Hi,
> >
> >> I have an external USB-3 drive with various system backups. There are
> >> 350 .tar files (not .tar.gz etc.), amounting to 2.5TB. I was sure I
> >> wouldn't need to compress them, so I didn't, but now I think I'm
> >> going to have to. Is there a reasonably efficient way to do this? I
> >> have 500GB spare space on /dev/sda, and the machine runs constantly.
> > Pick your favorite of gzip, bzip2, xz or lzip (I recommend lzip) and
> > then:
> > mount USB-3 /mnt; cd /mnt; lzip *
> >
> > The archiver you chose will compress the file and add the appropriate
> > extension all on its own and tar will use that (and the file magic) to
> > find the appropriate decompresser when you want to extract files later
> > (you can use `tar tf' to test if you want).
> >
> > --
> > Simon Thelen
> >
>
> --
> GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF
>
Re: [gentoo-user] How to compress lots of tarballs
In addition to this, you may want to use the parallel implementations of
"gzip", "xz", "bzip2" or the new "zstd" (zstandard), which are
"pigz"[1], "pixz"[2], "pbzip2"[3], or "zstmt" (within package
"app-arch/zstd")[4] in order to increase performance:
$ cd
$ for tar_archive in *.tar; do pixz "${tar_archive}"; done
-Ramon
[1]
* https://www.zlib.net/pigz/
[2]
* https://github.com/vasi/pixz
[3]
* https://launchpad.net/pbzip2
* http://compression.ca/pbzip2/
[4]
* https://facebook.github.io/zstd/
On 26/09/2021 13:36, Simon Thelen wrote:
[2021-09-26 11:57] Peter Humphrey
part text/plain 382
Hello list,
Hi,
I have an external USB-3 drive with various system backups. There are 350 .tar
files (not .tar.gz etc.), amounting to 2.5TB. I was sure I wouldn't need to
compress them, so I didn't, but now I think I'm going to have to. Is there a
reasonably efficient way to do this? I have 500GB spare space on /dev/sda, and
the machine runs constantly.
Pick your favorite of gzip, bzip2, xz or lzip (I recommend lzip) and
then:
mount USB-3 /mnt; cd /mnt; lzip *
The archiver you chose will compress the file and add the appropriate
extension all on its own and tar will use that (and the file magic) to
find the appropriate decompresser when you want to extract files later
(you can use `tar tf' to test if you want).
--
Simon Thelen
--
GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF
OpenPGP_signature
Description: OpenPGP digital signature
Re: [gentoo-user] How to compress lots of tarballs
Addendum:
To complete the list. Here the parallel implementation of "lzip":
"plzip": https://www.nongnu.org/lzip/plzip.html
-Ramon
On 26/09/2021 14:23, Ramon Fischer wrote:
In addition to this, you may want to use the parallel implementations
of "gzip", "xz", "bzip2" or the new "zstd" (zstandard), which are
"pigz"[1], "pixz"[2], "pbzip2"[3], or "zstmt" (within package
"app-arch/zstd")[4] in order to increase performance:
$ cd
$ for tar_archive in *.tar; do pixz "${tar_archive}"; done
-Ramon
[1]
* https://www.zlib.net/pigz/
[2]
* https://github.com/vasi/pixz
[3]
* https://launchpad.net/pbzip2
* http://compression.ca/pbzip2/
[4]
* https://facebook.github.io/zstd/
On 26/09/2021 13:36, Simon Thelen wrote:
[2021-09-26 11:57] Peter Humphrey
part text/plain 382
Hello list,
Hi,
I have an external USB-3 drive with various system backups. There
are 350 .tar
files (not .tar.gz etc.), amounting to 2.5TB. I was sure I wouldn't
need to
compress them, so I didn't, but now I think I'm going to have to. Is
there a
reasonably efficient way to do this? I have 500GB spare space on
/dev/sda, and
the machine runs constantly.
Pick your favorite of gzip, bzip2, xz or lzip (I recommend lzip) and
then:
mount USB-3 /mnt; cd /mnt; lzip *
The archiver you chose will compress the file and add the appropriate
extension all on its own and tar will use that (and the file magic) to
find the appropriate decompresser when you want to extract files later
(you can use `tar tf' to test if you want).
--
Simon Thelen
--
GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF
OpenPGP_signature
Description: OpenPGP digital signature
Re: [gentoo-user] How to compress lots of tarballs
On Sunday, 26 September 2021 13:25:24 BST Ramon Fischer wrote:
> Addendum:
>
> To complete the list. Here the parallel implementation of "lzip":
>
> "plzip": https://www.nongnu.org/lzip/plzip.html
>
> -Ramon
>
> On 26/09/2021 14:23, Ramon Fischer wrote:
> > In addition to this, you may want to use the parallel implementations
> > of "gzip", "xz", "bzip2" or the new "zstd" (zstandard), which are
> > "pigz"[1], "pixz"[2], "pbzip2"[3], or "zstmt" (within package
> > "app-arch/zstd")[4] in order to increase performance:
> >
> >$ cd
> >$ for tar_archive in *.tar; do pixz "${tar_archive}"; done
> >
> > -Ramon
> >
> > [1]
> > * https://www.zlib.net/pigz/
> >
> > [2]
> > * https://github.com/vasi/pixz
> >
> > [3]
> > * https://launchpad.net/pbzip2
> > * http://compression.ca/pbzip2/
> >
> > [4]
> > * https://facebook.github.io/zstd/
> >
> > On 26/09/2021 13:36, Simon Thelen wrote:
> >> [2021-09-26 11:57] Peter Humphrey
> >>
> >>> part text/plain 382
> >>> Hello list,
> >>
> >> Hi,
> >>
> >>> I have an external USB-3 drive with various system backups. There
> >>> are 350 .tar
> >>> files (not .tar.gz etc.), amounting to 2.5TB. I was sure I wouldn't
> >>> need to
> >>> compress them, so I didn't, but now I think I'm going to have to. Is
> >>> there a
> >>> reasonably efficient way to do this? I have 500GB spare space on
> >>> /dev/sda, and
> >>> the machine runs constantly.
> >>
> >> Pick your favorite of gzip, bzip2, xz or lzip (I recommend lzip) and
> >> then:
> >> mount USB-3 /mnt; cd /mnt; lzip *
> >>
> >> The archiver you chose will compress the file and add the appropriate
> >> extension all on its own and tar will use that (and the file magic) to
> >> find the appropriate decompresser when you want to extract files later
> >> (you can use `tar tf' to test if you want).
Thank you both. Now, as it's a single USB-3 drive, what advantage would a
parallel implementation confer? I assume I'd be better compressing from
external to SATA, then writing back, or is that wrong?
Or, I could connect a second USB-3 drive to a different interface, then read
from one and write to the other, with or without the SATA between.
--
Regards,
Peter.
