I've ran it with all the different compression levels on one of the smaller
db's now. And not sending any flags to it see is, as I've seen hinted on some
page on internet, same as level 6.
I do, somewhat, share the opinion that something is up with zlib. But at the
same time I haven't touch it since the 8.4 installation so it's a mystery how
it could've failed on its own. The only thing performed was an upgrade from 8.4
to 9.5. But yes, I can not really say exactly what that upgrade touched and
what it didn't touch. Will investigate further.
COMPRESSION LEVEL: 0
FILE SIZE: 6205982696
COMPRESSION LEVEL: 1
FILE SIZE: 1391475419
COMPRESSION LEVEL: 2
FILE SIZE: 1344563403
COMPRESSION LEVEL: 3
FILE SIZE: 1267601394
COMPRESSION LEVEL: 4
FILE SIZE: 1241632684
COMPRESSION LEVEL: 5
FILE SIZE: 1178377949
COMPRESSION LEVEL: 6
FILE SIZE: 1137727582
COMPRESSION LEVEL: 7
FILE SIZE: 1126257786
COMPRESSION LEVEL: 8
FILE SIZE: 1111804793
COMPRESSION LEVEL: 9
FILE SIZE: 1112194596
COMPRESSION LEVEL AT DEFAULT NO FLAG PASSED TO 'pg_dump'
FILE SIZE: 1140261276
cto | compositor
mobile [ + 46 (0)704 71 89 54 ]
skype [ cednert ]
On 22 Nov 2017, at 11:32, Matthew Hall
On Nov 21, 2017, at 10:18 PM, Henrik Cednert (Filmlance)
WHat's the normal way to deal with compression? Dump uncompressed and use
something that threads better to compress the dump?
I would say most likely your zlib is screwed up somehow, like maybe it didn't
get optimized right by the C compiler or something else sucks w/ the
compression settings. The CPU should easily blast away at that faster than
disks can read.
I did do some studies of this previously some years ago, and I found gzip -6
offered the best ratio between size reduction and CPU time out of a very wide
range of formats, but at the time xz was also not yet available.
If I were you I would first pipe the uncompressed output through a separate
compression command, then you can experiment with the flags and threads, and
you already get another separate process for the kernel to put on other CPUs as
an automatic bonus for multi-core with minimal work.
After that, xz is GNU standard now and has xz -T for cranking up some threads,
with little extra effort for the user. But it can be kind of slow so probably
need to lower the compression level somewhat depending a bit on some time
testing. I would try on some medium sized DB table, like a bit over the size of
system RAM, instead of dumping this great big DB, in order to benchmark a
couple times until it looks happy.