On Wed, Dec 28, 2016 at 6:58 AM, Michael Paquier <michael.paqu...@gmail.com> wrote:
> On Wed, Dec 28, 2016 at 3:12 AM, Magnus Hagander <mag...@hagander.net> > wrote: > > On Tue, Dec 27, 2016 at 1:16 PM, Michael Paquier < > michael.paqu...@gmail.com> > > wrote: > >> On Tue, Dec 27, 2016 at 6:34 PM, Magnus Hagander <mag...@hagander.net> > >> wrote: > >> > On Tue, Dec 27, 2016 at 2:23 AM, Michael Paquier > >> > <michael.paqu...@gmail.com> > >> > wrote: > >> >> Magnus, you have mentioned me as well that you had a couple of ideas > >> >> on the matter, feel free to jump in and let's mix our thoughts! > >> > > >> > > >> > Yeah, I've been wondering what the actual usecase is here :) > >> > >> There is value to compress segments finishing with trailing zeros, > >> even if they are not the species with the highest representation in > >> the WAL archive. > > > > Agreed on that part -- that's the value in compression though, and not > > necessarily the TAR format itself. > > > > Is there any value of the TAR format *without* compression in your > scenario? > > Hm. I cannot think of one. > > >> > I can see the point of being able to compress the individual segments > >> > directly in pg_receivexlog in smaller systems though, without the need > >> > to > >> > rely on an external compression program as well. But in that case, is > >> > there > >> > any reason we need to wrap it in a tarfile, and can't just write it to > >> > <segment>.gz natively? > >> > >> You mean having a --compress=0|9 option that creates individual gz > >> files for each segment? Definitely we could just do that. It would be > > > > Yes, that's what I meant. > > OK, I have bitten the bullet and attached is a patch to add just > --compress=0|9. So there is one .gz file generated per segment, and > things are rather straight-forward. Adding tests is unfortunately not > in scope as not all builds are compiled with libz. > Conditional tests? It probably wouldn't hurt to have them, but that would be something more generic (like we'd need something to actually validate it -- but it would make sense to have a test that, with compression enabled, would verify if the uncompressed file turns out to be exactly 16Mb for example). > A couple of things to be aware of though: > - gzopen, gzwrite and gzclose are used to handle the gz files. That's > unconsistent with the tar method that is based on mainly deflate() and > more low level routines. > But chosen for simplicity, I assume? > - I have switched the directory method to use a file pointer instead > of a file descriptor as gzwrite returns int as the number of > uncompressed bytes written. > I don't really follow that reasoning :) Why does the directory method have to change to use a filepointer because of that? > - history and timeline files are gzip'd as well. Perhaps we don't want > to do that. > I think it makes sense to compress everything. > What do you think about this approach? I'll add that to the next CF. I haven't reviweed the code in detail but yes, I think this approach is the right one. //Magnus
