Re: Introduce pg_receivewal gzip compression tests

2021-07-19 Thread Michael Paquier
On Mon, Jul 19, 2021 at 04:03:33PM +0900, Michael Paquier wrote: > Another advantage of this patch is the handling of ".gz" is reduced to > one code path instead of four. That makes a bit easier the > introduction of new compression methods. > > A second thing that was really confusing is that

Re: Introduce pg_receivewal gzip compression tests

2021-07-19 Thread Michael Paquier
On Fri, Jul 16, 2021 at 02:08:57PM +0900, Michael Paquier wrote: > This behavior is rather debatable, and it would be more instinctive to > me to just skip any business related to the pre-padding if compression > is enabled, at the cost of one extra callback in WalWriteMethod to > grab the

Re: Introduce pg_receivewal gzip compression tests

2021-07-15 Thread Michael Paquier
On Fri, Jul 16, 2021 at 08:59:11AM +0900, Michael Paquier wrote: > --compress is used and the sync fails for a non-compressed segment. > Looking at the code it is pretty obvious that open_walfile() is > getting confused with the handling of an existing .partial segment > while walmethods.c uses

Re: Introduce pg_receivewal gzip compression tests

2021-07-15 Thread Michael Paquier
On Thu, Jul 15, 2021 at 09:35:52PM +0900, Michael Paquier wrote: > For this one, I'll try to test harder on my own host. I am curious to > see if the other Windows members running the TAP tests have anything > to say. Looking at the code of zlib, this would come from gz_zero() > in gzflush(),

Re: Introduce pg_receivewal gzip compression tests

2021-07-15 Thread gkokolatos
‐‐‐ Original Message ‐‐‐ On Thursday, July 15th, 2021 at 14:35, Michael Paquier wrote: > On Thu, Jul 15, 2021 at 08:35:27PM +0900, Michael Paquier wrote: > > > 2. curculio: > > Looking at the OpenBSD code (usr.bin/compress/main.c), long options > > are supported, where --version

Re: Introduce pg_receivewal gzip compression tests

2021-07-15 Thread Michael Paquier
On Thu, Jul 15, 2021 at 08:35:27PM +0900, Michael Paquier wrote: > 1) bowerbird on Windows/MSVC: > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=bowerbird=2021-07-15%2010%3A30%3A36 > pg_receivewal: fatal: could not fsync existing write-ahead log file > "00010002.partial":

Re: Introduce pg_receivewal gzip compression tests

2021-07-15 Thread Michael Paquier
On Thu, Jul 15, 2021 at 07:48:08AM +, gkokola...@pm.me wrote: > Let us hope that it will prevent some bugs from happening. The buildfarm has two reports. 1) bowerbird on Windows/MSVC: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=bowerbird=2021-07-15%2010%3A30%3A36 pg_receivewal:

Re: Introduce pg_receivewal gzip compression tests

2021-07-15 Thread gkokolatos
‐‐‐ Original Message ‐‐‐ On Thursday, July 15th, 2021 at 09:00, Michael Paquier wrote: > On Wed, Jul 14, 2021 at 02:11:09PM +, gkokola...@pm.me wrote: > > > Please find v6 attached. > > Thanks. I have spent some time checking this stuff in details, and > I did some tests on

Re: Introduce pg_receivewal gzip compression tests

2021-07-15 Thread Michael Paquier
On Wed, Jul 14, 2021 at 02:11:09PM +, gkokola...@pm.me wrote: > Please find v6 attached. Thanks. I have spent some time checking this stuff in details, and I did some tests on Windows while on it. A run of pgperltidy was missing. A second thing is that you added one useless WAL segment

Re: Introduce pg_receivewal gzip compression tests

2021-07-14 Thread gkokolatos
‐‐‐ Original Message ‐‐‐ On Wednesday, July 14th, 2021 at 04:17, Michael Paquier wrote: > On Tue, Jul 13, 2021 at 11:16:06AM +, gkokola...@pm.me wrote: > > Agreed. For the record that is why I said v6 :) > Okay, thanks. Please find v6 attached. Cheers, //Georgios >

Re: Introduce pg_receivewal gzip compression tests

2021-07-13 Thread Michael Paquier
On Tue, Jul 13, 2021 at 11:16:06AM +, gkokola...@pm.me wrote: > Agreed. For the record that is why I said v6 :) Okay, thanks. -- Michael signature.asc Description: PGP signature

Re: Introduce pg_receivewal gzip compression tests

2021-07-13 Thread gkokolatos
‐‐‐ Original Message ‐‐‐ On Tuesday, July 13th, 2021 at 12:26, Michael Paquier wrote: > On Tue, Jul 13, 2021 at 08:28:44AM +, gkokola...@pm.me wrote: > > Sounds great. Let me cook up v6 for this. > > Thanks. Could you use v5 I posted upthread as a base? There were > some

Re: Introduce pg_receivewal gzip compression tests

2021-07-13 Thread Michael Paquier
On Tue, Jul 13, 2021 at 08:28:44AM +, gkokola...@pm.me wrote: > Sounds great. Let me cook up v6 for this. Thanks. Could you use v5 I posted upthread as a base? There were some improvements in the variable names, the comments and the test descriptions. -- Michael signature.asc Description:

Re: Introduce pg_receivewal gzip compression tests

2021-07-13 Thread gkokolatos
‐‐‐ Original Message ‐‐‐ On Tuesday, July 13th, 2021 at 10:14, Michael Paquier wrote: > On Tue, Jul 13, 2021 at 04:37:53PM +0900, Michael Paquier wrote: > > Poking at this problem, I partially take this statement back as this > requires an initial run of pg_receivewal --endpos to

Re: Introduce pg_receivewal gzip compression tests

2021-07-13 Thread gkokolatos
‐‐‐ Original Message ‐‐‐ On Tuesday, July 13th, 2021 at 09:37, Michael Paquier wrote: > On Tue, Jul 13, 2021 at 06:36:59AM +, gkokola...@pm.me wrote: > > > I am sorry this was not so clear. It is indeed running twice the binary > > with different flags. However the goal is not

Re: Introduce pg_receivewal gzip compression tests

2021-07-13 Thread Michael Paquier
On Tue, Jul 13, 2021 at 04:37:53PM +0900, Michael Paquier wrote: > Hmm. It looks like a waste in runtime once we mix LZ4 in that as that > would mean 5 runs of pg_receivewal, but we really need only three of > them with --endpos: > - One with ZLIB compression. > - One with LZ4 compression. > -

Re: Introduce pg_receivewal gzip compression tests

2021-07-13 Thread Michael Paquier
On Tue, Jul 13, 2021 at 06:36:59AM +, gkokola...@pm.me wrote: > I am sorry this was not so clear. It is indeed running twice the binary > with different flags. However the goal is not to check the flags, but > to make certain that the partial file has now been completed. That is > why there

Re: Introduce pg_receivewal gzip compression tests

2021-07-13 Thread gkokolatos
‐‐‐ Original Message ‐‐‐ On Tuesday, July 13th, 2021 at 03:53, Michael Paquier wrote: > On Mon, Jul 12, 2021 at 04:46:29PM +, gkokola...@pm.me wrote: > > > On Monday, July 12th, 2021 at 17:07, gkokola...@pm.me wrote: > > > > > ‐‐‐ Original Message ‐‐‐ > > Are you using

Re: Introduce pg_receivewal gzip compression tests

2021-07-12 Thread Michael Paquier
On Mon, Jul 12, 2021 at 04:46:29PM +, gkokola...@pm.me wrote: > On Monday, July 12th, 2021 at 17:07, wrote: >> ‐‐‐ Original Message ‐‐‐ Are you using outlook? The format of your messages gets blurry on the PG website, so does it for me. >> I am admittedly not so well versed on

Re: Introduce pg_receivewal gzip compression tests

2021-07-12 Thread gkokolatos
‐‐‐ Original Message ‐‐‐ On Monday, July 12th, 2021 at 17:07, wrote: > ‐‐‐ Original Message ‐‐‐ > > On Monday, July 12th, 2021 at 13:04, Michael Paquier mich...@paquier.xyz > wrote: > > > On Mon, Jul 12, 2021 at 09:42:32AM +, gkokola...@pm.me wrote: > > > > > This to my

Re: Introduce pg_receivewal gzip compression tests

2021-07-12 Thread gkokolatos
‐‐‐ Original Message ‐‐‐ On Monday, July 12th, 2021 at 13:04, Michael Paquier wrote: > On Mon, Jul 12, 2021 at 09:42:32AM +, gkokola...@pm.me wrote: > > > This to my understanding means that gzip is expected to exist. > > > > If this is correct, then simply checking for the

Re: Introduce pg_receivewal gzip compression tests

2021-07-12 Thread gkokolatos
‐‐‐ Original Message ‐‐‐ On Monday, July 12th, 2021 at 13:00, Gilles Darold wrote: > Le 12/07/2021 à 12:27, gkokola...@pm.me a écrit : > > > > > > Shouldn't this be coded as a loop going through @gzip_wals? > > > > > > > > > > I would hope that there is only one gz file created.

Re: Introduce pg_receivewal gzip compression tests

2021-07-12 Thread Michael Paquier
On Mon, Jul 12, 2021 at 09:42:32AM +, gkokola...@pm.me wrote: > This to my understanding means that gzip is expected to exist. > If this is correct, then simply checking for the headers should > suffice, since that is the only dependency for the files to be > created. You cannot expect this

Re: Introduce pg_receivewal gzip compression tests

2021-07-12 Thread Gilles Darold
Le 12/07/2021 à 12:27, gkokola...@pm.me a écrit : Shouldn't this be coded as a loop going through @gzip_wals? >>> I would hope that there is only one gz file created. There is a line >>> >>> further up that tests exactly that. >>> >>> - is (scalar(@gzip_wals), 1, "one gzip compressed

Re: Introduce pg_receivewal gzip compression tests

2021-07-12 Thread gkokolatos
‐‐‐ Original Message ‐‐‐ On Monday, July 12th, 2021 at 11:56, wrote: > ‐‐‐ Original Message ‐‐‐ > > On Monday, July 12th, 2021 at 11:42, gkokola...@pm.me wrote: > > > ‐‐‐ Original Message ‐‐‐ > > > > On Monday, July 12th, 2021 at 08:42, Michael Paquier

Re: Introduce pg_receivewal gzip compression tests

2021-07-12 Thread gkokolatos
‐‐‐ Original Message ‐‐‐ On Monday, July 12th, 2021 at 11:42, wrote: > ‐‐‐ Original Message ‐‐‐ > > On Monday, July 12th, 2021 at 08:42, Michael Paquier mich...@paquier.xyz > wrote: > > > On Fri, Jul 09, 2021 at 11:26:58AM +, Georgios wrote: > > > > > As suggested on a

Re: Introduce pg_receivewal gzip compression tests

2021-07-12 Thread gkokolatos
‐‐‐ Original Message ‐‐‐ On Monday, July 12th, 2021 at 08:42, Michael Paquier wrote: > On Fri, Jul 09, 2021 at 11:26:58AM +, Georgios wrote: > > > As suggested on a different thread [1], pg_receivewal can increase it's test > > > > coverage. There exists a non trivial amount of

Re: Introduce pg_receivewal gzip compression tests

2021-07-12 Thread Michael Paquier
On Fri, Jul 09, 2021 at 11:26:58AM +, Georgios wrote: > As suggested on a different thread [1], pg_receivewal can increase it's test > coverage. There exists a non trivial amount of code that handles gzip > compression. The current patch introduces tests that cover creation of gzip >