Re: More issues with pg_verify_checksums and checksum verification in base backups

2019-08-06 Thread Michael Paquier
On Tue, Aug 06, 2019 at 04:20:43PM -0400, Stephen Frost wrote: > On Tue, Aug 6, 2019 at 15:45 Magnus Hagander wrote: >> When agreement cannot be found, perhaps a parameter is in order? >> >> That is, have the tool complain about such files by default but with a >> HINT that it may or may not be a

Re: More issues with pg_verify_checksums and checksum verification in base backups

2019-08-06 Thread Stephen Frost
Greetings, On Tue, Aug 6, 2019 at 15:45 Magnus Hagander wrote: > On Tue, Aug 6, 2019 at 6:07 PM Stephen Frost wrote: > >> Greetings, >> >> * Andres Freund (and...@anarazel.de) wrote: >> > On 2019-08-06 10:58:15 -0400, Stephen Frost wrote: >> > > * Michael Banck (michael.ba...@credativ.de)

Re: More issues with pg_verify_checksums and checksum verification in base backups

2019-08-06 Thread Magnus Hagander
On Tue, Aug 6, 2019 at 6:07 PM Stephen Frost wrote: > Greetings, > > * Andres Freund (and...@anarazel.de) wrote: > > On 2019-08-06 10:58:15 -0400, Stephen Frost wrote: > > > * Michael Banck (michael.ba...@credativ.de) wrote: > > > > Independently of the whitelist/blacklist question, I believe >

Re: More issues with pg_verify_checksums and checksum verification in base backups

2019-08-06 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2019-08-06 10:58:15 -0400, Stephen Frost wrote: > > * Michael Banck (michael.ba...@credativ.de) wrote: > > > Independently of the whitelist/blacklist question, I believe > > > pg_checksums should not error out as soon as it encounters a

Re: More issues with pg_verify_checksums and checksum verification in base backups

2019-08-06 Thread Andres Freund
Hi, On 2019-08-06 10:58:15 -0400, Stephen Frost wrote: > * Michael Banck (michael.ba...@credativ.de) wrote: > > Independently of the whitelist/blacklist question, I believe > > pg_checksums should not error out as soon as it encounters a weird looking > > file, but either (i) still checksum it or

Re: More issues with pg_verify_checksums and checksum verification in base backups

2019-08-06 Thread Stephen Frost
Greetings, * Michael Banck (michael.ba...@credativ.de) wrote: > Independently of the whitelist/blacklist question, I believe > pg_checksums should not error out as soon as it encounters a weird looking > file, but either (i) still checksum it or (ii) skip it? Or is that to be > considered a pilot

Re: More issues with pg_verify_checksums and checksum verification in base backups

2019-08-05 Thread Michael Paquier
On Sat, Aug 03, 2019 at 06:47:48PM +0200, Michael Banck wrote: > first off, a bit of a meta-question: Did the whitelist approach die > completely, or are we going to tackle it again for v13 or later? At this stage, it is burried. Amen. > This is something I still have in the test suite of my

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-11-29 Thread Michael Paquier
On Thu, Nov 29, 2018 at 11:03:54AM +0900, Michael Paquier wrote: > Thanks David for the input. I think that I will be able to finish > wrapping and commit this stuff tomorrow. And done. I kept the split into two commits for clarity as suggested by Stephen, as bisect would actually complain only

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-11-28 Thread Michael Paquier
On Wed, Nov 28, 2018 at 12:32:29PM -0500, David Steele wrote: > Looks good to me. Thanks David for the input. I think that I will be able to finish wrapping and commit this stuff tomorrow. -- Michael signature.asc Description: PGP signature

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-11-28 Thread David Steele
On 11/27/18 8:32 PM, Michael Paquier wrote: > >>> Attached are two patches to fix all the mess: >>> - 0001 is a revert of the whitelist, minus the set of regression tests >>> checking after corrupted files and empty files. >>> - 0002 is a fix for all the issues reported on this thread, with tests

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-11-27 Thread Michael Paquier
On Tue, Nov 27, 2018 at 08:17:12PM -0500, Stephen Frost wrote: > * Michael Paquier (mich...@paquier.xyz) wrote: >> Please see 0002 attached, which moves the call to skipfile() where I >> think it should go. > > Alright, on a quick glance that seems ok. Thanks. >> Base backups are impacted as

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-11-27 Thread Stephen Frost
Greetings, * Michael Paquier (mich...@paquier.xyz) wrote: > On Tue, Nov 27, 2018 at 06:27:57PM -0500, Stephen Frost wrote: > > * Michael Paquier (mich...@paquier.xyz) wrote: > >> Believe me or not, but we have spent so much energy on this stuff that I > >> am ready to give up on the whitelist

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-11-27 Thread Michael Paquier
On Tue, Nov 27, 2018 at 06:27:57PM -0500, Stephen Frost wrote: > * Michael Paquier (mich...@paquier.xyz) wrote: >> Believe me or not, but we have spent so much energy on this stuff that I >> am ready to give up on the whitelist patch and focus on other business. > > I would have hoped that you'd

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-11-27 Thread Stephen Frost
Greetings, * Michael Paquier (mich...@paquier.xyz) wrote: > On Tue, Nov 27, 2018 at 05:45:41PM -0500, Stephen Frost wrote: > > This doesn't exactly change my opinion regarding this discussion and I'd > > rather we revert the "whitelist" patch and use the very minimal patch > > from here: > > > >

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-11-27 Thread Michael Paquier
On Tue, Nov 27, 2018 at 05:45:41PM -0500, Stephen Frost wrote: > This doesn't exactly change my opinion regarding this discussion and I'd > rather we revert the "whitelist" patch and use the very minimal patch > from here: > >

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-11-27 Thread Stephen Frost
Greetings, * Michael Paquier (mich...@paquier.xyz) wrote: > On Mon, Nov 19, 2018 at 10:17:19PM -0500, Stephen Frost wrote: > > Let's try to not conflate these two issues though, they're quite > > independent. > > This is a poke about a recent issue raised by Michael Banck here: >

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-11-27 Thread Michael Paquier
On Mon, Nov 19, 2018 at 10:17:19PM -0500, Stephen Frost wrote: > Let's try to not conflate these two issues though, they're quite > independent. This is a poke about a recent issue raised by Michael Banck here: https://www.postgresql.org/message-id/f1543332405.17247.9.ca...@credativ.de And for

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-11-19 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2018-11-19 21:18:43 -0500, Stephen Frost wrote: > > As has been mentioned elsewhere, there's really a 'right' way to do > > things and allowing PG to be 'extensible' by simply ignoring random > > files showing up isn't that- if we want

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-11-19 Thread Andres Freund
Hi, On 2018-11-19 21:18:43 -0500, Stephen Frost wrote: > As has been mentioned elsewhere, there's really a 'right' way to do > things and allowing PG to be 'extensible' by simply ignoring random > files showing up isn't that- if we want PG to be extensible in this way > then we need to provide a

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-11-19 Thread Stephen Frost
Greetings Michael, * Michael Paquier (mich...@paquier.xyz) wrote: > I am still a fan of the whitelist approach as there is no actual point > in restricting what people can do with Postgres in terms of > extensibility (relying on tablespace paths for storage plugin looks like > an important thing

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-11-19 Thread Michael Paquier
On Mon, Nov 19, 2018 at 08:45:29PM -0500, Stephen Frost wrote: > Michael, this obviously didn't happen and instead we ended up releasing > 11.1 with your changes, but I don't feel like this issue is closed and > I'm a bit disappointed that there hasn't been any further responses or > discussions

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-11-19 Thread Stephen Frost
Greetings, * David Steele (da...@pgmasters.net) wrote: > On 10/30/18 11:59 AM, Stephen Frost wrote: > > * Kyotaro HORIGUCHI (horiguchi.kyot...@lab.ntt.co.jp) wrote: > >> So I'm +1 for the Michael's current patch as (I think) we can't > >> make visible or large changes. > >> > >> That said, I

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-11-01 Thread Michael Paquier
On Thu, Nov 01, 2018 at 04:44:40PM +0530, Amit Kapila wrote: > This sounds like a good argument for having a whitelist approach, but > is it really a big problem if a user gets warning for files that the > utility is not able to verify checksums for? I think in some sense > this message can be

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-11-01 Thread Amit Kapila
On Sun, Oct 21, 2018 at 7:12 PM Michael Paquier wrote: > > Hi all, > > This is a follow-up of the following thread: > https://www.postgresql.org/message-id/20181012010411.re53cwcistcpi...@alap3.anarazel.de > > In a nutshell, b34e84f1 has added TAP tests for pg_verify_checksums, and > the

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-10-30 Thread David Steele
On 10/30/18 11:59 AM, Stephen Frost wrote: > > * Kyotaro HORIGUCHI (horiguchi.kyot...@lab.ntt.co.jp) wrote: >> >> So I'm +1 for the Michael's current patch as (I think) we can't >> make visible or large changes. >> >> That said, I agree with Stephen's concern on the point we could >> omit

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-10-30 Thread Stephen Frost
Greetings, * Kyotaro HORIGUCHI (horiguchi.kyot...@lab.ntt.co.jp) wrote: > At Wed, 24 Oct 2018 14:31:37 +0900, Michael Paquier > wrote in <20181024053137.gl1...@paquier.xyz> > > On Sun, Oct 21, 2018 at 08:56:32PM -0400, Stephen Frost wrote: > > > All of this pie-in-the-sky about what pluggable

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-10-25 Thread Kyotaro HORIGUCHI
Mmm. It took too long time than expected because I was repeatedly teased by git.. At Wed, 24 Oct 2018 14:31:37 +0900, Michael Paquier wrote in <20181024053137.gl1...@paquier.xyz> > On Sun, Oct 21, 2018 at 08:56:32PM -0400, Stephen Frost wrote: > > All of this pie-in-the-sky about what pluggable

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-10-23 Thread Michael Paquier
On Sun, Oct 21, 2018 at 08:56:32PM -0400, Stephen Frost wrote: > All of this pie-in-the-sky about what pluggable storage might have is > just hand-waving, in my opinion, and not worth much more than that. I > hope (and suspect..) that the actual pluggable storage that's being > worked on doesn't

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-10-21 Thread Stephen Frost
Greetings, * Michael Paquier (mich...@paquier.xyz) wrote: > On Sun, Oct 21, 2018 at 11:03:30AM -0400, Stephen Frost wrote: > > This doesn't change my opinion of the bigger question though, which is > > to what extent we should be implicitly supporting extensions and > > whatever else putting

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-10-21 Thread Michael Paquier
On Sun, Oct 21, 2018 at 11:03:30AM -0400, Stephen Frost wrote: > This doesn't change my opinion of the bigger question though, which is > to what extent we should be implicitly supporting extensions and > whatever else putting files into the database and tablespace > directories. Well, the whole

Re: More issues with pg_verify_checksums and checksum verification in base backups

2018-10-21 Thread Stephen Frost
Greetings, * Michael Paquier (mich...@paquier.xyz) wrote: > This is a follow-up of the following thread: > https://www.postgresql.org/message-id/20181012010411.re53cwcistcpi...@alap3.anarazel.de Thanks for starting this thread Michael. > pg_verify_checksums used first a blacklist to decide if