Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

Michael Paquier writes: > Oops. Are you using gcc to see that? I compiled with clang and on > Windows without noticing it. Peter already noted that you'd only see it on platforms that define PG_FLUSH_DATA_WORKS. regards, tom lane -- Sent via

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On Fri, Sep 30, 2016 at 1:31 AM, Jeff Janes wrote: > On Thu, Sep 29, 2016 at 8:33 AM, Peter Eisentraut > wrote: >> >> On 9/26/16 10:34 PM, Michael Paquier wrote: >> > I thought that as long as the error string is shown to the user, it >> >

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On Thu, Sep 29, 2016 at 8:33 AM, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 9/26/16 10:34 PM, Michael Paquier wrote: > > I thought that as long as the error string is shown to the user, it > > does not matter much if errno is still saved or not. All the callers > > of

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On 9/26/16 10:34 PM, Michael Paquier wrote: > I thought that as long as the error string is shown to the user, it > does not matter much if errno is still saved or not. All the callers > of durable_rename() on frontends don't check for strerrno(errno) > afterwards. Do you think it matters?

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On Tue, Sep 27, 2016 at 11:16 AM, Peter Eisentraut wrote: > On 9/23/16 11:15 AM, Michael Paquier wrote: >>> 0002: >>> > >>> > durable_rename needs close(fd) in non-error path (compare backend code). >> Oops, leak. > > Why did you remove the errno save and

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On 9/23/16 11:15 AM, Michael Paquier wrote: >> 0002: >> > >> > durable_rename needs close(fd) in non-error path (compare backend code). > Oops, leak. Why did you remove the errno save and restore? -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support,

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On Fri, Sep 23, 2016 at 10:54 AM, Peter Eisentraut wrote: > This is looking pretty good. More comments on this patch set: Thanks for the review. > 0001: > > Keep the file order alphabetical in Mkvcbuild.pm. > > Needs nls.mk updates for file move (in initdb and

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

This is looking pretty good. More comments on this patch set: 0001: Keep the file order alphabetical in Mkvcbuild.pm. Needs nls.mk updates for file move (in initdb and pg_basebackup directories). 0002: durable_rename needs close(fd) in non-error path (compare backend code). Changing from

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On Thu, Sep 15, 2016 at 9:44 AM, Peter Eisentraut wrote: > On 9/12/16 11:16 PM, Michael Paquier wrote: >>> I don't think tar file output in pg_basebackup needs to be fsynced. >>> > It's just a backup file like what pg_dump produces, and we don't fsync >>> > that

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On 9/12/16 11:16 PM, Michael Paquier wrote: >> I don't think tar file output in pg_basebackup needs to be fsynced. >> > It's just a backup file like what pg_dump produces, and we don't fsync >> > that either. The point of this change is to leave a data directory in >> > a synced state equivalent

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On Sat, Sep 10, 2016 at 6:27 PM, Peter Eisentraut wrote: > In fsync_pgdata(), you have removed the progname from one error > message, even though it is passed into the function. Right. Good catch. > Also, I think > fsync_pgdata() should not be printing initdb's

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On Tue, Sep 13, 2016 at 10:37 AM, Andres Freund wrote: > On 2016-09-13 10:35:38 +0900, Michael Paquier wrote: >> On Sat, Sep 10, 2016 at 7:36 PM, Craig Ringer >> wrote: >> > We need it for tap tests. More and more will use pg_basebackup and it'll

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On 2016-09-13 10:35:38 +0900, Michael Paquier wrote: > On Sat, Sep 10, 2016 at 7:36 PM, Craig Ringer > wrote: > > We need it for tap tests. More and more will use pg_basebackup and it'll > > start hurting test speeds badly. > > Ah yes, that's a good argument.

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On Sat, Sep 10, 2016 at 7:36 PM, Craig Ringer wrote: > We need it for tap tests. More and more will use pg_basebackup and it'll > start hurting test speeds badly. Ah yes, that's a good argument. hamster would suffer pretty badly after that if nothing is done. I'll

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On 3 Sep. 2016 9:22 pm, "Michael Paquier" wrote: > > On Sat, Sep 3, 2016 at 12:42 AM, Magnus Hagander wrote: > > On Fri, Sep 2, 2016 at 8:50 AM, Michael Paquier < michael.paqu...@gmail.com> > > wrote: > >> On Fri, Sep 2, 2016 at 2:20 AM, Peter

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On 9/3/16 9:23 AM, Michael Paquier wrote: > On Sat, Sep 3, 2016 at 10:22 PM, Michael Paquier > wrote: >> On Sat, Sep 3, 2016 at 10:21 PM, Michael Paquier >> wrote: >>> Oh, well. I have just implemented it on top of the two other patches >>>

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On Sat, Sep 3, 2016 at 10:26 PM, Magnus Hagander wrote: > Yes, we should definitely not allow that combination. In fact, maybe that > argument in itself is enough not to have it for pg_receivexlog -- the cause > of confusion. > > I can see how you might want to avoid it for

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On Sat, Sep 3, 2016 at 3:21 PM, Michael Paquier wrote: > On Sat, Sep 3, 2016 at 12:42 AM, Magnus Hagander > wrote: > > On Fri, Sep 2, 2016 at 8:50 AM, Michael Paquier < > michael.paqu...@gmail.com> > > wrote: > >> On Fri, Sep 2, 2016 at 2:20 AM,

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On Sat, Sep 3, 2016 at 10:22 PM, Michael Paquier wrote: > On Sat, Sep 3, 2016 at 10:21 PM, Michael Paquier > wrote: >> Oh, well. I have just implemented it on top of the two other patches >> for pg_basebackup. For pg_receivexlog, I am

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On Sat, Sep 3, 2016 at 10:21 PM, Michael Paquier wrote: > On Sat, Sep 3, 2016 at 12:42 AM, Magnus Hagander wrote: >> On Fri, Sep 2, 2016 at 8:50 AM, Michael Paquier >> wrote: >>> On Fri, Sep 2, 2016 at 2:20 AM, Peter

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On Sat, Sep 3, 2016 at 12:42 AM, Magnus Hagander wrote: > On Fri, Sep 2, 2016 at 8:50 AM, Michael Paquier > wrote: >> On Fri, Sep 2, 2016 at 2:20 AM, Peter Eisentraut >> wrote: >> > On 5/13/16 2:39 AM, Michael

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On Fri, Sep 2, 2016 at 8:50 AM, Michael Paquier wrote: > On Fri, Sep 2, 2016 at 2:20 AM, Peter Eisentraut > wrote: > > On 5/13/16 2:39 AM, Michael Paquier wrote: > >> So, attached are two patches that apply on HEAD to address the

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On Fri, Sep 2, 2016 at 2:20 AM, Peter Eisentraut wrote: > On 5/13/16 2:39 AM, Michael Paquier wrote: >> So, attached are two patches that apply on HEAD to address the problem >> of pg_basebackup that does not sync the data it writes. As >> pg_basebackup cannot

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On 5/13/16 2:39 AM, Michael Paquier wrote: > So, attached are two patches that apply on HEAD to address the problem > of pg_basebackup that does not sync the data it writes. As > pg_basebackup cannot use directly initdb -S because, as a client-side > utility, it may be installed while initdb is

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On Fri, May 13, 2016 at 11:49 PM, Alex Ignatov wrote: > > On 13.05.2016 9:39, Michael Paquier wrote: > Do we have any confidence that data file is not being corrupted? I.e > contains some corrupted page? Can pg_basebackup check page checksum (db init > with initdb -k)

Re: [HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

On 13.05.2016 9:39, Michael Paquier wrote: Hi all, Beginning a new thread because the ext4 issues are closed, and because pg_basebackup data durability meritates a new thread. And in short about the problem: pg_basebackup makes no effort in being sure that the data it backs up is on disk,

[HACKERS] pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)

Hi all, Beginning a new thread because the ext4 issues are closed, and because pg_basebackup data durability meritates a new thread. And in short about the problem: pg_basebackup makes no effort in being sure that the data it backs up is on disk, which is bad... One possible recommendation is to