Re: [HACKERS] pg_dump, pg_dumpall and data durability

2017-03-23 Thread Michael Paquier
On Wed, Mar 22, 2017 at 7:00 AM, Michael Paquier wrote: > On Wed, Mar 22, 2017 at 6:24 AM, Andrew Dunstan > wrote: >> This is really a pretty small patch all things considered, and pretty >> low-risk (although I haven;t been threough the code in fine detail yet). >> In the end I'm persuaded by An

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2017-03-21 Thread Michael Paquier
On Wed, Mar 22, 2017 at 6:24 AM, Andrew Dunstan wrote: > This is really a pretty small patch all things considered, and pretty > low-risk (although I haven;t been threough the code in fine detail yet). > In the end I'm persuaded by Andres' point that there's actually no > practical alternative way

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2017-03-21 Thread Andrew Dunstan
On 03/04/2017 01:08 AM, Robert Haas wrote: > On Thu, Mar 2, 2017 at 5:02 AM, Michael Paquier > wrote: >> On Thu, Mar 2, 2017 at 2:26 AM, David Steele wrote: >>> This patch is in need of a committer. Any takers? >>> I didn't see a lot of enthusiasm from committers on the thread >> Stephen at le

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2017-03-04 Thread Michael Paquier
On Sat, Mar 4, 2017 at 3:08 PM, Robert Haas wrote: > On Thu, Mar 2, 2017 at 5:02 AM, Michael Paquier > wrote: >> On Thu, Mar 2, 2017 at 2:26 AM, David Steele wrote: >>> This patch is in need of a committer. Any takers? >>> I didn't see a lot of enthusiasm from committers on the thread >> >> Ste

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2017-03-03 Thread Robert Haas
On Thu, Mar 2, 2017 at 5:02 AM, Michael Paquier wrote: > On Thu, Mar 2, 2017 at 2:26 AM, David Steele wrote: >> This patch is in need of a committer. Any takers? >> I didn't see a lot of enthusiasm from committers on the thread > > Stephen at least has showed interest. > >> so if nobody picks it

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2017-03-01 Thread Michael Paquier
On Thu, Mar 2, 2017 at 2:26 AM, David Steele wrote: > This patch is in need of a committer. Any takers? > I didn't see a lot of enthusiasm from committers on the thread Stephen at least has showed interest. > so if nobody picks it up by the end of the CF I'm going to mark the patch RWF. Yes, t

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2017-03-01 Thread David Steele
On 1/22/17 11:02 PM, Michael Paquier wrote: > On Tue, Nov 29, 2016 at 1:30 PM, Michael Paquier > wrote: >> On Mon, Nov 14, 2016 at 6:07 PM, Michael Paquier >> wrote: >>> On Mon, Nov 14, 2016 at 6:04 PM, Albe Laurenz >>> wrote: Michael Paquier wrote: > Meh. I forgot docs and --help outp

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2017-01-22 Thread Michael Paquier
On Tue, Nov 29, 2016 at 1:30 PM, Michael Paquier wrote: > On Mon, Nov 14, 2016 at 6:07 PM, Michael Paquier > wrote: >> On Mon, Nov 14, 2016 at 6:04 PM, Albe Laurenz >> wrote: >>> Michael Paquier wrote: Meh. I forgot docs and --help output updates. >>> >>> Looks good, except that you left t

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-12-06 Thread Michael Paquier
On Tue, Dec 6, 2016 at 6:00 PM, Fujii Masao wrote: > One idea is to provide the utility or extension which fsync's the specified > files and directories (including all the files under them). It may be > overkill, > though. But it would be useful for other purposes, for example, we can > improve t

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-12-06 Thread Fujii Masao
On Wed, Nov 16, 2016 at 1:27 AM, Robert Haas wrote: > On Sun, Nov 13, 2016 at 4:18 AM, Andres Freund wrote: >> On 2016-11-08 18:18:01 -0500, Tom Lane wrote: >>> I think this might be better addressed by adding something to backup.sgml >>> pointing out that you'd better fsync or sync your backups

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-28 Thread Michael Paquier
On Mon, Nov 14, 2016 at 6:07 PM, Michael Paquier wrote: > On Mon, Nov 14, 2016 at 6:04 PM, Albe Laurenz wrote: >> Michael Paquier wrote: >>> Meh. I forgot docs and --help output updates. >> >> Looks good, except that you left the "N" option in the getopt_long >> call for pg_dumpall. I fixed that

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-15 Thread Robert Haas
On Sun, Nov 13, 2016 at 4:18 AM, Andres Freund wrote: > On 2016-11-08 18:18:01 -0500, Tom Lane wrote: >> I think this might be better addressed by adding something to backup.sgml >> pointing out that you'd better fsync or sync your backups before assuming >> that they can't be lost. > > How does a

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-14 Thread Michael Paquier
On Mon, Nov 14, 2016 at 6:04 PM, Albe Laurenz wrote: > Michael Paquier wrote: >> Meh. I forgot docs and --help output updates. > > Looks good, except that you left the "N" option in the getopt_long > call for pg_dumpall. I fixed that in the attached patch. No, v5 has removed it, but it does not

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-14 Thread Albe Laurenz
Michael Paquier wrote: > Meh. I forgot docs and --help output updates. Looks good, except that you left the "N" option in the getopt_long call for pg_dumpall. I fixed that in the attached patch. I'll mark the patch "ready for committer". Yours, Laurenz Albe pgdump-sync-v5.patch Description: p

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-13 Thread Andres Freund
Hi, On 2016-11-08 18:18:01 -0500, Tom Lane wrote: > I think this might be better addressed by adding something to backup.sgml > pointing out that you'd better fsync or sync your backups before assuming > that they can't be lost. How does a normal user do that? I don't think there's a cross-platfo

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-12 Thread Peter Eisentraut
On 11/8/16 3:48 PM, Robert Haas wrote: > First question: Do we even want this? Generally, when a program > writes to a file, we rely on the operating system to decide when that > data should be written back to disk. We have to override that > distinction for things internal to PostgreSQL because

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-12 Thread Michael Paquier
On Sun, Nov 13, 2016 at 12:17 AM, Dilip Kumar wrote: > On Sat, Nov 12, 2016 at 10:16 AM, Michael Paquier > wrote: >>> - For pg_dumpall, a short option "-N" is added for "--no-sync", but not for >>> pg_dump (because -N is already taken there). >>> I'd opt for either using the same short option

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-12 Thread Dilip Kumar
On Sat, Nov 12, 2016 at 10:16 AM, Michael Paquier wrote: >> - For pg_dumpall, a short option "-N" is added for "--no-sync", but not for >> pg_dump (because -N is already taken there). >> I'd opt for either using the same short option for both or (IMO better) >> only offering a long option fo

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-12 Thread Michael Paquier
On Sat, Nov 12, 2016 at 1:46 PM, Michael Paquier wrote: > On Fri, Nov 11, 2016 at 11:03 PM, Albe Laurenz > wrote: >> This would avoid confusion, and we expect that few people will want to use >> this option anyway, right? > > Definitely a good point. Meh. I forgot docs and --help output upd

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-11 Thread Michael Paquier
On Fri, Nov 11, 2016 at 11:03 PM, Albe Laurenz wrote: > - In pg_dumpall.c, the result of fsync_fname() is cast to "void" to show that > the return code is ignored, but not anywhere else. Is that by design? Right. The patch is lacking consistency in this area. The main thought regarding this de

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-11 Thread Albe Laurenz
Michael Paquier wrote: > A typo s/pre_fsync_fname/pre_sync_fname, and a mistake from me because > I did not compile with -DPG_FLUSH_DATA_WORKS to check this code. > > v2 is attached, fixing those issues. The patch applies and compiles fine. I have tested it on Linux and MinGW and could see the f

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-08 Thread Tom Lane
Robert Haas writes: > First question: Do we even want this? Generally, when a program > writes to a file, we rely on the operating system to decide when that > data should be written back to disk. We have to override that > distinction for things internal to PostgreSQL because we need certain >

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-08 Thread Michael Paquier
On Wed, Nov 9, 2016 at 5:48 AM, Robert Haas wrote: > Kyotaro Horiguchi set this patch to "Ready for Committer" in the > CommitFest application, but that seems entirely premature to me > considering that v2 has had no review at all, or at least not that I > see on this thread. I'm setting it back

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-08 Thread Michael Paquier
On Wed, Nov 9, 2016 at 8:18 AM, Tom Lane wrote: > Robert Haas writes: >> First question: Do we even want this? Generally, when a program >> writes to a file, we rely on the operating system to decide when that >> data should be written back to disk. We have to override that >> distinction for t

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-08 Thread Robert Haas
On Mon, Nov 7, 2016 at 7:52 PM, Michael Paquier wrote: > On Tue, Nov 8, 2016 at 1:24 AM, Albe Laurenz wrote: >> The patch does not apply, I had to change the hunk for >> src/include/common/file_utils.h. > > Yes, the patch has rotten a bit because of f82ec32a. 5d58c07a has also > made the --noxxx

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-07 Thread Michael Paquier
On Tue, Nov 8, 2016 at 1:24 AM, Albe Laurenz wrote: > The patch does not apply, I had to change the hunk for > src/include/common/file_utils.h. Yes, the patch has rotten a bit because of f82ec32a. 5d58c07a has also made the --noxxx option names appearing as --no-xxx. > Also, compilation fails be

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-07 Thread Albe Laurenz
Michael Paquier wrote: >> In my quest of making the backup tools more compliant to data >> durability, here is a thread for pg_dump and pg_dumpall. > > Okay, here is a patch doing the above. I have added a new --nosync > option to pg_dump and pg_dumpall to switch to the pre-10 behavior. I > have a

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-10-13 Thread Michael Paquier
On Thu, Oct 13, 2016 at 2:49 PM, Michael Paquier wrote: > In my quest of making the backup tools more compliant to data > durability, here is a thread for pg_dump and pg_dumpall. Here is in a > couple of lines my proposal: > - Addition in _archiveHandle of a field to track if the dump generated >

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-10-13 Thread Michael Paquier
On Thu, Oct 13, 2016 at 2:49 PM, Michael Paquier wrote: > - This is effective for all modes, when the user specifies an output > file. In short that's when fileSpec is not NULL. I have sent the email too early here... In this case the sync is a no-op. It is missing a sentence. -- Michael -- S

[HACKERS] pg_dump, pg_dumpall and data durability

2016-10-12 Thread Michael Paquier
Hi all, In my quest of making the backup tools more compliant to data durability, here is a thread for pg_dump and pg_dumpall. Here is in a couple of lines my proposal: - Addition in _archiveHandle of a field to track if the dump generated should be synced or not. - This is effective for all modes