Re: [DOCS] Sample archive_command is still problematic

2014-08-21 Thread Josh Berkus
On 08/21/2014 12:20 PM, Bruce Momjian wrote: > On Thu, Aug 21, 2014 at 11:52:07AM -0700, Josh Berkus wrote: >> On 08/20/2014 04:12 PM, Bruce Momjian wrote: >>> On Tue, Aug 19, 2014 at 11:27:47AM -0700, Josh Berkus wrote: I'd also like to mention pg_receivexlog, Barman, WAL-E, and RepMgr as >>>

Re: [DOCS] Sample archive_command is still problematic

2014-08-21 Thread Bruce Momjian
On Thu, Aug 21, 2014 at 11:52:07AM -0700, Josh Berkus wrote: > On 08/20/2014 04:12 PM, Bruce Momjian wrote: > > On Tue, Aug 19, 2014 at 11:27:47AM -0700, Josh Berkus wrote: > >> I'd also like to mention pg_receivexlog, Barman, WAL-E, and RepMgr as > >> external utilites and alternatives. > > > > Y

Re: [DOCS] Sample archive_command is still problematic

2014-08-21 Thread Josh Berkus
On 08/20/2014 04:12 PM, Bruce Momjian wrote: > On Tue, Aug 19, 2014 at 11:27:47AM -0700, Josh Berkus wrote: >> I'd also like to mention pg_receivexlog, Barman, WAL-E, and RepMgr as >> external utilites and alternatives. > > Yes, please! We need to give users on what else is available, and the > s

Re: [DOCS] Sample archive_command is still problematic

2014-08-20 Thread Bruce Momjian
On Tue, Aug 19, 2014 at 11:27:47AM -0700, Josh Berkus wrote: > I'd also like to mention pg_receivexlog, Barman, WAL-E, and RepMgr as > external utilites and alternatives. Yes, please! We need to give users on what else is available, and the strengths of each. -- Bruce Momjian http://

Re: [DOCS] Sample archive_command is still problematic

2014-08-19 Thread Josh Berkus
On 08/19/2014 10:39 AM, Tom Lane wrote: > Josh Berkus writes: >> Well, if we want a sample command in the docs, then we should actually >> give a few different sample commands and even a sample shell script. If >> we give the users several alternatives, maybe they'll actually think >> about it in

Re: [DOCS] Sample archive_command is still problematic

2014-08-19 Thread Tom Lane
Josh Berkus writes: > Well, if we want a sample command in the docs, then we should actually > give a few different sample commands and even a sample shell script. If > we give the users several alternatives, maybe they'll actually think > about it instead of just C&P. Sure, if we can think of s

Re: [DOCS] Sample archive_command is still problematic

2014-08-19 Thread Josh Berkus
On 08/18/2014 10:31 AM, Tom Lane wrote: > Josh Berkus writes: >> 2) One reason users are using the "test -f" version of the archive >> command is that we put it in the same postgresql.conf. I would suggest >> that we don't put *any* archive command in the sample postgresql.conf, >> since there is

Re: [DOCS] Sample archive_command is still problematic

2014-08-18 Thread Tom Lane
Josh Berkus writes: > 2) One reason users are using the "test -f" version of the archive > command is that we put it in the same postgresql.conf. I would suggest > that we don't put *any* archive command in the sample postgresql.conf, > since there is no command we can supply which isn't a potent

Re: [DOCS] Sample archive_command is still problematic

2014-08-18 Thread Magnus Hagander
On Mon, Aug 18, 2014 at 7:12 PM, Josh Berkus wrote: > Magnus, Kevin: > > Two things: > > 1) pg_receivexlog doesn't help for users who need to archive from the > master to cold storage (e.g. remote SAN, S3, whatever). So we're going > to still need an archive_command. It can, but it's definitely

Re: [DOCS] Sample archive_command is still problematic

2014-08-18 Thread Josh Berkus
Magnus, Kevin: Two things: 1) pg_receivexlog doesn't help for users who need to archive from the master to cold storage (e.g. remote SAN, S3, whatever). So we're going to still need an archive_command. 2) One reason users are using the "test -f" version of the archive command is that we put it

Re: [DOCS] Sample archive_command is still problematic

2014-08-17 Thread Magnus Hagander
On Sun, Aug 17, 2014 at 9:50 PM, Kevin Grittner wrote: > Magnus Hagander wrote: > >> On Wed, Aug 13, 2014 at 11:23 PM, Kevin Grittner wrote: > >> >>> The above is regarding WAL file archiving -- I'm not putting down >>> streaming replication. Of course, what I would have *really* liked >>> is a

Re: [DOCS] Sample archive_command is still problematic

2014-08-17 Thread Kevin Grittner
Magnus Hagander wrote: > On Wed, Aug 13, 2014 at 11:23 PM, Kevin Grittner wrote: > >> The above is regarding WAL file archiving -- I'm not putting down >> streaming replication.  Of course, what I would have *really* liked >> is a WAL receiver that could write out normal-looking WAL files for >

Re: [DOCS] Sample archive_command is still problematic

2014-08-16 Thread Magnus Hagander
On Wed, Aug 13, 2014 at 11:23 PM, Kevin Grittner wrote: > The above is regarding WAL file archiving -- I'm not putting down > streaming replication. Of course, what I would have *really* liked > is a WAL receiver that could write out normal-looking WAL files for > archiving purposes and pass thr

Re: [DOCS] Sample archive_command is still problematic

2014-08-13 Thread MauMau
From: "Peter Eisentraut" I realize that there are about 128 different ways people set this up (which is itself a problem), but it appears to me that a solution like pg_copy only provides local copying, which implies the use of something like NFS. Which may be OK, but then we'd need to get into

Re: [DOCS] Sample archive_command is still problematic

2014-08-13 Thread MauMau
From: "Josh Berkus" Yah? Does it work on Windows? Yes. pg_copy is meant to be a replacement for cp/copy, not for the entire archive_command script. It just opens, reads, writes, syncs, and closes the file. Regards MauMau -- Sent via pgsql-docs mailing list (pgsql-docs@postgresql.org)

Re: [DOCS] Sample archive_command is still problematic

2014-08-13 Thread Kevin Grittner
Peter Eisentraut wrote: > On 8/11/14 6:23 PM, MauMau wrote: > >> I submitted a patch a patch for this a few months ago, which is pg_copy >> listed in the current CF.  The patch also addresses the problem that the >> archived file can get lost after power failure because it is not flushed >> to dis

Re: [DOCS] Sample archive_command is still problematic

2014-08-13 Thread Peter Eisentraut
On 8/11/14 6:23 PM, MauMau wrote: > I submitted a patch a patch for this a few months ago, which is pg_copy > listed in the current CF. The patch also addresses the problem that the > archived file can get lost after power failure because it is not flushed > to disk.The patch consists of a pro

Re: [DOCS] Sample archive_command is still problematic

2014-08-13 Thread Josh Berkus
On 08/11/2014 03:23 PM, MauMau wrote: > From: "Kevin Grittner" > The problem with the recommended command is that cp is not atomic. > The file can be read before the contents are materialized, causing > early end to recovery. I have seen it happen. The right way to do > this is to copy to a differ

Re: [DOCS] Sample archive_command is still problematic

2014-08-11 Thread Josh Berkus
On 08/11/2014 12:49 PM, Tom Lane wrote: > Kevin Grittner writes: >> Josh Berkus wrote: >>> Yeah, realistically, I think we need to start supplying a script or two >>> in /contrib and referencing that.� I'm not sure how to make it work for >>> the Windows users though. > >> That might work.� We s

Re: [DOCS] Sample archive_command is still problematic

2014-08-11 Thread MauMau
From: "Kevin Grittner" The problem with the recommended command is that cp is not atomic. The file can be read before the contents are materialized, causing early end to recovery. I have seen it happen. The right way to do this is to copy to a different name or directory and mv the file into plac

Re: [DOCS] Sample archive_command is still problematic

2014-08-11 Thread Tom Lane
Kevin Grittner writes: > Josh Berkus wrote: >> Yeah, realistically, I think we need to start supplying a script or two >> in /contrib and referencing that.  I'm not sure how to make it work for >> the Windows users though. > That might work.  We should do something, though.  The example we > giv

Re: [DOCS] Sample archive_command is still problematic

2014-08-11 Thread Kevin Grittner
Josh Berkus wrote: > On 08/11/2014 10:21 AM, Kevin Grittner wrote: >>> Is there some good reason why "test ! -f" was added to the >>> sample? >> >> In an environment with more than one cluster archiving, it is >> otherwise way too easy to copy a config file and have the WAL files >> of the two sys

Re: [DOCS] Sample archive_command is still problematic

2014-08-11 Thread Josh Berkus
On 08/11/2014 10:21 AM, Kevin Grittner wrote: >> Is there some good reason why "test ! -f" was added to the >> sample? > > In an environment with more than one cluster archiving, it is > otherwise way too easy to copy a config file and have the WAL files > of the two systems overwriting one anothe

Re: [DOCS] Sample archive_command is still problematic

2014-08-11 Thread Kevin Grittner
Josh Berkus wrote: > The example archive_command we give in the docs is this one: > > archive_command = 'test ! -f /mnt/server/archivedir/%f && cp %p > /mnt/server/archivedir/%f' > > This is a problematic recommendation. I agree with that statement, ... > If there's any reason why copying the

[DOCS] Sample archive_command is still problematic

2014-08-11 Thread Josh Berkus
All: The example archive_command we give in the docs is this one: archive_command = 'test ! -f /mnt/server/archivedir/%f && cp %p /mnt/server/archivedir/%f' This is a problematic recommendation. If there's any reason why copying the archive file gets interrupted (storage blip, for example), the