Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Andres Freund
On 2014-08-20 18:58:05 -0400, Bruce Momjian wrote: > On Wed, Aug 20, 2014 at 10:36:40AM -0400, Tom Lane wrote: > > Andres Freund writes: > > > On 2014-08-20 10:19:33 -0400, Tom Lane wrote: > > >> Alternatively, you could use the process PID as part of the temp file > > >> name; which is probably a

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Bruce Momjian
On Wed, Aug 20, 2014 at 10:36:40AM -0400, Tom Lane wrote: > Andres Freund writes: > > On 2014-08-20 10:19:33 -0400, Tom Lane wrote: > >> Alternatively, you could use the process PID as part of the temp file > >> name; which is probably a good idea anyway. > > > I think that's actually worse, beca

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Tom Lane
Andres Freund writes: > On 2014-08-20 10:19:33 -0400, Tom Lane wrote: >> Alternatively, you could use the process PID as part of the temp file >> name; which is probably a good idea anyway. > I think that's actually worse, because nothing will clean up those > unless you explicitly scan for all .

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Andres Freund
On 2014-08-20 10:19:33 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2014-08-20 09:50:56 -0400, Alvaro Herrera wrote: > >> Andres Freund wrote: > >>> Isn't this a solution looking for a problem? We're using tempfiles in > >>> dozens of other places and I really don't see why this is the pl

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Tom Lane
Andres Freund writes: > On 2014-08-20 09:50:56 -0400, Alvaro Herrera wrote: >> Andres Freund wrote: >>> Isn't this a solution looking for a problem? We're using tempfiles in >>> dozens of other places and I really don't see why this is the place to >>> stop doing so. Just copy to .tmp and move it

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Alvaro Herrera
Greg Stark wrote: > On Wed, Aug 20, 2014 at 2:27 PM, Alvaro Herrera > wrote: > > Hmm, the real trick here is linkat(... "/proc/self/foobar"), not the > > O_TMPFILE: you can have an open file descriptor to an "invisible" file > > simply by creating a normal file and unlinking it. I looked at linka

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Andres Freund
On 2014-08-20 09:50:56 -0400, Alvaro Herrera wrote: > Andres Freund wrote: > > On 2014-08-19 17:42:06 -0400, Alvaro Herrera wrote: > > > MauMau wrote: > > > > > > > With that said, copying to a temporary file like .tmp and > > > > renaming it to sounds worthwhile even as a basic copy > > > > util

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Greg Stark
On Wed, Aug 20, 2014 at 2:27 PM, Alvaro Herrera wrote: > Hmm, the real trick here is linkat(... "/proc/self/foobar"), not the > O_TMPFILE: you can have an open file descriptor to an "invisible" file > simply by creating a normal file and unlinking it. I looked at linkat() > yesterday but the idea

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Alvaro Herrera
Andres Freund wrote: > On 2014-08-19 17:42:06 -0400, Alvaro Herrera wrote: > > MauMau wrote: > > > > > With that said, copying to a temporary file like .tmp and > > > renaming it to sounds worthwhile even as a basic copy > > > utility. I want to avoid copying to a temporary file with a fixed > >

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Andres Freund
On 2014-08-19 17:42:06 -0400, Alvaro Herrera wrote: > MauMau wrote: > > > With that said, copying to a temporary file like .tmp and > > renaming it to sounds worthwhile even as a basic copy > > utility. I want to avoid copying to a temporary file with a fixed > > name like _copy.tmp, because som

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Alvaro Herrera
Greg Stark wrote: > char path[PATH_MAX]; > fd = open("/path/to/dir", O_TMPFILE | O_RDWR, > S_IRUSR | S_IWUSR); > > /* File I/O on 'fd'... */ > > snprintf(path, PATH_MAX, "/proc/self

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Greg Stark
c.f.: O_TMPFILE (since Linux 3.11) Create an unnamed temporary file. The pathname argument specifies a directory; an unnamed inode will be created in that directory's filesystem. Anything written to the resulting file will be lost wh

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Greg Stark
On Tue, Aug 19, 2014 at 10:42 PM, Alvaro Herrera wrote: > Is there a way to create a link to a file which only exists as an open > file descriptor? If there was, you could create a temp file, open an > fd, then delete the file. That would remove the issue with files being > leaked due to failur

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-19 Thread Alvaro Herrera
MauMau wrote: > With that said, copying to a temporary file like .tmp and > renaming it to sounds worthwhile even as a basic copy > utility. I want to avoid copying to a temporary file with a fixed > name like _copy.tmp, because some advanced utility may want to run > multiple instances of pg_co

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-19 Thread Josh Berkus
On 08/19/2014 12:37 PM, Peter Eisentraut wrote: > On 8/19/14 9:35 AM, MauMau wrote: >> pg_copy is for copying a file reliably by syncing. If sync is not >> necessary, people can use cp/copy. > > I'm getting mixed messages from this thread. > > I think there could be a fair amount of support for

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-19 Thread Peter Eisentraut
On 8/15/14 10:46 AM, Fujii Masao wrote: > At last, the big question is, is there really no OS command which provides > the same functionality as pg_copy does? If there is, I'd like to avoid > duplicate > work basically. If you look hard enough, you can maybe find an OS command that can fsync a fi

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-19 Thread Peter Eisentraut
On 8/19/14 9:35 AM, MauMau wrote: > pg_copy is for copying a file reliably by syncing. If sync is not > necessary, people can use cp/copy. I'm getting mixed messages from this thread. I think there could be a fair amount of support for a new tool that can serve as a universal plug-and-play archi

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-19 Thread MauMau
From: "Fujii Masao" What's the main purpose of this tool? If it's for WAL archiving, the tool name "pg_copy" sounds too generic. We already have pg_archivecleanup, so maybe "pg_archivecopy" or something is better for the consistency? pg_copy in the patch copies the file to the destination in a

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-15 Thread Kevin Grittner
Fujii Masao wrote: > Direct I/O and posix_fadvise are used only for destination file. > But why not source file? That might be useful especially for > restore_command case. That would prevent people from piping the file through a compression utility.  We should support piped I/O for input (and i

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-15 Thread Fujii Masao
On Thu, Aug 14, 2014 at 1:52 PM, MauMau wrote: > I fixed some minor mistakes. What's the main purpose of this tool? If it's for WAL archiving, the tool name "pg_copy" sounds too generic. We already have pg_archivecleanup, so maybe "pg_archivecopy" or something is better for the consistency? pg_c

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-13 Thread MauMau
I fixed some minor mistakes. Regards MauMau pg_copy_v3.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-06-17 Thread MauMau
From: "Joe Conway" That first hunk refers to dblink -- I'm guessing it does not belong with this patch. Ouch, what a careless mistake. The attached one is clean. I'll update the CommitFest entry when I'm back home from work. Regards MauMau pg_copy_v2.patch Description: Binary data --

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-06-17 Thread Abhijit Menon-Sen
At 2014-06-17 08:02:59 -0700, m...@joeconway.com wrote: > > That first hunk refers to dblink -- I'm guessing it does not belong > with this patch. Yes, it's a leftover of the dblink memory leak patch that's in this CF. -- Abhijit -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-06-17 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/17/2014 07:26 AM, MauMau wrote: > Hello, > > As I proposed before in the thread below, I've implemented a simple > command for reliable WAL archiving. I would appreciate it if you could > review and test the patch. > > http://www.postgresql.or

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-06-17 Thread Abhijit Menon-Sen
At 2014-06-17 23:26:37 +0900, maumau...@gmail.com wrote: > > Hello, > > As I proposed before in the thread below, I've implemented a simple > command for reliable WAL archiving. I would appreciate it if you > could review and test the patch. Please add your patch to the next CF (i.e. 2014-08), s

[HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-06-17 Thread MauMau
Hello, As I proposed before in the thread below, I've implemented a simple command for reliable WAL archiving. I would appreciate it if you could review and test the patch. http://www.postgresql.org/message-id/9C1EB95CA1F34DAB93DF549A51E3E874@maumau Regards MauMau pg_copy.patch Descriptio