Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Greg Stark
On Tue, Aug 19, 2014 at 10:42 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Is there a way to create a link to a file which only exists as an open file descriptor? If there was, you could create a temp file, open an fd, then delete the file. That would remove the issue with files being

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Greg Stark
c.f.: O_TMPFILE (since Linux 3.11) Create an unnamed temporary file. The pathname argument specifies a directory; an unnamed inode will be created in that directory's filesystem. Anything written to the resulting file will be lost

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Alvaro Herrera
Greg Stark wrote: char path[PATH_MAX]; fd = open(/path/to/dir, O_TMPFILE | O_RDWR, S_IRUSR | S_IWUSR); /* File I/O on 'fd'... */ snprintf(path, PATH_MAX, /proc/self/fd/%d,

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Andres Freund
On 2014-08-19 17:42:06 -0400, Alvaro Herrera wrote: MauMau wrote: With that said, copying to a temporary file like dest.tmp and renaming it to dest sounds worthwhile even as a basic copy utility. I want to avoid copying to a temporary file with a fixed name like _copy.tmp, because some

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Alvaro Herrera
Andres Freund wrote: On 2014-08-19 17:42:06 -0400, Alvaro Herrera wrote: MauMau wrote: With that said, copying to a temporary file like dest.tmp and renaming it to dest sounds worthwhile even as a basic copy utility. I want to avoid copying to a temporary file with a fixed name

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Greg Stark
On Wed, Aug 20, 2014 at 2:27 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Hmm, the real trick here is linkat(... /proc/self/foobar), not the O_TMPFILE: you can have an open file descriptor to an invisible file simply by creating a normal file and unlinking it. I looked at linkat()

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Andres Freund
On 2014-08-20 09:50:56 -0400, Alvaro Herrera wrote: Andres Freund wrote: On 2014-08-19 17:42:06 -0400, Alvaro Herrera wrote: MauMau wrote: With that said, copying to a temporary file like dest.tmp and renaming it to dest sounds worthwhile even as a basic copy utility. I want

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Alvaro Herrera
Greg Stark wrote: On Wed, Aug 20, 2014 at 2:27 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Hmm, the real trick here is linkat(... /proc/self/foobar), not the O_TMPFILE: you can have an open file descriptor to an invisible file simply by creating a normal file and unlinking it. I

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-08-20 09:50:56 -0400, Alvaro Herrera wrote: Andres Freund wrote: Isn't this a solution looking for a problem? We're using tempfiles in dozens of other places and I really don't see why this is the place to stop doing so. Just copy to

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Andres Freund
On 2014-08-20 10:19:33 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-08-20 09:50:56 -0400, Alvaro Herrera wrote: Andres Freund wrote: Isn't this a solution looking for a problem? We're using tempfiles in dozens of other places and I really don't see why this

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-08-20 10:19:33 -0400, Tom Lane wrote: Alternatively, you could use the process PID as part of the temp file name; which is probably a good idea anyway. I think that's actually worse, because nothing will clean up those unless you

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Bruce Momjian
On Wed, Aug 20, 2014 at 10:36:40AM -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-08-20 10:19:33 -0400, Tom Lane wrote: Alternatively, you could use the process PID as part of the temp file name; which is probably a good idea anyway. I think that's actually

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-20 Thread Andres Freund
On 2014-08-20 18:58:05 -0400, Bruce Momjian wrote: On Wed, Aug 20, 2014 at 10:36:40AM -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-08-20 10:19:33 -0400, Tom Lane wrote: Alternatively, you could use the process PID as part of the temp file name; which is

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-19 Thread MauMau
From: Fujii Masao masao.fu...@gmail.com What's the main purpose of this tool? If it's for WAL archiving, the tool name pg_copy sounds too generic. We already have pg_archivecleanup, so maybe pg_archivecopy or something is better for the consistency? pg_copy in the patch copies the file to the

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-19 Thread Peter Eisentraut
On 8/19/14 9:35 AM, MauMau wrote: pg_copy is for copying a file reliably by syncing. If sync is not necessary, people can use cp/copy. I'm getting mixed messages from this thread. I think there could be a fair amount of support for a new tool that can serve as a universal plug-and-play

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-19 Thread Peter Eisentraut
On 8/15/14 10:46 AM, Fujii Masao wrote: At last, the big question is, is there really no OS command which provides the same functionality as pg_copy does? If there is, I'd like to avoid duplicate work basically. If you look hard enough, you can maybe find an OS command that can fsync a file

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-19 Thread Josh Berkus
On 08/19/2014 12:37 PM, Peter Eisentraut wrote: On 8/19/14 9:35 AM, MauMau wrote: pg_copy is for copying a file reliably by syncing. If sync is not necessary, people can use cp/copy. I'm getting mixed messages from this thread. I think there could be a fair amount of support for a new

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-19 Thread Alvaro Herrera
MauMau wrote: With that said, copying to a temporary file like dest.tmp and renaming it to dest sounds worthwhile even as a basic copy utility. I want to avoid copying to a temporary file with a fixed name like _copy.tmp, because some advanced utility may want to run multiple instances of

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-15 Thread Fujii Masao
On Thu, Aug 14, 2014 at 1:52 PM, MauMau maumau...@gmail.com wrote: I fixed some minor mistakes. What's the main purpose of this tool? If it's for WAL archiving, the tool name pg_copy sounds too generic. We already have pg_archivecleanup, so maybe pg_archivecopy or something is better for the

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-15 Thread Kevin Grittner
Fujii Masao masao.fu...@gmail.com wrote: Direct I/O and posix_fadvise are used only for destination file. But why not source file? That might be useful especially for restore_command case. That would prevent people from piping the file through a compression utility.  We should support piped

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-13 Thread MauMau
I fixed some minor mistakes. Regards MauMau pg_copy_v3.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-06-17 Thread MauMau
Hello, As I proposed before in the thread below, I've implemented a simple command for reliable WAL archiving. I would appreciate it if you could review and test the patch. http://www.postgresql.org/message-id/9C1EB95CA1F34DAB93DF549A51E3E874@maumau Regards MauMau pg_copy.patch

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-06-17 Thread Abhijit Menon-Sen
At 2014-06-17 23:26:37 +0900, maumau...@gmail.com wrote: Hello, As I proposed before in the thread below, I've implemented a simple command for reliable WAL archiving. I would appreciate it if you could review and test the patch. Please add your patch to the next CF (i.e. 2014-08), so

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-06-17 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/17/2014 07:26 AM, MauMau wrote: Hello, As I proposed before in the thread below, I've implemented a simple command for reliable WAL archiving. I would appreciate it if you could review and test the patch.

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-06-17 Thread Abhijit Menon-Sen
At 2014-06-17 08:02:59 -0700, m...@joeconway.com wrote: That first hunk refers to dblink -- I'm guessing it does not belong with this patch. Yes, it's a leftover of the dblink memory leak patch that's in this CF. -- Abhijit -- Sent via pgsql-hackers mailing list

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-06-17 Thread MauMau
From: Joe Conway m...@joeconway.com That first hunk refers to dblink -- I'm guessing it does not belong with this patch. Ouch, what a careless mistake. The attached one is clean. I'll update the CommitFest entry when I'm back home from work. Regards MauMau pg_copy_v2.patch Description: