Re: [HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-29 Thread Jeff Janes
On Fri, Mar 21, 2014 at 2:22 PM, MauMau maumau...@gmail.com wrote: From: Jeff Janes jeff.ja...@gmail.com Do people really just copy the files from one directory of local storage to another directory of local storage? I don't see the point of that. It makes sense to archive WAL to a

Re: [HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-29 Thread Tom Lane
Jeff Janes jeff.ja...@gmail.com writes: But, it is hard to tell what the real solution is, because the doc doesn't explain why it should refuse (and fail) to overwrite an existing file. The only reason I can think of to make that recommendation is because it is easy to accidentally configure

Re: [HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-29 Thread Jeff Janes
On Saturday, March 29, 2014, Tom Lane t...@sss.pgh.pa.us wrote: Jeff Janes jeff.ja...@gmail.com javascript:; writes: But, it is hard to tell what the real solution is, because the doc doesn't explain why it should refuse (and fail) to overwrite an existing file. The only reason I can

Re: [HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-22 Thread Martijn van Oosterhout
On Sat, Mar 22, 2014 at 06:22:37AM +0900, MauMau wrote: From: Jeff Janes jeff.ja...@gmail.com Do people really just copy the files from one directory of local storage to another directory of local storage? I don't see the point of that. It makes sense to archive WAL to a directory of local

Re: [HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-21 Thread MauMau
From: Mitsumasa KONDO kondo.mitsum...@gmail.com 2014-03-17 21:12 GMT+09:00 Fujii Masao masao.fu...@gmail.com: On Mon, Mar 17, 2014 at 10:20 AM, Robert Haas robertmh...@gmail.com wrote: On Sun, Mar 16, 2014 at 6:23 AM, MauMau maumau...@gmail.com wrote: * Improve the example in the

Re: [HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-21 Thread Michael Paquier
On Fri, Mar 21, 2014 at 8:54 PM, MauMau maumau...@gmail.com wrote: * Create pg_copy in C so that it can be used on Windows as well as on UNIX/Linux. It just copies one file. Its source code is located in src/bin/pg_copy/. Please recommend a better name if you have one in mind. I'd rather see

Re: [HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-21 Thread MauMau
From: Michael Paquier michael.paqu...@gmail.com On Fri, Mar 21, 2014 at 8:54 PM, MauMau maumau...@gmail.com wrote: * Create pg_copy in C so that it can be used on Windows as well as on UNIX/Linux. It just copies one file. Its source code is located in src/bin/pg_copy/. Please recommend a

Re: [HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-21 Thread MauMau
From: Michael Paquier michael.paqu...@gmail.com On Fri, Mar 21, 2014 at 8:54 PM, MauMau maumau...@gmail.com wrote: * Create pg_copy in C so that it can be used on Windows as well as on UNIX/Linux. It just copies one file. Its source code is located in src/bin/pg_copy/. Please recommend a

Re: [HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-21 Thread Jeff Janes
On Sun, Mar 16, 2014 at 3:23 AM, MauMau maumau...@gmail.com wrote: Hello, The PostgreSQL documentation describes cp (on UNIX/Linux) or copy (on Windows) as an example for archive_command. However, cp/copy does not sync the copied data to disk. As a result, the completed WAL segments would

Re: [HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-21 Thread MauMau
From: Jeff Janes jeff.ja...@gmail.com Do people really just copy the files from one directory of local storage to another directory of local storage? I don't see the point of that. It makes sense to archive WAL to a directory of local storage for media recovery. Here, the local storage is

Re: [HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-21 Thread Bruce Momjian
On Fri, Mar 21, 2014 at 01:16:08PM -0700, Jeff Janes wrote: On Sun, Mar 16, 2014 at 3:23 AM, MauMau maumau...@gmail.com wrote: Hello, The PostgreSQL documentation describes cp (on UNIX/Linux) or copy (on Windows) as an example for archive_command. However, cp/copy does not

Re: [HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-17 Thread Fujii Masao
On Mon, Mar 17, 2014 at 10:20 AM, Robert Haas robertmh...@gmail.com wrote: On Sun, Mar 16, 2014 at 6:23 AM, MauMau maumau...@gmail.com wrote: The PostgreSQL documentation describes cp (on UNIX/Linux) or copy (on Windows) as an example for archive_command. However, cp/copy does not sync the

Re: [HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-17 Thread Mitsumasa KONDO
2014-03-17 21:12 GMT+09:00 Fujii Masao masao.fu...@gmail.com: On Mon, Mar 17, 2014 at 10:20 AM, Robert Haas robertmh...@gmail.com wrote: On Sun, Mar 16, 2014 at 6:23 AM, MauMau maumau...@gmail.com wrote: The PostgreSQL documentation describes cp (on UNIX/Linux) or copy (on Windows) as an

[HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-16 Thread MauMau
Hello, The PostgreSQL documentation describes cp (on UNIX/Linux) or copy (on Windows) as an example for archive_command. However, cp/copy does not sync the copied data to disk. As a result, the completed WAL segments would be lost in the following sequence: 1. A WAL segment fills up. 2.

Re: [HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-16 Thread Amit Kapila
On Sun, Mar 16, 2014 at 3:53 PM, MauMau maumau...@gmail.com wrote: Hello, The PostgreSQL documentation describes cp (on UNIX/Linux) or copy (on Windows) as an example for archive_command. However, cp/copy does not sync the copied data to disk. As a result, the completed WAL segments would

Re: [HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-16 Thread MauMau
From: Amit Kapila amit.kapil...@gmail.com How about using pg_receivexlog for archiving purpose? pg_receivexlog is good in that it does fsync(). But it seems difficult to use correctly, and I'm not sure if I can catch all WAL segments without any loss. pg_receivexlog must be started with

Re: [HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-16 Thread Greg Stark
On Sun, Mar 16, 2014 at 10:23 AM, MauMau maumau...@gmail.com wrote: The PostgreSQL documentation describes cp (on UNIX/Linux) or copy (on Windows) as an example for archive_command. However, cp/copy does not sync the copied data to disk I'm actually a lot less concerned about fsyncing the

Re: [HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-16 Thread Andreas Karlsson
On 03/16/2014 03:23 PM, MauMau wrote: From: Amit Kapila amit.kapil...@gmail.com How about using pg_receivexlog for archiving purpose? pg_receivexlog is good in that it does fsync(). But it seems difficult to use correctly, and I'm not sure if I can catch all WAL segments without any loss.

Re: [HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-16 Thread Robert Haas
On Sun, Mar 16, 2014 at 6:23 AM, MauMau maumau...@gmail.com wrote: The PostgreSQL documentation describes cp (on UNIX/Linux) or copy (on Windows) as an example for archive_command. However, cp/copy does not sync the copied data to disk. As a result, the completed WAL segments would be lost

Re: [HACKERS] [RFC] What should we do for reliable WAL archiving?

2014-03-16 Thread Amit Kapila
On Sun, Mar 16, 2014 at 7:53 PM, MauMau maumau...@gmail.com wrote: From: Amit Kapila amit.kapil...@gmail.com How about using pg_receivexlog for archiving purpose? pg_receivexlog is good in that it does fsync(). But it seems difficult to use correctly, and I'm not sure if I can catch all