Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-08-03 Thread Heikki Linnakangas
On 08/03/2015 07:01 AM, Michael Paquier wrote: On Sun, Aug 2, 2015 at 4:01 AM, Heikki Linnakangas wrote: Perhaps it's best if we copy all the WAL files from source in copy-mode, but not in libpq mode. Regarding old WAL files in the target, it's probably best to always leave them alone. They

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-08-03 Thread Fujii Masao
On Fri, Jul 17, 2015 at 12:28 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Wed, Jul 1, 2015 at 9:31 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Jul 1, 2015 at 2:21 AM, Heikki Linnakangas hlinn...@iki.fi wrote: On 06/29/2015 09:44 AM, Michael Paquier wrote: On Mon, Jun 29,

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-08-02 Thread Michael Paquier
On Sun, Aug 2, 2015 at 4:01 AM, Heikki Linnakangas wrote: Perhaps it's best if we copy all the WAL files from source in copy-mode, but not in libpq mode. Regarding old WAL files in the target, it's probably best to always leave them alone. They should do no harm, and as a general principle

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-08-01 Thread Heikki Linnakangas
On 07/17/2015 06:28 AM, Michael Paquier wrote: On Wed, Jul 1, 2015 at 9:31 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Jul 1, 2015 at 2:21 AM, Heikki Linnakangas hlinn...@iki.fi wrote: On 06/29/2015 09:44 AM, Michael Paquier wrote: On Mon, Jun 29, 2015 at 4:55 AM, Heikki Linnakangas

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-07-16 Thread Michael Paquier
On Wed, Jul 1, 2015 at 9:31 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Jul 1, 2015 at 2:21 AM, Heikki Linnakangas hlinn...@iki.fi wrote: On 06/29/2015 09:44 AM, Michael Paquier wrote: On Mon, Jun 29, 2015 at 4:55 AM, Heikki Linnakangas wrote: But we'll still need to handle the

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-07-01 Thread Fujii Masao
On Wed, Jul 1, 2015 at 2:21 AM, Heikki Linnakangas hlinn...@iki.fi wrote: On 06/29/2015 09:44 AM, Michael Paquier wrote: On Mon, Jun 29, 2015 at 4:55 AM, Heikki Linnakangas wrote: But we'll still need to handle the pg_xlog symlink case somehow. Perhaps it would be enough to special-case

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-07-01 Thread Michael Paquier
On Wed, Jul 1, 2015 at 2:21 AM, Heikki Linnakangas hlinn...@iki.fi wrote: Hmm. I'm starting to think that pg_rewind should ignore pg_xlog entirely. In any non-trivial scenarios, just copying all the files from pg_xlog isn't enough anyway, and you need to set up a recovery.conf after running

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-30 Thread Heikki Linnakangas
On 06/29/2015 09:44 AM, Michael Paquier wrote: On Mon, Jun 29, 2015 at 4:55 AM, Heikki Linnakangas wrote: But we'll still need to handle the pg_xlog symlink case somehow. Perhaps it would be enough to special-case pg_xlog for now. Well, sure, pg_rewind does not copy the soft links either way.

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-29 Thread Michael Paquier
On Mon, Jun 29, 2015 at 3:46 AM, Heikki Linnakangas hlinn...@iki.fi wrote: On 06/24/2015 09:43 AM, Michael Paquier wrote: Attached is a new set of patches. Except for the last ones that addresses one issue of pg_rewind (symlink management when streaming PGDATA), all the others introduce

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-29 Thread Michael Paquier
On Mon, Jun 29, 2015 at 3:53 PM, Vladimir Borodin r...@simply.name wrote: 28 июня 2015 г., в 21:46, Heikki Linnakangas hlinn...@iki.fi wrote: I've committed the additional option to the functions in genfile.c (I renamed it to missing_ok, for clarity), and the pg_rewind changes to use that

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-28 Thread Heikki Linnakangas
On 06/26/2015 10:10 PM, Andres Freund wrote: On 2015-06-26 15:07:59 -0400, Robert Haas wrote: I realize that the recent fsync fiasco demonstrated that people keep files not readable by PG in the data directory It wasn't unreadable files that were the primary problem, it was files with read

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-28 Thread Heikki Linnakangas
On 06/24/2015 09:43 AM, Michael Paquier wrote: Attached is a new set of patches. Except for the last ones that addresses one issue of pg_rewind (symlink management when streaming PGDATA), all the others introduce if_not_exists options for the functions of genfile.c. The pg_rewind stuff could be

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-26 Thread Robert Haas
On Wed, Jun 24, 2015 at 9:41 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Wed, Jun 24, 2015 at 3:43 PM, Michael Paquier wrote: I just realized another problem: We recently learned the hard way that some people have files in the data directory that are not writeable by the 'postgres'

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-26 Thread Robert Haas
On Fri, Jun 26, 2015 at 3:10 PM, Andres Freund and...@anarazel.de wrote: On 2015-06-26 15:07:59 -0400, Robert Haas wrote: I realize that the recent fsync fiasco demonstrated that people keep files not readable by PG in the data directory It wasn't unreadable files that were the primary

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-26 Thread Andres Freund
On 2015-06-26 15:07:59 -0400, Robert Haas wrote: I realize that the recent fsync fiasco demonstrated that people keep files not readable by PG in the data directory It wasn't unreadable files that were the primary problem, it was files with read only permissions, no? oops, I can't read this,

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-24 Thread Michael Paquier
On Wed, Jun 24, 2015 at 3:43 PM, Michael Paquier wrote: I just realized another problem: We recently learned the hard way that some people have files in the data directory that are not writeable by the 'postgres' user (http://www.postgresql.org/message-id/20150523172627.ga24...@msg.df7cb.de).

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-23 Thread Heikki Linnakangas
On 06/23/2015 07:51 AM, Michael Paquier wrote: So... Attached are a set of patches dedicated at fixing this issue: Thanks for working on this! - 0001, add if_not_exists to pg_tablespace_location, returning NULL if path does not exist - 0002, same with pg_stat_file, returning NULL if file

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-23 Thread Heikki Linnakangas
On 06/23/2015 05:03 PM, Fujii Masao wrote: On Tue, Jun 23, 2015 at 9:19 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 06/23/2015 07:51 AM, Michael Paquier wrote: So... Attached are a set of patches dedicated at fixing this issue: Thanks for working on this! - 0001, add if_not_exists

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-23 Thread Fujii Masao
On Tue, Jun 23, 2015 at 9:19 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 06/23/2015 07:51 AM, Michael Paquier wrote: So... Attached are a set of patches dedicated at fixing this issue: Thanks for working on this! - 0001, add if_not_exists to pg_tablespace_location, returning NULL if

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-23 Thread Fujii Masao
On Tue, Jun 23, 2015 at 11:21 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 06/23/2015 05:03 PM, Fujii Masao wrote: On Tue, Jun 23, 2015 at 9:19 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 06/23/2015 07:51 AM, Michael Paquier wrote: So... Attached are a set of patches dedicated at

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-23 Thread Michael Paquier
On Wed, Jun 24, 2015 at 1:40 AM, Fujii Masao masao.fu...@gmail.com wrote: On Tue, Jun 23, 2015 at 11:21 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 06/23/2015 05:03 PM, Fujii Masao wrote: On Tue, Jun 23, 2015 at 9:19 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 06/23/2015 07:51 AM,

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-23 Thread Fujii Masao
On Wed, Jun 24, 2015 at 3:36 AM, Michael Paquier michael.paqu...@gmail.com wrote: On Wed, Jun 24, 2015 at 1:40 AM, Fujii Masao masao.fu...@gmail.com wrote: On Tue, Jun 23, 2015 at 11:21 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 06/23/2015 05:03 PM, Fujii Masao wrote: On Tue, Jun 23,

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-22 Thread Michael Paquier
On Fri, Jun 19, 2015 at 9:22 PM, Robert Haas robertmh...@gmail.com wrote: On Fri, Jun 19, 2015 at 8:18 AM, Robert Haas robertmh...@gmail.com wrote: On Fri, Jun 19, 2015 at 12:14 AM, Michael Paquier michael.paqu...@gmail.com wrote: Listing the directories with pg_ls_dir() has the same problem.

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-19 Thread Robert Haas
On Fri, Jun 19, 2015 at 8:18 AM, Robert Haas robertmh...@gmail.com wrote: On Fri, Jun 19, 2015 at 12:14 AM, Michael Paquier michael.paqu...@gmail.com wrote: Listing the directories with pg_ls_dir() has the same problem. (After some discussion on IM with Heikki on this one). This is actually

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-19 Thread Robert Haas
On Fri, Jun 19, 2015 at 12:14 AM, Michael Paquier michael.paqu...@gmail.com wrote: Listing the directories with pg_ls_dir() has the same problem. (After some discussion on IM with Heikki on this one). This is actually more tricky because pg_ls_dir() does not return '.' or '..' that we could

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-18 Thread Heikki Linnakangas
On 06/16/2015 02:04 AM, Michael Paquier wrote: In order to reduce the risk of failure to a minimum and to preserve the performance of the tool when using --source-server, I think that we should add some check using pg_stat_file to see if a file is still present or not, and if it is missing we

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-18 Thread Michael Paquier
On Thu, Jun 18, 2015 at 10:55 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 06/16/2015 02:04 AM, Michael Paquier wrote: In order to reduce the risk of failure to a minimum and to preserve the performance of the tool when using --source-server, I think that we should add some check using

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-16 Thread Michael Paquier
On Fri, Jun 12, 2015 at 9:02 PM, Fujii Masao wrote: You want to draft a patch? Should I? Please feel free to try that! :) OK, so attached are a patch and a test case able to trigger easily the error. Apply the patch and run the test case to reproduce the following failure: $ ERROR: could not

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-12 Thread Michael Paquier
On Thu, Jun 11, 2015 at 5:48 PM, Fujii Masao masao.fu...@gmail.com wrote: On Thu, Jun 11, 2015 at 2:14 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Thu, Jun 11, 2015 at 1:51 AM, Fujii Masao masao.fu...@gmail.com wrote: Shouldn't pg_rewind ignore that failure of operation? If the

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-12 Thread Fujii Masao
On Fri, Jun 12, 2015 at 3:17 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Thu, Jun 11, 2015 at 5:48 PM, Fujii Masao masao.fu...@gmail.com wrote: On Thu, Jun 11, 2015 at 2:14 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Thu, Jun 11, 2015 at 1:51 AM, Fujii Masao

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-12 Thread Fujii Masao
On Fri, Jun 12, 2015 at 4:29 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Fri, Jun 12, 2015 at 3:50 PM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Jun 12, 2015 at 3:17 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Thu, Jun 11, 2015 at 5:48 PM, Fujii Masao

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-11 Thread Fujii Masao
On Thu, Jun 11, 2015 at 2:14 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Thu, Jun 11, 2015 at 1:51 AM, Fujii Masao masao.fu...@gmail.com wrote: Shouldn't pg_rewind ignore that failure of operation? If the file is not found in source server, the file doesn't need to be copied to

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-10 Thread Michael Paquier
On Thu, Jun 11, 2015 at 1:51 AM, Fujii Masao masao.fu...@gmail.com wrote: Shouldn't pg_rewind ignore that failure of operation? If the file is not found in source server, the file doesn't need to be copied to destination server obviously. So ISTM that pg_rewind safely can skip copying that