Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-08-03 Thread Fujii Masao
On Fri, Jul 17, 2015 at 12:28 PM, Michael Paquier wrote: > On Wed, Jul 1, 2015 at 9:31 PM, Fujii Masao wrote: >> On Wed, Jul 1, 2015 at 2:21 AM, Heikki Linnakangas wrote: >>> On 06/29/2015 09:44 AM, Michael Paquier wrote: On Mon, Jun 29, 2015 at 4:55 AM, Heikki Linnakangas wrote: >

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-08-03 Thread Heikki Linnakangas
On 08/03/2015 07:01 AM, Michael Paquier wrote: On Sun, Aug 2, 2015 at 4:01 AM, Heikki Linnakangas wrote: Perhaps it's best if we copy all the WAL files from source in copy-mode, but not in libpq mode. Regarding old WAL files in the target, it's probably best to always leave them alone. They shou

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-08-02 Thread Michael Paquier
On Sun, Aug 2, 2015 at 4:01 AM, Heikki Linnakangas wrote: > Perhaps it's best if we copy all the WAL files from source in copy-mode, but > not in libpq mode. Regarding old WAL files in the target, it's probably best > to always leave them alone. They should do no harm, and as a general > principle

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-08-01 Thread Heikki Linnakangas
On 07/17/2015 06:28 AM, Michael Paquier wrote: On Wed, Jul 1, 2015 at 9:31 PM, Fujii Masao wrote: On Wed, Jul 1, 2015 at 2:21 AM, Heikki Linnakangas wrote: On 06/29/2015 09:44 AM, Michael Paquier wrote: On Mon, Jun 29, 2015 at 4:55 AM, Heikki Linnakangas wrote: But we'll still need to han

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-07-16 Thread Michael Paquier
On Wed, Jul 1, 2015 at 9:31 PM, Fujii Masao wrote: > On Wed, Jul 1, 2015 at 2:21 AM, Heikki Linnakangas wrote: >> On 06/29/2015 09:44 AM, Michael Paquier wrote: >>> >>> On Mon, Jun 29, 2015 at 4:55 AM, Heikki Linnakangas wrote: But we'll still need to handle the pg_xlog symlink case som

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-07-01 Thread Fujii Masao
On Wed, Jul 1, 2015 at 2:21 AM, Heikki Linnakangas wrote: > On 06/29/2015 09:44 AM, Michael Paquier wrote: >> >> On Mon, Jun 29, 2015 at 4:55 AM, Heikki Linnakangas wrote: >>> >>> But we'll still need to handle the pg_xlog symlink case somehow. Perhaps >>> it >>> would be enough to special-case pg

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-07-01 Thread Michael Paquier
On Wed, Jul 1, 2015 at 2:21 AM, Heikki Linnakangas wrote: > Hmm. I'm starting to think that pg_rewind should ignore pg_xlog entirely. In > any non-trivial scenarios, just copying all the files from pg_xlog isn't > enough anyway, and you need to set up a recovery.conf after running > pg_rewind that

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-30 Thread Heikki Linnakangas
On 06/29/2015 09:44 AM, Michael Paquier wrote: On Mon, Jun 29, 2015 at 4:55 AM, Heikki Linnakangas wrote: But we'll still need to handle the pg_xlog symlink case somehow. Perhaps it would be enough to special-case pg_xlog for now. Well, sure, pg_rewind does not copy the soft links either way.

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-29 Thread Michael Paquier
On Mon, Jun 29, 2015 at 3:53 PM, Vladimir Borodin wrote: > 28 июня 2015 г., в 21:46, Heikki Linnakangas wrote: >> I've committed the additional option to the functions in genfile.c (I >> renamed it to "missing_ok", for clarity), and the pg_rewind changes to use >> that option. > > And since it ch

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-28 Thread Vladimir Borodin
> 28 июня 2015 г., в 21:46, Heikki Linnakangas написал(а): > > On 06/24/2015 09:43 AM, Michael Paquier wrote: >> Attached is a new set of patches. Except for the last ones that >> addresses one issue of pg_rewind (symlink management when streaming >> PGDATA), all the others introduce if_not_exis

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-28 Thread Michael Paquier
On Mon, Jun 29, 2015 at 4:55 AM, Heikki Linnakangas wrote: > After thinking about this some more, I think it'd be acceptable if we just > fail, if there are any non-writeable files in the data directory. The > typical scenario is that postgresql.conf, or an SSL cert file, is a symlink > to outside

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-28 Thread Michael Paquier
On Mon, Jun 29, 2015 at 3:46 AM, Heikki Linnakangas wrote: > On 06/24/2015 09:43 AM, Michael Paquier wrote: >> >> Attached is a new set of patches. Except for the last ones that >> addresses one issue of pg_rewind (symlink management when streaming >> PGDATA), all the others introduce if_not_exist

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-28 Thread Heikki Linnakangas
On 06/26/2015 10:10 PM, Andres Freund wrote: On 2015-06-26 15:07:59 -0400, Robert Haas wrote: I realize that the recent fsync fiasco demonstrated that people keep files not readable by PG in the data directory It wasn't unreadable files that were the primary problem, it was files with read onl

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-28 Thread Heikki Linnakangas
On 06/24/2015 09:43 AM, Michael Paquier wrote: Attached is a new set of patches. Except for the last ones that addresses one issue of pg_rewind (symlink management when streaming PGDATA), all the others introduce if_not_exists options for the functions of genfile.c. The pg_rewind stuff could be m

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-26 Thread Andres Freund
On 2015-06-26 15:07:59 -0400, Robert Haas wrote: > I realize that the recent fsync fiasco demonstrated that people keep > files not readable by PG in the data directory It wasn't unreadable files that were the primary problem, it was files with read only permissions, no? > "oops, I can't read thi

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-26 Thread Robert Haas
On Fri, Jun 26, 2015 at 3:10 PM, Andres Freund wrote: > On 2015-06-26 15:07:59 -0400, Robert Haas wrote: >> I realize that the recent fsync fiasco demonstrated that people keep >> files not readable by PG in the data directory > > It wasn't unreadable files that were the primary problem, it was fi

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-26 Thread Robert Haas
On Wed, Jun 24, 2015 at 9:41 PM, Michael Paquier wrote: > On Wed, Jun 24, 2015 at 3:43 PM, Michael Paquier wrote: >>> I just realized another problem: We recently learned the hard way that some >>> people have files in the data directory that are not writeable by the >>> 'postgres' user >>> (http:

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-24 Thread Michael Paquier
On Wed, Jun 24, 2015 at 3:43 PM, Michael Paquier wrote: >> I just realized another problem: We recently learned the hard way that some >> people have files in the data directory that are not writeable by the >> 'postgres' user >> (http://www.postgresql.org/message-id/20150523172627.ga24...@msg.df7c

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-23 Thread Michael Paquier
On Tue, Jun 23, 2015 at 9:19 PM, Heikki Linnakangas wrote: > On 06/23/2015 07:51 AM, Michael Paquier wrote: >> - 0001, add if_not_exists to pg_tablespace_location, returning NULL if >> path does not exist >> - 0002, same with pg_stat_file, returning NULL if file does not exist >> - 0003, same with

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-23 Thread Fujii Masao
On Wed, Jun 24, 2015 at 3:36 AM, Michael Paquier wrote: > On Wed, Jun 24, 2015 at 1:40 AM, Fujii Masao wrote: >> On Tue, Jun 23, 2015 at 11:21 PM, Heikki Linnakangas wrote: >>> On 06/23/2015 05:03 PM, Fujii Masao wrote: On Tue, Jun 23, 2015 at 9:19 PM, Heikki Linnakangas wrote: >

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-23 Thread Michael Paquier
On Wed, Jun 24, 2015 at 1:40 AM, Fujii Masao wrote: > On Tue, Jun 23, 2015 at 11:21 PM, Heikki Linnakangas wrote: >> On 06/23/2015 05:03 PM, Fujii Masao wrote: >>> >>> On Tue, Jun 23, 2015 at 9:19 PM, Heikki Linnakangas >>> wrote: On 06/23/2015 07:51 AM, Michael Paquier wrote: > >>

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-23 Thread Fujii Masao
On Tue, Jun 23, 2015 at 11:21 PM, Heikki Linnakangas wrote: > On 06/23/2015 05:03 PM, Fujii Masao wrote: >> >> On Tue, Jun 23, 2015 at 9:19 PM, Heikki Linnakangas >> wrote: >>> >>> On 06/23/2015 07:51 AM, Michael Paquier wrote: So... Attached are a set of patches dedicated at fixin

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-23 Thread Heikki Linnakangas
On 06/23/2015 05:03 PM, Fujii Masao wrote: On Tue, Jun 23, 2015 at 9:19 PM, Heikki Linnakangas wrote: On 06/23/2015 07:51 AM, Michael Paquier wrote: So... Attached are a set of patches dedicated at fixing this issue: Thanks for working on this! - 0001, add if_not_exists to pg_tablespace_

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-23 Thread Fujii Masao
On Tue, Jun 23, 2015 at 9:19 PM, Heikki Linnakangas wrote: > On 06/23/2015 07:51 AM, Michael Paquier wrote: >> >> So... Attached are a set of patches dedicated at fixing this issue: > > > Thanks for working on this! > >> - 0001, add if_not_exists to pg_tablespace_location, returning NULL if >> pat

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-23 Thread Heikki Linnakangas
On 06/23/2015 07:51 AM, Michael Paquier wrote: So... Attached are a set of patches dedicated at fixing this issue: Thanks for working on this! - 0001, add if_not_exists to pg_tablespace_location, returning NULL if path does not exist - 0002, same with pg_stat_file, returning NULL if file does

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-22 Thread Michael Paquier
On Fri, Jun 19, 2015 at 9:22 PM, Robert Haas wrote: > On Fri, Jun 19, 2015 at 8:18 AM, Robert Haas wrote: >> On Fri, Jun 19, 2015 at 12:14 AM, Michael Paquier >> wrote: Listing the directories with pg_ls_dir() has the same problem. >>> >>> (After some discussion on IM with Heikki on this on

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-19 Thread Robert Haas
On Fri, Jun 19, 2015 at 8:18 AM, Robert Haas wrote: > On Fri, Jun 19, 2015 at 12:14 AM, Michael Paquier > wrote: >>> Listing the directories with pg_ls_dir() has the same problem. >> >> (After some discussion on IM with Heikki on this one). >> This is actually more tricky because pg_ls_dir() does

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-19 Thread Robert Haas
On Fri, Jun 19, 2015 at 12:14 AM, Michael Paquier wrote: >> Listing the directories with pg_ls_dir() has the same problem. > > (After some discussion on IM with Heikki on this one). > This is actually more tricky because pg_ls_dir() does not return '.' > or '..' that we could use to identify that

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-18 Thread Michael Paquier
On Thu, Jun 18, 2015 at 10:55 PM, Heikki Linnakangas wrote: > On 06/16/2015 02:04 AM, Michael Paquier wrote: >> >> In order to reduce the risk of failure to a minimum and to preserve >> the performance of the tool when using --source-server, I think that >> we should add some check using pg_stat_f

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-18 Thread Heikki Linnakangas
On 06/16/2015 02:04 AM, Michael Paquier wrote: In order to reduce the risk of failure to a minimum and to preserve the performance of the tool when using --source-server, I think that we should add some check using pg_stat_file to see if a file is still present or not, and if it is missing we can

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-15 Thread Michael Paquier
On Fri, Jun 12, 2015 at 9:02 PM, Fujii Masao wrote: >> You want to draft a patch? Should I? > > Please feel free to try that! :) OK, so attached are a patch and a test case able to trigger easily the error. Apply the patch and run the test case to reproduce the following failure: $ ERROR: could n

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-12 Thread Fujii Masao
On Fri, Jun 12, 2015 at 4:29 PM, Michael Paquier wrote: > On Fri, Jun 12, 2015 at 3:50 PM, Fujii Masao wrote: >> On Fri, Jun 12, 2015 at 3:17 PM, Michael Paquier >> wrote: >>> On Thu, Jun 11, 2015 at 5:48 PM, Fujii Masao wrote: On Thu, Jun 11, 2015 at 2:14 PM, Michael Paquier wrote:

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-12 Thread Michael Paquier
On Fri, Jun 12, 2015 at 3:50 PM, Fujii Masao wrote: > On Fri, Jun 12, 2015 at 3:17 PM, Michael Paquier > wrote: >> On Thu, Jun 11, 2015 at 5:48 PM, Fujii Masao wrote: >>> On Thu, Jun 11, 2015 at 2:14 PM, Michael Paquier >>> wrote: On Thu, Jun 11, 2015 at 1:51 AM, Fujii Masao wrote: >

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-11 Thread Fujii Masao
On Fri, Jun 12, 2015 at 3:17 PM, Michael Paquier wrote: > On Thu, Jun 11, 2015 at 5:48 PM, Fujii Masao wrote: >> On Thu, Jun 11, 2015 at 2:14 PM, Michael Paquier >> wrote: >>> On Thu, Jun 11, 2015 at 1:51 AM, Fujii Masao wrote: Shouldn't pg_rewind ignore that failure of operation? If the f

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-11 Thread Michael Paquier
On Thu, Jun 11, 2015 at 5:48 PM, Fujii Masao wrote: > On Thu, Jun 11, 2015 at 2:14 PM, Michael Paquier > wrote: >> On Thu, Jun 11, 2015 at 1:51 AM, Fujii Masao wrote: >>> Shouldn't pg_rewind ignore that failure of operation? If the file is not >>> found in source server, the file doesn't need to

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-11 Thread Fujii Masao
On Thu, Jun 11, 2015 at 2:14 PM, Michael Paquier wrote: > On Thu, Jun 11, 2015 at 1:51 AM, Fujii Masao wrote: >> Shouldn't pg_rewind ignore that failure of operation? If the file is not >> found in source server, the file doesn't need to be copied to destination >> server obviously. So ISTM that

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-10 Thread Michael Paquier
On Thu, Jun 11, 2015 at 1:51 AM, Fujii Masao wrote: > Shouldn't pg_rewind ignore that failure of operation? If the file is not > found in source server, the file doesn't need to be copied to destination > server obviously. So ISTM that pg_rewind safely can skip copying that file. > Thought? I thi