Re: Two patches to speed up pg_rewind.

2021-08-19 Thread Michael Paquier
On Fri, Aug 20, 2021 at 11:33:33AM +0800, Paul Guo wrote: > On Tue, Aug 17, 2021 at 3:47 PM Michael Paquier wrote: > > + fsync_fname("global/pg_control", false); > > + fsync_fname("backup_label", false); > > + if (access("recovery.conf", F_OK) == 0) > > + fsync_fname("recovery.conf",

Re: Two patches to speed up pg_rewind.

2021-08-19 Thread Paul Guo
Thanks for reviewing, please see the replies below. On Tue, Aug 17, 2021 at 3:47 PM Michael Paquier wrote: > > On Thu, Aug 05, 2021 at 06:18:03PM +0800, Paul Guo wrote: > > I modified the copy_file_range() patch using the below logic: > > > > If the first call of copy_file_range() fails with

Re: Two patches to speed up pg_rewind.

2021-08-17 Thread Michael Paquier
On Tue, Aug 17, 2021 at 04:47:44PM +0900, Michael Paquier wrote: > One argument > against this approach is that if pg_rewind fails in the middle of its > operation then we would have done a set of fsync() for nothing, with > the data folder still unusable. I was skimming through the patch this

Re: Two patches to speed up pg_rewind.

2021-08-17 Thread Michael Paquier
On Thu, Aug 05, 2021 at 06:18:03PM +0800, Paul Guo wrote: > I modified the copy_file_range() patch using the below logic: > > If the first call of copy_file_range() fails with errno EXDEV or > ENOTSUP, pg_rewind > would not use copy_file_range() in rest code, and if copy_file_range() fails > we

Re: Two patches to speed up pg_rewind.

2021-08-05 Thread Paul Guo
On Tue, Jun 22, 2021 at 11:08 AM Paul Guo wrote: > > On Thu, Jun 17, 2021 at 3:19 PM Michael Paquier wrote: > > > > On Wed, Jun 02, 2021 at 05:02:10PM +0900, Michael Paquier wrote: > > > On Wed, Jun 02, 2021 at 06:20:30PM +1200, Thomas Munro wrote: > > > > The main thing I noticed was that Linux

Re: Two patches to speed up pg_rewind.

2021-06-21 Thread Paul Guo
On Thu, Jun 17, 2021 at 3:19 PM Michael Paquier wrote: > > On Wed, Jun 02, 2021 at 05:02:10PM +0900, Michael Paquier wrote: > > On Wed, Jun 02, 2021 at 06:20:30PM +1200, Thomas Munro wrote: > > > The main thing I noticed was that Linux < 5.3 can fail with EXDEV if > > > you cross a filesystem

Re: Two patches to speed up pg_rewind.

2021-06-17 Thread Paul Guo
No worry I’m work on this. On 2021/6/17, 3:18 PM, "Michael Paquier" wrote: On Wed, Jun 02, 2021 at 05:02:10PM +0900, Michael Paquier wrote: > On Wed, Jun 02, 2021 at 06:20:30PM +1200, Thomas Munro wrote: > > The main thing I noticed was that Linux < 5.3 can fail with EXDEV if > > you cross a

Re: Two patches to speed up pg_rewind.

2021-06-17 Thread Michael Paquier
On Wed, Jun 02, 2021 at 05:02:10PM +0900, Michael Paquier wrote: > On Wed, Jun 02, 2021 at 06:20:30PM +1200, Thomas Munro wrote: > > The main thing I noticed was that Linux < 5.3 can fail with EXDEV if > > you cross a filesystem boundary, is that something we need to worry > > about there? > >

Re: Two patches to speed up pg_rewind.

2021-06-02 Thread Michael Paquier
On Wed, Jun 02, 2021 at 06:20:30PM +1200, Thomas Munro wrote: > The main thing I noticed was that Linux < 5.3 can fail with EXDEV if > you cross a filesystem boundary, is that something we need to worry > about there? Hmm. Good point. That may justify having a switch to control that. -- Michael

Re: Two patches to speed up pg_rewind.

2021-06-02 Thread Thomas Munro
On Wed, Jun 2, 2021 at 5:20 PM Michael Paquier wrote: > Note: FreeBSD 13~ has support for copy_file_range(), nice.. Adding > Thomas in CC in case I am missing something. Yeah, so at least in theory Linux and FreeBSD can now both do tricks like pushing copies down to network filesystems, COW

Re: Two patches to speed up pg_rewind.

2021-06-01 Thread Michael Paquier
On Fri, May 28, 2021 at 05:30:51AM +, Paul Guo wrote: > Did some small modification and rebased the code. See attached for the new > version. Regarding patch 0002, I find the inter-dependencies between write_target_range() and copy_target_range() a bit confusing. There is also a bit of

Re: Two patches to speed up pg_rewind.

2021-05-27 Thread Paul Guo
> On 2021/2/19, 10:33 AM, "Paul Guo" wrote: > Refactored the code a bit along with fixes. Manually tested them on centos > & Ubuntu (the later has copy_file_range()) > For the first patch, actually I have some concerns. My assumption is that > the target pg_data directory should be fsync-ed

Re: Two patches to speed up pg_rewind.

2021-02-18 Thread Paul Guo
Refactored the code a bit along with fixes. Manually tested them on centos & Ubuntu (the later has copy_file_range()) For the first patch, actually I have some concerns. My assumption is that the target pg_data directory should be fsync-ed already. This should be correct normally but there is one

Re: Two patches to speed up pg_rewind.

2021-02-02 Thread Paul Guo
On Jan 28, 2021, at 3:31 PM, Michael Paquier mailto:mich...@paquier.xyz>> wrote: On Wed, Jan 27, 2021 at 09:18:48AM +, Paul Guo wrote: Second one is use copy_file_range() for the local rewind case to replace read()+write(). This introduces copy_file_range() check and HAVE_COPY_FILE_RANGE

Re: Two patches to speed up pg_rewind.

2021-01-27 Thread Michael Paquier
On Wed, Jan 27, 2021 at 09:18:48AM +, Paul Guo wrote: > Second one is use copy_file_range() for the local rewind case to replace > read()+write(). > This introduces copy_file_range() check and HAVE_COPY_FILE_RANGE so other > code could use copy_file_range() if needed. copy_file_range() was

Two patches to speed up pg_rewind.

2021-01-27 Thread Paul Guo
While reading pg_rewind code I found two things could speed up pg_rewind. Attached are the patches. First one: pg_rewind would fsync the whole pgdata directory on the target by default, but that is a waste since usually just part of the files/directories on the target are modified. Other files