Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-21 Thread Michael Paquier
On Fri, Jul 21, 2017 at 8:22 AM, Michael Paquier wrote: > On Thu, Jul 20, 2017 at 9:31 PM, Robert Haas wrote: >> Still, it can't be worse than the status quo, where instead of int64 >> we're using int and int32, so maybe we ought to back-patch it as-is >> for now and look at any further cleanup t

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-20 Thread Michael Paquier
On Thu, Jul 20, 2017 at 9:31 PM, Robert Haas wrote: > I was initially surprised that your testing managed to pass, but then > I noticed that this sanity test is using && where it should really be > using ||; it will only fail if ALL of the data types are wrong. Oops. Oh, oh. If this was right fr

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-20 Thread Robert Haas
On Thu, Jul 20, 2017 at 2:17 AM, Michael Paquier wrote: > Heikki, this bug is rather bad for anybody using pg_rewind with > relation file sizes larger than 2GB as this corrupts data of > instances. I think that you would be the best fit as a committer to > look at this patch as you implemented the

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-19 Thread Michael Paquier
On Fri, Jul 7, 2017 at 9:33 AM, Michael Paquier wrote: > On Fri, Jul 7, 2017 at 4:31 PM, Kuntal Ghosh > wrote: >> On Fri, Jul 7, 2017 at 7:49 AM, Michael Paquier >> wrote: >> I don't have any more inputs on this patch and it looks good to me. >> So, I'm moving the status to ready for committer.

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-07 Thread Michael Paquier
On Fri, Jul 7, 2017 at 4:31 PM, Kuntal Ghosh wrote: > On Fri, Jul 7, 2017 at 7:49 AM, Michael Paquier > wrote: > I don't have any more inputs on this patch and it looks good to me. > So, I'm moving the status to ready for committer. Thanks! >> At some point it would really make sense to group a

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-07 Thread Kuntal Ghosh
On Fri, Jul 7, 2017 at 7:49 AM, Michael Paquier wrote: > On Thu, Jul 6, 2017 at 8:51 PM, Dilip Kumar wrote: >> On Thu, Jul 6, 2017 at 4:18 PM, Kuntal Ghosh >> wrote: >>> But, I'm little concerned/doubt regarding the following part of the code. >>> +/* >>> + * Converts an int64 from network byte

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-06 Thread Michael Paquier
On Thu, Jul 6, 2017 at 8:51 PM, Dilip Kumar wrote: > On Thu, Jul 6, 2017 at 4:18 PM, Kuntal Ghosh > wrote: >> But, I'm little concerned/doubt regarding the following part of the code. >> +/* >> + * Converts an int64 from network byte order to native format. >> + */ >> +static int64 >> +pg_recvin

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-06 Thread Dilip Kumar
On Thu, Jul 6, 2017 at 4:18 PM, Kuntal Ghosh wrote: > On Wed, Jul 5, 2017 at 9:35 AM, Michael Paquier > wrote: >> On Tue, Jul 4, 2017 at 4:41 PM, Kuntal Ghosh >> wrote: >>> I've not yet started the patch and it may take some time for me to >>> understand and write >>> the patch in a correct way

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-06 Thread Kuntal Ghosh
On Wed, Jul 5, 2017 at 9:35 AM, Michael Paquier wrote: > On Tue, Jul 4, 2017 at 4:41 PM, Kuntal Ghosh > wrote: >> I've not yet started the patch and it may take some time for me to >> understand and write >> the patch in a correct way. Since, you've almost written the patch, >> IMHO, please go a

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-04 Thread Michael Paquier
On Tue, Jul 4, 2017 at 4:41 PM, Kuntal Ghosh wrote: > I've not yet started the patch and it may take some time for me to > understand and write > the patch in a correct way. Since, you've almost written the patch, > IMHO, please go ahead > and submit the patch. I'll happily review and test it. :-)

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-04 Thread Kuntal Ghosh
On Tue, Jul 4, 2017 at 12:44 PM, Michael Paquier wrote: > On Tue, Jul 4, 2017 at 3:25 PM, Kuntal Ghosh > wrote: >> On Mon, Jul 3, 2017 at 6:50 PM, Michael Paquier >> wrote: >>> pg_basebackup/ with fe_recvint64() has its own way to do things, as >>> does the large object things in libpq. I would

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-04 Thread Michael Paquier
On Tue, Jul 4, 2017 at 3:25 PM, Kuntal Ghosh wrote: > On Mon, Jul 3, 2017 at 6:50 PM, Michael Paquier > wrote: >> pg_basebackup/ with fe_recvint64() has its own way to do things, as >> does the large object things in libpq. I would think that at least on >> HEAD things could be gathered with a sm

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-03 Thread Kuntal Ghosh
On Mon, Jul 3, 2017 at 6:50 PM, Michael Paquier wrote: > pg_basebackup/ with fe_recvint64() has its own way to do things, as > does the large object things in libpq. I would think that at least on > HEAD things could be gathered with a small set of routines. It is > annoying to have a third copy o

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-03 Thread Kuntal Ghosh
On Tue, Jul 4, 2017 at 4:12 AM, Michael Paquier wrote: > On Tue, Jul 4, 2017 at 4:27 AM, Tom Lane wrote: >> Peter Eisentraut writes: >>> On 7/3/17 09:53, Tom Lane wrote: Hm. Before we add a bunch of code to deal with that, are we sure we *want* it to copy such files? Seems like that'

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-03 Thread Michael Paquier
On Tue, Jul 4, 2017 at 4:27 AM, Tom Lane wrote: > Peter Eisentraut writes: >> On 7/3/17 09:53, Tom Lane wrote: >>> Hm. Before we add a bunch of code to deal with that, are we sure we >>> *want* it to copy such files? Seems like that's expending a lot of >>> data-transfer work for zero added val

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-03 Thread Tom Lane
Peter Eisentraut writes: > On 7/3/17 09:53, Tom Lane wrote: >> Hm. Before we add a bunch of code to deal with that, are we sure we >> *want* it to copy such files? Seems like that's expending a lot of >> data-transfer work for zero added value --- consider e.g. a server >> with a bunch of old co

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-03 Thread Peter Eisentraut
On 7/3/17 09:53, Tom Lane wrote: > Hm. Before we add a bunch of code to deal with that, are we sure we > *want* it to copy such files? Seems like that's expending a lot of > data-transfer work for zero added value --- consider e.g. a server > with a bunch of old core files laying about in $PGDATA

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-03 Thread Tom Lane
Kuntal Ghosh writes: > pg_rewind throws the following error when there is a file of large > size available in the Slave server's data directory. Hm. Before we add a bunch of code to deal with that, are we sure we *want* it to copy such files? Seems like that's expending a lot of data-transfer w

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-03 Thread Michael Paquier
On Mon, Jul 3, 2017 at 8:22 PM, Kuntal Ghosh wrote: > pg_rewind throws the following error when there is a file of large > size available in the Slave server's data directory. Oops. > I guess we've to change the data type to bigint. Also, we need some > implementation of ntohl() for 8-byte data

[HACKERS] Error while copying a large file in pg_rewind

2017-07-03 Thread Kuntal Ghosh
Hello all, pg_rewind throws the following error when there is a file of large size available in the Slave server's data directory. unexpected result while sending file list: ERROR: value "214800" is out of range for type integer CONTEXT: COPY fetchchunks, line 2402, column begin: "21480