Re: [HACKERS] Include WAL in base backup

2011-01-30 Thread Heikki Linnakangas
On 29.01.2011 09:10, Fujii Masao wrote: On Sat, Jan 29, 2011 at 6:02 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 27.01.2011 06:44, Fujii Masao wrote: + XLByteToSeg(endptr, endlogid, endlogseg); snip + /* Have we reached our stop

Re: [HACKERS] Include WAL in base backup

2011-01-30 Thread Magnus Hagander
On Thu, Jan 27, 2011 at 05:44, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Jan 26, 2011 at 5:17 AM, Magnus Hagander mag...@hagander.net wrote: We should, and the easiest way is to actually use XLogRead() since the code is already there. How about the way in this patch? Thanks for the

Re: [HACKERS] Include WAL in base backup

2011-01-30 Thread Fujii Masao
On Mon, Jan 31, 2011 at 4:22 AM, Magnus Hagander mag...@hagander.net wrote: In pg_basebackup.sgml     termoption--checkpoint replaceable class=parameterfast|spread/replaceable/option/term Though this is not the problem of the patch, I found the inconsistency of the descriptions about options

Re: [HACKERS] Include WAL in base backup

2011-01-30 Thread Robert Haas
On Sun, Jan 30, 2011 at 9:34 PM, Fujii Masao masao.fu...@gmail.com wrote: On Mon, Jan 31, 2011 at 4:22 AM, Magnus Hagander mag...@hagander.net wrote: In pg_basebackup.sgml     termoption--checkpoint replaceable class=parameterfast|spread/replaceable/option/term Though this is not the problem

Re: [HACKERS] Include WAL in base backup

2011-01-28 Thread Heikki Linnakangas
On 27.01.2011 06:44, Fujii Masao wrote: + XLByteToSeg(endptr, endlogid, endlogseg); snip + /* Have we reached our stop position yet? */ + if (logid endlogid || + (logid == endlogid logseg= endlogseg)) +

Re: [HACKERS] Include WAL in base backup

2011-01-28 Thread Fujii Masao
On Sat, Jan 29, 2011 at 6:02 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 27.01.2011 06:44, Fujii Masao wrote: +               XLByteToSeg(endptr, endlogid, endlogseg); snip +                       /* Have we reached our stop position yet? */ +                      

Re: [HACKERS] Include WAL in base backup

2011-01-26 Thread Fujii Masao
On Wed, Jan 26, 2011 at 5:17 AM, Magnus Hagander mag...@hagander.net wrote: We should, and the easiest way is to actually use XLogRead() since the code is already there. How about the way in this patch? Thanks for the update. I reread the patch. + MemSet(statbuf, 0,

Re: [HACKERS] Include WAL in base backup

2011-01-25 Thread Fujii Masao
On Tue, Jan 25, 2011 at 12:33 AM, Magnus Hagander mag...@hagander.net wrote: Here's an updated version of the patch: * rebased on the current git master (including changing the switch from -w to -x since -w was used now) * includes some documentation * use XLogByteToSeg and uint32 as

Re: [HACKERS] Include WAL in base backup

2011-01-25 Thread Magnus Hagander
On Tue, Jan 25, 2011 at 10:56, Fujii Masao masao.fu...@gmail.com wrote: On Tue, Jan 25, 2011 at 12:33 AM, Magnus Hagander mag...@hagander.net wrote: Here's an updated version of the patch: * rebased on the current git master (including changing the switch from -w to -x since -w was used now)

Re: [HACKERS] Include WAL in base backup

2011-01-25 Thread Fujii Masao
On Tue, Jan 25, 2011 at 8:21 PM, Magnus Hagander mag...@hagander.net wrote: +               elog(DEBUG1, Going to send wal from %i.%i to %i.%i, +                        logid, logseg, +                        endlogid, endlogseg); %u should be used instead of %i. Or how about logging the

Re: [HACKERS] Include WAL in base backup

2011-01-25 Thread Fujii Masao
On Tue, Jan 25, 2011 at 10:28 PM, Fujii Masao masao.fu...@gmail.com wrote: (the discussed changse above have been applied and pushed to github) Thanks! I'll test and review that. WAL file might get recycled or removed while walsender is reading it. So the WAL file which pg_basebackup seemingly

Re: [HACKERS] Include WAL in base backup

2011-01-25 Thread Magnus Hagander
On Tue, Jan 25, 2011 at 15:04, Fujii Masao masao.fu...@gmail.com wrote: On Tue, Jan 25, 2011 at 10:28 PM, Fujii Masao masao.fu...@gmail.com wrote: (the discussed changse above have been applied and pushed to github) Thanks! I'll test and review that. WAL file might get recycled or removed

Re: [HACKERS] Include WAL in base backup

2011-01-25 Thread Magnus Hagander
On Tue, Jan 25, 2011 at 16:34, Magnus Hagander mag...@hagander.net wrote: On Tue, Jan 25, 2011 at 15:04, Fujii Masao masao.fu...@gmail.com wrote: On Tue, Jan 25, 2011 at 10:28 PM, Fujii Masao masao.fu...@gmail.com wrote: (the discussed changse above have been applied and pushed to github)

Re: [HACKERS] Include WAL in base backup

2011-01-24 Thread Fujii Masao
On Mon, Jan 24, 2011 at 4:47 PM, Magnus Hagander mag...@hagander.net wrote: On Mon, Jan 24, 2011 at 08:45, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Jan 21, 2011 at 12:28 AM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: - Why not initialize logid and

Re: [HACKERS] Include WAL in base backup

2011-01-24 Thread Magnus Hagander
On Mon, Jan 24, 2011 at 09:03, Fujii Masao masao.fu...@gmail.com wrote: On Mon, Jan 24, 2011 at 4:47 PM, Magnus Hagander mag...@hagander.net wrote: On Mon, Jan 24, 2011 at 08:45, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Jan 21, 2011 at 12:28 AM, Tom Lane t...@sss.pgh.pa.us wrote:

Re: [HACKERS] Include WAL in base backup

2011-01-23 Thread Fujii Masao
On Fri, Jan 21, 2011 at 12:28 AM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: - Why not initialize logid and logseg like so?:        int logid = startptr.xlogid;        int logseg = startptr.xrecoff / XLogSegSize;  Then use those in your elog?  Seems

Re: [HACKERS] Include WAL in base backup

2011-01-23 Thread Magnus Hagander
On Mon, Jan 24, 2011 at 08:45, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Jan 21, 2011 at 12:28 AM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: - Why not initialize logid and logseg like so?:        int logid = startptr.xlogid;        int logseg =

Re: [HACKERS] Include WAL in base backup

2011-01-20 Thread Magnus Hagander
On Thu, Jan 20, 2011 at 05:03, Stephen Frost sfr...@snowman.net wrote: Greetings, * Magnus Hagander (mag...@hagander.net) wrote: For now, you need to set wal_keep_segments to make it work properly, but if you do the idea is that the tar file/stream generated in the base backup will include

Re: [HACKERS] Include WAL in base backup

2011-01-20 Thread Tom Lane
Magnus Hagander mag...@hagander.net writes: - Why not initialize logid and logseg like so?:        int logid = startptr.xlogid;        int logseg = startptr.xrecoff / XLogSegSize;  Then use those in your elog?  Seems cleaner to me. Hmm. Yes. Agreed. Marginal complaint here: int is the

Re: [HACKERS] Include WAL in base backup

2011-01-19 Thread Stephen Frost
Greetings, * Magnus Hagander (mag...@hagander.net) wrote: For now, you need to set wal_keep_segments to make it work properly, but if you do the idea is that the tar file/stream generated in the base backup will include all the required WAL files. Is there some reason to not ERROR outright if

Re: [HACKERS] Include WAL in base backup

2011-01-19 Thread Stephen Frost
Magnus, * Stephen Frost (sfr...@snowman.net) wrote: mkay, I'm not going to try to make this ready for committer, but will provide my comments on it overall. Bit difficult to review when someone else is reviewing the base patch too. :/ I went ahead and marked it as 'waiting on author', since

Re: [HACKERS] Include WAL in base backup

2011-01-17 Thread Magnus Hagander
On Sun, Jan 16, 2011 at 20:13, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Magnus Hagander mag...@hagander.net writes: However, it's not quite that simple. just adding a fork() doesn't work on all our platforms, and you need to deal with feedback and such between them as well. I'd think

Re: [HACKERS] Include WAL in base backup

2011-01-17 Thread Dimitri Fontaine
Magnus Hagander mag...@hagander.net writes: But however we do it, it will be significantly more complex than just including the WAL. And I want to make sure we get *something* done in time for 9.1, and then improve upon it. If we can get the improvement into 9.1 that's great, but if not it

Re: [HACKERS] Include WAL in base backup

2011-01-16 Thread Dimitri Fontaine
Magnus Hagander mag...@hagander.net writes: What if you start a concurrent process that begins streaming the WAL segments just before you start the backup, and you stop it after having stopped the backup.  I would think that then, the local received files would be complete.  We would only need

Re: [HACKERS] Include WAL in base backup

2011-01-16 Thread Magnus Hagander
On Sun, Jan 16, 2011 at 18:45, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Magnus Hagander mag...@hagander.net writes: What if you start a concurrent process that begins streaming the WAL segments just before you start the backup, and you stop it after having stopped the backup.  I would

Re: [HACKERS] Include WAL in base backup

2011-01-16 Thread Dimitri Fontaine
Magnus Hagander mag...@hagander.net writes: However, it's not quite that simple. just adding a fork() doesn't work on all our platforms, and you need to deal with feedback and such between them as well. I'd think client-side, we have an existing implementation with the parallel pg_restore

Re: [HACKERS] Include WAL in base backup

2011-01-15 Thread Dimitri Fontaine
Magnus Hagander mag...@hagander.net writes: Here's a cutdown version of the idea about including WAL in the base backup. What I initially wanted was to introduce a way to guarantee that the required WAL (with some sort of cutoff of course) would be available for the backup, but I ran out of

Re: [HACKERS] Include WAL in base backup

2011-01-15 Thread Magnus Hagander
On Sat, Jan 15, 2011 at 23:32, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Magnus Hagander mag...@hagander.net writes: Here's a cutdown version of the idea about including WAL in the base backup. What I initially wanted was to introduce a way to guarantee that the required WAL (with some