Magnus Hagander mag...@hagander.net writes:
With pg_basebackup, you can set up streaming replication in what's
basically a single command (run the base backup, copy i na
recovery.conf file). In my first version I even had a switch that
would create the recovery.conf file for you - should we
On Mon, Jan 17, 2011 at 11:18, Dimitri Fontaine dimi...@2ndquadrant.fr wrote:
Magnus Hagander mag...@hagander.net writes:
With pg_basebackup, you can set up streaming replication in what's
basically a single command (run the base backup, copy i na
recovery.conf file). In my first version I
Magnus Hagander mag...@hagander.net writes:
Until we get integrated WAL streaming while the base backup is ongoing.
We don't know when that is (9.1 or future), but that's what we're aiming
to now, right?
Yeah, it does sound like a plan. But to still allow both - streaming
it in parallell
On Sat, Jan 15, 2011 at 8:33 PM, Tatsuo Ishii is...@postgresql.org wrote:
When do the standby launch its walreceiver? It would be extra-nice for
the base backup tool to optionally continue streaming WALs until the
standby starts doing it itself, so that wal_keep_segments is really
deprecated.
Good point. I have been always wondering why we can't use exiting WAL
transporting infrastructure for sending/receiving WAL archive
segments in streaming replication.
If my memory serves, Fujii has already proposed such an idea but was
rejected for some reason I don't understand.
I must be
On Mon, Jan 17, 2011 at 11:32 AM, Tatsuo Ishii is...@postgresql.org wrote:
Good point. I have been always wondering why we can't use exiting WAL
transporting infrastructure for sending/receiving WAL archive
segments in streaming replication.
If my memory serves, Fujii has already proposed such
On Mon, Jan 17, 2011 at 03:32, Tatsuo Ishii is...@postgresql.org wrote:
Good point. I have been always wondering why we can't use exiting WAL
transporting infrastructure for sending/receiving WAL archive
segments in streaming replication.
If my memory serves, Fujii has already proposed such an
On 14.01.2011 13:38, Magnus Hagander wrote:
On Fri, Jan 14, 2011 at 11:19, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 14.01.2011 08:45, Fujii Masao wrote:
1. Smart shutdown is requested while walsender is sending a backup.
2. Shutdown causes archiver to end.
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
BTW, I just spotted a small race condition between creating a new table
space and base backup. We take a snapshot of all the tablespaces in
pg_tblspc before calling pg_start_backup(). If someone creates a new
tablespace and puts
On 15.01.2011 17:30, Tom Lane wrote:
Heikki Linnakangasheikki.linnakan...@enterprisedb.com writes:
BTW, I just spotted a small race condition between creating a new table
space and base backup. We take a snapshot of all the tablespaces in
pg_tblspc before calling pg_start_backup(). If someone
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
On 15.01.2011 17:30, Tom Lane wrote:
So what? The needed actions will be covered by WAL replay.
No, they won't, if pg_base_backup() is called *after* getting the list
of tablespaces.
Ah. Then the fix is to change the order in
On Sat, Jan 15, 2011 at 16:54, Tom Lane t...@sss.pgh.pa.us wrote:
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes:
On 15.01.2011 17:30, Tom Lane wrote:
So what? The needed actions will be covered by WAL replay.
No, they won't, if pg_base_backup() is called *after* getting the
Magnus Hagander mag...@hagander.net writes:
Something like this to fix? or is this going to put those warnings by
stupid versions of gcc back?
Possibly. If so, I'll fix it --- I have an old gcc to test against
here.
regards, tom lane
--
Sent via pgsql-hackers mailing
On Sat, Jan 15, 2011 at 19:27, Tom Lane t...@sss.pgh.pa.us wrote:
Magnus Hagander mag...@hagander.net writes:
Something like this to fix? or is this going to put those warnings by
stupid versions of gcc back?
Possibly. If so, I'll fix it --- I have an old gcc to test against
here.
Ok,
When do the standby launch its walreceiver? It would be extra-nice for
the base backup tool to optionally continue streaming WALs until the
standby starts doing it itself, so that wal_keep_segments is really
deprecated. No idea how feasible that is, though.
Good point. I have been always
On 14.01.2011 08:45, Fujii Masao wrote:
On Fri, Jan 14, 2011 at 4:13 AM, Magnus Hagandermag...@hagander.net wrote:
At the end of the backup by walsender, it forces a switch to a new
WAL file and waits until the last WAL file has been archived. So we
should change postmaster so that it doesn't
On Fri, Jan 14, 2011 at 11:19, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 14.01.2011 08:45, Fujii Masao wrote:
On Fri, Jan 14, 2011 at 4:13 AM, Magnus Hagandermag...@hagander.net
wrote:
At the end of the backup by walsender, it forces a switch to a new
WAL file and
On Wed, Jan 12, 2011 at 10:39, Fujii Masao masao.fu...@gmail.com wrote:
On Mon, Jan 10, 2011 at 11:09 PM, Magnus Hagander mag...@hagander.net wrote:
I've committed the backend side of this, without that. Still working
on the client, and on cleaning up Heikki's patch for grammar/parser
support.
On Fri, Jan 14, 2011 at 4:13 AM, Magnus Hagander mag...@hagander.net wrote:
While walsender is sending a base backup, WalSndWakeup should
not send the signal to that walsender?
True, it's not necessary. How bad does it actually hurt things though?
Given that the walsender running the backup
On Mon, Jan 10, 2011 at 11:09 PM, Magnus Hagander mag...@hagander.net wrote:
I've committed the backend side of this, without that. Still working
on the client, and on cleaning up Heikki's patch for grammar/parser
support.
Great work!
I have some comments:
While walsender is sending a base
On Tue, Jan 11, 2011 at 01:28, Cédric Villemain
cedric.villemain.deb...@gmail.com wrote:
2011/1/10 Magnus Hagander mag...@hagander.net:
On Sun, Jan 9, 2011 at 23:33, Cédric Villemain
cedric.villemain.deb...@gmail.com wrote:
I've committed the backend side of this, without that. Still working
On Mon, Jan 10, 2011 at 09:09:28AM -0500, Magnus Hagander wrote:
On Sun, Jan 9, 2011 at 23:33, Cédric Villemain
cedric.villemain.deb...@gmail.com wrote:
2011/1/7 Magnus Hagander mag...@hagander.net:
On Fri, Jan 7, 2011 at 01:47, Cédric Villemain
cedric.villemain.deb...@gmail.com wrote:
2011/1/11 Garick Hamlin gham...@isc.upenn.edu:
On Mon, Jan 10, 2011 at 09:09:28AM -0500, Magnus Hagander wrote:
On Sun, Jan 9, 2011 at 23:33, Cédric Villemain
cedric.villemain.deb...@gmail.com wrote:
2011/1/7 Magnus Hagander mag...@hagander.net:
On Fri, Jan 7, 2011 at 01:47, Cédric
On Tue, Jan 11, 2011 at 11:39:20AM -0500, Cédric Villemain wrote:
2011/1/11 Garick Hamlin gham...@isc.upenn.edu:
On Mon, Jan 10, 2011 at 09:09:28AM -0500, Magnus Hagander wrote:
On Sun, Jan 9, 2011 at 23:33, Cédric Villemain
cedric.villemain.deb...@gmail.com wrote:
2011/1/7 Magnus
On Jan11, 2011, at 18:09 , Garick Hamlin wrote:
My gut was that direct io would likely work right on Linux
and Solaris, at least.
Didn't we discover recently that O_DIRECT fails for ext4 on linux
if ordered=data, or something like that?
best regards,
Florian Pflug
--
Sent via pgsql-hackers
2011/1/11 Garick Hamlin gham...@isc.upenn.edu:
On Tue, Jan 11, 2011 at 11:39:20AM -0500, Cédric Villemain wrote:
2011/1/11 Garick Hamlin gham...@isc.upenn.edu:
On Mon, Jan 10, 2011 at 09:09:28AM -0500, Magnus Hagander wrote:
On Sun, Jan 9, 2011 at 23:33, Cédric Villemain
Florian Pflug f...@phlo.org writes:
On Jan11, 2011, at 18:09 , Garick Hamlin wrote:
My gut was that direct io would likely work right on Linux
and Solaris, at least.
Didn't we discover recently that O_DIRECT fails for ext4 on linux
if ordered=data, or something like that?
Quite. Blithe
On Tue, Jan 11, 2011 at 12:45:02PM -0500, Tom Lane wrote:
Florian Pflug f...@phlo.org writes:
On Jan11, 2011, at 18:09 , Garick Hamlin wrote:
My gut was that direct io would likely work right on Linux
and Solaris, at least.
Didn't we discover recently that O_DIRECT fails for ext4 on
On Sun, Jan 9, 2011 at 23:33, Cédric Villemain
cedric.villemain.deb...@gmail.com wrote:
2011/1/7 Magnus Hagander mag...@hagander.net:
On Fri, Jan 7, 2011 at 01:47, Cédric Villemain
cedric.villemain.deb...@gmail.com wrote:
2011/1/5 Magnus Hagander mag...@hagander.net:
On Wed, Jan 5, 2011 at
2011/1/10 Magnus Hagander mag...@hagander.net:
On Sun, Jan 9, 2011 at 23:33, Cédric Villemain
cedric.villemain.deb...@gmail.com wrote:
2011/1/7 Magnus Hagander mag...@hagander.net:
On Fri, Jan 7, 2011 at 01:47, Cédric Villemain
cedric.villemain.deb...@gmail.com wrote:
2011/1/5 Magnus
On 01/10/2011 08:13 PM, Cédric Villemain wrote:
2011/1/10 Magnus Hagandermag...@hagander.net:
On Sun, Jan 9, 2011 at 23:33, Cédric Villemain
cedric.villemain.deb...@gmail.com wrote:
2011/1/7 Magnus Hagandermag...@hagander.net:
On Fri, Jan 7, 2011 at 01:47, Cédric Villemain
2011/1/10 Stefan Kaltenbrunner ste...@kaltenbrunner.cc:
On 01/10/2011 08:13 PM, Cédric Villemain wrote:
2011/1/10 Magnus Hagandermag...@hagander.net:
On Sun, Jan 9, 2011 at 23:33, Cédric Villemain
cedric.villemain.deb...@gmail.com wrote:
2011/1/7 Magnus Hagandermag...@hagander.net:
On
2011/1/10 Magnus Hagander mag...@hagander.net:
On Sun, Jan 9, 2011 at 23:33, Cédric Villemain
cedric.villemain.deb...@gmail.com wrote:
2011/1/7 Magnus Hagander mag...@hagander.net:
On Fri, Jan 7, 2011 at 01:47, Cédric Villemain
cedric.villemain.deb...@gmail.com wrote:
2011/1/5 Magnus
On 7.1.2011 15:45, Magnus Hagander wrote:
On Fri, Jan 7, 2011 at 02:15, Simon Riggssi...@2ndquadrant.com wrote:
One very useful feature will be some way of confirming the number and
size of files to transfer, so that the base backup client can find out
the progress.
The patch already does
On Sun, Jan 9, 2011 at 09:55, Hannu Krosing ha...@2ndquadrant.com wrote:
On 7.1.2011 15:45, Magnus Hagander wrote:
On Fri, Jan 7, 2011 at 02:15, Simon Riggssi...@2ndquadrant.com wrote:
One very useful feature will be some way of confirming the number and
size of files to transfer, so that
On 9.1.2011 10:44, Magnus Hagander wrote:
On Sun, Jan 9, 2011 at 09:55, Hannu Krosingha...@2ndquadrant.com wrote:
On 7.1.2011 15:45, Magnus Hagander wrote:
On Fri, Jan 7, 2011 at 02:15, Simon Riggssi...@2ndquadrant.comwrote:
One very useful feature will be some way of confirming the
On Sun, Jan 9, 2011 at 12:05, Hannu Krosing ha...@2ndquadrant.com wrote:
On 9.1.2011 10:44, Magnus Hagander wrote:
On Sun, Jan 9, 2011 at 09:55, Hannu Krosingha...@2ndquadrant.com wrote:
On 7.1.2011 15:45, Magnus Hagander wrote:
On Fri, Jan 7, 2011 at 02:15, Simon
2011/1/7 Garick Hamlin gham...@isc.upenn.edu:
On Thu, Jan 06, 2011 at 07:47:39PM -0500, Cédric Villemain wrote:
2011/1/5 Magnus Hagander mag...@hagander.net:
On Wed, Jan 5, 2011 at 22:58, Dimitri Fontaine dimi...@2ndquadrant.fr
wrote:
Magnus Hagander mag...@hagander.net writes:
* Stefan
2011/1/7 Magnus Hagander mag...@hagander.net:
On Fri, Jan 7, 2011 at 01:47, Cédric Villemain
cedric.villemain.deb...@gmail.com wrote:
2011/1/5 Magnus Hagander mag...@hagander.net:
On Wed, Jan 5, 2011 at 22:58, Dimitri Fontaine dimi...@2ndquadrant.fr
wrote:
Magnus Hagander
On Thu, Jan 6, 2011 at 23:57, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Looks like pg_streamrecv creates the pg_xlog and pg_tblspc directories,
because they're not included in the streamed tar. Wouldn't it be better to
include them in the tar as empty directories at the
On Fri, Jan 7, 2011 at 02:15, Simon Riggs si...@2ndquadrant.com wrote:
On Wed, 2011-01-05 at 14:54 +0100, Magnus Hagander wrote:
The basic implementation is: Add a new command to the replication mode called
BASE_BACKUP, that will initiate a base backup, stream the contents (in tar
compatible
On Fri, Jan 7, 2011 at 01:47, Cédric Villemain
cedric.villemain.deb...@gmail.com wrote:
2011/1/5 Magnus Hagander mag...@hagander.net:
On Wed, Jan 5, 2011 at 22:58, Dimitri Fontaine dimi...@2ndquadrant.fr
wrote:
Magnus Hagander mag...@hagander.net writes:
* Stefan mentiond it might be useful
On Thu, Jan 06, 2011 at 07:47:39PM -0500, Cédric Villemain wrote:
2011/1/5 Magnus Hagander mag...@hagander.net:
On Wed, Jan 5, 2011 at 22:58, Dimitri Fontaine dimi...@2ndquadrant.fr
wrote:
Magnus Hagander mag...@hagander.net writes:
* Stefan mentiond it might be useful to put some
On Fri, Jan 07, 2011 at 10:26:29AM -0500, Garick Hamlin wrote:
On Thu, Jan 06, 2011 at 07:47:39PM -0500, Cédric Villemain wrote:
2011/1/5 Magnus Hagander mag...@hagander.net:
On Wed, Jan 5, 2011 at 22:58, Dimitri Fontaine dimi...@2ndquadrant.fr
wrote:
Magnus Hagander
On 05.01.2011 15:54, Magnus Hagander wrote:
* Suggestion from Heikki: perhaps at some point we're going to need a full
bison grammar for walsender commands.
Here's a patch for this (Also available at
g...@github.com:hlinnaka/postgres.git, branch streaming_base). I
thought I know our
On 05.01.2011 15:54, Magnus Hagander wrote:
I've implemented a frontend for this in pg_streamrecv, based on the assumption
that we wanted to include this in bin/ for 9.1 - and that it seems like a
reasonable place to put it. This can obviously be moved elsewhere if we want to.
That code needs a
On 06.01.2011 00:27, Dimitri Fontaine wrote:
Magnus Hagandermag...@hagander.net writes:
What about pg_streamrecv | gzip …, which has the big advantage of
That's part of what I meant with easier and more useful.
Well…
One thing to keep in mind is that if you do compression in libpq for
On Wed, Jan 5, 2011 at 23:58, Dimitri Fontaine dimi...@2ndquadrant.fr wrote:
* Stefan mentiond it might be useful to put some
posix_fadvise(POSIX_FADV_DONTNEED)
in the process that streams all the files out. Seems useful, as long as
that
doesn't kick them out of the cache *completely*,
On Wed, Jan 5, 2011 at 23:27, Dimitri Fontaine dimi...@2ndquadrant.fr wrote:
Magnus Hagander mag...@hagander.net writes:
Well, I would guess that if you're streaming the WAL files in parallel
while the base backup is taken, then you're able to have it all without
archiving setup, and the
On 05.01.2011 15:54, Magnus Hagander wrote:
Attached is an updated streaming base backup patch, based off the work
that Heikki started.
...
I've implemented a frontend for this in pg_streamrecv, based on the assumption
that we wanted to include this in bin/ for 9.1 - and that it seems like a
On Thu, Jan 6, 2011 at 23:57, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 05.01.2011 15:54, Magnus Hagander wrote:
Attached is an updated streaming base backup patch, based off the work
that Heikki started.
...
I've implemented a frontend for this in pg_streamrecv, based
2011/1/5 Magnus Hagander mag...@hagander.net:
On Wed, Jan 5, 2011 at 22:58, Dimitri Fontaine dimi...@2ndquadrant.fr wrote:
Magnus Hagander mag...@hagander.net writes:
* Stefan mentiond it might be useful to put some
posix_fadvise(POSIX_FADV_DONTNEED)
in the process that streams all the
On Wed, 2011-01-05 at 14:54 +0100, Magnus Hagander wrote:
The basic implementation is: Add a new command to the replication mode called
BASE_BACKUP, that will initiate a base backup, stream the contents (in tar
compatible format) of the data directory and all tablespaces, and then end
the
On 01/05/2011 02:54 PM, Magnus Hagander wrote:
[..]
Some remaining thoughts and must-dos:
* Compression: Do we want to be able to compress the backups server-side? Or
defer that to whenever we get compression in libpq? (you can still tunnel it
through for example SSH to get compression if
Magnus Hagander mag...@hagander.net writes:
Attached is an updated streaming base backup patch, based off the work
Thanks! :)
* Compression: Do we want to be able to compress the backups server-side? Or
defer that to whenever we get compression in libpq? (you can still tunnel it
through
On Wed, Jan 5, 2011 at 22:58, Dimitri Fontaine dimi...@2ndquadrant.fr wrote:
Magnus Hagander mag...@hagander.net writes:
Attached is an updated streaming base backup patch, based off the work
Thanks! :)
* Compression: Do we want to be able to compress the backups server-side? Or
defer
Magnus Hagander mag...@hagander.net writes:
Compression in libpq would be a nice way to solve it, later.
Yeah, I'm pretty much set on postponing that one.
+1, in case it was not clear for whoever's counting the votes :)
What about pg_streamrecv | gzip …, which has the big advantage of
57 matches
Mail list logo
