Re: [GENERAL] Data Replication

2008-12-12 Thread Markus Wanner
Hi, Joshua D. Drake wrote: c) Postgres-R for multi-master data replication, appears to be a code fork of PostgreSQL Not stable as far as I know. Correct, it's not meant to be stable at this stage of development. I'm a bit disturbed by the tag code fork, which has a rather negative

Re: [GENERAL] Data Replication

2008-12-12 Thread Markus Wanner
Hi, Joshua D. Drake wrote: On Wed, 2008-12-10 at 17:18 -0500, Rutherdale, Will wrote: I attempted some searches in various areas and came up with a bewildering array of results but no clear answer. Let's extend the list even further: h) If you are up for Java, you might like Sequoia. i)

Re: [GENERAL] Data Replication

2008-12-12 Thread Rutherdale, Will
...@commandprompt.com Cc: Rutherdale, Will; pgsql-general@postgresql.org Subject: Re: [GENERAL] Data Replication Hi, Joshua D. Drake wrote: On Wed, 2008-12-10 at 17:18 -0500, Rutherdale, Will wrote: I attempted some searches in various areas and came up with a bewildering array of results

Re: [GENERAL] Data Replication

2008-12-12 Thread Joshua D. Drake
On Fri, 2008-12-12 at 16:31 +0100, Markus Wanner wrote: Hi, Let me simply point out and clarify, that I have absolutely no intent to fork from Postgres. Quite the opposite, I'm interested in working together with other Postgres hackers. I think the point is that right now Postgres-R (just

Re: [GENERAL] Data Replication

2008-12-12 Thread Markus Wanner
Hello Joshua, Joshua D. Drake wrote: I think the point is that right now Postgres-R (just like Replicator) keeps its own tree that incorporates the PostgreSQL code. ..as does every other patch for Postgres before possibly it lands on mainline. But that's neither good nor bad per se, IMO.

Re: [GENERAL] Data Replication

2008-12-12 Thread Joshua D. Drake
On Fri, 2008-12-12 at 18:45 +0100, Markus Wanner wrote: Hello Joshua, Well, yeah, maybe Postgres-R is going to loose that sale as well. But hey, it's not long ago since you've open sourced it. What makes you think that you've already lost that sale? I for example didn't find time to look at

Re: [GENERAL] Data Replication

2008-12-11 Thread Tim Uckun
No probably not. I mean they are all pretty easy (especially log shipping) but it is definitely true they are slow, depending on the size of the database. As an alternative is there a clustering or multi master replication scheme that would be useful in a WAN? Preferably with a prefered

Re: [GENERAL] Data Replication

2008-12-11 Thread Andrew Sullivan
On Wed, Dec 10, 2008 at 08:41:30PM -0700, Scott Marlowe wrote: one of the real time replication. Failover in slony is pretty easy to do and happens in seconds. But you do have to resubscribe the master as a slave and copy everything over again after a failover to make the old master the new

Re: [GENERAL] Data Replication

2008-12-11 Thread Simon Riggs
On Wed, 2008-12-10 at 18:34 -0500, Rutherdale, Will wrote: Thanks very much, Steve. The main (but not only) type of data replication activity I'm interested in right now would be the warm standby. Thus it appears from the documents you showed me that log shipping is one solution currently

Re: [GENERAL] Data Replication

2008-12-11 Thread Joshua D. Drake
On Thu, 2008-12-11 at 17:09 +, Simon Riggs wrote: On Wed, 2008-12-10 at 18:34 -0500, Rutherdale, Will wrote: Thanks very much, Steve. Yes, everything you need for log shipping has been contributed to the main project. If you read things elsewhere, please refer closely to the docs which

Re: [GENERAL] Data Replication

2008-12-11 Thread Simon Riggs
On Thu, 2008-12-11 at 09:14 -0800, Joshua D. Drake wrote: On Thu, 2008-12-11 at 17:09 +, Simon Riggs wrote: On Wed, 2008-12-10 at 18:34 -0500, Rutherdale, Will wrote: Thanks very much, Steve. Yes, everything you need for log shipping has been contributed to the main project. If

Re: [GENERAL] Data Replication

2008-12-11 Thread Joshua D. Drake
On Thu, 2008-12-11 at 17:37 +, Simon Riggs wrote: On Thu, 2008-12-11 at 09:14 -0800, Joshua D. Drake wrote: I think this statement is misleading. The only thing core contains is the ability to use a bunch of utilities (with the exception of pg_standby) that aren't in core to provide

Re: [GENERAL] Data Replication

2008-12-11 Thread Simon Riggs
On Thu, 2008-12-11 at 09:52 -0800, Joshua D. Drake wrote: On Thu, 2008-12-11 at 17:37 +, Simon Riggs wrote: On Thu, 2008-12-11 at 09:14 -0800, Joshua D. Drake wrote: I think this statement is misleading. The only thing core contains is the ability to use a bunch of utilities

Re: [GENERAL] Data Replication

2008-12-11 Thread Joshua D. Drake
On Thu, 2008-12-11 at 19:24 +, Simon Riggs wrote: True, we rely on the existence of rsync, scp etc.. and go to great pains to provide as much choice as possible. If you think other things are required you are welcome to contribute them so they can be verified fault free by the

Re: [GENERAL] Data Replication

2008-12-11 Thread Simon Riggs
On Thu, 2008-12-11 at 11:29 -0800, Joshua D. Drake wrote: As I said before, if you think something is missing, submit a software or a doc patch and submit it to peer review. Until then, I think its misleading to claim that only your magic spice makes replication work correctly and to

Re: [GENERAL] Data Replication

2008-12-11 Thread Joshua D. Drake
On Thu, 2008-12-11 at 19:33 +, Simon Riggs wrote: On Thu, 2008-12-11 at 11:29 -0800, Joshua D. Drake wrote: As I said before, if you think something is missing, submit a software or a doc patch and submit it to peer review. Until then, I think its misleading to claim that only your

[GENERAL] Data Replication

2008-12-10 Thread Rutherdale, Will
Hi. I am trying to determine what kind of data replication is currently available in PostgreSQL. This is for purposes of examining capabilities of PostgreSQL as compared to other RDBMSs. I attempted some searches in various areas and came up with a bewildering array of results but no clear

Re: [GENERAL] Data Replication

2008-12-10 Thread Steve Atkins
On Dec 10, 2008, at 2:18 PM, Rutherdale, Will wrote: Hi. I am trying to determine what kind of data replication is currently available in PostgreSQL. This is for purposes of examining capabilities of PostgreSQL as compared to other RDBMSs. I attempted some searches in various areas and

Re: [GENERAL] Data Replication

2008-12-10 Thread Joshua D. Drake
On Wed, 2008-12-10 at 17:18 -0500, Rutherdale, Will wrote: Hi. I am trying to determine what kind of data replication is currently available in PostgreSQL. This is for purposes of examining capabilities of PostgreSQL as compared to other RDBMSs. I attempted some searches in various areas

Re: [GENERAL] Data Replication

2008-12-10 Thread Rutherdale, Will
Hope you don't mind my quoting style. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Steve Atkins Sent: 10 December 2008 17:50 To: PostgreSQL General Subject: Re: [GENERAL] Data Replication On Dec 10, 2008, at 2:18 PM, Rutherdale, Will wrote: Hi

Re: [GENERAL] Data Replication

2008-12-10 Thread Joshua D. Drake
On Wed, 2008-12-10 at 18:34 -0500, Rutherdale, Will wrote: Thanks very much, Steve. The main (but not only) type of data replication activity I'm interested in right now would be the warm standby. Thus it appears from the documents you showed me that log shipping is one solution currently

Re: [GENERAL] Data Replication

2008-12-10 Thread Rutherdale, Will
: Joshua D. Drake [mailto:[EMAIL PROTECTED] Sent: 10 December 2008 17:52 To: Rutherdale, Will Cc: pgsql-general@postgresql.org Subject: Re: [GENERAL] Data Replication On Wed, 2008-12-10 at 17:18 -0500, Rutherdale, Will wrote: Hi. I am trying to determine what kind of data replication is currently

Re: [GENERAL] Data Replication

2008-12-10 Thread Joshua D. Drake
On Wed, 2008-12-10 at 18:45 -0500, Rutherdale, Will wrote: Thanks, Joshua. As I mentioned to Steve, warm standby / log shipping seems to be the main feature I'm looking for. The PITR solution you mention: is that an improvement over regular log shipping? Or do I misunderstand where that

Re: [GENERAL] Data Replication

2008-12-10 Thread David Wall
We've done warm standby as you indicate, and we've not needed anything special. On the primary's postgresql.conf we use: archive_command = '~/postgresql/bin/copyWAL %p %f' Our copyWAL script is just a wrapper for 'scp' since we want to copy the data encrypted over the network: #!/bin/bash

Re: [GENERAL] Data Replication

2008-12-10 Thread Tim Uckun
On Thu, Dec 11, 2008 at 1:05 PM, David Wall [EMAIL PROTECTED] wrote: We've done warm standby as you indicate, and we've not needed anything special. Thanks for sharing your configuation. I have one additional question thought... How do you handle the reverting? For example. Say I have a

Re: [GENERAL] Data Replication

2008-12-10 Thread Joshua D. Drake
On Thu, 2008-12-11 at 15:21 +1300, Tim Uckun wrote: What happens when I bring the primary back on line. I now want this to be primary again and catch up on all the transactions that were sent to the secondary. I want the secondary to resume it's backup status. You have to run a new base

Re: [GENERAL] Data Replication

2008-12-10 Thread Tim Uckun
You have to run a new base backup and have the slave ship logs to the master. Mmmm. Does this backup have to be a full backup? What if your database is very large? I am hoping to get a setup which is similar to SQL server mirroring. It uses a witness server to keep track of who got what logs

Re: [GENERAL] Data Replication

2008-12-10 Thread Scott Marlowe
On Wed, Dec 10, 2008 at 7:40 PM, Tim Uckun [EMAIL PROTECTED] wrote: You have to run a new base backup and have the slave ship logs to the master. Mmmm. Does this backup have to be a full backup? What if your database is very large? Yes. Your backup is very large. I am hoping to get a

Re: [GENERAL] Data Replication

2008-12-10 Thread Joshua D. Drake
On Wed, 2008-12-10 at 20:41 -0700, Scott Marlowe wrote: On Wed, Dec 10, 2008 at 7:40 PM, Tim Uckun [EMAIL PROTECTED] wrote: Log shipping doesn't really lends itself to switching back and forth between masters and slaves. Really? It seems to me that you can make a base backup just as fast as

Re: [GENERAL] Data Replication

2008-12-10 Thread Scott Marlowe
On Wed, Dec 10, 2008 at 8:43 PM, Joshua D. Drake [EMAIL PROTECTED] wrote: On Wed, 2008-12-10 at 20:41 -0700, Scott Marlowe wrote: On Wed, Dec 10, 2008 at 7:40 PM, Tim Uckun [EMAIL PROTECTED] wrote: Log shipping doesn't really lends itself to switching back and forth between masters and

Re: [GENERAL] Data Replication

2008-12-10 Thread Joshua D. Drake
On Wed, 2008-12-10 at 21:39 -0700, Scott Marlowe wrote: On Wed, Dec 10, 2008 at 8:43 PM, Joshua D. Drake [EMAIL PROTECTED] wrote: On Wed, 2008-12-10 at 20:41 -0700, Scott Marlowe wrote: On Wed, Dec 10, 2008 at 7:40 PM, Tim Uckun [EMAIL PROTECTED] wrote: Log shipping doesn't really lends

Re: [GENERAL] Data replication through disk replication

2007-05-20 Thread Thomas Lopatic
So what happens in those cases where the primary node gets in trouble but isn't actually dead yet? Hmmm. Is this really a problem? Couldn't the secondary DRBD node simply stop accepting replicated data from the primary node before firing up postmaster? Then the postmaster on the primary DRBD

Re: [GENERAL] Data replication through disk replication

2007-05-20 Thread Andrew Sullivan
On Sun, May 20, 2007 at 12:01:46PM +0200, Thomas Lopatic wrote: Hmmm. Is this really a problem? Couldn't the secondary DRBD node simply stop accepting replicated data from the primary node before firing up postmaster? Then the postmaster on the primary DRBD node would only write locally and

Re: [GENERAL] Data replication through disk replication

2007-05-20 Thread Ben
On May 20, 2007, at 3:01 AM, Thomas Lopatic wrote: So what happens in those cases where the primary node gets in trouble but isn't actually dead yet? Hmmm. Is this really a problem? The problem comes when the primary is cannot replicate to the secondary but can, for whatever reason, still

Re: [GENERAL] Data replication through disk replication

2007-05-20 Thread Hannes Dorbath
Thomas Lopatic wrote: So what happens in those cases where the primary node gets in trouble but isn't actually dead yet? Hmmm. Is this really a problem? Couldn't the secondary DRBD node simply stop accepting replicated data from the primary node before firing up postmaster? Then the

Re: [GENERAL] Data replication through disk replication

2007-05-20 Thread Hannes Dorbath
Ben wrote: On May 20, 2007, at 3:01 AM, Thomas Lopatic wrote: The problem comes when the primary is cannot replicate to the secondary but can, for whatever reason, still talk to clients. If a client is told something is committed but that commit isn't replicated, you have a problem. Right,

Re: [GENERAL] Data replication through disk replication

2007-05-19 Thread Andrew Sullivan
On Fri, May 18, 2007 at 05:03:30PM -0700, Ben wrote: that all changes are replicated, it won't say an fsync is finished until it's finished on the remote host too, and it won't let you mount the block device on the slave system (at least with 0.7x). How can it guarantee these things? The

Re: [GENERAL] Data replication through disk replication

2007-05-19 Thread Joris Dobbelsteen
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Andrew Sullivan Sent: zaterdag 19 mei 2007 15:28 To: pgsql-general@postgresql.org Subject: Re: [GENERAL] Data replication through disk replication On Fri, May 18, 2007 at 05:03:30PM -0700, Ben wrote

Re: [GENERAL] Data replication through disk replication

2007-05-19 Thread Ben
Er, yes, sorry, I didn't mean to imply that you should run without some kind of STONITH solution, to catch the case when the link DRDB uses goes down but the other network links are still working fine. It's in the common case, when everything is working, that DRBD won't accidentally let

Re: [GENERAL] Data replication through disk replication

2007-05-19 Thread Joshua D. Drake
Alvaro Herrera wrote: Ben wrote: If you're just looking for a way to have high availability and you're ok being tied to linux, DRBD is a good way to go. It keeps things simple in that all changes are replicated, it won't say an fsync is finished until it's finished on the remote host too,

Re: [GENERAL] Data replication through disk replication

2007-05-18 Thread Martijn van Oosterhout
On Fri, May 18, 2007 at 02:48:03PM +0200, Thomas Lopatic wrote: What I keep wondering: Isn't there substantial risk involved? I mean, suppose the master fails in the middle of a write. Isn't there the possibility that this corrupts the database? How robust is PostgreSQL's on-disk file format

Re: [GENERAL] Data replication through disk replication

2007-05-18 Thread Andrew Sullivan
On Fri, May 18, 2007 at 02:48:03PM +0200, Thomas Lopatic wrote: I am currently looking into replicated two-node master/slave PostgreSQL environments. Lately I've heard more and more people recommend replicating data from the master to the slave at the disk device level as opposed to the DBMS

Re: [GENERAL] Data replication through disk replication

2007-05-18 Thread Thomas Lopatic
[Disk-level replication instead of using Slony-I] What are the reasons they recommend this? (See my blathering in another thread about how often the hand-wavy recommendations that are made on this topic can really bite you hard if you don't know all the intimate details underneath.) The

Re: [GENERAL] Data replication through disk replication

2007-05-18 Thread Andrew Sullivan
On Fri, May 18, 2007 at 07:55:24PM +0200, Thomas Lopatic wrote: For Slony-I it seems to me that my risk is losing a couple of rows in my database, which is something that I could live with. For disk-level replication it seems to me that, in case of a master failure, I could easily end up with

Re: [GENERAL] Data replication through disk replication

2007-05-18 Thread Ben
If you're just looking for a way to have high availability and you're ok being tied to linux, DRBD is a good way to go. It keeps things simple in that all changes are replicated, it won't say an fsync is finished until it's finished on the remote host too, and it won't let you mount the block

Re: [GENERAL] Data replication through disk replication

2007-05-18 Thread Alvaro Herrera
Ben wrote: If you're just looking for a way to have high availability and you're ok being tied to linux, DRBD is a good way to go. It keeps things simple in that all changes are replicated, it won't say an fsync is finished until it's finished on the remote host too, Oh, so that's how it

Re: [GENERAL] Data replication through disk replication

2007-05-18 Thread Ben
You pay a price writes, but with write caching enabled on your (battery-backed, of course) RAID card and using gigabit, it's easy to get 100MB/s throughput. It's also easy to replicate different block devices over separate network links, if that becomes your bottleneck. On May 18, 2007,