Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-07-15 Thread Bruce Momjian
Alvaro Herrera wrote: > Bruce Momjian wrote: > > Alvaro Herrera wrote: > > > Bruce Momjian wrote: > > > > > > > Added to TODO: > > > > > > > > o Reduce PITR WAL file size by removing full page writes and > > > > by removing trailing bytes to improve compression > > > > > > If w

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-07-15 Thread Alvaro Herrera
Bruce Momjian wrote: > Alvaro Herrera wrote: > > Bruce Momjian wrote: > > > > > Added to TODO: > > > > > > o Reduce PITR WAL file size by removing full page writes and > > > by removing trailing bytes to improve compression > > > > If we remove full page writes, how does hint b

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-07-15 Thread Bruce Momjian
Alvaro Herrera wrote: > Bruce Momjian wrote: > > > Added to TODO: > > > > o Reduce PITR WAL file size by removing full page writes and > > by removing trailing bytes to improve compression > > If we remove full page writes, how does hint bit setting get propagated > to the slav

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-07-15 Thread Alvaro Herrera
Bruce Momjian wrote: > Added to TODO: > > o Reduce PITR WAL file size by removing full page writes and > by removing trailing bytes to improve compression If we remove full page writes, how does hint bit setting get propagated to the slave? -- Alvaro Herrera

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-07-15 Thread Bruce Momjian
Tom Lane wrote: > Bruce Momjian <[EMAIL PROTECTED]> writes: > > Agreed. I realize why we are not zeroing those bytes (for performance), > > but can't we have the archiver zero those bytes before calling the > > 'archive_command'? > > The archiver doesn't know any more about where the end-of-data

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-16 Thread Kevin Grittner
>>> On Mon, Jun 9, 2008 at 9:48 PM, in message <[EMAIL PROTECTED]>, Greg Smith <[EMAIL PROTECTED]> wrote: > On Mon, 9 Jun 2008, Tom Lane wrote: > >> It should also be pointed out that the whole thing becomes uninteresting >> if we get real-time log shipping implemented. So I see absolutely no >

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-10 Thread ITAGAKI Takahiro
Josh Berkus <[EMAIL PROTECTED]> wrote: > I still see having 2 different settings: > > Synchronous: XID visibility is pushed to the master. Maintains synchronous > failover, and users are expected to run *1* master to *1* slave for most > installations. > > Asynchronous: replication stops on

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-10 Thread Josh Berkus
All, > > For the slave to not interfere with the master at all, we would need to > > delay application of WAL files on each slave until visibility on that > > slave allows the WAL to be applied, but in that case we would have > > long-running transactions delay data visibility of all slave session

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-10 Thread Heikki Linnakangas
Gregory Stark wrote: Instead of zeroing bytes and depending on compression why not just pass an extra parameter to the archive command with the offset to the logical end of data. Because the archiver process doesn't have that information. -- Heikki Linnakangas EnterpriseDB http://www.ent

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-10 Thread Gregory Stark
"Greg Smith" <[EMAIL PROTECTED]> writes: > On Mon, 9 Jun 2008, Tom Lane wrote: > >> It should also be pointed out that the whole thing becomes uninteresting >> if we get real-time log shipping implemented. So I see absolutely no >> point in spending time integrating pg_clearxlogtail now. > > Ther

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-09 Thread Koichi Suzuki
Just for information. In terms of archive compression, I have archive log compression which will be found in http://pgfoundry.org/projects/pglesslog/ This feature is also included in NTT's synchronized log shipping replication presented in the last PGCon. 2008/6/10 Greg Smith <[EMAIL PROTECTED]>

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-09 Thread Greg Smith
On Mon, 9 Jun 2008, Tom Lane wrote: It should also be pointed out that the whole thing becomes uninteresting if we get real-time log shipping implemented. So I see absolutely no point in spending time integrating pg_clearxlogtail now. There are remote replication scenarios over a WAN (mainly

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-09 Thread Tom Lane
Bruce Momjian <[EMAIL PROTECTED]> writes: > Agreed. I realize why we are not zeroing those bytes (for performance), > but can't we have the archiver zero those bytes before calling the > 'archive_command'? The archiver doesn't know any more about where the end-of-data is than the archive_command

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-09 Thread Tom Lane
Bruce Momjian <[EMAIL PROTECTED]> writes: > Gurjeet Singh wrote: >> There could be multiple slaves following a master, some serving > For the slave to not interfere with the master at all, we would need to > delay application of WAL files on each slave until visibility on that > slave allows the W

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-09 Thread Alvaro Herrera
Bruce Momjian wrote: > Agreed. I realize why we are not zeroing those bytes (for performance), > but can't we have the archiver zero those bytes before calling the > 'archive_command'? Perhaps make the zeroing user-settable. -- Alvaro Herrerahttp://www.CommandPr

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-09 Thread Bruce Momjian
Andreas 'ads' Scherbaum wrote: > On Fri, 30 May 2008 16:22:41 -0400 (EDT) Greg Smith wrote: > > > On Fri, 30 May 2008, Andreas 'ads' Scherbaum wrote: > > > > > Then you ship 16 MB binary stuff every 30 second or every minute but > > > you only have some kbyte real data in the logfile. > > > > No

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-09 Thread Bruce Momjian
Gurjeet Singh wrote: > On Fri, May 30, 2008 at 10:40 AM, Tom Lane <[EMAIL PROTECTED]> wrote: > > > But since you mention it: one of the plausible answers for fixing the > > vacuum problem for read-only slaves is to have the slaves push an xmin > > back upstream to the master to prevent premature v

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-05 Thread Gregory Stark
"Jeff Davis" <[EMAIL PROTECTED]> writes: > On Wed, 2008-06-04 at 14:17 +0300, Heikki Linnakangas wrote: >> > Would that also cover possible differences in page size, 32bit OS vs. >> > 64bit OS, different timestamp flavour, etc. issues ? AFAIR, all these >> > things can have an influence on how the

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Koichi Suzuki
If the version of the master and the slave is different and we'd still like to allow log shipping replication, we need a negotiation if WAL format for the two is compatible. I hope it is not in our scope and I'm worrying too much. 2008/6/5 Tom Lane <[EMAIL PROTECTED]>: > "Koichi Suzuki" <[EMA

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Tom Lane
"Koichi Suzuki" <[EMAIL PROTECTED]> writes: > Well, WAL format doesn't only depend on WAL itself, but also depend on > each resource manager. If we introduce WAL format version > identification, ISTM that we have to take care of the matching of > resource manager in the master and the slave as we

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Koichi Suzuki
Well, WAL format doesn't only depend on WAL itself, but also depend on each resource manager. If we introduce WAL format version identification, ISTM that we have to take care of the matching of resource manager in the master and the slave as well. 2008/6/4 Heikki Linnakangas <[EMAIL PROTECTED]>

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Jeff Davis
On Wed, 2008-06-04 at 14:23 -0400, Tom Lane wrote: > That is covered by pg_control, at least to the extent of forcing the > same value of LC_COLLATE. But the same LC_COLLATE means different things on different systems. Even "en_US" means something different on Mac versus Linux. Regards, J

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Tom Lane
Jeff Davis <[EMAIL PROTECTED]> writes: > On Wed, 2008-06-04 at 14:17 +0300, Heikki Linnakangas wrote: >> These are already covered by the information in pg_control. > Another thing that can change between systems is the collation behavior, > which can corrupt indexes (and other bad things). That

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Jeff Davis
On Wed, 2008-06-04 at 14:17 +0300, Heikki Linnakangas wrote: > > Would that also cover possible differences in page size, 32bit OS vs. > > 64bit OS, different timestamp flavour, etc. issues ? AFAIR, all these > > things can have an influence on how the data is written and possibly > > make the WAL

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Simon Riggs
On Wed, 2008-06-04 at 11:37 -0400, Tom Lane wrote: > This thread is getting out of hand, actually. Agreed. We should start new threads for specific things. Please. > However, since by definition pg_control doesn't change in a minor > upgrade, there isn't any easy way to enforce a rule like "sla

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Tom Lane
Hannu Krosing <[EMAIL PROTECTED]> writes: > On Wed, 2008-06-04 at 10:40 -0400, Tom Lane wrote: >> "Heikki Linnakangas" <[EMAIL PROTECTED]> writes: > Hmm, WAL version compatibility is an interesting question. Most minor > releases hasn't changed the WAL format, and it would be nice to allow > runn

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Hannu Krosing
On Wed, 2008-06-04 at 10:40 -0400, Tom Lane wrote: > "Heikki Linnakangas" <[EMAIL PROTECTED]> writes: > > Hmm, WAL version compatibility is an interesting question. Most minor > > releases hasn't changed the WAL format, and it would be nice to allow > > running different minor versions in the mas

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Tom Lane
"Heikki Linnakangas" <[EMAIL PROTECTED]> writes: > Hmm, WAL version compatibility is an interesting question. Most minor > releases hasn't changed the WAL format, and it would be nice to allow > running different minor versions in the master and slave in those cases. > But it's certainly not unh

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Andrew Sullivan
On Wed, Jun 04, 2008 at 09:24:20AM +0200, Markus Schiltknecht wrote: > > Are the slides of your PgCon talk available for download somewhere? There weren't any slides, really (there were 4 that I put up in case the cases I was discussing needed back-references, but they didn't). Joshua tells me tha

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Heikki Linnakangas
Teodor Sigaev wrote: Is it possible to use catalog version number as WAL version? No, because we don't change the catalog version number in minor releases, even though we might change WAL format. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Heikki Linnakangas
Csaba Nagy wrote: On Wed, 2008-06-04 at 11:13 +0300, Heikki Linnakangas wrote: Hmm, WAL version compatibility is an interesting question. Most minor releases hasn't changed the WAL format, and it would be nice to allow running different minor versions in the master and slave in those cases. Bu

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Csaba Nagy
On Wed, 2008-06-04 at 11:13 +0300, Heikki Linnakangas wrote: > Hmm, WAL version compatibility is an interesting question. Most minor > releases hasn't changed the WAL format, and it would be nice to allow > running different minor versions in the master and slave in those cases. > But it's certa

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Teodor Sigaev
Hmm, WAL version compatibility is an interesting question. Most minor releases hasn't changed the WAL format, and it would be nice to allow As I remember, high minor version should read all WALs from lowers, but it isn't true for opposite case and between different major versions. running di

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Heikki Linnakangas
Stephen Denne wrote: Hannu Krosing wrote: The simplest form of synchronous wal shipping would not even need postgresql running on slave, just a small daemon which reports when wal blocks are a) received and b) synced to disk. While that does sound simple, I'd presume that most people would w

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Markus Schiltknecht
Hello Andrew, Andrew Sullivan wrote: Yes. And silent as ever. :-) Are the slides of your PgCon talk available for download somewhere? BTW: up until recently, there was yet another mailing list: [EMAIL PROTECTED] It was less focused on hooks and got at least some traffic. :-) Are those mail

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-03 Thread Stephen Denne
Hannu Krosing wrote: > The simplest form of synchronous wal shipping would not even need > postgresql running on slave, just a small daemon which > reports when wal > blocks are a) received and b) synced to disk. While that does sound simple, I'd presume that most people would want the guarante

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-03 Thread Andrew Sullivan
On Sun, Jun 01, 2008 at 01:43:22PM -0400, Tom Lane wrote: > power to him. (Is the replica-hooks-discuss list still working?) But Yes. And silent as ever. :-) A -- Andrew Sullivan [EMAIL PROTECTED] +1 503 667 4564 x104 http://www.commandprompt.com/ -- Sent via pgsql-hackers mailing list (pg

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-03 Thread Hannu Krosing
On Mon, 2008-06-02 at 22:40 +0200, Andreas 'ads' Scherbaum wrote: > On Mon, 02 Jun 2008 11:52:05 -0400 Chris Browne wrote: > > > [EMAIL PROTECTED] ("Andreas 'ads' Scherbaum") writes: > > > On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: > > > > > >> Well, yes, but you do know about archiv

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-02 Thread Andreas 'ads' Scherbaum
On Mon, 02 Jun 2008 11:52:05 -0400 Chris Browne wrote: > [EMAIL PROTECTED] ("Andreas 'ads' Scherbaum") writes: > > On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: > > > >> Well, yes, but you do know about archive_timeout, right? No need to wait > >> 2 hours. > > > > Then you ship 16 MB b

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-02 Thread Hannu Krosing
On Thu, 2008-05-29 at 23:37 +0200, Mathias Brossard wrote: > I pointed out that the NTT solution is synchronous because Tom said in > the first part of his email that: > > > In practice, simple asynchronous single-master-multiple-slave > > replication covers a respectable fraction of use cases

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-02 Thread Chris Browne
[EMAIL PROTECTED] ("Andreas 'ads' Scherbaum") writes: > On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: > >> Well, yes, but you do know about archive_timeout, right? No need to wait >> 2 hours. > > Then you ship 16 MB binary stuff every 30 second or every minute but > you only have some k

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-01 Thread Robert Hodges
Hi Hannu, Hi Hannu, On 6/1/08 2:14 PM, "Hannu Krosing" <[EMAIL PROTECTED]> wrote: > >> As a consequence, I don¹t see how you can get around doing some sort >> of row-based replication like all the other databases. > > Is'nt WAL-base replication "some sort of row-based replication" ? > Yes, in t

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-01 Thread Hannu Krosing
On Thu, 2008-05-29 at 12:05 -0700, Robert Hodges wrote: > Hi everyone, > > First of all, I’m absolutely delighted that the PG community is > thinking seriously about replication. > > Second, having a solid, easy-to-use database availability solution > that works more or less out of the box wou

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-01 Thread Hannu Krosing
On Thu, 2008-05-29 at 13:37 -0400, Tom Lane wrote: > David Fetter <[EMAIL PROTECTED]> writes: > > On Thu, May 29, 2008 at 08:46:22AM -0700, Joshua D. Drake wrote: > >> The only question I have is... what does this give us that PITR > >> doesn't give us? > > > It looks like a wrapper for PITR to me

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-01 Thread Hannu Krosing
On Fri, 2008-05-30 at 15:16 -0400, Robert Treat wrote: > On Friday 30 May 2008 01:10:20 Tom Lane wrote: > > Greg Smith <[EMAIL PROTECTED]> writes: > > > I fully accept that it may be the case that it doesn't make technical > > > sense to tackle them in any order besides sync->read-only slaves becau

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-01 Thread James Mansion
Aidan Van Dyk wrote: The whole single-threaded WAL replay problem is going to rear it's ugly head here too, and mean that a slave *won't* be able to keep up with a busy master if it's actually trying to apply all the changes in real-time. Is there a reason to commit at the same points that the ma

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-01 Thread James Mansion
David Fetter wrote: This part is a deal-killer. It's a giant up-hill slog to sell warm standby to those in charge of making resources available because the warm standby machine consumes SA time, bandwidth, power, rack space, etc., but provides no tangible benefit, and this feature would have exa

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-01 Thread Dawid Kuroczko
On Thu, May 29, 2008 at 4:12 PM, Tom Lane <[EMAIL PROTECTED]> wrote: > The Postgres core team met at PGCon to discuss a few issues, the largest > of which is the need for simple, built-in replication for PostgreSQL. [...] > We believe that the most appropriate base technology for this is 1> probabl

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-01 Thread Tom Lane
"Merlin Moncure" <[EMAIL PROTECTED]> writes: > On Sun, Jun 1, 2008 at 11:58 AM, Robert Hodges > <[EMAIL PROTECTED]> wrote: >> My point here is that with reasonably small extensions to the core you can >> build products that are a lot better than SLONY. > These issues are much discussed and well un

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-01 Thread Merlin Moncure
On Sun, Jun 1, 2008 at 11:58 AM, Robert Hodges <[EMAIL PROTECTED]> wrote: > Hi Merlin, > > My point here is that with reasonably small extensions to the core you can > build products that are a lot better than SLONY. Triggers do not cover > DDL, among other issues, and it's debatable whether they

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-01 Thread Robert Hodges
Hi Merlin, My point here is that with reasonably small extensions to the core you can build products that are a lot better than SLONY. Triggers do not cover DDL, among other issues, and it's debatable whether they are the best way to implement quorum policies like Google's semi-synchronous re

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-31 Thread Joshua D. Drake
Merlin Moncure wrote: On Sat, May 31, 2008 at 2:18 AM, Mike Rylander <[EMAIL PROTECTED]> wrote: On Fri, May 30, 2008 at 6:47 PM, Andreas 'ads' Scherbaum Compression especially is going to negate one of the big advantages of wal shipping, namely that it is cheap investment in terms of load to

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-31 Thread Merlin Moncure
On Sat, May 31, 2008 at 2:18 AM, Mike Rylander <[EMAIL PROTECTED]> wrote: > On Fri, May 30, 2008 at 6:47 PM, Andreas 'ads' Scherbaum > <[EMAIL PROTECTED]> wrote: >> On Fri, 30 May 2008 17:05:57 -0400 Andrew Dunstan wrote: >>> Andreas 'ads' Scherbaum wrote: >>> > On Thu, 29 May 2008 23:02:56 -0400 A

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Mike Rylander
On Fri, May 30, 2008 at 6:47 PM, Andreas 'ads' Scherbaum <[EMAIL PROTECTED]> wrote: > On Fri, 30 May 2008 17:05:57 -0400 Andrew Dunstan wrote: >> Andreas 'ads' Scherbaum wrote: >> > On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: >> > >> >> Well, yes, but you do know about archive_timeout,

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Dimitri Fontaine
Le vendredi 30 mai 2008, Dimitri Fontaine a écrit : > This way, no need to switch IP addresses, the clients just connect as usual > and get results back and do not have to know whether the host they're > qerying against is a slave or a master. This level of smartness is into > -core. Oh, and if yo

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Gurjeet Singh
On Sat, May 31, 2008 at 3:41 AM, Greg Smith <[EMAIL PROTECTED]> wrote: > On Sat, 31 May 2008, Gurjeet Singh wrote: > > Not if you use pg_clearxlogtail >>> >> >> This means we need to modify pg_standby to not check for filesize when >> reading XLogs. >> > > No, the idea is that you run the segment

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Andrew Dunstan
Tatsuo Ishii wrote: Andreas 'ads' Scherbaum wrote: On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: Well, yes, but you do know about archive_timeout, right? No need to wait 2 hours. Then you ship 16 MB binary stuff every 30 second or every minute but you

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Andreas 'ads' Scherbaum
On Fri, 30 May 2008 17:05:57 -0400 Andrew Dunstan wrote: > Andreas 'ads' Scherbaum wrote: > > On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: > > > >> Well, yes, but you do know about archive_timeout, right? No need to wait > >> 2 hours. > > > > Then you ship 16 MB binary stuff every 30 s

Fwd: Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Csaba Nagy
[Looks like this mail missed the hackers list on reply to all, I wonder how it could happen... so I forward it] On Thu, 2008-05-29 at 17:00 +0100, Dave Page wrote: > Yes, we're talking real-time streaming (synchronous) log shipping. Is there any design already how would this be implemented ? Som

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Josh Berkus
Andrew, > Sure there's a price to pay. But that doesn't mean the facility doesn't > exist. And I rather suspect that most of Josh's customers aren't too > concerned about traffic charges or affected by such bandwidth > restrictions. Certainly, none of my clients are, and they aren't in the > giant

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Simon Riggs
On Thu, 2008-05-29 at 10:12 -0400, Tom Lane wrote: > The Postgres core team met at PGCon to discuss a few issues, the largest > of which is the need for simple, built-in replication for PostgreSQL. > Historically the project policy has been to avoid putting replication > into core PostgreSQL, so as

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Andreas 'ads' Scherbaum
On Fri, 30 May 2008 16:22:41 -0400 (EDT) Greg Smith wrote: > On Fri, 30 May 2008, Andreas 'ads' Scherbaum wrote: > > > Then you ship 16 MB binary stuff every 30 second or every minute but > > you only have some kbyte real data in the logfile. > > Not if you use pg_clearxlogtail ( > http://www.2

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Dimitri Fontaine
Le vendredi 30 mai 2008, Tom Lane a écrit : > No, I think it would be a useless expenditure of energy. Failover > includes a lot of things that are not within our purview: switching > IP addresses to point to the new server, some kind of STONITH solution > to keep the original master from coming b

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Simon Riggs
On Fri, 2008-05-30 at 01:10 -0400, Tom Lane wrote: > Greg Smith <[EMAIL PROTECTED]> writes: > > I fully accept that it may be the case that it doesn't make technical > > sense to tackle them in any order besides sync->read-only slaves because > > of dependencies in the implementation between the

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Greg Smith
On Sat, 31 May 2008, Gurjeet Singh wrote: Not if you use pg_clearxlogtail This means we need to modify pg_standby to not check for filesize when reading XLogs. No, the idea is that you run the segments through pg_clearxlogtail | gzip, which then compresses lightly used segments massively be

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Joshua D. Drake
On Sat, 2008-05-31 at 02:48 +0530, Gurjeet Singh wrote: > > Not if you use pg_clearxlogtail > ( http://www.2ndquadrant.com/replication.htm ), which got lost > in the giant March commitfest queue but should probably wander > into contrib as part of 8.4. >

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Tatsuo Ishii
> Andreas 'ads' Scherbaum wrote: > > On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: > > > > > >> Well, yes, but you do know about archive_timeout, right? No need to wait > >> 2 hours. > >> > > > > Then you ship 16 MB binary stuff every 30 second or every minute but > > you only h

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Gurjeet Singh
On Sat, May 31, 2008 at 1:52 AM, Greg Smith <[EMAIL PROTECTED]> wrote: > On Fri, 30 May 2008, Andreas 'ads' Scherbaum wrote: > > Then you ship 16 MB binary stuff every 30 second or every minute but >> you only have some kbyte real data in the logfile. >> > > Not if you use pg_clearxlogtail ( > ht

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Robert Treat
On Friday 30 May 2008 01:10:20 Tom Lane wrote: > Greg Smith <[EMAIL PROTECTED]> writes: > > I fully accept that it may be the case that it doesn't make technical > > sense to tackle them in any order besides sync->read-only slaves because > > of dependencies in the implementation between the two. >

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Robert Treat
On Thursday 29 May 2008 20:31:31 Greg Smith wrote: > On Thu, 29 May 2008, Tom Lane wrote: > > There's no point in having read-only slave queries if you don't have a > > trustworthy method of getting the data to them. > > This is a key statement that highlights the difference in how you're > thinkin

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Andrew Dunstan
Andreas 'ads' Scherbaum wrote: On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: Well, yes, but you do know about archive_timeout, right? No need to wait 2 hours. Then you ship 16 MB binary stuff every 30 second or every minute but you only have some kbyte real data in the lo

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Robert Treat
On Thursday 29 May 2008 22:59:21 Merlin Moncure wrote: > On Thu, May 29, 2008 at 9:26 PM, Josh Berkus <[EMAIL PROTECTED]> wrote: > >> I fully accept that it may be the case that it doesn't make technical > >> sense to tackle them in any order besides sync->read-only slaves because > >> of dependenc

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Simon Riggs
On Fri, 2008-05-30 at 11:12 -0700, Robert Hodges wrote: > This is clearly an important use case but it also seems clear that > the WAL approach is not a general-purpose approach to replication. I think we cannot make such a statement yet, if ever. I would note that log-based replication is now t

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Gurjeet Singh
On Thu, May 29, 2008 at 7:42 PM, Tom Lane <[EMAIL PROTECTED]> wrote: > The big problem > is that long-running slave-side queries might still need tuples that are > vacuumable on the master, and so replication of vacuuming actions would > cause the slave's queries to deliver wrong answers. Anothe

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Greg Smith
On Fri, 30 May 2008, Andreas 'ads' Scherbaum wrote: Then you ship 16 MB binary stuff every 30 second or every minute but you only have some kbyte real data in the logfile. Not if you use pg_clearxlogtail ( http://www.2ndquadrant.com/replication.htm ), which got lost in the giant March commit

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Andreas 'ads' Scherbaum
On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: > Well, yes, but you do know about archive_timeout, right? No need to wait > 2 hours. Then you ship 16 MB binary stuff every 30 second or every minute but you only have some kbyte real data in the logfile. This must be taken into account,

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Andreas 'ads' Scherbaum
On Thu, 29 May 2008 18:29:01 -0400 Tom Lane wrote: > Dimitri Fontaine <[EMAIL PROTECTED]> writes: > > While at it, would it be possible for the "simple" part of the core > > team statement to include automatic failover? > > No, I think it would be a useless expenditure of energy. Failover > in

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Andreas 'ads' Scherbaum
On Thu, 29 May 2008 09:22:26 -0700 Steve Atkins wrote: > On May 29, 2008, at 9:12 AM, David Fetter wrote: > > > Either one of these would be great, but something that involves > > machines that stay useless most of the time is just not going to work. > > I have customers who are thinking about wa

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Simon Riggs
On Fri, 2008-05-30 at 12:31 +0530, Gurjeet Singh wrote: > On Fri, May 30, 2008 at 10:40 AM, Tom Lane <[EMAIL PROTECTED]> wrote: > But since you mention it: one of the plausible answers for > fixing the > vacuum problem for read-only slaves is to have the slaves push >

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Simon Riggs
On Fri, 2008-05-30 at 11:30 -0400, Andrew Dunstan wrote: > > Tom Lane wrote: > > Simon Riggs <[EMAIL PROTECTED]> writes: > > > >> On Fri, 2008-05-30 at 12:31 +0530, Gurjeet Singh wrote: > >> > >>> On Fri, May 30, 2008 at 10:40 AM, Tom Lane <[EMAIL PROTECTED]> wrote: > >>> > Bu

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Merlin Moncure
On Fri, May 30, 2008 at 9:31 AM, Marko Kreen <[EMAIL PROTECTED]> wrote: > On 5/30/08, Gurjeet Singh <[EMAIL PROTECTED]> wrote: >> >> I think it would be best to not make the slave interfere with the master's >> operations; that's only going to increase the operational complexity of such >> a soluti

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Robert Hodges
Hi Tom, Thanks for the reasoned reply. As you saw from point #2 in my comments, I think you should do this feature. I hope this answers Josh Berkus' concern about my comments. You make a very interesting comment which seems to go to the heart of this design approach: > About the only thing tha

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Tom Lane
Simon Riggs <[EMAIL PROTECTED]> writes: > On Fri, 2008-05-30 at 12:31 +0530, Gurjeet Singh wrote: >> On Fri, May 30, 2008 at 10:40 AM, Tom Lane <[EMAIL PROTECTED]> wrote: >>> But since you mention it: one of the plausible answers for fixing the >>> vacuum problem for read-only slaves is to have the

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Andrew Dunstan
Tom Lane wrote: Simon Riggs <[EMAIL PROTECTED]> writes: On Fri, 2008-05-30 at 12:31 +0530, Gurjeet Singh wrote: On Fri, May 30, 2008 at 10:40 AM, Tom Lane <[EMAIL PROTECTED]> wrote: But since you mention it: one of the plausible answers for fixing the vacuum problem for read-o

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Marko Kreen
On 5/30/08, Gurjeet Singh <[EMAIL PROTECTED]> wrote: > On Fri, May 30, 2008 at 10:40 AM, Tom Lane <[EMAIL PROTECTED]> wrote: > > But since you mention it: one of the plausible answers for fixing the > > vacuum problem for read-only slaves is to have the slaves push an xmin > > back upstream to the

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Andrew Sullivan
On Thu, May 29, 2008 at 01:58:34PM -0700, David Fetter wrote: > If people on core had come to the idea that we needed to build in > replication *before* 8.3 came out, they certainly didn't announce it. > > Now is a great time to mention this because it gives everybody time to: > > 1. Come to a

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Gurjeet Singh
On Fri, May 30, 2008 at 10:40 AM, Tom Lane <[EMAIL PROTECTED]> wrote: > But since you mention it: one of the plausible answers for fixing the > vacuum problem for read-only slaves is to have the slaves push an xmin > back upstream to the master to prevent premature vacuuming. The current > design

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Tom Lane
Greg Smith <[EMAIL PROTECTED]> writes: > I fully accept that it may be the case that it doesn't make technical > sense to tackle them in any order besides sync->read-only slaves because > of dependencies in the implementation between the two. Well, it's certainly not been my intention to suggest

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Andrew Dunstan
Josh Berkus wrote: Greg, I fully accept that it may be the case that it doesn't make technical sense to tackle them in any order besides sync->read-only slaves because of dependencies in the implementation between the two. If that's the case, it would be nice to explicitly spell out what

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Merlin Moncure
On Thu, May 29, 2008 at 9:26 PM, Josh Berkus <[EMAIL PROTECTED]> wrote: >> I fully accept that it may be the case that it doesn't make technical >> sense to tackle them in any order besides sync->read-only slaves because >> of dependencies in the implementation between the two. If that's the >> ca

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Andrew Sullivan
On Thu, May 29, 2008 at 07:02:56PM -0400, Tom Lane wrote: > People want the bits to go from point A to point B; they don't want > to have to research, design, test, and administer their own solution > for moving the bits. I agree with this. I think I probably know as well as most people -- per

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Josh Berkus
Greg, > I fully accept that it may be the case that it doesn't make technical > sense to tackle them in any order besides sync->read-only slaves because > of dependencies in the implementation between the two. If that's the > case, it would be nice to explicitly spell out what that was to deflect

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Joshua D. Drake
Tom Lane wrote: There's no point in having read-only slave queries if you don't have a trustworthy method of getting the data to them. O.k. I was with you until here. Log shipping ala pg_standby works fine now sans read-only slave. No, it isn't out of the box which I can see an argument for b

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Aidan Van Dyk
* Tom Lane <[EMAIL PROTECTED]> [080529 20:22]: > "Joshua D. Drake" <[EMAIL PROTECTED]> writes: > > I think maybe my actual argument isn't coming through. What I am arguing > > for is not shipping XY without Z. That is all. (and no, I don't think we > > should hold up 8.4). > > So we should keep al

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Greg Smith
On Thu, 29 May 2008, Tom Lane wrote: There's no point in having read-only slave queries if you don't have a trustworthy method of getting the data to them. This is a key statement that highlights the difference in how you're thinking about this compared to some other people here. As far as s

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Merlin Moncure
On Thu, May 29, 2008 at 7:12 PM, Joshua D. Drake <[EMAIL PROTECTED]> wrote: > On Thu, 2008-05-29 at 19:02 -0400, Tom Lane wrote: >> I think we have nontrivial >> work in front of us to build a simple, reliable, community-tested >> log shipping solution; and it's not very sexy work either. But it

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Tom Lane
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > I think maybe my actual argument isn't coming through. What I am arguing > for is not shipping XY without Z. That is all. (and no, I don't think we > should hold up 8.4). So we should keep all the work out of the tree until every part is done? No tha

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Chris Browne
[EMAIL PROTECTED] (Tom Lane) writes: > As I said originally, we have no expectation that the proposed features > will displace the existing replication projects for "high end" > replication problems ... and I'd characterize all of Robert's concerns > as "high end" problems. We are happy to let tho

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Joshua D. Drake
On Thu, 2008-05-29 at 19:02 -0400, Tom Lane wrote: > "Joshua D. Drake" <[EMAIL PROTECTED]> writes: > > One customer does not make a hundred. I am not saying that the shipping > > isn't valid, just that those that I talk to are more interested in the > > read only slave. Consider that we have any

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Tom Lane
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > One customer does not make a hundred. I am not saying that the shipping > isn't valid, just that those that I talk to are more interested in the > read only slave. Consider that we have any number of ways to solve the > problem we are considering impl

  1   2   >