Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-07-15 Thread Bruce Momjian
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Agreed. I realize why we are not zeroing those bytes (for performance), but can't we have the archiver zero those bytes before calling the 'archive_command'? The archiver doesn't know any more about where the end-of-data is than

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-07-15 Thread Alvaro Herrera
Bruce Momjian wrote: Added to TODO: o Reduce PITR WAL file size by removing full page writes and by removing trailing bytes to improve compression If we remove full page writes, how does hint bit setting get propagated to the slave? -- Alvaro Herrera

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-07-15 Thread Bruce Momjian
Alvaro Herrera wrote: Bruce Momjian wrote: Added to TODO: o Reduce PITR WAL file size by removing full page writes and by removing trailing bytes to improve compression If we remove full page writes, how does hint bit setting get propagated to the slave? We would

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-07-15 Thread Alvaro Herrera
Bruce Momjian wrote: Alvaro Herrera wrote: Bruce Momjian wrote: Added to TODO: o Reduce PITR WAL file size by removing full page writes and by removing trailing bytes to improve compression If we remove full page writes, how does hint bit setting get

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-07-15 Thread Bruce Momjian
Alvaro Herrera wrote: Bruce Momjian wrote: Alvaro Herrera wrote: Bruce Momjian wrote: Added to TODO: o Reduce PITR WAL file size by removing full page writes and by removing trailing bytes to improve compression If we remove full page writes, how

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-16 Thread Kevin Grittner
On Mon, Jun 9, 2008 at 9:48 PM, in message [EMAIL PROTECTED], Greg Smith [EMAIL PROTECTED] wrote: On Mon, 9 Jun 2008, Tom Lane wrote: It should also be pointed out that the whole thing becomes uninteresting if we get real-time log shipping implemented. So I see absolutely no point in

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-10 Thread Gregory Stark
Greg Smith [EMAIL PROTECTED] writes: On Mon, 9 Jun 2008, Tom Lane wrote: It should also be pointed out that the whole thing becomes uninteresting if we get real-time log shipping implemented. So I see absolutely no point in spending time integrating pg_clearxlogtail now. There are remote

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-10 Thread Heikki Linnakangas
Gregory Stark wrote: Instead of zeroing bytes and depending on compression why not just pass an extra parameter to the archive command with the offset to the logical end of data. Because the archiver process doesn't have that information. -- Heikki Linnakangas EnterpriseDB

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-10 Thread Josh Berkus
All, For the slave to not interfere with the master at all, we would need to delay application of WAL files on each slave until visibility on that slave allows the WAL to be applied, but in that case we would have long-running transactions delay data visibility of all slave sessions.

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-10 Thread ITAGAKI Takahiro
Josh Berkus [EMAIL PROTECTED] wrote: I still see having 2 different settings: Synchronous: XID visibility is pushed to the master. Maintains synchronous failover, and users are expected to run *1* master to *1* slave for most installations. Asynchronous: replication stops on the

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-09 Thread Bruce Momjian
Gurjeet Singh wrote: On Fri, May 30, 2008 at 10:40 AM, Tom Lane [EMAIL PROTECTED] wrote: But since you mention it: one of the plausible answers for fixing the vacuum problem for read-only slaves is to have the slaves push an xmin back upstream to the master to prevent premature vacuuming.

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-09 Thread Bruce Momjian
Andreas 'ads' Scherbaum wrote: On Fri, 30 May 2008 16:22:41 -0400 (EDT) Greg Smith wrote: On Fri, 30 May 2008, Andreas 'ads' Scherbaum wrote: Then you ship 16 MB binary stuff every 30 second or every minute but you only have some kbyte real data in the logfile. Not if you use

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-09 Thread Alvaro Herrera
Bruce Momjian wrote: Agreed. I realize why we are not zeroing those bytes (for performance), but can't we have the archiver zero those bytes before calling the 'archive_command'? Perhaps make the zeroing user-settable. -- Alvaro Herrera

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-09 Thread Tom Lane
Bruce Momjian [EMAIL PROTECTED] writes: Gurjeet Singh wrote: There could be multiple slaves following a master, some serving For the slave to not interfere with the master at all, we would need to delay application of WAL files on each slave until visibility on that slave allows the WAL to

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-09 Thread Tom Lane
Bruce Momjian [EMAIL PROTECTED] writes: Agreed. I realize why we are not zeroing those bytes (for performance), but can't we have the archiver zero those bytes before calling the 'archive_command'? The archiver doesn't know any more about where the end-of-data is than the archive_command

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-09 Thread Greg Smith
On Mon, 9 Jun 2008, Tom Lane wrote: It should also be pointed out that the whole thing becomes uninteresting if we get real-time log shipping implemented. So I see absolutely no point in spending time integrating pg_clearxlogtail now. There are remote replication scenarios over a WAN (mainly

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-09 Thread Koichi Suzuki
Just for information. In terms of archive compression, I have archive log compression which will be found in http://pgfoundry.org/projects/pglesslog/ This feature is also included in NTT's synchronized log shipping replication presented in the last PGCon. 2008/6/10 Greg Smith [EMAIL PROTECTED]:

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-05 Thread Gregory Stark
Jeff Davis [EMAIL PROTECTED] writes: On Wed, 2008-06-04 at 14:17 +0300, Heikki Linnakangas wrote: Would that also cover possible differences in page size, 32bit OS vs. 64bit OS, different timestamp flavour, etc. issues ? AFAIR, all these things can have an influence on how the data is

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Markus Schiltknecht
Hello Andrew, Andrew Sullivan wrote: Yes. And silent as ever. :-) Are the slides of your PgCon talk available for download somewhere? BTW: up until recently, there was yet another mailing list: [EMAIL PROTECTED] It was less focused on hooks and got at least some traffic. :-) Are those

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Heikki Linnakangas
Stephen Denne wrote: Hannu Krosing wrote: The simplest form of synchronous wal shipping would not even need postgresql running on slave, just a small daemon which reports when wal blocks are a) received and b) synced to disk. While that does sound simple, I'd presume that most people would

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Teodor Sigaev
Hmm, WAL version compatibility is an interesting question. Most minor releases hasn't changed the WAL format, and it would be nice to allow As I remember, high minor version should read all WALs from lowers, but it isn't true for opposite case and between different major versions. running

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Csaba Nagy
On Wed, 2008-06-04 at 11:13 +0300, Heikki Linnakangas wrote: Hmm, WAL version compatibility is an interesting question. Most minor releases hasn't changed the WAL format, and it would be nice to allow running different minor versions in the master and slave in those cases. But it's

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Heikki Linnakangas
Csaba Nagy wrote: On Wed, 2008-06-04 at 11:13 +0300, Heikki Linnakangas wrote: Hmm, WAL version compatibility is an interesting question. Most minor releases hasn't changed the WAL format, and it would be nice to allow running different minor versions in the master and slave in those cases.

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Heikki Linnakangas
Teodor Sigaev wrote: Is it possible to use catalog version number as WAL version? No, because we don't change the catalog version number in minor releases, even though we might change WAL format. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Andrew Sullivan
On Wed, Jun 04, 2008 at 09:24:20AM +0200, Markus Schiltknecht wrote: Are the slides of your PgCon talk available for download somewhere? There weren't any slides, really (there were 4 that I put up in case the cases I was discussing needed back-references, but they didn't). Joshua tells me that

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Tom Lane
Heikki Linnakangas [EMAIL PROTECTED] writes: Hmm, WAL version compatibility is an interesting question. Most minor releases hasn't changed the WAL format, and it would be nice to allow running different minor versions in the master and slave in those cases. But it's certainly not unheard of

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Hannu Krosing
On Wed, 2008-06-04 at 10:40 -0400, Tom Lane wrote: Heikki Linnakangas [EMAIL PROTECTED] writes: Hmm, WAL version compatibility is an interesting question. Most minor releases hasn't changed the WAL format, and it would be nice to allow running different minor versions in the master and

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Tom Lane
Hannu Krosing [EMAIL PROTECTED] writes: On Wed, 2008-06-04 at 10:40 -0400, Tom Lane wrote: Heikki Linnakangas [EMAIL PROTECTED] writes: Hmm, WAL version compatibility is an interesting question. Most minor releases hasn't changed the WAL format, and it would be nice to allow running

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Simon Riggs
On Wed, 2008-06-04 at 11:37 -0400, Tom Lane wrote: This thread is getting out of hand, actually. Agreed. We should start new threads for specific things. Please. However, since by definition pg_control doesn't change in a minor upgrade, there isn't any easy way to enforce a rule like slaves

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Jeff Davis
On Wed, 2008-06-04 at 14:17 +0300, Heikki Linnakangas wrote: Would that also cover possible differences in page size, 32bit OS vs. 64bit OS, different timestamp flavour, etc. issues ? AFAIR, all these things can have an influence on how the data is written and possibly make the WAL

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Tom Lane
Jeff Davis [EMAIL PROTECTED] writes: On Wed, 2008-06-04 at 14:17 +0300, Heikki Linnakangas wrote: These are already covered by the information in pg_control. Another thing that can change between systems is the collation behavior, which can corrupt indexes (and other bad things). That is

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Jeff Davis
On Wed, 2008-06-04 at 14:23 -0400, Tom Lane wrote: That is covered by pg_control, at least to the extent of forcing the same value of LC_COLLATE. But the same LC_COLLATE means different things on different systems. Even en_US means something different on Mac versus Linux. Regards, Jeff

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Koichi Suzuki
Well, WAL format doesn't only depend on WAL itself, but also depend on each resource manager. If we introduce WAL format version identification, ISTM that we have to take care of the matching of resource manager in the master and the slave as well. 2008/6/4 Heikki Linnakangas [EMAIL PROTECTED]:

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Tom Lane
Koichi Suzuki [EMAIL PROTECTED] writes: Well, WAL format doesn't only depend on WAL itself, but also depend on each resource manager. If we introduce WAL format version identification, ISTM that we have to take care of the matching of resource manager in the master and the slave as well.

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-04 Thread Koichi Suzuki
If the version of the master and the slave is different and we'd still like to allow log shipping replication, we need a negotiation if WAL format for the two is compatible. I hope it is not in our scope and I'm worrying too much. 2008/6/5 Tom Lane [EMAIL PROTECTED]: Koichi Suzuki [EMAIL

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-03 Thread Hannu Krosing
On Mon, 2008-06-02 at 22:40 +0200, Andreas 'ads' Scherbaum wrote: On Mon, 02 Jun 2008 11:52:05 -0400 Chris Browne wrote: [EMAIL PROTECTED] (Andreas 'ads' Scherbaum) writes: On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: Well, yes, but you do know about archive_timeout,

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-03 Thread Andrew Sullivan
On Sun, Jun 01, 2008 at 01:43:22PM -0400, Tom Lane wrote: power to him. (Is the replica-hooks-discuss list still working?) But Yes. And silent as ever. :-) A -- Andrew Sullivan [EMAIL PROTECTED] +1 503 667 4564 x104 http://www.commandprompt.com/ -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-03 Thread Stephen Denne
Hannu Krosing wrote: The simplest form of synchronous wal shipping would not even need postgresql running on slave, just a small daemon which reports when wal blocks are a) received and b) synced to disk. While that does sound simple, I'd presume that most people would want the guarantee

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-02 Thread Robert Hodges
Hi Hannu, Hi Hannu, On 6/1/08 2:14 PM, Hannu Krosing [EMAIL PROTECTED] wrote: As a consequence, I don¹t see how you can get around doing some sort of row-based replication like all the other databases. Is'nt WAL-base replication some sort of row-based replication ? Yes, in theory.

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-02 Thread Chris Browne
[EMAIL PROTECTED] (Andreas 'ads' Scherbaum) writes: On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: Well, yes, but you do know about archive_timeout, right? No need to wait 2 hours. Then you ship 16 MB binary stuff every 30 second or every minute but you only have some kbyte real

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-02 Thread Hannu Krosing
On Thu, 2008-05-29 at 23:37 +0200, Mathias Brossard wrote: I pointed out that the NTT solution is synchronous because Tom said in the first part of his email that: In practice, simple asynchronous single-master-multiple-slave replication covers a respectable fraction of use cases, so

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-02 Thread Andreas 'ads' Scherbaum
On Mon, 02 Jun 2008 11:52:05 -0400 Chris Browne wrote: [EMAIL PROTECTED] (Andreas 'ads' Scherbaum) writes: On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: Well, yes, but you do know about archive_timeout, right? No need to wait 2 hours. Then you ship 16 MB binary stuff

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-01 Thread Robert Hodges
Hi Merlin, My point here is that with reasonably small extensions to the core you can build products that are a lot better than SLONY. Triggers do not cover DDL, among other issues, and it's debatable whether they are the best way to implement quorum policies like Google's semi-synchronous

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-01 Thread Merlin Moncure
On Sun, Jun 1, 2008 at 11:58 AM, Robert Hodges [EMAIL PROTECTED] wrote: Hi Merlin, My point here is that with reasonably small extensions to the core you can build products that are a lot better than SLONY. Triggers do not cover DDL, among other issues, and it's debatable whether they are

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-01 Thread Tom Lane
Merlin Moncure [EMAIL PROTECTED] writes: On Sun, Jun 1, 2008 at 11:58 AM, Robert Hodges [EMAIL PROTECTED] wrote: My point here is that with reasonably small extensions to the core you can build products that are a lot better than SLONY. These issues are much discussed and well understood.

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-01 Thread Dawid Kuroczko
On Thu, May 29, 2008 at 4:12 PM, Tom Lane [EMAIL PROTECTED] wrote: The Postgres core team met at PGCon to discuss a few issues, the largest of which is the need for simple, built-in replication for PostgreSQL. [...] We believe that the most appropriate base technology for this is 1 probably

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-01 Thread James Mansion
David Fetter wrote: This part is a deal-killer. It's a giant up-hill slog to sell warm standby to those in charge of making resources available because the warm standby machine consumes SA time, bandwidth, power, rack space, etc., but provides no tangible benefit, and this feature would have

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-01 Thread James Mansion
Aidan Van Dyk wrote: The whole single-threaded WAL replay problem is going to rear it's ugly head here too, and mean that a slave *won't* be able to keep up with a busy master if it's actually trying to apply all the changes in real-time. Is there a reason to commit at the same points that the

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-01 Thread Hannu Krosing
On Thu, 2008-05-29 at 13:37 -0400, Tom Lane wrote: David Fetter [EMAIL PROTECTED] writes: On Thu, May 29, 2008 at 08:46:22AM -0700, Joshua D. Drake wrote: The only question I have is... what does this give us that PITR doesn't give us? It looks like a wrapper for PITR to me, so the gain

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-01 Thread Hannu Krosing
On Fri, 2008-05-30 at 15:16 -0400, Robert Treat wrote: On Friday 30 May 2008 01:10:20 Tom Lane wrote: Greg Smith [EMAIL PROTECTED] writes: I fully accept that it may be the case that it doesn't make technical sense to tackle them in any order besides sync-read-only slaves because of

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-06-01 Thread Hannu Krosing
On Thu, 2008-05-29 at 12:05 -0700, Robert Hodges wrote: Hi everyone, First of all, I’m absolutely delighted that the PG community is thinking seriously about replication. Second, having a solid, easy-to-use database availability solution that works more or less out of the box would be

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-31 Thread Dimitri Fontaine
Le vendredi 30 mai 2008, Dimitri Fontaine a écrit : This way, no need to switch IP addresses, the clients just connect as usual and get results back and do not have to know whether the host they're qerying against is a slave or a master. This level of smartness is into -core. Oh, and if you

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-31 Thread Mike Rylander
On Fri, May 30, 2008 at 6:47 PM, Andreas 'ads' Scherbaum [EMAIL PROTECTED] wrote: On Fri, 30 May 2008 17:05:57 -0400 Andrew Dunstan wrote: Andreas 'ads' Scherbaum wrote: On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: Well, yes, but you do know about archive_timeout, right? No

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-31 Thread Merlin Moncure
On Sat, May 31, 2008 at 2:18 AM, Mike Rylander [EMAIL PROTECTED] wrote: On Fri, May 30, 2008 at 6:47 PM, Andreas 'ads' Scherbaum [EMAIL PROTECTED] wrote: On Fri, 30 May 2008 17:05:57 -0400 Andrew Dunstan wrote: Andreas 'ads' Scherbaum wrote: On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-31 Thread Joshua D. Drake
Merlin Moncure wrote: On Sat, May 31, 2008 at 2:18 AM, Mike Rylander [EMAIL PROTECTED] wrote: On Fri, May 30, 2008 at 6:47 PM, Andreas 'ads' Scherbaum Compression especially is going to negate one of the big advantages of wal shipping, namely that it is cheap investment in terms of load to

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Gurjeet Singh
On Fri, May 30, 2008 at 10:40 AM, Tom Lane [EMAIL PROTECTED] wrote: But since you mention it: one of the plausible answers for fixing the vacuum problem for read-only slaves is to have the slaves push an xmin back upstream to the master to prevent premature vacuuming. The current design of

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Andrew Sullivan
On Thu, May 29, 2008 at 01:58:34PM -0700, David Fetter wrote: If people on core had come to the idea that we needed to build in replication *before* 8.3 came out, they certainly didn't announce it. Now is a great time to mention this because it gives everybody time to: 1. Come to a

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Marko Kreen
On 5/30/08, Gurjeet Singh [EMAIL PROTECTED] wrote: On Fri, May 30, 2008 at 10:40 AM, Tom Lane [EMAIL PROTECTED] wrote: But since you mention it: one of the plausible answers for fixing the vacuum problem for read-only slaves is to have the slaves push an xmin back upstream to the master to

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Andrew Dunstan
Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Fri, 2008-05-30 at 12:31 +0530, Gurjeet Singh wrote: On Fri, May 30, 2008 at 10:40 AM, Tom Lane [EMAIL PROTECTED] wrote: But since you mention it: one of the plausible answers for fixing the vacuum problem for read-only

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: On Fri, 2008-05-30 at 12:31 +0530, Gurjeet Singh wrote: On Fri, May 30, 2008 at 10:40 AM, Tom Lane [EMAIL PROTECTED] wrote: But since you mention it: one of the plausible answers for fixing the vacuum problem for read-only slaves is to have the slaves push

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Robert Hodges
Hi Tom, Thanks for the reasoned reply. As you saw from point #2 in my comments, I think you should do this feature. I hope this answers Josh Berkus' concern about my comments. You make a very interesting comment which seems to go to the heart of this design approach: About the only thing

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Merlin Moncure
On Fri, May 30, 2008 at 9:31 AM, Marko Kreen [EMAIL PROTECTED] wrote: On 5/30/08, Gurjeet Singh [EMAIL PROTECTED] wrote: I think it would be best to not make the slave interfere with the master's operations; that's only going to increase the operational complexity of such a solution. I

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Simon Riggs
On Fri, 2008-05-30 at 11:30 -0400, Andrew Dunstan wrote: Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Fri, 2008-05-30 at 12:31 +0530, Gurjeet Singh wrote: On Fri, May 30, 2008 at 10:40 AM, Tom Lane [EMAIL PROTECTED] wrote: But since you mention it: one

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Simon Riggs
On Fri, 2008-05-30 at 12:31 +0530, Gurjeet Singh wrote: On Fri, May 30, 2008 at 10:40 AM, Tom Lane [EMAIL PROTECTED] wrote: But since you mention it: one of the plausible answers for fixing the vacuum problem for read-only slaves is to have the slaves push an

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Andreas 'ads' Scherbaum
On Thu, 29 May 2008 09:22:26 -0700 Steve Atkins wrote: On May 29, 2008, at 9:12 AM, David Fetter wrote: Either one of these would be great, but something that involves machines that stay useless most of the time is just not going to work. I have customers who are thinking about warm

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Andreas 'ads' Scherbaum
On Thu, 29 May 2008 18:29:01 -0400 Tom Lane wrote: Dimitri Fontaine [EMAIL PROTECTED] writes: While at it, would it be possible for the simple part of the core team statement to include automatic failover? No, I think it would be a useless expenditure of energy. Failover includes a

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Andreas 'ads' Scherbaum
On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: Well, yes, but you do know about archive_timeout, right? No need to wait 2 hours. Then you ship 16 MB binary stuff every 30 second or every minute but you only have some kbyte real data in the logfile. This must be taken into account,

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Greg Smith
On Fri, 30 May 2008, Andreas 'ads' Scherbaum wrote: Then you ship 16 MB binary stuff every 30 second or every minute but you only have some kbyte real data in the logfile. Not if you use pg_clearxlogtail ( http://www.2ndquadrant.com/replication.htm ), which got lost in the giant March

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Gurjeet Singh
On Thu, May 29, 2008 at 7:42 PM, Tom Lane [EMAIL PROTECTED] wrote: The big problem is that long-running slave-side queries might still need tuples that are vacuumable on the master, and so replication of vacuuming actions would cause the slave's queries to deliver wrong answers. Another

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Simon Riggs
On Fri, 2008-05-30 at 11:12 -0700, Robert Hodges wrote: This is clearly an important use case but it also seems clear that the WAL approach is not a general-purpose approach to replication. I think we cannot make such a statement yet, if ever. I would note that log-based replication is now

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Robert Treat
On Thursday 29 May 2008 22:59:21 Merlin Moncure wrote: On Thu, May 29, 2008 at 9:26 PM, Josh Berkus [EMAIL PROTECTED] wrote: I fully accept that it may be the case that it doesn't make technical sense to tackle them in any order besides sync-read-only slaves because of dependencies in the

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Andrew Dunstan
Andreas 'ads' Scherbaum wrote: On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: Well, yes, but you do know about archive_timeout, right? No need to wait 2 hours. Then you ship 16 MB binary stuff every 30 second or every minute but you only have some kbyte real data in the

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Robert Treat
On Thursday 29 May 2008 20:31:31 Greg Smith wrote: On Thu, 29 May 2008, Tom Lane wrote: There's no point in having read-only slave queries if you don't have a trustworthy method of getting the data to them. This is a key statement that highlights the difference in how you're thinking about

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Robert Treat
On Friday 30 May 2008 01:10:20 Tom Lane wrote: Greg Smith [EMAIL PROTECTED] writes: I fully accept that it may be the case that it doesn't make technical sense to tackle them in any order besides sync-read-only slaves because of dependencies in the implementation between the two. Well,

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Gurjeet Singh
On Sat, May 31, 2008 at 1:52 AM, Greg Smith [EMAIL PROTECTED] wrote: On Fri, 30 May 2008, Andreas 'ads' Scherbaum wrote: Then you ship 16 MB binary stuff every 30 second or every minute but you only have some kbyte real data in the logfile. Not if you use pg_clearxlogtail (

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Tatsuo Ishii
Andreas 'ads' Scherbaum wrote: On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: Well, yes, but you do know about archive_timeout, right? No need to wait 2 hours. Then you ship 16 MB binary stuff every 30 second or every minute but you only have some kbyte real

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Joshua D. Drake
On Sat, 2008-05-31 at 02:48 +0530, Gurjeet Singh wrote: Not if you use pg_clearxlogtail ( http://www.2ndquadrant.com/replication.htm ), which got lost in the giant March commitfest queue but should probably wander into contrib as part of 8.4.

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Greg Smith
On Sat, 31 May 2008, Gurjeet Singh wrote: Not if you use pg_clearxlogtail This means we need to modify pg_standby to not check for filesize when reading XLogs. No, the idea is that you run the segments through pg_clearxlogtail | gzip, which then compresses lightly used segments massively

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Simon Riggs
On Fri, 2008-05-30 at 01:10 -0400, Tom Lane wrote: Greg Smith [EMAIL PROTECTED] writes: I fully accept that it may be the case that it doesn't make technical sense to tackle them in any order besides sync-read-only slaves because of dependencies in the implementation between the two.

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Dimitri Fontaine
Le vendredi 30 mai 2008, Tom Lane a écrit : No, I think it would be a useless expenditure of energy. Failover includes a lot of things that are not within our purview: switching IP addresses to point to the new server, some kind of STONITH solution to keep the original master from coming back

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Andreas 'ads' Scherbaum
On Fri, 30 May 2008 16:22:41 -0400 (EDT) Greg Smith wrote: On Fri, 30 May 2008, Andreas 'ads' Scherbaum wrote: Then you ship 16 MB binary stuff every 30 second or every minute but you only have some kbyte real data in the logfile. Not if you use pg_clearxlogtail (

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Simon Riggs
On Thu, 2008-05-29 at 10:12 -0400, Tom Lane wrote: The Postgres core team met at PGCon to discuss a few issues, the largest of which is the need for simple, built-in replication for PostgreSQL. Historically the project policy has been to avoid putting replication into core PostgreSQL, so as to

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Josh Berkus
Andrew, Sure there's a price to pay. But that doesn't mean the facility doesn't exist. And I rather suspect that most of Josh's customers aren't too concerned about traffic charges or affected by such bandwidth restrictions. Certainly, none of my clients are, and they aren't in the giant

Fwd: Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Csaba Nagy
[Looks like this mail missed the hackers list on reply to all, I wonder how it could happen... so I forward it] On Thu, 2008-05-29 at 17:00 +0100, Dave Page wrote: Yes, we're talking real-time streaming (synchronous) log shipping. Is there any design already how would this be implemented ?

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Andreas 'ads' Scherbaum
On Fri, 30 May 2008 17:05:57 -0400 Andrew Dunstan wrote: Andreas 'ads' Scherbaum wrote: On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: Well, yes, but you do know about archive_timeout, right? No need to wait 2 hours. Then you ship 16 MB binary stuff every 30 second or every

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Andrew Dunstan
Tatsuo Ishii wrote: Andreas 'ads' Scherbaum wrote: On Thu, 29 May 2008 23:02:56 -0400 Andrew Dunstan wrote: Well, yes, but you do know about archive_timeout, right? No need to wait 2 hours. Then you ship 16 MB binary stuff every 30 second or every minute but

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-30 Thread Gurjeet Singh
On Sat, May 31, 2008 at 3:41 AM, Greg Smith [EMAIL PROTECTED] wrote: On Sat, 31 May 2008, Gurjeet Singh wrote: Not if you use pg_clearxlogtail This means we need to modify pg_standby to not check for filesize when reading XLogs. No, the idea is that you run the segments through

[HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Tom Lane
The Postgres core team met at PGCon to discuss a few issues, the largest of which is the need for simple, built-in replication for PostgreSQL. Historically the project policy has been to avoid putting replication into core PostgreSQL, so as to leave room for development of competing solutions,

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Marko Kreen
On 5/29/08, Tom Lane [EMAIL PROTECTED] wrote: The Postgres core team met at PGCon to discuss a few issues, the largest of which is the need for simple, built-in replication for PostgreSQL. Historically the project policy has been to avoid putting replication into core PostgreSQL, so as to

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread David Fetter
On Thu, May 29, 2008 at 10:12:55AM -0400, Tom Lane wrote: The Postgres core team met at PGCon to discuss a few issues, the largest of which is the need for simple, built-in replication for PostgreSQL. Historically the project policy has been to avoid putting replication into core PostgreSQL,

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Joshua D. Drake
On Thu, 2008-05-29 at 08:21 -0700, David Fetter wrote: On Thu, May 29, 2008 at 10:12:55AM -0400, Tom Lane wrote: This part is a deal-killer. It's a giant up-hill slog to sell warm standby to those in charge of making resources available because the warm standby machine consumes SA time,

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Josh Berkus
Marko, But Tom's mail gave me impression core wants to wait until we get perfect read-only slave implementation so we wait with it until 8.6, which does not seem sensible. If we can do slightly inefficient (but simple) implementation right now, I see no reason to reject it, we can always

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread David Fetter
On Thu, May 29, 2008 at 08:46:22AM -0700, Joshua D. Drake wrote: On Thu, 2008-05-29 at 08:21 -0700, David Fetter wrote: This part is a deal-killer. It's a giant up-hill slog to sell warm standby to those in charge of making resources available because the warm standby machine consumes SA

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Douglas McNaught
On Thu, May 29, 2008 at 11:46 AM, Joshua D. Drake [EMAIL PROTECTED] wrote: The only question I have is... what does this give us that PITR doesn't give us? I think the idea is that WAL records would be shipped (possibly via socket) and applied as they're generated, rather than on a

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Bruce Momjian
Josh Berkus wrote: Marko, But Tom's mail gave me impression core wants to wait until we get perfect read-only slave implementation so we wait with it until 8.6, which does not seem sensible. If we can do slightly inefficient (but simple) implementation right now, I see no reason to

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Dave Page
On Thu, May 29, 2008 at 4:48 PM, Douglas McNaught [EMAIL PROTECTED] wrote: On Thu, May 29, 2008 at 11:46 AM, Joshua D. Drake [EMAIL PROTECTED] wrote: The only question I have is... what does this give us that PITR doesn't give us? I think the idea is that WAL records would be shipped

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Rick Vernam
On Thursday 29 May 2008 09:54:03 am Marko Kreen wrote: On 5/29/08, Tom Lane [EMAIL PROTECTED] wrote: The Postgres core team met at PGCon to discuss a few issues, the largest  of which is the need for simple, built-in replication for PostgreSQL.  Historically the project policy has been to

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Marko Kreen
On 5/29/08, David Fetter [EMAIL PROTECTED] wrote: On Thu, May 29, 2008 at 10:12:55AM -0400, Tom Lane wrote: Ideally this would be coupled with the ability to execute read-only queries on the slave servers, but we see technical difficulties that might prevent that from being completed

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Aidan Van Dyk
* Josh Berkus [EMAIL PROTECTED] [080529 11:52]: Marko, But Tom's mail gave me impression core wants to wait until we get perfect read-only slave implementation so we wait with it until 8.6, which does not seem sensible. If we can do slightly inefficient (but simple) implementation right

Re: [HACKERS] Core team statement on replication in PostgreSQL

2008-05-29 Thread Josh Berkus
Joshua D. Drake wrote: On Thu, 2008-05-29 at 08:21 -0700, David Fetter wrote: On Thu, May 29, 2008 at 10:12:55AM -0400, Tom Lane wrote: This part is a deal-killer. It's a giant up-hill slog to sell warm standby to those in charge of making resources available because the warm standby

  1   2   >