Re: [HACKERS] integration of pgcluster into postgresql
On Sun, Aug 27, 2006 at 12:47:12PM -0400, Tom Lane wrote: see some kind of joint proposal by multiple replication projects about what hooks to add. Anybody out there want to organize such a thing? We were attempting to define such a set of hooks as part of the Slony-II work, but that sort of fell off the rails. I am very strongly in favour of such a framework, though, and would love to see it. I am willing to do the co-ordination and project management slog work on this if those who need the hooks are willing to work on a set of common definitions. If anyone would like that, please let me know. If people want to contact me off-list, that's also fine; I'll summarise. A -- Andrew Sullivan | [EMAIL PROTECTED] Everything that happens in the world happens at some place. --Jane Jacobs ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] integration of pgcluster into postgresql
Yes, I forgot to include hackers on that mail. Anyway, relax Jim, I'm not trying to invade anyone's turf here. There seems to be support for the idea of providing an interface plug for replication modules, which is fine with me. If you have any constructive criticism towards that, I'd be most happy to consider it and try to find an accomodation. Best regards --- Jim Nasby [EMAIL PROTECTED] wrote: Adding -hackers back in... -Original Message- From: Chahine Hamila [mailto:[EMAIL PROTECTED] Sent: Fri 8/25/2006 8:36 PM To: Jim Nasby Subject: Re: [HACKERS] integration of pgcluster into postgresql First, you need to review all the past discussion about the very intentional decision not to build any replication into the core database. I would gladly do so. Can you send me any pointer? I don't really have any handy, but try searching the hackers archive for 'replication'. Second, pgcluster is (AFAIK) command-based replication, which has some very, very serious drawbacks. If PostgreSQL were to include a replication solution, I'd certainly hope it wouldn't be command-based. It's better than no replication at all... It's good enough for many uses. As is Slony. And dbmirror. And pgpool. So where do we draw the line? Should we include all four? ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] integration of pgcluster into postgresql
On Tue, Aug 29, 2006 at 07:19:09AM -0700, Chahine Hamila wrote: Yes, I forgot to include hackers on that mail. Anyway, relax Jim, I'm not trying to invade anyone's turf here. There seems to be support for the idea of providing an interface plug for replication modules, which is fine with me. If you have any constructive criticism towards that, I'd be most happy to consider it and try to find an accomodation. Well, the big challenge there is that each replication system uses a different methodology, so you're unlikely to come up with anything that would be common between any two systems. I think the best bet is to look for things that can be added that are either difficult or impossible to do outside the backend, or that have use beyond just replication. -- Jim C. Nasby, Sr. Engineering Consultant [EMAIL PROTECTED] Pervasive Software http://pervasive.comwork: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461 ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] integration of pgcluster into postgresql
Adding -hackers back in... -Original Message- From: Chahine Hamila [mailto:[EMAIL PROTECTED] Sent: Fri 8/25/2006 8:36 PM To: Jim Nasby Subject: Re: [HACKERS] integration of pgcluster into postgresql First, you need to review all the past discussion about the very intentional decision not to build any replication into the core database. I would gladly do so. Can you send me any pointer? I don't really have any handy, but try searching the hackers archive for 'replication'. Second, pgcluster is (AFAIK) command-based replication, which has some very, very serious drawbacks. If PostgreSQL were to include a replication solution, I'd certainly hope it wouldn't be command-based. It's better than no replication at all... It's good enough for many uses. As is Slony. And dbmirror. And pgpool. So where do we draw the line? Should we include all four? ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] integration of pgcluster into postgresql
Tom Lane [EMAIL PROTECTED] writes: That said, my company would feel more confortable with the idea that it's part of the postgresql mainstream distro for many obvious reasons - or we might drop postgresql altogether - which is why I'm proposing myself to do the necessary work to integrate it in postgresql if there's interest. The core development team has only a very finite number of cycles available. Would you rather we spend our time on fixing pgcluster than on fixing the core Postgres database? I'm beginning to wonder whether it would be better from a PR perspective to rename pgfoundry to something like modules.postgresql.org. While modules isn't necessarily technically right in postgresql vocabulary it's right in the more general sense And it doesn't imply the pieces of code are still in progress like projects.postgresql.org might and doesn't give the impression that they're living on their own without support from other postgres people like having a separate domain does. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] integration of pgcluster into postgresql
On 8/27/06, Gregory Stark [EMAIL PROTECTED] wrote: I'm beginning to wonder whether it would be better from a PR perspective to rename pgfoundry to something like modules.postgresql.org. While modules isn't necessarily technically right in postgresql vocabulary it's right in the more general sense And it doesn't imply the pieces of code are still in progress like projects.postgresql.org might and doesn't give the impression that they're living on their own without support from other postgres people like having a separate domain does. I don't know what the name should be but we should at least be consistent between pgFoundry and websites hosted on pgFoundry. Currently we have http://pgcluster.projects.postgresql.org/ for the website hosted on pgFoundry and http://pgfoundry.org/projects/pgcluster for the project itself (yes, Jonah, they are both pgFoundry stuff, it's just that the website is probably not maintained by pgcluster staff currently). I agree with Gregory that renaming pgfoundry.org to [whatever].postgresql.org could be a good idea to make it more official. And project sites should keep their projectname.[whatever].postgresql.org address. -- Guillaume ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] integration of pgcluster into postgresql
Tom Lane wrote: My take on all this is that there's no one-size-fits-all replication solution, and therefore the right approach is to have multiple active subprojects. Anybody knowing a little about the world of replication needs will agree with you here. Unfortunately, AFAICS pgcluster can't be added as module as e.g. Slony-I, since it's rather a not-so-small patch to the pgsql sources. So I wonder if it's possible to provide some not-too-intrusive hooks in core pgsql, enabling pgcluster to do most of the work in modules, to have the best of both worlds: core with as few modifications as possible, and modules extending the operation, profiting from backend development immediately. Regards, Andreas ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] integration of pgcluster into postgresql
Andreas Pflug [EMAIL PROTECTED] writes: Tom Lane wrote: My take on all this is that there's no one-size-fits-all replication solution, and therefore the right approach is to have multiple active subprojects. Anybody knowing a little about the world of replication needs will agree with you here. Unfortunately, AFAICS pgcluster can't be added as module as e.g. Slony-I, since it's rather a not-so-small patch to the pgsql sources. So I wonder if it's possible to provide some not-too-intrusive hooks in core pgsql, enabling pgcluster to do most of the work in modules, to have the best of both worlds: core with as few modifications as possible, and modules extending the operation, profiting from backend development immediately. I don't have any objection in principle to adding hooks that're needed by replication projects. But again, I don't want the core project to be seen as favoring some replication projects over others. So I'd want to see some kind of joint proposal by multiple replication projects about what hooks to add. Anybody out there want to organize such a thing? regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] integration of pgcluster into postgresql
Tom Lane wrote: So I'd want to see some kind of joint proposal by multiple replication projects about what hooks to add. Anybody out there want to organize such a thing? Well, at least the pgcluster group could come up with a rough list of required hooks, and then the other groups can judge whether that list can be shaped into something universally useful or whether it's completely useless to them. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] integration of pgcluster into postgresql
Peter Eisentraut wrote: Tom Lane wrote: So I'd want to see some kind of joint proposal by multiple replication projects about what hooks to add. Anybody out there want to organize such a thing? Well, at least the pgcluster group could come up with a rough list of required hooks, and then the other groups can judge whether that list can be shaped into something universally useful or whether it's completely useless to them. ... or the pgcluster group could check the hook list posted by the GORDA project guys. In fact IIRC that patch was committed already, without much discussion? -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] integration of pgcluster into postgresql
On 8/27/06, Alvaro Herrera [EMAIL PROTECTED] wrote: ... or the pgcluster group could check the hook list posted by the GORDA project guys. In fact IIRC that patch was committed already, without much discussion? I thought the GORDA patch got turned down because there was no communication between replication providers. Although, I do like the trigger hooks GORDA provides. -- Jonah H. Harris, Software Architect | phone: 732.331.1300 EnterpriseDB Corporation| fax: 732.331.1301 33 Wood Ave S, 2nd Floor| [EMAIL PROTECTED] Iselin, New Jersey 08830| http://www.enterprisedb.com/ ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] integration of pgcluster into postgresql
Jonah H. Harris [EMAIL PROTECTED] writes: On 8/27/06, Alvaro Herrera [EMAIL PROTECTED] wrote: ... or the pgcluster group could check the hook list posted by the GORDA project guys. In fact IIRC that patch was committed already, without much discussion? I thought the GORDA patch got turned down because there was no communication between replication providers. Exactly; we asked for some evidence that these particular hook definitions were generally useful. So it seems like a joint pgcluster/GORDA/Slony proposal would go over a lot better. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] integration of pgcluster into postgresql
The idea of hooks sounds quite good to me indeed. The issue is not PR, it's indeed pgcluster benefiting from the maintenance of postgresql and avoiding the hassle of having to resync its code at each postgresql change. I will propose something along those lines once I get a more stable pgcluster and have a better grasp at all details of its code. I could send a mail to the slony and gorda people at that point to see if they're interested in coordinating efforts. --- Tom Lane [EMAIL PROTECTED] wrote: Jonah H. Harris [EMAIL PROTECTED] writes: On 8/27/06, Alvaro Herrera [EMAIL PROTECTED] wrote: ... or the pgcluster group could check the hook list posted by the GORDA project guys. In fact IIRC that patch was committed already, without much discussion? I thought the GORDA patch got turned down because there was no communication between replication providers. Exactly; we asked for some evidence that these particular hook definitions were generally useful. So it seems like a joint pgcluster/GORDA/Slony proposal would go over a lot better. regards, tom lane __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] integration of pgcluster into postgresql
Second, pgcluster is (AFAIK) command-based replication, which has some very, very serious drawbacks. If PostgreSQL were to include a replication solution, I'd certainly hope it wouldn't be command-based. Support of PGCluster-I, which we're discussing here, is being dropped in favor of the shared-disk PGCluster-II which was demonstrated at the anniversary conference. IIRC, PGCluster-I does use command-based replication but is merged into the parser in such a way as to make it work quite well--unlike the man-in-the-middle approach taken by pgpool. Finally, pgcluster is very out-of-date. The last version uses 8.0.1 and was released on Mar. 7, 2005. If the author can't find the time to maintain it, I don't see why that burden should be put on the shoulders of this community. Umm, I don't know where you're looking Jim, but the last update was February 10, 2006 and it's for PostgreSQL 8.1.1. Frankly, it has had a very good track record of development and bug fixes... so let's not make assumptions on (very large PostgreSQL) projects we're unfamiliar with. -- Jonah H. Harris, Software Architect | phone: 732.331.1300 EnterpriseDB Corporation| fax: 732.331.1301 33 Wood Ave S, 2nd Floor| [EMAIL PROTECTED] Iselin, New Jersey 08830| http://www.enterprisedb.com/ ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] integration of pgcluster into postgresql
Jonah H. Harris wrote: Support of PGCluster-I, which we're discussing here, is being dropped in favor of the shared-disk PGCluster-II which was demonstrated at the anniversary conference. IIRC, PGCluster-I does use command-based replication but is merged into the parser in such a way as to make it work quite well--unlike the man-in-the-middle approach taken by pgpool. Didn't Atsushi Mitani say he wanted to continue PgCluster-I? As they serve quite different needs that would make sense. Regards Markus ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] integration of pgcluster into postgresql
On 8/26/06, Markus Schiltknecht [EMAIL PROTECTED] wrote: Didn't Atsushi Mitani say he wanted to continue PgCluster-I? As they serve quite different needs that would make sense. Hmm... I was pretty sure he said that he couldn't devote time to both projects. -- Jonah H. Harris, Software Architect | phone: 732.331.1300 EnterpriseDB Corporation| fax: 732.331.1301 33 Wood Ave S, 2nd Floor| [EMAIL PROTECTED] Iselin, New Jersey 08830| http://www.enterprisedb.com/ ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] integration of pgcluster into postgresql
Support of PGCluster-I, which we're discussing here, is being dropped in favor of the shared-disk PGCluster-II which was demonstrated at the anniversary conference. IIRC, PGCluster-I does use command-based replication but is merged into the parser in such a way as to make it work quite well--unlike the man-in-the-middle approach taken by pgpool. A shared disk approach doesn't fullfill the needs of everyone. So I guess PGCluster I and II would answer different needs and can co-exist. Finally, pgcluster is very out-of-date. The last version uses 8.0.1 and was released on Mar. 7, 2005. If the author can't find the time to maintain it, I don't see why that burden should be put on the shoulders of this community. Umm, I don't know where you're looking Jim, but the last update was February 10, 2006 and it's for PostgreSQL 8.1.1. Frankly, it has had a very good track record of development and bug fixes... so let's not make assumptions on (very large PostgreSQL) projects we're unfamiliar with. 8.1.2 actually, which I have updated to apply to 8.1.4. I posted a patch on the pgcluster mailing list but I already have two significant fixes related to pgcluster and one minor change related to the upgrade itself. I am to use PGCluster in a real time embedded fault-tolerant system, so I'm likely to emit a few more patches in the way to make it more robust and performant on some aspects. That said, my company would feel more confortable with the idea that it's part of the postgresql mainstream distro for many obvious reasons - or we might drop postgresql altogether - which is why I'm proposing myself to do the necessary work to integrate it in postgresql if there's interest. It's pretty non intrusive for your average postgresql user who won't see a difference, and very little so for a postgresql developer. At the same time, anyone wanting replication will have that option in standard. It's all benefits. So, should I give it a try? __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] integration of pgcluster into postgresql
On Sat, Aug 26, 2006 at 08:44:07AM -0400, Jonah H. Harris wrote: Second, pgcluster is (AFAIK) command-based replication, which has some very, very serious drawbacks. If PostgreSQL were to include a replication solution, I'd certainly hope it wouldn't be command-based. Support of PGCluster-I, which we're discussing here, is being dropped in favor of the shared-disk PGCluster-II which was demonstrated at the anniversary conference. IIRC, PGCluster-I does use command-based replication but is merged into the parser in such a way as to make it work quite well--unlike the man-in-the-middle approach taken by pgpool. Ahh, I didn't realize that. Good to know. Finally, pgcluster is very out-of-date. The last version uses 8.0.1 and was released on Mar. 7, 2005. If the author can't find the time to maintain it, I don't see why that burden should be put on the shoulders of this community. Umm, I don't know where you're looking Jim, but the last update was February 10, 2006 and it's for PostgreSQL 8.1.1. Frankly, it has had a very good track record of development and bug fixes... so let's not make assumptions on (very large PostgreSQL) projects we're unfamiliar with. http://pgcluster.projects.postgresql.org/; the latest date I see there is Mar. 7, 2005, and the newest version is 8.0.1. -- Jim C. Nasby, Sr. Engineering Consultant [EMAIL PROTECTED] Pervasive Software http://pervasive.comwork: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461 ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] integration of pgcluster into postgresql
On Sat, Aug 26, 2006 at 11:18:04AM -0700, Chahine Hamila wrote: 8.1.2 actually, which I have updated to apply to 8.1.4. I posted a patch on the pgcluster mailing list but I already have two significant fixes related to pgcluster and one minor change related to the upgrade itself. I am to use PGCluster in a real time embedded fault-tolerant system, so I'm likely to emit a few more patches in the way to make it more robust and performant on some aspects. That said, my company would feel more confortable with the idea that it's part of the postgresql mainstream distro for many obvious reasons - or we might drop postgresql altogether - which is why I'm proposing myself to do If they're that concerned, why don't they just pay for something like mammouth replicator? Or pay someone for a support contract. the necessary work to integrate it in postgresql if there's interest. It's pretty non intrusive for your average postgresql user who won't see a difference, and very little so for a postgresql developer. At the same time, anyone wanting replication will have that option in standard. It's all benefits. So, should I give it a try? From an advocacy standpoint, I'd love to see built-in replication... but I just don't see it happening. The communities been pretty clear on this... -- Jim C. Nasby, Sr. Engineering Consultant [EMAIL PROTECTED] Pervasive Software http://pervasive.comwork: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461 ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] integration of pgcluster into postgresql
Caution! Blatant use of sarcasm ahead. On 8/26/06, Jim C. Nasby [EMAIL PROTECTED] wrote: Umm, I don't know where you're looking Jim, but the last update was February 10, 2006 http://pgcluster.projects.postgresql.org/; the latest date I see there is Mar. 7, 2005, and the newest version is 8.0.1. sarcasm *Everyone* knows that pgfoundry is the source for all things PostgreSQL! Google has led you astray... pgfoundry is the search engine of the future. Don't trust the top Google links my friend! /sarcasm In all reality, I'm just kiddin' with ya Jim. I have to be sarcastic as I've been beaten lately (by multiple people) into believing that everyone knows about and uses pgfoundry :( For gory details see [PATCHES] Adding fulldisjunctions to the contrib. Nevertheless, here's the new link to PGCluster: http://pgfoundry.org/projects/pgcluster /me is done writing emails for the night. Being sick is a killer on my patience and email diplomacy :( -Jonah ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] integration of pgcluster into postgresql
Chahine Hamila [EMAIL PROTECTED] writes: I posted a patch on the pgcluster mailing list but I already have two significant fixes related to pgcluster and one minor change related to the upgrade itself. I am to use PGCluster in a real time embedded fault-tolerant system, so I'm likely to emit a few more patches in the way to make it more robust and performant on some aspects. That all sounds great. That said, my company would feel more confortable with the idea that it's part of the postgresql mainstream distro for many obvious reasons - or we might drop postgresql altogether - which is why I'm proposing myself to do the necessary work to integrate it in postgresql if there's interest. The core development team has only a very finite number of cycles available. Would you rather we spend our time on fixing pgcluster than on fixing the core Postgres database? My take on all this is that there's no one-size-fits-all replication solution, and therefore the right approach is to have multiple active subprojects. pgcluster sounds like it's steaming along nicely where it is. regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] integration of pgcluster into postgresql
On 8/27/06, Tom Lane [EMAIL PROTECTED] wrote: My take on all this is that there's no one-size-fits-all replication solution, and therefore the right approach is to have multiple active subprojects. Can't help but agree there. Maybe someday the subprojects will get together and come up with a standard framework each of them could use... but PGCluster, as good as it is, certainly doesn't address much of the replication arena; primarily asynchronous replication for both multimaster and master-slave. -- Jonah H. Harris, Software Architect | phone: 732.331.1300 EnterpriseDB Corporation| fax: 732.331.1301 33 Wood Ave S, 2nd Floor| [EMAIL PROTECTED] Iselin, New Jersey 08830| http://www.enterprisedb.com/ ---(end of broadcast)--- TIP 6: explain analyze is your friend
[HACKERS] integration of pgcluster into postgresql
Hi there, I guess many - if not most - here have tried pgcluster. For those who didn't, postgresql is pretty much the equivalent of pgcluster configured without load balancer or replicator, in read-write standalone mode. From a user point of view, that's three maximum additional configuration files, which can be basically set to those default values and distributed as is in standard (making installs/upgrades transparent to non-cluster environments). From a developer point of view, the pgcluster code is quite easy to take a hold on. pgcluster still has quite a few pending issues, but it's good enough for many users. Integrating it as part of a standard postgresql distribution would likely not disrupt standard postgresql functionning, while giving it the replication features it lacks as is. It's also likely to accelerate its maturing by a more widespread adoption and as a result overcome most of its issues. If the idea of its integration in the main postgresql code is of any interest to the postgresql team, I'm willing to invest some effort on it. __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
