Re: [HACKERS] xReader, double-effort (was: Temporary tables under hot standby)
Sure Kevin, will get the wiki page ready asap, and reply back. Thanks. On Thu, Apr 26, 2012 at 8:10 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: [resending because of postgresql.org bounces on first try] Simon Riggs si...@2ndquadrant.com wrote: Kevin Grittner kevin.gritt...@wicourts.gov wrote: The GSoC xReader project is intended to be a major step toward that, by providing a way to translate the WAL stream to a series of notifications of logical events to clients which register with xReader. This is already nearly finished in prototype and will be published in May. Andres Freund is working on it, copied here. URL? It looks like there is significant overlap there. Hard for me to know without more information. It sounds like there is at least some overlap. I hope that can involve cooperation, with the efforts of Andres forming the basis of Aakash's GSoC effort. That might leave him more time to polish up the user filters. Aakash: It seems like we need that Wiki page rather sooner than later. Can you get to that quickly? I would think that just copying the text from your approved GSoC proposal would be a very good start. If you need help figuring out how to embed the images from your proposal, let me know. -Kevin
Re: [HACKERS] xReader, double-effort (was: Temporary tables under hot standby)
All, the wiki page is now up at http://wiki.postgresql.org/wiki/XReader. On Sat, Apr 28, 2012 at 1:19 AM, Aakash Goel aakash.b...@gmail.com wrote: Sure Kevin, will get the wiki page ready asap, and reply back. Thanks. On Thu, Apr 26, 2012 at 8:10 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: [resending because of postgresql.org bounces on first try] Simon Riggs si...@2ndquadrant.com wrote: Kevin Grittner kevin.gritt...@wicourts.gov wrote: The GSoC xReader project is intended to be a major step toward that, by providing a way to translate the WAL stream to a series of notifications of logical events to clients which register with xReader. This is already nearly finished in prototype and will be published in May. Andres Freund is working on it, copied here. URL? It looks like there is significant overlap there. Hard for me to know without more information. It sounds like there is at least some overlap. I hope that can involve cooperation, with the efforts of Andres forming the basis of Aakash's GSoC effort. That might leave him more time to polish up the user filters. Aakash: It seems like we need that Wiki page rather sooner than later. Can you get to that quickly? I would think that just copying the text from your approved GSoC proposal would be a very good start. If you need help figuring out how to embed the images from your proposal, let me know. -Kevin
Re: [HACKERS] xReader, double-effort
On 04/26/2012 12:57 PM, Robert Haas wrote: I don't recall seeing any previous postings about the 2Q project, either. We should try to keep these discussions on-list. First brought up two months ago to the list it seemed most relevant to at the time: http://archives.postgresql.org/pgsql-cluster-hackers/2012-02/msg1.php and the solid commentary started coming out yesterday, right near the approach of May as promised. There hasn't been much posted on hackers about this out of respect for the extended CommitFest. I personally suggested Aakash vent ideas out on pgsql-students (or here) at the beginning of the month: http://archives.postgresql.org/pgsql-students/2012-04/msg2.php but the next time I heard about the subject was the acceptance notice. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xReader, double-effort
Greg Smith g...@2ndquadrant.com wrote: On 04/26/2012 12:57 PM, Robert Haas wrote: I don't recall seeing any previous postings about the 2Q project, either. We should try to keep these discussions on-list. First brought up two months ago to the list it seemed most relevant to at the time: http://archives.postgresql.org/pgsql-cluster-hackers/2012-02/msg1.php and the solid commentary started coming out yesterday, right near the approach of May as promised. There hasn't been much posted on hackers about this out of respect for the extended CommitFest. I personally suggested Aakash vent ideas out on pgsql-students (or here) at the beginning of the month: http://archives.postgresql.org/pgsql-students/2012-04/msg2.php but the next time I heard about the subject was the acceptance notice. That's largely my fault. Aakash's original proposal was naively assuming that logical records were available in our xlog stream and wanted to work on tools to do something useful with that data. There was some confusion among the various core team members, committers, and other contributors who volunteered to review the proposals about whether he was talking about analyzing the server log or xlog. We had a wide-ranging discussion on the freenode #postgresql channel about it, and brainstormed to the current idea -- at which point Aakash had about a week and a half to rewrite his proposal along the lines discussed on irc. A week after that the acceptance notices came out. If we had had any idea that 2Q was working on something along these lines, we would have coordinated or guided him in another direction. The post you cite from a couple months ago was way too vague to have even an inkling that it had anything to do with providing feeds of the logical interpretation of the WAL stream. I don't recall having heard the idea even mentioned before. The timing in regard to the release process seems to have had an effect on both groups here, since I suggested he read up on walsender and walreceiver and a few other areas, put together a Wiki page with the plan, and *then* describe the preliminary plan to the community (a month or so before starting to code), so that details could be hashed out in public before time was spent coding. Had there not been so many people heads-down trying to get the release out, I might have advised him to post a hand-wavy description of the effort before he got the Wiki page set up and reviewed the relevant code. As Josh points out, his first choice was to write utilities to *do* something with the logical xlog information, and he only switched over to working on *producing* it when he heard that no such thing was currently available. I think there's a lot of room to coordinate these efforts so that there *isn't* a waste of effort as long as communication is open enough. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xReader, double-effort
On 04/27/2012 06:48 PM, Kevin Grittner wrote: That's largely my fault. It may not have come out that way, but I was trying more to point out the complexity of the story rather than assign blame. I find it hard to point at any obvious don't do that again target here. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xReader, double-effort (was: Temporary tables under hot standby)
On Thu, Apr 26, 2012 at 3:40 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: [resending because of postgresql.org bounces on first try] Simon Riggs si...@2ndquadrant.com wrote: Kevin Grittner kevin.gritt...@wicourts.gov wrote: The GSoC xReader project is intended to be a major step toward that, by providing a way to translate the WAL stream to a series of notifications of logical events to clients which register with xReader. sigh Yes, it appears very similar. I didn't know about this. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xReader, double-effort (was: Temporary tables under hot standby)
On Thu, Apr 26, 2012 at 11:02 AM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, Apr 26, 2012 at 3:40 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: [resending because of postgresql.org bounces on first try] Simon Riggs si...@2ndquadrant.com wrote: Kevin Grittner kevin.gritt...@wicourts.gov wrote: The GSoC xReader project is intended to be a major step toward that, by providing a way to translate the WAL stream to a series of notifications of logical events to clients which register with xReader. sigh Yes, it appears very similar. I didn't know about this. I don't recall seeing any previous postings about the 2Q project, either. We should try to keep these discussions on-list. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xReader, double-effort (was: Temporary tables under hot standby)
On Thu, Apr 26, 2012 at 5:57 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, Apr 26, 2012 at 11:02 AM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, Apr 26, 2012 at 3:40 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: [resending because of postgresql.org bounces on first try] Simon Riggs si...@2ndquadrant.com wrote: Kevin Grittner kevin.gritt...@wicourts.gov wrote: The GSoC xReader project is intended to be a major step toward that, by providing a way to translate the WAL stream to a series of notifications of logical events to clients which register with xReader. sigh Yes, it appears very similar. I didn't know about this. I don't recall seeing any previous postings about the 2Q project, either. We should try to keep these discussions on-list. While that's true, it wasn't much of a stretch to think that I might be planning to do more work on replication since I've done that for years and years now. Also, I do keep requesting GSOC doesn't put such major tasks out there, but not much heed taken. Beautiful illustration of a multi-master conflict case though. Not sure just yet what the resolution is, and it will probably take a few weeks to resolve that happily. Glad we're all facing in the same direction, which is the most important thing. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] xReader, double-effort
The GSoC xReader project is intended to be a major step toward that, by providing a way to translate the WAL stream to a series of notifications of logical events to clients which register with xReader. sigh Yes, it appears very similar. I didn't know about this. This isn't a problem. In fact, it's the *exact opposite* of a problem. The student wanted to do more advanced things with the xlog stream, and we convinced him to narrow the scope in order to make the project achievable. With 2Q having done some/most of the basic work, the student can focus on more advanced applications of the data again. Win! However, it would be great if someone from 2Q co-mentored him. --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
