On Wed, Oct 29, 2014 at 10:21 AM, Simon Riggs <si...@2ndquadrant.com> wrote: > There is a real danger that your "ta-dah" moment sometime in the > future contains flaws which need to be addressed, but we now have > piles of questionable infrastructure lieing around. If you have > similar doubts about anything I'm doing, please just ask.
I think the implication that the infrastructure created to date is of questionable utility for parallelism is false and unjustifed. Every infrastructure patch thus far committed, and every patch proposed other than this one, has been accompanied by code that demonstrates what it does and how that's useful. Most of them have had enough interest by extension authors to generate bug reports from other people - most notably Petr, who has obviously been playing with all of the existing modules enough to notice bugs and come to conclusions about the shortcomings of the existing technology similar to my own. Interest in dynamic background workers seems to go considerably further, to the point where it made 9.4's list of major features. > If we could see the 6-12 patch descriptions and understand where you > are going it would help. OK, here's what I think we need (roughly): 1. The remaining pg_background patches. The main things in there are (a) the ability to propagate errors from a background worker back to the user backend and (b) the ability to synchronize the background worker's GUC state with the user backend. 2. Group locking. 3. Patches to synchronize the (a) snapshot, (b) combo CID hash, and (c) transaction state from the user backend to a background worker. Draft patches for (a) and (b) are already completed yet, but not published because I didn't think anyone would be very excited about them just yet. 4. A patch to add "parallel mode" in which a whole bunch of things are prohibited in the user backend and even more things are prohibited in the background worker - specifically, things that won't be safe in those contexts. Noah and Amit Kapila worked on this and we have draft patches for this as well, also not published and for the same reasons. 5. A patch to add a "proisparallel" flag or similar so that we can label parallel-safe functions. 6. A patch to add a way for a background worker that hits an error to kick the user backend, probably via the ProcSignal mechanism, so that the user doesn't have to poll the shm_mq structures it's using to communicate with those backends for errors continuously (or alternative sit idle while those background workers do all the work). This might be something that can be skipped for early versions, but eventually I think we'll need it. 7. Eventually, a dynamic shared memory allocator. A lot of things can be done without this, but some can't. I think that's most of it. No doubt we'll discover a few other needs along the way, but if we had that infrastructure, then I think it would be quite feasible for an experienced hacker to parallelize most of what we want to parallelize without having to invent too much that's really fundamental. > If I did, I don't think it would be that useful to interfere. > Cooperation seems better use of my time, if possible. I am not asking anyone to interfere, in the sense of getting in the way, but I certainly think it would be useful for people to contribute code and/or reviews. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
