Re: [HACKERS] Parallel query execution with SPI

On 31.03.2017 13:48, Robert Haas wrote: On Fri, Mar 31, 2017 at 3:33 AM, Konstantin Knizhnik wrote: It is possible to execute query concurrently using SPI? If so, how it can be enforced? I tried to open cursor with CURSOR_OPT_PARALLEL_OK flag but it doesn't help:

Re: [HACKERS] Parallel query execution with SPI

On Fri, Mar 31, 2017 at 4:18 PM, Robert Haas wrote: > On Fri, Mar 31, 2017 at 3:33 AM, Konstantin Knizhnik > wrote: >> It is possible to execute query concurrently using SPI? >> If so, how it can be enforced? >> I tried to open cursor with

Re: [HACKERS] Parallel query execution with SPI

On Fri, Mar 31, 2017 at 3:33 AM, Konstantin Knizhnik wrote: > It is possible to execute query concurrently using SPI? > If so, how it can be enforced? > I tried to open cursor with CURSOR_OPT_PARALLEL_OK flag but it doesn't help: > query is executed by single backend

Re: [HACKERS] Parallel query execution

On Tuesday, January 15, 2013 at 2:14 PM, Bruce Momjian wrote: I mentioned last year that I wanted to start working on parallelism: https://wiki.postgresql.org/wiki/Parallel_Query_Execution I believe it is time to start adding parallel execution to the backend. We already have some

Re: [HACKERS] Parallel query execution

On Thu, Jan 24, 2013 at 02:34:49PM -0800, Paul Ramsey wrote: On Tuesday, January 15, 2013 at 2:14 PM, Bruce Momjian wrote: I mentioned last year that I wanted to start working on parallelism: https://wiki.postgresql.org/wiki/Parallel_Query_Execution I believe it is time to

Re: [HACKERS] Parallel query execution

On Tue, Jan 15, 2013 at 11:07 PM, Tom Lane t...@sss.pgh.pa.us wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: There are still 34 items needing attention in CF3. I suggest that, if you have some spare time, your help would be very much appreciated there. The commitfest that started on

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 12:03 AM, Bruce Momjian br...@momjian.us wrote: On Tue, Jan 15, 2013 at 11:01:04PM +, Simon Riggs wrote: On 15 January 2013 22:55, Bruce Momjian br...@momjian.us wrote: Why is this being discussed now? It is for 9.4 and will take months. I didn't think there

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 2:07 AM, Tom Lane t...@sss.pgh.pa.us wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: There are still 34 items needing attention in CF3. I suggest that, if you have some spare time, your help would be very much appreciated there. The commitfest that started on

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 6:52 AM, Magnus Hagander mag...@hagander.net wrote: On Wed, Jan 16, 2013 at 12:03 AM, Bruce Momjian br...@momjian.us wrote: On Tue, Jan 15, 2013 at 11:01:04PM +, Simon Riggs wrote: On 15 January 2013 22:55, Bruce Momjian br...@momjian.us wrote: Why is this being

Re: [HACKERS] Parallel query execution

* Claudio Freire (klaussfre...@gmail.com) wrote: Well, there's the fault in your logic. It won't be as linear. I really don't see how this has become so difficult to communicate. It doesn't have to be linear. We're currently doing massive amounts of parallel processing by hand using

Re: [HACKERS] Parallel query execution

* Tom Lane (t...@sss.pgh.pa.us) wrote: In case you hadn't noticed, we've totally lost control of the CF process. I concur. Quite aside from the lack of progress on closing CF3, major hackers who should know better are submitting significant new feature patches now, despite our agreement

Re: [HACKERS] Parallel query execution

* Daniel Farina (dan...@heroku.com) wrote: I have been skimming the commitfest application, and unlike some of the previous commitfests a huge number of patches have had review at some point in time, but probably need more...so looking for the red Nobody in the 'reviewers' column probably

Re: [HACKERS] Parallel query execution

On 01/15/2013 11:32 PM, Bruce Momjian wrote: On Wed, Jan 16, 2013 at 01:28:18PM +0900, Michael Paquier wrote: On Wed, Jan 16, 2013 at 1:22 PM, Josh Berkus j...@agliodbs.com wrote: Claudio, Stephen, It really seems like the areas where we could get the most bang for the buck

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 10:33 AM, Stephen Frost sfr...@snowman.net wrote: * Claudio Freire (klaussfre...@gmail.com) wrote: Well, there's the fault in your logic. It won't be as linear. I really don't see how this has become so difficult to communicate. It doesn't have to be linear. We're

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 08:42:29AM -0500, Stephen Frost wrote: * Daniel Farina (dan...@heroku.com) wrote: I have been skimming the commitfest application, and unlike some of the previous commitfests a huge number of patches have had review at some point in time, but probably need more...so

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 01:37:28PM +0900, Michael Paquier wrote: On Wed, Jan 16, 2013 at 1:32 PM, Bruce Momjian br...@momjian.us wrote: On Wed, Jan 16, 2013 at 01:28:18PM +0900, Michael Paquier wrote: On Wed, Jan 16, 2013 at 1:22 PM, Josh Berkus j...@agliodbs.com

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 01:48:29AM -0300, Alvaro Herrera wrote: Bruce Momjian escribió: On Wed, Jan 16, 2013 at 01:28:18PM +0900, Michael Paquier wrote: On Wed, Jan 16, 2013 at 1:22 PM, Josh Berkus j...@agliodbs.com wrote: Claudio, Stephen, It really seems like

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 08:11:06AM -0500, Robert Haas wrote: We kind of do - when in a CF we should do reviewing of existing patches, when outside a CF we should do discussions and work on new features. It's on http://wiki.postgresql.org/wiki/CommitFest. It doesn't specifically say do this

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 09:05:39AM -0500, Andrew Dunstan wrote: On 01/15/2013 11:32 PM, Bruce Momjian wrote: On Wed, Jan 16, 2013 at 01:28:18PM +0900, Michael Paquier wrote: On Wed, Jan 16, 2013 at 1:22 PM, Josh Berkus j...@agliodbs.com wrote: Claudio, Stephen, It really

Re: [HACKERS] Parallel query execution

On 01/16/2013 12:20 PM, Bruce Momjian wrote: On Wed, Jan 16, 2013 at 09:05:39AM -0500, Andrew Dunstan wrote: On 01/15/2013 11:32 PM, Bruce Momjian wrote: On Wed, Jan 16, 2013 at 01:28:18PM +0900, Michael Paquier wrote: On Wed, Jan 16, 2013 at 1:22 PM, Josh Berkus j...@agliodbs.com wrote:

Re: [HACKERS] Parallel query execution

* Bruce Momjian (br...@momjian.us) wrote: I am not sure how a COPY could be easily parallelized, but I supposed it could be done as part of the 1GB segment feature. People have complained that COPY is CPU-bound, so it might be very interesting to see if we could offload some of that parsing

Re: [HACKERS] Parallel query execution

2013/1/16 Stephen Frost sfr...@snowman.net: * Bruce Momjian (br...@momjian.us) wrote: I am not sure how a COPY could be easily parallelized, but I supposed it could be done as part of the 1GB segment feature. People have complained that COPY is CPU-bound, so it might be very interesting to

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 10:06:51PM +0100, Pavel Stehule wrote: 2013/1/16 Stephen Frost sfr...@snowman.net: * Bruce Momjian (br...@momjian.us) wrote: I am not sure how a COPY could be easily parallelized, but I supposed it could be done as part of the 1GB segment feature. People have

Re: [HACKERS] Parallel query execution

2013/1/16 Bruce Momjian br...@momjian.us: Wiki updated: https://wiki.postgresql.org/wiki/Parallel_Query_Execution Could we add CTE to that opportunities list? I think that some kind of queries in CTE queries could be easilly parallelized. []s -- Dickson S. Guedes mail/xmpp:

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 07:57:01PM -0200, Dickson S. Guedes wrote: 2013/1/16 Bruce Momjian br...@momjian.us: Wiki updated: https://wiki.postgresql.org/wiki/Parallel_Query_Execution Could we add CTE to that opportunities list? I think that some kind of queries in CTE queries

Re: [HACKERS] Parallel query execution

On Tuesday, January 15, 2013, Stephen Frost wrote: * Gavin Flower (gavinflo...@archidevsys.co.nz javascript:;) wrote: How about being aware of multiple spindles - so if the requested data covers multiple spindles, then data could be extracted in parallel. This may, or may not, involve

Re: [HACKERS] Parallel query execution

On Tuesday, January 15, 2013, Gavin Flower wrote: On 16/01/13 11:14, Bruce Momjian wrote: I mentioned last year that I wanted to start working on parallelism: https://wiki.postgresql.org/wiki/Parallel_Query_Execution Years ago I added thread-safety to libpq. Recently I added two

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 10:04 PM, Jeff Janes jeff.ja...@gmail.com wrote: Hmm... How about being aware of multiple spindles - so if the requested data covers multiple spindles, then data could be extracted in parallel. This may, or may not, involve multiple I/O channels?

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 05:04:05PM -0800, Jeff Janes wrote: On Tuesday, January 15, 2013, Stephen Frost wrote: * Gavin Flower (gavinflo...@archidevsys.co.nz) wrote: How about being aware of multiple spindles - so if the requested data covers multiple spindles, then data could

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 11:44 PM, Bruce Momjian br...@momjian.us wrote: On Wed, Jan 16, 2013 at 05:04:05PM -0800, Jeff Janes wrote: On Tuesday, January 15, 2013, Stephen Frost wrote: * Gavin Flower (gavinflo...@archidevsys.co.nz) wrote: How about being aware of multiple spindles - so

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 11:56:21PM -0300, Claudio Freire wrote: On Wed, Jan 16, 2013 at 11:44 PM, Bruce Momjian br...@momjian.us wrote: On Wed, Jan 16, 2013 at 05:04:05PM -0800, Jeff Janes wrote: On Tuesday, January 15, 2013, Stephen Frost wrote: * Gavin Flower

Re: [HACKERS] Parallel query execution

On Wednesday, January 16, 2013, Stephen Frost wrote: * Bruce Momjian (br...@momjian.us javascript:;) wrote: I am not sure how a COPY could be easily parallelized, but I supposed it could be done as part of the 1GB segment feature. People have complained that COPY is CPU-bound, so it might

Re: [HACKERS] Parallel query execution

* Bruce Momjian (br...@momjian.us) wrote: Parallelism isn't going to help all queries, in fact it might be just a small subset, but it will be the larger queries. The pg_upgrade parallelism only helps clusters with multiple databases or tablespaces, but the improvements are significant. This

Re: [HACKERS] Parallel query execution

On 15 January 2013 22:14, Bruce Momjian br...@momjian.us wrote: I believe it is time to start adding parallel execution to the backend. We already have some parallelism in the backend: effective_io_concurrency and helper processes. I think it is time we start to consider additional options.

Re: [HACKERS] Parallel query execution

On Tue, Jan 15, 2013 at 10:39:10PM +, Peter Geoghegan wrote: On 15 January 2013 22:14, Bruce Momjian br...@momjian.us wrote: I believe it is time to start adding parallel execution to the backend. We already have some parallelism in the backend: effective_io_concurrency and helper

Re: [HACKERS] Parallel query execution

On Tue, Jan 15, 2013 at 10:53:29PM +, Simon Riggs wrote: On 15 January 2013 22:14, Bruce Momjian br...@momjian.us wrote: I mentioned last year that I wanted to start working on parallelism: We don't normally begin discussing topics for next release just as a CF is starting. Why is

Re: [HACKERS] Parallel query execution

On 15 January 2013 22:14, Bruce Momjian br...@momjian.us wrote: I mentioned last year that I wanted to start working on parallelism: We don't normally begin discussing topics for next release just as a CF is starting. Why is this being discussed now? -- Simon Riggs

Re: [HACKERS] Parallel query execution

On 15 January 2013 22:55, Bruce Momjian br...@momjian.us wrote: Why is this being discussed now? It is for 9.4 and will take months. I didn't think there was a better time. We don't usually discuss features during beta testing. Bruce, there are many, many patches on the queue. How will we

Re: [HACKERS] Parallel query execution

On 16/01/13 11:14, Bruce Momjian wrote: I mentioned last year that I wanted to start working on parallelism: https://wiki.postgresql.org/wiki/Parallel_Query_Execution Years ago I added thread-safety to libpq. Recently I added two parallel execution paths to pg_upgrade. The first

Re: [HACKERS] Parallel query execution

On Tue, Jan 15, 2013 at 11:01:04PM +, Simon Riggs wrote: On 15 January 2013 22:55, Bruce Momjian br...@momjian.us wrote: Why is this being discussed now? It is for 9.4 and will take months. I didn't think there was a better time. We don't usually discuss features during beta

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 12:03:50PM +1300, Gavin Flower wrote: On 16/01/13 11:14, Bruce Momjian wrote: I mentioned last year that I wanted to start working on parallelism: https://wiki.postgresql.org/wiki/Parallel_Query_Execution Years ago I added thread-safety to

Re: [HACKERS] Parallel query execution

* Gavin Flower (gavinflo...@archidevsys.co.nz) wrote: How about being aware of multiple spindles - so if the requested data covers multiple spindles, then data could be extracted in parallel. This may, or may not, involve multiple I/O channels? Yes, this should dovetail with partitioning and

Re: [HACKERS] Parallel query execution

On Tue, Jan 15, 2013 at 06:15:57PM -0500, Stephen Frost wrote: * Gavin Flower (gavinflo...@archidevsys.co.nz) wrote: How about being aware of multiple spindles - so if the requested data covers multiple spindles, then data could be extracted in parallel. This may, or may not, involve

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 7:14 AM, Bruce Momjian br...@momjian.us wrote: I mentioned last year that I wanted to start working on parallelism: https://wiki.postgresql.org/wiki/Parallel_Query_Execution Years ago I added thread-safety to libpq. Recently I added two parallel execution

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 09:11:20AM +0900, Michael Paquier wrote: On Wed, Jan 16, 2013 at 7:14 AM, Bruce Momjian br...@momjian.us wrote: I mentioned last year that I wanted to start working on parallelism: https://wiki.postgresql.org/wiki/Parallel_Query_Execution

Re: [HACKERS] Parallel query execution

On Tue, Jan 15, 2013 at 8:19 PM, Bruce Momjian br...@momjian.us wrote: Given our row-based storage architecture, I can't imagine we'd do anything other than take a row-based approach to this.. I would think we'd do two things: parallelize based on partitioning, and parallelize seqscan's

Re: [HACKERS] Parallel query execution

* Claudio Freire (klaussfre...@gmail.com) wrote: On Tue, Jan 15, 2013 at 8:19 PM, Bruce Momjian br...@momjian.us wrote: The 1GB idea is interesting. I found in pg_upgrade that file copy would just overwhelm the I/O channel, and that doing multiple copies on the same device had no win, but

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 12:13 AM, Stephen Frost sfr...@snowman.net wrote: * Claudio Freire (klaussfre...@gmail.com) wrote: On Tue, Jan 15, 2013 at 8:19 PM, Bruce Momjian br...@momjian.us wrote: The 1GB idea is interesting. I found in pg_upgrade that file copy would just overwhelm the I/O

Re: [HACKERS] Parallel query execution

* Claudio Freire (klaussfre...@gmail.com) wrote: On Wed, Jan 16, 2013 at 12:13 AM, Stephen Frost sfr...@snowman.net wrote: Sequentially scanning the *same* data over and over is certainly counterprouctive. Synchroscans fixed that, yes. That's not what we're talking about though- we're

Re: [HACKERS] Parallel query execution

but there will be quite a few cases where it's much, much better. Just cached segments. Actually, thanks to much faster storage (think SSD, SAN), it's easily possible for PostgreSQL to become CPU-limited on a seq scan query, even when reading from disk. -- Josh Berkus PostgreSQL Experts

Re: [HACKERS] Parallel query execution

* Josh Berkus (j...@agliodbs.com) wrote: Actually, thanks to much faster storage (think SSD, SAN), it's easily possible for PostgreSQL to become CPU-limited on a seq scan query, even when reading from disk. Particularly with a complex filter being applied or if it's feeding into something

Re: [HACKERS] Parallel query execution

Claudio, Stephen, It really seems like the areas where we could get the most bang for the buck in parallelism would be: 1. Parallel sort 2. Parallel aggregation (for commutative aggregates) 3. Parallel nested loop join (especially for expression joins, like GIS) -- Josh Berkus PostgreSQL

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 1:22 PM, Josh Berkus j...@agliodbs.com wrote: Claudio, Stephen, It really seems like the areas where we could get the most bang for the buck in parallelism would be: 1. Parallel sort 2. Parallel aggregation (for commutative aggregates) 3. Parallel nested loop join

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 01:28:18PM +0900, Michael Paquier wrote: On Wed, Jan 16, 2013 at 1:22 PM, Josh Berkus j...@agliodbs.com wrote: Claudio, Stephen, It really seems like the areas where we could get the most bang for the buck in parallelism would be: 1. Parallel

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 1:32 PM, Bruce Momjian br...@momjian.us wrote: On Wed, Jan 16, 2013 at 01:28:18PM +0900, Michael Paquier wrote: On Wed, Jan 16, 2013 at 1:22 PM, Josh Berkus j...@agliodbs.com wrote: Claudio, Stephen, It really seems like the areas where we could get

Re: [HACKERS] Parallel query execution

On Wed, Jan 16, 2013 at 12:55 AM, Stephen Frost sfr...@snowman.net wrote: If memory serves me correctly (and it does, I suffered it a lot), the performance hit is quite considerable. Enough to make it a lot worse rather than not as good. I feel like we must not be communicating very well.

Re: [HACKERS] Parallel query execution

Bruce Momjian escribió: On Wed, Jan 16, 2013 at 01:28:18PM +0900, Michael Paquier wrote: On Wed, Jan 16, 2013 at 1:22 PM, Josh Berkus j...@agliodbs.com wrote: Claudio, Stephen, It really seems like the areas where we could get the most bang for the buck in

Re: [HACKERS] Parallel query execution

On Tuesday, January 15, 2013, Simon Riggs wrote: On 15 January 2013 22:55, Bruce Momjian br...@momjian.us javascript:; wrote: Why is this being discussed now? It is for 9.4 and will take months. I didn't think there was a better time. We don't usually discuss features during beta

Re: [HACKERS] Parallel query execution

Alvaro Herrera alvhe...@2ndquadrant.com writes: There are still 34 items needing attention in CF3. I suggest that, if you have some spare time, your help would be very much appreciated there. The commitfest that started on Jan 15th has 65 extra items. Anything currently listed in CF3 can

Re: [HACKERS] Parallel Query Execution Project

Hi, On 09/28/2010 07:24 AM, Li Jie wrote: I'm interested in this parallel project, http://wiki.postgresql.org/wiki/Parallel_Query_Execution But I can't find any discussion and current progress in the website, it seems to stop for nearly a year? Yeah, I don't know of anybody really working

Re: [HACKERS] Parallel Query Execution Project

On Sep 28, 2010, at 10:15 AM, Markus Wanner wrote: Hi, On 09/28/2010 07:24 AM, Li Jie wrote: I'm interested in this parallel project, http://wiki.postgresql.org/wiki/Parallel_Query_Execution But I can't find any discussion and current progress in the website, it seems to stop for nearly