Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-12-02 Thread Bruce Momjian
On Fri, Nov 14, 2014 at 02:51:32PM +1300, David Rowley wrote: Likely for most aggregates, like count, sum, max, min, bit_and and bit_or the merge function would be the same as the transition function, as the state type is just the same as the input type. It would only be aggregates like avg(),

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-12-02 Thread Kouhei Kaigai
On Fri, Nov 14, 2014 at 02:51:32PM +1300, David Rowley wrote: Likely for most aggregates, like count, sum, max, min, bit_and and bit_or the merge function would be the same as the transition function, as the state type is just the same as the input type. It would only be aggregates like

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-12-02 Thread Michael Paquier
On Wed, Dec 3, 2014 at 3:23 PM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: On Fri, Nov 14, 2014 at 02:51:32PM +1300, David Rowley wrote: Likely for most aggregates, like count, sum, max, min, bit_and and bit_or the merge function would be the same as the transition function, as the state

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-17 Thread Robert Haas
On Thu, Nov 13, 2014 at 7:27 PM, Simon Riggs si...@2ndquadrant.com wrote: On 12 November 2014 00:54, Robert Haas robertmh...@gmail.com wrote: On Tue, Nov 11, 2014 at 3:29 AM, Simon Riggs si...@2ndquadrant.com wrote: * only functions marked as CONTAINS NO SQL We don't really know what

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-17 Thread Simon Riggs
On 14 November 2014 11:02, Kouhei Kaigai kai...@ak.jp.nec.com wrote: I'd like to throw community folks a question. Did someone have a discussion to the challenge of aggregate push-down across relations join in the past? It potentially reduces number of rows to be joined. If we already had,

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-17 Thread Simon Riggs
On 17 November 2014 16:01, Robert Haas robertmh...@gmail.com wrote: I still don't know what CONTAINS NO SQL means. It's a function marking that would indicate we aren't allowed to take snapshots or run SQL. I think you should publish the scheme for marking functions as safe for parallelism, so

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-17 Thread David Rowley
On 18 November 2014 05:19, Simon Riggs si...@2ndquadrant.com wrote: On 14 November 2014 11:02, Kouhei Kaigai kai...@ak.jp.nec.com wrote: I'd like to throw community folks a question. Did someone have a discussion to the challenge of aggregate push-down across relations join in the past?

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-17 Thread Kouhei Kaigai
On 18 November 2014 05:19, Simon Riggs si...@2ndquadrant.com wrote: On 14 November 2014 11:02, Kouhei Kaigai kai...@ak.jp.nec.com wrote: I'd like to throw community folks a question. Did someone have a discussion to the challenge of aggregate push-down across

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-14 Thread David Rowley
On 14 November 2014 20:37, Jim Nasby jim.na...@bluetreble.com wrote: On 11/12/14, 1:54 AM, David Rowley wrote: We'd also need to add some infrastructure to merge aggregate states together for this to work properly. This means that could also work for avg() and stddev etc. For max() and

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-14 Thread Simon Riggs
On 14 November 2014 07:37, Jim Nasby jim.na...@bluetreble.com wrote: On 11/12/14, 1:54 AM, David Rowley wrote: On Tue, Nov 11, 2014 at 9:29 PM, Simon Riggs si...@2ndquadrant.com mailto:si...@2ndquadrant.com wrote: This plan type is widely used in reporting queries, so will hit the

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-14 Thread Simon Riggs
On 14 November 2014 01:51, David Rowley dgrowle...@gmail.com wrote: When I mentioned this, I didn't mean to appear to be placing a road block.I was just bringing to the table the information that COUNT(*) + COUNT(*) works ok for merging COUNT(*)'s sub totals, but AVG(n) + AVG(n) does not.

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-14 Thread Kouhei Kaigai
On 14 November 2014 07:37, Jim Nasby jim.na...@bluetreble.com wrote: On 11/12/14, 1:54 AM, David Rowley wrote: On Tue, Nov 11, 2014 at 9:29 PM, Simon Riggs si...@2ndquadrant.com mailto:si...@2ndquadrant.com wrote: This plan type is widely used in reporting queries, so will hit

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-14 Thread David Rowley
On Fri, Nov 14, 2014 at 1:27 PM, Simon Riggs si...@2ndquadrant.com wrote: On 12 November 2014 00:54, Robert Haas robertmh...@gmail.com wrote: Interestingly, I have a fairly solid idea of what proisparallel is, but I have no clear idea what CONTAINS NO SQL is or why it's relevant. I would

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-13 Thread Simon Riggs
On 12 November 2014 00:54, Robert Haas robertmh...@gmail.com wrote: I'm going to aim for the simpler: Hash Aggregate - Parallel Seq Scan Workers: 4 Yeah, I know that won't perform as well as what you're proposing, but I'm fairly sure it's simpler. Simple is best, so +1. -- Simon

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-13 Thread Simon Riggs
On 12 November 2014 07:54, David Rowley dgrowle...@gmail.com wrote: On Tue, Nov 11, 2014 at 9:29 PM, Simon Riggs si...@2ndquadrant.com wrote: This plan type is widely used in reporting queries, so will hit the mainline of BI applications and many Mat View creations. This will allow SELECT

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-13 Thread Simon Riggs
On 12 November 2014 00:54, Robert Haas robertmh...@gmail.com wrote: On Tue, Nov 11, 2014 at 3:29 AM, Simon Riggs si...@2ndquadrant.com wrote: * only functions marked as CONTAINS NO SQL We don't really know what proisparallel is, but we do know what CONTAINS NO SQL means and can easily check

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-13 Thread David Rowley
On Fri, Nov 14, 2014 at 1:19 PM, Simon Riggs si...@2ndquadrant.com wrote: On 12 November 2014 07:54, David Rowley dgrowle...@gmail.com wrote: On Tue, Nov 11, 2014 at 9:29 PM, Simon Riggs si...@2ndquadrant.com wrote: This plan type is widely used in reporting queries, so will hit the

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-13 Thread Kouhei Kaigai
On 12 November 2014 07:54, David Rowley dgrowle...@gmail.com wrote: On Tue, Nov 11, 2014 at 9:29 PM, Simon Riggs si...@2ndquadrant.com wrote: This plan type is widely used in reporting queries, so will hit the mainline of BI applications and many

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-13 Thread Kouhei Kaigai
On Fri, Nov 14, 2014 at 2:12 PM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: On 12 November 2014 07:54, David Rowley dgrowle...@gmail.com Take int4_avg_accum() for example it does: transdata-count++; transdata-sum += newval;

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-13 Thread Jim Nasby
On 11/12/14, 1:54 AM, David Rowley wrote: On Tue, Nov 11, 2014 at 9:29 PM, Simon Riggs si...@2ndquadrant.com mailto:si...@2ndquadrant.com wrote: This plan type is widely used in reporting queries, so will hit the mainline of BI applications and many Mat View creations. This will

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-12 Thread Atri Sharma
On Wed, Nov 12, 2014 at 1:24 PM, David Rowley dgrowle...@gmail.com wrote: On Tue, Nov 11, 2014 at 9:29 PM, Simon Riggs si...@2ndquadrant.com wrote: This plan type is widely used in reporting queries, so will hit the mainline of BI applications and many Mat View creations. This will allow

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-12 Thread Robert Haas
On Tue, Nov 11, 2014 at 7:48 PM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Isn't provolatile = PROVOLATILE_IMMUTABLE sufficient? There are certainly things that are parallel-safe that are not immutable. It might be the case that everything immutable is parallel-safe. -- Robert Haas

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-12 Thread Michael Paquier
On Wed, Nov 12, 2014 at 9:49 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Nov 11, 2014 at 7:48 PM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Isn't provolatile = PROVOLATILE_IMMUTABLE sufficient? There are certainly things that are parallel-safe that are not immutable. It might be the

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-11 Thread Simon Riggs
On 10 November 2014 15:57, Robert Haas robertmh...@gmail.com wrote: On Wed, Oct 15, 2014 at 2:55 PM, Simon Riggs si...@2ndquadrant.com wrote: Something usable, with severe restrictions, is actually better than we have now. I understand the journey this work represents, so don't be embarrassed

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-11 Thread Kouhei Kaigai
Given where we are with the infrastructure, there would be a number of unhandled problems, such as deadlock detection (needs group locking or similar), assessment of quals as to parallel-safety (needs proisparallel or similar), general waterproofing to make sure that pushing down a

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-11 Thread Robert Haas
On Tue, Nov 11, 2014 at 3:29 AM, Simon Riggs si...@2ndquadrant.com wrote: * only functions marked as CONTAINS NO SQL We don't really know what proisparallel is, but we do know what CONTAINS NO SQL means and can easily check for it. Plus I already have a patch for this, slightly bitrotted.

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-11 Thread David Rowley
On Tue, Nov 11, 2014 at 9:29 PM, Simon Riggs si...@2ndquadrant.com wrote: This plan type is widely used in reporting queries, so will hit the mainline of BI applications and many Mat View creations. This will allow SELECT count(*) FROM foo to go faster also. We'd also need to add some

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-10 Thread Andres Freund
Hi Robert, All, On 2014-11-10 10:57:16 -0500, Robert Haas wrote: On Wed, Oct 15, 2014 at 2:55 PM, Simon Riggs si...@2ndquadrant.com wrote: Something usable, with severe restrictions, is actually better than we have now. I understand the journey this work represents, so don't be embarrassed

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-10 Thread Haribabu Kommi
On Tue, Nov 11, 2014 at 10:21 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-11-10 10:57:16 -0500, Robert Haas wrote: Does parallelism help at all? I'm pretty damn sure. We can't even make a mildly powerfull storage fully busy right now. Heck, I can't make my workstation's storage

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-10 Thread Amit Kapila
On Tue, Nov 11, 2014 at 5:30 AM, Haribabu Kommi kommi.harib...@gmail.com wrote: On Tue, Nov 11, 2014 at 10:21 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-11-10 10:57:16 -0500, Robert Haas wrote: Does parallelism help at all? I'm pretty damn sure. We can't even make a mildly

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-10 Thread Haribabu Kommi
On Tue, Nov 11, 2014 at 2:35 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Tue, Nov 11, 2014 at 5:30 AM, Haribabu Kommi kommi.harib...@gmail.com wrote: On Tue, Nov 11, 2014 at 10:21 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-11-10 10:57:16 -0500, Robert Haas wrote: Does

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-10 Thread Amit Kapila
On Tue, Nov 11, 2014 at 9:42 AM, Haribabu Kommi kommi.harib...@gmail.com wrote: On Tue, Nov 11, 2014 at 2:35 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Tue, Nov 11, 2014 at 5:30 AM, Haribabu Kommi kommi.harib...@gmail.com wrote: On Tue, Nov 11, 2014 at 10:21 AM, Andres Freund

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-10 Thread Kouhei Kaigai
Subject: Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan On Tue, Nov 11, 2014 at 2:35 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Tue, Nov 11, 2014 at 5:30 AM, Haribabu Kommi kommi.harib...@gmail.com wrote: On Tue, Nov 11, 2014 at 10:21 AM, Andres Freund