Re: [HACKERS] parallelize queries containing subplans

2017-02-14 Thread Amit Kapila
On Wed, Feb 15, 2017 at 4:38 AM, Robert Haas wrote: > On Tue, Feb 14, 2017 at 4:24 AM, Amit Kapila wrote: >> On further evaluation, it seems this patch has one big problem which >> is that it will allow forming parallel plans which can't be supported >> with current infrastructure. For ex. marki

Re: [HACKERS] parallelize queries containing subplans

2017-02-14 Thread Amit Kapila
On Wed, Feb 15, 2017 at 4:46 AM, Robert Haas wrote: > On Mon, Feb 13, 2017 at 11:07 PM, Amit Kapila wrote: >> I have removed the check of AlternativeSubPlan so that it can be >> handled by expression_tree_walker. > ... >> Attached patch fixes all the comments raised till now. > > Committed after

Re: [HACKERS] parallelize queries containing subplans

2017-02-14 Thread Robert Haas
On Mon, Feb 13, 2017 at 11:07 PM, Amit Kapila wrote: > I have removed the check of AlternativeSubPlan so that it can be > handled by expression_tree_walker. ... > Attached patch fixes all the comments raised till now. Committed after removing the reference to AlternativeSubPlan from the comment.

Re: [HACKERS] parallelize queries containing subplans

2017-02-14 Thread Robert Haas
On Tue, Feb 14, 2017 at 4:24 AM, Amit Kapila wrote: > On further evaluation, it seems this patch has one big problem which > is that it will allow forming parallel plans which can't be supported > with current infrastructure. For ex. marking immediate level params > as parallel safe can generate

Re: [HACKERS] parallelize queries containing subplans

2017-02-14 Thread Amit Kapila
On Tue, Jan 3, 2017 at 4:19 PM, Amit Kapila wrote: > On Wed, Dec 28, 2016 at 11:47 AM, Amit Kapila wrote: >> >> Now, we can further extend this to parallelize queries containing >> correlated subplans like below: >> >> explain select * from t1 where t1.i in (select t2.i from t2 where t2.i=t1.i);

Re: [HACKERS] parallelize queries containing subplans

2017-02-13 Thread Amit Kapila
On Thu, Feb 2, 2017 at 9:23 AM, Amit Kapila wrote: > On Wed, Feb 1, 2017 at 10:04 PM, Robert Haas wrote: >> On Tue, Jan 31, 2017 at 6:01 AM, Amit Kapila wrote: >>> Moved this patch to next CF. >> >> So here is what seems to be the key hunk from this patch: >> >> /* >> - * Since we don't

Re: [HACKERS] parallelize queries containing subplans

2017-02-01 Thread Amit Kapila
On Wed, Feb 1, 2017 at 10:04 PM, Robert Haas wrote: > On Tue, Jan 31, 2017 at 6:01 AM, Amit Kapila wrote: >> Moved this patch to next CF. > > So here is what seems to be the key hunk from this patch: > > /* > - * Since we don't have the ability to push subplans down to workers at > -

Re: [HACKERS] parallelize queries containing subplans

2017-02-01 Thread Robert Haas
On Tue, Jan 31, 2017 at 6:01 AM, Amit Kapila wrote: > Moved this patch to next CF. So here is what seems to be the key hunk from this patch: /* - * Since we don't have the ability to push subplans down to workers at - * present, we treat subplan references as parallel-restricted. W

Re: [HACKERS] parallelize queries containing subplans

2017-01-31 Thread Amit Kapila
On Tue, Jan 24, 2017 at 10:59 AM, Amit Kapila wrote: > On Thu, Jan 19, 2017 at 4:51 PM, Dilip Kumar wrote: >> On Thu, Jan 19, 2017 at 3:05 PM, Dilip Kumar wrote: >>> During debugging I found that subplan created for below part of the >>> query is parallel_unsafe, Is it a problem or there is some

Re: [HACKERS] parallelize queries containing subplans

2017-01-24 Thread Amit Kapila
On Tue, Jan 24, 2017 at 10:59 AM, Amit Kapila wrote: > On Thu, Jan 19, 2017 at 4:51 PM, Dilip Kumar wrote: >> On Thu, Jan 19, 2017 at 3:05 PM, Dilip Kumar wrote: >>> During debugging I found that subplan created for below part of the >>> query is parallel_unsafe, Is it a problem or there is some

Re: [HACKERS] parallelize queries containing subplans

2017-01-23 Thread Amit Kapila
On Thu, Jan 19, 2017 at 4:51 PM, Dilip Kumar wrote: > On Thu, Jan 19, 2017 at 3:05 PM, Dilip Kumar wrote: >> During debugging I found that subplan created for below part of the >> query is parallel_unsafe, Is it a problem or there is some explanation >> of why it's not parallel_safe, > > Okay, so

Re: [HACKERS] parallelize queries containing subplans

2017-01-19 Thread Kuntal Ghosh
On Thu, Jan 19, 2017 at 4:51 PM, Dilip Kumar wrote: > On Thu, Jan 19, 2017 at 3:05 PM, Dilip Kumar wrote: >> During debugging I found that subplan created for below part of the >> query is parallel_unsafe, Is it a problem or there is some explanation >> of why it's not parallel_safe, > > Okay, so

Re: [HACKERS] parallelize queries containing subplans

2017-01-19 Thread Dilip Kumar
On Thu, Jan 19, 2017 at 3:05 PM, Dilip Kumar wrote: > During debugging I found that subplan created for below part of the > query is parallel_unsafe, Is it a problem or there is some explanation > of why it's not parallel_safe, Okay, so basically we don't have any mechanism to perform parallel sc

Re: [HACKERS] parallelize queries containing subplans

2017-01-19 Thread Kuntal Ghosh
On Thu, Jan 19, 2017 at 3:19 PM, Kuntal Ghosh wrote: > Since Param is not parallel_safe till now, the SubPlan is also not > parallel_safe. This is why CTE subplans will not be pushed under > Gather. Specifically, Params which are generated using generate_new_param() are not parallel_safe. In the p

Re: [HACKERS] parallelize queries containing subplans

2017-01-19 Thread Kuntal Ghosh
On Thu, Jan 19, 2017 at 3:05 PM, Dilip Kumar wrote: > @@ -1213,6 +1216,7 @@ SS_process_ctes(PlannerInfo *root) > &splan->firstColCollation); > splan->useHashTable = false; > splan->unknownEqFalse = false; > + splan->parallel_safe = best_path->parallel_safe; > > I noticed that if path for C

Re: [HACKERS] parallelize queries containing subplans

2017-01-19 Thread Dilip Kumar
On Mon, Jan 16, 2017 at 9:13 PM, Amit Kapila wrote: > > After commit-ab1f0c8, this patch needs a rebase. Attached find > rebased version of the patch. > > Thanks to Kuntal for informing me offlist that this patch needs rebase. In this patch, I have observed some changes while creating subplan fo

Re: [HACKERS] parallelize queries containing subplans

2017-01-16 Thread Amit Kapila
On Thu, Jan 12, 2017 at 7:56 PM, Amit Kapila wrote: > On Thu, Jan 12, 2017 at 8:51 AM, Robert Haas wrote: >> On Wed, Jan 11, 2017 at 9:58 PM, Amit Kapila wrote: >>> The other alternative is to remember this information in SubPlan. We >>> can retrieve parallel_safe information from best_path and

Re: [HACKERS] parallelize queries containing subplans

2017-01-14 Thread Amit Kapila
On Fri, Jan 13, 2017 at 7:13 PM, Tom Lane wrote: > Dilip Kumar writes: >> ERROR: did not find '}' at end of input node at character 762 > I could reproduce this error with simple query like: SELECT * FROM information_schema.role_usage_grants WHERE object_type LIKE 'FOREIGN%' AND object_name IN

Re: [HACKERS] parallelize queries containing subplans

2017-01-13 Thread Tom Lane
Dilip Kumar writes: > ERROR: did not find '}' at end of input node at character 762 I've not looked at the patches, but just seeing this error message, this looks like somebody's fat-fingered the correspondence between outfuncs.c and readfuncs.c processing. regards, tom

Re: [HACKERS] parallelize queries containing subplans

2017-01-13 Thread Dilip Kumar
On Thu, Jan 12, 2017 at 7:56 PM, Amit Kapila wrote: > Okay, done that way. I have fixed the review comments raised by Dilip > as well and added the test case in attached patch. I have tested with pq_pushdown_subplan_v2.patch + pq_pushdown_correl_subplan_v1.patch sqlsmith. is reporting below err

Re: [HACKERS] parallelize queries containing subplans

2017-01-12 Thread Amit Kapila
On Thu, Jan 12, 2017 at 8:51 AM, Robert Haas wrote: > On Wed, Jan 11, 2017 at 9:58 PM, Amit Kapila wrote: >> The other alternative is to remember this information in SubPlan. We >> can retrieve parallel_safe information from best_path and can use it >> while generating SubPlan. The main reason

Re: [HACKERS] parallelize queries containing subplans

2017-01-11 Thread Dilip Kumar
On Thu, Jan 12, 2017 at 9:22 AM, Amit Kapila wrote: > Valid point, but I think we can avoid that by returning false after > foreach(..) loop. I think one improvement could be that instead of > manually checking the parallel safety of each subplan, we can > recursively call max_parallel_hazard_wal

Re: [HACKERS] parallelize queries containing subplans

2017-01-11 Thread Amit Kapila
On Tue, Jan 10, 2017 at 11:55 AM, Dilip Kumar wrote: > On Wed, Dec 28, 2016 at 11:47 AM, Amit Kapila wrote: >> Attached patch implements the above idea. This will enable >> parallelism for queries containing un-correlated subplans, an example >> of which is as follows: > > I have reviewed the pa

Re: [HACKERS] parallelize queries containing subplans

2017-01-11 Thread Robert Haas
On Wed, Jan 11, 2017 at 9:58 PM, Amit Kapila wrote: > The other alternative is to remember this information in SubPlan. We > can retrieve parallel_safe information from best_path and can use it > while generating SubPlan. The main reason for storing it in the plan > was to avoid explicitly passi

Re: [HACKERS] parallelize queries containing subplans

2017-01-11 Thread Amit Kapila
On Tue, Jan 10, 2017 at 10:55 PM, Robert Haas wrote: > On Wed, Dec 28, 2016 at 1:17 AM, Amit Kapila wrote: >> Currently, queries that have references to SubPlans or >> AlternativeSubPlans are considered parallel-restricted. I think we >> can lift this restriction in many cases especially when Su

Re: [HACKERS] parallelize queries containing subplans

2017-01-10 Thread Robert Haas
On Wed, Dec 28, 2016 at 1:17 AM, Amit Kapila wrote: > Currently, queries that have references to SubPlans or > AlternativeSubPlans are considered parallel-restricted. I think we > can lift this restriction in many cases especially when SubPlans are > parallel-safe. To make this work, we need to

Re: [HACKERS] parallelize queries containing subplans

2017-01-09 Thread Dilip Kumar
On Wed, Dec 28, 2016 at 11:47 AM, Amit Kapila wrote: > Attached patch implements the above idea. This will enable > parallelism for queries containing un-correlated subplans, an example > of which is as follows: I have reviewed the patch (pq_pushdown_subplan_v1.patch), Mostly patch looks clean t

Re: [HACKERS] parallelize queries containing subplans

2017-01-03 Thread Amit Kapila
On Wed, Dec 28, 2016 at 11:47 AM, Amit Kapila wrote: > > Now, we can further extend this to parallelize queries containing > correlated subplans like below: > > explain select * from t1 where t1.i in (select t2.i from t2 where t2.i=t1.i); > QUERY PLAN > ---

Re: [HACKERS] parallelize queries containing subplans

2016-12-28 Thread Rafia Sabih
On Wed, Dec 28, 2016 at 11:47 AM, Amit Kapila wrote: > > Currently, queries that have references to SubPlans or > AlternativeSubPlans are considered parallel-restricted. I think we > can lift this restriction in many cases especially when SubPlans are > parallel-safe. To make this work, we need