Re: [HACKERS] A GUC to prevent leader processes from running subplans?

2017-11-11 Thread Amit Kapila
as a storage option for tables, so users might want leader to participate in parallelism only for some of the tables. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] GatherMerge misses to push target list

2017-11-11 Thread Amit Kapila
hatever Gather Merge > produces as output is the same as what you put into it. > Agreed. Your change looks good to me. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] parallelize queries containing initplans

2017-11-11 Thread Amit Kapila
On Sat, Nov 11, 2017 at 12:15 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Tue, Nov 7, 2017 at 4:45 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> As mentioned, changed the status of the patch in CF app. > > I spent some time reviewing this patch today and

Re: [HACKERS] [POC] Faster processing at Gather node

2017-11-10 Thread Amit Kapila
On Fri, Nov 10, 2017 at 9:48 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Fri, Nov 10, 2017 at 8:36 AM, Robert Haas <robertmh...@gmail.com> wrote: >> On Thu, Nov 9, 2017 at 9:31 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >>> Have you set force_p

Re: [HACKERS] [POC] Faster processing at Gather node

2017-11-09 Thread Amit Kapila
On Fri, Nov 10, 2017 at 8:36 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Thu, Nov 9, 2017 at 9:31 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> Have you set force_parallel_mode=regress; before running the >> statement? > > Yes, I tried that fir

Re: [HACKERS] [POC] Faster processing at Gather node

2017-11-09 Thread Amit Kapila
On Fri, Nov 10, 2017 at 12:05 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Thu, Nov 9, 2017 at 12:08 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> This change looks suspicious to me. I think here we can't use the >> tupDesc constructed from targetlist

Re: [HACKERS] Runtime Partition Pruning

2017-11-09 Thread Amit Kapila
ng on it. > > I don't think it's really the job of this patch to do anything about > that problem. > +1. I think if we really want to do something about plan choice when partitions are involved that should be done as a separate patch. -- With Regards, Amit Kapila. EnterpriseDB: htt

Re: [HACKERS] why not parallel seq scan for slow functions

2017-11-09 Thread Amit Kapila
On Wed, Nov 8, 2017 at 6:48 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Wed, Nov 8, 2017 at 7:26 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> We do want to generate it later when there isn't inheritance involved, >> but only if there is a single rel invo

Re: [HACKERS] [POC] Faster processing at Gather node

2017-11-08 Thread Amit Kapila
On Wed, Nov 8, 2017 at 1:02 AM, Andres Freund <and...@anarazel.de> wrote: > Hi, > > On 2017-11-06 10:56:43 +0530, Amit Kapila wrote: >> On Sun, Nov 5, 2017 at 6:54 AM, Andres Freund <and...@anarazel.de> wrote >> > On 2017-11-05 01:05:59 +0100, Robert Haas wrote

Re: [HACKERS] why not parallel seq scan for slow functions

2017-11-08 Thread Amit Kapila
On Wed, Nov 8, 2017 at 4:34 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Tue, Nov 7, 2017 at 9:41 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> This is required to prohibit generating gather path for top rel in >> case of inheritence (Append node) at this

Re: [HACKERS] why not parallel seq scan for slow functions

2017-11-07 Thread Amit Kapila
On Wed, Nov 8, 2017 at 2:51 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Nov 6, 2017 at 9:57 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: > >>> Also, even if inheritance is used, we might still be the >>> topmost scan/join target. >> &g

Re: [HACKERS] parallelize queries containing initplans

2017-11-07 Thread Amit Kapila
On Mon, Oct 30, 2017 at 9:00 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Wed, Oct 11, 2017 at 9:24 PM, Robert Haas <robertmh...@gmail.com> wrote: >> On Mon, Oct 9, 2017 at 5:56 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >>> How about alw

Re: [HACKERS] Parallel Plans and Cost of non-filter functions

2017-11-06 Thread Amit Kapila
core! Is there any way I > can productively help? You have already helped a lot by providing the use case, but feel free to ping on that thread if you find it is not moving. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsq

Re: [HACKERS] why not parallel seq scan for slow functions

2017-11-06 Thread Amit Kapila
On Mon, Nov 6, 2017 at 7:05 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Nov 6, 2017 at 11:20 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> On Mon, Nov 6, 2017 at 3:51 AM, Robert Haas <robertmh...@gmail.com> wrote: >>> This looks like it's o

Re: [HACKERS] why not parallel seq scan for slow functions

2017-11-06 Thread Amit Kapila
On Mon, Nov 6, 2017 at 3:51 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Sun, Nov 5, 2017 at 12:57 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> Thanks for the confirmation. Find rebased patch attached. > > This looks like it's on the right track t

Re: [HACKERS] [POC] Faster processing at Gather node

2017-11-05 Thread Amit Kapila
al time=0.025..649.887 rows=3373722 loops=8) > │ Filter: (l_suppkey > 5012) > │ Rows Removed by Filter: 376252 > │ Planning time: 0.076 ms > │ Execution time: 5986.171 ms > └ > (9 rows) > > so there cl

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-11-05 Thread Amit Kapila
On Sat, Nov 4, 2017 at 2:03 AM, Tom Lane <t...@sss.pgh.pa.us> wrote: > Michael Paquier <michael.paqu...@gmail.com> writes: >> On Fri, Nov 3, 2017 at 1:10 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >>> On Fri, Nov 3, 2017 at 2:54 AM, Tom Lane <t...

Re: [HACKERS] Parallel Plans and Cost of non-filter functions

2017-11-04 Thread Amit Kapila
h the recent patch [1] posted on the thread mentioned by Tom. [1] - https://www.postgresql.org/message-id/CAA4eK1%2B1H5Urm0_Wp-n5XszdLX1YXBqS_zW0f-vvWKwdh3eCJA%40mail.gmail.com -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing lis

Re: [HACKERS] why not parallel seq scan for slow functions

2017-11-04 Thread Amit Kapila
On Thu, Sep 21, 2017 at 2:35 AM, Jeff Janes <jeff.ja...@gmail.com> wrote: > On Tue, Sep 19, 2017 at 9:15 PM, Amit Kapila <amit.kapil...@gmail.com> > wrote: >> >> On Wed, Sep 20, 2017 at 3:05 AM, Jeff Janes <jeff.ja...@gmail.com> wrote: >> >

Re: [HACKERS] parallelize queries containing initplans

2017-11-03 Thread Amit Kapila
On Mon, Oct 30, 2017 at 10:07 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Oct 30, 2017 at 9:00 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> Now that the PARAM_EXTERN issue is fixed, I have rebased this patch. >> This patch had been switched to Rea

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-11-02 Thread Amit Kapila
B7nPqQBtDW43ABnWEdoHP6A2ToedzDFdpykbGjpO2wuZNiQnw%40mail.gmail.com -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com change_metapage_usage_btree-v3.patch Description: Binary data change_metapage_usage_hash-v3.patch Description: Binary data -- Sent via pgsql-hackers mailing list (

Re: [HACKERS] Parallel worker error

2017-10-30 Thread Amit Kapila
On Mon, Oct 30, 2017 at 10:04 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Oct 30, 2017 at 8:25 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> Thanks. I have closed this entry in CF app, however, I am not sure >> what is the best way to deal with the

Re: [HACKERS] parallelize queries containing initplans

2017-10-29 Thread Amit Kapila
On Wed, Oct 18, 2017 at 2:06 PM, tushar <tushar.ah...@enterprisedb.com> wrote: > On 10/11/2017 12:42 PM, tushar wrote: >> >> On 10/09/2017 03:26 PM, Amit Kapila wrote: >>> >>> I have reverted the check >>> in the attached patch. >> >> &

Re: [HACKERS] parallelize queries containing initplans

2017-10-29 Thread Amit Kapila
On Wed, Oct 11, 2017 at 9:24 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Oct 9, 2017 at 5:56 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> How about always returning false for PARAM_EXTERN? > > Yeah, I think that's what we should do. Let's do that

Re: [HACKERS] Parallel worker error

2017-10-29 Thread Amit Kapila
On Sun, Oct 29, 2017 at 1:15 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Sun, Oct 29, 2017 at 12:02 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> This patch no longer applies, so attached rebased patches. I have >> also created patches for v10 as we migh

Re: [HACKERS] Parallel safety for extern params

2017-10-29 Thread Amit Kapila
On Sun, Oct 29, 2017 at 9:55 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Sat, Oct 28, 2017 at 8:02 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> I think we need to make changes in exec_simple_recheck_plan to make >> the behavior similar to HEAD. With th

Re: [HACKERS] Parallel worker error

2017-10-29 Thread Amit Kapila
On Sat, Sep 9, 2017 at 7:56 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Fri, Sep 8, 2017 at 3:13 PM, Robert Haas <robertmh...@gmail.com> wrote: >> On Fri, Sep 8, 2017 at 1:17 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >>> You are right. I

Re: [HACKERS] Parallel safety for extern params

2017-10-28 Thread Amit Kapila
On Sat, Oct 28, 2017 at 2:02 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Oct 16, 2017 at 12:59 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: > When I tried back-porting the patch to v10 I discovered that the > plpgsql changes conflict heavily and that ripping

Re: [HACKERS] Re: Burst in WAL size when UUID is used as PK while full_page_writes are enabled

2017-10-27 Thread Amit Kapila
dex with a hash index, because hash indexes > don't support uniqueness. > That's true, but it hasn't been mentioned in the mail that the usage of hash index is the for primary key. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailin

Re: [HACKERS] Burst in WAL size when UUID is used as PK while full_page_writes are enabled

2017-10-27 Thread Amit Kapila
On Fri, Oct 27, 2017 at 5:36 PM, Alvaro Herrera <alvhe...@2ndquadrant.com> wrote: > Amit Kapila wrote: > >> You might want to give a try with the hash index if you are planning >> to use PG10 and your queries involve equality operations. > > So, btree indexes on mono

Re: [HACKERS] parallel.c oblivion of worker-startup failures

2017-10-27 Thread Amit Kapila
On Tue, Sep 19, 2017 at 8:47 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Mon, Sep 18, 2017 at 10:00 PM, Tom Lane <t...@sss.pgh.pa.us> wrote: >> Amit Kapila <amit.kapil...@gmail.com> writes: >>> Attached patch fixes these problems. >> >> H

Re: [HACKERS] Burst in WAL size when UUID is used as PK while full_page_writes are enabled

2017-10-27 Thread Amit Kapila
in > size. > You might want to give a try with the hash index if you are planning to use PG10 and your queries involve equality operations. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql

Re: [HACKERS] Parallel safety for extern params

2017-10-27 Thread Amit Kapila
On Mon, Oct 16, 2017 at 4:29 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Sat, Oct 14, 2017 at 1:51 AM, Robert Haas <robertmh...@gmail.com> wrote: > >> I think the bug is in ExecGather(Merge): it assumes that if we're in >> parallel mode, it's OK t

Re: [HACKERS] CurTransactionContext freed before transaction COMMIT ???

2017-10-26 Thread Amit Kapila
think there is a good chance that this is some of your application issues where you probably haven't used memory context as required, so probably you need to figure out a way to reproduce this issue, otherwise, it might be difficult to track down the actual cause. -- With Regards, Amit Kapila. Ente

Re: [HACKERS] CurTransactionContext freed before transaction COMMIT ???

2017-10-25 Thread Amit Kapila
ically CurTransactionContext shouldn't get freed > before transaction COMMIT. >--> So what has actually happened here??? Kindly help us with > your insights! > > Can you check if CurTransactionContext is valid at that point? To see, if this problem is related to Cur

Re: [HACKERS] Pluggable storage

2017-10-25 Thread Amit Kapila
On Wed, Oct 25, 2017 at 11:37 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Fri, Oct 20, 2017 at 5:47 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> I think what we need here is a way to register satisfies function >> (SnapshotSatisfiesFunc) in SnapshotData fo

Re: [HACKERS] BLK_DONE state in XLogReadBufferForRedoExtended

2017-10-23 Thread Amit Kapila
On Tue, Oct 17, 2017 at 7:53 AM, Michael Paquier <michael.paqu...@gmail.com> wrote: > On Mon, Oct 16, 2017 at 9:50 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> If above analysis is correct, then I think we can say that row state >> in a page will be same dur

Re: [HACKERS] [POC] Faster processing at Gather node

2017-10-23 Thread Amit Kapila
the thread [1] has done something similar where the tuples are sent till there is a space in shared memory queue and then turn to batching the tuples using local queues. [1] - https://www.postgresql.org/message-id/CAOGQiiNiMhq5Pg3LiYxjfi2B9eAQ_q5YjS%3DfHiBJmbSOF74aBQ%40mail.gmail.com -

Re: [HACKERS] [POC] Faster processing at Gather node

2017-10-23 Thread Amit Kapila
ss them by adding another layer of queuing. > That is done to use batching the tuples while sending them. Sure, we can do some of the other things as well, but I think the main advantage is from batching the tuples in a smart way while sending them and once that is done, we might not need many o

Re: [HACKERS] Pluggable storage

2017-10-19 Thread Amit Kapila
to it. Currently, EState->es_epqTuple is a HeapTuple which is filled as part of EvalPlanQual mechanism and then later used during the scan. We need to make it pluggable in some way so that other heaps can work. We also need some work for EvalPlanQualFetchRowMarks as that also seems to be ti

Re: [HACKERS] Pluggable storage

2017-10-19 Thread Amit Kapila
D that are visible to >> the scan snapshot". Then for the current heap it will do >> heap_hot_search_buffer, and for zheap it will walk the undo chain and >> return the relevant tuple from the chain. > > > OK, Understood. > I will check along these lines and co

Re: [HACKERS] BLK_DONE state in XLogReadBufferForRedoExtended

2017-10-16 Thread Amit Kapila
On Fri, Oct 13, 2017 at 11:57 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Fri, Oct 13, 2017 at 10:32 AM, Michael Paquier > <michael.paqu...@gmail.com> wrote: >> On Thu, Oct 12, 2017 at 10:47 PM, Amit Kapila <amit.kapil...@gmail.com> >> wrote: >>

Re: [HACKERS] PATCH: enabling parallel execution for cursors explicitly (experimental)

2017-10-16 Thread Amit Kapila
w we could do better. > One idea could be that whenever someone uses Parallel option, we can fetch and store all the data locally before returning control to the user (something like WITH HOLD option). -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via

Re: [HACKERS] Parallel safety for extern params

2017-10-16 Thread Amit Kapila
On Sat, Oct 14, 2017 at 1:51 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Fri, Oct 13, 2017 at 1:19 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> After fixing this problem, when I ran the regression tests with >> force_parallel_mode = regress, I saw mu

Re: [HACKERS] BLK_DONE state in XLogReadBufferForRedoExtended

2017-10-13 Thread Amit Kapila
On Fri, Oct 13, 2017 at 10:32 AM, Michael Paquier <michael.paqu...@gmail.com> wrote: > On Thu, Oct 12, 2017 at 10:47 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> Today, I was trying to think about cases when we can return BLK_DONE >> in XLogReadBuffer

Re: [HACKERS] parallelize queries containing initplans

2017-10-12 Thread Amit Kapila
On Wed, Oct 11, 2017 at 9:24 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Oct 9, 2017 at 5:56 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> How about always returning false for PARAM_EXTERN? > > Yeah, I think that's what we should do. Let's do that

[HACKERS] Parallel safety for extern params

2017-10-12 Thread Amit Kapila
lem. Thanks to Kuntal who has helped in investigating the regression failures which happened as a result of making param extern params as parallel-safe. [1] - https://www.postgresql.org/message-id/CA%2BTgmobSN66o2_c76rY12JvS_sZa17zpKvpuyG-QzRvVLYE8Vg%40mail.gmail.com -- With Regards, Amit Kapila.

[HACKERS] BLK_DONE state in XLogReadBufferForRedoExtended

2017-10-12 Thread Amit Kapila
am missing? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Parallel Append implementation

2017-10-09 Thread Amit Kapila
On Fri, Oct 6, 2017 at 12:03 PM, Amit Khandekar <amitdkhan...@gmail.com> wrote: > On 6 October 2017 at 08:49, Amit Kapila <amit.kapil...@gmail.com> wrote: >> >> Okay, but why not cheapest partial path? > > I gave some thought on this point. Overall I feel it doe

Re: [HACKERS] parallelize queries containing initplans

2017-10-09 Thread Amit Kapila
On Sat, Oct 7, 2017 at 7:22 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Fri, Oct 6, 2017 at 7:08 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> I have fixed the other review comment related to using safe_param_list >> in the attached patch. I think I ha

Re: [HACKERS] parallelize queries containing initplans

2017-10-06 Thread Amit Kapila
On Fri, Oct 6, 2017 at 2:32 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Thu, Oct 5, 2017 at 1:16 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> On Thu, Oct 5, 2017 at 6:08 PM, Robert Haas <robertmh...@gmail.com> wrote: >>> On Thu, Oct 5, 2017

Re: [HACKERS] Parallel Append implementation

2017-10-05 Thread Amit Kapila
); + } It seems either you want to add a comment in above part of patch or you just left /**/ mistakenly. > > > > Regarding a mix of partial and non-partial paths, I feel it always > makes sense for the leader to choose the partial path. > Okay, but why not cheapest partia

Re: [HACKERS] parallelize queries containing initplans

2017-10-05 Thread Amit Kapila
On Thu, Oct 5, 2017 at 6:08 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Thu, Oct 5, 2017 at 5:52 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> Now, unless, I am missing something here, it won't be possible to >> detect params in such cases during formin

Re: [HACKERS] Parallel Append implementation

2017-10-05 Thread Amit Kapila
On Thu, Oct 5, 2017 at 6:14 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Thu, Oct 5, 2017 at 6:29 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> Okay, but can't we try to pick the cheapest partial path for master >> backend and maybe master backend can tr

Re: [HACKERS] Parallel Append implementation

2017-10-05 Thread Amit Kapila
On Mon, Oct 2, 2017 at 6:21 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Sun, Oct 1, 2017 at 9:55 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> Isn't it for both? I mean it is about comparing the non-partial paths >> for child relations of the

Re: [HACKERS] parallelize queries containing initplans

2017-10-05 Thread Amit Kapila
On Wed, Oct 4, 2017 at 12:55 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Wed, Oct 4, 2017 at 3:40 AM, Robert Haas <robertmh...@gmail.com> wrote: >> On Tue, Oct 3, 2017 at 7:33 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> >> Having said all t

Re: [HACKERS] parallelize queries containing initplans

2017-10-04 Thread Amit Kapila
On Wed, Oct 4, 2017 at 12:55 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Wed, Oct 4, 2017 at 3:40 AM, Robert Haas <robertmh...@gmail.com> wrote: >> On Tue, Oct 3, 2017 at 7:33 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> >> Having said all t

Re: [HACKERS] parallelize queries containing initplans

2017-10-04 Thread Amit Kapila
On Wed, Oct 4, 2017 at 3:40 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Tue, Oct 3, 2017 at 7:33 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > > Having said all that, I think that this patch only wants to handle the > subset of cases (2) and (4) where the relev

Re: [HACKERS] parallelize queries containing initplans

2017-10-03 Thread Amit Kapila
On Mon, Oct 2, 2017 at 8:13 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Thu, Sep 14, 2017 at 10:45 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> The latest patch again needs to be rebased. Find rebased patch >> attached with this email. > > I rea

Re: [HACKERS] [sqlsmith] crash in RestoreLibraryState during low-memory testing

2017-10-02 Thread Amit Kapila
On Tue, Oct 3, 2017 at 8:31 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Tue, Oct 3, 2017 at 3:04 AM, Andreas Seltenreich <seltenre...@gmx.de> > wrote: >> Hi, >> >> doing low-memory testing with REL_10_STABLE at 1f19550a87 also produced >&

Re: [HACKERS] [sqlsmith] crash in RestoreLibraryState during low-memory testing

2017-10-02 Thread Amit Kapila
ng shared_preload_libraries or by some other way)? I think without that we shouldn't try to load anything in the parallel worker. Also, if you can get the failed query (check in server log), it would be great. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-

Re: [HACKERS] Parallel Append implementation

2017-10-01 Thread Amit Kapila
On Sat, Sep 30, 2017 at 9:25 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Sat, Sep 30, 2017 at 12:20 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> Okay, but the point is whether it will make any difference >> practically. Let us try to see with an example

Re: [HACKERS] Parallel Append implementation

2017-09-30 Thread Amit Kapila
On Wed, Sep 20, 2017 at 10:59 AM, Amit Khandekar <amitdkhan...@gmail.com> wrote: > On 16 September 2017 at 10:42, Amit Kapila <amit.kapil...@gmail.com> wrote: >> >> At a broader level, the idea is good, but I think it won't turn out >> exactly like that consi

Re: [HACKERS] Parallel Append implementation

2017-09-29 Thread Amit Kapila
On Sat, Sep 30, 2017 at 4:02 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Fri, Sep 29, 2017 at 7:48 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> I think in general the non-partial paths should be cheaper as compared >> to partial paths as that is the reason

Re: [HACKERS] Parallel Append implementation

2017-09-29 Thread Amit Kapila
On Wed, Sep 20, 2017 at 10:59 AM, Amit Khandekar <amitdkhan...@gmail.com> wrote: > On 16 September 2017 at 10:42, Amit Kapila <amit.kapil...@gmail.com> wrote: >> On Thu, Sep 14, 2017 at 9:41 PM, Robert Haas <robertmh...@gmail.com> wrote: >>> On Mon, Sep 11, 2017

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-25 Thread Amit Kapila
On Mon, Sep 25, 2017 at 12:18 PM, Amit Langote <langote_amit...@lab.ntt.co.jp> wrote: > Hi. > > Trying to catch up. > > On 2017/09/25 13:43, Michael Paquier wrote: >> On Sun, Sep 24, 2017 at 2:25 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >>> Add

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-24 Thread Amit Kapila
On Mon, Sep 25, 2017 at 10:13 AM, Michael Paquier <michael.paqu...@gmail.com> wrote: > On Sun, Sep 24, 2017 at 2:25 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> Added and updated the comments for both btree and hash index patches. > > I don't have real

Re: [HACKERS] Page Scan Mode in Hash Index

2017-09-24 Thread Amit Kapila
ving pin on the overflow page doesn't guarantee that vacuum won't delete any items.). That part of the comment has been written to indicate that we have to check LSN in this function unconditonally. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com typo_hash_readme_v

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-23 Thread Amit Kapila
On Wed, Sep 20, 2017 at 9:19 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Wed, Sep 20, 2017 at 4:25 AM, Michael Paquier > <michael.paqu...@gmail.com> wrote: >> On Wed, Sep 20, 2017 at 12:47 AM, Tom Lane <t...@sss.pgh.pa.us> wrote: >>> Amit

Re: [HACKERS] hash index on unlogged tables doesn't behave as expected

2017-09-21 Thread Amit Kapila
indexes are not WAL-logged and their use is >> discouraged"))); > > Using !RelationUsesLocalBuffers instead fixes that and the > attached patch is for 9.6. I'm a bit unconfident on the usage of > logical meaning of the macro but what it does fits there. > I think giving an error

Re: [HACKERS] Page Scan Mode in Hash Index

2017-09-20 Thread Amit Kapila
On Wed, Sep 20, 2017 at 6:44 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Wed, Sep 20, 2017 at 7:45 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> Right, I was thinking from the perspective of the index entry. Before >> marking index entry as dead, w

Re: [HACKERS] Page Scan Mode in Hash Index

2017-09-20 Thread Amit Kapila
On Wed, Sep 20, 2017 at 4:56 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Wed, Sep 20, 2017 at 7:19 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >>> Page-at-a-time index vacuum as in _hash_vacuum_one_page doesn't matter >>> because such an operatio

Re: [HACKERS] Page Scan Mode in Hash Index

2017-09-20 Thread Amit Kapila
On Wed, Sep 20, 2017 at 4:04 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Tue, Sep 19, 2017 at 11:34 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> This point has been discussed above [1] and to avoid this problem we >> are keeping the scan always

Re: [HACKERS] why not parallel seq scan for slow functions

2017-09-19 Thread Amit Kapila
On Wed, Sep 20, 2017 at 3:05 AM, Jeff Janes <jeff.ja...@gmail.com> wrote: > On Tue, Sep 19, 2017 at 1:17 PM, Thomas Munro > <thomas.mu...@enterprisedb.com> wrote: >> >> On Thu, Sep 14, 2017 at 3:19 PM, Amit Kapila <amit.kapil...@gmail.com> >> wrote: >

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Amit Kapila
n we avoid maintaining additional information (in_use, generation,..) in LogicalRepWorker which is similar to bgworker worker machinery (which in turn can also avoid all the housekeeping for those variables) if we have access to BackgroundWorkerHandle? -- With Regards, Amit Kapila. EnterpriseDB: http://www

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-19 Thread Amit Kapila
On Wed, Sep 20, 2017 at 4:25 AM, Michael Paquier <michael.paqu...@gmail.com> wrote: > On Wed, Sep 20, 2017 at 12:47 AM, Tom Lane <t...@sss.pgh.pa.us> wrote: >> Amit Kapila <amit.kapil...@gmail.com> writes: >>> On Tue, Sep 19, 2017 at 9:27 AM, Michael Paquier &

Re: [HACKERS] Page Scan Mode in Hash Index

2017-09-19 Thread Amit Kapila
iPsAdS0%2Bp5PoKFf1R2yVjTwrY_4snA%40mail.gmail.com -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Amit Kapila
On Tue, Sep 19, 2017 at 6:51 PM, Petr Jelinek <petr.jeli...@2ndquadrant.com> wrote: > On 19/09/17 15:08, Amit Kapila wrote: >> >> I am not much aware of this area. Can you explain what other usages >> it has apart from in the process that has launched the wor

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Amit Kapila
On Tue, Sep 19, 2017 at 3:34 PM, Petr Jelinek <petr.jeli...@2ndquadrant.com> wrote: > n 18/09/17 18:42, Tom Lane wrote: >> Amit Kapila <amit.kapil...@gmail.com> writes: >>> On Mon, Sep 18, 2017 at 7:46 PM, Tom Lane <t...@sss.pgh.pa.us> wrote: >>>> T

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-19 Thread Amit Kapila
ppropriately. > > Another thing that we could consider as well is adding an assertion in > XLogRegisterBuffer & friends so as the combination of REGBUF_STANDARD > and REGBUF_NO_IMAGE is forbidden. That's bugging me as well. > Is that related to this patch? If not, then may

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-19 Thread Amit Kapila
On Tue, Sep 19, 2017 at 9:27 AM, Michael Paquier <michael.paqu...@gmail.com> wrote: > On Tue, Sep 19, 2017 at 12:40 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> On Mon, Sep 18, 2017 at 4:03 PM, Michael Paquier >> <michael.paqu...@gmail.com> wrote: >>&

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Amit Kapila
On Tue, Sep 19, 2017 at 6:29 PM, Petr Jelinek <petr.jeli...@2ndquadrant.com> wrote: > On 19/09/17 14:33, Amit Kapila wrote: >> On Tue, Sep 19, 2017 at 3:34 PM, Petr Jelinek >> <petr.jeli...@2ndquadrant.com> wrote: >>> n 18/09/17 18:42, Tom Lane wrote: >>&g

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Amit Kapila
ormation. You can check the logic of execparallel.c and parallel.c to see how we do all such stuff for parallel query. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-18 Thread Amit Kapila
On Mon, Sep 18, 2017 at 4:03 PM, Michael Paquier <michael.paqu...@gmail.com> wrote: > On Mon, Sep 18, 2017 at 11:16 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> On Sat, Sep 16, 2017 at 7:15 PM, Michael Paquier >> <michael.paqu...@gmail.com> wrote: >>

Re: [HACKERS] parallel.c oblivion of worker-startup failures

2017-09-18 Thread Amit Kapila
On Mon, Sep 18, 2017 at 10:00 PM, Tom Lane <t...@sss.pgh.pa.us> wrote: > Amit Kapila <amit.kapil...@gmail.com> writes: >> Attached patch fixes these problems. > > Hmm, this patch adds a kill(notify_pid) after one call to > ForgetBackgroundWorker, but the postmast

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-18 Thread Amit Kapila
//www.postgresql.org/message-id/CAA4eK1KDfKkvrjxsKJi3WPyceVi3dH1VCkbTJji2fuwKuB%3D3uw%40mail.gmail.com -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] GatherMerge misses to push target list

2017-09-18 Thread Amit Kapila
ter in CF app > Here is the updated patch - where added test-case clean up. > oops, missed dropping the function. Thanks for the review. [1] - https://commitfest.postgresql.org/15/1293/ -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers

[HACKERS] parallel.c oblivion of worker-startup failures

2017-09-17 Thread Amit Kapila
://www.postgresql.org/message-id/4905.1492813...@sss.pgh.pa.us -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com fix_worker_startup_failures_v1.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-17 Thread Amit Kapila
ommitter. I have added it based on Tom's suggestion above thread about explaining why it is inessential or essential to set pd_lower. I think Amit Langote just tried to mimic what I have done in hash and btree patches to maintain consistency. I am also not very sure if we should write some detailed comment or leave the existing comment as it is. I think it is just a matter of different perspective. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Parallel Append implementation

2017-09-16 Thread Amit Kapila
On Thu, Sep 14, 2017 at 8:30 PM, Amit Khandekar <amitdkhan...@gmail.com> wrote: > On 11 September 2017 at 18:55, Amit Kapila <amit.kapil...@gmail.com> wrote: >>> >> >> How? See, if you have four partial subpaths and two non-partial >> subpaths, then for

Re: [HACKERS] Parallel Append implementation

2017-09-15 Thread Amit Kapila
On Thu, Sep 14, 2017 at 9:41 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Sep 11, 2017 at 9:25 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> I think the patch stores only non-partial paths in decreasing order, >> what if partial paths having mor

Re: [HACKERS] parallelize queries containing initplans

2017-09-14 Thread Amit Kapila
On Thu, Aug 31, 2017 at 11:23 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Mon, Aug 21, 2017 at 2:40 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> On Mon, Aug 21, 2017 at 1:44 PM, Haribabu Kommi >> <kommi.harib...@gmail.com> wrote: >>&

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-14 Thread Amit Kapila
PG10. > Why do we need to change metapage at every place for btree or hash? Any index that is upgraded should have pd_lower set, do you have any case in mind where it won't be set? For hash, if someone upgrades from a version lower than 9.6, it might not have set, but we already give warning

Re: [HACKERS] GatherMerge misses to push target list

2017-09-13 Thread Amit Kapila
On Wed, Sep 13, 2017 at 5:30 PM, Rushabh Lathia <rushabh.lat...@gmail.com> wrote: > On Wed, Sep 6, 2017 at 10:04 AM, Amit Kapila <amit.kapil...@gmail.com> > wrote: >> > > > This seems like a good optimization. I tried to simulate the test given > in the mai

Re: [HACKERS] why not parallel seq scan for slow functions

2017-09-13 Thread Amit Kapila
On Wed, Sep 13, 2017 at 9:39 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Tue, Sep 12, 2017 at 5:47 PM, Amit Khandekar <amitdkhan...@gmail.com> > wrote: >> On 5 September 2017 at 14:04, Amit Kapila <amit.kapil...@gmail.com> wrote: >> >>

Re: [HACKERS] why not parallel seq scan for slow functions

2017-09-12 Thread Amit Kapila
On Tue, Sep 12, 2017 at 5:47 PM, Amit Khandekar <amitdkhan...@gmail.com> wrote: > On 5 September 2017 at 14:04, Amit Kapila <amit.kapil...@gmail.com> wrote: > > I started with a quick review ... a couple of comments below : > > - * If this is a baserel, consider gatheri

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-12 Thread Amit Kapila
o contains a reference to meta page buffer and when that is modified (ex. in revmap_physical_extend), then also I think you need to consider using REGBUF_STNADARD flag. > > Did I miss something from the discussion? > I think one point which might be missed is that the patch needs to mod

Re: [HACKERS] GatherMerge misses to push target list

2017-09-11 Thread Amit Kapila
On Wed, Sep 6, 2017 at 10:04 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > During my recent work on costing of parallel paths [1], I noticed that > we are missing to push target list below GatherMerge in some simple > cases like below. > I think this should be consi

Re: [HACKERS] Parallel Append implementation

2017-09-11 Thread Amit Kapila
On Mon, Sep 11, 2017 at 4:49 PM, Amit Khandekar <amitdkhan...@gmail.com> wrote: > On 8 September 2017 at 19:17, Amit Kapila <amit.kapil...@gmail.com> wrote: >> >> In that case, why can't we keep the workers also process in same >> order, what is the harm in that?

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-11 Thread Amit Kapila
On Mon, Sep 11, 2017 at 12:43 PM, Michael Paquier <michael.paqu...@gmail.com> wrote: > On Mon, Sep 11, 2017 at 4:07 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: >> I have prepared separate patches for hash and btree index. I think >> for another type of indexes

  1   2   3   4   5   6   7   8   9   10   >