Re: [HACKERS] A GUC to prevent leader processes from running subplans?

2017-11-11 Thread Amit Kapila
_locally? If yes, then I think whatever you said is right. > I've added this to the January Commitfest. > +1 to this idea. Do you think such an option at table level can be meaningful? We have a parallel_workers as a storage option for tables, so users might want leader to pa

Re: [HACKERS] GatherMerge misses to push target list

2017-11-11 Thread Amit Kapila
as what you put into it. > Agreed. Your change looks good to me. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] parallelize queries containing initplans

2017-11-11 Thread Amit Kapila
On Sat, Nov 11, 2017 at 12:15 AM, Robert Haas wrote: > On Tue, Nov 7, 2017 at 4:45 AM, Amit Kapila wrote: >> As mentioned, changed the status of the patch in CF app. > > I spent some time reviewing this patch today and found myself still > quite uncomfortable with the fact

Re: [HACKERS] [POC] Faster processing at Gather node

2017-11-10 Thread Amit Kapila
On Fri, Nov 10, 2017 at 9:48 AM, Amit Kapila wrote: > On Fri, Nov 10, 2017 at 8:36 AM, Robert Haas wrote: >> On Thu, Nov 9, 2017 at 9:31 PM, Amit Kapila wrote: >>> Have you set force_parallel_mode=regress; before running the >>> statement? >> >> Yes, I t

Re: [HACKERS] [POC] Faster processing at Gather node

2017-11-09 Thread Amit Kapila
On Fri, Nov 10, 2017 at 8:36 AM, Robert Haas wrote: > On Thu, Nov 9, 2017 at 9:31 PM, Amit Kapila wrote: >> Have you set force_parallel_mode=regress; before running the >> statement? > > Yes, I tried that first. > >> If so, then why you need to tune other parall

Re: [HACKERS] [POC] Faster processing at Gather node

2017-11-09 Thread Amit Kapila
On Fri, Nov 10, 2017 at 12:05 AM, Robert Haas wrote: > On Thu, Nov 9, 2017 at 12:08 AM, Amit Kapila wrote: >> This change looks suspicious to me. I think here we can't use the >> tupDesc constructed from targetlist. One problem, I could see is that >> the c

Re: [HACKERS] Runtime Partition Pruning

2017-11-09 Thread Amit Kapila
ob of this patch to do anything about > that problem. > +1. I think if we really want to do something about plan choice when partitions are involved that should be done as a separate patch. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsq

Re: [HACKERS] why not parallel seq scan for slow functions

2017-11-09 Thread Amit Kapila
On Wed, Nov 8, 2017 at 6:48 PM, Robert Haas wrote: > On Wed, Nov 8, 2017 at 7:26 AM, Amit Kapila wrote: >> We do want to generate it later when there isn't inheritance involved, >> but only if there is a single rel involved (simple_rel_array_size >> <=2). The ru

Re: [HACKERS] [POC] Faster processing at Gather node

2017-11-08 Thread Amit Kapila
On Wed, Nov 8, 2017 at 1:02 AM, Andres Freund wrote: > Hi, > > On 2017-11-06 10:56:43 +0530, Amit Kapila wrote: >> On Sun, Nov 5, 2017 at 6:54 AM, Andres Freund wrote >> > On 2017-11-05 01:05:59 +0100, Robert Haas wrote: >> >> skip-gather-project-v1.patch do

Re: [HACKERS] why not parallel seq scan for slow functions

2017-11-08 Thread Amit Kapila
On Wed, Nov 8, 2017 at 4:34 PM, Robert Haas wrote: > On Tue, Nov 7, 2017 at 9:41 PM, Amit Kapila wrote: >> This is required to prohibit generating gather path for top rel in >> case of inheritence (Append node) at this place (we want to generate >> it later when scan/joi

Re: [HACKERS] why not parallel seq scan for slow functions

2017-11-07 Thread Amit Kapila
On Wed, Nov 8, 2017 at 2:51 AM, Robert Haas wrote: > On Mon, Nov 6, 2017 at 9:57 PM, Amit Kapila wrote: > >>> Also, even if inheritance is used, we might still be the >>> topmost scan/join target. >> >> Sure, but in that case, it won't generate the gathe

Re: [HACKERS] parallelize queries containing initplans

2017-11-07 Thread Amit Kapila
On Mon, Oct 30, 2017 at 9:00 AM, Amit Kapila wrote: > On Wed, Oct 11, 2017 at 9:24 PM, Robert Haas wrote: >> On Mon, Oct 9, 2017 at 5:56 AM, Amit Kapila wrote: >>> How about always returning false for PARAM_EXTERN? >> >> Yeah, I think that's what we s

Re: [HACKERS] Parallel Plans and Cost of non-filter functions

2017-11-06 Thread Amit Kapila
can productively help? You have already helped a lot by providing the use case, but feel free to ping on that thread if you find it is not moving. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To m

Re: [HACKERS] why not parallel seq scan for slow functions

2017-11-06 Thread Amit Kapila
On Mon, Nov 6, 2017 at 7:05 PM, Robert Haas wrote: > On Mon, Nov 6, 2017 at 11:20 AM, Amit Kapila wrote: >> On Mon, Nov 6, 2017 at 3:51 AM, Robert Haas wrote: >>> This looks like it's on the right track to me. I hope Tom will look >>> into it, but if he doesn

Re: [HACKERS] why not parallel seq scan for slow functions

2017-11-06 Thread Amit Kapila
On Mon, Nov 6, 2017 at 3:51 AM, Robert Haas wrote: > On Sun, Nov 5, 2017 at 12:57 AM, Amit Kapila wrote: >> Thanks for the confirmation. Find rebased patch attached. > > This looks like it's on the right track to me. I hope Tom will look > into it, but if he does

Re: [HACKERS] [POC] Faster processing at Gather node

2017-11-05 Thread Amit Kapila
st=0.00..635802.67 > rows=3375405 width=127) (actual time=0.025..649.887 rows=3373722 loops=8) > │ Filter: (l_suppkey > 5012) > │ Rows Removed by Filter: 376252 > │ Planning time: 0.076 ms > │ Execution time: 5986.171 ms > └──────── >

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-11-05 Thread Amit Kapila
On Sat, Nov 4, 2017 at 2:03 AM, Tom Lane wrote: > Michael Paquier writes: >> On Fri, Nov 3, 2017 at 1:10 AM, Amit Kapila wrote: >>> On Fri, Nov 3, 2017 at 2:54 AM, Tom Lane wrote: >>>> I've marked the CF entry closed. However, I'm not sure if we'

Re: [HACKERS] Parallel Plans and Cost of non-filter functions

2017-11-04 Thread Amit Kapila
tioned by Tom. [1] - https://www.postgresql.org/message-id/CAA4eK1%2B1H5Urm0_Wp-n5XszdLX1YXBqS_zW0f-vvWKwdh3eCJA%40mail.gmail.com -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] why not parallel seq scan for slow functions

2017-11-04 Thread Amit Kapila
On Thu, Sep 21, 2017 at 2:35 AM, Jeff Janes wrote: > On Tue, Sep 19, 2017 at 9:15 PM, Amit Kapila > wrote: >> >> On Wed, Sep 20, 2017 at 3:05 AM, Jeff Janes wrote: >> > On Tue, Sep 19, 2017 at 1:17 PM, Thomas Munro >> > wrote: >> >> &g

Re: [HACKERS] parallelize queries containing initplans

2017-11-03 Thread Amit Kapila
On Mon, Oct 30, 2017 at 10:07 AM, Robert Haas wrote: > On Mon, Oct 30, 2017 at 9:00 AM, Amit Kapila wrote: >> Now that the PARAM_EXTERN issue is fixed, I have rebased this patch. >> This patch had been switched to Ready For Committer in last CF, then >> Robert had comments w

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-11-02 Thread Amit Kapila
stgresql.org/message-id/CAA4eK1%2BTKg6ZK%3DmF14x_wf2KrmOxoMJ6z7YUK3-78acaYLwQ8Q%40mail.gmail.com [2] - https://www.postgresql.org/message-id/CAB7nPqTE4-GCaLtDh%3DJBcgUKR6B5WkvRLC-NpOqkgybi4FhHPw%40mail.gmail.com [3] - https://www.postgresql.org/message-id/CAB7nPqQBtDW43ABnWEdoHP6A2ToedzDFdpykbG

Re: [HACKERS] Parallel worker error

2017-10-30 Thread Amit Kapila
On Mon, Oct 30, 2017 at 10:04 AM, Robert Haas wrote: > On Mon, Oct 30, 2017 at 8:25 AM, Amit Kapila wrote: >> Thanks. I have closed this entry in CF app, however, I am not sure >> what is the best way to deal with the entry present in PostgreSQL 10 >> Open Items page[1]. T

Re: [HACKERS] parallelize queries containing initplans

2017-10-29 Thread Amit Kapila
On Wed, Oct 18, 2017 at 2:06 PM, tushar wrote: > On 10/11/2017 12:42 PM, tushar wrote: >> >> On 10/09/2017 03:26 PM, Amit Kapila wrote: >>> >>> I have reverted the check >>> in the attached patch. >> >> >> I have applied this patch again

Re: [HACKERS] parallelize queries containing initplans

2017-10-29 Thread Amit Kapila
On Wed, Oct 11, 2017 at 9:24 PM, Robert Haas wrote: > On Mon, Oct 9, 2017 at 5:56 AM, Amit Kapila wrote: >> How about always returning false for PARAM_EXTERN? > > Yeah, I think that's what we should do. Let's do that first as a > separate patch, which we might e

Re: [HACKERS] Parallel worker error

2017-10-29 Thread Amit Kapila
On Sun, Oct 29, 2017 at 1:15 PM, Robert Haas wrote: > On Sun, Oct 29, 2017 at 12:02 PM, Amit Kapila wrote: >> This patch no longer applies, so attached rebased patches. I have >> also created patches for v10 as we might want to backpatch the fix. >> Added t

Re: [HACKERS] Parallel safety for extern params

2017-10-29 Thread Amit Kapila
On Sun, Oct 29, 2017 at 9:55 PM, Robert Haas wrote: > On Sat, Oct 28, 2017 at 8:02 PM, Amit Kapila wrote: >> I think we need to make changes in exec_simple_recheck_plan to make >> the behavior similar to HEAD. With the attached patch, all tests >> passed with force_paralle

Re: [HACKERS] Parallel worker error

2017-10-28 Thread Amit Kapila
On Sat, Sep 9, 2017 at 7:56 AM, Amit Kapila wrote: > On Fri, Sep 8, 2017 at 3:13 PM, Robert Haas wrote: >> On Fri, Sep 8, 2017 at 1:17 AM, Amit Kapila wrote: >>> You are right. I have changed the ordering and passed OuterUserId via >>> FixedParallelState. >&g

Re: [HACKERS] Parallel safety for extern params

2017-10-28 Thread Amit Kapila
On Sat, Oct 28, 2017 at 2:02 AM, Robert Haas wrote: > On Mon, Oct 16, 2017 at 12:59 PM, Amit Kapila wrote: > When I tried back-porting the patch to v10 I discovered that the > plpgsql changes conflict heavily and that ripping them out all the way > produces regression fa

Re: [HACKERS] Re: Burst in WAL size when UUID is used as PK while full_page_writes are enabled

2017-10-27 Thread Amit Kapila
because hash indexes > don't support uniqueness. > That's true, but it hasn't been mentioned in the mail that the usage of hash index is the for primary key. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgs

Re: [HACKERS] Burst in WAL size when UUID is used as PK while full_page_writes are enabled

2017-10-27 Thread Amit Kapila
On Fri, Oct 27, 2017 at 5:36 PM, Alvaro Herrera wrote: > Amit Kapila wrote: > >> You might want to give a try with the hash index if you are planning >> to use PG10 and your queries involve equality operations. > > So, btree indexes on monotonically increasing sequences do

Re: [HACKERS] parallel.c oblivion of worker-startup failures

2017-10-27 Thread Amit Kapila
On Tue, Sep 19, 2017 at 8:47 AM, Amit Kapila wrote: > On Mon, Sep 18, 2017 at 10:00 PM, Tom Lane wrote: >> Amit Kapila writes: >>> Attached patch fixes these problems. >> >> Hmm, this patch adds a kill(notify_pid) after one call to >> ForgetBackgroundWorker,

Re: [HACKERS] Burst in WAL size when UUID is used as PK while full_page_writes are enabled

2017-10-27 Thread Amit Kapila
want to give a try with the hash index if you are planning to use PG10 and your queries involve equality operations. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscr

Re: [HACKERS] Parallel safety for extern params

2017-10-27 Thread Amit Kapila
On Mon, Oct 16, 2017 at 4:29 PM, Amit Kapila wrote: > On Sat, Oct 14, 2017 at 1:51 AM, Robert Haas wrote: > >> I think the bug is in ExecGather(Merge): it assumes that if we're in >> parallel mode, it's OK to start workers. But actually, it shouldn't >> d

Re: [HACKERS] CurTransactionContext freed before transaction COMMIT ???

2017-10-26 Thread Amit Kapila
nce that this is some of your application issues where you probably haven't used memory context as required, so probably you need to figure out a way to reproduce this issue, otherwise, it might be difficult to track down the actual cause. -- With Regards, Amit Kapila. EnterpriseDB: http

Re: [HACKERS] CurTransactionContext freed before transaction COMMIT ???

2017-10-25 Thread Amit Kapila
Context shouldn't get freed > before transaction COMMIT. >--> So what has actually happened here??? Kindly help us with > your insights! > > Can you check if CurTransactionContext is valid at that point? To see, if this problem is related to CurTransactionContext, can you try to populate the list in TopMemoryContext and see if that works. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Pluggable storage

2017-10-25 Thread Amit Kapila
On Wed, Oct 25, 2017 at 11:37 AM, Robert Haas wrote: > On Fri, Oct 20, 2017 at 5:47 AM, Amit Kapila wrote: >> I think what we need here is a way to register satisfies function >> (SnapshotSatisfiesFunc) in SnapshotData for different storage engines. > > I don't se

Re: [HACKERS] BLK_DONE state in XLogReadBufferForRedoExtended

2017-10-23 Thread Amit Kapila
On Tue, Oct 17, 2017 at 7:53 AM, Michael Paquier wrote: > On Mon, Oct 16, 2017 at 9:50 PM, Amit Kapila wrote: >> If above analysis is correct, then I think we can say that row state >> in a page will be same during recovery as it was when the original >> operation

Re: [HACKERS] [POC] Faster processing at Gather node

2017-10-23 Thread Amit Kapila
milar where the tuples are sent till there is a space in shared memory queue and then turn to batching the tuples using local queues. [1] - https://www.postgresql.org/message-id/CAOGQiiNiMhq5Pg3LiYxjfi2B9eAQ_q5YjS%3DfHiBJmbSOF74aBQ%40mail.gmail.com -- With Regards, Amit Kapila. EnterpriseDB:

Re: [HACKERS] [POC] Faster processing at Gather node

2017-10-23 Thread Amit Kapila
the structural issues, instead opting to > address them by adding another layer of queuing. > That is done to use batching the tuples while sending them. Sure, we can do some of the other things as well, but I think the main advantage is from batching the tuples in a smart way while sending them

Re: [HACKERS] Pluggable storage

2017-10-19 Thread Amit Kapila
lanQual mechanism and then later used during the scan. We need to make it pluggable in some way so that other heaps can work. We also need some work for EvalPlanQualFetchRowMarks as that also seems to be tightly coupled with HeapTuple. -- With Regards, Amit Kapila. EnterpriseDB: http://www.en

Re: [HACKERS] Pluggable storage

2017-10-19 Thread Amit Kapila
eap it will do >> heap_hot_search_buffer, and for zheap it will walk the undo chain and >> return the relevant tuple from the chain. > > > OK, Understood. > I will check along these lines and come up with storage API's. > I think what we need here is a

Re: [HACKERS] BLK_DONE state in XLogReadBufferForRedoExtended

2017-10-16 Thread Amit Kapila
On Fri, Oct 13, 2017 at 11:57 AM, Amit Kapila wrote: > On Fri, Oct 13, 2017 at 10:32 AM, Michael Paquier > wrote: >> On Thu, Oct 12, 2017 at 10:47 PM, Amit Kapila >> wrote: >>> Today, I was trying to think about cases when we can return BLK_DONE >>> in XLogR

Re: [HACKERS] PATCH: enabling parallel execution for cursors explicitly (experimental)

2017-10-16 Thread Amit Kapila
> One idea could be that whenever someone uses Parallel option, we can fetch and store all the data locally before returning control to the user (something like WITH HOLD option). -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing lis

Re: [HACKERS] Parallel safety for extern params

2017-10-16 Thread Amit Kapila
On Sat, Oct 14, 2017 at 1:51 AM, Robert Haas wrote: > On Fri, Oct 13, 2017 at 1:19 AM, Amit Kapila wrote: >> After fixing this problem, when I ran the regression tests with >> force_parallel_mode = regress, I saw multiple other failures. All the >> failures boil down

Re: [HACKERS] BLK_DONE state in XLogReadBufferForRedoExtended

2017-10-12 Thread Amit Kapila
On Fri, Oct 13, 2017 at 10:32 AM, Michael Paquier wrote: > On Thu, Oct 12, 2017 at 10:47 PM, Amit Kapila wrote: >> Today, I was trying to think about cases when we can return BLK_DONE >> in XLogReadBufferForRedoExtended. One thing that occurred to me is >> that it can hap

Re: [HACKERS] parallelize queries containing initplans

2017-10-12 Thread Amit Kapila
On Wed, Oct 11, 2017 at 9:24 PM, Robert Haas wrote: > On Mon, Oct 9, 2017 at 5:56 AM, Amit Kapila wrote: >> How about always returning false for PARAM_EXTERN? > > Yeah, I think that's what we should do. Let's do that first as a > separate patch, which we might e

[HACKERS] Parallel safety for extern params

2017-10-12 Thread Amit Kapila
on_v1.patch fixes this problem. Thanks to Kuntal who has helped in investigating the regression failures which happened as a result of making param extern params as parallel-safe. [1] - https://www.postgresql.org/message-id/CA%2BTgmobSN66o2_c76rY12JvS_sZa17zpKvpuyG-QzRvVLYE8Vg%40mail.gmail.com -

[HACKERS] BLK_DONE state in XLogReadBufferForRedoExtended

2017-10-12 Thread Amit Kapila
am missing? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Parallel Append implementation

2017-10-09 Thread Amit Kapila
On Fri, Oct 6, 2017 at 12:03 PM, Amit Khandekar wrote: > On 6 October 2017 at 08:49, Amit Kapila wrote: >> >> Okay, but why not cheapest partial path? > > I gave some thought on this point. Overall I feel it does not matter > which partial path it should pick up. RIght n

Re: [HACKERS] parallelize queries containing initplans

2017-10-09 Thread Amit Kapila
On Sat, Oct 7, 2017 at 7:22 PM, Robert Haas wrote: > On Fri, Oct 6, 2017 at 7:08 AM, Amit Kapila wrote: >> I have fixed the other review comment related to using safe_param_list >> in the attached patch. I think I have fixed all comments given by >> you, but let me

Re: [HACKERS] parallelize queries containing initplans

2017-10-06 Thread Amit Kapila
On Fri, Oct 6, 2017 at 2:32 AM, Robert Haas wrote: > On Thu, Oct 5, 2017 at 1:16 PM, Amit Kapila wrote: >> On Thu, Oct 5, 2017 at 6:08 PM, Robert Haas wrote: >>> On Thu, Oct 5, 2017 at 5:52 AM, Amit Kapila wrote: >>>> Now, unless, I am missing something

Re: [HACKERS] Parallel Append implementation

2017-10-05 Thread Amit Kapila
to add a comment in above part of patch or you just left /**/ mistakenly. > > > > Regarding a mix of partial and non-partial paths, I feel it always > makes sense for the leader to choose the partial path. > Okay, but why not cheapest partial path? -- With Regards, Amit

Re: [HACKERS] parallelize queries containing initplans

2017-10-05 Thread Amit Kapila
On Thu, Oct 5, 2017 at 6:08 PM, Robert Haas wrote: > On Thu, Oct 5, 2017 at 5:52 AM, Amit Kapila wrote: >> Now, unless, I am missing something here, it won't be possible to >> detect params in such cases during forming of join rels and hence we >> need the tests in gene

Re: [HACKERS] Parallel Append implementation

2017-10-05 Thread Amit Kapila
On Thu, Oct 5, 2017 at 6:14 PM, Robert Haas wrote: > On Thu, Oct 5, 2017 at 6:29 AM, Amit Kapila wrote: >> Okay, but can't we try to pick the cheapest partial path for master >> backend and maybe master backend can try to work on a partial path >> which is alread

Re: [HACKERS] Parallel Append implementation

2017-10-05 Thread Amit Kapila
On Mon, Oct 2, 2017 at 6:21 PM, Robert Haas wrote: > On Sun, Oct 1, 2017 at 9:55 AM, Amit Kapila wrote: >> Isn't it for both? I mean it is about comparing the non-partial paths >> for child relations of the same relation and also when there are >> different relation

Re: [HACKERS] parallelize queries containing initplans

2017-10-05 Thread Amit Kapila
On Wed, Oct 4, 2017 at 12:55 PM, Amit Kapila wrote: > On Wed, Oct 4, 2017 at 3:40 AM, Robert Haas wrote: >> On Tue, Oct 3, 2017 at 7:33 AM, Amit Kapila wrote: >> >> Having said all that, I think that this patch only wants to handle the >> subset of cases (2) and (4)

Re: [HACKERS] parallelize queries containing initplans

2017-10-04 Thread Amit Kapila
On Wed, Oct 4, 2017 at 12:55 PM, Amit Kapila wrote: > On Wed, Oct 4, 2017 at 3:40 AM, Robert Haas wrote: >> On Tue, Oct 3, 2017 at 7:33 AM, Amit Kapila wrote: >> >> Having said all that, I think that this patch only wants to handle the >> subset of cases (2) and (4)

Re: [HACKERS] parallelize queries containing initplans

2017-10-04 Thread Amit Kapila
On Wed, Oct 4, 2017 at 3:40 AM, Robert Haas wrote: > On Tue, Oct 3, 2017 at 7:33 AM, Amit Kapila wrote: > > Having said all that, I think that this patch only wants to handle the > subset of cases (2) and (4) where the relevant InitPlan is attached > ABOVE the Gather node -- w

Re: [HACKERS] parallelize queries containing initplans

2017-10-03 Thread Amit Kapila
On Mon, Oct 2, 2017 at 8:13 PM, Robert Haas wrote: > On Thu, Sep 14, 2017 at 10:45 PM, Amit Kapila wrote: >> The latest patch again needs to be rebased. Find rebased patch >> attached with this email. > > I read through this patch this morning. Copying Tom in the hopes

Re: [HACKERS] [sqlsmith] crash in RestoreLibraryState during low-memory testing

2017-10-02 Thread Amit Kapila
On Tue, Oct 3, 2017 at 8:31 AM, Amit Kapila wrote: > On Tue, Oct 3, 2017 at 3:04 AM, Andreas Seltenreich > wrote: >> Hi, >> >> doing low-memory testing with REL_10_STABLE at 1f19550a87 also produced >> a couple of parallel worker core dumps with the backtrace

Re: [HACKERS] [sqlsmith] crash in RestoreLibraryState during low-memory testing

2017-10-02 Thread Amit Kapila
y some other way)? I think without that we shouldn't try to load anything in the parallel worker. Also, if you can get the failed query (check in server log), it would be great. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers m

Re: [HACKERS] Parallel Append implementation

2017-10-01 Thread Amit Kapila
On Sat, Sep 30, 2017 at 9:25 PM, Robert Haas wrote: > On Sat, Sep 30, 2017 at 12:20 AM, Amit Kapila wrote: >> Okay, but the point is whether it will make any difference >> practically. Let us try to see with an example, consider there are >> two children (just taking two f

Re: [HACKERS] Parallel Append implementation

2017-09-30 Thread Amit Kapila
On Wed, Sep 20, 2017 at 10:59 AM, Amit Khandekar wrote: > On 16 September 2017 at 10:42, Amit Kapila wrote: >> >> At a broader level, the idea is good, but I think it won't turn out >> exactly like that considering your below paragraph which indicates >> that it i

Re: [HACKERS] Parallel Append implementation

2017-09-29 Thread Amit Kapila
On Sat, Sep 30, 2017 at 4:02 AM, Robert Haas wrote: > On Fri, Sep 29, 2017 at 7:48 AM, Amit Kapila wrote: >> I think in general the non-partial paths should be cheaper as compared >> to partial paths as that is the reason planner choose not to make a >> partial plan at fir

Re: [HACKERS] Parallel Append implementation

2017-09-29 Thread Amit Kapila
On Wed, Sep 20, 2017 at 10:59 AM, Amit Khandekar wrote: > On 16 September 2017 at 10:42, Amit Kapila wrote: >> On Thu, Sep 14, 2017 at 9:41 PM, Robert Haas wrote: >>> On Mon, Sep 11, 2017 at 9:25 AM, Amit Kapila >>> wrote: >>>> I think the patch stor

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-25 Thread Amit Kapila
On Mon, Sep 25, 2017 at 12:18 PM, Amit Langote wrote: > Hi. > > Trying to catch up. > > On 2017/09/25 13:43, Michael Paquier wrote: >> On Sun, Sep 24, 2017 at 2:25 PM, Amit Kapila wrote: >>> Added and updated the comments for both btree and hash index patches. >

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-24 Thread Amit Kapila
On Mon, Sep 25, 2017 at 10:13 AM, Michael Paquier wrote: > On Sun, Sep 24, 2017 at 2:25 PM, Amit Kapila wrote: >> Added and updated the comments for both btree and hash index patches. > > I don't have real complaints about this patch, this looks fine to me. > > +* C

Re: [HACKERS] Page Scan Mode in Hash Index

2017-09-24 Thread Amit Kapila
guarantee that vacuum won't delete any items.). That part of the comment has been written to indicate that we have to check LSN in this function unconditonally. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com typo_hash_readme_v1.patch Description: Binary data --

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-23 Thread Amit Kapila
On Wed, Sep 20, 2017 at 9:19 AM, Amit Kapila wrote: > On Wed, Sep 20, 2017 at 4:25 AM, Michael Paquier > wrote: >> On Wed, Sep 20, 2017 at 12:47 AM, Tom Lane wrote: >>> Amit Kapila writes: >>>> On Tue, Sep 19, 2017 at 9:27 AM, Michael Paquier >>>>

Re: [HACKERS] hash index on unlogged tables doesn't behave as expected

2017-09-21 Thread Amit Kapila
t;> discouraged"))); > > Using !RelationUsesLocalBuffers instead fixes that and the > attached patch is for 9.6. I'm a bit unconfident on the usage of > logical meaning of the macro but what it does fits there. > I think giving an error message like "ha

Re: [HACKERS] Page Scan Mode in Hash Index

2017-09-20 Thread Amit Kapila
On Wed, Sep 20, 2017 at 6:44 PM, Robert Haas wrote: > On Wed, Sep 20, 2017 at 7:45 AM, Amit Kapila wrote: >> Right, I was thinking from the perspective of the index entry. Before >> marking index entry as dead, we do check for heaptid. So, as heaptid >> can't be reuse

Re: [HACKERS] Page Scan Mode in Hash Index

2017-09-20 Thread Amit Kapila
On Wed, Sep 20, 2017 at 4:56 PM, Robert Haas wrote: > On Wed, Sep 20, 2017 at 7:19 AM, Amit Kapila wrote: >>> Page-at-a-time index vacuum as in _hash_vacuum_one_page doesn't matter >>> because such an operation doesn't allow TIDs to be reused. >> >> P

Re: [HACKERS] Page Scan Mode in Hash Index

2017-09-20 Thread Amit Kapila
On Wed, Sep 20, 2017 at 4:04 PM, Robert Haas wrote: > On Tue, Sep 19, 2017 at 11:34 PM, Amit Kapila wrote: >> This point has been discussed above [1] and to avoid this problem we >> are keeping the scan always behind vacuum for unlogged and temporary >> tables as we are doi

Re: [HACKERS] why not parallel seq scan for slow functions

2017-09-19 Thread Amit Kapila
On Wed, Sep 20, 2017 at 3:05 AM, Jeff Janes wrote: > On Tue, Sep 19, 2017 at 1:17 PM, Thomas Munro > wrote: >> >> On Thu, Sep 14, 2017 at 3:19 PM, Amit Kapila >> wrote: >> > The attached patch fixes both the review comments as discussed above. > > > th

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Amit Kapila
alRepWorker which is similar to bgworker worker machinery (which in turn can also avoid all the housekeeping for those variables) if we have access to BackgroundWorkerHandle? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-19 Thread Amit Kapila
On Wed, Sep 20, 2017 at 4:25 AM, Michael Paquier wrote: > On Wed, Sep 20, 2017 at 12:47 AM, Tom Lane wrote: >> Amit Kapila writes: >>> On Tue, Sep 19, 2017 at 9:27 AM, Michael Paquier >>> wrote: >>>> I am not saying that no index AMs take advantage FPW co

Re: [HACKERS] Page Scan Mode in Hash Index

2017-09-19 Thread Amit Kapila
are going to mark as dead. This has been taken care in patch 0003. I understand that this is slightly ugly, but the other alternative (as mentioned in the email [1]) is much worse. [1] - https://www.postgresql.org/message-id/CAA4eK1J6xiJUOidBaOt0iPsAdS0%2Bp5PoKFf1R2yVjTwrY_4s

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Amit Kapila
On Tue, Sep 19, 2017 at 6:51 PM, Petr Jelinek wrote: > On 19/09/17 15:08, Amit Kapila wrote: >> >> I am not much aware of this area. Can you explain what other usages >> it has apart from in the process that has launched the worker and in >> worker itself? >> &g

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Amit Kapila
On Tue, Sep 19, 2017 at 3:34 PM, Petr Jelinek wrote: > n 18/09/17 18:42, Tom Lane wrote: >> Amit Kapila writes: >>> On Mon, Sep 18, 2017 at 7:46 PM, Tom Lane wrote: >>>> The subscriber log includes >>>> 2017-09-18 08:43:08.240 UTC [15672] WARN

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-19 Thread Amit Kapila
s well is adding an assertion in > XLogRegisterBuffer & friends so as the combination of REGBUF_STANDARD > and REGBUF_NO_IMAGE is forbidden. That's bugging me as well. > Is that related to this patch? If not, then maybe we can discuss it in a separate thread. -- With Regards

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-19 Thread Amit Kapila
On Tue, Sep 19, 2017 at 9:27 AM, Michael Paquier wrote: > On Tue, Sep 19, 2017 at 12:40 PM, Amit Kapila wrote: >> On Mon, Sep 18, 2017 at 4:03 PM, Michael Paquier >> wrote: >>> On Mon, Sep 18, 2017 at 11:16 AM, Amit Kapila >>> wrote: >>>> On

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Amit Kapila
On Tue, Sep 19, 2017 at 6:29 PM, Petr Jelinek wrote: > On 19/09/17 14:33, Amit Kapila wrote: >> On Tue, Sep 19, 2017 at 3:34 PM, Petr Jelinek >> wrote: >>> n 18/09/17 18:42, Tom Lane wrote: >>> >>>> So, frankly, I think we would be best off losin

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Amit Kapila
n? >> > > I am not quite sure I understand this question, we need to store > additional info about workers in shared memory so we need slots for that. > Yeah, but you could have used the way we do for parallel query where we setup dsm and share all such information. You can check the

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-18 Thread Amit Kapila
On Mon, Sep 18, 2017 at 4:03 PM, Michael Paquier wrote: > On Mon, Sep 18, 2017 at 11:16 AM, Amit Kapila wrote: >> On Sat, Sep 16, 2017 at 7:15 PM, Michael Paquier >> wrote: >>> >> >> You have already noticed above that it will help when >> wal_checking

Re: [HACKERS] parallel.c oblivion of worker-startup failures

2017-09-18 Thread Amit Kapila
On Mon, Sep 18, 2017 at 10:00 PM, Tom Lane wrote: > Amit Kapila writes: >> Attached patch fixes these problems. > > Hmm, this patch adds a kill(notify_pid) after one call to > ForgetBackgroundWorker, but the postmaster has several more such calls. > Shouldn't they

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-18 Thread Amit Kapila
us of the worker which won't give the correct status in case of fork failure. The patch I have posted has a fix for the issue, however, this could be an entirely different issue altogether as it appears from your next email in this thread. [1] - https://www.postgresql.org/message

Re: [HACKERS] GatherMerge misses to push target list

2017-09-18 Thread Amit Kapila
the updated patch - where added test-case clean up. > oops, missed dropping the function. Thanks for the review. [1] - https://commitfest.postgresql.org/15/1293/ -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@po

[HACKERS] parallel.c oblivion of worker-startup failures

2017-09-17 Thread Amit Kapila
s? [1] - https://www.postgresql.org/message-id/4905.1492813...@sss.pgh.pa.us -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com fix_worker_startup_failures_v1.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your sub

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-17 Thread Amit Kapila
you. If your only worry about these patches is that you want this sentence to be removed from the comment because you think it is obvious or doesn't make much sense, then I think we can leave this decision to committer. I have added it based on Tom's suggestion above thread about exp

Re: [HACKERS] Parallel Append implementation

2017-09-15 Thread Amit Kapila
On Thu, Sep 14, 2017 at 8:30 PM, Amit Khandekar wrote: > On 11 September 2017 at 18:55, Amit Kapila wrote: >>> >> >> How? See, if you have four partial subpaths and two non-partial >> subpaths, then for parallel-aware append it considers all six paths in &g

Re: [HACKERS] Parallel Append implementation

2017-09-15 Thread Amit Kapila
On Thu, Sep 14, 2017 at 9:41 PM, Robert Haas wrote: > On Mon, Sep 11, 2017 at 9:25 AM, Amit Kapila wrote: >> I think the patch stores only non-partial paths in decreasing order, >> what if partial paths having more costs follows those paths? > > The general picture here is t

Re: [HACKERS] parallelize queries containing initplans

2017-09-14 Thread Amit Kapila
On Thu, Aug 31, 2017 at 11:23 AM, Amit Kapila wrote: > On Mon, Aug 21, 2017 at 2:40 PM, Amit Kapila wrote: >> On Mon, Aug 21, 2017 at 1:44 PM, Haribabu Kommi >> wrote: >>> >>> >>> Thanks for adding more details. It is easy to understand. >>> &

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-14 Thread Amit Kapila
ce for btree or hash? Any index that is upgraded should have pd_lower set, do you have any case in mind where it won't be set? For hash, if someone upgrades from a version lower than 9.6, it might not have set, but we already give warning to reindex the hash indexes upgraded from a version lo

Re: [HACKERS] GatherMerge misses to push target list

2017-09-13 Thread Amit Kapila
On Wed, Sep 13, 2017 at 5:30 PM, Rushabh Lathia wrote: > On Wed, Sep 6, 2017 at 10:04 AM, Amit Kapila > wrote: >> > > > This seems like a good optimization. I tried to simulate the test given > in the mail, initially wasn't able to generate the exact test - as index

Re: [HACKERS] why not parallel seq scan for slow functions

2017-09-13 Thread Amit Kapila
On Wed, Sep 13, 2017 at 9:39 AM, Amit Kapila wrote: > On Tue, Sep 12, 2017 at 5:47 PM, Amit Khandekar > wrote: >> On 5 September 2017 at 14:04, Amit Kapila wrote: >> >> I started with a quick review ... a couple of comments below : >> >> - * If this is a bas

Re: [HACKERS] why not parallel seq scan for slow functions

2017-09-12 Thread Amit Kapila
On Tue, Sep 12, 2017 at 5:47 PM, Amit Khandekar wrote: > On 5 September 2017 at 14:04, Amit Kapila wrote: > > I started with a quick review ... a couple of comments below : > > - * If this is a baserel, consider gathering any partial paths we may have > - * created for it

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-12 Thread Amit Kapila
reference to meta page buffer and when that is modified (ex. in revmap_physical_extend), then also I think you need to consider using REGBUF_STNADARD flag. > > Did I miss something from the discussion? > I think one point which might be missed is that the patch needs to modify pd_lower

Re: [HACKERS] GatherMerge misses to push target list

2017-09-11 Thread Amit Kapila
On Wed, Sep 6, 2017 at 10:04 AM, Amit Kapila wrote: > During my recent work on costing of parallel paths [1], I noticed that > we are missing to push target list below GatherMerge in some simple > cases like below. > I think this should be considered as a bug-fix for 10.0, but it d

Re: [HACKERS] Parallel Append implementation

2017-09-11 Thread Amit Kapila
On Mon, Sep 11, 2017 at 4:49 PM, Amit Khandekar wrote: > On 8 September 2017 at 19:17, Amit Kapila wrote: >> >> In that case, why can't we keep the workers also process in same >> order, what is the harm in that? > > Because of the way the logic of queuing works,

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-09-11 Thread Amit Kapila
On Mon, Sep 11, 2017 at 12:43 PM, Michael Paquier wrote: > On Mon, Sep 11, 2017 at 4:07 PM, Amit Kapila wrote: >> I have prepared separate patches for hash and btree index. I think >> for another type of indexes, it is better to first fix the pd_lower >> issue. > > J

  1   2   3   4   5   6   7   8   9   10   >