Re: Parallel Inserts in CREATE TABLE AS

On Sat, May 29, 2021 at 9:46 AM Amit Kapila wrote: > While looking at results, I have observed one more thing that we are > trying to parallelize I/O due to which we might not be seeing benefit > in such cases. I think even for non-write queries there won't be any > (much) benefit if we can't

Re: Parallel Inserts in CREATE TABLE AS

On Fri, May 28, 2021 at 8:53 AM Amit Kapila wrote: > > On Thu, May 27, 2021 at 7:37 PM Bharath Rupireddy > > > > I captured below information with the attached patch > > 0001-test-times-and-block-counts.patch applied on top of CTAS v23 > > patch set. Testing details are attached in the file named

RE: Parallel Inserts in CREATE TABLE AS

From: Bharath Rupireddy Sent: Thursday, May 27, 2021 10:07 PM > On Thu, May 27, 2021 at 9:53 AM Bharath Rupireddy > wrote: > > > One idea to find this out could be that we have three counters for > > > each worker which counts the number of times each worker extended > > > the relation in bulk,

Re: Parallel Inserts in CREATE TABLE AS

On Fri, May 28, 2021 at 6:24 AM tsunakawa.ta...@fujitsu.com wrote: > > From: Bharath Rupireddy > > I'm still not sure why the execution time with 0 workers (or serial > > execution or > > no parallelism involved) on my testing system is 112 sec compared to 58 sec > > on > > Hou-San's system

Re: Parallel Inserts in CREATE TABLE AS

On Thu, May 27, 2021 at 7:37 PM Bharath Rupireddy wrote: > > On Thu, May 27, 2021 at 9:53 AM Bharath Rupireddy > wrote: > > > One idea to find this out could be that we have three counters for > > > each worker which counts the number of times each worker extended the > > > relation in bulk, the

RE: Parallel Inserts in CREATE TABLE AS

From: Tsunakawa, Takayuki/綱川 貴之 Sent: Friday, May 28, 2021 8:55 AM > To: 'Bharath Rupireddy' ; Hou, > Zhijie/侯 志杰 > Cc: Amit Kapila ; Tang, Haiying/唐 海英 > ; PostgreSQL-development > ; Zhihong Yu ; Luc > Vlaming ; Dilip Kumar ; > vignesh C > Subject: RE: Parallel

RE: Parallel Inserts in CREATE TABLE AS

From: Bharath Rupireddy > I'm still not sure why the execution time with 0 workers (or serial execution > or > no parallelism involved) on my testing system is 112 sec compared to 58 sec on > Hou-San's system for the same use case. Maybe the testing system I'm using > is not of the latest

Re: Parallel Inserts in CREATE TABLE AS

On Thu, May 27, 2021 at 9:53 AM Bharath Rupireddy wrote: > > One idea to find this out could be that we have three counters for > > each worker which counts the number of times each worker extended the > > relation in bulk, the number of times each worker extended the > > relation by one block,

RE: Parallel Inserts in CREATE TABLE AS

From: Bharath Rupireddy Sent: Thursday, May 27, 2021 3:41 PM > > On Thu, May 27, 2021 at 1:03 PM tsunakawa.ta...@fujitsu.com > wrote: > > > > From: houzj.f...@fujitsu.com > > > Although, the 4 workers case still has performance degradation > > > compared to serial case. > > > > > > SERIAL:

Re: Parallel Inserts in CREATE TABLE AS

On Thu, May 27, 2021 at 2:26 PM Amit Kapila wrote: > > I think some other cause of contention on relation extension locks are > > 1. CTAS is using a buffer strategy and due to that, it might need to > > evict out the buffer frequently for getting the new block in. Maybe > > we can identify by

Re: Parallel Inserts in CREATE TABLE AS

On Thu, May 27, 2021 at 10:27 AM Dilip Kumar wrote: > > On Thu, May 27, 2021 at 10:16 AM Bharath Rupireddy > wrote: > > > > On Thu, May 27, 2021 at 7:12 AM houzj.f...@fujitsu.com > > wrote: > > > I am afraid that the using the FSM seems not get a stable performance > > > gain(at least on my

RE: Parallel Inserts in CREATE TABLE AS

From: Bharath Rupireddy > I think we can discuss this in a separate thread and see what other > hackers think. OK, unless we won't get stuck in the current direction. (Our goal is to not degrade in performance, but to outperform serial execution, isn't it?) > If the idea is to give the user

Re: Parallel Inserts in CREATE TABLE AS

On Thu, May 27, 2021 at 1:03 PM tsunakawa.ta...@fujitsu.com wrote: > > From: houzj.f...@fujitsu.com > > Although, the 4 workers case still has performance degradation compared to > > serial case. > > > > SERIAL: 58759.213 ms > > PARALLEL 2 WORKER [NOT SKIP FSM]: 68390.221 ms [SKIP FSM]: > >

RE: Parallel Inserts in CREATE TABLE AS

From: houzj.f...@fujitsu.com > Although, the 4 workers case still has performance degradation compared to > serial case. > > SERIAL: 58759.213 ms > PARALLEL 2 WORKER [NOT SKIP FSM]: 68390.221 ms [SKIP FSM]: > 58633.924 ms > PARALLEL 4 WORKER [NOT SKIP FSM]: 67448.142 ms [SKIP FSM]: >

Re: Parallel Inserts in CREATE TABLE AS

On Thu, May 27, 2021 at 12:46 PM tsunakawa.ta...@fujitsu.com wrote: > > From: Dilip Kumar > Basically you are creating a new table and loading data to it and that means > you will be less likely to access those data soon so for such thing spoiling > buffer cache may not be a good idea. >

RE: Parallel Inserts in CREATE TABLE AS

From: Dilip Kumar Basically you are creating a new table and loading data to it and that means you will be less likely to access those data soon so for such thing spoiling buffer cache may not be a good idea. -- Some people, including me, would

RE: Parallel Inserts in CREATE TABLE AS

From: Bharath Rupireddy Sent: Thursday, May 27, 2021 2:59 PM > On Thu, May 27, 2021 at 12:19 PM houzj.f...@fujitsu.com > wrote: > > BTW, I checked my test results, I was testing INSERT INTO unlogged table. > > What do you mean by "testing INSERT INTO"? Is it that you are testing the > timings

Re: Parallel Inserts in CREATE TABLE AS

On Thu, May 27, 2021 at 12:19 PM houzj.f...@fujitsu.com wrote: > BTW, I checked my test results, I was testing INSERT INTO unlogged table. What do you mean by "testing INSERT INTO"? Is it that you are testing the timings for parallel inserts in INSERT INTO ... SELECT command? If so, why should

RE: Parallel Inserts in CREATE TABLE AS

From: Bharath Rupireddy Sent: Thursday, May 27, 2021 12:46 PM > On Thu, May 27, 2021 at 7:12 AM houzj.f...@fujitsu.com > wrote: > > I am afraid that the using the FSM seems not get a stable performance > > gain(at least on my machine), I will take a deep look into this to > > figure out the

Re: Parallel Inserts in CREATE TABLE AS

On Thu, 27 May 2021 at 11:32 AM, tsunakawa.ta...@fujitsu.com < tsunakawa.ta...@fujitsu.com> wrote: > From: Dilip Kumar > > I think some other cause of contention on relation extension locks are > > 1. CTAS is using a buffer strategy and due to that, it might need to > > evict out the buffer

RE: Parallel Inserts in CREATE TABLE AS

From: Dilip Kumar > I think some other cause of contention on relation extension locks are > 1. CTAS is using a buffer strategy and due to that, it might need to > evict out the buffer frequently for getting the new block in. Maybe > we can identify by turning off the buffer strategy for CTAS

Re: Parallel Inserts in CREATE TABLE AS

On Thu, May 27, 2021 at 10:16 AM Bharath Rupireddy wrote: > > On Thu, May 27, 2021 at 7:12 AM houzj.f...@fujitsu.com > wrote: > > I am afraid that the using the FSM seems not get a stable performance > > gain(at least on my machine), > > I will take a deep look into this to figure out the

Re: Parallel Inserts in CREATE TABLE AS

On Thu, May 27, 2021 at 7:12 AM houzj.f...@fujitsu.com wrote: > I am afraid that the using the FSM seems not get a stable performance gain(at > least on my machine), > I will take a deep look into this to figure out the difference. A naive idea > it that the benefit that bulk extension > bring

Re: Parallel Inserts in CREATE TABLE AS

On Thu, May 27, 2021 at 9:43 AM Amit Kapila wrote: > > I have read it but I think we should try to ensure practically what is > > happening because it is possible that first time worker checked in FSM > > without taking relation extension lock, it didn't find any free page, > > and then when it

Re: Parallel Inserts in CREATE TABLE AS

On Wed, May 26, 2021 at 5:51 PM Amit Kapila wrote: > > On Wed, May 26, 2021 at 5:28 PM Bharath Rupireddy > wrote: > > > > On Fri, May 21, 2021 at 3:46 PM Amit Kapila wrote: > > > > > > On Fri, Mar 19, 2021 at 11:02 AM Bharath Rupireddy > > > wrote: > > > > > > > > > The other possibility could

Re: Parallel Inserts in CREATE TABLE AS

On Thu, May 27, 2021 at 7:12 AM houzj.f...@fujitsu.com wrote: > I followed your above test steps and the below configuration, but my test > results are a little different from yours. > I am not sure the exact reason, maybe because of the hardware.. > > Test INSERT 1000 rows((2 bigint(of 8

RE: Parallel Inserts in CREATE TABLE AS

Thank you for the detailed analysis, I'll look into it too. (The times have changed...) From: Bharath Rupireddy > Well, one might think to add more blocks at a time, say > Min(1024, lockWaiters * 128/256/512) than currently extraBlocks = > Min(512, lockWaiters * 20);. This will work (i.e. we

RE: Parallel Inserts in CREATE TABLE AS

From: Bharath Rupireddy Sent: Wednesday, May 26, 2021 7:22 PM > Thanks for trying that out. > > Please see the code around the use_fsm flag in RelationGetBufferForTuple for > more understanding of the points below. > > What happens if FSM is skipped i.e. myState->ti_options = >

Re: Parallel Inserts in CREATE TABLE AS

On Wed, May 26, 2021 at 5:28 PM Bharath Rupireddy wrote: > > On Fri, May 21, 2021 at 3:46 PM Amit Kapila wrote: > > > > On Fri, Mar 19, 2021 at 11:02 AM Bharath Rupireddy > > wrote: > > > > > > The other possibility could > > be that the free pages added to FSM by one worker are not being used

Re: Parallel Inserts in CREATE TABLE AS

On Fri, May 21, 2021 at 3:46 PM Amit Kapila wrote: > > On Fri, Mar 19, 2021 at 11:02 AM Bharath Rupireddy > wrote: > > > > On Wed, Jan 27, 2021 at 1:47 PM Bharath Rupireddy > > wrote: > > > > > > > I analyzed performance of parallel inserts in CTAS for different cases > > with tuple size

Re: Parallel Inserts in CREATE TABLE AS

On Tue, May 25, 2021 at 1:50 PM tsunakawa.ta...@fujitsu.com wrote: > > From: houzj.f...@fujitsu.com > > + /* > > + * We don't need to skip contacting FSM while inserting tuples > > for > > + * parallel mode, while extending the relations, workers > > instead

Re: Parallel Inserts in CREATE TABLE AS

On Tue, May 25, 2021 at 1:10 PM tsunakawa.ta...@fujitsu.com wrote: > > Although this should be a controversial and may be crazy idea, the following > change brought 4-11% speedup. This is because I thought parallel workers > might contend for WAL flush as a result of them using the limited

Re: Parallel Inserts in CREATE TABLE AS

On Tue, May 25, 2021 at 12:05 PM houzj.f...@fujitsu.com wrote: > > I noticed one place which could be one of the reasons that cause the > performance degradation. > > + /* > +* We don't need to skip contacting FSM while inserting > tuples for > +*

RE: Parallel Inserts in CREATE TABLE AS

From: houzj.f...@fujitsu.com > + /* > + * We don't need to skip contacting FSM while inserting tuples > for > + * parallel mode, while extending the relations, workers > instead of > + * blocking on a page while another worker is inserting, can >

RE: Parallel Inserts in CREATE TABLE AS

Bharath-san, all, Hmm, I didn't experience performance degradation on my poor-man's Linux VM (4 CPU, 4 GB RAM, HDD)... [benchmark preparation] autovacuum = off shared_buffers = 1GB checkpoint_timeout = 1h max_wal_size = 8GB min_wal_size = 8GB (other settings to enable parallelism) CREATE

RE: Parallel Inserts in CREATE TABLE AS

Hi Bharath-san, From: Bharath Rupireddy Sent: Friday, May 21, 2021 6:49 PM > > On Fri, May 21, 2021 at 3:46 PM Amit Kapila wrote: > > > > On Fri, Mar 19, 2021 at 11:02 AM Bharath Rupireddy > > wrote: > > > > > > On Wed, Jan 27, 2021 at 1:47 PM Bharath Rupireddy > > > wrote: > > > > > > > > >

Re: Parallel Inserts in CREATE TABLE AS

On Fri, May 21, 2021 at 3:46 PM Amit Kapila wrote: > > On Fri, Mar 19, 2021 at 11:02 AM Bharath Rupireddy > wrote: > > > > On Wed, Jan 27, 2021 at 1:47 PM Bharath Rupireddy > > wrote: > > > > > > > I analyzed performance of parallel inserts in CTAS for different cases > > with tuple size

Re: Parallel Inserts in CREATE TABLE AS

On Fri, Mar 19, 2021 at 11:02 AM Bharath Rupireddy wrote: > > On Wed, Jan 27, 2021 at 1:47 PM Bharath Rupireddy > wrote: > > > > I analyzed performance of parallel inserts in CTAS for different cases > with tuple size 32bytes, 59bytes, 241bytes and 1064bytes. We could > gain if the tuple sizes

Re: Parallel Inserts in CREATE TABLE AS

On Fri, Mar 19, 2021 at 4:33 PM Bharath Rupireddy wrote: > > In an offlist discussion with Robert and Dilip, using fallocate to > extend the relation may help to extend the relation faster. In regards > to this, it looks like the AIO/DIO patch set of Andres [1] which > involves using fallocate()

RE: Parallel Inserts in CREATE TABLE AS

> > They are pretty simple though. I think someone can also check if the > > same regression exists for parallel inserts in "INSERT INTO SELECT" > > patch set as well for larger tuple sizes. > > Thanks for reminding. > I did some performance tests for parallel inserts in " INSERT INTO SELECT " >

RE: Parallel Inserts in CREATE TABLE AS

> They are pretty simple though. I think someone can also check if the same > regression exists for parallel inserts in "INSERT INTO SELECT" > patch set as well for larger tuple sizes. Thanks for reminding. I did some performance tests for parallel inserts in " INSERT INTO SELECT " with the

Re: Parallel Inserts in CREATE TABLE AS

On Fri, Mar 19, 2021 at 12:45 PM tanghy.f...@fujitsu.com wrote: > > From: Bharath Rupireddy > >I analyzed performance of parallel inserts in CTAS for different cases > >with tuple size 32bytes, 59bytes, 241bytes and 1064bytes. We could > >gain if the tuple sizes are lower. But if the tuple size

RE: Parallel Inserts in CREATE TABLE AS

From: Bharath Rupireddy >I analyzed performance of parallel inserts in CTAS for different cases >with tuple size 32bytes, 59bytes, 241bytes and 1064bytes. We could >gain if the tuple sizes are lower. But if the tuple size is larger >i..e 1064bytes, there's a regression with parallel inserts.

Re: Parallel Inserts in CREATE TABLE AS

On Wed, Jan 27, 2021 at 1:47 PM Bharath Rupireddy wrote: > > On Wed, Jan 27, 2021 at 1:25 PM Tang, Haiying > wrote: > > I choose 5 cases which pick parallel insert plan in CTAS to measure the > > patched performance. Each case run 30 times. > > > > Most of the tests execution become faster with

RE: Parallel Inserts in CREATE TABLE AS

> > Seems like v22 patch was failing in cfbot for one of the unstable test > > cases. > > Attaching v23 patch set with modification in 0003 and 0004 patches. No > > changes to 0001 and 0002 patches. Hopefully cfbot will be happy with v23. > > > > Please consider v23 for further review. > Hi, >

RE: Parallel Inserts in CREATE TABLE AS

> Seems like v22 patch was failing in cfbot for one of the unstable test cases. > Attaching v23 patch set with modification in 0003 and 0004 patches. No > changes to 0001 and 0002 patches. Hopefully cfbot will be happy with v23. > > Please consider v23 for further review. Hi, I was looking into

Re: Parallel Inserts in CREATE TABLE AS

On Wed, Jan 27, 2021 at 1:25 PM Tang, Haiying wrote: > I choose 5 cases which pick parallel insert plan in CTAS to measure the > patched performance. Each case run 30 times. > > Most of the tests execution become faster with this patch. > > However, Test NO 4(create table xxx as table xxx.)

RE: Parallel Inserts in CREATE TABLE AS

Hi Bharath, I choose 5 cases which pick parallel insert plan in CTAS to measure the patched performance. Each case run 30 times. Most of the tests execution become faster with this patch. However, Test NO 4(create table xxx as table xxx.) appears performance degradation. I tested various

RE: Parallel Inserts in CREATE TABLE AS

>Thanks a lot for the tests. In your test case, parallel insertions are not >being picked because the Gather node has > some projections(floor(((random() * '1'::double precision) + >'1'::double > precision)) to perform. That's expected. >Whenever parallel insertions are chosen for CTAS, we

Re: Parallel Inserts in CREATE TABLE AS

On Fri, Jan 22, 2021 at 5:16 PM Tang, Haiying wrote: > > Hi Bharath, > > I'm trying to take some performance measurements on you patch v23. > But when I started, I found an issue about the tuples unbalance distribution among workers(99% tuples read by one worker) under specified case which lead

RE: Parallel Inserts in CREATE TABLE AS

Hi Bharath, I'm trying to take some performance measurements on you patch v23. But when I started, I found an issue about the tuples unbalance distribution among workers(99% tuples read by one worker) under specified case which lead the "parallel select" part makes no performance gain. Then I

Re: Parallel Inserts in CREATE TABLE AS

On 06-01-2021 09:32, Bharath Rupireddy wrote: On Tue, Jan 5, 2021 at 1:25 PM Luc Vlaming wrote: wrt v18-0002patch: It looks like this introduces a state machine that goes like: - starts at CTAS_PARALLEL_INS_UNDEF - possibly moves to CTAS_PARALLEL_INS_SELECT -

RE: Parallel Inserts in CREATE TABLE AS

> Attaching v21 patch set, which has following changes: > 1) 0001 - changed fpes->ins_cmd_type == > PARALLEL_INSERT_CMD_CREATE_TABLE_AS to fpes->ins_cmd_type != > PARALLEL_INSERT_CMD_UNDEF > 2) 0002 - reworded the commit message. > 3) 0003 - added cmin, xmin test case to one of the parallel insert

Re: Parallel Inserts in CREATE TABLE AS

Thanks for the clarification. w.r.t. the commit message, maybe I was momentarily sidetracked by the phrase: With this change. It seems the scenarios you listed are known parallel safety constraints. Probably rephrase that sentence a little bit to make this clearer. Cheers On Wed, Jan 6, 2021

Re: Parallel Inserts in CREATE TABLE AS

On Thu, Jan 7, 2021 at 5:12 AM Zhihong Yu wrote: > > Hi, > For v20-0002-Tuple-Cost-Adjustment-for-Parallel-Inserts-in-CTAS.patch : > > workers to Gather node to 0. With this change, there are chances > that the planner may choose the parallel plan. > > It would be nice if the scenarios where a

Re: Parallel Inserts in CREATE TABLE AS

Hi, For v20-0002-Tuple-Cost-Adjustment-for-Parallel-Inserts-in-CTAS.patch : workers to Gather node to 0. With this change, there are chances that the planner may choose the parallel plan. It would be nice if the scenarios where parallel plan is not chosen are listed. + if

Re: Parallel Inserts in CREATE TABLE AS

On Tue, Jan 5, 2021 at 1:25 PM Luc Vlaming wrote: > wrt v18-0002patch: > > It looks like this introduces a state machine that goes like: > - starts at CTAS_PARALLEL_INS_UNDEF > - possibly moves to CTAS_PARALLEL_INS_SELECT > - CTAS_PARALLEL_INS_TUP_COST_CAN_IGN

RE: Parallel Inserts in CREATE TABLE AS

> > > > + /* Okay to parallelize inserts, so mark it. */ > > + if (ins_cmd == PARALLEL_INSERT_CMD_CREATE_TABLE_AS) > > + ((DR_intorel *) dest)->is_parallel = true; > > > > + if (ins_cmd == PARALLEL_INSERT_CMD_CREATE_TABLE_AS) > > +

Re: Parallel Inserts in CREATE TABLE AS

On 05-01-2021 13:57, Bharath Rupireddy wrote: On Tue, Jan 5, 2021 at 1:00 PM Luc Vlaming wrote: Reviewing further v20-0001: I would still opt for moving the code for the parallel worker into a separate function, and then setting rStartup of the dest receiver to that function in

Re: Parallel Inserts in CREATE TABLE AS

On Wed, Jan 6, 2021 at 11:30 AM Hou, Zhijie wrote: > > > > I think it makes sense. > > > > > > And if the check about ' ins_cmd == xxx1 || ins_cmd == xxx2' may be > > > used in some places, How about define a generic function with some comment > > to mention the purpose. > > > > > > An example in

Re: Parallel Inserts in CREATE TABLE AS

On Wed, Jan 6, 2021 at 11:26 AM Bharath Rupireddy wrote: > > On Wed, Jan 6, 2021 at 10:17 AM Dilip Kumar wrote: > > > > On Wed, Jan 6, 2021 at 9:23 AM Bharath Rupireddy > > wrote: > > > > > > > +/* > > + * List the commands here for which parallel insertions are possible. > > + */ > > +typedef

RE: Parallel Inserts in CREATE TABLE AS

> > I think it makes sense. > > > > And if the check about ' ins_cmd == xxx1 || ins_cmd == xxx2' may be > > used in some places, How about define a generic function with some comment > to mention the purpose. > > > > An example in INSERT INTO SELECT patch: > > +/* > > + *

Re: Parallel Inserts in CREATE TABLE AS

On Wed, Jan 6, 2021 at 10:17 AM Dilip Kumar wrote: > > On Wed, Jan 6, 2021 at 9:23 AM Bharath Rupireddy > wrote: > > > > +/* > + * List the commands here for which parallel insertions are possible. > + */ > +typedef enum ParallelInsertCmdKind > +{ > + PARALLEL_INSERT_CMD_UNDEF = 0, > +

Re: Parallel Inserts in CREATE TABLE AS

On Wed, Jan 6, 2021 at 11:06 AM Hou, Zhijie wrote: > > > > For v20-0001-Parallel-Inserts-in-CREATE-TABLE-AS.patch : > > > > > > ParallelInsCmdEstimate : > > > > > > + Assert(pcxt && ins_info && > > > + (ins_cm

Re: Parallel Inserts in CREATE TABLE AS

On Wed, Jan 6, 2021 at 10:05 AM Zhihong Yu wrote: > > The plan sounds good. > > Before the second command type is added, can you leave out the 'if (ins_cmd > == PARALLEL_INSERT_CMD_CREATE_TABLE_AS)' and keep the pair of curlies ? > > You can add the if condition back when the second command type

RE: Parallel Inserts in CREATE TABLE AS

> > For v20-0001-Parallel-Inserts-in-CREATE-TABLE-AS.patch : > > > > ParallelInsCmdEstimate : > > > > + Assert(pcxt && ins_info && > > + (ins_cmd == PARALLEL_INSERT_CMD_CREATE_TABLE_AS)); > > + > > + if (ins_cmd == PA

Re: Parallel Inserts in CREATE TABLE AS

On Wed, Jan 6, 2021 at 9:23 AM Bharath Rupireddy wrote: > +/* + * List the commands here for which parallel insertions are possible. + */ +typedef enum ParallelInsertCmdKind +{ + PARALLEL_INSERT_CMD_UNDEF = 0, + PARALLEL_INSERT_CMD_CREATE_TABLE_AS +} ParallelInsertCmdKind; I see there is some

Re: Parallel Inserts in CREATE TABLE AS

Rupireddy < bharath.rupireddyforpostg...@gmail.com> wrote: > On Wed, Jan 6, 2021 at 8:19 AM Zhihong Yu wrote: > > For v20-0001-Parallel-Inserts-in-CREATE-TABLE-AS.patch : > > > > ParallelInsCmdEstimate : > > > > + Assert(pcxt

Re: Parallel Inserts in CREATE TABLE AS

On Wed, Jan 6, 2021 at 8:19 AM Zhihong Yu wrote: > For v20-0001-Parallel-Inserts-in-CREATE-TABLE-AS.patch : > > ParallelInsCmdEstimate : > > + Assert(pcxt && ins_info && > + (ins_cmd == PARALLEL_INSERT_CMD_CREATE_TABLE_AS)); > + > + if (ins_cm

Re: Parallel Inserts in CREATE TABLE AS

For v20-0001-Parallel-Inserts-in-CREATE-TABLE-AS.patch : ParallelInsCmdEstimate : + Assert(pcxt && ins_info && + (ins_cmd == PARALLEL_INSERT_CMD_CREATE_TABLE_AS)); + + if (ins_cmd == PARALLEL_INSERT_CMD_CREATE_TABLE_AS) Sinc the if condition is covered by the ass

Re: Parallel Inserts in CREATE TABLE AS

On Tue, Jan 5, 2021 at 1:00 PM Luc Vlaming wrote: > Reviewing further v20-0001: > > I would still opt for moving the code for the parallel worker into a > separate function, and then setting rStartup of the dest receiver to > that function in ExecParallelGetInsReceiver, as its completely >

Re: Parallel Inserts in CREATE TABLE AS

On 05-01-2021 11:32, Dilip Kumar wrote: On Tue, Jan 5, 2021 at 12:43 PM Luc Vlaming wrote: On 04-01-2021 14:32, Bharath Rupireddy wrote: On Mon, Jan 4, 2021 at 4:22 PM Luc Vlaming mailto:l...@swarm64.com>> wrote: > Sorry it took so long to get back to reviewing this. Thanks for the

Re: Parallel Inserts in CREATE TABLE AS

On Tue, Jan 5, 2021 at 12:43 PM Luc Vlaming wrote: > > On 04-01-2021 14:32, Bharath Rupireddy wrote: > > On Mon, Jan 4, 2021 at 4:22 PM Luc Vlaming > > wrote: > > > Sorry it took so long to get back to reviewing this. > > > > Thanks for the comments. > > > > > wrt

Re: Parallel Inserts in CREATE TABLE AS

On Tue, Jan 5, 2021 at 10:08 AM vignesh C wrote: > On Mon, Jan 4, 2021 at 3:07 PM Bharath Rupireddy > wrote: > > > > On Wed, Dec 30, 2020 at 5:28 PM vignesh C wrote: > > > Few comments: > > > - /* > > > -* To allow parallel inserts, we need to ensure that they are > > > safe to

Re: Parallel Inserts in CREATE TABLE AS

On 04-01-2021 14:53, Bharath Rupireddy wrote: On Mon, Jan 4, 2021 at 5:44 PM Luc Vlaming wrote: On 04-01-2021 12:16, Hou, Zhijie wrote: wrt v18-0002patch: It looks like this introduces a state machine that goes like: - starts at CTAS_PARALLEL_INS_UNDEF - possibly moves

Re: Parallel Inserts in CREATE TABLE AS

On 05-01-2021 04:59, Bharath Rupireddy wrote: On Mon, Jan 4, 2021 at 7:02 PM Bharath Rupireddy wrote: + if (IS_PARALLEL_CTAS_DEST(gstate->dest) && + ((DR_intorel *) gstate->dest)->into->rel && +

Re: Parallel Inserts in CREATE TABLE AS

On 04-01-2021 14:32, Bharath Rupireddy wrote: On Mon, Jan 4, 2021 at 4:22 PM Luc Vlaming > wrote: > Sorry it took so long to get back to reviewing this. Thanks for the comments. > wrt v18-0001patch: > > +               /* > +                * If the worker is

Re: Parallel Inserts in CREATE TABLE AS

On Mon, Jan 4, 2021 at 3:07 PM Bharath Rupireddy wrote: > > On Wed, Dec 30, 2020 at 5:28 PM vignesh C wrote: > > Few comments: > > - /* > > -* To allow parallel inserts, we need to ensure that they are safe > > to be > > -* performed in workers. We have the infrastructure

Re: Parallel Inserts in CREATE TABLE AS

On Mon, Jan 4, 2021 at 5:44 PM Luc Vlaming wrote: > On 04-01-2021 12:16, Hou, Zhijie wrote: > >> > >> wrt v18-0002patch: > >> > >> It looks like this introduces a state machine that goes like: > >> - starts at CTAS_PARALLEL_INS_UNDEF > >> - possibly moves to

Re: Parallel Inserts in CREATE TABLE AS

On Mon, Jan 4, 2021 at 4:22 PM Luc Vlaming wrote: > Sorry it took so long to get back to reviewing this. Thanks for the comments. > wrt v18-0001patch: > > + /* > +* If the worker is for parallel insert in CTAS, then use the proper > +* dest

Re: Parallel Inserts in CREATE TABLE AS

On 04-01-2021 12:16, Hou, Zhijie wrote: Hi wrt v18-0002patch: It looks like this introduces a state machine that goes like: - starts at CTAS_PARALLEL_INS_UNDEF - possibly moves to CTAS_PARALLEL_INS_SELECT - CTAS_PARALLEL_INS_TUP_COST_CAN_IGN can be added - if both were

RE: Parallel Inserts in CREATE TABLE AS

Hi > > wrt v18-0002patch: > > It looks like this introduces a state machine that goes like: > - starts at CTAS_PARALLEL_INS_UNDEF > - possibly moves to CTAS_PARALLEL_INS_SELECT > - CTAS_PARALLEL_INS_TUP_COST_CAN_IGN can be added > - if both were added at some stage, we can

Re: Parallel Inserts in CREATE TABLE AS

On 30-12-2020 04:55, Bharath Rupireddy wrote: On Wed, Dec 30, 2020 at 5:22 AM Zhihong Yu wrote: w.r.t. v17-0004-Enable-CTAS-Parallel-Inserts-For-Append.patch + * Push the dest receiver to Gather node when it is either at the top of the + * plan or under top Append node unless it does not have

Re: Parallel Inserts in CREATE TABLE AS

On Wed, Dec 30, 2020 at 5:28 PM vignesh C wrote: > Few comments: > - /* > -* To allow parallel inserts, we need to ensure that they are safe to > be > -* performed in workers. We have the infrastructure to allow parallel > -* inserts in general except for the cases

Re: Parallel Inserts in CREATE TABLE AS

On Wed, Dec 30, 2020 at 5:26 PM vignesh C wrote: > > On Wed, Dec 30, 2020 at 10:47 AM Bharath Rupireddy > wrote: > > > > On Wed, Dec 30, 2020 at 10:32 AM Dilip Kumar wrote: > > > I have completed reviewing 0001, I don't have more comments, just one > > > question. Soon I will review the

Re: Parallel Inserts in CREATE TABLE AS

On Wed, Dec 30, 2020 at 7:47 PM Bharath Rupireddy wrote: > > Thanks for the comments. > > How about naming like below more generically and placing them in > parallel.h so that it will also be used for refresh materialized view? > > +typedef enum ParallelInsertTupleCostOpt > +{ > +

Re: Parallel Inserts in CREATE TABLE AS

Thanks for the comments. How about naming like below more generically and placing them in parallel.h so that it will also be used for refresh materialized view? +typedef enum ParallelInsertTupleCostOpt +{ + PINS_SELECT_QUERY = 1 << 0, /* turn on this before planning */ + /* + * Turn on this

Re: Parallel Inserts in CREATE TABLE AS

On Wed, Dec 30, 2020 at 9:25 AM Bharath Rupireddy wrote: > > On Wed, Dec 30, 2020 at 5:22 AM Zhihong Yu wrote: > > w.r.t. v17-0004-Enable-CTAS-Parallel-Inserts-For-Append.patch > > > > + * Push the dest receiver to Gather node when it is either at the top of > > the > > + * plan or under top

Re: Parallel Inserts in CREATE TABLE AS

On Wed, Dec 30, 2020 at 10:47 AM Bharath Rupireddy wrote: > > On Wed, Dec 30, 2020 at 10:32 AM Dilip Kumar wrote: > > I have completed reviewing 0001, I don't have more comments, just one > > question. Soon I will review the remaining patches. > > Thanks. > > > +/* If parallel inserts are

Re: Parallel Inserts in CREATE TABLE AS

On Wed, Dec 30, 2020 at 10:49 AM Dilip Kumar wrote: > > On Wed, 30 Dec 2020 at 10:47 AM, Bharath Rupireddy > wrote: >> >> On Wed, Dec 30, 2020 at 10:32 AM Dilip Kumar wrote: >> > I have completed reviewing 0001, I don't have more comments, just one >> > question. Soon I will review the

Re: Parallel Inserts in CREATE TABLE AS

On Wed, 30 Dec 2020 at 10:47 AM, Bharath Rupireddy < bharath.rupireddyforpostg...@gmail.com> wrote: > On Wed, Dec 30, 2020 at 10:32 AM Dilip Kumar > wrote: > > I have completed reviewing 0001, I don't have more comments, just one > > question. Soon I will review the remaining patches. > >

Re: Parallel Inserts in CREATE TABLE AS

On Wed, Dec 30, 2020 at 10:32 AM Dilip Kumar wrote: > I have completed reviewing 0001, I don't have more comments, just one > question. Soon I will review the remaining patches. Thanks. > +/* If parallel inserts are to be allowed, set a few extra information. */ > +if

Re: Parallel Inserts in CREATE TABLE AS

On Mon, Dec 28, 2020 at 10:45 AM Dilip Kumar wrote: > > On Sun, Dec 27, 2020 at 2:20 PM Bharath Rupireddy > wrote: > > > > On Sat, Dec 26, 2020 at 11:11 AM Dilip Kumar wrote: > > > I have reviewed part of v15-0001 patch, I have a few comments, I will > > > continue to review this. > > > >

Re: Parallel Inserts in CREATE TABLE AS

w.r.t. v17-0004-Enable-CTAS-Parallel-Inserts-For-Append.patch + * Push the dest receiver to Gather node when it is either at the top of the + * plan or under top Append node unless it does not have any projections to do. I think the 'unless' should be 'if'. As can be seen from the body of the

Re: Parallel Inserts in CREATE TABLE AS

On Mon, Dec 28, 2020 at 11:24 AM vignesh C wrote: > Test comments are detailed in a few cases and in few others it is not > detailed for similar kinds of parallelism selected tests. I felt we could > make the test comments consistent across the file. Modified the test case description in the

Re: Parallel Inserts in CREATE TABLE AS

On Mon, Dec 28, 2020 at 1:16 AM Zhihong Yu wrote: > For v16-0002-Tuple-Cost-Adjustment-for-Parallel-Inserts-in-CTAS.patch: > > + if (ignore && > + (root->parse->CTASParallelInsInfo & > +CTAS_PARALLEL_INS_TUP_COST_CAN_IGN)) > > I wonder why

Re: Parallel Inserts in CREATE TABLE AS

On Sun, Dec 27, 2020 at 2:28 PM Bharath Rupireddy < bharath.rupireddyforpostg...@gmail.com> wrote: > > On Sat, Dec 26, 2020 at 9:20 PM vignesh C wrote: > > +-- parallel inserts must occur > > +select explain_pictas( > > +'create table parallel_write as select length(stringu1) from tenk1;'); > >

Re: Parallel Inserts in CREATE TABLE AS

On Sun, Dec 27, 2020 at 2:20 PM Bharath Rupireddy wrote: > > On Sat, Dec 26, 2020 at 11:11 AM Dilip Kumar wrote: > > I have reviewed part of v15-0001 patch, I have a few comments, I will > > continue to review this. > > Thanks a lot. > > > 1. > > Why is this temporary hack? and what is the plan

Re: Parallel Inserts in CREATE TABLE AS

For v16-0002-Tuple-Cost-Adjustment-for-Parallel-Inserts-in-CTAS.patch: + if (ignore && + (root->parse->CTASParallelInsInfo & +CTAS_PARALLEL_INS_TUP_COST_CAN_IGN)) I wonder why CTAS_PARALLEL_INS_TUP_COST_CAN_IGN is checked again in the above if since when

Re: Parallel Inserts in CREATE TABLE AS

On Sat, Dec 26, 2020 at 9:20 PM vignesh C wrote: > +-- parallel inserts must occur > +select explain_pictas( > +'create table parallel_write as select length(stringu1) from tenk1;'); > +select count(*) from parallel_write; > +drop table parallel_write; > > We can change comment "parallel inserts

  1   2   >