Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

On Fri, 18 Mar 2016 20:40:16 +0300 Dmitry Ivanov wrote: > - Currently there's no documentation regarding parallel gin build > feature and provided GUC variables. > You could replace direct access to 'rd_id' field with the > RelationGetRelid macro. > Parameter 'r' is

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

On Fri, 18 Mar 2016 20:40:16 +0300 Dmitry Ivanov wrote: > Hi Constantin, > > I did a quick review of your patch, and here are my comments: > > - This patch applies cleanly to the current HEAD (61d2ebdbf91). > > - Code compiles without warnings. > > - Currently

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

On Wed, Mar 16, 2016 at 7:50 PM, Constantin S. Pan wrote: > > On Wed, 16 Mar 2016 18:08:38 +0530 > Amit Kapila wrote: > > > > > Why backend just waits, why can't it does the same work as any worker > > does? In general, for other parallelism features

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

On Thu, 17 Mar 2016 13:21:32 +0530 Amit Kapila wrote: > On Wed, Mar 16, 2016 at 7:50 PM, Constantin S. Pan > wrote: > > > > On Wed, 16 Mar 2016 18:08:38 +0530 > > Amit Kapila wrote: > > > > > > > > Why backend just waits, why

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

On Wed, Mar 16, 2016 at 2:55 PM, Constantin S. Pan wrote: > > On Wed, 16 Mar 2016 12:14:51 +0530 > Amit Kapila wrote: > > > On Wed, Mar 16, 2016 at 5:41 AM, Constantin S. Pan > > wrote: > > > 3. Tested on some real data (GIN index on

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

On Wed, 16 Mar 2016 18:08:38 +0530 Amit Kapila wrote: > On Wed, Mar 16, 2016 at 2:55 PM, Constantin S. Pan > wrote: > > > > On Wed, 16 Mar 2016 12:14:51 +0530 > > Amit Kapila wrote: > > > > > On Wed, Mar 16, 2016 at 5:41 AM,

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

On Thu, Mar 17, 2016 at 2:56 PM, Constantin S. Pan wrote: > > On Thu, 17 Mar 2016 13:21:32 +0530 > Amit Kapila wrote: > > > On Wed, Mar 16, 2016 at 7:50 PM, Constantin S. Pan > > wrote: > > > > > > On Wed, 16 Mar 2016 18:08:38 +0530 >

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

Hi Constantin, I did a quick review of your patch, and here are my comments: - This patch applies cleanly to the current HEAD (61d2ebdbf91). - Code compiles without warnings. - Currently there's no documentation regarding parallel gin build feature and provided GUC variables. - Built indexes

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

On Thu, Mar 17, 2016 at 11:42 PM, Amit Kapila wrote: > I think here the comparison should be between the case of (active backend + > 1 worker) with (passive backend + 1 worker) or (active backend + 2 worker) > with (passive backend + 2 workers). I don't think it is good

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

On Wed, 16 Mar 2016 02:43:47 -0700 Peter Geoghegan wrote: > On Wed, Mar 16, 2016 at 2:25 AM, Constantin S. Pan > wrote: > > The backend just waits for the results from the workers and merges > > them (in case wnum > 0). So the 1-worker configuration should

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

On Wed, Mar 16, 2016 at 2:25 AM, Constantin S. Pan wrote: > The backend just waits for the results from the workers and merges them > (in case wnum > 0). So the 1-worker configuration should never be used, > because it is as sequential as the 0-worker, but adds data transfer.

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

On Wed, 16 Mar 2016 12:14:51 +0530 Amit Kapila wrote: > On Wed, Mar 16, 2016 at 5:41 AM, Constantin S. Pan > wrote: > > > On Mon, 14 Mar 2016 08:42:26 -0400 > > David Steele wrote: > > > > > On 2/18/16 10:10 AM, Constantin S. Pan

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

On Wed, Mar 16, 2016 at 5:41 AM, Constantin S. Pan wrote: > On Mon, 14 Mar 2016 08:42:26 -0400 > David Steele wrote: > > > On 2/18/16 10:10 AM, Constantin S. Pan wrote: > > > On Wed, 17 Feb 2016 23:01:47 +0300 > > > Oleg Bartunov

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

On Mon, 14 Mar 2016 08:42:26 -0400 David Steele wrote: > On 2/18/16 10:10 AM, Constantin S. Pan wrote: > > On Wed, 17 Feb 2016 23:01:47 +0300 > > Oleg Bartunov wrote: > > > >> My feedback is (Mac OS X 10.11.3) > >> > >> set gin_parallel_workers=2; > >>

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

On 2/18/16 10:10 AM, Constantin S. Pan wrote: On Wed, 17 Feb 2016 23:01:47 +0300 Oleg Bartunov wrote: My feedback is (Mac OS X 10.11.3) set gin_parallel_workers=2; create index message_body_idx on messages using gin(body_tsvector); LOG: worker process: parallel worker

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

On Wed, 17 Feb 2016 23:01:47 +0300 Oleg Bartunov wrote: > My feedback is (Mac OS X 10.11.3) > > set gin_parallel_workers=2; > create index message_body_idx on messages using gin(body_tsvector); > LOG: worker process: parallel worker for PID 5689 (PID 6906) was > terminated

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

I was testing with number of workers starting at 0. The 0 case is (in theory) equivalent to master branch. But I should certainly compare to the master branch, you are right. Will do that shortly. On Wed, 17 Feb 2016 12:26:05 -0800 Peter Geoghegan wrote: > On Wed, Feb 17, 2016

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

On Wed, Feb 17, 2016 at 7:55 AM, Constantin S. Pan wrote: > 4. Hit the 8x speedup limit. Made some analysis of the reasons (see the > attached plot or the data file). Did you actually compare this to the master branch? I wouldn't like to assume that the one worker case was

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

On Wed, Feb 17, 2016 at 6:55 PM, Constantin S. Pan wrote: > On Sat, 16 Jan 2016 01:38:39 +0300 > "Constantin S. Pan" wrote: > > > The task of building GIN can require lots of time and eats 100 % CPU, > > but we could easily make it use more than a 100 %,

Re: [HACKERS] [WIP] speeding up GIN build with parallel workers

On Sat, 16 Jan 2016 01:38:39 +0300 "Constantin S. Pan" wrote: > The task of building GIN can require lots of time and eats 100 % CPU, > but we could easily make it use more than a 100 %, especially since we > now have parallel workers in postgres. > > The process of building