Re: Postgres with pthread

On 27.12.2017 13:08, Andres Freund wrote: On December 27, 2017 11:05:52 AM GMT+01:00, james wrote: All threads are blocked in semaphores. That they are blocked is inevitable - I guess the issue is that they are thrashing. I guess it would be necessary to

Re: Postgres with pthread

On December 27, 2017 11:05:52 AM GMT+01:00, james wrote: > > All threads are blocked in semaphores. >That they are blocked is inevitable - I guess the issue is that they >are >thrashing. >I guess it would be necessary to separate the internals to have some

Re: Postgres with pthread

> All threads are blocked in semaphores. That they are blocked is inevitable - I guess the issue is that they are thrashing. I guess it would be necessary to separate the internals to have some internal queueing and effectively reduce the number of actively executing threads. In effect make

Re: Postgres with pthread

On 21.12.2017 16:25, Konstantin Knizhnik wrote: I continue experiments with my pthread prototype. Latest results are the following: 1. I have eliminated all (I hope) calls of non-reentrant functions (getopt, setlocale, setitimer, localtime, ...). So now parallel tests are passed. 2. I

Re: Postgres with pthread

2017-12-21 14:25 GMT+01:00 Konstantin Knizhnik : > I continue experiments with my pthread prototype. > Latest results are the following: > > 1. I have eliminated all (I hope) calls of non-reentrant functions > (getopt, setlocale, setitimer, localtime, ...). So now

Re: Postgres with pthread

I continue experiments with my pthread prototype. Latest results are the following: 1. I have eliminated all (I hope) calls of non-reentrant functions (getopt, setlocale, setitimer, localtime, ...). So now parallel tests are passed. 2. I have implemented deallocation of top memory context

Re: Postgres with pthread

On 06/12/2017 17:26, Andreas Karlsson wrote: An additional issue is that this could break a lot of extensions and in a way that it is not apparent at compile time. This means we may need to break all extensions to force extensions authors to check if they are thread safe. I do not like

Re: Postgres with pthread

On Sat, Dec 9, 2017 at 1:09 AM, konstantin knizhnik < k.knizh...@postgrespro.ru> wrote: > I am not going to show stack traces of all 1000 threads. > But you may notice that proc array lock really seems be be a bottleneck. > Yes, proc array lock easily becomes a bottleneck on multicore machine

Re: Postgres with pthread

On Dec 7, 2017, at 10:41 AM, Simon Riggs wrote: >> But it is a theory. The main idea of this prototype was to prove or disprove >> this expectation at practice. > >> But please notice that it is very raw prototype. A lot of stuff is not >> working yet. > >> And supporting all of exited

Re: Postgres with pthread

On 8 December 2017 at 03:58, Andres Freund wrote: > On 2017-12-07 11:26:07 +0800, Craig Ringer wrote: > > PostgreSQL's architecture conflates "connection", "session" and > "executor" > > into one somewhat muddled mess. > > How is the executor entangled in the other two? > >

Re: Postgres with pthread

Hi, On 2017-12-07 20:48:06 +, Greg Stark wrote: > But then I thought about it a bit and I do wonder. I don't know how > well we test having multiple portals doing all kinds of different > query plans with their execution interleaved. Cursors test that pretty well. > And I definitely have

Re: Postgres with pthread

On 2017-12-07 11:26:07 +0800, Craig Ringer wrote: > PostgreSQL's architecture conflates "connection", "session" and "executor" > into one somewhat muddled mess. How is the executor entangled in the other two? Greetings, Andres Freund

Re: Postgres with pthread

On Wed, Dec 6, 2017 at 10:20 PM, Craig Ringer wrote: > Personally I think it's a pity we didn't land up here before the foundations > for parallel query went in - DSM, shm_mq, DSA, etc. I know the EDB folks at > least looked into it though, and presumably there were good

Re: Postgres with pthread

On 7 December 2017 at 19:55, Konstantin Knizhnik wrote: > > Pros: > 1. Simplified memory model: no need in DSM, shm_mq, DSA, etc > shm_mq would remain useful, and the others could only be dropped if you also dropped process-model support entirely. > 1. Breaks

Re: Postgres with pthread

On 07.12.2017 00:58, Thomas Munro wrote: Using a ton of thread local variables may be a useful stepping stone, but if we want to be able to separate threads/processes from sessions eventually then I guess we'll want to model sessions as first class objects and pass them around explicitly or

Re: Postgres with pthread

Hi On 06.12.2017 20:08, Andres Freund wrote: 4. Rewrite file descriptor cache to be global (shared by all threads). That one I'm very unconvinced of, that's going to add a ton of new contention. Do you mean lock contention because of mutex I used to synchronize access to shared file

Re: Postgres with pthread

> But it is a theory. The main idea of this prototype was to prove or disprove > this expectation at practice. > But please notice that it is very raw prototype. A lot of stuff is not > working yet. > And supporting all of exited Postgres functionality requires > much more efforts (and even more

Re: Postgres with pthread

On 7 December 2017 at 11:44, Tsunakawa, Takayuki < tsunakawa.ta...@jp.fujitsu.com> wrote: > From: Craig Ringer [mailto:cr...@2ndquadrant.com] > > I'd personally expect that an immediate conversion would result > > in very > > little speedup, a bunch of code deleted, a bunch of

Re: Postgres with pthread

On 7 December 2017 at 05:58, Thomas Munro wrote: > > Using a ton of thread local variables may be a useful stepping stone, > but if we want to be able to separate threads/processes from sessions > eventually then I guess we'll want to model sessions as first class

Re: Postgres with pthread

On 7 December 2017 at 01:17, Andres Freund wrote: > > > I think you've done us a very substantial service by pursuing > > this far enough to get some quantifiable performance results. > > But now that we have some results in hand, I think we're best > > off sticking with the

Re: Postgres with pthread

On Thu, Dec 7, 2017 at 6:08 AM, Andres Freund wrote: > On 2017-12-06 19:40:00 +0300, Konstantin Knizhnik wrote: >> As far as I remember, several years ago when implementation of intra-query >> parallelism was just started there was discussion whether to use threads or >> leave

Re: Postgres with pthread

Hi, On 2017-12-06 12:28:29 -0500, Robert Haas wrote: > > Possibly we even want to continue having various > > processes around besides that, the most interesting cases involving > > threads are around intra-query parallelism, and pooling, and for both a > > hybrid model could be beneficial. > >

Re: Postgres with pthread

On 12/06/2017 06:08 PM, Andres Freund wrote: I think the biggest problem with doing this for real is that it's a huge project, and that it'll take a long time. An additional issue is that this could break a lot of extensions and in a way that it is not apparent at compile time. This means we

Re: Postgres with pthread

> "barely a 50% speedup" - Hah. I don't believe the numbers, but that'd be > huge. They are numbers derived from a benchmark that any sane person would be using a connection pool for in a production environment, but impressive if true none the less.

Re: Postgres with pthread

Hi, On 2017-12-06 11:53:21 -0500, Tom Lane wrote: > Konstantin Knizhnik writes: > However, if I guess at which numbers are supposed to be what, > it looks like even the best case is barely a 50% speedup. "barely a 50% speedup" - Hah. I don't believe the numbers, but

Re: Postgres with pthread

On Wed, Dec 6, 2017 at 11:53 AM, Tom Lane wrote: > barely a 50% speedup. I think that's an awfully strange choice of adverb. This is, by its authors own admission, a rough cut at this, probably with very little of the optimization that could ultimately done, and it's already

Re: Postgres with pthread

Hi! On 2017-12-06 19:40:00 +0300, Konstantin Knizhnik wrote: > As far as I remember, several years ago when implementation of intra-query > parallelism was just started there was discussion whether to use threads or > leave traditional Postgres process architecture. The decision was made to >

Re: Postgres with pthread

Here it is formatted a little better. ​ So a little over 50% performance improvement for a couple of the test cases. On Wed, Dec 6, 2017 at 11:53 AM, Tom Lane wrote: > Konstantin Knizhnik writes: > > Below are some results (1000xTPS) of