Re: Postgres with pthread

On 27.12.2017 13:08, Andres Freund wrote: On December 27, 2017 11:05:52 AM GMT+01:00, james wrote: All threads are blocked in semaphores. That they are blocked is inevitable - I guess the issue is that they are thrashing. I guess it would be necessary to

Re: Postgres with pthread

On December 27, 2017 11:05:52 AM GMT+01:00, james wrote: > > All threads are blocked in semaphores. >That they are blocked is inevitable - I guess the issue is that they >are >thrashing. >I guess it would be necessary to separate the internals to have some

Re: Postgres with pthread

> All threads are blocked in semaphores. That they are blocked is inevitable - I guess the issue is that they are thrashing. I guess it would be necessary to separate the internals to have some internal queueing and effectively reduce the number of actively executing threads. In effect make

Re: Postgres with pthread

far from completion. 4. I have performed experiments with replacing synchronization primitives used in Postgres with pthread analogues. Unfortunately it has almost now influence on performance. 5. Handling large number of connections. The maximal number of postgres connections is almost

Re: Postgres with pthread

3. Now my prototype is passing all regression tests now. But handling of > errors is still far from completion. > > 4. I have performed experiments with replacing synchronization primitives > used in Postgres with pthread analogues. > Unfortunately it has almost now influence

Re: Postgres with pthread

experiments with replacing synchronization primitives used in Postgres with pthread analogues. Unfortunately it has almost now influence on performance. 5. Handling large number of connections. The maximal number of postgres connections is almost the same: 100k. But memory footprint in case of pthreads

Re: Postgres with pthread

On 06/12/2017 17:26, Andreas Karlsson wrote: An additional issue is that this could break a lot of extensions and in a way that it is not apparent at compile time. This means we may need to break all extensions to force extensions authors to check if they are thread safe. I do not like

Re: Postgres with pthread

On Sat, Dec 9, 2017 at 1:09 AM, konstantin knizhnik < k.knizh...@postgrespro.ru> wrote: > I am not going to show stack traces of all 1000 threads. > But you may notice that proc array lock really seems be be a bottleneck. > Yes, proc array lock easily becomes a bottleneck on multicore machine

Re: Postgres with pthread

On Dec 7, 2017, at 10:41 AM, Simon Riggs wrote: >> But it is a theory. The main idea of this prototype was to prove or disprove >> this expectation at practice. > >> But please notice that it is very raw prototype. A lot of stuff is not >> working yet. > >> And supporting all of exited

Re: Postgres with pthread

On 8 December 2017 at 03:58, Andres Freund wrote: > On 2017-12-07 11:26:07 +0800, Craig Ringer wrote: > > PostgreSQL's architecture conflates "connection", "session" and > "executor" > > into one somewhat muddled mess. > > How is the executor entangled in the other two? > >

Re: Postgres with pthread

Hi, On 2017-12-07 20:48:06 +, Greg Stark wrote: > But then I thought about it a bit and I do wonder. I don't know how > well we test having multiple portals doing all kinds of different > query plans with their execution interleaved. Cursors test that pretty well. > And I definitely have

Re: Postgres with pthread

On 2017-12-07 11:26:07 +0800, Craig Ringer wrote: > PostgreSQL's architecture conflates "connection", "session" and "executor" > into one somewhat muddled mess. How is the executor entangled in the other two? Greetings, Andres Freund

Re: Postgres with pthread

On Wed, Dec 6, 2017 at 10:20 PM, Craig Ringer wrote: > Personally I think it's a pity we didn't land up here before the foundations > for parallel query went in - DSM, shm_mq, DSA, etc. I know the EDB folks at > least looked into it though, and presumably there were good

Re: Postgres with pthread

On 7 December 2017 at 19:55, Konstantin Knizhnik wrote: > > Pros: > 1. Simplified memory model: no need in DSM, shm_mq, DSA, etc > shm_mq would remain useful, and the others could only be dropped if you also dropped process-model support entirely. > 1. Breaks

Re: Postgres with pthread

On 07.12.2017 00:58, Thomas Munro wrote: Using a ton of thread local variables may be a useful stepping stone, but if we want to be able to separate threads/processes from sessions eventually then I guess we'll want to model sessions as first class objects and pass them around explicitly or

Re: Postgres with pthread

Hi On 06.12.2017 20:08, Andres Freund wrote: 4. Rewrite file descriptor cache to be global (shared by all threads). That one I'm very unconvinced of, that's going to add a ton of new contention. Do you mean lock contention because of mutex I used to synchronize access to shared file

Re: Postgres with pthread

> But it is a theory. The main idea of this prototype was to prove or disprove > this expectation at practice. > But please notice that it is very raw prototype. A lot of stuff is not > working yet. > And supporting all of exited Postgres functionality requires > much more efforts (and even more

Re: Postgres with pthread

On 7 December 2017 at 11:44, Tsunakawa, Takayuki < tsunakawa.ta...@jp.fujitsu.com> wrote: > From: Craig Ringer [mailto:cr...@2ndquadrant.com] > > I'd personally expect that an immediate conversion would result > > in very > > little speedup, a bunch of code deleted, a bunch of

Re: Postgres with pthread

On 7 December 2017 at 05:58, Thomas Munro wrote: > > Using a ton of thread local variables may be a useful stepping stone, > but if we want to be able to separate threads/processes from sessions > eventually then I guess we'll want to model sessions as first class

Re: Postgres with pthread

On 7 December 2017 at 01:17, Andres Freund wrote: > > > I think you've done us a very substantial service by pursuing > > this far enough to get some quantifiable performance results. > > But now that we have some results in hand, I think we're best > > off sticking with the

Re: Postgres with pthread

On Thu, Dec 7, 2017 at 6:08 AM, Andres Freund wrote: > On 2017-12-06 19:40:00 +0300, Konstantin Knizhnik wrote: >> As far as I remember, several years ago when implementation of intra-query >> parallelism was just started there was discussion whether to use threads or >> leave

Re: Postgres with pthread

Hi, On 2017-12-06 12:28:29 -0500, Robert Haas wrote: > > Possibly we even want to continue having various > > processes around besides that, the most interesting cases involving > > threads are around intra-query parallelism, and pooling, and for both a > > hybrid model could be beneficial. > >

Re: Postgres with pthread

On 12/06/2017 06:08 PM, Andres Freund wrote: I think the biggest problem with doing this for real is that it's a huge project, and that it'll take a long time. An additional issue is that this could break a lot of extensions and in a way that it is not apparent at compile time. This means we

Re: Postgres with pthread

> "barely a 50% speedup" - Hah. I don't believe the numbers, but that'd be > huge. They are numbers derived from a benchmark that any sane person would be using a connection pool for in a production environment, but impressive if true none the less.

Re: Postgres with pthread

Hi, On 2017-12-06 11:53:21 -0500, Tom Lane wrote: > Konstantin Knizhnik writes: > However, if I guess at which numbers are supposed to be what, > it looks like even the best case is barely a 50% speedup. "barely a 50% speedup" - Hah. I don't believe the numbers, but

Re: Postgres with pthread

On Wed, Dec 6, 2017 at 11:53 AM, Tom Lane wrote: > barely a 50% speedup. I think that's an awfully strange choice of adverb. This is, by its authors own admission, a rough cut at this, probably with very little of the optimization that could ultimately done, and it's already

Re: Postgres with pthread

Hi! On 2017-12-06 19:40:00 +0300, Konstantin Knizhnik wrote: > As far as I remember, several years ago when implementation of intra-query > parallelism was just started there was discussion whether to use threads or > leave traditional Postgres process architecture. The decision was made to >

Re: Postgres with pthread

Here it is formatted a little better. ​ So a little over 50% performance improvement for a couple of the test cases. On Wed, Dec 6, 2017 at 11:53 AM, Tom Lane wrote: > Konstantin Knizhnik writes: > > Below are some results (1000xTPS) of

Postgres with pthread

Hi hackers, As far as I remember, several years ago when implementation of intra-query parallelism was just started there was discussion whether to use threads or leave traditional Postgres process architecture. The decision was made to leave processes. So now we have bgworkers, shared