Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-15 Thread Alvaro Herrera
Tom Lane wrote: > Robert Haas writes: > > I really, really strongly encourage you to rip the use of DSA out here > > entirely. It is reducing the reliability of a critical part of the > > system for no actual benefit other than speculation that this is going > > to be better in the future, and it

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-15 Thread Tom Lane
Robert Haas writes: > I really, really strongly encourage you to rip the use of DSA out here > entirely. It is reducing the reliability of a critical part of the > system for no actual benefit other than speculation that this is going > to be better in the future, and it adds a bunch of failure c

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-15 Thread Robert Haas
On Mon, Aug 14, 2017 at 7:16 PM, Alvaro Herrera wrote: >> Yeah, and the other question -- which Thomas asked before you >> originally committed originally, and which I just now asked again is >> "Why in the world are you using DSA for this at all?". There are >> serious problems with that which b

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-14 Thread Alvaro Herrera
Robert Haas wrote: > On Mon, Aug 14, 2017 at 1:12 PM, Alvaro Herrera > wrote: > > Yeah, the problem that lwlocks aren't released is because the launcher > > is not in a transaction at that point, so AbortCurrentTransaction() > > doesn't release locks like it normally would. The simplest fix (in t

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-14 Thread Robert Haas
On Mon, Aug 14, 2017 at 1:12 PM, Alvaro Herrera wrote: > Yeah, the problem that lwlocks aren't released is because the launcher > is not in a transaction at that point, so AbortCurrentTransaction() > doesn't release locks like it normally would. The simplest fix (in the > attached 0001 patch) is

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-14 Thread David Fetter
On Sun, Aug 13, 2017 at 05:56:56PM -0700, Andres Freund wrote: > Hi, > > Since we're getting a bit into the weeds of a different topic, and since > I think it's an interesting feature, I'm detaching this into a separate > thread. > > On 2017-08-12 23:37:27 -0400, Tom Lane wrote: > > >> On 2017-08

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-14 Thread Tom Lane
Robert Haas writes: > I think it would be a bad idea to remove both > dynamic_shared_memory_type=none and dynamic_shared_memory_type=mmap. > If you do that, then somebody who doesn't have root and whose system > configuration is messed up can't start PostgreSQL at all. I'd be happy to leave the m

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-14 Thread Andres Freund
On 2017-08-14 13:06:48 -0400, Robert Haas wrote: > On Mon, Aug 14, 2017 at 1:02 PM, Andres Freund wrote: > > I previously thought that an option to occasionally WAL log the stats > > file would be useful (e.g. just before a checkpoint). That'd make them > > persistent, and available on the standby

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-14 Thread Alvaro Herrera
Robert Haas wrote: > Actually, now that you mention it, I think it *is* broken already, or > more to the point, if you configure that value, the autovacuum > launcher hangs up, because of the AutovacuumWorkItem stuff that Alvaro > added. When I just tested it, the AV launcher somehow ended up > w

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-14 Thread Robert Haas
On Mon, Aug 14, 2017 at 1:02 PM, Andres Freund wrote: > I previously thought that an option to occasionally WAL log the stats > file would be useful (e.g. just before a checkpoint). That'd make them > persistent, and available on the standby. But that'd still require > somehow dealing with stats b

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-14 Thread Andres Freund
On 2017-08-14 18:57:39 +0200, Magnus Hagander wrote: > On Mon, Aug 14, 2017 at 5:46 PM, Robert Haas wrote: > > > On Sun, Aug 13, 2017 at 9:59 PM, Tom Lane wrote: > > >> b) I think our tendency to dump all stats whenever we crash isn't really > > >>tenable, given how autovacuum etc are tied t

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-14 Thread Robert Haas
On Mon, Aug 14, 2017 at 12:36 PM, Andres Freund wrote: >> If so, why isn't choose_dsm_implementation() trying it; and if not, >> why are we carrying it? > > I think the idea was that there might be platforms that require it, but > ... Right. So, for example, POSIX shared memory will fail on Linu

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-14 Thread Magnus Hagander
On Mon, Aug 14, 2017 at 5:46 PM, Robert Haas wrote: > On Sun, Aug 13, 2017 at 9:59 PM, Tom Lane wrote: > >> b) I think our tendency to dump all stats whenever we crash isn't really > >>tenable, given how autovacuum etc are tied to them. > > > > Eh, maybe. I don't think crashes are really so

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-14 Thread Andres Freund
On 2017-08-14 12:28:39 -0400, Tom Lane wrote: > Robert Haas writes: > > On Mon, Aug 14, 2017 at 12:16 PM, Tom Lane wrote: > >> Just FYI, the only values being reported by buildfarm animals are > >> "posix", "sysv", and "windows". So while mmap may be a thing, > >> it's an untested thing. > > >

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-14 Thread Tom Lane
Robert Haas writes: > On Mon, Aug 14, 2017 at 12:16 PM, Tom Lane wrote: >> Just FYI, the only values being reported by buildfarm animals are >> "posix", "sysv", and "windows". So while mmap may be a thing, >> it's an untested thing. > I'm pretty sure I dev-tested it before committing anything,

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-14 Thread Andres Freund
On 2017-08-14 12:21:30 -0400, Robert Haas wrote: > >> ... If somebody has a system where no other form of shared > >> memory, works dynamic_shared_memory_type = mmap is still a thing, so > >> the use case for "none" seems very thin indeed. I'd vote for just > >> ripping it out in v11. > > > > Just

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-14 Thread Robert Haas
On Mon, Aug 14, 2017 at 12:16 PM, Tom Lane wrote: > Robert Haas writes: >> On Sun, Aug 13, 2017 at 9:59 PM, Tom Lane wrote: >>> In principle we could keep the existing mechanism as a fallback. >>> Whether that's worth the trouble is debatable. The current code >>> in initdb believes that every

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-14 Thread Tom Lane
Robert Haas writes: > On Sun, Aug 13, 2017 at 9:59 PM, Tom Lane wrote: >> In principle we could keep the existing mechanism as a fallback. >> Whether that's worth the trouble is debatable. The current code >> in initdb believes that every platform has some type of DSM support >> (see choose_dsm_

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-14 Thread Alvaro Herrera
Robert Haas wrote: > Actually, now that you mention it, I think it *is* broken already, or > more to the point, if you configure that value, the autovacuum > launcher hangs up, because of the AutovacuumWorkItem stuff that Alvaro > added. When I just tested it, the AV launcher somehow ended up > w

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-14 Thread Andres Freund
Hi, On 2017-08-14 11:46:10 -0400, Robert Haas wrote: > On Sun, Aug 13, 2017 at 9:59 PM, Tom Lane wrote: > > Andres Freund writes: > >> You both are obviously right. Another point of potential concern could > >> be that we'd pretyt much fully rely on dsm/dht's being available, for > >> the serve

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-14 Thread Robert Haas
On Sun, Aug 13, 2017 at 9:59 PM, Tom Lane wrote: > Andres Freund writes: >> You both are obviously right. Another point of potential concern could >> be that we'd pretyt much fully rely on dsm/dht's being available, for >> the server to function correctly. Are we ok with that? Right now >> postg

Re: [HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-13 Thread Tom Lane
Andres Freund writes: > You both are obviously right. Another point of potential concern could > be that we'd pretyt much fully rely on dsm/dht's being available, for > the server to function correctly. Are we ok with that? Right now > postgres still works perfectly well, leaving parallelism asid

[HACKERS] shared memory based stat collector (was: Sharing record typmods between backends)

2017-08-13 Thread Andres Freund
Hi, Since we're getting a bit into the weeds of a different topic, and since I think it's an interesting feature, I'm detaching this into a separate thread. On 2017-08-12 23:37:27 -0400, Tom Lane wrote: > >> On 2017-08-12 22:52:57 -0400, Robert Haas wrote: > >>> I think it'd be pretty interesting