Re: removal of dangling temp tables

2018-12-28 Thread Alvaro Herrera
On 2018-Dec-28, Tom Lane wrote: > I wrote: > > Alvaro Herrera writes: > >> I looked at > >> https://github.com/postgrespro/pg_wait_sampling/blob/master/pg_wait_sampling.c > >> https://github.com/citusdata/citus/search?q=pgproc_q=pgproc > >> and skimmed a few others can't find any instance where

Re: removal of dangling temp tables

2018-12-28 Thread Tom Lane
I wrote: > Alvaro Herrera writes: >> I looked at >> https://github.com/postgrespro/pg_wait_sampling/blob/master/pg_wait_sampling.c >> https://github.com/citusdata/citus/search?q=pgproc_q=pgproc >> and skimmed a few others can't find any instance where the full struct >> is used, as opposed to

Re: removal of dangling temp tables

2018-12-28 Thread Tom Lane
Alvaro Herrera writes: > On 2018-Dec-28, Alvaro Herrera wrote: >> On 2018-Dec-28, Michael Paquier wrote: >>> There are many exotic extensions which could be using sizeof(PGPROC) >>> as that's a popular structure, >> Can you show one instance of this? > I looked at >

Re: removal of dangling temp tables

2018-12-28 Thread Alvaro Herrera
On 2018-Dec-28, Alvaro Herrera wrote: > On 2018-Dec-28, Michael Paquier wrote: > > > On Fri, Dec 28, 2018 at 12:05:34AM -0300, Alvaro Herrera wrote: > > > The server code allocates arrays, but that's fine because that code is > > > recompiled. Extensions only pass pointers around -- they don't

Re: removal of dangling temp tables

2018-12-28 Thread Alvaro Herrera
On 2018-Dec-28, Michael Paquier wrote: > On Fri, Dec 28, 2018 at 12:05:34AM -0300, Alvaro Herrera wrote: > > The server code allocates arrays, but that's fine because that code is > > recompiled. Extensions only pass pointers around -- they don't create > > any additional arrays. > > There are

Re: removal of dangling temp tables

2018-12-27 Thread Michael Paquier
On Fri, Dec 28, 2018 at 12:05:34AM -0300, Alvaro Herrera wrote: > The server code allocates arrays, but that's fine because that code is > recompiled. Extensions only pass pointers around -- they don't create > any additional arrays. There are many exotic extensions which could be using

Re: removal of dangling temp tables

2018-12-27 Thread Alvaro Herrera
On 2018-Dec-28, Michael Paquier wrote: > On Thu, Dec 27, 2018 at 04:30:21PM -0300, Alvaro Herrera wrote: > > We allow structs to receive new members at the end of the struct, since > > this doesn't affect the offset of existing members; thus code already > > compiled with the previous struct

Re: removal of dangling temp tables

2018-12-27 Thread Michael Paquier
On Thu, Dec 27, 2018 at 04:30:21PM -0300, Alvaro Herrera wrote: > We allow structs to receive new members at the end of the struct, since > this doesn't affect the offset of existing members; thus code already > compiled with the previous struct definition does not break. AFAICS > there is no

Re: removal of dangling temp tables

2018-12-27 Thread Alvaro Herrera
On 2018-Dec-27, Michael Paquier wrote: > On Wed, Dec 26, 2018 at 08:51:56PM -0300, Alvaro Herrera wrote: > > Having been victim of ABI incompatibility myself, I loathe giving too > > much priority to other issues that can be resolved in other ways, so I > > don't necessarily support your view on

Re: removal of dangling temp tables

2018-12-26 Thread Michael Paquier
On Wed, Dec 26, 2018 at 08:51:56PM -0300, Alvaro Herrera wrote: > Having been victim of ABI incompatibility myself, I loathe giving too > much priority to other issues that can be resolved in other ways, so I > don't necessarily support your view on bugs. > That said, I think in this case it

RE: removal of dangling temp tables

2018-12-26 Thread Tsunakawa, Takayuki
From: Alvaro Herrera [mailto:alvhe...@2ndquadrant.com] > Having been victim of ABI incompatibility myself, I loathe giving too much > priority to other issues that can be resolved in other ways, so I don't > necessarily support your view on bugs. > That said, I think in this case it shouldn't be a

Re: removal of dangling temp tables

2018-12-26 Thread Alvaro Herrera
On 2018-Dec-26, Tsunakawa, Takayuki wrote: > From: Alvaro Herrera [mailto:alvhe...@2ndquadrant.com] > > The more aggressive action is to backpatch 943576bddcb5 ("Make autovacuum > > more aggressive to remove orphaned temp tables") which is currently only > > in pg11. We would put the new PGPROC

RE: removal of dangling temp tables

2018-12-26 Thread Tsunakawa, Takayuki
From: Alvaro Herrera [mailto:alvhe...@2ndquadrant.com] > The more aggressive action is to backpatch 943576bddcb5 ("Make autovacuum > more aggressive to remove orphaned temp tables") which is currently only > in pg11. We would put the new PGPROC member at the end of the struct, to > avoid ABI

Re: removal of dangling temp tables

2018-12-26 Thread Alvaro Herrera
On 2018-Dec-16, Michael Paquier wrote: > On Sat, Dec 15, 2018 at 09:51:31AM -0500, Tom Lane wrote: > > Alvaro Herrera writes: > >> Oh, we already have it! Sorry, I overlooked it. With that, it seems > >> the patch is fairly simple ... I wonder about the locking implications > >> in autovacuum,

Re: removal of dangling temp tables

2018-12-15 Thread Michael Paquier
On Sat, Dec 15, 2018 at 09:51:31AM -0500, Tom Lane wrote: > Alvaro Herrera writes: >> Oh, we already have it! Sorry, I overlooked it. With that, it seems >> the patch is fairly simple ... I wonder about the locking implications >> in autovacuum, though -- the value is set in backends without

Re: removal of dangling temp tables

2018-12-15 Thread Tom Lane
Alvaro Herrera writes: > On 2018-Dec-15, Michael Paquier wrote: >> Isn't that what tempNamespaceId can be used for in PGPROC now? The flag >> would be set only when a backend creates a new temporary schema so as it >> can be tracked as the owner of the schema. > Oh, we already have it! Sorry,

Re: removal of dangling temp tables

2018-12-15 Thread Alvaro Herrera
On 2018-Dec-15, Michael Paquier wrote: > On Fri, Dec 14, 2018 at 11:06:32PM -0300, Alvaro Herrera wrote: > > I did propose in my OP the idea of a PGPROC boolean flag that indicates > > whether the temp namespace has been set up. If not, have autovac remove > > those tables. I like this option

Re: removal of dangling temp tables

2018-12-14 Thread Michael Paquier
On Fri, Dec 14, 2018 at 11:06:32PM -0300, Alvaro Herrera wrote: > I did propose in my OP the idea of a PGPROC boolean flag that indicates > whether the temp namespace has been set up. If not, have autovac remove > those tables. I like this option better, but I fear it adds more > ProcArrayLock

Re: removal of dangling temp tables

2018-12-14 Thread Alvaro Herrera
On 2018-Dec-14, Robert Haas wrote: > On Fri, Dec 14, 2018 at 6:35 PM Tom Lane wrote: > > Hm. It *could*, if we wanted it to run some transactions after > > finishing recovery. > > It'd have to launch a separate process per database. That would be > useful infrastructure for other things, too,

Re: removal of dangling temp tables

2018-12-14 Thread Alvaro Herrera
On 2018-Dec-14, Robert Haas wrote: > On Fri, Dec 14, 2018 at 12:27 PM Alvaro Herrera > wrote: > > Maybe it'd be better to change temp table removal to always drop > > max_locks_per_transaction objects at a time (ie. commit/start a new > > transaction every so many objects). > > We're basically

Re: removal of dangling temp tables

2018-12-14 Thread Robert Haas
On Fri, Dec 14, 2018 at 6:35 PM Tom Lane wrote: > Hm. It *could*, if we wanted it to run some transactions after > finishing recovery. It'd have to launch a separate process per database. That would be useful infrastructure for other things, too, like automatic catalog upgrades in minor

Re: removal of dangling temp tables

2018-12-14 Thread Andres Freund
Hi, On 2018-12-14 13:35:50 -0500, Tom Lane wrote: > Hm. It *could*, if we wanted it to run some transactions after > finishing recovery. How, we don't have infrastructure for changing databases yet? > But I guess I wonder why bother; if the disk > space is gone then there's not that much

Re: removal of dangling temp tables

2018-12-14 Thread Tom Lane
Robert Haas writes: > On Fri, Dec 14, 2018 at 12:57 PM Tom Lane wrote: >> I seem to recall discussions about having crash recovery go around >> and clean out temp tables. That seems like a better plan than >> penalizing every session start with this. > Well, crash recovery already removes the

Re: removal of dangling temp tables

2018-12-14 Thread Robert Haas
On Fri, Dec 14, 2018 at 12:57 PM Tom Lane wrote: > I seem to recall discussions about having crash recovery go around > and clean out temp tables. That seems like a better plan than > penalizing every session start with this. Well, crash recovery already removes the files, but it can't really

Re: removal of dangling temp tables

2018-12-14 Thread Tom Lane
Robert Haas writes: > On Fri, Dec 14, 2018 at 11:29 AM Alvaro Herrera > wrote: >> I think the best way to fix this is to call RemoveTempRelations() >> unconditionally at session start (without doing the rest of the temp >> table setup, just the removal.) > That would certainly simplify things.

Re: removal of dangling temp tables

2018-12-14 Thread Robert Haas
On Fri, Dec 14, 2018 at 12:27 PM Alvaro Herrera wrote: > Hmm, I think in the case covered by your commit, that is a session that > crashes with a few thousands of temp tables, this new patch might cause > a failure to open a new session altogether. Oh, good point. Or if the catalog is

Re: removal of dangling temp tables

2018-12-14 Thread Alvaro Herrera
On 2018-Dec-14, Robert Haas wrote: > On Fri, Dec 14, 2018 at 11:29 AM Alvaro Herrera > wrote: > > I think the best way to fix this is to call RemoveTempRelations() > > unconditionally at session start (without doing the rest of the temp > > table setup, just the removal.) > > That would

Re: removal of dangling temp tables

2018-12-14 Thread Robert Haas
On Fri, Dec 14, 2018 at 11:29 AM Alvaro Herrera wrote: > I think the best way to fix this is to call RemoveTempRelations() > unconditionally at session start (without doing the rest of the temp > table setup, just the removal.) That would certainly simplify things. I think I thought about that

removal of dangling temp tables

2018-12-14 Thread Alvaro Herrera
We recently ran into a funny situation, where autovacuum would not remove very old temp tables. The relfrozenxid of those tables was about to reach the max freeze age, so monitoring started to complain. It turned out that autovacuum saw that the backendId was used by a live backend ... but that