Re: removal of dangling temp tables

2018-12-28 Thread Alvaro Herrera
On 2018-Dec-28, Tom Lane wrote: > I wrote: > > Alvaro Herrera writes: > >> I looked at > >> https://github.com/postgrespro/pg_wait_sampling/blob/master/pg_wait_sampling.c > >> https://github.com/citusdata/citus/search?q=pgproc_q=pgproc > >> and skimmed a few others can't find any instance where

Re: removal of dangling temp tables

2018-12-28 Thread Tom Lane
I wrote: > Alvaro Herrera writes: >> I looked at >> https://github.com/postgrespro/pg_wait_sampling/blob/master/pg_wait_sampling.c >> https://github.com/citusdata/citus/search?q=pgproc_q=pgproc >> and skimmed a few others can't find any instance where the full struct >> is used, as opposed to

Re: removal of dangling temp tables

2018-12-28 Thread Tom Lane
Alvaro Herrera writes: > On 2018-Dec-28, Alvaro Herrera wrote: >> On 2018-Dec-28, Michael Paquier wrote: >>> There are many exotic extensions which could be using sizeof(PGPROC) >>> as that's a popular structure, >> Can you show one instance of this? > I looked at >

Re: removal of dangling temp tables

2018-12-28 Thread Alvaro Herrera
On 2018-Dec-28, Alvaro Herrera wrote: > On 2018-Dec-28, Michael Paquier wrote: > > > On Fri, Dec 28, 2018 at 12:05:34AM -0300, Alvaro Herrera wrote: > > > The server code allocates arrays, but that's fine because that code is > > > recompiled. Extensions only pass pointers around -- they don't

Re: removal of dangling temp tables

2018-12-28 Thread Alvaro Herrera
On 2018-Dec-28, Michael Paquier wrote: > On Fri, Dec 28, 2018 at 12:05:34AM -0300, Alvaro Herrera wrote: > > The server code allocates arrays, but that's fine because that code is > > recompiled. Extensions only pass pointers around -- they don't create > > any additional arrays. > > There are

Re: removal of dangling temp tables

2018-12-27 Thread Michael Paquier
On Fri, Dec 28, 2018 at 12:05:34AM -0300, Alvaro Herrera wrote: > The server code allocates arrays, but that's fine because that code is > recompiled. Extensions only pass pointers around -- they don't create > any additional arrays. There are many exotic extensions which could be using

Re: removal of dangling temp tables

2018-12-27 Thread Alvaro Herrera
On 2018-Dec-28, Michael Paquier wrote: > On Thu, Dec 27, 2018 at 04:30:21PM -0300, Alvaro Herrera wrote: > > We allow structs to receive new members at the end of the struct, since > > this doesn't affect the offset of existing members; thus code already > > compiled with the previous struct

Re: removal of dangling temp tables

2018-12-27 Thread Michael Paquier
On Thu, Dec 27, 2018 at 04:30:21PM -0300, Alvaro Herrera wrote: > We allow structs to receive new members at the end of the struct, since > this doesn't affect the offset of existing members; thus code already > compiled with the previous struct definition does not break. AFAICS > there is no

Re: removal of dangling temp tables

2018-12-27 Thread Alvaro Herrera
On 2018-Dec-27, Michael Paquier wrote: > On Wed, Dec 26, 2018 at 08:51:56PM -0300, Alvaro Herrera wrote: > > Having been victim of ABI incompatibility myself, I loathe giving too > > much priority to other issues that can be resolved in other ways, so I > > don't necessarily support your view on

Re: removal of dangling temp tables

2018-12-26 Thread Michael Paquier
On Wed, Dec 26, 2018 at 08:51:56PM -0300, Alvaro Herrera wrote: > Having been victim of ABI incompatibility myself, I loathe giving too > much priority to other issues that can be resolved in other ways, so I > don't necessarily support your view on bugs. > That said, I think in this case it

RE: removal of dangling temp tables

2018-12-26 Thread Tsunakawa, Takayuki
From: Alvaro Herrera [mailto:alvhe...@2ndquadrant.com] > Having been victim of ABI incompatibility myself, I loathe giving too much > priority to other issues that can be resolved in other ways, so I don't > necessarily support your view on bugs. > That said, I think in this case it shouldn't be a

Re: removal of dangling temp tables

2018-12-26 Thread Alvaro Herrera
On 2018-Dec-26, Tsunakawa, Takayuki wrote: > From: Alvaro Herrera [mailto:alvhe...@2ndquadrant.com] > > The more aggressive action is to backpatch 943576bddcb5 ("Make autovacuum > > more aggressive to remove orphaned temp tables") which is currently only > > in pg11. We would put the new PGPROC

RE: removal of dangling temp tables

2018-12-26 Thread Tsunakawa, Takayuki
From: Alvaro Herrera [mailto:alvhe...@2ndquadrant.com] > The more aggressive action is to backpatch 943576bddcb5 ("Make autovacuum > more aggressive to remove orphaned temp tables") which is currently only > in pg11. We would put the new PGPROC member at the end of the struct, to > avoid ABI

Re: removal of dangling temp tables

2018-12-26 Thread Alvaro Herrera
On 2018-Dec-16, Michael Paquier wrote: > On Sat, Dec 15, 2018 at 09:51:31AM -0500, Tom Lane wrote: > > Alvaro Herrera writes: > >> Oh, we already have it! Sorry, I overlooked it. With that, it seems > >> the patch is fairly simple ... I wonder about the locking implications > >> in autovacuum,

Re: removal of dangling temp tables

2018-12-15 Thread Michael Paquier
On Sat, Dec 15, 2018 at 09:51:31AM -0500, Tom Lane wrote: > Alvaro Herrera writes: >> Oh, we already have it! Sorry, I overlooked it. With that, it seems >> the patch is fairly simple ... I wonder about the locking implications >> in autovacuum, though -- the value is set in backends without

Re: removal of dangling temp tables

2018-12-15 Thread Tom Lane
Alvaro Herrera writes: > On 2018-Dec-15, Michael Paquier wrote: >> Isn't that what tempNamespaceId can be used for in PGPROC now? The flag >> would be set only when a backend creates a new temporary schema so as it >> can be tracked as the owner of the schema. > Oh, we already have it! Sorry,

Re: removal of dangling temp tables

2018-12-15 Thread Alvaro Herrera
On 2018-Dec-15, Michael Paquier wrote: > On Fri, Dec 14, 2018 at 11:06:32PM -0300, Alvaro Herrera wrote: > > I did propose in my OP the idea of a PGPROC boolean flag that indicates > > whether the temp namespace has been set up. If not, have autovac remove > > those tables. I like this option

Re: removal of dangling temp tables

2018-12-14 Thread Michael Paquier
On Fri, Dec 14, 2018 at 11:06:32PM -0300, Alvaro Herrera wrote: > I did propose in my OP the idea of a PGPROC boolean flag that indicates > whether the temp namespace has been set up. If not, have autovac remove > those tables. I like this option better, but I fear it adds more > ProcArrayLock

Re: removal of dangling temp tables

2018-12-14 Thread Alvaro Herrera
On 2018-Dec-14, Robert Haas wrote: > On Fri, Dec 14, 2018 at 6:35 PM Tom Lane wrote: > > Hm. It *could*, if we wanted it to run some transactions after > > finishing recovery. > > It'd have to launch a separate process per database. That would be > useful infrastructure for other things, too,

Re: removal of dangling temp tables

2018-12-14 Thread Alvaro Herrera
On 2018-Dec-14, Robert Haas wrote: > On Fri, Dec 14, 2018 at 12:27 PM Alvaro Herrera > wrote: > > Maybe it'd be better to change temp table removal to always drop > > max_locks_per_transaction objects at a time (ie. commit/start a new > > transaction every so many objects). > > We're basically

Re: removal of dangling temp tables

2018-12-14 Thread Robert Haas
On Fri, Dec 14, 2018 at 6:35 PM Tom Lane wrote: > Hm. It *could*, if we wanted it to run some transactions after > finishing recovery. It'd have to launch a separate process per database. That would be useful infrastructure for other things, too, like automatic catalog upgrades in minor

Re: removal of dangling temp tables

2018-12-14 Thread Andres Freund
Hi, On 2018-12-14 13:35:50 -0500, Tom Lane wrote: > Hm. It *could*, if we wanted it to run some transactions after > finishing recovery. How, we don't have infrastructure for changing databases yet? > But I guess I wonder why bother; if the disk > space is gone then there's not that much

Re: removal of dangling temp tables

2018-12-14 Thread Tom Lane
Robert Haas writes: > On Fri, Dec 14, 2018 at 12:57 PM Tom Lane wrote: >> I seem to recall discussions about having crash recovery go around >> and clean out temp tables. That seems like a better plan than >> penalizing every session start with this. > Well, crash recovery already removes the

Re: removal of dangling temp tables

2018-12-14 Thread Robert Haas
On Fri, Dec 14, 2018 at 12:57 PM Tom Lane wrote: > I seem to recall discussions about having crash recovery go around > and clean out temp tables. That seems like a better plan than > penalizing every session start with this. Well, crash recovery already removes the files, but it can't really

Re: removal of dangling temp tables

2018-12-14 Thread Tom Lane
Robert Haas writes: > On Fri, Dec 14, 2018 at 11:29 AM Alvaro Herrera > wrote: >> I think the best way to fix this is to call RemoveTempRelations() >> unconditionally at session start (without doing the rest of the temp >> table setup, just the removal.) > That would certainly simplify things.

Re: removal of dangling temp tables

2018-12-14 Thread Robert Haas
On Fri, Dec 14, 2018 at 12:27 PM Alvaro Herrera wrote: > Hmm, I think in the case covered by your commit, that is a session that > crashes with a few thousands of temp tables, this new patch might cause > a failure to open a new session altogether. Oh, good point. Or if the catalog is

Re: removal of dangling temp tables

2018-12-14 Thread Alvaro Herrera
On 2018-Dec-14, Robert Haas wrote: > On Fri, Dec 14, 2018 at 11:29 AM Alvaro Herrera > wrote: > > I think the best way to fix this is to call RemoveTempRelations() > > unconditionally at session start (without doing the rest of the temp > > table setup, just the removal.) > > That would

Re: removal of dangling temp tables

2018-12-14 Thread Robert Haas
On Fri, Dec 14, 2018 at 11:29 AM Alvaro Herrera wrote: > I think the best way to fix this is to call RemoveTempRelations() > unconditionally at session start (without doing the rest of the temp > table setup, just the removal.) That would certainly simplify things. I think I thought about that