Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-16 Thread Tomas Vondra
On 08/11/2018 04:18 PM, Tomas Vondra wrote: > On 08/11/2018 04:15 PM, Tom Lane wrote: >> Tomas Vondra writes: > On 08/09/2018 07:47 PM, Alvaro Herrera wrote: >> Actually, it seems to me that ApplyLogicalMappingFile is just leaking >> the file descriptor for no good reason. >> >>> I

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-14 Thread Tomas Vondra
On 8/14/18 10:05 AM, Tomas Vondra wrote: ... > I take that back - I can reproduce the crashes, both with and without the patch, all the way back to 9.6. Attached is a bunch of backtraces from various versions. There's a bit of variability depending on which pgbench script gets started

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-14 Thread Tomas Vondra
On 08/14/2018 01:49 PM, Tomas Vondra wrote: On 08/13/2018 04:49 PM, Andres Freund wrote: Hi, On 2018-08-13 11:46:30 -0300, Alvaro Herrera wrote: On 2018-Aug-11, Tomas Vondra wrote: Hmmm, it's difficult to compare "bt full" output, but my backtraces look somewhat different (and all the

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-14 Thread Tomas Vondra
On 08/13/2018 04:49 PM, Andres Freund wrote: Hi, On 2018-08-13 11:46:30 -0300, Alvaro Herrera wrote: On 2018-Aug-11, Tomas Vondra wrote: Hmmm, it's difficult to compare "bt full" output, but my backtraces look somewhat different (and all the backtraces I'm seeing are 100% exactly the same).

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-13 Thread Andres Freund
On 2018-08-13 11:50:41 -0300, Alvaro Herrera wrote: > On 2018-Aug-11, Andres Freund wrote: > > > On 2018-08-11 00:46:25 -0700, Andres Freund wrote: > > > Below you can find the bt full showing a bunch of nested invalidations. > > > Looking. > > > > Hm. I wonder if the attached fixes the issue

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-13 Thread Alvaro Herrera
On 2018-Aug-11, Andres Freund wrote: > On 2018-08-11 00:46:25 -0700, Andres Freund wrote: > > Below you can find the bt full showing a bunch of nested invalidations. > > Looking. > > Hm. I wonder if the attached fixes the issue for you. If it's this I > don't understand why this doesn't occur

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-13 Thread Andres Freund
Hi, On 2018-08-13 11:46:30 -0300, Alvaro Herrera wrote: > On 2018-Aug-11, Tomas Vondra wrote: > > > Hmmm, it's difficult to compare "bt full" output, but my backtraces look > > somewhat different (and all the backtraces I'm seeing are 100% exactly > > the same). Attached for comparison. > >

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-13 Thread Alvaro Herrera
On 2018-Aug-11, Tomas Vondra wrote: > Hmmm, it's difficult to compare "bt full" output, but my backtraces look > somewhat different (and all the backtraces I'm seeing are 100% exactly > the same). Attached for comparison. Hmm, looks similar enough to me -- at the bottom you have the executor

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-11 Thread Tomas Vondra
On 08/11/2018 04:15 PM, Tom Lane wrote: > Tomas Vondra writes: On 08/09/2018 07:47 PM, Alvaro Herrera wrote: > Actually, it seems to me that ApplyLogicalMappingFile is just leaking > the file descriptor for no good reason. > >> I think the fix can be as simple as attached ... I'm

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-11 Thread Tom Lane
Tomas Vondra writes: >>> On 08/09/2018 07:47 PM, Alvaro Herrera wrote: Actually, it seems to me that ApplyLogicalMappingFile is just leaking the file descriptor for no good reason. > I think the fix can be as simple as attached ... I'm mostly afk for the > weekend, so I'll commit &

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-11 Thread Tomas Vondra
On 08/10/2018 11:13 PM, Andres Freund wrote: > On 2018-08-10 22:57:57 +0200, Tomas Vondra wrote: >> >> >> On 08/09/2018 07:47 PM, Alvaro Herrera wrote: >>> On 2018-Aug-09, Tomas Vondra wrote: >>> I suppose there are reasons why it's done this way, and admittedly the test that happens to

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-11 Thread Tomas Vondra
On 08/11/2018 09:46 AM, Andres Freund wrote: > On 2018-08-11 00:34:15 -0700, Andres Freund wrote: >> I've run this numerous times now, and I haven't triggered it yet :/ > > Heh, I hit it literally seconds after hitting send: > > 2018-08-11 00:35:52.804 PDT [2196][9/14864] LOG: automatic

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-11 Thread Tomas Vondra
On 08/11/2018 09:34 AM, Andres Freund wrote: > Hi, > > On 2018-08-11 01:55:43 +0200, Tomas Vondra wrote: >> On 08/10/2018 11:59 PM, Tomas Vondra wrote: >>> >>> ... >>> >>> I suspect there's some other ingredient, e.g. some manipulation with the >>> subscription. Or maybe it's not needed at all

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-11 Thread Andres Freund
Hi, On 2018-08-11 00:46:25 -0700, Andres Freund wrote: > Below you can find the bt full showing a bunch of nested invalidations. > Looking. Hm. I wonder if the attached fixes the issue for you. If it's this I don't understand why this doesn't occur on older branches... I've not yet been able

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-11 Thread Andres Freund
On 2018-08-11 00:34:15 -0700, Andres Freund wrote: > I've run this numerous times now, and I haven't triggered it yet :/ Heh, I hit it literally seconds after hitting send: 2018-08-11 00:35:52.804 PDT [2196][9/14864] LOG: automatic analyze of table "postgres.pg_catalog.pg_depend" system usage:

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-11 Thread Andres Freund
Hi, On 2018-08-11 01:55:43 +0200, Tomas Vondra wrote: > On 08/10/2018 11:59 PM, Tomas Vondra wrote: > > > > ... > > > > I suspect there's some other ingredient, e.g. some manipulation with the > > subscription. Or maybe it's not needed at all and I'm just imagining things. > > > > Indeed, the

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-11 Thread Peter Geoghegan
On Sat, Aug 11, 2018 at 12:06 AM, Andres Freund wrote: > To the point that I wonder if we shouldn't just change the ERROR into a > PANIC on master (but not REL_11_STABLE), so the buildfarm gives us > feedback. I don't think the problem can fundamentally be related to > subscriptions, given the

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-11 Thread Andres Freund
Hi, On 2018-08-11 01:55:43 +0200, Tomas Vondra wrote: > On 08/10/2018 11:59 PM, Tomas Vondra wrote: > > > > ... > > > > I suspect there's some other ingredient, e.g. some manipulation with the > > subscription. Or maybe it's not needed at all and I'm just imagining things. > > > > Indeed, the

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-10 Thread Tomas Vondra
On 08/10/2018 11:59 PM, Tomas Vondra wrote: > > ... > > I suspect there's some other ingredient, e.g. some manipulation with the > subscription. Or maybe it's not needed at all and I'm just imagining things. > Indeed, the manipulation with the subscription seems to be the key here. I pretty

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-10 Thread Tomas Vondra
On 08/10/2018 11:13 PM, Andres Freund wrote: > On 2018-08-10 22:57:57 +0200, Tomas Vondra wrote: >> >> >> On 08/09/2018 07:47 PM, Alvaro Herrera wrote: >>> On 2018-Aug-09, Tomas Vondra wrote: >>> I suppose there are reasons why it's done this way, and admittedly the test that happens

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-10 Thread Andres Freund
On 2018-08-10 22:57:57 +0200, Tomas Vondra wrote: > > > On 08/09/2018 07:47 PM, Alvaro Herrera wrote: > > On 2018-Aug-09, Tomas Vondra wrote: > > > >> I suppose there are reasons why it's done this way, and admittedly the test > >> that happens to trigger this is a bit extreme (essentially

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-10 Thread Tomas Vondra
On 08/09/2018 07:47 PM, Alvaro Herrera wrote: > On 2018-Aug-09, Tomas Vondra wrote: > >> I suppose there are reasons why it's done this way, and admittedly the test >> that happens to trigger this is a bit extreme (essentially running pgbench >> concurrently with 'vacuum full pg_class' in a

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-09 Thread Tomas Vondra
On 08/09/2018 07:47 PM, Alvaro Herrera wrote: On 2018-Aug-09, Tomas Vondra wrote: I suppose there are reasons why it's done this way, and admittedly the test that happens to trigger this is a bit extreme (essentially running pgbench concurrently with 'vacuum full pg_class' in a loop). I'm

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-09 Thread Alvaro Herrera
On 2018-Aug-09, Tom Lane wrote: > It sounds like whoever wrote that code was completely impervious to the > API spec for AllocateFile(): > > * Note that files that will be open for any significant length of time > * should NOT be handled this way, since they cannot share kernel file > *

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-09 Thread Alvaro Herrera
On 2018-Aug-09, Tomas Vondra wrote: > I suppose there are reasons why it's done this way, and admittedly the test > that happens to trigger this is a bit extreme (essentially running pgbench > concurrently with 'vacuum full pg_class' in a loop). I'm not sure it's > extreme enough to deem it not

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-09 Thread Tom Lane
Tomas Vondra writes: > While investigating an issue with rewrite maps in logical decoding, I > found it's pretty darn trivial to hit this: > ERROR: 53000: exceeded maxAllocatedDescs (492) while trying to open > file "pg_logical/mappings/map-4000-4eb-1_60DE1E08-5376b5-537c6b" > This