Re: Should pg 11 use a lot more memory building an spgist index?

2018-11-02 Thread Bruno Wolff III
I see that a fix got committed. Thanks! I'll double check it after the point release comes out (which looks like it will be next week) and let you know if there is still a problem.

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-31 Thread Tom Lane
Amit Langote writes: > Maybe, we don't need to spoil the interface of index_beginscan with the > new memory context argument like my patch does if the simple following of > its contract by amendscan would suffice. Yeah, I'm not enamored of changing the API of index_beginscan for this; the

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-30 Thread Amit Langote
On 2018/10/30 21:27, Amit Langote wrote: > On Tue, Oct 30, 2018 at 7:11 PM Amit Langote >> I've tried to fix that with the attached patches. >> >> 0001 adds the ability for the callers of index_beginscan to specify a >> memory context. index_beginscan_internals switches to that context before >>

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-30 Thread Amit Langote
On Tue, Oct 30, 2018 at 7:11 PM Amit Langote wrote: > > On 2018/10/30 4:48, Tom Lane wrote: > > I wrote: > >> Alvaro Herrera writes: > >>> How about modifying SysScanDescData to have a memory context member, > >>> which is created by systable_beginscan and destroyed by endscan? > > > >> I think

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-30 Thread Amit Langote
On 2018/10/30 4:48, Tom Lane wrote: > I wrote: >> Alvaro Herrera writes: >>> How about modifying SysScanDescData to have a memory context member, >>> which is created by systable_beginscan and destroyed by endscan? > >> I think it would still have problems, in that it would affect in which >>

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-30 Thread Amit Langote
On 2018/10/30 4:48, Tom Lane wrote: > I was confused about why the memory leak in Bruno's example is so much > larger in HEAD than v11; spgbeginscan does allocate more stuff than > before, but only if numberOfOrderBys > 0, which surely doesn't apply for > the exclusion-check code path. Eventually

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-29 Thread Tom Lane
I wrote: > Alvaro Herrera writes: >> How about modifying SysScanDescData to have a memory context member, >> which is created by systable_beginscan and destroyed by endscan? > I think it would still have problems, in that it would affect in which > context index_getnext delivers its output. We

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Bruno Wolff III
On Fri, Oct 26, 2018 at 13:44:07 +0100, Tom Lane wrote: Bruno Wolff III writes: As a short term work around, could I create the index first and use insert statements, each in their own transaction, to get the table loaded with the index? Yes; it might also be that you don't even need to

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Tom Lane
Alvaro Herrera writes: > On 2018-Oct-26, Tom Lane wrote: >> After a quick look around, I think that making systable_begin/endscan >> do this is a nonstarter; there are just too many call sites that would >> be affected. Now, you could imagine specifying that indexes on system >> catalogs (in

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Alvaro Herrera
On 2018-Oct-26, Tom Lane wrote: > After a quick look around, I think that making systable_begin/endscan > do this is a nonstarter; there are just too many call sites that would > be affected. Now, you could imagine specifying that indexes on system > catalogs (in practice, only btree) have to

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Tom Lane
Bruno Wolff III writes: > Tom Lane wrote: >> Hmm, in my hands this produces the same size leak (~28GB) in either v10 >> or v11. In HEAD, somebody's made it even worse (~43GB). So this is >> certainly pretty broken, but I'm not sure why it seems worse to you in >> v11 than before. > As a

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Bruno Wolff III
On Fri, Oct 26, 2018 at 10:16:09 +0100, Tom Lane wrote: Bruno Wolff III writes: I have something that seems to produce it on rhel7. Fedora isn't working well either, but the difference may be due to postgresql.conf being different or some difference in the Fedora build. Hmm, in my hands

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Tom Lane
Amit Langote writes: > On 2018/10/26 18:59, Tom Lane wrote: >> After a quick look around, I think that making systable_begin/endscan >> do this is a nonstarter; there are just too many call sites that would >> be affected. Now, you could imagine specifying that indexes on system >> catalogs (in

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Amit Langote
On 2018/10/26 18:59, Tom Lane wrote: > Amit Langote writes: >> On 2018/10/26 18:16, Tom Lane wrote: >>> A quick review of the other index AM endscan methods seems to indicate >>> that they all try to clean up their mess. So maybe we should just make >>> spgendscan do likewise. Alternatively, we

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Tom Lane
Amit Langote writes: > On 2018/10/26 18:16, Tom Lane wrote: >> A quick review of the other index AM endscan methods seems to indicate >> that they all try to clean up their mess. So maybe we should just make >> spgendscan do likewise. Alternatively, we could decide that requiring >> endscan

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Amit Langote
On 2018/10/26 18:16, Tom Lane wrote: > The core of the problem I see is that check_exclusion_or_unique_constraint > does index_beginscan/index_rescan/index_endscan in a long-lived context, > while spgendscan seems to have employed dice while deciding which of > spgbeginscan's allocations it would

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Tom Lane
Bruno Wolff III writes: > I have something that seems to produce it on rhel7. Fedora isn't working > well either, but the difference may be due to postgresql.conf being > different or some difference in the Fedora build. Hmm, in my hands this produces the same size leak (~28GB) in either v10