Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Bruno Wolff III
On Fri, Oct 26, 2018 at 13:44:07 +0100, Tom Lane wrote: Bruno Wolff III writes: As a short term work around, could I create the index first and use insert statements, each in their own transaction, to get the table loaded with the index? Yes; it might also be that you don't even need to

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Tom Lane
Alvaro Herrera writes: > On 2018-Oct-26, Tom Lane wrote: >> After a quick look around, I think that making systable_begin/endscan >> do this is a nonstarter; there are just too many call sites that would >> be affected. Now, you could imagine specifying that indexes on system >> catalogs (in

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Alvaro Herrera
On 2018-Oct-26, Tom Lane wrote: > After a quick look around, I think that making systable_begin/endscan > do this is a nonstarter; there are just too many call sites that would > be affected. Now, you could imagine specifying that indexes on system > catalogs (in practice, only btree) have to

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Tom Lane
Bruno Wolff III writes: > Tom Lane wrote: >> Hmm, in my hands this produces the same size leak (~28GB) in either v10 >> or v11. In HEAD, somebody's made it even worse (~43GB). So this is >> certainly pretty broken, but I'm not sure why it seems worse to you in >> v11 than before. > As a

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Bruno Wolff III
On Fri, Oct 26, 2018 at 10:16:09 +0100, Tom Lane wrote: Bruno Wolff III writes: I have something that seems to produce it on rhel7. Fedora isn't working well either, but the difference may be due to postgresql.conf being different or some difference in the Fedora build. Hmm, in my hands

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Tom Lane
Amit Langote writes: > On 2018/10/26 18:59, Tom Lane wrote: >> After a quick look around, I think that making systable_begin/endscan >> do this is a nonstarter; there are just too many call sites that would >> be affected. Now, you could imagine specifying that indexes on system >> catalogs (in

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Amit Langote
On 2018/10/26 18:59, Tom Lane wrote: > Amit Langote writes: >> On 2018/10/26 18:16, Tom Lane wrote: >>> A quick review of the other index AM endscan methods seems to indicate >>> that they all try to clean up their mess. So maybe we should just make >>> spgendscan do likewise. Alternatively, we

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Tom Lane
Amit Langote writes: > On 2018/10/26 18:16, Tom Lane wrote: >> A quick review of the other index AM endscan methods seems to indicate >> that they all try to clean up their mess. So maybe we should just make >> spgendscan do likewise. Alternatively, we could decide that requiring >> endscan

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Amit Langote
On 2018/10/26 18:16, Tom Lane wrote: > The core of the problem I see is that check_exclusion_or_unique_constraint > does index_beginscan/index_rescan/index_endscan in a long-lived context, > while spgendscan seems to have employed dice while deciding which of > spgbeginscan's allocations it would

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-26 Thread Tom Lane
Bruno Wolff III writes: > I have something that seems to produce it on rhel7. Fedora isn't working > well either, but the difference may be due to postgresql.conf being > different or some difference in the Fedora build. Hmm, in my hands this produces the same size leak (~28GB) in either v10

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-25 Thread Bruno Wolff III
On Wed, Oct 24, 2018 at 10:21:11 +0100, Tom Lane wrote: Bruno Wolff III writes: If I have a substantial database dump file to provide for reproducing this do you prefer it on a web server somewhere? I expect that mailing very large attachments to the lists is a bad idea. No, don't do that.

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-25 Thread Bruno Wolff III
It looks like it got past creating the exclude constraint based on the ordering of commands in the dump file. However creating a more normal spgist index is taking a very long time with a lot of disk wait time. CPU usage seems pretty low for the amount of time it has been working on building

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-24 Thread Bruno Wolff III
On Tue, Oct 23, 2018 at 20:23:14 -0500, Bruno Wolff III wrote: While reloading a database cluster to move from 10.5 to 11, I'm getting out of memory crashes that I did see when doing reloads on pg 10. The statement flagged in the log is this: 2018-10-23 16:44:34.815 CDT [126839] STATEMENT:

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-24 Thread Tom Lane
Bruno Wolff III writes: > If I have a substantial database dump file to provide for reproducing this > do you prefer it on a web server somewhere? I expect that mailing very > large attachments to the lists is a bad idea. No, don't do that. If you can make sample data available for download,

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-24 Thread Bruno Wolff III
On Wed, Oct 24, 2018 at 09:33:48 +0100, Tom Lane wrote: Bruno Wolff III writes: While reloading a database cluster to move from 10.5 to 11, I'm getting out of memory crashes that I did see when doing reloads on pg 10. The statement flagged in the log is this: 2018-10-23 16:44:34.815 CDT

Re: Should pg 11 use a lot more memory building an spgist index?

2018-10-24 Thread Tom Lane
Bruno Wolff III writes: > While reloading a database cluster to move from 10.5 to 11, I'm getting > out of memory crashes that I did see when doing reloads on pg 10. > The statement flagged in the log is this: > 2018-10-23 16:44:34.815 CDT [126839] STATEMENT: ALTER TABLE ONLY >