Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-07-11 Thread Amit Langote
Horiguchi-san, Thanks for the comment. My reply is a bit late now, but On Wed, Jul 10, 2019 at 5:39 PM Kyotaro Horiguchi wrote: > At Wed, 10 Jul 2019 16:35:18 +0900, Amit Langote > wrote: > - * Switch to appropriate context for constructing querytrees (again, > - * these must

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-07-11 Thread Amit Langote
On Thu, Jul 11, 2019 at 3:46 AM Tom Lane wrote: > > Amit Langote writes: > > Attached updated patch. Thanks again. > > Pushed with a bit of further cleanup Thanks a lot. > --- most notably, the way > you had execute_sql_string(), it was still leaking any cruft > ProcessUtility might generate.

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-07-10 Thread Tom Lane
Amit Langote writes: > Attached updated patch. Thanks again. Pushed with a bit of further cleanup --- most notably, the way you had execute_sql_string(), it was still leaking any cruft ProcessUtility might generate. We can fix that by running ProcessUtility in the per-statement context too. I

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-07-10 Thread Kyotaro Horiguchi
Hi, At Wed, 10 Jul 2019 16:35:18 +0900, Amit Langote wrote in > On Tue, Jul 9, 2019 at 6:21 AM Tom Lane wrote: > > > > Amit Langote writes: > > > [ parse-plan-memcxt_v2.patch ] > > > > I got around to looking at this finally. > > Thanks for the review. > > > I'm not at all happy with > >

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-07-10 Thread Amit Langote
On Tue, Jul 9, 2019 at 6:21 AM Tom Lane wrote: > > Amit Langote writes: > > [ parse-plan-memcxt_v2.patch ] > > I got around to looking at this finally. Thanks for the review. > I'm not at all happy with > the fact that it's added a plantree copy step to the only execution > path through

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-07-08 Thread Tom Lane
Amit Langote writes: > [ parse-plan-memcxt_v2.patch ] I got around to looking at this finally. I'm not at all happy with the fact that it's added a plantree copy step to the only execution path through exec_simple_query. That's a very significant overhead, in many use-cases, to solve something

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-07-08 Thread Julien Rouhaud
Hi Amit, On Mon, Jul 8, 2019 at 10:52 AM Amit Langote wrote: > > On Thu, Jul 4, 2019 at 6:52 PM Julien Rouhaud wrote: > > On Tue, May 28, 2019 at 6:57 AM Amit Langote wrote: > > > >> Maybe like the attached? I'm not sure if we need to likewise be > > > >> concerned > > > >> about

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-07-08 Thread Amit Langote
Hi Julien, Thanks for taking a look at this. On Thu, Jul 4, 2019 at 6:52 PM Julien Rouhaud wrote: > On Tue, May 28, 2019 at 6:57 AM Amit Langote wrote: > > >> Maybe like the attached? I'm not sure if we need to likewise be > > >> concerned > > >> about exec_sql_string() being handed

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-07-04 Thread Julien Rouhaud
On Tue, May 28, 2019 at 6:57 AM Amit Langote wrote: > > On 2019/05/27 21:56, Tom Lane wrote: > > Amit Langote writes: > >> On 2019/05/24 23:28, Tom Lane wrote: > >>> So my thought, if we want to do something about this, is not "find > >>> some things we can pfree at the end of planning" but

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-05-27 Thread Amit Langote
On 2019/05/27 21:56, Tom Lane wrote: > Amit Langote writes: >> On 2019/05/24 23:28, Tom Lane wrote: >>> So my thought, if we want to do something about this, is not "find >>> some things we can pfree at the end of planning" but "find a way >>> to use a separate context for each statement in the

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-05-27 Thread Tom Lane
Amit Langote writes: > On 2019/05/24 23:28, Tom Lane wrote: >> So my thought, if we want to do something about this, is not "find >> some things we can pfree at the end of planning" but "find a way >> to use a separate context for each statement in the query string". > Maybe like the attached?

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-05-27 Thread Amit Langote
Hi, On 2019/05/24 21:18, Joe Conway wrote: > On 5/24/19 1:47 AM, Amit Langote wrote: >> I too have had similar thoughts on the matter. If the planner had built >> all its subsidiary data structures in its own private context (or tree of >> contexts) which is reset once a plan for a given query

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-05-26 Thread Amit Langote
On 2019/05/24 23:28, Tom Lane wrote: > So my thought, if we want to do something about this, is not "find > some things we can pfree at the end of planning" but "find a way > to use a separate context for each statement in the query string". > Maybe multi-query strings could be handled by setting

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-05-24 Thread Joe Conway
On 5/24/19 10:28 AM, Tom Lane wrote: > Joe Conway writes: >> On 5/24/19 9:33 AM, David Rowley wrote: >>> For it to have regressed it would have had to once have been better, >>> but where was that mentioned? The only thing I saw was >>> non-partitioned tables compared to partitioned tables, but

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-05-24 Thread Tom Lane
Joe Conway writes: > On 5/24/19 9:33 AM, David Rowley wrote: >> For it to have regressed it would have had to once have been better, >> but where was that mentioned? The only thing I saw was >> non-partitioned tables compared to partitioned tables, but you can't >> really say it's a regression

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-05-24 Thread Joe Conway
On 5/24/19 9:33 AM, David Rowley wrote: > On Sat, 25 May 2019 at 00:18, Joe Conway wrote: >> I admittedly haven't followed this thread too closely, but if having 100 >> partitions causes out of memory on pg11, that sounds like a massive >> regression to me. > > For it to have regressed it would

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-05-24 Thread David Rowley
On Sat, 25 May 2019 at 00:18, Joe Conway wrote: > I admittedly haven't followed this thread too closely, but if having 100 > partitions causes out of memory on pg11, that sounds like a massive > regression to me. For it to have regressed it would have had to once have been better, but where was

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-05-24 Thread Joe Conway
On 5/24/19 1:47 AM, Amit Langote wrote: > On 2019/05/23 4:15, Andreas Seltenreich wrote: >> …but when doing it on the parent relation, even 100 statements are >> enough to exceed the limit: >> >> , >> | $ psql -c "$(yes update t set c=c where c=6 \; | head -n 100)" >> | FEHLER: Speicher

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-05-23 Thread Amit Langote
Hi, On 2019/05/23 4:15, Andreas Seltenreich wrote: > …but when doing it on the parent relation, even 100 statements are > enough to exceed the limit: > > , > | $ psql -c "$(yes update t set c=c where c=6 \; | head -n 100)" > | FEHLER: Speicher aufgebraucht > | DETAIL: Failed on request of

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-05-23 Thread Julian Schauder
Hey David, > "multiple statements in a single query", did you mean to write > session > or maybe transaction there? Maybe the wording isn't perfect. It is required that the querys are sent as a single batch. Try the exact bash-script Andreas used for updating the parent. > Which version?

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-05-23 Thread David Rowley
On Thu, 23 May 2019 at 21:19, Julian Schauder wrote: > > "multiple statements in a single query", did you mean to write > > session > > or maybe transaction there? > > Maybe the wording isn't perfect. It is required that the querys are > sent as a single batch. Try the exact bash-script Andreas

Re: Excessive memory usage in multi-statement queries w/ partitioning

2019-05-23 Thread David Rowley
On Thu, 23 May 2019 at 17:55, Andreas Seltenreich wrote: > a customer reported excessive memory usage and out-of-memory ERRORs > after introducing native partitioning in one of their databases. We > could narrow it down to the overhead introduced by the partitioning when > issuing multiple

Excessive memory usage in multi-statement queries w/ partitioning

2019-05-22 Thread Andreas Seltenreich
Hi, a customer reported excessive memory usage and out-of-memory ERRORs after introducing native partitioning in one of their databases. We could narrow it down to the overhead introduced by the partitioning when issuing multiple statements in a single query. I could reduce the problem to the