Re: A performance issue with Memoize

2024-02-01 Thread Richard Guo
On Thu, Feb 1, 2024 at 3:43 PM Anthonin Bonnefoy < anthonin.bonne...@datadoghq.com> wrote: > Hi, > > I've seen a similar issue with the following query (tested on the current > head): > > EXPLAIN ANALYZE SELECT * FROM tenk1 t1 > LEFT JOIN LATERAL (SELECT t1.two, tenk2.hundred, tenk2.two FROM tenk2

Re: A performance issue with Memoize

2024-01-31 Thread Anthonin Bonnefoy
Hi, I've seen a similar issue with the following query (tested on the current head): EXPLAIN ANALYZE SELECT * FROM tenk1 t1 LEFT JOIN LATERAL (SELECT t1.two, tenk2.hundred, tenk2.two FROM tenk2) t2 ON t1.hundred = t2.hundred WHERE t1.hundred < 5;

Re: A performance issue with Memoize

2024-01-26 Thread Alexander Lakhin
Hello, 27.01.2024 00:09, Tom Lane wrote: David Rowley writes: On Sat, 27 Jan 2024 at 09:41, Tom Lane wrote: drongo and fairywren are consistently failing the test case added by this commit. I'm not quite sure why the behavior of Memoize would be platform-specific when we're dealing with int

Re: A performance issue with Memoize

2024-01-26 Thread Tom Lane
David Rowley writes: > On Sat, 27 Jan 2024 at 09:41, Tom Lane wrote: >> drongo and fairywren are consistently failing the test case added >> by this commit. I'm not quite sure why the behavior of Memoize >> would be platform-specific when we're dealing with integers, >> but ... > Maybe snprintf

Re: A performance issue with Memoize

2024-01-26 Thread David Rowley
On Sat, 27 Jan 2024 at 09:41, Tom Lane wrote: > > David Rowley writes: > > I've adjusted the comments to what you mentioned and also leaned out > > the pretty expensive test case to something that'll run much faster > > and pushed the result. > > drongo and fairywren are consistently failing the

Re: A performance issue with Memoize

2024-01-26 Thread Tom Lane
David Rowley writes: > I've adjusted the comments to what you mentioned and also leaned out > the pretty expensive test case to something that'll run much faster > and pushed the result. drongo and fairywren are consistently failing the test case added by this commit. I'm not quite sure why the

Re: A performance issue with Memoize

2024-01-25 Thread David Rowley
On Fri, 26 Jan 2024 at 19:03, Richard Guo wrote: > At first I wondered if we should assume that the same param expr must > have the same equality operator. If not, we should also check the > operator to tell if the cache key is a duplicate, like > > - if (!list_member(*param_exprs, expr)

Re: A performance issue with Memoize

2024-01-25 Thread Richard Guo
On Fri, Jan 26, 2024 at 12:18 PM David Rowley wrote: > On Fri, 26 Jan 2024 at 16:51, Tom Lane wrote: > > >> However ... it seems like we're not out of the woods yet. Why > > >> is Richard's proposed test case still showing > > >> + -> Memoize (actual rows=5000 loops=N) > > >> +

Re: A performance issue with Memoize

2024-01-25 Thread Richard Guo
On Fri, Jan 26, 2024 at 1:22 AM Tom Lane wrote: > Apologies for not having noticed this thread before. I'm taking > a look at it now. However, while sniffing around this I found > what seems like an oversight in paramassign.c's > assign_param_for_var(): it says it should compare all the same >

Re: A performance issue with Memoize

2024-01-25 Thread David Rowley
On Fri, 26 Jan 2024 at 16:51, Tom Lane wrote: > >> However ... it seems like we're not out of the woods yet. Why > >> is Richard's proposed test case still showing > >> + -> Memoize (actual rows=5000 loops=N) > >> + Cache Key: t1.two, t1.two > >> Seems like there is missing

Re: A performance issue with Memoize

2024-01-25 Thread Tom Lane
David Rowley writes: > I've adjusted the comments to what you mentioned and also leaned out > the pretty expensive test case to something that'll run much faster > and pushed the result. +1, I was wondering if the test could be cheaper. It wasn't horrid as Richard had it, but core regression tes

Re: A performance issue with Memoize

2024-01-25 Thread David Rowley
On Fri, 26 Jan 2024 at 07:32, Tom Lane wrote: > > David Rowley writes: > > I'd feel better about doing it your way if Tom could comment on if > > there was a reason he put the function calls that way around in > > 5ebaaa494. > > I'm fairly sure I thought it wouldn't matter because of the Param >

Re: A performance issue with Memoize

2024-01-25 Thread Richard Guo
On Fri, Jan 26, 2024 at 2:32 AM Tom Lane wrote: > I'm fairly sure I thought it wouldn't matter because of the Param > de-duplication done in paramassign.c. However, Richard's example > shows that's not so, because process_subquery_nestloop_params is > picky about the param ID assigned to a parti

Re: A performance issue with Memoize

2024-01-25 Thread Tom Lane
David Rowley writes: > I think fixing it your way makes sense. I don't really see any reason > why we should have two. However... > Another way it *could* be fixed would be to get rid of pull_paramids() > and change create_memoize_plan() to set keyparamids to all the param > IDs that match are e

Re: A performance issue with Memoize

2024-01-25 Thread Tom Lane
David Rowley writes: > I'd feel better about doing it your way if Tom could comment on if > there was a reason he put the function calls that way around in > 5ebaaa494. Apologies for not having noticed this thread before. I'm taking a look at it now. However, while sniffing around this I found

Re: A performance issue with Memoize

2024-01-24 Thread David Rowley
On Fri, 20 Oct 2023 at 23:40, Richard Guo wrote: > The Memoize runtime stats 'Hits: 0 Misses: 1 Evictions: ' > seems suspicious to me, so I've looked into it a little bit, and found > that the MemoizeState's keyparamids and its outerPlan's chgParam are > always different, and that makes

Re: A performance issue with Memoize

2024-01-17 Thread Arne Roland
Hi Richard, I can tell this a real world problem. I have seen this multiple times in production. The fix seems surprisingly simple. I hope my questions here aren't completely off. I still struggle to think about the implications. I wonder, if there is any stuff we are breaking by bluntly f

Re: A performance issue with Memoize

2023-10-30 Thread Richard Guo
On Tue, Oct 31, 2023 at 1:36 PM Andrei Lepikhov wrote: > On 30/10/2023 14:55, Richard Guo wrote: > > > > On Thu, Oct 26, 2023 at 12:07 PM Andrei Lepikhov > > mailto:a.lepik...@postgrespro.ru>> wrote: > > > > Do you've thought about the case, fixed with the commit 1db5667? As I > > see, th

Re: A performance issue with Memoize

2023-10-30 Thread Andrei Lepikhov
On 30/10/2023 14:55, Richard Guo wrote: On Thu, Oct 26, 2023 at 12:07 PM Andrei Lepikhov mailto:a.lepik...@postgrespro.ru>> wrote: Do you've thought about the case, fixed with the commit 1db5667? As I see, that bugfix still isn't covered by regression tests. Could your approach of

Re: A performance issue with Memoize

2023-10-30 Thread Richard Guo
On Thu, Oct 26, 2023 at 12:07 PM Andrei Lepikhov wrote: > Do you've thought about the case, fixed with the commit 1db5667? As I > see, that bugfix still isn't covered by regression tests. Could your > approach of a PARAM_EXEC slot reusing break that case? Hm, I don't think so. The issue fixed

Re: A performance issue with Memoize

2023-10-25 Thread Andrei Lepikhov
On 20/10/2023 17:40, Richard Guo wrote: I noticed $subject with the query below. set enable_memoize to off; explain (analyze, costs off) select * from tenk1 t1 left join lateral     (select t1.two as t1two, * from tenk1 t2 offset 0) s on t1.two = s.two;                                      QU

Re: A performance issue with Memoize

2023-10-24 Thread Richard Guo
On Fri, Oct 20, 2023 at 7:43 PM Pavel Stehule wrote: > +1 > > it would be great to fix this problem - I've seen this issue a few times. > Thanks for the input. I guess this is not rare in the real world. If the subquery contains lateral reference to a Var that also appears in the subquery's jo

Re: A performance issue with Memoize

2023-10-24 Thread Richard Guo
On Fri, Oct 20, 2023 at 6:40 PM Richard Guo wrote: > I haven't thought thoroughly about the fix yet. But one way I'm > thinking is that in create_subqueryscan_plan() we can first add the > subquery's subplan_params to root->curOuterParams, and then replace > outer-relation Vars in scan_clauses a

Re: A performance issue with Memoize

2023-10-20 Thread Pavel Stehule
pá 20. 10. 2023 v 13:01 odesílatel Richard Guo napsal: > I noticed $subject with the query below. > > set enable_memoize to off; > > explain (analyze, costs off) > select * from tenk1 t1 left join lateral > (select t1.two as t1two, * from tenk1 t2 offset 0) s > on t1.two = s.two; >

A performance issue with Memoize

2023-10-20 Thread Richard Guo
I noticed $subject with the query below. set enable_memoize to off; explain (analyze, costs off) select * from tenk1 t1 left join lateral (select t1.two as t1two, * from tenk1 t2 offset 0) s on t1.two = s.two; QUERY PLAN ---