Re: using extended statistics to improve join estimates

2024-05-22 Thread Andrei Lepikhov
On 5/23/24 09:04, Andy Fan wrote: Andrei Lepikhov writes: * c) No extended stats with MCV. If there are multiple join clauses, * we can try using ndistinct coefficients and do what eqjoinsel does. OK, I didn't pay enough attention to this comment before. and yes, I get the same conclusion

Re: POC: GROUP BY optimization

2024-05-21 Thread Andrei Lepikhov
of sorted columns. -- regards, Andrei Lepikhov Postgres Professional

Re: using extended statistics to improve join estimates

2024-05-21 Thread Andrei Lepikhov
On 5/20/24 16:40, Andrei Lepikhov wrote: On 20/5/2024 15:52, Andy Fan wrote: +    if (clauselist_selectivity_hook) +    *return* clauselist_selectivity_hook(root, clauses, ..) Of course - library may estimate not all the clauses - it is a reason, why I added input/output parameter

Re: using extended statistics to improve join estimates

2024-05-20 Thread Andrei Lepikhov
he clauses - it is a reason, why I added input/output parameter 'estimatedclauses' by analogy with statext_clauselist_selectivity. -- regards, Andrei Lepikhov Postgres Professional

Re: using extended statistics to improve join estimates

2024-05-20 Thread Andrei Lepikhov
. -- regards, Andrei Lepikhov Postgres Professional diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c index 0ab021c1e8..271d36a522 100644 --- a/src/backend/optimizer/path/clausesel.c +++ b/src/backend/optimizer/path/clausesel.c @@ -128,6 +128,9

Re: POC: GROUP BY optimization

2024-05-16 Thread Andrei Lepikhov
ebuke - does behavior still the same?. 2. Could you try to find the reason? -- regards, Andrei Lepikhov Postgres Professional

Re: query_id, pg_stat_activity, extended query protocol

2024-05-15 Thread Andrei Lepikhov
se to move pgstat_report_query_id to the ExecutorRun routine? -- regards, Andrei Lepikhov

Re: query_id, pg_stat_activity, extended query protocol

2024-05-15 Thread Andrei Lepikhov
hat's what I precisely wanted to understand: queryId doesn't have semantics to show the job that consumes resources right now—it is mostly about convenience to know that the backend processes nothing except (probably) this query. -- regards, Andrei Lepikhov

Re: query_id, pg_stat_activity, extended query protocol

2024-05-08 Thread Andrei Lepikhov
reset on abortion. -- regards, Andrei Lepikhov diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index 4d7c92d63c..a4d38a157f 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -470,6 +470,12 @@ ExecutorEnd(QueryDesc *queryDesc

Re: Removing unneeded self joins

2024-05-06 Thread Andrei Lepikhov
On 6/5/2024 21:44, Robert Haas wrote: On Sat, May 4, 2024 at 10:46 PM Andrei Lepikhov wrote: Having no objective negative feedback, we have no reason to change anything in the design or any part of the code. It looks regrettable and unusual. To me, this sounds like you think it's someone

Re: Removing unneeded self joins

2024-05-04 Thread Andrei Lepikhov
migration policy in some countries, it would be better to work through the online presence as equivalent to offline. Without an online part of the conference, the only way to communicate and discuss is through this mailing list. -- regards, Andrei Lepikhov

Re: Removing unneeded self joins

2024-05-02 Thread Andrei Lepikhov
ements in the planner. As a result, I vote to stay with the feature. But remember, as an author, I'm not entirely objective, so let's wait for alternative opinions. [1] Make Vars be outer-join-aware [2] Improve performance of ORDER BY / DISTINCT aggregates -- regards, Andrei Lepikhov Postgres Professional

Re: Removing unneeded self joins

2024-05-02 Thread Andrei Lepikhov
this subquery and don't execute SJE. Do we need more code, as you have written in the first patch? -- regards, Andrei Lepikhov Postgres Professional From dac8afd2095586921dfcf5564e7f2979e89b77de Mon Sep 17 00:00:00 2001 From: "Andrei V. Lepikhov" Date: Thu, 2 May 2024 16:17:52 +07

Re: query_id, pg_stat_activity, extended query protocol

2024-04-27 Thread Andrei Lepikhov
be considered invalid at that point and raise an error to the user? I don't think so. It may be any object, even stored procedure, that can be changed. IMO, the right option here is to report zero (like the undefined value of queryId) until the end of the parsing stage. -- regards, Andrei Lepikhov

Re: query_id, pg_stat_activity, extended query protocol

2024-04-23 Thread Andrei Lepikhov
On 4/23/24 12:49, Michael Paquier wrote: On Tue, Apr 23, 2024 at 11:42:41AM +0700, Andrei Lepikhov wrote: On 23/4/2024 11:16, Imseih (AWS), Sami wrote: + pgstat_report_query_id(linitial_node(Query, psrc->query_list)->queryId, true); set_ps_display("BIND"); @@

Re: query_id, pg_stat_activity, extended query protocol

2024-04-22 Thread Andrei Lepikhov
hedPlan(), which will validate and lock the plan? What if some OIDs were altered in the middle? -- regards, Andrei Lepikhov

Re: POC: GROUP BY optimization

2024-04-21 Thread Andrei Lepikhov
don't need any ordering procedure at all. And the last thing here — this code introduces the basics needed to add more sophisticated strategies, like ordering according to uniqueness or preferring to set constant-width columns first in grouping. -- regards, Andrei Lepikhov Postgres Professional

Re: POC: GROUP BY optimization

2024-04-18 Thread Andrei Lepikhov
of code more resistant to blunders is another story. -- regards, Andrei Lepikhov Postgres Professional diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index aa80f6486c..9c079270ec 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/o

Re: POC: GROUP BY optimization

2024-04-17 Thread Andrei Lepikhov
ly one reason to do this: adjust_group_pathkeys_for_groupagg. Curious about how much profit we get here, I think we can discover it later with no hurry. A good outcome here will be a test case that can show the importance of arranging GROUP-BY and ORDER-BY at an early stage. -- regards, Andrei Lepikhov Postgre

Re: POC: GROUP BY optimization

2024-04-16 Thread Andrei Lepikhov
. -- regards, Andrei Lepikhov Postgres Professional diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 5d9597adcd..aa80f6486c 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -380,15 +380,18

Re: POC: GROUP BY optimization

2024-04-16 Thread Andrei Lepikhov
style. To avoid possible usage issues (in extensions, for example), I propose setting it up afterwards, explicitly forcing this action by input parameter - see my attempt in the attachment. -- regards, Andrei Lepikhov Postgres Professional diff --git a/src/backend/optimizer/path/equivclass.c b

Re: POC: GROUP BY optimization

2024-04-11 Thread Andrei Lepikhov
passed through the code being steered by the issues explained in detail. I see seven issues. Two of them definitely should be scrutinised right now, and I'm ready to do that. -- regards, Andrei Lepikhov Postgres Professional

Re: post-freeze damage control

2024-04-09 Thread Andrei Lepikhov
On 9/4/2024 12:55, Tom Lane wrote: Andrei Lepikhov writes: * I really, really dislike jamming this logic into prepqual.c, where it has no business being. I note that it was shoved into process_duplicate_ors without even the courtesy of expanding the header comment: Yeah, I preferred to do

Re: post-freeze damage control

2024-04-08 Thread Andrei Lepikhov
any bugs or strong performance issues, and all the issues can be resolved further; on the other hand, I've got your good review and some ideas to work out. -- regards, Andrei Lepikhov Postgres Professional

Re: type cache cleanup improvements

2024-04-03 Thread Andrei Lepikhov
and reducing the hash table's size might be more efficient. In toto, the 0001-* patch looks good, and I would be glad to see it in the core. [1] https://www.postgresql.org/message-id/flat/CAKcux6ktu-8tefLWtQuuZBYFaZA83vUzuRd7c1YHC-yEWyYFpg%40mail.gmail.com -- regards, Andrei Lepikhov Postgres

Re: Asymmetric partition-wise JOIN

2024-04-01 Thread Andrei Lepikhov
. But as I can predict, it is easier to implement and looks more natural for the architecture. What do you think about that? -- regards, Andrei Lepikhov Postgres Professional

Re: Table AM Interface Enhancements

2024-04-01 Thread Andrei Lepikhov
and avoid huge underestimation. This makes sense, especially for multicolumn filters/clauses. Having a probing AM method, we may invent something for this challenge. -- regards, Andrei Lepikhov Postgres Professional

Re: POC, WIP: OR-clause support for indexes

2024-04-01 Thread Andrei Lepikhov
rray element as an OR clause; we can also provide a BitmapOr path, where SAOP is covered with a minimal number of partial indexes (likewise, previous version). -- regards, Andrei Lepikhov Postgres Professional From e42a7111a12ef82eecdb2e692d65e7ba6e43ad79 Mon Sep 17 00:00:00 2001 From: Alena Rybakina Date

Re: POC, WIP: OR-clause support for indexes

2024-03-18 Thread Andrei Lepikhov
On 14/3/2024 16:31, Alexander Korotkov wrote: On Wed, Mar 13, 2024 at 2:16 PM Andrei Lepikhov As you can see this case is not related to partial indexes.  Just no index selective for the whole query.  However, splitting scan by the OR qual lets use a combination of two selective indexes. I

Re: POC, WIP: OR-clause support for indexes

2024-03-14 Thread Andrei Lepikhov
ust no > index selective for the whole query. However, splitting scan by the > OR qual lets use a combination of two selective indexes. Thanks for the case. I will try to resolve it. -- regards, Andrei Lepikhov Postgres Professional From 156c00c820a38e5e1856f07363af87b3109b5d77 Mon Sep 17

Re: POC, WIP: OR-clause support for indexes

2024-03-14 Thread Andrei Lepikhov
On 14/3/2024 16:31, Alexander Korotkov wrote: On Wed, Mar 13, 2024 at 2:16 PM Andrei Lepikhov mailto:a.lepik...@postgrespro.ru>> wrote: > On 13/3/2024 18:05, Alexander Korotkov wrote: > > On Wed, Mar 13, 2024 at 7:52 AM Andrei Lepikhov > > Given all of the

Re: POC, WIP: OR-clause support for indexes

2024-03-13 Thread Andrei Lepikhov
On 13/3/2024 18:05, Alexander Korotkov wrote: On Wed, Mar 13, 2024 at 7:52 AM Andrei Lepikhov Given all of the above, I think moving transformation to the canonicalize_qual() would be the right way to go. Ok, I will try to move the code. I have no idea about the timings so far. I recall

Re: POC, WIP: OR-clause support for indexes

2024-03-12 Thread Andrei Lepikhov
On 12/3/2024 22:20, Alexander Korotkov wrote: On Mon, Mar 11, 2024 at 2:43 PM Andrei Lepikhov I think you are right. It is probably a better place than any other to remove duplicates in an array. I just think we should sort and remove duplicates from entry->consts in one pass. T

Re: a wrong index choose when statistics is out of date

2024-03-12 Thread Andrei Lepikhov
-picking technique less dependent on the ordering of index lists [1]. [1] https://www.postgresql.org/message-id/9b3dbf6d-776a-4953-a5a4-609929393...@postgrespro.ru -- regards, Andrei Lepikhov Postgres Professional diff --git a/src/backend/utils/adt/like_support.c b/src/backend/utils/adt

Re: POC, WIP: OR-clause support for indexes

2024-03-11 Thread Andrei Lepikhov
kly as possible to increase the effect of the optimization. -- regards, Andrei Lepikhov Postgres Professional

Re: POC, WIP: OR-clause support for indexes

2024-03-10 Thread Andrei Lepikhov
On 7/3/2024 21:51, Alexander Korotkov wrote: Hi! On Tue, Mar 5, 2024 at 9:59 AM Andrei Lepikhov mailto:a.lepik...@postgrespro.ru>> wrote: > On 5/3/2024 12:30, Andrei Lepikhov wrote: > > On 4/3/2024 09:26, jian he wrote: > ... and the new version of the patchset is atta

Re: a wrong index choose when statistics is out of date

2024-03-08 Thread Andrei Lepikhov
On 7/3/2024 17:32, David Rowley wrote: On Thu, 7 Mar 2024 at 21:17, Andrei Lepikhov wrote: I would like to ask David why the var_eq_const estimator doesn't have an option for estimation with a histogram. Having that would relieve a problem with skewed data. Detecting the situation

Re: a wrong index choose when statistics is out of date

2024-03-07 Thread Andrei Lepikhov
al rows=3501 EXPLAIN ANALYZE SELECT * FROM norm_test WHERE val = 700; -- result: planned rows=8604, actual rows=1705 EXPLAIN ANALYZE SELECT * FROM norm_test WHERE val = 1000; -- result: planned rows=8604, actual rows=91 -- regards, Andrei Lepikhov Postgres Professional

Re: "type with xxxx does not exist" when doing ExecMemoize()

2024-03-05 Thread Andrei Lepikhov
On 6/3/2024 10:10, Tender Wang wrote: Andrei Lepikhov <mailto:a.lepik...@postgrespro.ru>> 于2024年3月5日周二 17:36写道: On 1/3/2024 14:18, Tender Wang wrote: > During debug, I learned that numeric_add doesn't have type check like > rangetype, so aboved query will n

Re: Hooking into ExplainOneQuery() complicated by missing standard_ExplainOneQuery

2024-03-05 Thread Andrei Lepikhov
%40garret.ru -- regards, Andrei Lepikhov Postgres Professional

Re: "type with xxxx does not exist" when doing ExecMemoize()

2024-03-05 Thread Andrei Lepikhov
On 1/3/2024 14:18, Tender Wang wrote: During debug, I learned that numeric_add doesn't have type check like rangetype, so aboved query will not report "type with xxx does not exist". And I realize that  the test case added by Andrei Lepikhov  in v3 is right. So in v6 patch I

Re: POC, WIP: OR-clause support for indexes

2024-03-04 Thread Andrei Lepikhov
On 5/3/2024 12:30, Andrei Lepikhov wrote: On 4/3/2024 09:26, jian he wrote: ... and the new version of the patchset is attached. -- regards, Andrei Lepikhov Postgres Professional From 1c3ac3e006cd66ff40f1ddaaa09e3fc0f3a75ca5 Mon Sep 17 00:00:00 2001 From: Alena Rybakina Date: Fri, 2 Feb 2024

Re: POC, WIP: OR-clause support for indexes

2024-03-04 Thread Andrei Lepikhov
On 4/3/2024 09:26, jian he wrote: On Thu, Feb 29, 2024 at 4:59 PM Andrei Lepikhov Feel free to add, change or totally rewrite these changes. On replacement of static ScalarArrayOpExpr dest with dynamic allocated one: After discussion [1] I agree with that replacement. Some style (and language

Re: a wrong index choose when statistics is out of date

2024-03-03 Thread Andrei Lepikhov
of the boundaries when an index shows us that we have min/max outside known statistics? Because it would be used for the values out of the histogram, it should only add an overhead with a reason. -- regards, Andrei Lepikhov Postgres Professional

Re: a wrong index choose when statistics is out of date

2024-03-03 Thread Andrei Lepikhov
in pg_type and a lot of additional code for such a rare case. I'm looking forward to the demo patch. -- regards, Andrei Lepikhov Postgres Professional

Re: POC, WIP: OR-clause support for indexes

2024-03-03 Thread Andrei Lepikhov
more cycles after detection of TYPCATEGORY_ARRAY. I haven't done it yet because have a second thought: what if to combine arrays into the larger one? I'm unsure on that, so we can forbid it too. -- regards, Andrei Lepikhov Postgres Professional

Re: POC, WIP: OR-clause support for indexes

2024-02-29 Thread Andrei Lepikhov
othly. All modifications are integrated into the two new patches. Feel free to add, change or totally rewrite these changes. -- regards, Andrei Lepikhov Postgres Professional From 015a564cc784139c806a7004f25bf5f7a4b4a29d Mon Sep 17 00:00:00 2001 From: Alena Rybakina Date: Fri, 2 Feb 2024 22:01:09 +03

Re: POC, WIP: OR-clause support for indexes

2024-02-28 Thread Andrei Lepikhov
transformBoolExprOr and generate_saop_pathlist (including cross-referencing each other). These are starting points to understand the transformation and, therefore, a good place for a detailed explanation. -- regards, Andrei Lepikhov Postgres Professional

Re: "type with xxxx does not exist" when doing ExecMemoize()

2024-02-27 Thread Andrei Lepikhov
. Also, You don't need to initialize tts_values[i] at all if tts_isnull[i] set to true. -- regards, Andrei Lepikhov Postgres Professional

Re: POC, WIP: OR-clause support for indexes

2024-02-27 Thread Andrei Lepikhov
. We should describe the second part of the feature, where the optimiser can split an array to fit the optimal BitmapOr scan path. -- regards, Andrei Lepikhov Postgres Professional

Re: The const expression evaluation routine should always return a copy

2024-02-27 Thread Andrei Lepikhov
On 28/2/2024 04:19, Tom Lane wrote: Andrei Lepikhov writes: IMO, the routine eval_const_expressions_mutator contains some stale code: I'd like to push back against the idea that eval_const_expressions is expected to return a freshly-copied tree. Its API specification contains no such claim

The const expression evaluation routine should always return a copy

2024-02-26 Thread Andrei Lepikhov
and, as a result, allow to return a Const instead of SubPlan? But at least we can return a flat copy of the SubPplan node just for the convention — the same thing for the AlternativeSubPlan. See the patch in the attachment. -- regards, Andrei Lepikhov Postgres ProfessionalFrom

Re: "type with xxxx does not exist" when doing ExecMemoize()

2024-02-26 Thread Andrei Lepikhov
On 26/2/2024 18:34, Richard Guo wrote: On Mon, Feb 26, 2024 at 3:54 PM Andrei Lepikhov mailto:a.lepik...@postgrespro.ru>> wrote: On 26/2/2024 12:44, Tender Wang wrote: > Make sense. I found MemoizeState already has a MemoryContext, so I used it. > I upda

Re: "type with xxxx does not exist" when doing ExecMemoize()

2024-02-25 Thread Andrei Lepikhov
On 26/2/2024 12:44, Tender Wang wrote: Andrei Lepikhov <mailto:a.lepik...@postgrespro.ru>> 于2024年2月26日周一 10:57写道: On 25/2/2024 20:32, Tender Wang wrote: > I think in prepare_probe_slot(), should called datumCopy as the attached > patch does. >

Re: "type with xxxx does not exist" when doing ExecMemoize()

2024-02-25 Thread Andrei Lepikhov
On 26/2/2024 09:52, Andrei Lepikhov wrote: On 25/2/2024 20:32, Tender Wang wrote: I think in prepare_probe_slot(), should called datumCopy as the attached patch does. Any thoughts? Thanks. Thanks for the report. I think it is better to invent a Runtime Memory Context; likewise

Re: "type with xxxx does not exist" when doing ExecMemoize()

2024-02-25 Thread Andrei Lepikhov
. Here, you just allocate the value in some upper memory context. Also, I'm curious why such a trivial error hasn't been found for a long time -- regards, Andrei Lepikhov Postgres Professional

Re: Removing unneeded self joins

2024-02-22 Thread Andrei Lepikhov
style, but technically, it's still OK. -- regards, Andrei Lepikhov Postgres Professional

Re: POC: GROUP BY optimization

2024-02-21 Thread Andrei Lepikhov
Maxim will answer that it's enough to resolve his issue, why not? -- regards, Andrei Lepikhov Postgres Professional

Re: Unlinking Parallel Hash Join inner batch files sooner

2024-02-21 Thread Andrei Lepikhov
On 22/2/2024 06:42, Thomas Munro wrote: On Wed, Feb 21, 2024 at 7:34 PM Andrei Lepikhov wrote: I see in [1] that the reporter mentioned a delay between the error message in parallel HashJoin and the return control back from PSQL. Your patch might reduce this delay. Also, I have the same

Re: POC: GROUP BY optimization

2024-02-21 Thread Andrei Lepikhov
org/message-id/CACG=ezaYM1tr6Lmp8PRH1aeZq=rbkxeotwgzmclad5mphfw...@mail.gmail.com -- regards, Andrei Lepikhov Postgres Professional

Re: Removing unneeded self joins

2024-02-21 Thread Andrei Lepikhov
need two different subtrees for the same query. I will look into your fix. -- regards, Andrei Lepikhov Postgres Professional

Re: Unlinking Parallel Hash Join inner batch files sooner

2024-02-20 Thread Andrei Lepikhov
stuck into the unlink of tons of temporary files. So, are you going to do something with this code? [1] https://www.postgresql.org/message-id/18349-83d33dd3d0c855c3%40postgresql.org -- regards, Andrei Lepikhov Postgres Professional

Re: [POC] Allow flattening of subquery with a link to upper query

2024-02-20 Thread Andrei Lepikhov
On 20/2/2024 17:43, David Rowley wrote: On Tue, 20 Feb 2024 at 22:57, Andrei Lepikhov wrote: I agree that it would be nice to teach the planner how to do this, but I think it just has to be a cost-based decision. Imagine how the transformed query would perform of pg_am had a billion rows

Re: [POC] Allow flattening of subquery with a link to upper query

2024-02-20 Thread Andrei Lepikhov
links from a non-parent query block. So, in my opinion, the reason for this patch still exists, and we can continue this work further, maybe elaborating on flattening LATERAL references - this needs some research. [1] https://www.postgresql.org/message-id/35c8a3e8-d080-dfa8-2be3-cf5fe702010a%40postgr

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Andrei Lepikhov
design small memory contexts - one per scalable direction of memory utilization, like selectivity or partitioning (appending ?). My coding experience shows that short-lived GEQO memory context forces people to learn on Postgres internals more precisely :). -- regards, Andrei Lepikhov Postgres

Re: POC, WIP: OR-clause support for indexes

2024-02-19 Thread Andrei Lepikhov
eliminate many calls of the equal() routine, too. `leftop operator rightop` the operator can also be volatile. Do we need to check (op_volatile(opno) == PROVOLATILE_VOLATILE) within transformBoolExprOr? As usual, could you provide a test case to discuss it more objectively? -- regards, Andrei

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Andrei Lepikhov
for the next array element. It finds this piece more quickly than before that optimization. -- regards, Andrei Lepikhov Postgres Professional From 2e89dc8b743953068174c777d7a014e1ea71f659 Mon Sep 17 00:00:00 2001 From: "Andrey V. Lepikhov" Date: Tue, 20 Feb 2024 11:05:51 +0700 Subject: [PATC

Re: POC, WIP: OR-clause support for indexes

2024-02-19 Thread Andrei Lepikhov
ate in this operation */ + continue; + arrayconst = lsecond_node(Const, saop->args); + dest = makeNode(ScalarArrayOpExpr); Thanks for the review! I'm not sure I understand you clearly. Does the patch in attachment fix the issue you raised? -- regards, Andrei Lepikhov Postgres Professional diff --git

Re: Memory consumed by paths during partitionwise join planning

2024-02-19 Thread Andrei Lepikhov
On 19/2/2024 19:25, Ashutosh Bapat wrote: On Fri, Feb 16, 2024 at 8:42 AM Andrei Lepikhov wrote: Live example: right now, I am working on the code like MSSQL has - a combination of NestLoop and HashJoin paths and switching between them in real-time. It requires both paths in the path list

Re: POC, WIP: OR-clause support for indexes

2024-02-19 Thread Andrei Lepikhov
. As I see it, the only general explanation of the idea is not addressed. I'm not sure how deeply we should explain it. -- regards, Andrei Lepikhov Postgres Professional From 3a3b6aa36320a06b64f2f608e3526255e53ed655 Mon Sep 17 00:00:00 2001 From: Alena Rybakina Date: Fri, 2 Feb 2024 22:01:09 +0300

Re: Removing unneeded self joins

2024-02-18 Thread Andrei Lepikhov
, please, recheck? I reviewed this patch. Why do you check only the target list? I guess these links can be everywhere. See the patch in the attachment with the elaborated test and slightly changed code. -- regards, Andrei Lepikhov Postgres Professional From 7f94a3c96fd410522b87e570240cdb96b300dd31

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2024-02-18 Thread Andrei Lepikhov
partitions, we should introduce per-partition memory context and reset it in between. GEQO already has a short-lived memory context, making designing extensions a bit more challenging but nothing too painful. -- regards, Andrei Lepikhov Postgres Professional

Re: POC, WIP: OR-clause support for indexes

2024-02-15 Thread Andrei Lepikhov
On 16/2/2024 07:00, jian he wrote: On Wed, Feb 14, 2024 at 11:21 AM Andrei Lepikhov wrote: My OS: Ubuntu 22.04.3 LTS I already set the max_parallel_workers_per_gather to 10. So for all cases, it should use parallelism first? a better question would be: how to make the number of OR less than 29

Re: Memory consumed by paths during partitionwise join planning

2024-02-15 Thread Andrei Lepikhov
On 15/2/2024 19:06, Ashutosh Bapat wrote: On Thu, Feb 15, 2024 at 9:41 AM Andrei Lepikhov But I'm not sure about freeing unreferenced paths. I would have to see alternatives in the pathlist. I didn't understand this. Can you please elaborate? A path in any pathlist is referenced

Re: planner chooses incremental but not the best one

2024-02-15 Thread Andrei Lepikhov
On 15/2/2024 18:10, Tomas Vondra wrote: On 2/15/24 07:50, Andrei Lepikhov wrote: On 18/12/2023 19:53, Tomas Vondra wrote: On 12/18/23 11:40, Richard Guo wrote: The challenge is where to get usable information about correlation between columns. I only have a couple very rought ideas of what

Re: planner chooses incremental but not the best one

2024-02-14 Thread Andrei Lepikhov
directly checking appendrel->tuples correction. -- regards, Andrei Lepikhov Postgres Professional

Re: planner chooses incremental but not the best one

2024-02-14 Thread Andrei Lepikhov
f grouping? -- regards, Andrei Lepikhov Postgres Professional

Re: Memory consumed by paths during partitionwise join planning

2024-02-14 Thread Andrei Lepikhov
/CAOP8fzaVL_2SCJayLL9kj5pCA46PJOXXjuei6-3aFUV45j4LJQ%40mail.gmail.com [2] https://www.postgresql.org/message-id/flat/CAMbWs496%2BN%3DUAjOc%3DrcD3P7B6oJe4rZw08e_TZRUsWbPxZW3Tw%40mail.gmail.com -- regards, Andrei Lepikhov Postgres Professional

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-02-13 Thread Andrei Lepikhov
On 14/2/2024 13:32, Ashutosh Bapat wrote: On Wed, Feb 14, 2024 at 9:50 AM Andrei Lepikhov wrote: On 30/1/2024 12:44, Ashutosh Bapat wrote: Thanks Vignesh. PFA patches rebased on the latest HEAD. The patch addressing Amit's comments is still a separate patch for him to review. Thanks

Re: Propagate pathkeys from CTEs up to the outer query

2024-02-13 Thread Andrei Lepikhov
parameterisation through the GROUP-BY and the query block fence up to the JOIN searching code of the parent query. -- regards, Andrei Lepikhov Postgres Professional

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-02-13 Thread Andrei Lepikhov
, such as pg_hint_plan, call build_child_join_sjinfo. It is OK to break the interface with a major version. But what if they need child_sjinfo a bit longer and collect links to this structure? I don't think it is a real stopper, but it is worth additional analysis. -- regards, Andrei Lepikhov

Re: POC, WIP: OR-clause support for indexes

2024-02-13 Thread Andrei Lepikhov
On 13/2/2024 17:03, Andrei Lepikhov wrote: On 13/2/2024 07:00, jian he wrote: The time is the last result of the 10 iterations. I'm not sure about the origins of such behavior, but it seems to be an issue of parallel workers, not this specific optimization. Having written that, I'd got

Re: POC, WIP: OR-clause support for indexes

2024-02-13 Thread Andrei Lepikhov
On 12/2/2024 17:51, Alena Rybakina wrote: On 12.02.2024 12:01, Andrei Lepikhov wrote: On 12/2/2024 15:55, Alena Rybakina wrote: As I understand it, match_clauses_to_index is necessary if you have a RestrictInfo (rinfo1) variable, so maybe we should run it after the make_restrictonfo procedure

Re: POC, WIP: OR-clause support for indexes

2024-02-13 Thread Andrei Lepikhov
ue of parallel workers, not this specific optimization. -- regards, Andrei Lepikhov Postgres Professional

Re: POC, WIP: OR-clause support for indexes

2024-02-12 Thread Andrei Lepikhov
a RestrictInfo (rinfo1) variable, so maybe we should run it after the make_restrictonfo procedure, otherwise calling it, I think, is useless. I think you must explain your note in more detail. Before this call, we already called make_restrictinfo() and built rinfo1, haven't we? -- regards, Andrei Lepikhov

Re: POC, WIP: OR-clause support for indexes

2024-02-11 Thread Andrei Lepikhov
Thanks for the review! It was the first version for discussion. Of course, refactoring and polishing cycles will be needed, even so we can discuss the general idea earlier. On 10/2/2024 12:00, jian he wrote: On Thu, Feb 8, 2024 at 1:34 PM Andrei Lepikhov 1235 | PredicatesData

Re: pg_stat_advisor extension

2024-02-07 Thread Andrei Lepikhov
. -- regards, Andrei Lepikhov Postgres Professional

Re: POC, WIP: OR-clause support for indexes

2024-02-07 Thread Andrei Lepikhov
] https://www.postgresql.org/message-id/CAPpHfduJtO0s9E%3DSHUTzrCD88BH0eik0UNog1_q3XBF2wLmH6g%40mail.gmail.com -- regards, Andrei Lepikhov Postgres Professional From 0ac511ab94959e87d1525d5ea8c4855643a6ccbe Mon Sep 17 00:00:00 2001 From: Alena Rybakina Date: Fri, 2 Feb 2024 22:01:09 +0300 Subject

Re: POC: GROUP BY optimization

2024-02-01 Thread Andrei Lepikhov
ort on (x,z,w) instead of full sort? 2. For memo, IMO, this test shows us the future near perspective of this feature: It is cheaper to use grouping order (w,z) instead of (z,w). -- regards, Andrei Lepikhov Postgres Professional

Re: POC: GROUP BY optimization

2024-02-01 Thread Andrei Lepikhov
particular reason for pg_class to be especially suited to the test. Yeah, It is my fault. Please, see in the attachment the patch fixing that. -- regards, Andrei Lepikhov Postgres Professional From 11a049d95ee48e38ad569aab7663d8de91f946ad Mon Sep 17 00:00:00 2001 From: "Andrey V. Lepikhov&q

Re: POC: GROUP BY optimization

2024-01-18 Thread Andrei Lepikhov
Just forgotten second attachment. -- regards, Andrei Lepikhov Postgres Professional diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 1095b73dac..b612420547 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path

Re: POC: GROUP BY optimization

2024-01-18 Thread Andrei Lepikhov
sortref' is valid before calling get_sortgroupref_clause_noerr with it. Also, how can we guarantee that the returned GROUP BY item is sortable? Should we check that with OidIsValid(sgc->sortop)? Added. Reviewed it, looks good. -- regards, Andrei Lepikhov Postgres Professional diff --git a/sr

Re: POC: GROUP BY optimization

2024-01-15 Thread Andrei Lepikhov
set looks good to me.  I'm going to push it if there are no objections. Seems I'm late for the party.  Can we hold for several more days?  I'd like to have a review on this patch. Get on board! It looks like this feature needs as much review as possible (likewise SJE). -- regards, Andrei

Re: POC: GROUP BY optimization

2024-01-14 Thread Andrei Lepikhov
On 13/1/2024 22:00, Alexander Korotkov wrote: On Sat, Jan 13, 2024 at 11:09 AM Andrei Lepikhov wrote: On 11/1/2024 18:30, Alexander Korotkov wrote: On Tue, Jan 9, 2024 at 1:14 PM Pavel Borisov wrote: Hmm, I don't see this old code in these patches. Resend 0002-* because of trailing spaces

Re: POC: GROUP BY optimization

2024-01-13 Thread Andrei Lepikhov
comments and tests to make the code more understandable (see attachment). I intended to look into this part of the code more, but the tests show a difference between PostgreSQL v.15 and v.16, which causes changes in the code of this feature. -- regards, Andrei Lepikhov Postgres Professional

Re: introduce dynamic shared memory registry

2024-01-10 Thread Andrei Lepikhov
to tie the key size to > NAMEDATALEN. IMO, we should avoid magic numbers whenever possible. Current logic according to which first users of this feature will be extensions naturally bonds this size to the size of the 'name' type. And one more point. I think the commit already deserves a mor

Re: Custom explain options

2024-01-10 Thread Andrei Lepikhov
On 10/1/2024 20:27, Konstantin Knizhnik wrote: On 10/01/2024 8:46 am, Michael Paquier wrote: On Wed, Jan 10, 2024 at 01:29:30PM +0700, Andrei Lepikhov wrote: What do you think about this really useful feature? Do you wish to develop it further? I am biased here.  This seems like a lot

Re: Multidimensional Histograms

2024-01-10 Thread Andrei Lepikhov
combination, building the dependency statistics is still massive work. So, in the multicolumn case, having something like a histogram may be more effective. -- regards, Andrei Lepikhov Postgres Professional

Re: Custom explain options

2024-01-09 Thread Andrei Lepikhov
this code in my PR to send actual amount of custom instrumentation data. But it can not help with the cases above. What do you think about this really useful feature? Do you wish to develop it further? -- regards, Andrei Lepikhov Postgres Professional

Re: POC: GROUP BY optimization

2024-01-09 Thread Andrei Lepikhov
On 9/1/2024 16:45, vignesh C wrote: On Tue, 9 Jan 2024 at 14:31, Andrei Lepikhov wrote: Here is a new version of GROUP-BY optimization without sort model. On 21/12/2023 17:53, Alexander Korotkov wrote: I'd like to make some notes. 1) As already mentioned, there is clearly a repetitive

  1   2   >