Re: Asymmetric partition-wise JOIN

2023-10-16 Thread Ashutosh Bapat
#x27;t be able to know the number of such relations involved in the query unless we start planning such a join. It's late to add new base relations and assign them new Relids. Of course I haven't thought hard about it. I haven't looked at the patch to see whether this problem is solved and how. -- Best Wishes, Ashutosh Bapat

Re: BRIN minmax multi - incorrect distance for infinite timestamp/date

2023-10-16 Thread Ashutosh Bapat
On Mon, Oct 16, 2023 at 7:33 PM Tomas Vondra wrote: > > On 10/16/23 11:25, Ashutosh Bapat wrote: > > Thanks Tomas for bringing this discussion to hackers. > > > > > > On Fri, Oct 13, 2023 at 8:58 PM Dean Rasheed > > wrote: > >> > >>

Re: RFC: Logging plan of the running query

2023-10-16 Thread Ashutosh Bapat
On Thu, Oct 12, 2023 at 6:51 PM torikoshia wrote: > > On 2023-10-11 16:22, Ashutosh Bapat wrote: > > > > Considering the similarity with auto_explain I wondered whether this > > function should be part of auto_explain contrib module itself? If we > > do that users

Re: BRIN minmax multi - incorrect distance for infinite timestamp/date

2023-10-16 Thread Ashutosh Bapat
uldn't find an example where that is not true. Tomas, you may want to incorporate this in your patch. If not, I will incorporate it in my infinite interval patchset in [1]. [1] https://www.postgresql.org/message-id/CAExHW5u1JE7dxK=wlzqhcszntoxqzjdiermhrepw6r8w6kc...@mail.gmail.com -- Best Wishes, Ashutosh Bapat

Re: Use virtual tuple slot for Unique node

2023-10-12 Thread Ashutosh Bapat
ceding) a, b from t_text) q HEAD: 16070.62 ms patched: 16182.16 ms This one is surprising though. May be the advantage of using the same tuple table slot is so narrow when large data needs to be copied that the execution times almost match. The patched and unpatched execution times differ by the margin of error either way. -- Best Wishes, Ashutosh Bapat

Re: RFC: Logging plan of the running query

2023-10-11 Thread Ashutosh Bapat
f we could add some safety net that will be great but may not be necessary for the first cut. -- Best Wishes, Ashutosh Bapat

Re: Check each of base restriction clauses for constant-FALSE-or-NULL

2023-10-09 Thread Ashutosh Bapat
ight. Do you have an example where this could be demonstrated? -- Best Wishes, Ashutosh Bapat

Re: Draft LIMIT pushdown to Append and MergeAppend patch

2023-10-09 Thread Ashutosh Bapat
On Mon, Oct 9, 2023 at 4:33 PM David Rowley wrote: > > On Mon, 9 Oct 2023 at 23:35, Ashutosh Bapat > wrote: > > > > On Mon, Oct 9, 2023 at 6:25 AM David Rowley wrote: > > > > > > However, it may also be worth you reading over [3] and the ultimate > &g

Re: Draft LIMIT pushdown to Append and MergeAppend patch

2023-10-09 Thread Ashutosh Bapat
in needs to be explicitly turned ON. The solution we will develop here will solve problem with partitionwise join as well. It's hard to solve this problem. If there's a real case where LIMIT pushdown helps without fixing Sort pushdown case, it might help proceeding with the same. -- Best Wishes, Ashutosh Bapat

Re: Check each of base restriction clauses for constant-FALSE-or-NULL

2023-10-09 Thread Ashutosh Bapat
ur code would just duplicate that. > > -- patched > explain (costs off) > select * from t t1 left join (select * from t t2 where false) s on s.a = 1; >QUERY PLAN > > Nested Loop Left Join >-> Seq Scan on t t1 >-> Result > One-Time Filter: false > (4 rows) Does your code have any other benefits like deeming an inner join as empty? -- Best Wishes, Ashutosh Bapat

Re: FDW LIM IT pushdown

2023-10-09 Thread Ashutosh Bapat
er it's being used for costing or for pushing it down to the partitions. This isn't a full answer, but I hope these pointers would help you. -- Best Wishes, Ashutosh Bapat

Re: ALTER COLUMN ... SET EXPRESSION to alter stored generated column's expression

2023-10-06 Thread Ashutosh Bapat
. > Should we treat it the same fashion as ALTER COLUMN ... TYPE which rewrites the column values? Of course that rewrites the whole table, but logically they are comparable. -- Best Wishes, Ashutosh Bapat

Re: Two Window aggregate node for logically same over clause

2023-10-06 Thread Ashutosh Bapat
rse tree resulting in the creation of two different window aggregate > node. > > This check can be modified to form a single window aggregate node for the > above results in faster query execution. Is there any reason for creating two > different window aggregate node? I don&#x

Re: FDW pushdown of non-collated functions

2023-10-06 Thread Ashutosh Bapat
lved for 7 years at least, may be such a simple fix is not enough. Another solution would be to attach another attribute to a function indicating whether the output of that function depends upon the input collations or not. Doing that just for FDW may not be acceptable though. -- Best Wishes, Ashutosh Bapat

Re: Infinite Interval

2023-10-04 Thread Ashutosh Bapat
3J6ZXf3BOM79pFsFNcRjwA%40mail.gmail.com [2] Table 8.9 at https://www.postgresql.org/docs/current/datatype-datetime.html#DATATYPE-DATETIME [3] https://www.postgresql.org/message-id/CAAvxfHf0-T99i%3DOrve_xfonVCvsCuPy7C4avVm%3D%2Byu128ujSGg%40mail.gmail.com [4] https://www.postgresql.org/message-id/26022.1545087636%40sss.pgh.pa.us [5] https://www.postgresql.org/message-id/CAAvxfHdzd5JLRBXDAW7OPhsNNACvhsCP3f5R4LNhRVaDuQG0gg%40mail.gmail.com -- Best Wishes, Ashutosh Bapat

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-10-03 Thread Ashutosh Bapat
Ds and the list of expressions may be > -* referenced somewhere else. Do not free those. > -*/ > + /* semi_rhs_exprs may be referenced, so don't free. */ > } Works for me. PFA patchset with these changes. I have still left the changes addressing y

Re: Doc: Minor update for enable_partitionwise_aggregate

2023-10-03 Thread Ashutosh Bapat
eels > slightly off to me to say the "partitioned tables" instead of the partitions. It's technically incorrect as well. Aggregation is performed on a single relation always - a join or subquery or simple relation. A join may have multiple tables in it but the aggregation is performed on its result and not individual tables and hence not on partitions of individual tables. -- Best Wishes, Ashutosh Bapat

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-09-27 Thread Ashutosh Bapat
atch set 0002 - 0002 and 0003 from previous patch set squashed together 0003 - changes addressing above comments. -- Best Wishes, Ashutosh Bapat From d7e65bf7c277c3dd3afbd1b7a152b86d24bd6904 Mon Sep 17 00:00:00 2001 From: Ashutosh Bapat Date: Wed, 27 Sep 2023 16:29:11 +0530 Subject: [PATCH 3/3] A

Re: Avoid a possible out-of-bounds access (src/backend/optimizer/util/relnode.c)

2023-09-26 Thread Ashutosh Bapat
verhead to find_base_rel() as this would make the case for using > PlannerInfo.simple_rel_array directly even stronger. I am curious, is the overhead in find_base_rel() impacting overall performance? -- Best Wishes, Ashutosh Bapat

Re: Avoid a possible out-of-bounds access (src/backend/optimizer/util/relnode.c)

2023-09-25 Thread Ashutosh Bapat
On Mon, Sep 25, 2023 at 7:14 PM Ranier Vilela wrote: > > Em seg., 25 de set. de 2023 às 08:23, Ashutosh Bapat > escreveu: >> >> On Sat, Sep 23, 2023 at 7:29 PM Ranier Vilela wrote: >> > >> > Hi, >> > >> > Per Coverity. &

Re: Avoid a possible out-of-bounds access (src/backend/optimizer/util/relnode.c)

2023-09-25 Thread Ashutosh Bapat
lity. bms_make_singleton() has an assertion at the end: Assert(result >= 0); bms_make_singleton(), bms_add_member() explicitly forbids negative values. It looks like we have covered all the places which can add a negative value to a bitmapset. May be we are missing a place or two. It will be goo

Re: Infinite Interval

2023-09-22 Thread Ashutosh Bapat
le NULL inputs if they are *not* strict. #select proisstrict from pg_proc where proname = 'interval_avg_serialize'; proisstrict - t (1 row) #select proisstrict from pg_proc where proname = 'interval_avg_deserialize'; proisstrict ----- t (1 row) -- Best Wishes, Ashutosh Bapat

Re: Report planning memory in EXPLAIN ANALYZE

2023-09-22 Thread Ashutosh Bapat
t's out of scope of this work. "used" and "allocated" are MemoryContext terms indicating memory actually used vs memory allocated. > > Personally I am pretty like patch 1, we just need to refactor some words > to make everything clearer. Thanks. -- Best Wishes, Ashutosh Bapat

Re: Infinite Interval

2023-09-21 Thread Ashutosh Bapat
On Wed, Sep 20, 2023 at 8:23 PM Dean Rasheed wrote: > > On Wed, 20 Sept 2023 at 15:00, Ashutosh Bapat > wrote: > > > > 0005 - Refactored Jian's code fixing window functions. Does not > > contain the changes for serialization and deserialization. Jian, > &g

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-09-20 Thread Ashutosh Bapat
On Thu, Sep 21, 2023 at 6:37 AM Amit Langote wrote: > > On Wed, Sep 20, 2023 at 10:24 PM Ashutosh Bapat > wrote: > > On Wed, Sep 20, 2023 at 5:24 PM Amit Langote > > wrote: > > > Just one comment on 0003: > > > > > > + /* > > > +*

Re: Infinite Interval

2023-09-20 Thread Ashutosh Bapat
at finite values don't sum to our representation of infinity, > as in interval_pl(). Fixed in the latest patch set. -- Best Wishes, Ashutosh Bapat

Re: Infinite Interval

2023-09-20 Thread Ashutosh Bapat
On Mon, Sep 18, 2023 at 5:09 PM Dean Rasheed wrote: > > On Wed, 13 Sept 2023 at 11:13, Ashutosh Bapat > wrote: > > > > On Tue, Sep 12, 2023 at 2:39 PM Dean Rasheed > > wrote: > > > > > > and it looks like the infinite interval > > > inp

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-09-20 Thread Ashutosh Bapat
ction build the rest of the code will free Relids which are referenced somewhere else causing dangling pointers. -- Best Wishes, Ashutosh Bapat

Re: persist logical slots to disk during shutdown checkpoint

2023-09-20 Thread Ashutosh Bapat
for the updated version. > > > > Pushed. Commitfest entry "https://commitfest.postgresql.org/44/4536/ is in "Ready for committer" state. Is there something remaining here? We should probably set it as "committed". -- Best Wishes, Ashutosh Bapat

Re: [PoC] Reducing planning time when tables have many partitions

2023-09-20 Thread Ashutosh Bapat
l be simplified by your approach. [1] https://www.postgresql.org/message-id/caexhw5u0yyyr2mwvlrvvy_qnld65kpc9u-bo0ox7bglkgba...@mail.gmail.com -- Best Wishes, Ashutosh Bapat

Re: Infinite Interval

2023-09-18 Thread Ashutosh Bapat
erval >later than all other time stamps > > it seems we have forgotten to mention the -infinity case, we can fix > the doc together, since timestamptz also applies to > +/-infinity. Your point about -infinity is right. But timestamp corresponds to both timestamp with and without timezone as per table 8.9 on the same page . https://www.postgresql.org/docs/current/datatype-datetime.html#DATATYPE-DATETIME-TABLE. So I don't see a need to specify timestamptz separately. [1] https://www.postgresql.org/message-id/CAExHW5ut4bR4KSNWAhXb_EZ8PyY=j100gua6zumnhvoia1z...@mail.gmail.com -- Best Wishes, Ashutosh Bapat

Re: ALTER COLUMN ... SET EXPRESSION to alter stored generated column's expression

2023-09-14 Thread Ashutosh Bapat
where it's reasonable - following precedence is usually good. But I haven't reviewed the code to comment on it. [1] https://www.postgresql.org/docs/16/ddl-generated-columns.html -- Best Wishes, Ashutosh Bapat

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-09-14 Thread Ashutosh Bapat
Hi All, On Fri, Aug 11, 2023 at 6:24 PM Ashutosh Bapat wrote: > > Obtaining child clauses from parent clauses by translation and > tracking the translations is less complex and may be more efficient > too. I will post a patch on those lines soon. > PFA patch set to add infrastr

Re: logical decoding and replication of sequences, take 2

2023-09-13 Thread Ashutosh Bapat
On Fri, Aug 18, 2023 at 4:28 PM Amit Kapila wrote: > > On Fri, Aug 18, 2023 at 10:37 AM Amit Kapila wrote: > > > > On Thu, Aug 17, 2023 at 7:13 PM Ashutosh Bapat > > wrote: > > > > > > On Wed, Aug 16, 2023 at 7:56 PM Tomas Vondra > > > wr

Re: logical decoding and replication of sequences, take 2

2023-09-13 Thread Ashutosh Bapat
On Thu, Aug 17, 2023 at 7:13 PM Ashutosh Bapat wrote: > > > > The attached patch merges the earlier improvements, except for the part > > that experimented with adding a "fake" transaction (which turned out to > > have a number of difficult issues). > > 000

Reuse child_relids in try_partitionwise_join was Re: Assert failure on bms_equal(child_joinrel->relids, child_joinrelids)

2023-09-08 Thread Ashutosh Bapat
On Fri, Jul 28, 2023 at 3:16 PM Ashutosh Bapat wrote: > > Hi Tom, Richard, > > On Mon, Jul 24, 2023 at 8:17 AM Richard Guo wrote: > > > > Thanks for pushing it! > > With this fix, I saw a noticeable increase in the memory consumption > of planner. I was running

Re: Oversight in reparameterize_path_by_child leading to executor crash

2023-09-08 Thread Ashutosh Bapat
tree and be evaluated at appropriate level. But then why the query returns the right results is a mystery. I am not sure where we are taking the original bug fix with this investigation. Is it required to fix this problem in order to fix the original problem OR we should commit the fix for the original problem and then investigate this further? -- Best Wishes, Ashutosh Bapat

Re: [BackendXidGetPid] only access allProcs when xid matches

2023-09-07 Thread Ashutosh Bapat
On Thu, Sep 7, 2023 at 3:05 PM Junwang Zhao wrote: > > On Thu, Sep 7, 2023 at 5:07 PM Ashutosh Bapat > wrote: > > > > Forgot to attach the patch. > > LGTM > > Should I change the status to ready for committer now? > I would still love to see some numbers.

Re: persist logical slots to disk during shutdown checkpoint

2023-09-07 Thread Ashutosh Bapat
On Thu, Sep 7, 2023 at 4:11 PM Amit Kapila wrote: > > On Thu, Sep 7, 2023 at 3:38 PM Ashutosh Bapat > wrote: > > > > * This needn't actually be part of a checkpoint, but it's a convenient > > - * location. > > + * location. is_shutdown is true in cas

Re: persist logical slots to disk during shutdown checkpoint

2023-09-07 Thread Ashutosh Bapat
used is commented upon in the relevant code. * This needn't actually be part of a checkpoint, but it's a convenient - * location. + * location. is_shutdown is true in case of a shutdown checkpoint. Relying on the first sentence, if we decide to not persist the replication slot at the time of checkpoint, would that be OK? It doesn't look like a convenience thing to me any more. -- Best Wishes, Ashutosh Bapat

Re: [BackendXidGetPid] only access allProcs when xid matches

2023-09-07 Thread Ashutosh Bapat
Forgot to attach the patch. On Thu, Sep 7, 2023 at 1:22 PM Ashutosh Bapat wrote: > > Hi Junwang, > We leave a line blank after variable declaration as in the attached patch. > > Otherwise the patch looks good to me. > > The function modified by the patch is only used by e

Re: [BackendXidGetPid] only access allProcs when xid matches

2023-09-07 Thread Ashutosh Bapat
able to catch some miniscule improvement per row. If the performance is measurable, we can mark the CF entry as ready for committer. -- Best Wishes, Ashutosh Bapat On Thu, Aug 10, 2023 at 1:48 PM Junwang Zhao wrote: > > On Thu, Aug 10, 2023 at 4:11 PM Ashutosh Bapat > wrote: >

Re: [PoC] Reducing planning time when tables have many partitions

2023-09-06 Thread Ashutosh Bapat
tive > solution if we can overcome several problems, including what I > mentioned above. It seems that you are still investigating and fixing issues. But the CF entry is marked as "needs review". I think a better status is "WoA". Do you agree with that? -- Best Wishes, Ashutosh Bapat

Re: Optimize planner memory consumption for huge arrays

2023-09-06 Thread Ashutosh Bapat
ring to create a Const node for each element. > > The diff in the attachment is proof of concept showing how to reduce wasting > of memory. Having benchmarked a bit, I didn't find any overhead. > You might want to include your benchmarking results as well. -- Best Wishes, Ashutosh Bapat

Re: Speaker Bureau

2023-09-04 Thread Ashutosh Bapat
Added my name to the list. I am available to travel for meetups. -- Best Wishes, Ashutosh Bapat On Sat, Sep 2, 2023 at 12:30 AM Dave Cramer wrote: > > Greetings, > > If you are on the speaker list can you send an email to ugc...@postgresql.us > indicating whether you are avai

Re: [17] CREATE SUBSCRIPTION ... SERVER

2023-09-04 Thread Ashutosh Bapat
ons, what you are saying makes sense. But it's not. We will continue to support current mechanism which doesn't require postgres_fdw to be installed on every PostgreSQL cluster. What security problems do you foresee if postgres_fdw is used in addition to the current mechanism? -- Best Wishes, Ashutosh Bapat

Re: [17] CREATE SUBSCRIPTION ... SERVER

2023-09-04 Thread Ashutosh Bapat
On Sat, Sep 2, 2023 at 12:24 AM Jeff Davis wrote: > > On Fri, 2023-09-01 at 12:28 +0530, Ashutosh Bapat wrote: > > Thinking larger, how about we allow any FDW to be used here. > > That's a possibility, but I think that means the subscription would > need to constant

Re: persist logical slots to disk during shutdown checkpoint

2023-09-01 Thread Ashutosh Bapat
On Thu, Aug 31, 2023 at 7:28 PM Amit Kapila wrote: > > On Thu, Aug 31, 2023 at 6:12 PM Ashutosh Bapat > wrote: > > > > On Thu, Aug 31, 2023 at 2:52 PM Amit Kapila wrote: > > > > > > I > > > > > > think we should shut down subs

Re: [17] CREATE SUBSCRIPTION ... SERVER

2023-08-31 Thread Ashutosh Bapat
On Fri, Sep 1, 2023 at 2:47 AM Joe Conway wrote: > > On 8/31/23 12:52, Jeff Davis wrote: > > On Thu, 2023-08-31 at 10:59 +0530, Ashutosh Bapat wrote: > >> The server's FDW has to be postgres_fdw. So we have to handle the > >> awkward dependency between

Re: persist logical slots to disk during shutdown checkpoint

2023-08-31 Thread Ashutosh Bapat
idea. > > > > We call ReplicatioinSlotMarkDirty() followed by ReplicationSlotSave() > > at all the places, even those which are more frequent than this. > > > > All but one. Normally, the idea of marking dirty is to indicate that > we will actually write/flush the contents at a later point (except > when required for correctness) as even indicated in the comments atop > ReplicatioinSlotMarkDirty(). However, I see your point that we use > that protocol at all the current places including CreateSlotOnDisk(). > So, we can probably do it here as well. yes I didn't see this entry in commitfest. Since we are discussing it and the next CF is about to begin, probably it's good to add one there. -- Best Wishes, Ashutosh Bapat

Re: Statistics Import and Export

2023-08-31 Thread Ashutosh Bapat
building it locally. If it's known that the stats on foreign and local servers match for a foreign table, we will be one step closer to accurately estimating the cost of a foreign plan locally rather than through EXPLAIN. -- Best Wishes, Ashutosh Bapat

Re: persist logical slots to disk during shutdown checkpoint

2023-08-30 Thread Ashutosh Bapat
ithout > much benefit. Sure, it may not be performance-sensitive but still > adding another pair of lock/release doesn't seem like a better idea. We call ReplicatioinSlotMarkDirty() followed by ReplicationSlotSave() at all the places, even those which are more frequent than this. So I think it's better to stick to that protocol rather than adding a new flag. -- Best Wishes, Ashutosh Bapat

Re: [17] CREATE SUBSCRIPTION ... SERVER

2023-08-30 Thread Ashutosh Bapat
On Wed, Aug 30, 2023 at 9:00 PM Jeff Davis wrote: > > On Wed, 2023-08-30 at 19:11 +0530, Ashutosh Bapat wrote: > > Are you suggesting that SERVERs created with FDW can not be used as > > publishers? > > Correct. Without that, how would the subscription know that the FDW &

Re: [17] CREATE SUBSCRIPTION ... SERVER

2023-08-30 Thread Ashutosh Bapat
wn but by restricting this feature to only CONNECTION ONLY, we are restricting the possibility of such a push down. -- Best Wishes, Ashutosh Bapat

Re: persist logical slots to disk during shutdown checkpoint

2023-08-30 Thread Ashutosh Bapat
On Tue, Aug 29, 2023 at 5:40 PM Ashutosh Bapat wrote: > > I am looking at it. If you can wait till the end of the week, that > will be great. /* * Successfully wrote, unset dirty bit, unless somebody dirtied again - * already. + * already and remember the confirmed_

Re: persist logical slots to disk during shutdown checkpoint

2023-08-29 Thread Ashutosh Bapat
e comments or suggestions, I would like to push > this in a day or two. I am looking at it. If you can wait till the end of the week, that will be great. -- Best Wishes, Ashutosh Bapat

Re: Report planning memory in EXPLAIN ANALYZE

2023-08-24 Thread Ashutosh Bapat
ture, we will need a way to create a new memory context with the same properties as the CurrentMemoryContext, a functionality missing right now. Once we agree upon the approach, the patch will need to be merged into 0002 and in turn 0001. -- Best Wishes, Ashutosh Bapat explain_planning_memory.o

Re: Oversight in reparameterize_path_by_child leading to executor crash

2023-08-23 Thread Ashutosh Bapat
1b = s.a; > count > --- > 5 > (1 row) > > > For another query it would error out during planning. > > regression=# explain (verbose, costs off) > select count(*) from prt1 t1 left join lateral > (select t1.b as t1b, t2.* from prt2 t2) s > on t1.a = s.b where s.t1b = s.b; > ERROR: variable not found in subplan target list > > To fix it we may need to modify RelOptInfos for Path, BitmapHeapPath, > ForeignPath and CustomPath, and modify IndexOptInfos for IndexPath. It > seems that that is not easily done without postponing reparameterization > of paths until createplan.c. Maybe I am missing something here but why aren't we copying these restrictinfos in the path somewhere? I have a similar question for tablesample clause as well. -- Best Wishes, Ashutosh Bapat

Re: Oversight in reparameterize_path_by_child leading to executor crash

2023-08-22 Thread Ashutosh Bapat
o which need to be outside the macros and actually look better to me. Let me know what you think of this. FWIW I ran make check and all the tests pass. I agree that we can not consider this for backbranches but we are anyway thinking of disabling reparameterization for tablesa

Re: persist logical slots to disk during shutdown checkpoint

2023-08-22 Thread Ashutosh Bapat
better to do that > optimization (persist confirm_flush_lsn at a regular interval) as a > separate patch as we need to test and prove its value separately. Fine with me. -- Best Wishes, Ashutosh Bapat

Re: persist logical slots to disk during shutdown checkpoint

2023-08-22 Thread Ashutosh Bapat
(1) right now, we might just forget to do that small incremental change in future. My preference is 1. Do both (1) and (2) together 2. Do (2) first and then (1) as a separate commit. 3. Just implement (2) if we don't have time at all for first two options. -- Best Wishes, Ashutosh Bapat

Re: persist logical slots to disk during shutdown checkpoint

2023-08-21 Thread Ashutosh Bapat
s 1. to mark a slot dirty and persist if the last confirm_flush_lsn when slot was persisted was too far from the current confirm_flush_lsn of the slot. 2. at shutdown checkpoint, persist all the slots which have these two confirm_flush_lsns different. -- Best Wishes, Ashutosh Bapat

Re: Oversight in reparameterize_path_by_child leading to executor crash

2023-08-21 Thread Ashutosh Bapat
Essentially combing reparameterize_path_by_child() and path_is_reparameterizable_by_child(). The name of such a function can be chosen appropriately. The mode will be controlled by a fourth argument to the function. When assessing a path no translations happen and no extra memory is allocated. -- Best Wishes, Ashutosh Bapat

Re: logical decoding and replication of sequences, take 2

2023-08-17 Thread Ashutosh Bapat
o review the impact of not setting replorigin_session_origin_timestamp. What fake transaction experiment are you talking about? -- Best Wishes, Ashutosh Bapat

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-08-15 Thread Ashutosh Bapat
On Fri, Aug 4, 2023 at 2:11 PM Ashutosh Bapat wrote: > > Attached patchset fixing those. > 0001 - patch to report planning memory, with to explain.out regression output > fix. We may consider committing this as well. > 0002 - with your comment addressed above. 0003 - Added

Re: AssertLog instead of Assert in some places

2023-08-14 Thread Ashutosh Bapat
't help. Those might make things worse. We need two separate incantations for non-recoverable and recoverable Asserts respectively. I like Peter's idea of having a new elevel, however it still requires adding conditional USE_ASSERT, an if testing the condition and then writing an error message. AssertLog() in the patch uses just a few more letters. It won't help to expand the scope of the problem since that will reduce the chances of getting anything done. -- Best Wishes, Ashutosh Bapat

Re: AssertLog instead of Assert in some places

2023-08-14 Thread Ashutosh Bapat
und > process that just restarts. Fair point. Our regression doesn't check server error logs for unwanted errors. How about restricting it to only client backends? I don't know how to identify those from others but there must be a way. -- Best Wishes, Ashutosh Bapat

Re: Report planning memory in EXPLAIN ANALYZE

2023-08-14 Thread Ashutosh Bapat
racking itself might make the planning in-efficient and we might still miss the spikes in memory allocations, if they are very short lived. If the planner runs for more than a few minutes, maybe we could add some tracking. -- Best Wishes, Ashutosh Bapat

Re: Report planning memory in EXPLAIN ANALYZE

2023-08-14 Thread Ashutosh Bapat
On Mon, Aug 14, 2023 at 5:23 AM David Rowley wrote: > > On Thu, 10 Aug 2023 at 20:33, Ashutosh Bapat > wrote: > > My point is what's relevant here is how much net memory planner asked > > for. > > But that's not what your patch is reporting. All you'

Re: Report planning memory in EXPLAIN ANALYZE

2023-08-11 Thread Ashutosh Bapat
other hand, developers can watch it easily in different > ways, if needed. So, I vote for the 'memory used' metric only. > > The patch looks good, passes the tests. Thanks Andrey. David, are you against reporting "memory used"? If you are not against reporting used me

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-08-11 Thread Ashutosh Bapat
I spent some time on 4th point below but also looked at other points. Here's what I have found so far On Thu, Jul 27, 2023 at 7:35 PM Ashutosh Bapat wrote: > > 1. The patch uses RestrictInfo::required_relids as the key for > searching child RelOptInfos. I am not sure which

AssertLog instead of Assert in some places

2023-08-11 Thread Ashutosh Bapat
r terminated abnormally before or while processing the request. The connection to the server was lost. Attempting reset: Failed. The connection to the server was lost. Attempting reset: Failed. @!#\q ``` If there is an interest in accepting the patch, I will add it to the commitfest an

Re: logical decoding and replication of sequences, take 2

2023-08-10 Thread Ashutosh Bapat
downstream as a principle. I think a way to achieve this is to update the replorigin_session_origin_lsn so that a sequence change applied once is not requested (and hence sent) again. [1] https://en.wikipedia.org/wiki/Rubber_banding -- Best Wishes, Ashutosh Bapat

Re: Report planning memory in EXPLAIN ANALYZE

2023-08-10 Thread Ashutosh Bapat
Hi David, On Wed, Aug 9, 2023 at 8:09 PM Ashutosh Bapat wrote: > > I need to just make sure that the Planning Memory is reported with SUMMARY ON. > The patch reports planning memory in EXPLAIN without ANALYZE when SUMMARY = ON. #explain (summary on) select * from a, b where a.c1 = b.c1

Re: Report planning memory in EXPLAIN ANALYZE

2023-08-10 Thread Ashutosh Bapat
On Wed, Aug 9, 2023 at 8:56 PM David Rowley wrote: > > On Thu, 10 Aug 2023 at 03:12, Ashutosh Bapat > wrote: > > Thinking more about it, I think memory used is the only right metrics. > > It's an optimization in MemoryContext implementation that malloc'ed >

Re: [BackendXidGetPid] only access allProcs when xid matches

2023-08-10 Thread Ashutosh Bapat
Please add this to commitfest so that it's not forgotten. On Wed, Aug 9, 2023 at 8:37 PM Junwang Zhao wrote: > > On Wed, Aug 9, 2023 at 10:46 PM Ashutosh Bapat > wrote: > > > > On Wed, Aug 9, 2023 at 9:30 AM Junwang Zhao wrote: > > > > > > In fun

Re: Report planning memory in EXPLAIN ANALYZE

2023-08-09 Thread Ashutosh Bapat
On Wed, Aug 9, 2023 at 8:09 PM Ashutosh Bapat wrote: > > Hi David, > Replying to your comments on this thread. > > > On Tue, Aug 8, 2023 at 11:40 AM Ashutosh Bapat > > wrote: > >> > >> Hi All, > >> I have been looking at memory consumed when

Re: [BackendXidGetPid] only access allProcs when xid matches

2023-08-09 Thread Ashutosh Bapat
can optimize this kind of inefficiency, I > believe we should ship with better code. > Looks good to me. However, I would just move the variable declaration with their assignments inside the if () rather than combing the expressions. It more readable that way. -- Best Wishes, Ashutosh Bapat

Re: Report planning memory in EXPLAIN ANALYZE

2023-08-09 Thread Ashutosh Bapat
Hi David, Replying to your comments on this thread. > On Tue, Aug 8, 2023 at 11:40 AM Ashutosh Bapat > wrote: >> >> Hi All, >> I have been looking at memory consumed when planning a partitionwise join >> [1], [2] and [3]. I am using the attached patch to measure t

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-08-09 Thread Ashutosh Bapat
On Wed, Aug 9, 2023 at 10:12 AM David Rowley wrote: > > On Fri, 28 Jul 2023 at 02:06, Ashutosh Bapat > wrote: > > 0001 - to measure memory consumption during planning. This is the same > > one as attached to [1]. > I have started a separate thread to discuss this

Re: Oversight in reparameterize_path_by_child leading to executor crash

2023-08-09 Thread Ashutosh Bapat
nt of outer rel. > Your changes in create_nestloop_path() only affect the check for parameterized path not the unparameterized paths. So I think we are good there. My worry was misplaced. -- Best Wishes, Ashutosh Bapat

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-08-08 Thread Ashutosh Bapat
lso figure out answers to 1, 2, 3 there. Please let me know your opinions if any. -- Best Wishes, Ashutosh Bapat [1] https://www.postgresql.org/message-id/CAExHW5s=bclmmq8n_bn6iu+pjau0ds3z_6dn6ile69esmsp...@mail.gmail.com From 4abdfd676a31d885ccdbe1803c60c1df1d0c1df2 Mon Sep 17 00:00:00 2001 Fro

Re: Oversight in reparameterize_path_by_child leading to executor crash

2023-08-08 Thread Ashutosh Bapat
hat the > path must be reparameterizable. Works for me. > > Attaching v3 patch to address all the reviews above. The patch looks good to me. Please add this to the next commitfest. -- Best Wishes, Ashutosh Bapat

Re: [PoC] Reducing planning time when tables have many partitions

2023-08-07 Thread Ashutosh Bapat
resql.org/message-id/CAExHW5sZA=5lj_zppro-w09ck8z9p7eayaqq3ks9gdfhrxe...@mail.gmail.com -- Best Wishes, Ashutosh Bapat

Report planning memory in EXPLAIN ANALYZE

2023-08-07 Thread Ashutosh Bapat
=-xlgrs9zgetr1nxwkgft2j-om...@mail.gmail.com -- Best Wishes, Ashutosh Bapat From 1dcadcc9cb38702f9e8882bf16173715982e36da Mon Sep 17 00:00:00 2001 From: Ashutosh Bapat Date: Wed, 12 Jul 2023 14:34:14 +0530 Subject: [PATCH] Report memory used for planning a query in EXPLAIN ANALYZE The memory used in the

Re: Oversight in reparameterize_path_by_child leading to executor crash

2023-08-07 Thread Ashutosh Bapat
On Fri, Aug 4, 2023 at 6:08 AM Richard Guo wrote: > > On Thu, Aug 3, 2023 at 7:20 PM Ashutosh Bapat < > ashutosh.bapat@gmail.com> wrote: > >> However, if reparameterization can *not* happen at the planning time, we >> have chosen a plan which can not be real

Re: [PoC] Reducing planning time when tables have many partitions

2023-08-07 Thread Ashutosh Bapat
ilar will be done to the planning memory. I will propose it as a separate patch in the next commitfest and will seek opinions from other hackers. -- Best Wishes, Ashutosh Bapat

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-08-04 Thread Ashutosh Bapat
On Fri, Jul 28, 2023 at 7:28 PM Ashutosh Bapat wrote: > > > > + bms_free(child_sjinfo->commute_above_l); > > + bms_free(child_sjinfo->commute_above_r); > > + bms_free(child_sjinfo->commute_below_l); > > + bms_free(child_sjinfo->commu

Re: [PoC] Reducing planning time when tables have many partitions

2023-08-03 Thread Ashutosh Bapat
On Wed, Aug 2, 2023 at 12:11 PM Yuya Watari wrote: > Hello, > > I really appreciate sharing very useful scripts and benchmarking results. > > On Fri, Jul 28, 2023 at 6:51 PM Ashutosh Bapat > wrote: > > Given that most of the developers run assert enabled builds it wo

Re: Oversight in reparameterize_path_by_child leading to executor crash

2023-08-03 Thread Ashutosh Bapat
On Thu, Aug 3, 2023 at 4:14 PM Richard Guo wrote: > > On Thu, Aug 3, 2023 at 12:38 PM Ashutosh Bapat < > ashutosh.bapat@gmail.com> wrote: > > Sometimes it would help to avoid the translation at all if we postpone > the reparameterization until createplan.c, such as

Re: Adding a LogicalRepWorker type field

2023-08-03 Thread Ashutosh Bapat
quired by all RelOptKinds crammed under the same structure. If we can avoid that here at the beginning itself, that will be great. May be we should create a union of type specific members while we are introducing the type. This will also provide some protection against unrelated members being (mis)used in the future. -- Best Wishes, Ashutosh Bapat

Re: Inquiry about Functionality Availability in PostgreSQL

2023-08-03 Thread Ashutosh Bapat
eSQL database, could you kindly let me know and provide any relevant insights about their potential implementation in future versions? > Core PostgreSQL may not provide some of the functionalities by itself but extensions like pg_audit may provide those functionalities. Maybe you want to lo

Re: Oversight in reparameterize_path_by_child leading to executor crash

2023-08-02 Thread Ashutosh Bapat
ed of in nearby threads. > As long as the translation happens only once, it should be fine. It's the repeated translations which should be avoided. -- Best Wishes, Ashutosh Bapat

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2023-07-28 Thread Ashutosh Bapat
y parent relids of the child rels. 2. Do not free those. 3. Add a comment about keeping the build and free functions in sync. I will work on those next. -- Best Wishes, Ashutosh Bapat

Re: logical decoding and replication of sequences, take 2

2023-07-28 Thread Ashutosh Bapat
ame of the function doesn't leave much space for expanding its functionality. So we are good with a new one. Probably some code deduplication. -- Best Wishes, Ashutosh Bapat

Re: logical decoding and replication of sequences, take 2

2023-07-28 Thread Ashutosh Bapat
On Tue, Jul 25, 2023 at 10:02 PM Tomas Vondra wrote: > > On 7/24/23 14:57, Ashutosh Bapat wrote: > > ... > > > >> > >> > >> 2) Currently, the sequences hash table is in reorderbuffer, i.e. global. > >> I was thinking maybe we should have it

Re: [PoC] Reducing planning time when tables have many partitions

2023-07-28 Thread Ashutosh Bapat
5pmt83r8uxvfcph+pvo5dnpdrvcsbgxe...@mail.gmail.com [2] https://www.postgresql.org/message-id/CAExHW5s4EqY43oB%3Dne6B2%3D-xLgrs9ZGeTr1NXwkGFt2j-OmaQQ%40mail.gmail.com -- Best Wishes, Ashutosh Bapat planning time measurement.ods Description: application/vnd.oasis.opendocument.spreadsheet

Re: Assert failure on bms_equal(child_joinrel->relids, child_joinrelids)

2023-07-28 Thread Ashutosh Bapat
ught I would mention it where I found it. [1] https://www.postgresql.org/message-id/caexhw5stmouobe55pmt83r8uxvfcph+pvo5dnpdrvcsbgxe...@mail.gmail.com -- Best Wishes, Ashutosh Bapat From 0f00fc48b9ebf73b54724ba82cec4a69d5f63846 Mon Sep 17 00:00:00 2001 From: Ashutosh Bapat Date: Wed, 12 Jul 2023 14:

Re: unrecognized node type while displaying a Path due to dangling pointer

2023-07-27 Thread Ashutosh Bapat
patch set uses references to free memory consumed by paths which remain unused. The memory consumed is substantial when partitionwise join is used and there are thousands of partitions. [1] https://www.postgresql.org/message-id/CAExHW5tUcVsBkq9qT%3DL5vYz4e-cwQNw%3DKAGJrtSyzOp3F%3DXacA%40mail.gmail.com -- Best Wishes, Ashutosh Bapat

Re: [May be a bug] double free or corruption

2023-07-27 Thread Ashutosh Bapat
pt > connections > [postgres@fedora postgres]$ psql > psql (17devel) > Type "help" for help. > > postgres=# \q > [postgres@fedora postgres]$ I guess gdb hit this bug after detaching from the backend you were debugging. So it continued running. In case it crashed because some

<    1   2   3   4   5   6   7   8   9   10   >