Hello,
Thank you for reviewing these patches.
On Thu, May 2, 2024 at 11:35 PM jian he wrote:
> on v24-0001:
> +/*
> + * add_eq_member - build a new EquivalenceMember and add it to an EC
> + */
> +static EquivalenceMember *
> +add_eq_member(EquivalenceClass *ec, Expr *expr, Relids relids,
> +
On Thu, May 2, 2024 at 3:57 PM Yuya Watari wrote:
>
hi. sorry to bother you, maybe a dumb question.
trying to understand something under the hood.
currently I only applied
v24-0001-Speed-up-searches-for-child-EquivalenceMembers.patch.
on v24-0001:
+/*
+ * add_eq_member - build a new
Hello Ashutosh,
Thank you for your email and for reviewing the patch. I sincerely
apologize for the delay in responding.
On Wed, Mar 6, 2024 at 11:16 PM Ashutosh Bapat
wrote:
> here's summary of observations
> 1. The patch improves planning time significantly (3X to 20X) and the
> improvement
Hi Yuya
On Wed, Feb 28, 2024 at 4:48 PM Yuya Watari wrote:
> Hello,
>
> On Tue, Feb 13, 2024 at 6:19 AM Alena Rybakina
> wrote:
> >
> > Yes, it is working correctly now with the assertion check. I suppose
> > it's better to add this code with an additional comment and a
> > recommendation for
Hi! Sorry my delayed reply too.
On 17.01.2024 12:33, Yuya Watari wrote:
Hello Alena,
Thank you for your quick response, and I'm sorry for my delayed reply.
On Sun, Dec 17, 2023 at 12:41 AM Alena Rybakina
wrote:
I thought about this earlier and was worried that the index links of the
Hi!
On 13.12.2023 09:21, Yuya Watari wrote:
Hello Alena, Andrei, and all,
I am sorry for my late response. I found that the current patches do
not apply to the master, so I have rebased those patches. I have
attached v22. For this later discussion, I separated the rebasing and
bug fixing that
Hello,
On Wed, Nov 22, 2023 at 2:32 PM Yuya Watari wrote:
> Unfortunately, I've been busy due to work, so I won't be able to
> respond for several weeks. I'm really sorry for not being able to see
> the patches. As soon as I'm not busy, I will look at them, consider
> the above approach, and
Hello Alena, Andrei, and all,
Thank you for reviewing this patch. I really apologize for not
updating this thread for a while.
On Sat, Nov 18, 2023 at 6:04 AM Alena Rybakina wrote:
> Hi, all!
>
> While I was reviewing the patches, I noticed that they needed some rebasing,
> and in one of the
On 27/9/2023 14:28, Yuya Watari wrote:
Thank you for pointing it out. The ec_source_indexes and
ec_derive_indexes are just picked up from the previous patch, and I
have not changed their design. I think a similar approach to
EquivalenceMembers may be applied to RestrictInfos. I will experiment
John Naylor writes:
> On Sat, Nov 18, 2023 at 4:04 AM Alena Rybakina
> wrote:
>> All diff files have already been added to
>> v21-0002-Introduce-indexes-for-RestrictInfo patch.
> Unfortunately, the patch tester is too smart for its own good, and
> will try to apply .diff files as well.
Yeah
On Sat, Nov 18, 2023 at 4:04 AM Alena Rybakina wrote:
>
> All diff files have already been added to
> v21-0002-Introduce-indexes-for-RestrictInfo patch.
Unfortunately, the patch tester is too smart for its own good, and
will try to apply .diff files as well. Since
Hello Ashutosh and Andrey,
On Wed, Sep 20, 2023 at 8:03 PM Ashutosh Bapat
wrote:
> While working on RestrictInfo translations patch I was thinking on
> these lines. [1] uses hash table for storing translated RestrictInfo.
> An EC can have a hash table to store ec_member translations. The same
>
On Wed, Sep 20, 2023, at 5:04 PM, Yuya Watari wrote:
> On Tue, Sep 19, 2023 at 5:21 PM Andrey Lepikhov
> wrote:
>> Working on self-join removal in the thread [1] nearby, I stuck into the
>> problem, which made an additional argument to work in this new direction
>> than a couple of previous ones.
On Wed, Sep 20, 2023 at 3:35 PM Yuya Watari wrote:
> I think we may be able to remove the eclass_child_members field by
> making child members on demand. v20 makes child members at
> add_[child_]join_rel_equivalences() and adds them into
> RelOptInfo->eclass_child_members. Instead of doing that,
Hello Ashutosh and Andrey,
Thank you for your email, and I really apologize for my late response.
On Thu, Sep 7, 2023 at 3:43 PM Ashutosh Bapat
wrote:
> It seems that you are still investigating and fixing issues. But the
> CF entry is marked as "needs review". I think a better status is
>
On 25/8/2023 14:39, Yuya Watari wrote:
Hello,
On Wed, Aug 9, 2023 at 8:54 PM David Rowley wrote:
I think the best way to move this forward is to explore not putting
partitioned table partitions in EMs and instead see if we can
translate to top-level parent before lookups. This might just be
Hi Yuya,
On Fri, Aug 25, 2023 at 1:09 PM Yuya Watari wrote:
>
> 3. Future works
>
> 3.1. Redundant memory allocation of Lists
>
> When we need child EquivalenceMembers in a loop over ec_members, v20
> adds them to the list. However, since we cannot modify the ec_members,
> v20 always copies it.
Hello David,
I really appreciate your quick reply.
On Wed, Aug 9, 2023 at 7:28 PM David Rowley wrote:
> If 0004 is adding an em_index to mark the index into
> PlannerInfo->eq_members, can't you use that in
> setup_eclass_member[_strict]_iterator to loop to verify that the two
> methods yield
On Wed, 9 Aug 2023 at 20:15, Yuya Watari wrote:
> I agree with your opinion that my patch lacks some explanations, so I
> will consider adding more comments. However, I received the following
> message from David in March.
>
> On Thu, Mar 9, 2023 at 6:23 AM David Rowley wrote:
> > For the main
On Wed, 9 Aug 2023 at 22:28, David Rowley wrote:
> i.e:
>
> + Bitmapset *matching_ems = NULL;
> + memcpy(_iter, iter, sizeof(EquivalenceMemberIterator));
> + memcpy(_iter, iter, sizeof(EquivalenceMemberIterator));
> +
> + idx_iter.use_index = true;
> + noidx_iter.use_index = false;
> +
> + while
On Wed, 5 Jul 2023 at 21:58, Yuya Watari wrote:
>
> Hello,
>
> On Fri, Mar 10, 2023 at 5:38 PM Yuya Watari wrote:
> > Thank you for pointing it out. I have attached the rebased version to
> > this email.
>
> Recent commits, such as a8c09daa8b [1], have caused conflicts and
> compilation errors
Hello Andrey, Ashutosh, and David,
Thank you for your reply and for reviewing the patch.
On Mon, Aug 7, 2023 at 5:51 PM Andrey Lepikhov
wrote:
> One more thing: I think, you should add comments to
> add_child_rel_equivalences() and add_child_join_rel_equivalences()
> on replacing of:
>
> if
Hi Andrey,
On Tue, Aug 8, 2023 at 8:52 AM Andrey Lepikhov
wrote:
> It is a positive thing to access some planner internals from the
> console, of course. My point is dedicated to the structuration of an
> EXPLAIN output and is caused by two reasons:
> 1. I use the EXPLAIN command daily to
On 7/8/2023 19:15, Ashutosh Bapat wrote:
On Mon, Aug 7, 2023 at 2:21 PM Andrey Lepikhov
mailto:a.lepik...@postgrespro.ru>> wrote:
>> Do you think that the memory measurement patch I have shared in
those threads is useful in itself? If so, I will start another
proposal to
On Mon, Aug 7, 2023 at 2:21 PM Andrey Lepikhov
wrote:
> >> Do you think that the memory measurement patch I have shared in those
> threads is useful in itself? If so, I will start another proposal to
> address it.
> >
> > For me, who is developing the planner in this thread, the memory
> >
On 7/8/2023 15:19, Yuya Watari wrote:
Hello,
Thank you for your reply.
On Thu, Aug 3, 2023 at 10:29 PM Ashutosh Bapat
wrote:
If you think that the verification is important to catch bugs, you may want to
encapsulate it with an #ifdef .. #endif such that the block within is not
compiled by
Hello,
Thank you for your reply.
On Thu, Aug 3, 2023 at 10:29 PM Ashutosh Bapat
wrote:
> If you think that the verification is important to catch bugs, you may want
> to encapsulate it with an #ifdef .. #endif such that the block within is not
> compiled by default. See OPTIMIZER_DEBUG for
On Wed, Aug 2, 2023 at 12:11 PM Yuya Watari wrote:
> Hello,
>
> I really appreciate sharing very useful scripts and benchmarking results.
>
> On Fri, Jul 28, 2023 at 6:51 PM Ashutosh Bapat
> wrote:
> > Given that most of the developers run assert enabled builds it would
> > be good to bring
Hello,
On Wed, Aug 2, 2023 at 6:43 PM Andrey Lepikhov
wrote:
> You introduced list_ptr_cmp as an extern function of a List, but use it
> the only under USE_ASSERT_CHECKING ifdef.
> Maybe you hide it under USE_ASSERT_CHECKING or remove all the stuff?
Thank you for your quick reply and for
On 2/8/2023 13:40, Yuya Watari wrote:
As seen from the above, verifying iteration results was the cause of
the performance degradation. I agree that we should avoid such
degradation because it negatively affects the development of
PostgreSQL. Removing the verification when committing this patch
Hello,
I really appreciate sharing very useful scripts and benchmarking results.
On Fri, Jul 28, 2023 at 6:51 PM Ashutosh Bapat
wrote:
> Given that most of the developers run assert enabled builds it would
> be good to bring down the degradation there while keeping the
> excellent speedup in
Hi Yuya, Andrey,
On Fri, Jul 28, 2023 at 9:58 AM Andrey Lepikhov
wrote:
> >>
> > Discovering quality of partition pruning at the stage of execution
> > initialization and using your set of patches I have found some dubious
> > results with performance degradation. Look into the test case in
> >
Hello,
On Fri, Jul 28, 2023 at 1:27 PM Andrey Lepikhov
wrote:
> Sorry for this. It was definitely a false alarm. In this patch,
> assertion checking adds much overhead. After switching it off, I found
> out that this feature solves my problem with a quick pass through the
> members of an
On 5/7/2023 16:57, Yuya Watari wrote:
Hello,
On Fri, Mar 10, 2023 at 5:38 PM Yuya Watari wrote:
Thank you for pointing it out. I have attached the rebased version to
this email.
Recent commits, such as a8c09daa8b [1], have caused conflicts and
compilation errors in these patches. I have
On Thu, 9 Mar 2023 at 01:34, Alvaro Herrera wrote:
> David, do you intend to continue to be involved in reviewing this one?
Yes. I'm currently trying to make a few Bitmapset improvements which
include the change made in this thread's 0001 patch over on [1].
For the main patch, I've been
Hello Watari-san, this patch does not currently apply. Could you please
rebase?
David, do you intend to continue to be involved in reviewing this one?
Thanks to both,
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"All rings of power are equal,
But some rings
On 2/6/23 06:47, Yuya Watari wrote:
Of course, I'm not sure if my approach in v16-0003 is ideal, but it
may help solve your concern above. Since simple_rel_array[0] is no
longer necessary with my patch, I removed the setup_append_rel_entry()
function in v16-0004. However, to work the patch, I
On Mon, 5 Dec 2022 at 21:28, David Rowley wrote:
>
> On Tue, 6 Dec 2022 at 04:45, Thom Brown wrote:
> > Testing your patches with the same 1024 partitions, each with 64
> > sub-partitions, I get a planning time of 205.020 ms, which is now a
> > 1,377x speedup. This has essentially reduced the
On Tue, 6 Dec 2022 at 04:45, Thom Brown wrote:
> Testing your patches with the same 1024 partitions, each with 64
> sub-partitions, I get a planning time of 205.020 ms, which is now a
> 1,377x speedup. This has essentially reduced the planning time from a
> catastrophe to a complete non-issue.
On Sun, 4 Dec 2022 at 00:35, David Rowley wrote:
>
> On Tue, 29 Nov 2022 at 21:59, Yuya Watari wrote:
> > Thank you for testing the patch with an actual query. This speedup is
> > very impressive. When I used an original query with 1024 partitions,
> > its planning time was about 200ms. Given
Dear Andrey and Thom,
Thank you for reviewing and testing the patch. I really apologize for
my late response.
On Tue, Nov 8, 2022 at 8:31 PM Andrey Lepikhov
wrote:
> Looking into find_em_for_rel() changes I see that you replaced
> if (bms_is_subset(em->em_relids, rel->relids)
> with assertion
On Thu, 17 Nov 2022 at 11:20, Thom Brown wrote:
>
> On Thu, 17 Nov 2022 at 09:31, Alvaro Herrera wrote:
> >
> > On 2022-Nov-16, Thom Brown wrote:
> >
> > > Once the issue Tom identified has been resolved, I'd like to test
> > > drive newer patches.
> >
> > What issue? If you mean the one from
On Thu, 17 Nov 2022 at 09:31, Alvaro Herrera wrote:
>
> On 2022-Nov-16, Thom Brown wrote:
>
> > Once the issue Tom identified has been resolved, I'd like to test
> > drive newer patches.
>
> What issue? If you mean the one from the thread "Reducing
> duplicativeness of EquivalenceClass-derived
On 2022-Nov-16, Thom Brown wrote:
> Once the issue Tom identified has been resolved, I'd like to test
> drive newer patches.
What issue? If you mean the one from the thread "Reducing
duplicativeness of EquivalenceClass-derived clauses", that patch is
already applied (commit a5fc46414deb), and
On Mon, 7 Nov 2022 at 06:33, Zhang Mingli wrote:
>
> HI,
>
> Regards,
> Zhang Mingli
> On Nov 7, 2022, 14:26 +0800, Tom Lane , wrote:
>
> Andrey Lepikhov writes:
>
> I'm still in review of your patch now. At most it seems ok, but are you
> really need both eq_sources and eq_derives lists now?
>
On 2/11/2022 15:27, Yuya Watari wrote:
I noticed that the previous patch does not apply to the current HEAD.
I attached the rebased version to this email.
Looking into find_em_for_rel() changes I see that you replaced
if (bms_is_subset(em->em_relids, rel->relids)
with assertion statement.
HI,
Regards,
Zhang Mingli
On Nov 7, 2022, 14:26 +0800, Tom Lane , wrote:
> Andrey Lepikhov writes:
> > I'm still in review of your patch now. At most it seems ok, but are you
> > really need both eq_sources and eq_derives lists now?
>
> Didn't we just have this conversation? eq_sources needs to
Andrey Lepikhov writes:
> I'm still in review of your patch now. At most it seems ok, but are you
> really need both eq_sources and eq_derives lists now?
Didn't we just have this conversation? eq_sources needs to be kept
separate to support the "broken EC" logic. We don't want to be
On 2/11/2022 15:27, Yuya Watari wrote:
Hello,
I noticed that the previous patch does not apply to the current HEAD.
I attached the rebased version to this email.
I'm still in review of your patch now. At most it seems ok, but are you
really need both eq_sources and eq_derives lists now? As I
Hello,
I noticed that the previous patch does not apply to the current HEAD.
I attached the rebased version to this email.
--
Best regards,
Yuya Watari
v8-0001-Apply-eclass_member_speedup_v3.patch.patch
Description: Binary data
v8-0002-Revert-one-of-the-optimizations.patch
Description:
Hello,
On Wed, Sep 21, 2022 at 6:43 PM Yuya Watari wrote:
> 1.1. Revert one of our optimizations (v5)
>
> As I mentioned in the comment in
> v[5|6|7]-0002-Revert-one-of-the-optimizations.patch, I reverted one of
> our optimizations. This code tries to find EquivalenceMembers that do
> not
Dear David,
On Fri, Aug 26, 2022 at 12:18 PM David Rowley wrote:
> How about instead of doing this caching like this, why don't we code
> up some iterators that we can loop over to fetch the required EMs.
Thank you very much for your quick reply and for sharing your idea
with code. I also think
On Fri, 26 Aug 2022 at 12:40, Yuya Watari wrote:
> This performance degradation is due to the heavy processing of the
> get_ec***_indexes***() functions. These functions are the core part of
> the optimization we are working on in this thread, but they are
> relatively heavy when the number of
esOn Tue, 9 Aug 2022 at 19:10, David Rowley wrote:
> I've not had a chance to look at the 0003 patch yet.
I've looked at the 0003 patch now.
The performance numbers look quite impressive, however, there were a
few things about the patch that I struggled to figure what they were
done the way you
On Mon, 8 Aug 2022 at 23:28, Yuya Watari wrote:
> If you have already applied David's patch, please start the 'git am'
> command from 0002-Fix-bugs.patch. All regression tests passed with
> this patch on my environment.
Thanks for fixing those scope bugs.
In regards to the 0002 patch, you have;
On Wed, 27 Jul 2022 at 18:07, Yuya Watari wrote:
> I agree that introducing a Bitmapset field may solve this problem. I
> will try this approach in addition to previous ones.
I've attached a very half-done patch that might help you get started
on this. There are still 2 failing regression tests
Dear David,
Thank you for sharing your new idea.
I agree that introducing a Bitmapset field may solve this problem. I
will try this approach in addition to previous ones.
Thank you again for helping me.
--
Best regards,
Yuya Watari
On Mon, 4 Jul 2022 at 09:28, Tom Lane wrote:
> For the bms_equal class of lookups, I wonder if we could get anywhere
> by adding an additional List field to every RelOptInfo that chains
> all EquivalenceMembers that match that RelOptInfo's relids.
> The trick here would be to figure out when to
Dear Andrey Lepikhov,
Thank you for replying and being a reviewer for this patch. I really
appreciate it.
> Are you still working on this patch?
Yes, I’m working on improving this patch. It is not easy to address
the problems that this patch has, but I’m hoping to send a new version
of it in a
On 7/5/22 13:57, Yuya Watari wrote:
On Mon, Jul 4, 2022 at 6:28 AM Tom Lane wrote:
Perhaps the bms_is_subset class could be handled in a similar
way, ie do a one-time pass to make a List of all EquivalenceMembers
that use a RelOptInfo.
Thank you for giving your idea. I will try to polish up
Dear Tom,
Thank you for replying to my email.
On Mon, Jul 4, 2022 at 6:28 AM Tom Lane wrote:
> I'm not real thrilled with trying to throw hashtables at the problem,
> though. As David noted, they'd be counterproductive for simple
> queries.
As you said, my approach that utilizes hash tables
Yuya Watari writes:
> On Thu, Mar 24, 2022 at 11:03 AM David Rowley wrote:
>> I think a better way to solve this would be just to have a single hash
>> table over all EquivalenceClasses that allows fast lookups of
>> EquivalenceMember->em_expr.
> If the predicate were "em->em_expr ==
Dear David,
Thank you for your comments on my patch. I really apologize for my
late response.
On Thu, Mar 24, 2022 at 11:03 AM David Rowley wrote:
> I think a better way to solve this would be just to have a single hash
> table over all EquivalenceClasses that allows fast lookups of
>
On Fri, 18 Mar 2022 at 23:32, Yuya Watari wrote:
> I found a problem that planning takes too much time when the tables
> have many child partitions. According to my observation, the planning
> time increases in the order of O(n^2). Here, n is the number of child
> partitions. I attached the patch
64 matches
Mail list logo
