Re: [HACKERS] Runtime Partition Pruning

2020-10-11 Thread Andy Fan
Hi Ashutosh: On Thu, Oct 8, 2020 at 7:25 PM Ashutosh Bapat wrote: > On Wed, Oct 7, 2020 at 7:00 PM Andy Fan wrote: > > > > > > > > On Wed, Oct 7, 2020 at 5:05 PM Andy Fan > wrote: > >> > >> > >> > >> On Sun, Oct 4, 2020 at

Wired if-statement in gen_partprune_steps_internal

2020-10-08 Thread Andy Fan
p_combine(context, step_ids, PARTPRUNE_COMBINE_INTERSECT); result = lappend(result, step); } } -- Best Regards Andy Fan

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-10-07 Thread Andy Fan
On Wed, Oct 7, 2020 at 9:55 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > On Wed, Sep 09, 2020 at 07:51:12AM +0800, Andy Fan wrote: > > > > Thank you Michael for checking it, I can reproduce the same locally after > > rebasing to the latest master. The attached

Re: [HACKERS] Runtime Partition Pruning

2020-10-07 Thread Andy Fan
On Wed, Oct 7, 2020 at 5:05 PM Andy Fan wrote: > > > On Sun, Oct 4, 2020 at 3:10 PM Andy Fan wrote: > >> >>> >>> Now, in my experience, the current system for custom plans vs. generic >>> plans doesn't approach the problem in this way at all, and in

Re: Improve choose_custom_plan for initial partition prune case

2020-10-07 Thread Andy Fan
Thank you Amit and Ashutosh for your reply! On Wed, Oct 7, 2020 at 8:41 PM Amit Langote wrote: > Hi Ashutosh, > > On Wed, Oct 7, 2020 at 8:40 PM Ashutosh Bapat > wrote: > > On Wed, Oct 7, 2020 at 11:20 AM Andy Fan > wrote: > > > On Mon, Oct 5, 2020 at 9:27 PM Ash

Re: [HACKERS] Runtime Partition Pruning

2020-10-07 Thread Andy Fan
On Sun, Oct 4, 2020 at 3:10 PM Andy Fan wrote: > >> >> Now, in my experience, the current system for custom plans vs. generic >> plans doesn't approach the problem in this way at all, and in my >> experience that results in some pretty terrible behavior. It will do &

Re: Improve choose_custom_plan for initial partition prune case

2020-10-07 Thread Andy Fan
On Wed, Oct 7, 2020 at 2:43 PM Andy Fan wrote: > >> 2. Associate them with RelationOid, and we can record such information in >> the >> Append node as well. The bad part is the same relation Oid may appear >> multiple >> times in a query. for example: SELEC

Re: Improve choose_custom_plan for initial partition prune case

2020-10-07 Thread Andy Fan
tkey2 = $2; > > I just came up with a new idea. Since this situation should be rare, we can just come back to our original method (totally ignore the cost reduction) or use the average number. Fixing the 99% cases would be a big winner as well IMO. So any hint on this will be appreciated..

Re: Improve choose_custom_plan for initial partition prune case

2020-10-07 Thread Andy Fan
On Sat, Oct 3, 2020 at 10:05 AM Andy Fan wrote: > Hi Amit: > > Very glad to see your comment! > > On Fri, Oct 2, 2020 at 4:21 PM Amit Langote > wrote: > >> Hi Andy, >> >> On Fri, Oct 2, 2020 at 1:04 AM Andy Fan wrote: >> > >> > G

Re: Improve choose_custom_plan for initial partition prune case

2020-10-06 Thread Andy Fan
Hi Ashutosh: Thanks for coming. On Mon, Oct 5, 2020 at 9:27 PM Ashutosh Bapat wrote: > On Thu, Oct 1, 2020 at 9:34 PM Andy Fan wrote: > > > > Given the plan example: > > > > CREATE TABLE measurement ( > > city_id int not null, >

Re: [HACKERS] Runtime Partition Pruning

2020-10-04 Thread Andy Fan
Any thought about this? I'd be sorry if I missed some already existing discussion on this topic. [1] https://www.postgresql.org/message-id/CA%2BHiwqGsP2L0BW1ad58HRSj1NouNSVHLfL5pm7%3DPBTvL0b%2B-BQ%40mail.gmail.com -- Best Regards Andy Fan

Re: Improve choose_custom_plan for initial partition prune case

2020-10-02 Thread Andy Fan
Hi Amit: Very glad to see your comment! On Fri, Oct 2, 2020 at 4:21 PM Amit Langote wrote: > Hi Andy, > > On Fri, Oct 2, 2020 at 1:04 AM Andy Fan wrote: > > > > Given the plan example: > > > > CREATE TABLE measurement ( > > city_id int

Improve choose_custom_plan for initial partition prune case

2020-10-01 Thread Andy Fan
and then reducing such costs from the generic plan. I just went through the related code but didn't write anything now. I'd like to see if this is a correct direction to go. -- Best Regards Andy Fan

Re: BLOB / CLOB support in PostgreSQL

2020-09-28 Thread Andy Fan
ods on the Clob interface must be fully implemented if the JDBC Spec> * JDBC driver supports the data type. What would be the sense behind this? This is not reasonable based on limited experience. To be short, I am supporting Andrew's proposal for now. -- Best Regards Andy Fan

Re: Partition prune with stable Expr

2020-09-28 Thread Andy Fan
On Mon, Sep 28, 2020 at 9:17 PM Thomas Kellerer wrote: > Andy Fan schrieb am 28.09.2020 um 02:54: > > Well, that's very interesting. Specific to my user case, > > SELECT * FROM p WHERE pkey = to_date('2018-12-13', '-mm-dd)'; > > Why use to_date() at all for a constant v

Re: Partition prune with stable Expr

2020-09-28 Thread Andy Fan
On Mon, Sep 28, 2020 at 7:15 AM Tom Lane wrote: > Andy Fan writes: > > On Mon, Sep 28, 2020 at 4:46 AM David Rowley > wrote: > >> Thanks for showing an interest in partition pruning. Unfortunately, > >> it's not possible to use stable functions to prune p

Re: Partition prune with stable Expr

2020-09-28 Thread Andy Fan
On Mon, Sep 28, 2020 at 2:44 PM Jesse Zhang wrote: > On Sun, Sep 27, 2020 at 7:52 PM Andy Fan wrote: > > > > > > On Mon, Sep 28, 2020 at 9:15 AM Tom Lane wrote: > >> > >> Andy Fan writes: > >> > Well, that's very interesting. Specific to my use

Re: Partition prune with stable Expr

2020-09-27 Thread Andy Fan
On Mon, Sep 28, 2020 at 9:15 AM Tom Lane wrote: > Andy Fan writes: > > Well, that's very interesting. Specific to my user case, > > SELECT * FROM p WHERE pkey = to_date('2018-12-13', '-mm-dd)'; > > p has 1500+ partitions and planning takes lots of time, which is so

Re: Partition prune with stable Expr

2020-09-27 Thread Andy Fan
On Mon, Sep 28, 2020 at 7:15 AM Tom Lane wrote: > Andy Fan writes: > > On Mon, Sep 28, 2020 at 4:46 AM David Rowley > wrote: > >> Thanks for showing an interest in partition pruning. Unfortunately, > >> it's not possible to use stable functions to prune p

Re: Partition prune with stable Expr

2020-09-27 Thread Andy Fan
Thank you David for coming:) On Mon, Sep 28, 2020 at 4:46 AM David Rowley wrote: > On Mon, 28 Sep 2020 at 08:59, Andy Fan wrote: > > I find we can't prune partitions in the planner if the qual is a stable > function. > > > IMO, we should do it. Why not? > > Tha

Partition prune with stable Expr

2020-09-27 Thread Andy Fan
yyy-mm-dd'::text)) (4 rows) IMO, we should do it. Why not? The attached is used to show the things in my mind. Btw, why the to_date function is declared as stable rather than immutable since it always delivers the same result for the same inputs. -- Best Regards Andy Fan v1-0001-Allow-planner

Re: Dynamic gathering the values for seq_page_cost/xxx_cost

2020-09-26 Thread Andy Fan
On Sat, Sep 26, 2020 at 1:51 PM Julien Rouhaud wrote: > On Sat, Sep 26, 2020 at 8:17 AM Andy Fan wrote: > > > > As for the testing with cache considered, I found how to estimate cache > hit > > ratio is hard or how to control a hit ratio to test is hard. Recently I &g

Re: Dynamic gathering the values for seq_page_cost/xxx_cost

2020-09-25 Thread Andy Fan
On Fri, Sep 25, 2020 at 5:15 PM Ashutosh Bapat wrote: > On Tue, Sep 22, 2020 at 10:57 AM Andy Fan > wrote: > > > > > > My tools set the random_page_cost to 8.6, but based on the fio data, it > should be > > set to 12.3 on the same hardware. and I do see th

Re: Dynamic gathering the values for seq_page_cost/xxx_cost

2020-09-22 Thread Andy Fan
0%. [1] https://www.postgresql.org/message-id/CAKU4AWpRv50k8E3tC3tiLWGe2DbKaoZricRh_YJ8y_zK%2BHdSjQ%40mail.gmail.com -- Best Regards Andy Fan

Re: Dynamic gathering the values for seq_page_cost/xxx_cost

2020-09-21 Thread Andy Fan
Thanks Ashutosh for coming:) On Mon, Sep 21, 2020 at 9:03 PM Ashutosh Bapat wrote: > On Mon, Sep 21, 2020 at 9:11 AM Andy Fan wrote: > > > > Here are some changes for my detection program. > > > > | | seq_read_lat (

Re: Dynamic gathering the values for seq_page_cost/xxx_cost

2020-09-20 Thread Andy Fan
/u/y/g/fdirect> sudo ./main fs_cache_lat = 0.569031us, seq_read_lat = 18.901749us, random_page_lat = 148.650589us cache hit ratio: 1.000000 random_page_cost 1.00 cache hit ratio: 0.90 random_page_cost 6.401019 cache hit ratio: 0.50 random_page_cost 7.663772 cache hit ratio: 0.10 random_page_cost 7.841498 cache hit ratio: 0.00 random_page_cost 7.864383 This result looks much different from "we should use 1.1 ~ 1.5 for SSD". The attached is the modified detection program. [1] https://www.cdw.com/product/lsi-megaraid-sas-9271-8i-storage-controller-raid-sas-pcie-3.0-x8/4576538#PO -- Best Regards Andy Fan main.c Description: Binary data

Re: Get rid of runtime handling of AlternativeSubPlan?

2020-08-31 Thread Andy Fan
On Tue, Sep 1, 2020 at 1:42 AM Tom Lane wrote: > Andy Fan writes: > > Thank you for this code! I still have some confusion about when a > SubPlan > > should be executed when a join is involved. I care about this because > this > > has an impact on when we can get

Re: Get rid of runtime handling of AlternativeSubPlan?

2020-08-30 Thread Andy Fan
art PARTITION OF hash_parted2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); psql:src/test/regress/sql/create_table.sql:622: ERROR: partition "fail_part" would overlap partition "h2part_4" [1] https://www.postgresql.org/message-id/07b3fa88-aa4e-2e13-423d-8389eb1712cf%40imap.cc -- Best Regards Andy Fan

Re: Improve planner cost estimations for alternative subplans

2020-08-28 Thread Andy Fan
On Wed, Aug 26, 2020 at 4:21 PM Andy Fan wrote: > > > On Mon, Aug 17, 2020 at 10:12 PM Andy Fan > wrote: > >> >> >> On Mon, Jun 22, 2020 at 9:39 AM Tom Lane wrote: >> >>> I wrote: >>> > Nope. The entire reason why we have that kluge

Re: How is bushy plans generated in join_search_one_lev

2020-08-26 Thread Andy Fan
On Thu, Aug 27, 2020 at 8:05 AM Tom Lane wrote: > Andy Fan writes: > > I do see the README says we support bushy plans and I also see bushy > > plans in real life (for example tpc-h Q20) like below. However I don't > know > > how it is generated with the algorithm in j

How is bushy plans generated in join_search_one_lev

2020-08-26 Thread Andy Fan
Index Cond: (p_partkey = partsupp.ps_partkey) Filter: ((p_name)::text ~~ 'lavender%'::text) (21 rows) -- Best Regards Andy Fan

Re: Improve planner cost estimations for alternative subplans

2020-08-26 Thread Andy Fan
On Mon, Aug 17, 2020 at 10:12 PM Andy Fan wrote: > > > On Mon, Jun 22, 2020 at 9:39 AM Tom Lane wrote: > >> I wrote: >> > Nope. The entire reason why we have that kluge is that we don't know >> > until much later how many times we expect to execute the sub

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-08-26 Thread Andy Fan
On Wed, Aug 26, 2020 at 8:14 AM David Rowley wrote: > On Wed, 26 Aug 2020 at 05:18, Andy Fan wrote: > > > > > > On Tue, Aug 25, 2020 at 11:53 PM Andres Freund > wrote: > >> > >> On 2020-08-25 20:48:37 +1200, David Rowley wrote: > >> > Als

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-08-25 Thread Andy Fan
> > Will you stand in my way if I want to push ahead with the separate > > node? Will anyone else? > > I feel pretty darn strongly about this. If there's plenty people on your > side I'll not stand in your way, but I think this is a bad design based on > pretty flimsy reasons. > > Greetings, > > Andres Freund > > > -- Best Regards Andy Fan

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-08-25 Thread Andy Fan
the node? > > Will you stand in my way if I want to push ahead with the separate > > node? Will anyone else? > > I feel pretty darn strongly about this. If there's plenty people on your > side I'll not stand in your way, but I think this is a bad design based on > pretty flimsy reasons. > > Nice to see the different opinions from two great guys and interesting to see how this can be resolved at last:) -- Best Regards Andy Fan

Re: Pulling up direct-correlated ANY_SUBLINK

2020-08-18 Thread Andy Fan
l. 4). no limit/offset cause. 5). No volatile function involved for safety. I can't tell how often it is, I just run into this by my own and search the maillist and get only 1 report [1]. Is it something worth doing or do we have a better strategy to handle it? Thanks! [1] https://www.postgresql.org/message-id/3691.1342650...@sss.pgh.pa.us -- Best Regards Andy Fan

Re: Improve planner cost estimations for alternative subplans

2020-08-17 Thread Andy Fan
we save relevant rowcounts rather > than recomputing them in createplan.c. Is it worth doing? I dunno. > AlternativeSubPlan is pretty much a backwater, I think --- if it > were interesting performance-wise to a lot of people, more would > have been done with it by now. > > regards, tom lane > > -- Best Regards Andy Fan v1-0001-Convert-the-AlternativeSubplan-to-Subplan-as-soon.patch Description: Binary data

Re: Allows Extend Protocol support CURSOR_OPT_HOLD with prepared stmt.

2020-08-12 Thread Andy Fan
On Wed, Aug 12, 2020 at 8:21 PM Dave Cramer wrote: > > > On Wed, 12 Aug 2020 at 08:14, Andy Fan wrote: > >> >> >> On Wed, Aug 12, 2020 at 8:11 PM Andy Fan >> wrote: >> >>> >>> >>> On Wed, Aug 12, 2020 at 5:54 PM Dave Cramer

Re: Allows Extend Protocol support CURSOR_OPT_HOLD with prepared stmt.

2020-08-12 Thread Andy Fan
On Wed, Aug 12, 2020 at 8:11 PM Andy Fan wrote: > > > On Wed, Aug 12, 2020 at 5:54 PM Dave Cramer > wrote: > >> >> >> >> On Tue, 11 Aug 2020 at 22:33, Andy Fan wrote: >> >>> >>> >>> On Mon, Jul 27, 2020 at 11:57 AM

Re: Allows Extend Protocol support CURSOR_OPT_HOLD with prepared stmt.

2020-08-12 Thread Andy Fan
On Wed, Aug 12, 2020 at 5:54 PM Dave Cramer wrote: > > > > On Tue, 11 Aug 2020 at 22:33, Andy Fan wrote: > >> >> >> On Mon, Jul 27, 2020 at 11:57 AM Andy Fan >> wrote: >> >>> >>>> 2. Currently I want to add a new GUC parameter,

Re: Can I test Extended Query in core test framework

2020-08-11 Thread Andy Fan
On Tue, Aug 11, 2020 at 11:22 PM Tom Lane wrote: > Andy Fan writes: > > I want to write some test cases with extended query in core test system. > > Why? (That is, what is it you need to test exactly?) > > Thanks for your attention. The background is I hacked exec_bind_me

Re: Allows Extend Protocol support CURSOR_OPT_HOLD with prepared stmt.

2020-08-11 Thread Andy Fan
On Mon, Jul 27, 2020 at 11:57 AM Andy Fan wrote: > >> 2. Currently I want to add a new GUC parameter, if set it to true, server >> will >> create a holdable portal, or else nothing changed. Then let the user set >> it to true in the above case and reset it to false

Re: Can I test Extended Query in core test framework

2020-08-11 Thread Andy Fan
ed queries. [1] https://www.postgresql.org/message-id/CAKU4AWqvwmo=nlpga_ohxb4f+u4ts1_3yry9m6xtjlt9dkh...@mail.gmail.com -- Best Regards Andy Fan

Can I test Extended Query in core test framework

2020-08-10 Thread Andy Fan
\->prepare' and '\->execute' and get nothing. am I miss something? -- Best Regards Andy Fan

Re: FailedAssertion("pd_idx == pinfo->nparts", File: "execPartition.c", Line: 1689)

2020-08-06 Thread Andy Fan
On Fri, Aug 7, 2020 at 8:32 AM Tom Lane wrote: > Andy Fan writes: > > Attached is the v2 patch. > > Forgot to mention that I'd envisioned adding this as a src/test/modules/ > module; contrib/ is for things that we intend to expose to users, which > I think this isn't

Re: FailedAssertion("pd_idx == pinfo->nparts", File: "execPartition.c", Line: 1689)

2020-08-06 Thread Andy Fan
On Thu, Aug 6, 2020 at 10:42 PM Tom Lane wrote: > Andy Fan writes: > > On Thu, Aug 6, 2020 at 12:02 PM Tom Lane wrote: > >> See my straw-man proposal downthread. > > > Thanks for your explanation, I checked it again and it looks a very clean > > method. Th

Re: FailedAssertion("pd_idx == pinfo->nparts", File: "execPartition.c", Line: 1689)

2020-08-06 Thread Andy Fan
On Thu, Aug 6, 2020 at 12:02 PM Tom Lane wrote: > Andy Fan writes: > > On Thu, Aug 6, 2020 at 2:22 AM Tom Lane wrote: > >> In the longer term, it's annoying that we have no test methodology > >> for this other than "manually set a breakpoint here". > >

Re: FailedAssertion("pd_idx == pinfo->nparts", File: "execPartition.c", Line: 1689)

2020-08-05 Thread Andy Fan
ree later > (even after relcache inval would normally have forced replanning). > Obviously that could not be a normal SQL-accessible feature, because > some types of invals would make the plan completely wrong, but for > testing purposes it'd be mighty helpful to check that a stale plan > still works. > > regards, tom lane > > > -- Best Regards Andy Fan

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-08-03 Thread Andy Fan
y access of unique_index->opfamily[c] – this array only has nkeycolumns > entries. > You are correct, I would include this in the next version patch, Thank you for this checking! -- Andy Fan Best Regards > > > > > *From:* Andy Fan > *Sent:* Sunday 19 July 2020 5:

Re: Index Skip Scan (new UniqueKeys)

2020-08-02 Thread Andy Fan
s. As for the DISTINCT/GROUP BY case, we should build the UniqueKeys from root->distinct_pathkeys and root->group_pathkeys where the EquivalenceClasses are already there. I am still not insisting on either Expr or EquivalenceClasses right now, if we need to change it to EquivalenceClasses, I'd see if we need to have more places to take care before doing that. -- Best Regards Andy Fan

Re: Allows Extend Protocol support CURSOR_OPT_HOLD with prepared stmt.

2020-07-26 Thread Andy Fan
hod? > > I forget to say in this case, the user has to drop the holdable portal explicitly. -- Best Regards Andy Fan

Allows Extend Protocol support CURSOR_OPT_HOLD with prepared stmt.

2020-07-26 Thread Andy Fan
server will create a holdable portal, or else nothing changed. Then let the user set it to true in the above case and reset it to false afterward. Is there any issue with this method? -- Best Regards Andy Fan

Re: Difference for Binary format vs Text format for client-server communication

2020-07-26 Thread Andy Fan
On Sun, Jul 26, 2020 at 1:49 AM Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 2020-07-16 18:52, Andy Fan wrote: > > The reason I ask this is because I have a task to make numeric output > > similar to oracle. > > > > Oracle: >

Difference for Binary format vs Text format for client-server communication

2020-07-16 Thread Andy Fan
umn? 2. (1 row) If the user uses text format, I can just hack some numeric_out function, but if they use binary format, looks I have to change the driver they used for it. Am I understand it correctly? -- Best Regards Andy Fan

Re: Implementing Incremental View Maintenance

2020-07-09 Thread Andy Fan
quot;not" word:( > > >> - inheritance parent table > >> ... > >> - targetlist containing IVM column > >> - simple subquery is only supported > >> > > > > How to understand 3 items above? > > The best way to understand them is looking into regression test. > Thanks for sharing, I will look into it. -- Best Regards Andy Fan

Re: Implementing Incremental View Maintenance

2020-07-06 Thread Andy Fan
Thanks for the patch! > Query checks for following restrictions are added: Are all known supported cases listed below? > - inheritance parent table > ... > - targetlist containing IVM column > - simple subquery is only supported > How to understand 3 items above? - Best Regards Andy Fan

Re: Commitfest 2020-07

2020-06-28 Thread Andy Fan
poken up that I've missed, I'm happy to >> volunteer to >> > run CFM for this one. >> >> No one has volunteered that I recall, so the baton is yours. >> >> > Enjoy! > > //Magnus > > Thanks for the volunteering! -- Best Regards Andy Fan

Re: hash as an search key and hash collision

2020-06-19 Thread Andy Fan
On Sat, Jun 20, 2020 at 12:34 AM Tomas Vondra wrote: > On Fri, Jun 19, 2020 at 04:24:01PM +0800, Andy Fan wrote: > >I want to maintain an internal table which the primary key is sql_text and > >planstmt::text, it is efficient since it both may be very long. So a > >gene

hash as an search key and hash collision

2020-06-19 Thread Andy Fan
/pg_stat_statements/pg_stat_statements.c#L154 -- Best Regards Andy Fan

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-06-11 Thread Andy Fan
On Wed, Jun 3, 2020 at 10:36 AM Andy Fan wrote: > >> Thanks for running those tests. I had a quick look at the results and >> I think to say that all 4 are better is not quite right. One is >> actually a tiny bit slower and one is only faster due to a plan >> ch

Re: Index Skip Scan (new UniqueKeys)

2020-06-11 Thread Andy Fan
mo=# set enable_indexskipscan to on; SET demo=# select distinct im5, i, im100 from j1 where i < 2; im5 | i | im100 -+---+--- 1 | 1 | 0 (1 row) -- Best Regards Andy Fan

Re: A wrong index choose issue because of inaccurate statistics

2020-06-08 Thread Andy Fan
á 5. 6. 2020 v 8:19 odesílatel David Rowley > napsal: > >> > >> On Mon, 1 Jun 2020 at 01:24, Andy Fan wrote: > >> > The one-line fix describe the exact idea in my mind: > >> > > >> > +++ b/src/backend/optimizer/path/costsize.c > &g

Re: A wrong index choose issue because of inaccurate statistics

2020-06-08 Thread Andy Fan
On Fri, Jun 5, 2020 at 2:30 PM Pavel Stehule wrote: > > > pá 5. 6. 2020 v 8:19 odesílatel David Rowley > napsal: > >> On Mon, 1 Jun 2020 at 01:24, Andy Fan wrote: >> > The one-line fix describe the exact idea in my mind: >> > >> > +++ b/src/

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-06-04 Thread Andy Fan
On Fri, Jun 5, 2020 at 10:57 AM David Rowley wrote: > On Fri, 5 Jun 2020 at 14:36, Andy Fan wrote: > > On Mon, May 25, 2020 at 2:34 AM David Rowley > wrote: > >> > >> On Sun, 24 May 2020 at 04:14, Dmitry Dolgov <9erthali...@gmail.com> > wrote: > >&g

Re: A wrong index choose issue because of inaccurate statistics

2020-06-04 Thread Andy Fan
worth the troubles. My current implementation looks not cool, so any suggestion to research further is pretty welcome. -- Best Regards Andy Fan

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-06-04 Thread Andy Fan
ied to think it as root->distinct_pathkeys, however I didn't fully understand where root->distinct_pathkeys is used for as well. [1] https://www.postgresql.org/message-id/CAKU4AWq%3DwWkAo-CDOQ5Ea6UwYvZCgb501w6iqU0rtnTT-zg6bQ%40mail.gmail.com -- Best Regards Andy Fan

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-06-02 Thread Andy Fan
s://github.com/zhihuiFan/tpch-postgres -- Best Regards Andy Fan

Re: Index Skip Scan

2020-06-02 Thread Andy Fan
On Tue, Jun 2, 2020 at 9:38 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > On Tue, Jun 02, 2020 at 08:36:31PM +0800, Andy Fan wrote: > > > > > Other than that to summarize current open points for future readers > > > (this thread somehow became quite b

Re: Index Skip Scan

2020-06-02 Thread Andy Fan
on from David, like blinking twice or something). > > * Suspicious performance difference between different type of workload, > mentioned by Tomas (unfortunately I had no chance yet to investigate). > > * Thinking about supporting conditions, that are not covered by the index, > to make skipping more flexible (one of the potential next steps in the > future, as suggested by Floris). > Looks this is the latest patch, which commit it is based on? Thanks -- Best Regards Andy Fan

A wrong index choose issue because of inaccurate statistics

2020-05-31 Thread Andy Fan
list/id/8810.1590714...@sss.pgh.pa.us [2] https://ankane.org/tpc-h -- Best Regards Andy Fan normal.log Description: Binary data patched.log Description: Binary data reproduce.sql Description: Binary data

Re: Make the qual cost on index Filter slightly higher than qual cost on index Cond.

2020-05-29 Thread Andy Fan
its in cost, which is not accurate enough to show the difference. However with a nest loop, the overall plan shows the cost difference. [1] https://ankane.org/tpc-h -- Best Regards Andy Fan normal.log Description: Binary data patched.log Description: Binary data

Re: Make the qual cost on index Filter slightly higher than qual cost on index Cond.

2020-05-29 Thread Andy Fan
On Fri, May 29, 2020 at 9:37 PM Ashutosh Bapat wrote: > On Fri, May 29, 2020 at 6:40 AM Andy Fan wrote: > > > > > >> > >> >so we need to optimize the cost model for such case, the method is the > >> >patch I mentioned above. > >> > &

Re: Make the qual cost on index Filter slightly higher than qual cost on index Cond.

2020-05-28 Thread Andy Fan
Thanks all of you for your feedback. On Fri, May 29, 2020 at 9:04 AM Tom Lane wrote: > Tomas Vondra writes: > > On Wed, May 27, 2020 at 09:58:04PM +0800, Andy Fan wrote: > >> so we need to optimize the cost model for such case, the method is the > >> patch I me

Re: Make the qual cost on index Filter slightly higher than qual cost on index Cond.

2020-05-28 Thread Andy Fan
e it cpu_tuple_cost + qpqual_cost.per_tuple + (0.0001) * list_lenght(qpquals). -- Best Regards Andy Fan

Re: Make the qual cost on index Filter slightly higher than qual cost on index Cond.

2020-05-27 Thread Andy Fan
On Wed, May 27, 2020 at 8:01 PM Ashutosh Bapat < ashutosh.ba...@2ndquadrant.com> wrote: > > > On Wed, 27 May 2020 at 04:43, Andy Fan wrote: > >> You can use the attached sql to reproduce this issue, but I'm not sure >> you can >> get the above result at

Re: Make the qual cost on index Filter slightly higher than qual cost on index Cond.

2020-05-26 Thread Andy Fan
. -- Best Regards Andy Fan index_choose.sql Description: Binary data

Re: Make the qual cost on index Filter slightly higher than qual cost on index Cond.

2020-05-26 Thread Andy Fan
On Tue, May 26, 2020 at 9:59 PM Ashutosh Bapat wrote: > On Tue, May 26, 2020 at 1:52 PM Andy Fan wrote: > > > > > > Consider the below example: > > > > create table j1(i int, im5 int, im100 int, im1000 int); > > insert into j1 select i, i%5, i%100, i%10

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-05-26 Thread Andy Fan
On Fri, May 22, 2020 at 9:51 PM Fujii Masao wrote: > > > On 2020/05/22 15:10, Andy Fan wrote: > > > > > > On Thu, May 21, 2020 at 3:49 PM Julien Rouhaud <mailto:rjuju...@gmail.com>> wrote: > > > > Le jeu. 21 mai 2020 à 09:17, Michae

Make the qual cost on index Filter slightly higher than qual cost on index Cond.

2020-05-26 Thread Andy Fan
, however I'm open for suggestion on that as well. Any suggestions? -- Best Regards Andy Fan

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-05-22 Thread Andy Fan
On Thu, May 21, 2020 at 3:49 PM Julien Rouhaud wrote: > Le jeu. 21 mai 2020 à 09:17, Michael Paquier a > écrit : > >> On Thu, May 21, 2020 at 08:49:53AM +0200, Julien Rouhaud wrote: >> > On Tue, May 19, 2020 at 4:29 AM Andy Fan >> wrote: >> >> Thanks

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-05-22 Thread Andy Fan
On Thu, May 21, 2020 at 3:17 PM Michael Paquier wrote: > On Thu, May 21, 2020 at 08:49:53AM +0200, Julien Rouhaud wrote: > > On Tue, May 19, 2020 at 4:29 AM Andy Fan > wrote: > >> Thanks for the excellent extension. I want to add 5 more fields to > satisfy the >

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-05-21 Thread Andy Fan
e where j1o.i = 1; -- Best Regards Andy Fan

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-21 Thread Andy Fan
On Fri, May 22, 2020 at 4:52 AM David Rowley wrote: > On Thu, 14 May 2020 at 14:39, Andy Fan wrote: > > > > On Thu, May 14, 2020 at 6:20 AM David Rowley > wrote: > >> Having the "onerow" flag was not how I intended it to work. > >> > > Th

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-21 Thread Andy Fan
> if I understand correctly those two are introducing the concept, and others are just using it You are understand it correctly. -- Best Regards Andy Fan

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-21 Thread Andy Fan
ht be useful to try those > when planning joins. > > Yes, In current implementation, we also add UniqueKey during create_xxx_paths, xxx may be grouping/union. after the index skipscan patch, we can do the similar things in create_indexskip_path. -- Best Regards Andy Fan

Re: Subplan result caching

2020-05-20 Thread Andy Fan
On Wed, May 20, 2020 at 7:47 PM David Rowley wrote: > On Mon, 27 Apr 2020 at 00:37, Andy Fan wrote: > > I was feeling that we may have to maintain some extra status if we use > hash > > table rather than tuple store, but that might be not a major concern. I > can > &

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-05-18 Thread Andy Fan
in nature. However I don't think the above 2 will cause big issues. I added the columns to V1_8 rather than adding a new version. this can be changed at final patch. Any suggestions? Best Regards Andy Fan v1-0001-Add-query-characters-information-to-pg_stat_state.patch Description: Binary data

Find query characters in respect of optimizer for develop purpose

2020-05-18 Thread Andy Fan
of standard_planner, and the values are increased at make_subplan, set_subquery_pathlist, make_one_rel, create_grouping_paths. later it can be tracked and viewed in pg_stat_statements. What do you think about the requirement and the method I am thinking? Any kind of feedback is welcome. -- Best Regards Andy

Re: Add "-Wimplicit-fallthrough" to default flags

2020-05-14 Thread Andy Fan
On Wed, May 13, 2020 at 10:02 PM Tom Lane wrote: > Andy Fan writes: > >> FWIW, I got a warning for jsonpath_gram.c. > > Ugh. Confirmed here on Fedora 30 (bison 3.0.5). > > > I just found this just serval minutes ago. Upgrading your bison to the > > latest vers

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-13 Thread Andy Fan
On Wed, May 13, 2020 at 11:48 PM Andy Fan wrote: > >> My impression about the one row stuff, is that there is too much >> special casing around it. We should somehow structure the UniqueKey >> data so that one row unique keys come naturally rather than special >&

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-13 Thread Andy Fan
On Thu, May 14, 2020 at 6:20 AM David Rowley wrote: > On Thu, 14 May 2020 at 03:48, Andy Fan wrote: > > On Wed, May 13, 2020 at 8:04 PM Ashutosh Bapat < > ashutosh.bapat@gmail.com> wrote: > >> My impression about the one row stuff, is that there is too much > &

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-13 Thread Andy Fan
ge the multi_nullvals to true at this case */ >> >> Need a comment explaining this. Generally, I feel, this and other >> functions in >> this file need good comments explaining the logic esp. "why" instead of >> "what". > > > Exactly. > Done in v8. >> Will continue reviewing your new set of patches as time permits. >> > > Thank you! Actually there is no big difference between v6 and v7 > regarding the > UniqueKey part except 2 bug fix. However v7 has some more documents, > comments improvement and code refactor/split, which may be helpful > for review. You may try v7 next time if v8 has not come yet:) > > v8 has come :) Best Regards Andy Fan

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-13 Thread Andy Fan
On Wed, May 13, 2020 at 8:04 PM Ashutosh Bapat wrote: > On Fri, May 8, 2020 at 7:27 AM Andy Fan wrote: > > >> +else if (inner_is_onerow) > >> +{ > >> +/* Since rows in innerrel can't be duplicat

Re: Add "-Wimplicit-fallthrough" to default flags

2020-05-13 Thread Andy Fan
; > > /* Fall through. */ > > default: > > It is generated code by bison. > > $ bison --version > bison (GNU Bison) 3.0.4 > > I just found this just serval minutes ago. Upgrading your bison to the latest version (3.6) is ok. I'd like we have a better way to share this knowledge through. I spend ~30 minutes to troubleshooting this issue. Best Regards Andy Fan

Re: No core file generated after PostgresNode->start

2020-05-12 Thread Andy Fan
On Tue, May 12, 2020 at 3:36 AM Robert Haas wrote: > On Sun, May 10, 2020 at 11:21 PM Andy Fan > wrote: > > Looks this doesn't mean a crash. If the test case(subscription/t/ > 013_partition.pl) > > failed, test framework kill some process, which leads the above &

Re: No core file generated after PostgresNode->start

2020-05-10 Thread Andy Fan
On Mon, May 11, 2020 at 9:48 AM Andy Fan wrote: > Hi: > > > 2020-05-11 09:37:40.778 CST [69541] sub_viaroot WARNING: terminating > connection because of crash of another server process > > Looks this doesn't mean a crash. If the test case(subscription/t/ 013_partit

No core file generated after PostgresNode->start

2020-05-10 Thread Andy Fan
gt;data_dir, '-l', $self->logfile, '-o', "--cluster-name=$name", 'start'); } Best Regards Andy Fan

Re: Implementing Incremental View Maintenance

2020-05-07 Thread Andy Fan
id you test it? AFAIK, we can test it with: 1. For any query like SELECT xxx, we create view like CREATE MATERIAL VIEW mv_name as SELECT xxx; to test if the features in the query are supported. 2. Update the data and then compare the result with SELECT XXX with SELECT * from mv_name to test if the data is correctly sync. Best Regards Andy Fan

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-05-07 Thread Andy Fan
h the positions across > subquery and the outer query. > but I think it should be rel->subroot->processed_tlist rather than reltarget? Actually I still a bit of uneasy about rel->subroot->processed_tlist for some DML case, which the processed_tlist is different and I still not figure out its impact. > Will continue reviewing your new set of patches as time permits. > Thank you! Actually there is no big difference between v6 and v7 regarding the UniqueKey part except 2 bug fix. However v7 has some more documents, comments improvement and code refactor/split, which may be helpful for review. You may try v7 next time if v8 has not come yet:) Best Regards Andy Fan

Can we remove the other_rels_list parameter for make_rels_by_clause_joins

2020-04-29 Thread Andy Fan
and use root->inital_rels directly. I did the same for make_rels_by_clauseless_joins. The attached is my proposal which should be semantic correctly and more explicitly. Best Regards Andy Fan v1-0001-Remove-other_rels_list-from-make_rels_by_clause_j.patch Description: Binary data

<    1   2   3   4   5   6   >