Re: Speed up transaction completion faster after many relations are accessed in a transaction

2022-03-14 Thread David Rowley
On Sat, 1 Jan 2022 at 15:40, Zhihong Yu wrote: > + locallock->nLocks -= locallockowner->nLocks; > + Assert(locallock->nLocks >= 0); > > I think the assertion is not needed since the above code is in if block : > > + if (locallockowner->nLocks < locallock->nLocks) > > the

Re: Use generation context to speed up tuplesorts

2022-03-22 Thread David Rowley
On Wed, 23 Feb 2022 at 15:25, Andres Freund wrote: > On 2022-02-18 12:10:51 +1300, David Rowley wrote: > > The other way I thought to fix it was by changing the logic for when > > generation blocks are freed. In the problem case mentioned above, the > > block being freed

Re: Window Function "Run Conditions"

2022-03-22 Thread David Rowley
On Thu, 26 Aug 2021 at 14:54, Andy Fan wrote: > > On Thu, Aug 19, 2021 at 2:35 PM David Rowley wrote: > > > > On Thu, 19 Aug 2021 at 00:20, Andy Fan wrote: > > > In the current master, the result is: > > > > > > empno | salary | c | dr > > &g

Re: Window Function "Run Conditions"

2022-03-22 Thread David Rowley
On Wed, 16 Mar 2022 at 10:24, Greg Stark wrote: > > This looks like an awesome addition. Thanks > I have one technical questions... > > Is it possible to actually transform the row_number case into a LIMIT > clause or make the planner support for this case equivalent to it (in > which case we ca

Re: Window Function "Run Conditions"

2022-03-22 Thread David Rowley
On Thu, 17 Mar 2022 at 17:04, Corey Huinker wrote: > It seems like this effort would aid in implementing what some other databases > implement via the QUALIFY clause, which is to window functions what HAVING is > to aggregate functions. > example: > https://cloud.google.com/bigquery/docs/refere

Re: Window Function "Run Conditions"

2022-03-22 Thread David Rowley
It seems the code for checking left / right is the same. It would be better > to extract and reuse the code. I've moved some of that code into find_window_run_conditions() which removes about 10 lines of code. Updated patch attached. Thanks for looking. David From db393b01ce6dd48f3a4

Re: Window Function "Run Conditions"

2022-03-22 Thread David Rowley
On Wed, 23 Mar 2022 at 11:24, David Rowley wrote: > I think it's safer to just disable the optimisation when there are > multiple window clauses. Multiple matching clauses are merged > already, so it's perfectly valid to have multiple window functions, > it's just they

Re: [PoC] Reducing planning time when tables have many partitions

2022-03-23 Thread David Rowley
On Fri, 18 Mar 2022 at 23:32, Yuya Watari wrote: > I found a problem that planning takes too much time when the tables > have many child partitions. According to my observation, the planning > time increases in the order of O(n^2). Here, n is the number of child > partitions. I attached the patch

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-12-06 Thread David Rowley
Thanks for this review. I somehow missed addressing what's mentioned here for the v10 patch. Comments below. On Mon, 23 Nov 2020 at 02:21, Andy Fan wrote: > 1. modified src/include/utils/selfuncs.h > @@ -70,9 +70,9 @@ > * callers to provide further details about some assumptions which were m

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-12-06 Thread David Rowley
On Sun, 6 Dec 2020 at 03:52, Andy Fan wrote: > > On Fri, Dec 4, 2020 at 10:41 PM David Rowley wrote: >> >> I also >> noticed that the code I'd written to build the cache lookup expression >> included a step to deform the outer tuple. This was unnecessar

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-12-06 Thread David Rowley
Thanks for having a look at this. On Sat, 5 Dec 2020 at 14:08, Zhihong Yu wrote: > +#define SH_EQUAL(tb, a, b) ResultCacheHash_equal(tb, a, b) == 0 > > I think it would be safer if the comparison is enclosed in parentheses (in > case the macro appears in composite condition). That seems fair.

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-12-06 Thread David Rowley
On Sat, 5 Dec 2020 at 16:51, Zhihong Yu wrote: > > There are two blocks with almost identical code (second occurrence in > cache_store_tuple): > > + if (rcstate->mem_used > rcstate->mem_upperlimit) > + { > It would be nice if the code can be extracted to a method and shared. It's true, they'

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-12-07 Thread David Rowley
On Mon, 7 Dec 2020 at 14:25, Zhihong Yu wrote: > > > + /* Make a guess at a good size when we're not given a valid size. */ > > + if (size == 0) > > + size = 1024; > > > > Should the default size be logged ? > > > I'm not too sure if I know what you mean here. Should it be a power of > >

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-12-08 Thread David Rowley
On Sun, 6 Dec 2020 at 04:10, Andy Fan wrote: > For anyone who is interested with these patchsets, here is my plan about this > now. 1). I will try EquivalenceClass rather than Expr in UniqueKey and add > opfamily > if needed. I agree that we should be storing them in EquivalenceClasses. Apart

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-12-09 Thread David Rowley
On Tue, 8 Dec 2020 at 20:15, David Rowley wrote: > I've attached another patchset that addresses some comments left by > Zhihong Yu over on [1]. The version number got bumped to v12 instead > of v11 as I still have a copy of the other version of the patch which > I made so

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-12-10 Thread David Rowley
Thanks a lot for testing this patch. It's good to see it run through a benchmark that exercises quite a few join problems. On Fri, 11 Dec 2020 at 05:44, Konstantin Knizhnik wrote: > For most queries performance is the same, some queries are executed > faster but > one query is 150 times slower: >

Re: Reduce the number of special cases to build contrib modules on windows

2020-12-22 Thread David Rowley
On Wed, 11 Nov 2020 at 13:44, Michael Paquier wrote: > > On Wed, Nov 11, 2020 at 11:01:57AM +1300, David Rowley wrote: > > I'm still working through some small differences in some of the > > .vcxproj files. I've been comparing these by copying *.vcxproj out to > &

Re: Reduce the number of special cases to build contrib modules on windows

2020-12-29 Thread David Rowley
On Wed, 23 Dec 2020 at 18:46, Michael Paquier wrote: > I have begun a new thread about this point as that's a separate > topic. I did not see other places in need of a similar cleanup: > https://www.postgresql.org/message-id/x+lqpflyk7jgz...@paquier.xyz Thanks. I'll look at that shortly. > > I

Re: Reduce the number of special cases to build contrib modules on windows

2020-12-29 Thread David Rowley
On Wed, 30 Dec 2020 at 10:03, David Rowley wrote: > > On Wed, 23 Dec 2020 at 18:46, Michael Paquier wrote: > > I have tested your patch, and this is causing compilation failures for > > hstore_plpython, jsonb_plpython and ltree_plpython. So > > AddTransformModule is mis

Re: Cleanup some -I$(libpq_srcdir) in makefiles

2020-12-29 Thread David Rowley
On Wed, 23 Dec 2020 at 18:07, Michael Paquier wrote: > While looking at a patch from David, I have noticed $subject: > https://www.postgresql.org/message-id/CAApHDvpgB+vxk=w6opkidwzzeo6knifqidnomzr8p4rotyk...@mail.gmail.com > > adminpack and old_snapshot have no need for those references as they >

Re: Keep notnullattrs in RelOptInfo (Was part of UniqueKey patch series)

2021-02-15 Thread David Rowley
On Fri, 12 Feb 2021 at 15:18, Andy Fan wrote: > > On Fri, Feb 12, 2021 at 9:02 AM David Rowley wrote: >> The reason I don't really like this is that it really depends where >> you want to use RelOptInfo.notnullattrs. If someone wants to use it >> to optimise someth

Re: Tid scan improvements

2021-02-16 Thread David Rowley
On Thu, 4 Feb 2021 at 23:51, David Rowley wrote: > Updated patch attached. I made another pass over this patch and did a bit of renaming work around the heap_* functions and the tableam functions. I think the new names are a bit more aligned to the existing names. I don't really see

Re: Tid scan improvements

2021-02-17 Thread David Rowley
Thanks for having a look at this. On Wed, 17 Feb 2021 at 11:05, Andres Freund wrote: > > On 2021-02-04 23:51:39 +1300, David Rowley wrote: > > and > > bool (*scan_tid_range_nextslot) (TableScanDesc sscan, > > ScanDirection direction, > > TupleTableSlot *slot);

Re: Tid scan improvements

2021-02-18 Thread David Rowley
On Thu, 18 Feb 2021 at 09:45, David Rowley wrote: > > On Wed, 17 Feb 2021 at 11:05, Andres Freund wrote: > > Architecturally it feels like this is something that really belongs more > > into plan time? > > Possibly. It would mean TidOpExpr would have to become a Node t

Re: Tid scan improvements

2021-02-27 Thread David Rowley
On Fri, 19 Feb 2021 at 20:37, David Rowley wrote: > > On Thu, 18 Feb 2021 at 09:45, David Rowley wrote: > > > > On Wed, 17 Feb 2021 at 11:05, Andres Freund wrote: > > > How does this interact with rescans? > > > > We must call table_rescan() before calli

Re: Side effect of remove_useless_groupby_columns

2021-02-28 Thread David Rowley
On Sun, 28 Feb 2021 at 20:52, Richard Guo wrote: > When looking at [1], I realized we may have a side effect when removing > redundant columns in the GROUP BY clause. Suppose we have a query with > ORDER BY 'b', and meanwhile column 'b' is also a group key. If we decide > that 'b' is redundant due

Re: We should stop telling users to "vacuum that database in single-user mode"

2021-03-01 Thread David Rowley
On Tue, 2 Mar 2021 at 04:32, Hannu Krosing wrote: > > It looks like we are unnecessarily instructing our usiers to vacuum their > databases in single-user mode when just vacuuming would be enough. > > We should fix the error message to be less misleading. It would be good to change the message as

Re: We should stop telling users to "vacuum that database in single-user mode"

2021-03-02 Thread David Rowley
On Wed, 3 Mar 2021 at 01:12, Magnus Hagander wrote: > > On Tue, Mar 2, 2021 at 7:52 AM David Rowley wrote: > > I have seen it happen that an instance has a vacuum_cost_limit set and > > someone did start the database in single-user mode, per the advice of > > the error m

Re: Increase value of OUTER_VAR

2021-03-03 Thread David Rowley
On Wed, 3 Mar 2021 at 21:29, Andrey Lepikhov wrote: > > Playing with a large value of partitions I caught the limit with 65000 > table entries in a query plan: > > if (IS_SPECIAL_VARNO(list_length(glob->finalrtable))) > ereport(ERROR, > (errcode(ERRCODE_PROGRAM_LIMIT_EXCEED

Re: Reduce the number of special cases to build contrib modules on windows

2021-03-03 Thread David Rowley
Thank you for having another look at this. On Tue, 12 Jan 2021 at 20:18, Michael Paquier wrote: > > On Wed, Dec 30, 2020 at 10:07:29AM +1300, David Rowley wrote: > > -#ifdef LOWER_NODE > > +/* > > + * Below we ignore the fact that LOWER_NODE is defined when compiling

Re: We should stop telling users to "vacuum that database in single-user mode"

2021-03-03 Thread David Rowley
On Wed, 3 Mar 2021 at 21:44, Magnus Hagander wrote: > > On Tue, Mar 2, 2021 at 10:07 PM David Rowley wrote: > > > > On Wed, 3 Mar 2021 at 01:12, Magnus Hagander wrote: > > > > > > On Tue, Mar 2, 2021 at 7:52 AM David Rowley wrote: > > > &

Re: Why OR-clauses not getting copied into baserestrictinfo of another table whose columns are in the same EquivalenceClass?

2021-03-03 Thread David Rowley
On Wed, 3 Mar 2021 at 23:26, Mohamed Insaf wrote: > I have a question regarding distributing the filter clause(baserestrictinfo) > of one table into another table(Keys belong to the same EquivalenceClass). > > In the following query, why PG is not copying the filter (t1.pk=1 OR t1.pk=2) > into t

Re: [patch] bit XOR aggregate functions

2021-03-07 Thread David Rowley
On Sun, 7 Mar 2021 at 23:24, Pavel Stehule wrote: The mandatory ORDER BY clause should be necessary for operations when the result depends on the order. You need an order for calculation of median. And you don't need to know an order for average. More if the result is one number and is not possib

Re: Extend more usecase for planning time partition pruning and init partition pruning.

2021-03-07 Thread David Rowley
On Thu, 4 Mar 2021 at 22:07, Amit Langote wrote: > * Or maybe have you considered generalizing what > build_implied_pruning_quals() does so that other places like > indxpath.c can use the facility? I agree with doing it another way. There's plenty of other queries which we could produce a better

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2021-03-11 Thread David Rowley
Thanks for these suggestions. On Mon, 22 Feb 2021 at 14:21, Justin Pryzby wrote: > > On Tue, Feb 16, 2021 at 11:15:51PM +1300, David Rowley wrote: > > To summarise here, the planner performance gets a fair bit worse with > > the patched code. With master, summing the average

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2021-03-11 Thread David Rowley
On Tue, 23 Feb 2021 at 14:22, Andy Fan wrote: > > On Mon, Feb 22, 2021 at 9:21 AM Justin Pryzby wrote: >> - Maybe this should be integrated into nestloop rather than being a separate >>plan node. That means that it could be dynamically enabled during >>execution, maybe after a few loops

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2021-03-11 Thread David Rowley
On Fri, 12 Mar 2021 at 14:59, Tom Lane wrote: > > David Rowley writes: > > The 0001 patch adds a has_volatile bool field to RestrictInfo and sets > > it when building the RestrictInfo. > > I'm -1 on doing it exactly that way, because you're expending > the co

Re: Parallel Append can break run-time partition pruning

2020-11-01 Thread David Rowley
On Tue, 27 Oct 2020 at 19:40, Amit Langote wrote: > Some comments: Thanks for having a look at this. I've made some adjustments to those comments and pushed. David

Reduce the number of special cases to build contrib modules on windows

2020-11-01 Thread David Rowley
Hi, At the moment we do very basic parsing of makefiles to build the visual studio project file in order to build our contrib modules on MSVC. This parsing is quite basic and still requires a number of special cases to get enough information into the project file in order for the build to succeed

Re: Collation versioning

2020-11-02 Thread David Rowley
On Tue, 3 Nov 2020 at 09:43, Thomas Munro wrote: > Fortunately David Rowley is able to repro this on his Windows box (it > fails even with strings that are succeeding on the other BF machines), > so we have something to work with. The name mangling that is done in > get_iso_locale

Re: Collation versioning

2020-11-02 Thread David Rowley
On Tue, 3 Nov 2020 at 12:29, David Rowley wrote: > Running low on ideas for now, so thought I'd post this in case it > someone thinks of something else. FWIW, the attached does fix the issue for me. It basically just calls the function that converts the windows-type "English_

Re: Split copy.c

2020-11-02 Thread David Rowley
On Tue, 3 Nov 2020 at 07:35, Andres Freund wrote: > > On 2020-11-02 19:43:38 +0200, Heikki Linnakangas wrote: > > On 02/11/2020 19:23, Andres Freund wrote: > > > On 2020-11-02 11:03:29 +0200, Heikki Linnakangas wrote: > > > > There isn't much common code between COPY FROM and COPY TO, so I propose

Re: Keep elog(ERROR) and ereport(ERROR) calls in the cold path

2020-11-03 Thread David Rowley
On Tue, 3 Nov 2020 at 20:08, Peter Eisentraut wrote: > > On 2020-09-29 11:26, David Rowley wrote: > > I've marked this patch back as waiting for review. It would be good if > > someone could run some tests on some intel hardware and see if they > > can see any speedup

Re: Collation versioning

2020-11-03 Thread David Rowley
On Wed, 4 Nov 2020 at 14:21, Thomas Munro wrote: > > On Wed, Nov 4, 2020 at 10:56 AM Thomas Munro wrote: > > On Wed, Nov 4, 2020 at 10:52 AM Tom Lane wrote: > > > Thomas Munro writes: > > > > We want the same algorithm that Windows uses internally to resolve the > > > > old style name to a coll

Re: Use of "long" in incremental sort code

2020-11-04 Thread David Rowley
On Wed, 4 Nov 2020 at 10:42, Tomas Vondra wrote: > IMHO this should simply switch the current int64 variable to long, as it > was before. Not sure about about the hashagg uint64 variable. IMO, we should just get rid of the use of "long" here. As far as I'm concerned, using long in the core code

Re: Reduce the number of special cases to build contrib modules on windows

2020-11-05 Thread David Rowley
Thank you for looking at this. On Tue, 3 Nov 2020 at 09:49, Andres Freund wrote: > > diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm > > index 90594bd41b..491a465e2f 100644 > > --- a/src/tools/msvc/Mkvcbuild.pm > > +++ b/src/tools/msvc/Mkvcbuild.pm > > @@ -32,16 +32,13 @@ m

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-05 Thread David Rowley
On Mon, 2 Nov 2020 at 20:43, David Rowley wrote: > > On Tue, 20 Oct 2020 at 22:30, David Rowley wrote: > I did some further tests this time with some tuple deforming. Again, > it does seem that v9 is slower than v8. > > Graphs attached > > Looking at profiles, I don&

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-08 Thread David Rowley
On Mon, 9 Nov 2020 at 03:52, Andy Fan wrote: > then I did a perf on the 2 version, Is it possible that you called > tts_minimal_clear twice in > the v9 version? Both ExecClearTuple and ExecStoreMinimalTuple called > tts_minimal_clear > on the same slot. > > With the following changes: Thank

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-09 Thread David Rowley
On Mon, 9 Nov 2020 at 16:29, Andy Fan wrote: > I think either version is OK for me and I like this patch overall. That's good to know. Thanks. > However I believe v9 > should be no worse than v8 all the time, Is there any theory to explain > your result? Nothing jumps out at me from looking a

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-09 Thread David Rowley
On Tue, 10 Nov 2020 at 12:49, Tom Lane wrote: > > Alvaro Herrera writes: > > Are you taking into account the possibility that generated machine code > > is a small percent slower out of mere bad luck? I remember someone > > suggesting that they can make code 2% faster or so by inserting random >

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-09 Thread David Rowley
On Tue, 10 Nov 2020 at 15:38, Andy Fan wrote: > While I have interest about what caused the tiny difference, I admit that > what direction > this patch should go is more important. Not sure if anyone is convinced that > v8 and v9 have a similar performance. The current data show it is similar.

Re: Reduce the number of special cases to build contrib modules on windows

2020-11-10 Thread David Rowley
On Tue, 10 Nov 2020 at 03:07, Alvaro Herrera wrote: > > On 2020-Nov-06, David Rowley wrote: > > > +# Handle makefile rules for when file to be added to the project > > +# does not exist. Returns 1 when the original file add should be > > +# skipped. > &g

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-11 Thread David Rowley
On Tue, 10 Nov 2020 at 12:55, David Rowley wrote: > > On Tue, 10 Nov 2020 at 12:49, Tom Lane wrote: > > > > Alvaro Herrera writes: > > > Are you taking into account the possibility that generated machine code > > > is a small percent slower out o

Re: Keep elog(ERROR) and ereport(ERROR) calls in the cold path

2020-11-23 Thread David Rowley
On Sat, 21 Nov 2020 at 03:26, Peter Eisentraut wrote: > > I did tests of elog_ereport_attribute_cold_v4.patch on an oldish Mac > Intel laptop with pgbench scale 1 (default), and then: > > pgbench -S -T 60 > > master: tps = 8251.883229 (excluding connections establishing) > patched: tps = 9556.836

Re: Keep elog(ERROR) and ereport(ERROR) calls in the cold path

2020-11-23 Thread David Rowley
On Tue, 24 Nov 2020 at 09:36, David Rowley wrote: > Well, that makes it look pretty good. If we can get 10-15% on some > machines without making things slower on any other machines, then that > seems like a good win to me. Pushed. Thank you both for reviewing this. David

Re: Keep elog(ERROR) and ereport(ERROR) calls in the cold path

2020-11-23 Thread David Rowley
On Tue, 24 Nov 2020 at 12:50, Greg Nancarrow wrote: > Hmmm, unfortunately this seems to break my build ... > I think your commit needs to be fixed based on the following documentation: > > https://gcc.gnu.org/onlinedocs/cpp/_005f_005fhas_005fattribute.html#g_t_005f_005fhas_005fattribute Agreed.

Re: Keep elog(ERROR) and ereport(ERROR) calls in the cold path

2020-11-24 Thread David Rowley
On Wed, 25 Nov 2020 at 04:48, Peter Eisentraut wrote: > > On 2020-11-24 01:52, Dagfinn Ilmari Mannsåker wrote: > > The Clang documentation¹ suggest an even neater solution, which would > > eliminate the repetitive empty pg_attribute_foo #defines in the trailing > > #else/#endif block in commit 1fa

Re: Keep elog(ERROR) and ereport(ERROR) calls in the cold path

2020-11-24 Thread David Rowley
On Wed, 25 Nov 2020 at 04:55, Tom Lane wrote: > > walleye's been failing since this patchset went in: > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=walleye&dt=2020-11-24%2000%3A25%3A31 > > ccache gcc -Wall -Wmissing-prototypes -Wpointer-arith > -Wdeclaration-after-statement -Werror=

Re: Keep elog(ERROR) and ereport(ERROR) calls in the cold path

2020-11-24 Thread David Rowley
On Wed, 25 Nov 2020 at 14:28, Tom Lane wrote: > So maybe, rather than hacking up the attribute stuff for > a bug that might bite us again anyway in future, we ought > to press walleye's owner to install a more recent compiler. I think that seems like a better idea. I had thoughts about installin

Re: Keep elog(ERROR) and ereport(ERROR) calls in the cold path

2020-11-24 Thread David Rowley
On Wed, 25 Nov 2020 at 14:35, David Rowley wrote: > > On Wed, 25 Nov 2020 at 14:28, Tom Lane wrote: > > So maybe, rather than hacking up the attribute stuff for > > a bug that might bite us again anyway in future, we ought > > to press walleye's owner to install

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-26 Thread David Rowley
Thanks for having another look at this. > On Sun, Nov 22, 2020 at 9:21 PM Andy Fan wrote: > add 2 more comments. > > 1. I'd suggest adding Assert(false); in RC_END_OF_SCAN case to make the > error clearer. > > case RC_END_OF_SCAN: > /* > * We've already returned NULL for this scan, but just in

Heap's backwards scan scans the incorrect pages with heap_setscanlimits()

2021-01-20 Thread David Rowley
Hackers, It looks like both heapgettup() and heapgettup_pagemode() are coded incorrectly when setting the page to start the scan on for a backwards scan when heap_setscanlimits() has been used. It looks like the code was not updated during 7516f5259. The current code is: /* start from last page

Re: Tid scan improvements

2021-01-20 Thread David Rowley
ed which can be incorrect. This is a precursor fix to a future patch which allows TID Range scans to scan a subset of a heap table. Proper adjustment of the heap scan code seems to have been missed when heap_setscanlimits() was added in 7516f5259. Author: David Rowley Discussion: https://p

Re: Heap's backwards scan scans the incorrect pages with heap_setscanlimits()

2021-01-24 Thread David Rowley
On Thu, 21 Jan 2021 at 13:16, David Rowley wrote: > Proposed patch attached. I ended up pushing a slightly revised version of this which kept the code the same as before when rs_numblocks had not been changed. I backpatched to 9.5 as it seemed low risk and worthy of stopping some head-scratch

Re: Tid scan improvements

2021-01-25 Thread David Rowley
On Thu, 21 Jan 2021 at 18:16, David Rowley wrote: > I've implemented this in the attached. The bug fix in 0001 is now committed, so I'm just attaching the 0002 patch again after having rebased... This is mostly just to keep the CFbot happy. David From e459b522d0599602188fcb1cc9e

Re: Tid scan improvements

2021-01-25 Thread David Rowley
Thanks for having a look at this. On Tue, 26 Jan 2021 at 15:48, Zhihong Yu wrote: > bq. within this range. Table AMs where scanning ranges of TIDs does not make > sense or is difficult to implement efficiently may choose to not implement > > Is there criterion on how to judge efficiency ? For e

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2021-01-28 Thread David Rowley
Thanks for having a look at this. I've taken most of your suggestions. The things quoted below are just the ones I didn't agree with or didn't understand. On Thu, 28 Jan 2021 at 18:43, Justin Pryzby wrote: > > On Tue, Dec 08, 2020 at 08:15:52PM +1300, David Rowley wrote

Re: [sqlsmith] Failed assertion during partition pruning

2021-01-31 Thread David Rowley
On Sun, 31 Jan 2021 at 11:42, Tom Lane wrote: > This fixes the cases reported by Andreas and Jaime, leaving me > more confident that there's nothing wrong with David's Assert. I agree that there is nothing wrong with the Assert. The commit message of a929e17e5 mentions: > Here we tighten that u

Re: [sqlsmith] Failed assertion during partition pruning

2021-01-31 Thread David Rowley
On Sun, 31 Jan 2021 at 11:42, Tom Lane wrote: > > For simplicity of review I divided the patch into two parts. > 0001 revises make_partition_pruneinfo() and children to identify > the relevant parent partitions for themselves, which is not too > hard to do by chasing up the child-to-parent AppendR

Re: [sqlsmith] Failed assertion during partition pruning

2021-01-31 Thread David Rowley
On Mon, 1 Feb 2021 at 18:57, Tom Lane wrote: > > I wrote: > > David Rowley writes: > >> Parent RT indexes are guaranteed to be lower than their children RT > >> indexes, > > > I was intentionally avoiding that assumption ;-). Maybe it buys enough >

Re: [sqlsmith] Failed assertion during partition pruning

2021-02-01 Thread David Rowley
On Tue, 2 Feb 2021 at 08:58, Tom Lane wrote: > I renamed things that way, did some more work on the comments, > and pushed it. Thanks for reviewing! Thanks for working on this and coming up with the idea to nuke partitioned_rels. David

Re: Should we make Bitmapsets a kind of Node?

2021-02-01 Thread David Rowley
On Tue, 2 Feb 2021 at 09:23, Tom Lane wrote: > > Now that commit f003a7522 did away with the partitioned_rels fields, > my original motivation for doing $SUBJECT is gone. It might still be > worth doing, but I'm not planning to tackle it right now. I'm not sure if the misuse of Lists to store no

Re: Tid scan improvements

2021-02-03 Thread David Rowley
Thanks for looking at this. On Thu, 4 Feb 2021 at 10:19, Andres Freund wrote: > Perhaps something like > > typedef struct TableScanTidRange TableScanTidRange; > > TableScanTidRange* table_scan_tid_range_start(TableScanDesc sscan, > ItemPointer mintid, ItemPointer maxtid); > bool table_scan_tid_r

Re: Tid scan improvements

2021-02-04 Thread David Rowley
On Thu, 4 Feb 2021 at 10:31, David Rowley wrote: > > Thanks for looking at this. > > On Thu, 4 Feb 2021 at 10:19, Andres Freund wrote: > > Perhaps something like > > > > typedef struct TableScanTidRange TableScanTidRange; > > > > TableScanTidRange* tabl

Re: Keep notnullattrs in RelOptInfo (Was part of UniqueKey patch series)

2021-02-11 Thread David Rowley
On Wed, 10 Feb 2021 at 16:18, Andy Fan wrote: > v1-0001-Introduce-notnullattrs-field-in-RelOptInfo-to-ind.patch > > Introduce notnullattrs field in RelOptInfo to indicate which attr are not null > in current query. The not null is judged by checking pg_attribute and query's > restrictinfo. The inf

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2021-12-31 Thread David Rowley
On Fri, 3 Dec 2021 at 20:36, Michael Paquier wrote: > Two months later, this has been switched to RwF. I was discussing this patch with Andres. He's not very keen on my densehash hash table idea and suggested that instead of relying on trying to make the hash table iteration faster, why don't we

Re: Fix BUG #17335: Duplicate result rows in Gather node

2021-12-31 Thread David Rowley
On Fri, 31 Dec 2021 at 00:14, Yura Sokolov wrote: > Problem: > - Append path is created with explicitely parallel_aware = true > - It has two child, one is trivial, other is parallel_aware = false . > Trivial child is dropped. > - Gather/GatherMerge path takes Append path as a child and thinks >

Re: A reloption for partitioned tables - parallel_workers

2021-03-23 Thread David Rowley
On Fri, 19 Mar 2021 at 02:07, Amit Langote wrote: > Attached a new version rebased over c8f78b616, with the grouping > relation partitioning enhancements as a separate patch 0001. Sorry > about the delay. I had a quick look at this and wondered if the partitioned table's parallel workers shouldn

Re: Getting better results from valgrind leak tracking

2021-03-28 Thread David Rowley
On Wed, 17 Mar 2021 at 15:31, Andres Freund wrote: > I'm a bit confused about the precise design of rs_private / > ParallelBlockTableScanWorkerData, specifically why it's been added to > TableScanDesc, instead of just adding it to HeapScanDesc? And why is it > allocated unconditionally, instead of

Re: UniqueKey on Partitioned table.

2021-03-29 Thread David Rowley
On Tue, 30 Mar 2021 at 02:27, Ashutosh Bapat wrote: > > On Sat, Mar 27, 2021 at 11:44 AM Andy Fan wrote: > > > > On Sat, Mar 27, 2021 at 3:07 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > >> Thanks for the patch. After a short look through it I'm a bit confused > >> and wanted to clarify, now

Re: Getting better results from valgrind leak tracking

2021-03-29 Thread David Rowley
On Tue, 30 Mar 2021 at 06:38, Andres Freund wrote: > On 2021-03-29 11:48:47 +1300, David Rowley wrote: > > I've done that in the attached. I added the > > ParallelBlockTableScanWorkerData as a pointer field in > > HeapScanDescData and change it so we only allocate

What to call an executor node which lazily caches tuples in a hash table?

2021-03-30 Thread David Rowley
Hackers, Over on [1] I've been working on adding a new type of executor node which caches tuples in a hash table belonging to a given cache key. The current sole use of this node type is to go between a parameterized nested loop and the inner node in order to cache previously seen sets of paramet

Re: What to call an executor node which lazily caches tuples in a hash table?

2021-03-30 Thread David Rowley
On Wed, 31 Mar 2021 at 14:43, Andy Fan wrote: > When naming it, we may also think about some non native English speakers, so > some too advanced words may make them uncomfortable. Actually when I read > "Reactive", I googled to find what its meaning is. I knew reactive > programming, but I > d

Re: What to call an executor node which lazily caches tuples in a hash table?

2021-03-30 Thread David Rowley
On Wed, 31 Mar 2021 at 14:43, Andy Fan wrote: > At last, I still want to vote for "Tuple(s) Cache", which sounds simple and > enough. > I was thinking if we need to put "Lazy" in the node name since we do build > cache > lazily, then I found we didn't call "Materialize" as "Lazy Materialize",

Re: Shared buffers advice for windows in the wiki

2021-03-31 Thread David Rowley
On Wed, 31 Mar 2021 at 22:39, talk to ben wrote: > Is it possible to remove this advice or add that since pg10 it no longer > holds true [3] ? I've just removed all mention of it from the wiki. David

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2021-03-31 Thread David Rowley
On Wed, 31 Mar 2021 at 05:34, Zhihong Yu wrote: > > Hi, > In paraminfo_get_equal_hashops(), > > + /* Reject if there are any volatile functions */ > + if (contain_volatile_functions(expr)) > + { > > You can move the above code to just ahead of: > > + if (IsA(expr, Var)) > +

Re: making update/delete of inheritance trees scale better

2021-03-31 Thread David Rowley
On Thu, 1 Apr 2021 at 15:09, Amit Langote wrote: > Note that the patch over there doesn't do anything about > AcquireExecutorLocks() bottleneck, as there are some yet-unsolved race > conditions that were previously discussed here: > > https://www.postgresql.org/message-id/flat/CAKJS1f_kfRQ3ZpjQyHC

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2021-04-01 Thread David Rowley
On Thu, 1 Apr 2021 at 23:41, houzj.f...@fujitsu.com wrote: > > > I've attached the updated patch. I'll let the CFbot grab this to ensure > > it's > > happy with it before I go looking to push it again. > > Hi, > > I took a look into the patch and noticed some minor things. > > 1. > +

Re: Binary search in ScalarArrayOpExpr for OR'd constant arrays

2021-04-05 Thread David Rowley
On Sat, 20 Mar 2021 at 09:41, James Coleman wrote: > I've attached a cleaned up patch. Last CF it was listed in is > https://commitfest.postgresql.org/29/2542/ -- what's the appropriate > step to take here given it's an already existing patch, but not yet > moved into recent CFs? I had a look at

Re: UniqueKey on Partitioned table.

2021-04-06 Thread David Rowley
On Tue, 6 Apr 2021 at 22:31, Andy Fan wrote: > On Wed, Mar 31, 2021 at 9:12 PM Ashutosh Bapat > wrote: >> I think the reason we add ECs for sort expressions is to use >> transitive relationship. The EC may start with a single member but >> later in the planning that member might find partners wh

Re: Wired if-statement in gen_partprune_steps_internal

2021-04-07 Thread David Rowley
On Thu, 4 Mar 2021 at 19:03, Amit Langote wrote: > > On Tue, Oct 20, 2020 at 9:46 PM Amit Langote wrote: > > I had updated the patch last week to address Horiguchi-san's comments > > but didn't manage to post a polished-enough version. I will try again > > this week. > > Sorry, this seems to hav

Re: Wired if-statement in gen_partprune_steps_internal

2021-04-07 Thread David Rowley
On Wed, 7 Apr 2021 at 21:04, Amit Langote wrote: > > On Wed, Apr 7, 2021 at 4:43 PM David Rowley wrote: > > However, it does change the meaning of what PARTCLAUSE_MATCH_STEPS > > does. If we ever needed to expand what PARTCLAUSE_MATCH_STEPS does, > > then we'll h

Re: Wired if-statement in gen_partprune_steps_internal

2021-04-07 Thread David Rowley
On Wed, 7 Apr 2021 at 21:53, David Rowley wrote: > If canonicalize_qual() had been unable to rewrite that WHERE clause > then I could see that we might want to combine steps from other > recursive quals. I'm thinking right now that I'm glad > canonicalize_qual() does that

Re: ModifyTable overheads in generic plans

2021-04-07 Thread David Rowley
On Thu, 8 Apr 2021 at 15:32, Amit Langote wrote: > There's 10-20% improvement in this case too for various partition > counts, which really has more to do with 86dc90056 than the work done > here. I'm not sure of the exact query you're running, but I imagine the reason that it wasn't that slow wi

Re: Binary search in ScalarArrayOpExpr for OR'd constant arrays

2021-04-07 Thread David Rowley
On Thu, 8 Apr 2021 at 05:54, Tomas Vondra wrote: > I only ran that with a single client, the machine only has 4 cores and > this should not be related to concurrency, so 1 client seems fine. The > average of 10 runs, 15 seconds each look like this: Thanks for running these tests. The reason I a

Re: Wired if-statement in gen_partprune_steps_internal

2021-04-08 Thread David Rowley
On Thu, 8 Apr 2021 at 00:49, Amit Langote wrote: > > Thanks David. Actually, I was busy updating the patch to revert to > gen_partprune_steps_internal() returning a list and was almost done > with it when I saw your message. > > I read through v3 and can say that it certainly looks better than v2

Re: Wired if-statement in gen_partprune_steps_internal

2021-04-08 Thread David Rowley
On Thu, 8 Apr 2021 at 21:04, Amit Langote wrote: > + * These partition pruning steps come in 2 forms; operation steps and combine > + * steps. > > Maybe you meant "operator" steps? IIRC, the reason why we named it > PartitionPruneStepOp is that an op step is built to prune based on the > semantic

Re: Binary search in ScalarArrayOpExpr for OR'd constant arrays

2021-04-08 Thread David Rowley
On Thu, 8 Apr 2021 at 18:50, David Rowley wrote: > I've not done any further performance tests yet but will start those now. I ran a set of tests on this: select * from a where a in( < 1 to 10 > ); and select * from a where a in( < 1 to 100 > ); the table "a"

Re: Binary search in ScalarArrayOpExpr for OR'd constant arrays

2021-04-08 Thread David Rowley
On Thu, 8 Apr 2021 at 22:54, David Rowley wrote: > > I think the changes in the patch are fairly isolated and the test > coverage is now pretty good. I'm planning on looking at the patch > again now and will consider pushing it for PG14. I push this with some minor cleanup fr

Lots of incorrect comments in nodeFuncs.c

2021-04-08 Thread David Rowley
I noticed that nodeFuncs.c appears to have some pretty sloppy work done in many of the comments. Many look like they've just not been updated from a copy/paste/edit from another node function. The attached aims to clean these up. I plan to push this a later today unless anyone has anything they'

  1   2   3   4   5   6   7   8   9   10   >