Re: meson uses stale pg_config_paths.h left over from make

2023-08-23 Thread David Rowley
On Thu, 24 Aug 2023 at 00:52, David Rowley wrote: > Are there any objections to the attached being applied? Pushed. David

Re: PostgreSQL 16 release announcement draft

2023-08-23 Thread David Rowley
On Thu, 24 Aug 2023 at 05:55, Jonathan S. Katz wrote: > We could add something about 1349d2790 -- do you have suggested wording? I think it's worth a mention. See the text added in square brackets below: PostgreSQL 16 improves the performance of existing PostgreSQL functionality through new

meson uses stale pg_config_paths.h left over from make

2023-08-23 Thread David Rowley
I've been having some problems running the regression tests using meson on Windows. This seems to be down to initdb being compiled with a version of pg_config_paths.h left over from the msvc build which had been used on that source tree previously. Generally when there are files left over the

Re: PostgreSQL 16 release announcement draft

2023-08-23 Thread David Rowley
On Wed, 23 Aug 2023 at 22:21, jian he wrote: > > >>> > PostgreSQL 16 improves the performance of existing PostgreSQL functionality > through new query planner optimizations. In this latest release, the query > planner can parallelize `FULL` and `RIGHT` joins, utilize incremental sorts > for >

Re: Support run-time partition pruning for hash join

2023-08-22 Thread David Rowley
On Tue, 22 Aug 2023 at 00:34, Andy Fan wrote: > > On Mon, Aug 21, 2023 at 11:48 AM Richard Guo wrote: >> 1. All the join partition prunning decisions are made in createplan.c >>where the best path tree has been decided. This is not great. Maybe >>it's better to make it happen when we

Re: PG 16 draft release notes ready

2023-08-21 Thread David Rowley
s://www.postgresql.org/docs/16/release-16.html#RELEASE-16-PERFORMANCE > > > > Same as here: > > https://momjian.us/pgsql_docs/release-16.html#RELEASE-16-PERFORMANCE > > > > Allow window functions to use ROWS mode internally when RANGE mode is > > specified but un

Re: Report planning memory in EXPLAIN ANALYZE

2023-08-13 Thread David Rowley
On Thu, 10 Aug 2023 at 20:33, Ashutosh Bapat wrote: > My point is what's relevant here is how much net memory planner asked > for. But that's not what your patch is reporting. All you're reporting is the difference in memory that's *currently* palloc'd from before and after the planner ran. If

Re: Report planning memory in EXPLAIN ANALYZE

2023-08-09 Thread David Rowley
On Thu, 10 Aug 2023 at 03:12, Ashutosh Bapat wrote: > Thinking more about it, I think memory used is the only right metrics. > It's an optimization in MemoryContext implementation that malloc'ed > memory is not freed when it is returned using free(). I guess it depends on the problem you're

Re: Separate memory contexts for relcache and catcache

2023-08-09 Thread David Rowley
On Thu, 10 Aug 2023 at 01:23, Alvaro Herrera wrote: > > On 2023-Aug-09, Melih Mutlu wrote: > > > --Patch > > name | used_bytes | free_bytes | total_bytes > > ---+++- > > RelCacheMemoryContext |4706464 |3682144 |

Re: [PoC] Reducing planning time when tables have many partitions

2023-08-09 Thread David Rowley
On Wed, 9 Aug 2023 at 20:15, Yuya Watari wrote: > I agree with your opinion that my patch lacks some explanations, so I > will consider adding more comments. However, I received the following > message from David in March. > > On Thu, Mar 9, 2023 at 6:23 AM David Rowley wrote: &

Re: [PoC] Reducing planning time when tables have many partitions

2023-08-09 Thread David Rowley
On Wed, 9 Aug 2023 at 22:28, David Rowley wrote: > i.e: > > + Bitmapset *matching_ems = NULL; > + memcpy(_iter, iter, sizeof(EquivalenceMemberIterator)); > + memcpy(_iter, iter, sizeof(EquivalenceMemberIterator)); > + > + idx_iter.use_index = true; > + noidx_iter.use_inde

Re: [PoC] Reducing planning time when tables have many partitions

2023-08-09 Thread David Rowley
On Wed, 5 Jul 2023 at 21:58, Yuya Watari wrote: > > Hello, > > On Fri, Mar 10, 2023 at 5:38 PM Yuya Watari wrote: > > Thank you for pointing it out. I have attached the rebased version to > > this email. > > Recent commits, such as a8c09daa8b [1], have caused conflicts and > compilation errors

Re: Avoid stack frame setup in performance critical routines using tail calls

2023-08-09 Thread David Rowley
On Fri, 21 Jul 2023 at 14:03, David Rowley wrote: > I'll reply back with a more detailed review next week. Here's a review of v2-0001: 1. /* * XXX: Should this also be moved into alloc()? We could possibly avoid * zeroing in some cases (e.g. if we used mmap() ourselves. */ MemSetAligned(ret

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-08-08 Thread David Rowley
On Fri, 28 Jul 2023 at 02:06, Ashutosh Bapat wrote: > 0001 - to measure memory consumption during planning. This is the same > one as attached to [1]. I see you're recording the difference in the CurrentMemoryContext of palloc'd memory before and after planning. That won't really alert us to

Re: [PATCH] [zh_CN.po] fix a typo in simplified Chinese translation file

2023-08-07 Thread David Rowley
On Wed, 2 Aug 2023 at 15:45, jian he wrote: > I think it's pretty obviously. anyway. I created an commitfest entry. > https://commitfest.postgresql.org/44/4470/ I saw that there were two CF entries for this patch. I marked one as committed and the other as withdrawn. For the future, I believe

Re: 2023-08-10 release announcement draft

2023-08-07 Thread David Rowley
On Tue, 8 Aug 2023 at 13:49, Jonathan S. Katz wrote: > > On 8/7/23 9:45 PM, David Rowley wrote: > > >> * Fix a performance regression when running concurrent > >> [`COPY`](https://www.postgresql.org/docs/16/sql-copy.html) statements on a > >> single

Re: 2023-08-10 release announcement draft

2023-08-07 Thread David Rowley
On Tue, 8 Aug 2023 at 13:15, Jonathan S. Katz wrote: > Attached is the release announcement draft for the 2023-08-10 update > release, which also includes the release of PostgreSQL 16 Beta 3. Thanks for drafting this. > * Fix a performance regression when running concurrent >

Re: Check volatile functions in ppi_clauses for memoize node

2023-08-07 Thread David Rowley
On Fri, 4 Aug 2023 at 22:26, Richard Guo wrote: > explain (costs off) > select * from t t1 left join lateral > (select t1.a as t1a, t2.a as t2a from t t2) s > on t1.a = s.t2a + random(); > QUERY PLAN > --- > Nested Loop Left Join

Re: Fix a comment in paraminfo_get_equal_hashops

2023-08-07 Thread David Rowley
On Fri, 4 Aug 2023 at 18:48, Richard Guo wrote: > As stated in [1], there is a typo in the comment in > paraminfo_get_equal_hashops. > [1] > https://www.postgresql.org/message-id/cambws49dehrpe8pom_k39r2uosaozcg+y0b5a8tf7vw3uvr...@mail.gmail.com Thanks. Pushed. David

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-08-07 Thread David Rowley
On Wed, 2 Aug 2023 at 13:35, David Rowley wrote: > So, it looks like this item can be closed off. I'll hold off from > doing that for a few days just in case anyone else wants to give > feedback or test themselves. Alright, closed. David

Re: Improve join_search_one_level readibilty (one line change)

2023-08-06 Thread David Rowley
On Fri, 4 Aug 2023 at 16:05, Richard Guo wrote: > > > On Fri, Aug 4, 2023 at 10:36 AM David Rowley wrote: >> >> The whole lnext() stuff all feels a bit old now that Lists are arrays. >> I think we'd be better adjusting the code to pass the List index where &

Re: Improve join_search_one_level readibilty (one line change)

2023-08-03 Thread David Rowley
On Tue, 1 Aug 2023 at 01:48, Julien Rouhaud wrote: > Apart from that +1 from me for the patch, I think it helps focusing the > attention on what actually matters here. I think it's worth doing something to improve this code. However, I think we should go a bit further than what the proposed

Re: Fix incorrect start up costs for WindowAgg paths (bug #17862)

2023-08-03 Thread David Rowley
On Fri, 4 Aug 2023 at 11:54, Nathan Bossart wrote: > I'm seeing some reliable test failures for 32-bit builds on cfbot [0]. At > a glance, it looks like the relations are swapped in the plan. Thank you for the report. I've just pushed a patch which I'm hoping will fix it. David

Re: Fix incorrect start up costs for WindowAgg paths (bug #17862)

2023-08-03 Thread David Rowley
On Fri, 4 Aug 2023 at 02:02, Andy Fan wrote: > I have checked the updated patch and LGTM. Thank you for reviewing. I've pushed the patch to master only. David

Re: Fix incorrect start up costs for WindowAgg paths (bug #17862)

2023-08-03 Thread David Rowley
Thanks for having a look at this. On Thu, 3 Aug 2023 at 18:49, Andy Fan wrote: > 1. ORDER BY or PARTITION BY > > select *, count(two) over (order by unique1) from tenk1 limit 1; > DEBUG: startup_tuples = 1 > DEBUG: startup_tuples = 1 > > select *, count(two) over (partition by unique1) from

Re: Fix incorrect start up costs for WindowAgg paths (bug #17862)

2023-08-02 Thread David Rowley
On Wed, 31 May 2023 at 12:59, David Rowley wrote: > > On Wed, 12 Apr 2023 at 21:03, David Rowley wrote: > > I'll add this to the "Older bugs affecting stable branches" section of > > the PG 16 open items list > > When I wrote the above, it was very soon afte

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-08-01 Thread David Rowley
On Wed, 2 Aug 2023 at 12:25, David Rowley wrote: > master @ 3845577cb > latency average = 1575.879 ms > >6.79% postgres [.] pg_strtoint32_safe > > master~1 > latency average = 1968.004 ms > > 14.28% postgres [.] pg_strtoint32_safe > >

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-08-01 Thread David Rowley
On Wed, 2 Aug 2023 at 07:38, Dean Rasheed wrote: > Running the new test on slightly older Intel hardware (i9-9900K, gcc > 11.3), I get the following: Thanks for running those tests. I've now pushed the fastpath4.patch after making a few adjustments to the header comments to mention the new stuff

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-08-01 Thread David Rowley
On Wed, 2 Aug 2023 at 01:26, Dean Rasheed wrote: > > On Tue, 1 Aug 2023 at 13:55, David Rowley wrote: > > > > I tried adding the "at least 1 digit check" by adding an else { goto > > slow; } in the above code, but it seems to generate slower code than >

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-08-01 Thread David Rowley
On Tue, 1 Aug 2023 at 13:25, Andres Freund wrote: > There's a lot of larger numbers in the file, which likely reduces the impact > some. And there's the overhead of actually inserting the rows into the table, > making the difference appear smaller than it is. It might be worth special casing the

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-08-01 Thread David Rowley
On Mon, 31 Jul 2023 at 21:39, John Naylor wrote: > master + pg_strtoint_fastpath1.patch > latency average = 938.146 ms > latency stddev = 9.354 ms > > master + pg_strtoint_fastpath2.patch > latency average = 902.808 ms > latency stddev = 3.957 ms Thanks for checking those two on your machine.

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-27 Thread David Rowley
On Thu, 27 Jul 2023 at 14:51, David Rowley wrote: > Just to keep this moving and to make it easier for people to test the > pg_strtoint patches, I've pushed the fix_COPY_DEFAULT.patch patch. > The only thing I changed was to move the line that was allocating the > array to a location

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-26 Thread David Rowley
On Wed, 26 Jul 2023 at 03:50, Andres Freund wrote: > On 2023-07-25 23:37:08 +1200, David Rowley wrote: > > On Tue, 25 Jul 2023 at 17:34, Andres Freund wrote: > > I've not really studied the fix_COPY_DEFAULT.patch patch. Is there a > > reason to delay committing th

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-26 Thread David Rowley
> On 2023-07-25 23:37:08 +1200, David Rowley wrote: > > On Tue, 25 Jul 2023 at 17:34, Andres Freund wrote: > > > HEAD: 812.690 > > > > > > your patch: 821.354 > > > > > > strtoint from 8692f6644

Re: Partition pruning not working on updates

2023-07-25 Thread David Rowley
On Tue, 25 Jul 2023 at 20:45, Mr.Bim wrote: > Partition pruning is not working on the updates query, am I missing something? In PG13, partition pruning for UPDATE and DELETE only works during query planning. Because you're using CURRENT_TIMESTAMP, that's not an immutable expression which can be

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-25 Thread David Rowley
On Tue, 25 Jul 2023 at 17:34, Andres Freund wrote: > prep: > COPY (SELECT generate_series(1, 200) a, (random() * 10 - 5)::int > b, 3243423 c) TO '/tmp/lotsaints.copy'; > DROP TABLE lotsaints; CREATE UNLOGGED TABLE lotsaints(a int, b int, c int); > > benchmark: > psql -qX -c 'truncate

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-20 Thread David Rowley
On Thu, 20 Jul 2023 at 20:37, Dean Rasheed wrote: > > On Thu, 20 Jul 2023 at 00:56, David Rowley wrote: > I agree with the principal though. In the attached updated patch, I > replaced that test with a simpler one: > > +/* > + * Process the number's digits. We optim

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-19 Thread David Rowley
On Wed, 19 Jul 2023 at 23:14, Dean Rasheed wrote: > Hmm, I'm somewhat sceptical about this second patch. It's not obvious > why adding such tests would speed it up, and indeed, testing on my > machine with 50M rows, I see a noticeable speed-up from patch 1, and a > slow-down from patch 2: I

Re: Add TOAST support for more system tables

2023-07-17 Thread David Rowley
On Tue, 18 Jul 2023 at 10:31, Tom Lane wrote: > I wonder whether we'd be better off shoving the ACL data out of > these catalogs and putting it somewhere else (compare pg_attrdef). relpartbound is another column that could cause a pg_class row to grow too large. I did have a patch [1] to move

Re: unrecognized node type while displaying a Path due to dangling pointer

2023-07-17 Thread David Rowley
On Mon, 17 Jul 2023 at 15:31, Tom Lane wrote: > > I also didn't do anything about ExtensibleNode types. I assume just > > copying the ExtensibleNode isn't good enough. To flat copy the actual > > node I think would require adding a new function to > > ExtensibleNodeMethods. > > Yeah, the problem

Re: unrecognized node type while displaying a Path due to dangling pointer

2023-07-16 Thread David Rowley
On Wed, 12 Jul 2023 at 14:50, David Rowley wrote: > > On Wed, 12 Jul 2023 at 14:23, Tom Lane wrote: > > I did think about that, but "shallow copy a Path" seems nontrivial > > because the Path structs are all different sizes. Maybe it is worth > > build

Re: Changing types of block and chunk sizes in memory contexts

2023-07-16 Thread David Rowley
On Fri, 14 Jul 2023 at 18:53, Melih Mutlu wrote: > David Rowley , 13 Tem 2023 Per, 08:04 tarihinde şunu > yazdı: >> >> I looked at your v2 patch. The only thing that really looked wrong >> were the (Size) casts in the context creation functions. These should >> ha

Re: Changing types of block and chunk sizes in memory contexts

2023-07-12 Thread David Rowley
gt;chunksPerBlock = chunksPerBlock; @@ -506,7 +506,7 @@ SlabAlloc(MemoryContext context, Size size) /* make sure we only allow correct request size */ if (unlikely(size != slab->chunkSize)) - elog(ERROR, "unexpected alloc chunk size %zu (expected %zu)", +

Re: Correct the documentation for work_mem

2023-07-12 Thread David Rowley
On Tue, 25 Apr 2023 at 04:20, Imseih (AWS), Sami wrote: > > Based on the feedback, here is a v1 of the suggested doc changes. > > I modified Gurjeets suggestion slightly to make it clear that a specific > query execution could have operations simultaneously using up to > work_mem. > -

Re: document the need to analyze partitioned tables

2023-07-12 Thread David Rowley
On Wed, 25 Jan 2023 at 21:43, David Rowley wrote: > While I agree that the majority of partitions are likely to be > relkind='r', which you might ordinarily consider a "normal table", you > just might change your mind when you try to INSERT or UPDATE records > that would

Re: unrecognized node type while displaying a Path due to dangling pointer

2023-07-11 Thread David Rowley
On Wed, 12 Jul 2023 at 14:23, Tom Lane wrote: > I did think about that, but "shallow copy a Path" seems nontrivial > because the Path structs are all different sizes. Maybe it is worth > building some infrastructure to support that? It seems a reasonable thing to have to do. It seems the

Re: unrecognized node type while displaying a Path due to dangling pointer

2023-07-11 Thread David Rowley
On Wed, 12 Jul 2023 at 08:46, Tom Lane wrote: > A low-cost fix perhaps could be to unlink the lower rel's whole > path list (set input_rel->pathlist = NIL, also zero the related > fields such as cheapest_path) once we've finished selecting the > paths we want for the upper rel. That's not great

Re: Add bump memory context type and use it for tuplesorts

2023-07-10 Thread David Rowley
On Tue, 27 Jun 2023 at 21:19, David Rowley wrote: > I've attached the bump allocator patch and also the script I used to > gather the performance results in the first 2 tabs in the attached > spreadsheet. I've attached a v2 patch which changes the BumpContext a little to re

Re: Check lateral references within PHVs for memoize cache keys

2023-07-08 Thread David Rowley
On Sat, 8 Jul 2023 at 17:24, Paul A Jungwirth wrote: > One adjacent thing I noticed is that when we renamed "Result Cache" to > "Memoize" this bit of the docs in config.sgml got skipped (probably > because of the line break): > > Hash tables are used in hash joins, hash-based aggregation,

Re: Check lateral references within PHVs for memoize cache keys

2023-07-08 Thread David Rowley
On Sun, 9 Jul 2023 at 05:28, Tom Lane wrote: > More generally, it's not clear to me why we should need to look inside > lateral PHVs in the first place. Wouldn't the lateral PHV itself > serve fine as a cache key? For Memoize specifically, I purposefully made it so the expression was used as a

Re: Improve the performance of nested loop join in the case of partitioned inner table

2023-07-04 Thread David Rowley
On Thu, 13 Apr 2023 at 03:00, Alexandr Nikulin wrote: > explain analyze select * from ids join test_part on ids.id=test_part.id where > ascii(ids.name)=ascii('best case'); > explain analyze select * from ids join test_part on ids.id=test_part.id where > ascii(ids.name)=ascii('worst case'); > >

Re: Incremental sort for access method with ordered scan support (amcanorderbyop)

2023-07-04 Thread David Rowley
On Tue, 4 Jul 2023 at 20:12, Richard Guo wrote: > The v4 patch looks good to me (maybe some cosmetic tweaks are still > needed for the comments). I think it's now 'Ready for Committer'. I agree. I went and hit the comments with a large hammer and while there also adjusted the regression tests.

Re: Making empty Bitmapsets always be NULL

2023-07-03 Thread David Rowley
On Mon, 3 Jul 2023 at 18:10, Yuya Watari wrote: > Thank you for your reply. I am +1 to your change. I think these > assertions will help someone who changes the Bitmapset implementations > in the future. I've now pushed the patch. Thanks for all your reviews and detailed benchmarks. David

Re: Remove WindowClause PARTITION BY items belonging to redundant pathkeys

2023-07-02 Thread David Rowley
On Mon, 12 Jun 2023 at 20:20, Richard Guo wrote: > So now the v2 patch looks good to me. Thank you for reviewing this. I've just pushed the patch. David

Re: Making empty Bitmapsets always be NULL

2023-07-02 Thread David Rowley
On Mon, 3 Jul 2023 at 09:27, David Rowley wrote: > If nobody else wants to take a look, then I plan to push the v4 + the > asserts in the next day or so. Here's the patch which includes those Asserts. I also made some small tweaks to a comment. I understand that Tom thought that the A

Re: Changing types of block and chunk sizes in memory contexts

2023-06-28 Thread David Rowley
On Thu, 29 Jun 2023 at 09:26, Tomas Vondra wrote: > AllocSetContext 224B -> 208B (4 cachelines) > GenerationContext 152B -> 136B (3 cachelines) > SlabContext 200B -> 200B (no change, adds 4B hole) > > Nothing else changes, AFAICS. I don't think a lack of a reduction in the number

Re: Making empty Bitmapsets always be NULL

2023-06-28 Thread David Rowley
Thank you for running the profiles. On Tue, 27 Jun 2023 at 21:11, Yuya Watari wrote: > On Sat, Jun 24, 2023 at 1:15 PM David Rowley wrote: > > I think it's also important to check we don't slow anything down for > > more normal-sized sets. The vast majority of sets wi

Re: Changing types of block and chunk sizes in memory contexts

2023-06-28 Thread David Rowley
On Wed, 28 Jun 2023 at 20:13, Peter Eisentraut wrote: > size_t (= Size) is the correct type in C to store the size of an object > in memory. This is partially a self-documentation issue: If I see > size_t in a function signature, I know what is intended; if I see > uint32, I have to wonder what

Re: Speeding Up Bitmapset

2023-06-25 Thread David Rowley
On Mon, 26 Jun 2023 at 12:55, Ranier Vilela wrote: > > Em dom., 25 de jun. de 2023 às 17:49, David Rowley > escreveu: >> >> There's no reason in the world >> that those will speed up Bitmapsets, so why include them? > > Of course optimization is the most im

Re: Speeding Up Bitmapset

2023-06-25 Thread David Rowley
On Mon, 26 Jun 2023 at 12:55, Ranier Vilela wrote: > Have you seen bms_compare? > For some reason someone thought it would be better to loop backward the array. That's nothing to do with efficiency. It's related to behaviour. Have a look at the function's header comment, it's trying to find the

Re: Speeding Up Bitmapset

2023-06-25 Thread David Rowley
On Mon, 26 Jun 2023 at 00:29, Ranier Vilela wrote: > Yuya Watari presented a series of patches, with the objective of improving > the Bitmapset [1]. > After reading through the patches, I saw a lot of good ideas and thought I'd > help. > Unfortunately, my suggestions were not well received.

Re: Stampede of the JIT compilers

2023-06-24 Thread David Rowley
On Sun, 25 Jun 2023 at 05:54, Tom Lane wrote: > > James Coleman writes: > > On Sat, Jun 24, 2023 at 7:40 AM Tomas Vondra > > wrote: > >> On 6/24/23 02:33, David Rowley wrote: > >>> On Sat, 24 Jun 2023 at 02:28, James Coleman wrote: > >>>>

Re: Allow ordered partition scans in more cases

2023-06-24 Thread David Rowley
On Sat, 4 Mar 2023 at 00:56, David Rowley wrote: > What's on my mind now is if turning 1 Sort into N Sorts is a > particularly good idea from a work_mem standpoint. I see that we don't > do tuplesort_end() until executor shutdown, so that would mean that we > could end up using 1 x

Re: Making empty Bitmapsets always be NULL

2023-06-23 Thread David Rowley
On Thu, 22 Jun 2023 at 20:59, Yuya Watari wrote: > Table 1: Planning time and its speedup of Join Order Benchmark > (n: the number of partitions of each table) > (Speedup: higher is better) > 64 | 115.7% > 128 | 142.9% > 256 | 187.7% Thanks for benchmarking. It certainly

Re: Making empty Bitmapsets always be NULL

2023-06-23 Thread David Rowley
On Sat, 24 Jun 2023 at 07:43, Ranier Vilela wrote: > I worked a bit more on the v4 version and made a new v6 version, with some > changes. > I can see some improvement, would you mind testing v6 and reporting back? Please don't bother. I've already mentioned that I'm not going to consider any

Re: Stampede of the JIT compilers

2023-06-23 Thread David Rowley
On Sat, 24 Jun 2023 at 02:28, James Coleman wrote: > There are a couple of issues here. I'm sure it's been discussed > before, and it's not the point of my thread, but I can't help but note > that the default value of jit_above_cost of 10 seems absurdly low. > On good hardware like we have

Re: Making empty Bitmapsets always be NULL

2023-06-21 Thread David Rowley
On Thu, 22 Jun 2023 at 00:16, Ranier Vilela wrote: > 2. Only compute BITNUM when necessary. I doubt this will help. The % 64 done by BITNUM will be transformed to an AND operation by the compiler which is likely going to be single instruction latency on most CPUs which probably amounts to it

Re: extended statistics n-distinct on multiple columns not used when join two tables

2023-06-21 Thread David Rowley
On Tue, 13 Jun 2023 at 23:29, Pavel Stehule wrote: >> I think it's probably worth adjusting the docs to mention this. It >> seems like it might be something that could surprise someone. >> >> Something like the attached, maybe? > > +1 Ok, I pushed that patch. Thanks. David

Re: Making empty Bitmapsets always be NULL

2023-06-19 Thread David Rowley
On Thu, 15 Jun 2023 at 20:57, Yuya Watari wrote: > > On Tue, Jun 13, 2023 at 8:07 PM David Rowley wrote: > > For the fix in the 0004 patch, I think we can do what you did more > > simply. I don't think there's any need to perform the loop to find > > the last non-zero wo

Re: extended statistics n-distinct on multiple columns not used when join two tables

2023-06-13 Thread David Rowley
(moving to -hackers) On Tue, 13 Jun 2023 at 21:30, Pavel Stehule wrote: > út 13. 6. 2023 v 11:21 odesílatel James Pang (chaolpan) > napsal: >> When join two table on multiple columns equaljoin, rows estimation >> always use selectivity = multiplied by distinct multiple individual

Re: Making empty Bitmapsets always be NULL

2023-06-13 Thread David Rowley
On Tue, 13 Jun 2023 at 00:32, Yuya Watari wrote: > In March, I reported that David's patch caused a degradation in > planning performance. I have investigated this issue further and found > some bugs in the patch. Due to these bugs, Bitmapset operations in the > original patch computed incorrect

Re: Remove WindowClause PARTITION BY items belonging to redundant pathkeys

2023-06-11 Thread David Rowley
On Fri, 9 Jun 2023 at 20:57, Richard Guo wrote: > > On Fri, Jun 9, 2023 at 8:13 AM David Rowley wrote: >> It might be possible to make adjustments in nodeWindowAgg.c to have >> the equality checks come out as true when there is no ORDER BY. >> update_frameheadpos() is

Re: Remove WindowClause PARTITION BY items belonging to redundant pathkeys

2023-06-08 Thread David Rowley
Thank you for having a look at this. On Thu, 8 Jun 2023 at 21:11, Richard Guo wrote: > > On Thu, Jun 8, 2023 at 7:37 AM David Rowley wrote: >> >> What the attached patch does is process each WindowClause and removes >> any items from the PARTITION BY clause that ar

Re: Use COPY for populating all pgbench tables

2023-06-07 Thread David Rowley
On Thu, 8 Jun 2023 at 07:16, Tristan Partin wrote: > > master: > > 5000 of 5000 tuples (100%) done (elapsed 260.93 s, remaining 0.00 s)) > vacuuming... > creating primary keys... > done in 1414.26 s (drop tables 0.20 s, create tables 0.82 s, client-side > generate 1280.43 s, vacuum 2.55

Remove WindowClause PARTITION BY items belonging to redundant pathkeys

2023-06-07 Thread David Rowley
Recently Markus Winand pointed out to me that the PG15 changes made in [1] to teach the query planner about monotonic window functions improved the situation for PostgreSQL on his feature/optimization timeline for PostgreSQL. These can be seen in [2]. Unfortunately, if you look at the timeline

Re: [16Beta1][doc] pgstat: Track time of the last scan of a relation

2023-06-04 Thread David Rowley
On Wed, 31 May 2023 at 15:57, Shinoda, Noriyoshi (PN Japan FSIP) wrote: > PostgreSQL 16 Beta1, added last access time to pg_stat_all_tables and > pg_stat_all_indexes views by this patch [1]. > According to the documentation [2], the data type of the columns added to > these views is

Re: [16Beta1][doc] pgstat: Track time of the last scan of a relation

2023-05-31 Thread David Rowley
On Wed, 31 May 2023 at 15:57, Shinoda, Noriyoshi (PN Japan FSIP) wrote: > According to the documentation [2], the data type of the columns added to > these views is 'timestamptz'. > However, columns of the same data type in pg_stat_all_tables.last_vacuum, > last_analyze and other tables are

Re: PG 16 draft release notes ready

2023-05-31 Thread David Rowley
On Wed, 31 May 2023 at 11:32, Bruce Momjian wrote: > > On Thu, May 25, 2023 at 05:57:25PM +1200, David Rowley wrote: > > On 64-bit builds, it was 16 bytes for AllocSet contexts, 24 bytes for > > generation contexts and 16 bytes for slab contexts. > > Okay, item added

Re: Fix incorrect start up costs for WindowAgg paths (bug #17862)

2023-05-30 Thread David Rowley
On Wed, 12 Apr 2023 at 21:03, David Rowley wrote: > I'm not sure if we should consider backpatching a fix for this bug. > We tend not to commit stuff that would destabilise plans in the back > branches. On the other hand, it's fairly hard to imagine how we > could make this muc

Re: benchmark results comparing versions 15.2 and 16

2023-05-28 Thread David Rowley
On Tue, 23 May 2023 at 07:40, MARK CALLAGHAN wrote: (pg15) > --- Q2.10k : explain analyze SELECT c FROM sbtest1 WHERE id BETWEEN 1000 > AND 1001 order by c; > QUERY PLAN >

Re: PG 16 draft release notes ready

2023-05-24 Thread David Rowley
On Thu, 25 May 2023 at 05:45, Bruce Momjian wrote: > > On Wed, May 24, 2023 at 01:43:50PM -0400, Bruce Momjian wrote: > > > * Reduce palloc() memory overhead for all memory allocations down to 8 > > > bytes on all platforms. (Andres Freund, David Rowley) > > > >

Re: PG 16 draft release notes ready

2023-05-23 Thread David Rowley
On Wed, 24 May 2023 at 15:54, Bruce Momjian wrote: > > On Wed, May 24, 2023 at 08:37:45AM +1200, David Rowley wrote: > > On Mon, 22 May 2023 at 07:05, Jonathan S. Katz wrote: > > > * Parallel execution of queries that use `FULL` and `OUTER` joins > > > > I t

Re: PG 16 draft release notes ready

2023-05-23 Thread David Rowley
g which items to include. > Can you suggest some text? This? > > Improve efficiency of memory usage to allow for better scaling Maybe something like: * Reduce palloc() memory overhead for all memory allocations down to 8 bytes on all platforms. (Andres Freund, David Rowley

Re: PG 16 draft release notes ready

2023-05-23 Thread David Rowley
On Mon, 22 May 2023 at 07:05, Jonathan S. Katz wrote: > * Parallel execution of queries that use `FULL` and `OUTER` joins I think this should be `RIGHT` joins rather than `OUTER` joins. LEFT joins have been parallelizable I think for a long time now. David

Re: benchmark results comparing versions 15.2 and 16

2023-05-10 Thread David Rowley
On Thu, 11 May 2023 at 01:00, Alexander Lakhin wrote: > This time `git bisect` pointed at 3c6fc5820. Having compared execution plans > (both attached), I see the following differences (3c6fc5820~1 vs 3c6fc5820): Based on what you've sent, I'm uninspired to want to try to do anything about it.

Re: 2023-05-11 release announcement draft

2023-05-07 Thread David Rowley
Thanks for working on this. On Sun, 7 May 2023 at 15:37, Jonathan S. Katz wrote: > Please provide any suggestions, corrections, or notable omissions no > later than 2023-05-11 0:00 AoE. For this one: > * Fix partition pruning logic for partitioning on boolean columns when using a > `IS NOT

Re: Rename 'lpp' to 'lp' in heapam.c

2023-05-02 Thread David Rowley
On Wed, 3 May 2023 at 12:16, Yaphters W wrote: > I just found the naming of the ItemId variables is not consistent in > heapam.c. There are 13 'lpp's and 112 'lp's. Technically 'lpp' is correct as > ItemId is a line pointer's pointer and there used to be code like "++lpp" for > line pointer

Re: Doc limitation update proposal: include out-of-line OID usage per TOAST-ed columns

2023-04-26 Thread David Rowley
On Sun, 23 Apr 2023, 3:42 am Gurjeet Singh, wrote: > I anticipate that edits to Appendix K Postgres Limits will prompt > improving the note in there about the maximum column limit, That note > is too wordy, and sometimes confusing, especially for the audience > that it's written for: newcomers

Re: Option to not use ringbuffer in VACUUM, using it in failsafe mode

2023-04-26 Thread David Rowley
On Wed, 26 Apr 2023, 8:48 pm Masahiko Sawada, wrote: > I realized that the value of vacuum_buffer_usage_limit parameter in > postgresql.conf.sample doesn't have the unit: > > #vacuum_buffer_usage_limit = 256 # size of vacuum and analyze buffer > access strategy ring. >

Re: Improve list manipulation in several places

2023-04-21 Thread David Rowley
On Fri, 21 Apr 2023 at 23:16, Ranier Vilela wrote: > Perhaps list_delete_nth_cell needs to check NIL too? > + if (list == NIL) > + return NIL; Which cell would you be deleting from an empty list? David

Re: Fix typos and inconsistencies for v16

2023-04-20 Thread David Rowley
On Wed, 19 Apr 2023 at 07:00, Alexander Lakhin wrote: > please look at the similar list for v15+ (596b5af1d..HEAD). I've now pushed most of these but didn't include the following ones: > 3. BufFileOpenShared -> BufFileOpenFileSet // see dcac5e7ac Maybe I need to spend longer, but I just didn't

Re: New committers: Nathan Bossart, Amit Langote, Masahiko Sawada

2023-04-20 Thread David Rowley
On Fri, 21 Apr 2023 at 05:40, Tom Lane wrote: > > The Core Team would like to extend our congratulations to > Nathan Bossart, Amit Langote, and Masahiko Sawada, who have > accepted invitations to become our newest Postgres committers. > > Please join me in wishing them much success and few

Re: Incremental sort for access method with ordered scan support (amcanorderbyop)

2023-04-20 Thread David Rowley
On Thu, 20 Apr 2023 at 18:46, Richard Guo wrote: > > > On Thu, Apr 20, 2023 at 6:38 AM David Rowley wrote: >> >> That function is pretty new and was exactly added so we didn't have to >> write list_truncate(list_copy(...), n) anymore. That gets pretty >> was

Re: Should we put command options in alphabetical order in the doc?

2023-04-20 Thread David Rowley
On Wed, 19 Apr 2023 at 22:04, Alvaro Herrera wrote: > > On 2023-Apr-18, Peter Geoghegan wrote: > > > While I'm certain that nobody will agree with me on every little > > detail, I have to imagine that most would find my preferred ordering > > quite understandable and unsurprising, at a high level

Re: Note new NULLS NOT DISTINCT on unique index tutorial page

2023-04-20 Thread David Rowley
On Thu, 20 Apr 2023 at 12:04, David Gilman wrote: > The revised patch is good. Please go ahead and commit whatever > phrasing you or the other committers find acceptable. I don't really > have any preferences in how this is exactly phrased, I just think it > should be mentioned in the docs.

Re: Incremental sort for access method with ordered scan support (amcanorderbyop)

2023-04-19 Thread David Rowley
On Wed, 19 Apr 2023 at 16:53, Miroslav Bendik wrote: > > 2. You can use list_copy_head(root->query_pathkeys, > > list_length(orderbyclauses)); instead of: > > > > + useful_pathkeys = list_truncate(list_copy(root->query_pathkeys), > > + list_length(orderbyclauses)); > > This code will crash if

Re: Should we put command options in alphabetical order in the doc?

2023-04-18 Thread David Rowley
On Tue, 18 Apr 2023 at 18:53, Peter Geoghegan wrote: > Take the VACUUM command. Right now FULL, FREEZE, and VERBOSE all come > first. Those options are approximately the most important options -- > especially VERBOSE. But your patch places VERBOSE dead last. hmm, how can we verify that the

Re: Incremental sort for access method with ordered scan support (amcanorderbyop)

2023-04-18 Thread David Rowley
On Tue, 18 Apr 2023 at 19:29, Miroslav Bendik wrote: > here is an updated patch with proposed changes. Here's a quick review: 1. I don't think this is required. match_pathkeys_to_index() sets these to NIL and they're set accordingly by the other code paths. - List*orderbyclauses; - List

Re: Option to not use ringbuffer in VACUUM, using it in failsafe mode

2023-04-17 Thread David Rowley
On Tue, 18 Apr 2023 at 09:21, Melanie Plageman wrote: > Are we still thinking that reordering the VACUUM (and ANALYZE) options > makes sense. And, if so, should it be alphabetical within parameter > category? That is, all actual parameters (e.g. FULL and FREEZE) are > alphabetically organized

Should we put command options in alphabetical order in the doc?

2023-04-17 Thread David Rowley
Over on [1], Peter mentions that we might want to consider putting the VACUUM options into some order that's better than the apparent random order that they're currently in. VACUUM is certainly one command that's grown a fairly good number of options over the years and it appears we've not given

<    1   2   3   4   5   6   7   8   9   10   >