Re: Add memory context type to pg_backend_memory_contexts view

2024-04-23 Thread David Rowley
On Tue, 16 Apr 2024 at 13:30, David Rowley wrote: > In [1] Andres mentioned that there's no way to determine the memory > context type in pg_backend_memory_contexts. This is a bit annoying as > I'd like to add a test to exercise BumpStats(). > > Having the context type in the

Re: Streaming I/O, vectored I/O (WIP)

2024-04-23 Thread David Rowley
I've attached a patch with a few typo fixes and what looks like an incorrect type for max_ios. It's an int16 and I think it needs to be an int. Doing "max_ios = Min(max_ios, PG_INT16_MAX);" doesn't do anything when max_ios is int16. David diff --git a/src/backend/storage/aio/read_stream.c

Re: Minor document typo

2024-04-23 Thread David Rowley
On Wed, 24 Apr 2024 at 00:11, Tatsuo Ishii wrote: > Just out of a curiosity, is it possible to say "low a wal_level on the > primary"? (just "too" is removed) Prefixing the adjective with "too" means it's beyond the acceptable range. "This coffee is too hot".

Re: Minor document typo

2024-04-23 Thread David Rowley
On Tue, 23 Apr 2024 at 23:17, Tatsuo Ishii wrote: >Number of uses of logical slots in this database that have been >canceled due to old snapshots or too low a linkend="guc-wal-level"/> >on the primary > > I think "too low a" should be "too low" ('a' is not > necessary).

Re: Typos in the code and README

2024-04-19 Thread David Rowley
On Fri, 19 Apr 2024 at 20:13, Daniel Gustafsson wrote: > Thanks, I incorporated these into 0001 before pushing. All the commits in > this > patchset are now applied. Here are a few more to see if it motivates anyone else to do a more thorough search for another batch. Fixes duplicate words

Re: Stability of queryid in minor versions

2024-04-19 Thread David Rowley
On Tue, 16 Apr 2024 at 15:16, Michael Paquier wrote: > > On Tue, Apr 16, 2024 at 02:04:22PM +1200, David Rowley wrote: > > It makes sense to talk about the hashing variations closer to the > > object identifier part. > > Mostly what I had in mind. A separate for the n

Re: Why is parula failing?

2024-04-16 Thread David Rowley
On Tue, 16 Apr 2024 at 18:58, Robins Tharakan wrote: > The last 25 consecutive runs have passed [1] after switching > REL_12_STABLE to -O0 ! So I am wondering whether that confirms that > the compiler version is to blame, and while we're still here, > is there anything else I could try? I don't

Re: Add bump memory context type and use it for tuplesorts

2024-04-16 Thread David Rowley
On Mon, 8 Apr 2024 at 00:37, David Rowley wrote: > I've now pushed all 3 patches. Thank you for all the reviews on > these and for the extra MemoryContextMethodID bit, Matthias. I realised earlier today when working on [1] that bump makes a pretty brain-dead move when adding dedicated

Re: Add bump memory context type and use it for tuplesorts

2024-04-16 Thread David Rowley
On Tue, 16 Apr 2024 at 17:13, Amul Sul wrote: > Attached is a small patch adding the missing BumpContext description to the > README. Thanks for noticing and working on the patch. There were a few things that were not quite accurate or are misleading: 1. > +These three memory contexts aim to

Re: Differential code coverage between 16 and HEAD

2024-04-15 Thread David Rowley
On Tue, 16 Apr 2024 at 14:29, Andres Freund wrote: > I think total_nblocks might also not be entirely stable? I think it is stable for this test. However, I'll let the buildfarm make the final call on that. The reason I want to include it is that I'd like to push the large allocations to the

Re: Stability of queryid in minor versions

2024-04-15 Thread David Rowley
On Tue, 16 Apr 2024 at 12:10, Michael Paquier wrote: > Not sure that this is an improvement in clarity. There are a few > bullet points that treat about the instability of the query ID, and > your patch is now mixing the query ID being different for two > mostly-identical queries on the same

Re: Differential code coverage between 16 and HEAD

2024-04-15 Thread David Rowley
On Tue, 16 Apr 2024 at 10:57, Andres Freund wrote: > I guess was thinking more about BumpIsEmpty() and BumpStats() then the "bogus" > cases. But BumpIsEmpty() likely is unreachable as well. The only call to MemoryContextIsEmpty() I see is AtSubCommit_Memory() and it's on an aset.c context type.

Add memory context type to pg_backend_memory_contexts view

2024-04-15 Thread David Rowley
0a58697a4b88bc3ac80f09ed78b56ebe903a2aae Mon Sep 17 00:00:00 2001 From: David Rowley Date: Tue, 16 Apr 2024 13:05:34 +1200 Subject: [PATCH v1] Add context type field to pg_backend_memory_contexts --- doc/src/sgml/system-views.sgml | 9 + src/backend/utils/adt/mcxtfuncs.c | 50

Re: Differential code coverage between 16 and HEAD

2024-04-15 Thread David Rowley
On Mon, 15 Apr 2024 at 10:33, Andres Freund wrote: > - The new bump allocator has a fair amount of uncovered functionality: > > https://anarazel.de/postgres/cov/16-vs-HEAD-2024-04-14/src/backend/utils/mmgr/bump.c.gcov.html#L293 The attached adds a test to tuplesort to exercise

Re: Why is parula failing?

2024-04-14 Thread David Rowley
On Mon, 15 Apr 2024 at 16:10, Robins Tharakan wrote: > - I now have 2 separate runs stuck on pg_sleep() - HEAD / REL_16_STABLE > - I'll keep them (stuck) for this week, in case there's more we can get > from them (and to see how long they take) > - Attached are 'bt full' outputs for both (b.txt -

Re: Stability of queryid in minor versions

2024-04-14 Thread David Rowley
On Mon, 15 Apr 2024 at 14:09, Michael Paquier wrote: > > On Mon, Apr 15, 2024 at 01:31:47PM +1200, David Rowley wrote: > > I'm unsure if "Rule of thumb" is the correct way to convey that. We > > can't really write "We endeavour to", as who is "We&quo

Re: Fix out-of-bounds in the function GetCommandTagName

2024-04-14 Thread David Rowley
On Mon, 15 Apr 2024 at 12:12, Ranier Vilela wrote: > > Em dom., 14 de abr. de 2024 às 20:38, David Rowley > escreveu: >> You seem to have forgotten to attach it, but my comments above were >> written with the assumption that the patch is what I've attached here. >

Re: Stability of queryid in minor versions

2024-04-14 Thread David Rowley
On Mon, 15 Apr 2024 at 13:37, David G. Johnston wrote: > Seems we can improve things by simply removing the "rule of thumb" sentence > altogether. The prior paragraph states the things the queryid depends upon > at the level of detail the reader needs. I don't think that addresses the

Re: Stability of queryid in minor versions

2024-04-14 Thread David Rowley
On Mon, 15 Apr 2024 at 13:19, Tom Lane wrote: > > Michael Paquier writes: > > On Mon, Apr 15, 2024 at 11:20:16AM +1200, David Rowley wrote: > >> 1. We cannot change Node enums in minor versions > >> 2. We're *unlikely* to add fields to Node types in minor versions,

Re: Stability of queryid in minor versions

2024-04-14 Thread David Rowley
On Mon, 15 Apr 2024 at 12:04, Michael Paquier wrote: > Since 16 these new fields would be added by default unless the node > attribute query_jumble_ignore is appended to it. I agree that this > may not be entirely intuitive when it comes to force compatibility > across the same major version.

Re: Fix out-of-bounds in the function GetCommandTagName

2024-04-14 Thread David Rowley
On Mon, 15 Apr 2024 at 11:54, Tom Lane wrote: > would this also allow us to get rid of any default: > cases in switches on command tags? git grep "case CMDTAG_" does not yield any results. As far as I understand, we'd only be able to get rid of a default case if we had a switch that included

Re: Stability of queryid in minor versions

2024-04-14 Thread David Rowley
On Mon, 15 Apr 2024 at 11:47, Peter Geoghegan wrote: > > On Sun, Apr 14, 2024 at 7:20 PM David Rowley wrote: > > It's the "underlying server version" that I think needs some > > clarification. It's unclear if the minor version must match or just > > the m

Re: Fix out-of-bounds in the function GetCommandTagName

2024-04-14 Thread David Rowley
On Mon, 15 Apr 2024 at 11:17, Ranier Vilela wrote: > Coverity has reported some out-of-bounds bugs > related to the GetCommandTagName function. > > The size of the array is defined by COMMAND_TAG_NEXTTAG enum, > whose value currently corresponds to 193. > Since enum items are evaluated starting

Stability of queryid in minor versions

2024-04-14 Thread David Rowley
I was recently asked internally about the stability guarantees we offer for queryid. My answer consisted of: 1. We cannot change Node enums in minor versions 2. We're *unlikely* to add fields to Node types in minor versions, and if we did we'd likely be leaving them out of the jumble calc, plus

Re: Typos in the code and README

2024-04-14 Thread David Rowley
On Sat, 13 Apr 2024 at 09:17, Daniel Gustafsson wrote: > > > On 12 Apr 2024, at 23:15, Heikki Linnakangas wrote: > > Here's a few more. I've accumulate these over the past couple of months, > > keeping them stashed in a branch, adding to it whenever I've spotted a > > minor typo while reading

SupportRequestRows support function for generate_series_timestamptz

2024-04-13 Thread David Rowley
this doesn't help the case in [1] as the generate_series inputs are not const there) David [1] https://www.postgresql.org/message-id/CAMPYKo0FouB-HZ1k-_Ur2v%2BkK71q0T5icQGrp%2BSPbQJGq0H2Rw%40mail.gmail.com From ca0e982215d1335d015a2b515f038b7186935af0 Mon Sep 17 00:00:00 2001 From: David Rowley

Re: Incorrect handling of IS [NOT] NULL quals on inheritance parents

2024-04-12 Thread David Rowley
On Thu, 11 Apr 2024 at 18:49, Richard Guo wrote: > I agree with both of your points. But I also think we do not need to > skip applying the NOT NULL qual optimization for partitioned tables. > For partitioned tables, if the parent is marked NOT NULL, then all its > children must also be marked

Re: Typos in the code and README

2024-04-11 Thread David Rowley
On Fri, 12 Apr 2024, 1:05 am Daniel Gustafsson, wrote: > Now that the tree has settled down a bit post-freeze I ran some tooling to > check spelling. I was primarily interested in docs and README* which were > mostly free from simply typos, while the code had some in various comments > and >

Re: Incorrect handling of IS [NOT] NULL quals on inheritance parents

2024-04-10 Thread David Rowley
On Wed, 10 Apr 2024 at 19:12, Richard Guo wrote: > And I think recording NOT NULL columns for traditional inheritance > parents can be error-prone for some future optimization where we look > at an inheritance parent's notnullattnums and make decisions based on > the assumption that the included

Re: Incorrect handling of IS [NOT] NULL quals on inheritance parents

2024-04-09 Thread David Rowley
On Wed, 10 Apr 2024 at 17:18, Tom Lane wrote: > > David Rowley writes: > > I think a better fix is just to not apply the optimisation for > > inheritance RTEs in add_base_clause_to_rel(). > > Is it worth paying attention to whether the constraint is marked > conn

Re: Incorrect handling of IS [NOT] NULL quals on inheritance parents

2024-04-09 Thread David Rowley
On Tue, 9 Apr 2024 at 21:55, Richard Guo wrote: > > In b262ad440e we introduced an optimization that drops IS NOT NULL quals > on a NOT NULL column, and reduces IS NULL quals on a NOT NULL column to > constant-FALSE. I happened to notice that this is not working correctly > for traditional

Re: Why is parula failing?

2024-04-09 Thread David Rowley
On Tue, 9 Apr 2024 at 15:48, David Rowley wrote: > Still no partition_prune failures on master since the compiler version > change. There has been one [1] in REL_16_STABLE. I'm thinking it > might be worth backpatching the partition_prune debug to REL_16_STABLE > to see if we can le

Re: "an SQL" vs. "a SQL"

2024-04-09 Thread David Rowley
On Tue, 9 Apr 2024 at 16:18, David Rowley wrote: > There's just 1 instance of "a SQL" that crept into PG16 after > d866f0374. This probably means I'd be better off doing this in June a > few weeks before branching... > > Patch attached. Pushed. David

Re: Fixup some StringInfo usages

2024-04-09 Thread David Rowley
On Tue, 9 Apr 2024 at 14:27, Tom Lane wrote: > > David Rowley writes: > > Similar to f736e188c, I've attached a patch that fixes up a few > > misusages of the StringInfo functions. These just swap one function > > call for another function that is more suited to the use

Re: "an SQL" vs. "a SQL"

2024-04-08 Thread David Rowley
On Tue, 11 Apr 2023 at 17:43, David Rowley wrote: > > On Fri, 11 Jun 2021 at 13:44, David Rowley wrote: > > Anyway, I'll set an alarm for this time next year so I can check on > > how many inconsistencies have crept back in over the development > > cycle. >

Re: Why is parula failing?

2024-04-08 Thread David Rowley
On Mon, 8 Apr 2024 at 23:56, Robins Tharakan wrote: > #3 0x0083ed84 in WaitLatch (latch=, > wakeEvents=wakeEvents@entry=41, timeout=60, > wait_event_info=wait_event_info@entry=150994946) at latch.c:538 > #4 0x00907404 in pg_sleep (fcinfo=) at misc.c:406 > #17

Fixup a few 2023 copyright years

2024-04-08 Thread David Rowley
Attached is a patch which adjusts the copyright years of 2023 that have crept in this year from patches that were written last year and committed without adjusting this to 2024. The patch isn't produced by src/tools/copyright.pl as that'll transform files which are new and only contain "2023" to

Fixup some StringInfo usages

2024-04-08 Thread David Rowley
ab6ffda29c9d1d258a8918190dc3b84cbe5efecf Mon Sep 17 00:00:00 2001 From: David Rowley Date: Tue, 5 Mar 2024 09:07:05 +1300 Subject: [PATCH v1] Fixup various StringInfo function usages --- contrib/postgres_fdw/deparse.c | 4 ++-- src/backend/replication/logical/slotsync.c | 2 +- src/backend/replication

Re: enhance the efficiency of migrating particularly large tables

2024-04-08 Thread David Rowley
On Tue, 9 Apr 2024 at 11:02, Tom Lane wrote: > > David Rowley writes: > > Unsure if such a feature is worthwhile. I think maybe not for just > > min(ctid)/max(ctid). However, there could be other reasons, such as > > the transform OR to UNION stuff that Tom w

Re: enhance the efficiency of migrating particularly large tables

2024-04-08 Thread David Rowley
On Tue, 9 Apr 2024 at 09:52, David Zhang wrote: > However, when executing SELECT min(ctid) and max(ctid), it performs a > Seq Scan, which can be slow for a large table. Is there a way to > retrieve the minimum and maximum ctid other than using the system > functions min() and max()? Finding the

Re: Improve heapgetpage() performance, overhead from serializable

2024-04-07 Thread David Rowley
On Mon, 8 Apr 2024 at 16:08, Andres Freund wrote: > > On 2024-04-08 15:43:12 +1200, David Rowley wrote: > > I understand wanting to avoid the long name. I'd rather stay clear of > > "visible", but don't feel as strongly about this as it's static. >

Re: Improve heapgetpage() performance, overhead from serializable

2024-04-07 Thread David Rowley
On Mon, 8 Apr 2024 at 15:13, Andres Freund wrote: > I kinda don't like heap_prepare_pagescan(), but heapgetpage() is worse. And I > don't have a great alternative suggestion. It came around from having nothing better. I was keen not to have the name indicate it was only for checking visibility

Re: Improve heapgetpage() performance, overhead from serializable

2024-04-07 Thread David Rowley
On Sun, 7 Apr 2024 at 19:30, Andres Freund wrote: > Good call. Added and pushed. I understand you're already aware of the reference in the comment to heapgetpage(), which no longer exists as of 44086b097. Melanie and I had discussed the heap_prepare_pagescan() name while I was reviewing that

Re: Add bump memory context type and use it for tuplesorts

2024-04-07 Thread David Rowley
On Mon, 8 Apr 2024 at 09:09, Andres Freund wrote: > I suspect that KeeperBlock() isn't returning true, because IsKeeperBlock > misses > the MAXALIGN(). I think that about fits with: Thanks for investigating that. I've just pushed a fix for the macro and also adjusted a location which was

Re: Flushing large data immediately in pqcomm

2024-04-07 Thread David Rowley
On Sun, 7 Apr 2024 at 22:05, Jelte Fennema-Nio wrote: > > On Sun, 7 Apr 2024 at 03:39, Andres Freund wrote: > > Changing the global vars to size_t seems mildly bogus to me. All it's > > achieving is to use slightly more memory. It also just seems unrelated to > > the > > change. > > I took a

Re: Add bump memory context type and use it for tuplesorts

2024-04-07 Thread David Rowley
On Sun, 7 Apr 2024 at 22:05, John Naylor wrote: > > On Sat, Apr 6, 2024 at 7:37 PM David Rowley wrote: > > > I'm planning on pushing these, pending a final look at 0002 and 0003 > > on Sunday morning NZ time (UTC+12), likely in about 10 hours time. > > +1 I've now pu

Re: Flushing large data immediately in pqcomm

2024-04-07 Thread David Rowley
On Sun, 7 Apr 2024 at 08:21, Andres Freund wrote: > I added WITH BINARY, SET STORAGE EXTERNAL and tested both unix socket and > localhost. I also reduced row counts and iteration counts, because I am > impatient, and I don't think it matters much here. Attached the modified > version. Thanks for

Re: Add bump memory context type and use it for tuplesorts

2024-04-06 Thread David Rowley
On Sun, 7 Apr 2024 at 05:45, Matthias van de Meent wrote: > Malloc's docs specify the minimum chunk size at 4*sizeof(void*) and itself > uses , so using powers of 2 for chunks would indeed fail to detect 1s in the > 4th bit. I suspect you'll get different results when you check the allocation

Re: Flushing large data immediately in pqcomm

2024-04-06 Thread David Rowley
On Sat, 6 Apr 2024 at 23:17, Jelte Fennema-Nio wrote: > Weird that on your machines you don't see a difference. Are you sure > you didn't make a silly mistake, like not restarting postgres or > something? I'm sure. I spent quite a long time between the AMD and an Apple m2 trying. I did see the

Re: Add bump memory context type and use it for tuplesorts

2024-04-05 Thread David Rowley
On Sat, 6 Apr 2024 at 03:24, Tom Lane wrote: > OK. I did not read the patch very closely, but at least in principle > I have no further objections. David, are you planning to take point > on getting this in? Yes. I'll be looking soon. David

Re: Popcount optimization using AVX512

2024-04-05 Thread David Rowley
On Sat, 6 Apr 2024 at 14:17, Nathan Bossart wrote: > > On Sat, Apr 06, 2024 at 12:08:14PM +1300, David Rowley wrote: > > Won't Valgrind complain about this? > > > > +pg_popcount_avx512(const char *buf, int bytes) > > > > + buf = (const char *) TYPEALIGN_DOWN(

Re: Flushing large data immediately in pqcomm

2024-04-05 Thread David Rowley
On Fri, 5 Apr 2024 at 03:28, Melih Mutlu wrote: > > Jelte Fennema-Nio , 4 Nis 2024 Per, 16:34 tarihinde şunu > yazdı: >> >> On Thu, 4 Apr 2024 at 13:08, Melih Mutlu wrote: >> > I changed internal_flush() to an inline function, results look better this >> > way. >> >> It seems you also change

Re: Popcount optimization using AVX512

2024-04-05 Thread David Rowley
On Sat, 6 Apr 2024 at 04:38, Nathan Bossart wrote: > This seems to provide a small performance boost, so I've incorporated it > into v27. Won't Valgrind complain about this? +pg_popcount_avx512(const char *buf, int bytes) + buf = (const char *) TYPEALIGN_DOWN(sizeof(__m512i), buf); + val =

Re: Streaming read-ready sequential scan code

2024-04-04 Thread David Rowley
On Thu, 4 Apr 2024 at 16:45, David Rowley wrote: > I've pushed the v9-0001 with that rename done. I've now just pushed the 0002 patch with some revisions: 1. The function declarations you added for heapgettup_advance_block() and heapgettup_initial_block() didn't match the propert

Re: Streaming read-ready sequential scan code

2024-04-03 Thread David Rowley
On Thu, 4 Apr 2024 at 14:38, Melanie Plageman wrote: > Looking at it in the code, I am wondering if we should call > heap_page_prep() heap_scan_page_prep(). Just wondering if it is clear > that it is prepping a page to be scanned. You choose whatever you > think is best. I ended up calling it

Re: Popcount optimization using AVX512

2024-04-03 Thread David Rowley
On Thu, 4 Apr 2024 at 11:50, Nathan Bossart wrote: > If we can verify this approach won't cause segfaults and can stomach the > regression between 8 and 16 bytes, I'd happily pivot to this approach so > that we can avoid the function call dance that I have in v25. > > Thoughts? If we're worried

Re: On disable_cost

2024-04-03 Thread David Rowley
On Thu, 4 Apr 2024 at 10:15, David Rowley wrote: > In short, I don't find it strange that disabling one node type results > in considering another type that we'd otherwise not consider in cases > where we assume that the disabled node type is always superior and > should always

Re: PostgreSQL 17 Release Management Team & Feature Freeze

2024-04-03 Thread David Rowley
On Mon, 18 Mar 2024 at 15:50, Michael Paquier wrote: > Additionally, the RMT has set the feature freeze to be **April 8, 2024 > at 0:00 AoE** (see [1]). This is the last time to commit features for > PostgreSQL 17. In other words, no new PostgreSQL 17 feature can be > committed after April 8,

Re: Parent/child context relation in pg_get_backend_memory_contexts()

2024-04-03 Thread David Rowley
On Thu, 4 Apr 2024 at 12:34, Michael Paquier wrote: > I've been re-reading the patch again to remember what this is about, > and I'm OK with having this "path" column in the catalog. However, > I'm somewhat confused by the choice of having a temporary number that > shows up in the catalog

Re: Streaming read-ready sequential scan code

2024-04-03 Thread David Rowley
On Thu, 4 Apr 2024 at 06:03, Melanie Plageman wrote: > Attached v8 is rebased over current master (which now has the > streaming read API). I've looked at the v8-0001 patch. I wasn't too keen on heapbuildvis() as a function name for an external function. Since it also does pruning work, it

Re: On disable_cost

2024-04-03 Thread David Rowley
On Thu, 4 Apr 2024 at 08:21, Robert Haas wrote: > I wanted to further explore the idea of just not generating plans of > types that are currently disabled. I looked into doing this for > enable_indexscan and enable_indexonlyscan. As a first step, I > investigated how those settings work now, and

Re: Why is parula failing?

2024-04-01 Thread David Rowley
On Sat, 30 Mar 2024 at 09:17, Tom Lane wrote: > > I wrote: > > I'd not looked closely enough at the previous failure, because > > now that I have, this is well out in WTFF territory: how can > > reltuples be greater than zero when relpages is zero? This can't > > be a state that autovacuum would

Re: Crash on UNION with PG 17

2024-04-01 Thread David Rowley
On Thu, 28 Mar 2024 at 04:34, Regina Obe wrote: > CREATE TABLE edge_data AS > SELECT i AS edge_id, i + 1 AS start_node, i + 2 As end_node > FROM generate_series(1,10) AS i; > > WITH edge AS ( > SELECT start_node, end_node > FROM edge_data > WHERE edge_id = 1 > ) > SELECT

Re: Properly pathify the union planner

2024-04-01 Thread David Rowley
On Fri, 29 Mar 2024 at 08:53, Tom Lane wrote: > On third thought, I'm not at all convinced that we even want to > invent this struct as compared to just adding another parameter > to subquery_planner. The problem with a struct is what happens > the next time we need to add a parameter? If we

Re: Properly pathify the union planner

2024-03-27 Thread David Rowley
On Thu, 28 Mar 2024 at 15:56, Tom Lane wrote: > I haven't studied the underlying problem yet, so I'm not quite > buying into whether we need this struct at all ... The problem is, when planning a UNION child query, we want to try and produce some Paths that would suit the top-level UNION query

Re: Crash on UNION with PG 17

2024-03-27 Thread David Rowley
On Thu, 28 Mar 2024 at 04:34, Regina Obe wrote: > The issue can be exercised without postgis installed as follows: > > > CREATE TABLE edge_data AS > SELECT i AS edge_id, i + 1 AS start_node, i + 2 As end_node > FROM generate_series(1,10) AS i; > > WITH edge AS ( > SELECT start_node,

Re: incorrect results and different plan with 2 very similar queries

2024-03-27 Thread David Rowley
On Thu, 28 Mar 2024 at 10:33, Dave Cramer wrote: > There is a report on the pgjdbc github JDBC Driver shows erratic behavior > when filtering on CURRENT_DATE · pgjdbc/pgjdbc · Discussion #3184 (github.com) > > Here are the plans. > > JDBC - Nested Loop (incorrect result) > >

Re: Flushing large data immediately in pqcomm

2024-03-27 Thread David Rowley
On Fri, 22 Mar 2024 at 12:46, Melih Mutlu wrote: > I did all of the above changes and it seems like those resolved the > regression issue. Thanks for adjusting the patch. The numbers do look better, but on looking at your test.sh script from [1], I see: meson setup --buildtype debug

Re: Parallel Aggregates for string_agg and array_agg

2024-03-27 Thread David Rowley
On Wed, 27 Mar 2024 at 03:00, Alexander Lakhin wrote: > I've discovered that the test query: > -- Ensure parallel aggregation is actually being used. > explain (costs off) select * from v_pagg_test order by y; > > added by 16fd03e95 fails sometimes. For instance: >

Re: Properly pathify the union planner

2024-03-27 Thread David Rowley
On Wed, 27 Mar 2024 at 22:47, David Rowley wrote: > I did wonder when first working on this patch if subquery_planner() > should grow an extra parameter, or maybe consolidate some existing > ones by passing some struct that provides the planner with a bit more > context about the q

Re: Properly pathify the union planner

2024-03-27 Thread David Rowley
On Wed, 27 Mar 2024 at 16:15, Richard Guo wrote: >if (root->parent_root != NULL && >root->parent_root->parse->setOperations != NULL && >IsA(root->parent_root->parse->setOperations, SetOperationStmt)) >qp_extra.setop = >(SetOperationStmt *)

Re: Why is parula failing?

2024-03-27 Thread David Rowley
On Wed, 27 Mar 2024 at 18:28, Tom Lane wrote: > > David Rowley writes: > > Unfortunately, REL_16_STABLE does not have the additional debugging, > > so don't get to know what reltuples was set to. > > Let's wait a bit to see if it fails in HEAD ... but if not, would &g

Re: Propagate pathkeys from CTEs up to the outer query

2024-03-27 Thread David Rowley
On Wed, 27 Mar 2024 at 06:20, Tom Lane wrote: > That's not the fault of anything we did here; the IndexOnlyScan path > in the subquery is in fact not marked with any pathkeys, even though > clearly its result is sorted. I believe that's an intentional > decision from way way back, that pathkeys

Re: Why is parula failing?

2024-03-26 Thread David Rowley
On Tue, 26 Mar 2024 at 21:03, Tharakan, Robins wrote: > > > David Rowley writes: > > It would be good to have log_autovacuum_min_duration = 0 on this machine > > for a while. > > - Have set log_autovacuum_min_duration=0 on parula and a test run came out > okay

Re: Properly pathify the union planner

2024-03-26 Thread David Rowley
On Wed, 27 Mar 2024 at 06:00, Alexander Lakhin wrote: > SELECT count(*) FROM ( >WITH q1(x) AS (SELECT 1) >SELECT FROM q1 UNION SELECT FROM q1 > ) qu; > > TRAP: failed Assert("lg != NULL"), File: "planner.c", Line: 7941, PID: 1133017 Thanks for finding that. There's something weird going

Re: Add bump memory context type and use it for tuplesorts

2024-03-25 Thread David Rowley
On Tue, 26 Mar 2024 at 03:53, Tom Lane wrote: > I agree with this completely. However, the current design for chunk > headers is mighty restrictive about how many kinds of contexts we can > have. We need to open that back up. Andres mentioned how we could do this in [1]. One possible issue

Re: Why is parula failing?

2024-03-25 Thread David Rowley
On Thu, 21 Mar 2024 at 14:19, Tom Lane wrote: > > David Rowley writes: > > We could also do something like the attached just in case we're > > barking up the wrong tree. > > Yeah, checking indisvalid isn't a bad idea. I'd put another > one further down, just before

Re: Add bump memory context type and use it for tuplesorts

2024-03-25 Thread David Rowley
On Tue, 5 Mar 2024 at 15:42, David Rowley wrote: > The query I ran was: > > select chksz,mtype,pg_allocate_memory_test_reset(chksz, > 1024*1024,1024*1024*1024, mtype) from (values(8),(16),(32),(64)) > sizes(chksz),(values('aset'),('generation'),('slab'),('bump')) > cxt(mtype) or

Re: Properly pathify the union planner

2024-03-24 Thread David Rowley
On Tue, 12 Mar 2024 at 23:21, David Rowley wrote: > I've attached v3. I spent quite a bit more time looking at this. I discovered that the dNumGroups wasn't being set as it should have been for INTERSECT and EXCEPT queries. There was a plan change as a result of this. I've fi

Re: Flushing large data immediately in pqcomm

2024-03-21 Thread David Rowley
On Thu, 21 Mar 2024 at 22:44, Jelte Fennema-Nio wrote: > > On Thu, 21 Mar 2024 at 01:45, David Rowley wrote: > > As I understand the code, there's no problem calling > > internal_flush_buffer() when the buffer is empty and I suspect that if > > we're sending

Re: Why is parula failing?

2024-03-20 Thread David Rowley
On Thu, 21 Mar 2024 at 12:36, Tom Lane wrote: > So yeah, if we could have log_autovacuum_min_duration = 0 perhaps > that would yield a clue. FWIW, I agree with your earlier statement about it looking very much like auto-vacuum has run on that table, but equally, if something like the pg_index

Re: Flushing large data immediately in pqcomm

2024-03-20 Thread David Rowley
On Thu, 21 Mar 2024 at 13:24, Melih Mutlu wrote: > What if I do a simple comparison like PqSendStart == PqSendPointer instead of > calling pq_is_send_pending() as Heikki suggested, then this check should not > hurt that much. Right? Does that make sense? As I understand the code, there's no

Re: Popcount optimization using AVX512

2024-03-20 Thread David Rowley
On Wed, 20 Mar 2024 at 11:56, Amonson, Paul D wrote: > Changed in this patch set. > > * Rebased. > * Direct *slow* calls via macros as shown in example patch. > * Changed the choose filename to be platform specific as suggested. > * Falls back to intermediate "Fast" methods if AVX512 is not

Re: Trying to build x86 version on windows using meson

2024-03-20 Thread David Rowley
On Thu, 21 Mar 2024 at 11:00, Andres Freund wrote: > > On 2024-03-20 17:49:14 -0400, Dave Cramer wrote: > > First off this is on an ARM64 machine > > Uh, that's a fairly crucial bit - you're actually trying to cross compile > then. I don't know much about cross compiling on windows, so it's

Re: Flushing large data immediately in pqcomm

2024-03-20 Thread David Rowley
On Fri, 15 Mar 2024 at 01:46, Heikki Linnakangas wrote: > - the "(int *) )" cast is not ok, and will break visibly on > big-endian systems where sizeof(int) != sizeof(size_t). I think fixing this requires adjusting the signature of internal_flush_buffer() to use size_t instead of int. That

Re: Flushing large data immediately in pqcomm

2024-03-20 Thread David Rowley
On Fri, 15 Mar 2024 at 02:03, Jelte Fennema-Nio wrote: > > On Thu, 14 Mar 2024 at 13:12, Robert Haas wrote: > > > > On Thu, Mar 14, 2024 at 7:22 AM Melih Mutlu wrote: > > > 1- Even though I expect both the patch and HEAD behave similarly in case > > > of small data (case 1: 100 bytes), the

Re: Why is parula failing?

2024-03-20 Thread David Rowley
On Wed, 20 Mar 2024 at 08:58, Tom Lane wrote: > I suppose we could attach "autovacuum=off" settings to these tables, > but it doesn't seem to me that that should be necessary. These test > cases are several years old and haven't given trouble before. > Moreover, if that's necessary then there

Re: Popcount optimization using AVX512

2024-03-19 Thread David Rowley
On Wed, 20 Mar 2024 at 11:56, Amonson, Paul D wrote: > Changed in this patch set. Thanks for rebasing. I don't think there's any need to mention Intel in each of the following comments: +# Check for Intel AVX512 intrinsics to do POPCNT calculations. +# Newer Intel processors can use AVX-512

Re: pg16: XX000: could not find pathkey item to sort

2024-03-19 Thread David Rowley
On Mon, 18 Mar 2024 at 18:50, Ashutosh Bapat wrote: > If the problem you speculate is different from this one, I am not able to see > it. It might help give an example query or explain more. I looked at this again and I might have been wrong about there being a problem. I set a breakpoint in

Re: Popcount optimization using AVX512

2024-03-18 Thread David Rowley
On Tue, 19 Mar 2024 at 11:08, Nathan Bossart wrote: > > On Mon, Mar 18, 2024 at 04:29:19PM -0500, Nathan Bossart wrote: > > Agreed. Will send an updated patch shortly. > > As promised... Looks good. David

Re: Popcount optimization using AVX512

2024-03-18 Thread David Rowley
On Tue, 19 Mar 2024 at 10:08, Nathan Bossart wrote: > > On Tue, Mar 19, 2024 at 10:02:18AM +1300, David Rowley wrote: > > The only thing I'd question in the patch is in pg_popcount_fast(). It > > looks like you've opted to not do the 32-bit processing on 32-bit > > machine

Re: Popcount optimization using AVX512

2024-03-18 Thread David Rowley
On Tue, 19 Mar 2024 at 06:30, Nathan Bossart wrote: > Here is a more fleshed-out version of what I believe David is proposing. > On my machine, the gains aren't quite as impressive (~8.8s to ~5.2s for the > test_popcount benchmark). I assume this is because this patch turns > pg_popcount() into

Re: Popcount optimization using AVX512

2024-03-17 Thread David Rowley
On Sat, 16 Mar 2024 at 04:06, Nathan Bossart wrote: > I ran John Naylor's test_popcount module [0] with the following command on > an i7-1195G7: > > time psql postgres -c 'select drive_popcount(1000, 1024)' > > Without your patches, this seems to take somewhere around 8.8 seconds. >

Re: Inconsistent printf placeholders

2024-03-14 Thread David Rowley
On Fri, 15 Mar 2024 at 15:27, Kyotaro Horiguchi wrote: > I have considered only the two messages. Actually, buffile.c and md.c > are already like that. The attached aligns the messages in > pg_combinebackup.c and reconstruct.c with the precedents. This looks like a worthy cause to make

Re: Add bump memory context type and use it for tuplesorts

2024-03-14 Thread David Rowley
ot all at once. (I've not studied your algorithm for the rebalance.) David [1] https://github.com/david-rowley/postgres/tree/malloccache

Re: pg16: XX000: could not find pathkey item to sort

2024-03-14 Thread David Rowley
On Thu, 14 Mar 2024 at 12:00, David Rowley wrote: > I've attached a patch which fixes the problem for me. I've pushed the patch to fix gather_grouping_paths(). The issue with the RelOptInfo having the incorrect PathTarget->exprs after the partial phase of partition-wise aggregate r

Re: JIT compilation per plan node

2024-03-14 Thread David Rowley
Thanks for chipping in here. On Fri, 15 Mar 2024 at 08:14, Robert Haas wrote: > A slightly subtler way the patch could lose is if the new threshold is > harder to adjust than the old one. For example, imagine that you have > a query that does a Cartesian join. That makes the cost of the input >

Re: pg16: XX000: could not find pathkey item to sort

2024-03-14 Thread David Rowley
On Thu, 14 Mar 2024 at 18:23, Ashutosh Bapat wrote: > I don't understand why root->query_pathkeys has both a and b. "a" is there > because of GROUP BY and ORDER BY clause. But why "b"? So that the ORDER BY aggregate function can be evaluated without nodeAgg.c having to perform the sort. See

Re: pg16: XX000: could not find pathkey item to sort

2024-03-13 Thread David Rowley
On Thu, 14 Mar 2024 at 06:00, Alexander Lakhin wrote: > I've stumbled upon the same error, but this time it apparently has another > cause. It can be produced (on REL_16_STABLE and master) as follows: > CREATE TABLE t (a int, b int) PARTITION BY RANGE (a); > CREATE TABLE td PARTITION OF t

Re: On disable_cost

2024-03-12 Thread David Rowley
On Wed, 13 Mar 2024 at 08:55, Robert Haas wrote: > But in the absence of that, we need some way to privilege the > non-disabled paths over the disabled ones -- and I'd prefer to have > something more principled than disable_cost, if we can work out the > details. The primary place I see issues

  1   2   3   4   5   6   7   8   9   10   >