Re: Use generation memory context for tuplestore.c

2024-05-10 Thread Matthias van de Meent
On Sat, 4 May 2024 at 04:02, David Rowley wrote: > > On Sat, 4 May 2024 at 03:51, Matthias van de Meent > wrote: > > Was a bump context considered? If so, why didn't it make the cut? > > If tuplestore_trim is the only reason why the type of context in patch > >

Re: SQL:2011 application time

2024-05-09 Thread Matthias van de Meent
NTLY modifier, nor an INVALID modifier. This means temporal unique constraints have much less administrative wiggle room than normal unique constraints, and I think that's not great. Kind regards, Matthias van de Meent.

Re: Parallel CREATE INDEX for GIN indexes

2024-05-09 Thread Matthias van de Meent
or the patch, Agreed. > and I don't think I'll have time to work on this > anytime soon. The patch is not extremely complex, but it's not trivial > either. But if someone wants to take a stab at extending tuplesort to > allow this, I won't object ... Same here: While I do have some

Expand applicability of aggregate's sortop optimization

2024-05-08 Thread Matthias van de Meent
ths to the aggregate's sort operator, but that'd significantly increase complexity here. Kind regards, Matthias van de Meent Neon (https://neon.tech) From 215600d12164f214aae8f0de94b16373bd202d69 Mon Sep 17 00:00:00 2001 From: Matthias van de Meent Date: Thu, 25 Apr 2024 15:23:57 +0200 Subject: [PA

Re: Use generation memory context for tuplestore.c

2024-05-03 Thread Matthias van de Meent
quire rewind capabilities (i.e. where _trim is never effectively executed)? > master @ 8f0a97dff > Storage: Memory Maximum Storage: 16577kB > > patched: > Storage: Memory Maximum Storage: 8577kB Those are some impressive numbers. Kind regards, Matthias van de Meent Neon

Re: Why is FOR ORDER BY function getting called when the index is handling ordering?

2024-05-02 Thread Matthias van de Meent
ecute that projection. This happens regardless of it's use in downstream nodes due to planner or executor limitations. See also Heikki's thread over at [0], and a comment of me about the same issue over at pgvector's issue board at [1]. Kind regards, Matthias van de Meent Neon (https://neon.tech

Re: Parallel CREATE INDEX for GIN indexes

2024-05-02 Thread Matthias van de Meent
thing similar - it tracks if the TID value goes > backward in the callback, and if it does it dumps the state into the > tuplesort before processing the first tuple from the beginning of the > table. Which means we end up with two separate "narrow" TID list, not > one very wide one. See note above: We may still need a merge phase, just to make sure we handle all TAM parallel scans correctly, even if that merge join phase wouldn't get hit in vanilla PostgreSQL. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Possible to get LIMIT in an index access method?

2024-04-29 Thread Matthias van de Meent
nk we can know in advance how many tuples are going to be extracted from an index scan. Kind regards, Matthias van de Meent.

Re: Extend ALTER DEFAULT PRIVILEGES for large objects

2024-04-26 Thread Matthias van de Meent
we'll have to figure out how we're going to transfer those larger values to (and from) clients. For large objects, this is much less of an issue because the IO operations are already chunked by design, but this may not work well for types that you want to use in your table's columns. Kind regards,

Re: Statistics Import and Export

2024-04-24 Thread Matthias van de Meent
On Wed, 24 Apr 2024 at 21:31, Bruce Momjian wrote: > > On Tue, Apr 23, 2024 at 06:33:48PM +0200, Matthias van de Meent wrote: > > I've heard of use cases where dumping stats without data would help > > with production database planner debugging on a non-prod system. > >

Re: Cleanup: remove unused fields from nodes

2024-04-24 Thread Matthias van de Meent
On Wed, 24 Apr 2024 at 09:28, Michael Paquier wrote: > > On Tue, Apr 23, 2024 at 11:03:40PM -0400, Tom Lane wrote: > > Hah. Seems like the comment for isall needs to explain that it > > exists for this purpose, so we don't make this mistake again. > > How about something like the attached?

Re: Statistics Import and Export

2024-04-23 Thread Matthias van de Meent
larger with the current stats. As for other planner inputs: table size is relatively easy to shim with sparse files; cumulative statistics can be copied from a donor replica if needed, and btree indexes only really really need to contain their highest and lowest values (and need their height set correctly). Kind regards, Matthias van de Meent

Re: Cleanup: remove unused fields from nodes

2024-04-22 Thread Matthias van de Meent
On Mon, 22 Apr 2024 at 17:41, Tom Lane wrote: > > Matthias van de Meent writes: > > 0002/0004 remove fields in ExecRowMark which were added for FDWs to > > use, but there are no FDWs which use this: I could only find two FDWs > > who implement RefetchForeignRow

Cleanup: remove unused fields from nodes

2024-04-22 Thread Matthias van de Meent
. /cc Tom Lane and Etsuro Fujita: 2 and 4 were introduced with your commit afb9249d in 2015. /cc Amit Kapila: 0003 was introduced with your spearheaded commit 6185c973 this year. Kind regards, Matthias van de Meent 0001 removes two old fields that are not in use anywhere anymore, but at some point

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-04-10 Thread Matthias van de Meent
ng SERIALIZE to the default set of explain features enabled with ANALYZE? I'll add this to "Decisions to Recheck Mid-Beta"-section if nobody has a clear objection. Kind regards, Matthias van de Meent Neon (https://neon.tech) [0] https://postgr.es/m/ea885631-21f1-425a-97ed-c4bfb8cf9c63%40gmx.de

Re: Parallel Recovery in PostgreSQL

2024-04-09 Thread Matthias van de Meent
the heap tuple of the index entry wasn't even inserted yet. Index-only scans could thus return invalid results. See also the wiki page [0] on parallel recovery, and Koichi-san's repository [1] with his code for parallel replay based on PG 14.6. Kind regards, Matthias van de Meent [0] https://wiki.po

Re: PostgreSQL 17 Release Management Team & Feature Freeze

2024-04-09 Thread Matthias van de Meent
On Mon, 8 Apr 2024 at 20:15, Tomas Vondra wrote: > > > On 4/8/24 17:48, Matthias van de Meent wrote: >> On Mon, 8 Apr 2024 at 17:21, Tomas Vondra >> wrote: >>> >>> Maybe it'd be better to start by expanding the existing rule about not >>>

Re: PostgreSQL 17 Release Management Team & Feature Freeze

2024-04-08 Thread Matthias van de Meent
On Mon, 8 Apr 2024 at 17:21, Tomas Vondra wrote: > > > > On 4/8/24 16:59, Tom Lane wrote: > > Heikki Linnakangas writes: > >> On 08/04/2024 16:43, Tom Lane wrote: > >>> I was just about to pen an angry screed along the same lines. > >>> The commit flux over the past couple days, and even the

Re: Add bump memory context type and use it for tuplesorts

2024-04-06 Thread Matthias van de Meent
On Sun, 7 Apr 2024, 01:59 David Rowley, wrote: > On Sun, 7 Apr 2024 at 05:45, Matthias van de Meent > wrote: > > Malloc's docs specify the minimum chunk size at 4*sizeof(void*) and > itself uses , so using powers of 2 for chunks would indeed fail to detect > 1s in the 4th bit

Re: Add bump memory context type and use it for tuplesorts

2024-04-06 Thread Matthias van de Meent
On Sat, 6 Apr 2024, 14:36 David Rowley, wrote: > On Sat, 6 Apr 2024 at 02:30, Matthias van de Meent > wrote: > > > > On Thu, 4 Apr 2024 at 22:43, Tom Lane wrote: > > > > > > Matthias van de Meent writes: > > > > It extends memory context IDs

Re: Add bump memory context type and use it for tuplesorts

2024-04-05 Thread Matthias van de Meent
On Thu, 4 Apr 2024 at 22:43, Tom Lane wrote: > > Matthias van de Meent writes: > > On Mon, 25 Mar 2024 at 22:44, Tom Lane wrote: > >> Basically, I'm not happy with consuming the last reasonably-available > >> pattern for a memory context type that has little claim

Re: Add bump memory context type and use it for tuplesorts

2024-04-04 Thread Matthias van de Meent
regards, Matthias From 4c0b4b1af98fcfecf80a30ea1834668b59d543a5 Mon Sep 17 00:00:00 2001 From: Matthias van de Meent Date: Thu, 4 Apr 2024 21:34:46 +0200 Subject: [PATCH v0] Add bitspace for more memory context types in MemoryChunk's hdrmask Assuming we don't want to use patterns from unused mem

Re: Experiments with Postgres and SSL

2024-04-04 Thread Matthias van de Meent
On Thu, 28 Mar 2024, 13:37 Heikki Linnakangas, wrote: > > On 28/03/2024 13:15, Matthias van de Meent wrote: > > On Tue, 5 Mar 2024 at 15:08, Heikki Linnakangas wrote: > >> > >> I hope I didn't joggle your elbow reviewing this, Jacob, but I spent > >>

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-04-04 Thread Matthias van de Meent
On Wed, 3 Apr 2024 at 23:50, Tom Lane wrote: > > Matthias van de Meent writes: >> On Tue, 2 Apr 2024 at 17:47, Tom Lane wrote: >>> IIUC, it's not possible to use the SERIALIZE option when explaining >>> CREATE TABLE AS, because you can't install the instrumen

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-04-03 Thread Matthias van de Meent
On Tue, 2 Apr 2024 at 17:47, Tom Lane wrote: > > Matthias van de Meent writes: > > Attached is v9, which is rebased on master to handle 24eebc65's > > changed signature of pq_sendcountedtext. > > It now also includes autocompletion, and a second patch which adds > &g

Re: Experiments with Postgres and SSL

2024-03-28 Thread Matthias van de Meent
Could you provide a new version of the patchset so that it can be reviewed in the context of current HEAD? Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: DRAFT: Pass sk_attno to consistent function

2024-03-21 Thread Matthias van de Meent
rts of the scankey are consistent with the table constraints" is a good reason to expose the index column to operator classes. Note that operator classes were built specifically so that they're independent from their column position. It doens't really make sense to expose this. Maybe my suggestion up above helps? Kind regards, Matthias van de Meent

Re: 13dev failed assert: comparetup_index_btree(): ItemPointer values should never be equal

2024-03-21 Thread Matthias van de Meent
ersion been upgraded? Considering the age of this thread, and thus potential for v14 pre-.4: Did this cluster use REINDEX (concurrently) for the relevant indexes? Kind regards, Matthias van de Meent

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2024-03-20 Thread Matthias van de Meent
On Sat, 16 Mar 2024 at 01:12, Peter Geoghegan wrote: > > On Fri, Mar 8, 2024 at 9:00 AM Matthias van de Meent > wrote: > > I've attached v14, where 0001 is v13, 0002 is a patch with small > > changes + some large comment suggestions, and 0003 which contains >

Re: Why is parula failing?

2024-03-20 Thread Matthias van de Meent
On Wed, 20 Mar 2024 at 11:50, Matthias van de Meent wrote: > > On Tue, 19 Mar 2024 at 20:58, Tom Lane wrote: > > > > For the last few days, buildfarm member parula has been intermittently > > failing the partition_prune regression test, due to unexpected plan

Re: Reducing output size of nodeToString

2024-03-20 Thread Matthias van de Meent
On Wed, 20 Mar 2024 at 12:49, Peter Eisentraut wrote: > > On 19.03.24 17:13, Peter Eisentraut wrote: > > On 11.03.24 21:52, Matthias van de Meent wrote: > >>> * v7-0003-gen_node_support.pl-Mark-location-fields-as-type-.patch > >>> > >>> This looks s

Re: Why is parula failing?

2024-03-20 Thread Matthias van de Meent
On Tue, 19 Mar 2024 at 20:58, Tom Lane wrote: > > For the last few days, buildfarm member parula has been intermittently > failing the partition_prune regression test, due to unexpected plan > changes [1][2][3][4]. The symptoms can be reproduced exactly by > inserting a "vacuum" of one or

Re: Reducing output size of nodeToString

2024-03-19 Thread Matthias van de Meent
On Tue, 19 Mar 2024 at 17:13, Peter Eisentraut wrote: > > On 11.03.24 21:52, Matthias van de Meent wrote: > >> * v7-0003-gen_node_support.pl-Mark-location-fields-as-type-.patch > >> > >> This looks sensible, but maybe making Location a global type is a bit

Re: documentation structure

2024-03-18 Thread Matthias van de Meent
s users can use vs those accessible to database owners and administrators, which is not currently organized as such in the SQL Commands section. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2024-03-18 Thread Matthias van de Meent
On Sat, 16 Mar 2024 at 01:12, Peter Geoghegan wrote: > > On Fri, Mar 8, 2024 at 9:00 AM Matthias van de Meent > wrote: > > I've attached v14, where 0001 is v13, 0002 is a patch with small > > changes + some large comment suggestions, and 0003 which contains >

Re: Disabling Heap-Only Tuples

2024-03-15 Thread Matthias van de Meent
On Wed, 13 Mar 2024 at 14:27, Laurenz Albe wrote: > > On Thu, 2023-09-21 at 16:18 -0700, Andres Freund wrote: > > I think a minimal working approach could be to have the configuration be > > based > > on the relation size vs space known to the FSM. If the target block of an > > update is higher

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-03-12 Thread Matthias van de Meent
. No problem. Remarks from users (when built on solid arguments) are always welcome, even if we may not always agree on the specifics. -->8-- Attached is v9, which is rebased on master to handle 24eebc65's changed signature of pq_sendcountedtext. It now also includes autocompletion, and

Re: Revisiting {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2024-03-12 Thread Matthias van de Meent
Do you know any less-hacky way? Or is it a nice way to go? I suppose we could be resetting the snapshot every so often? Or use multiple successive TID range scans with a new snapshot each? Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Reducing output size of nodeToString

2024-03-11 Thread Matthias van de Meent
On Mon, 11 Mar 2024 at 14:19, Peter Eisentraut wrote: > > On 22.02.24 16:07, Matthias van de Meent wrote: > > On Thu, 22 Feb 2024 at 13:37, Matthias van de Meent > > wrote: > >> > >> On Mon, 19 Feb 2024 at 14:19, Matthias van de Meent > >> wrote: &g

Re: btree: downlink right separator/HIKEY optimization

2024-03-11 Thread Matthias van de Meent
On Fri, 8 Mar 2024 at 20:14, Peter Geoghegan wrote: > > On Thu, Feb 22, 2024 at 10:42 AM Matthias van de Meent > wrote: > > I forgot to address this in the previous patch, so here's v3 which > > fixes the issue warning. > > What benchmarking have you done here? I

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2024-03-06 Thread Matthias van de Meent
On Wed, 6 Mar 2024 at 22:46, Matthias van de Meent wrote: > > On Wed, 6 Mar 2024 at 01:50, Peter Geoghegan wrote: > > At one point Heikki suggested that I just get rid of > > BTScanOpaqueData.arrayKeyData (which has been there for as long as > > nbtree had native sup

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2024-03-06 Thread Matthias van de Meent
On Wed, 6 Mar 2024 at 01:50, Peter Geoghegan wrote: > > On Mon, Mar 4, 2024 at 3:51 PM Matthias van de Meent > wrote: > > > +that searches for multiple values together. Queries that use certain > > > +SQL constructs to search for rows matching any > >

Re: Statistics Import and Export

2024-03-06 Thread Matthias van de Meent
On Wed, 6 Mar 2024 at 11:33, Stephen Frost wrote: > On Wed, Mar 6, 2024 at 11:07 Matthias van de Meent > wrote: >> Or even just one VALUES for the whole statistics loading? > > > I don’t think we’d want to go beyond one relation at a time as then it can be > parallel

Re: Statistics Import and Export

2024-03-06 Thread Matthias van de Meent
? I suspect the main issue with combining this into one statement (transaction) is that failure to load one column's statistics implies you'll have to redo all the other statistics (or fail to load the statistics at all), which may be problematic at the scale of thousands of relations with tens of columns each. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Revisiting {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2024-03-05 Thread Matthias van de Meent
On Wed, 21 Feb 2024 at 12:37, Michail Nikolaev wrote: > > Hi! > > > How do you suppose this would work differently from a long-lived > > normal snapshot, which is how it works right now? > > Difference in the ability to take new visibility snapshot periodically > during the second phase with

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2024-03-04 Thread Matthias van de Meent
continuescan field, in > the new page state struct). Cool. I'm planning to review the rest of this patch this week/tomorrow, could you take some time to review some of my btree patches, too? Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Avoid is possible a expensive function call (src/backend/utils/adt/varlena.c)

2024-03-04 Thread Matthias van de Meent
the user has access to the locale (and throw an error if not) before we return that the two strings are equal. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: btree: implement dynamic prefix truncation (was: Improving btree performance through specializing by key shape, take 2)

2024-03-01 Thread Matthias van de Meent
On Wed, 24 Jan 2024 at 13:02, Matthias van de Meent wrote: > > 1. > > Commit message refers to a non-existing reference '(see [0])'. > > Noted, I'll update that. > > > 2. > > +When we do a binary search on a sorted set (such as a BTree), we know that > >

Re: src/include/miscadmin.h outdated comments

2024-03-01 Thread Matthias van de Meent
oric value, there is something to be said about it delineating sections of the file. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Regardign RecentFlushPtr in WalSndWaitForWal()

2024-03-01 Thread Matthias van de Meent
idea of the currently flushed position. */ > else if (!RecoveryInProgress()) I can't find where this "else" of this "else if" clause came from, as this piece of code hasn't changed in years. But apart from that, your observation seems accurate, yes. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Experiments with Postgres and SSL

2024-02-28 Thread Matthias van de Meent
On Thu, 22 Feb 2024 at 18:02, Heikki Linnakangas wrote: > > On 22/02/2024 01:43, Matthias van de Meent wrote: >> On Wed, 10 Jan 2024 at 09:31, Heikki Linnakangas wrote: >>> 4. The number of combinations of sslmode, gssencmode and sslnegotiation >>> settings is sca

Re: Detoasting optionally to make Explain-Analyze less misleading

2024-02-26 Thread Matthias van de Meent
on format and names, but overall I think the patch is in a very decent shape. Stepan, thank you for your earlier work, and feel free to check it out or pick it up again if you want to; else I'll try to get this done. Kind regards, Matthias van de Meent [0] https://commitfest.postgresql.org/47/4852

Re: Sequence Access Methods, round two

2024-02-26 Thread Matthias van de Meent
on is not supported for tables. We seem to lose some details here: We can most definitely open tables. We just can't open them while treating them as sequences, which is not mentioned in the error message. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: btree: downlink right separator/HIKEY optimization

2024-02-22 Thread Matthias van de Meent
ixes the issue warning. Kind regards, Matthias van de Meent Neon (https://neon.tech) v3-0001-btree-optimize-_bt_moveright-using-downlink-s-rig.patch Description: Binary data

Re: Reducing output size of nodeToString

2024-02-22 Thread Matthias van de Meent
On Thu, 22 Feb 2024 at 13:37, Matthias van de Meent wrote: > > On Mon, 19 Feb 2024 at 14:19, Matthias van de Meent > wrote: > > Attached the updated version of the patch on top of 5497daf3, which > > incorporates this last round of feedback. > > Now attached rebase

Re: btree: downlink right separator/HIKEY optimization

2024-02-22 Thread Matthias van de Meent
On Tue, 5 Dec 2023 at 08:43, Heikki Linnakangas wrote: > > On 01/11/2023 00:08, Matthias van de Meent wrote: > > By caching the right separator index tuple in _bt_search, we can > > compare the downlink's right separator and the HIKEY, and when they > > are equal (memc

Re: Reducing output size of nodeToString

2024-02-22 Thread Matthias van de Meent
On Mon, 19 Feb 2024 at 14:19, Matthias van de Meent wrote: > Attached the updated version of the patch on top of 5497daf3, which > incorporates this last round of feedback. Now attached rebased on top of 93db6cbd to fix conflicts with fbc93b8b and an issue in the previous patchset: I at

Re: Experiments with Postgres and SSL

2024-02-21 Thread Matthias van de Meent
in ProcessStartupPacket. > ALPN Does the TLS ALPN spec allow protocol versions in the protocol tag? It would be very useful to detect clients with new capabilities at the first connection, rather than having to wait for one round trip, and would allow one avenue for changing the protocol version. Apart from this, I didn't really find any serious problems in the sum of these patches. The intermediate states were not great though, with various broken states in between. Kind regards, Matthias van de Meent 001_negotiate_encryption.pl Description: Binary data

Re: Revisiting {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2024-02-21 Thread Matthias van de Meent
is how it works right now? Would it be exclusively for that relation? How would this be integrated with e.g. heap_page_prune_opt? Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Revisiting {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2024-02-21 Thread Matthias van de Meent
TID can (and probably will) take O(indexsize) when the tuple is not in the index, which is one reason for ambulkdelete() to exist. Kind regards, Matthias van de Meent

Re: Add bump memory context type and use it for tuplesorts

2024-02-20 Thread Matthias van de Meent
On Tue, 20 Feb 2024 at 11:02, David Rowley wrote: > On Fri, 26 Jan 2024 at 01:29, Matthias van de Meent > wrote: > > >> +allocSize = MAXALIGN(sizeof(BumpContext)) + Bump_BLOCKHDRSZ + > > >> +if (minContextSize != 0) > > >> +

Re: Add bump memory context type and use it for tuplesorts

2024-02-20 Thread Matthias van de Meent
On Tue, 20 Feb 2024 at 10:41, David Rowley wrote: > On Tue, 7 Nov 2023 at 07:55, Matthias van de Meent > wrote: > > > +++ b/src/backend/utils/mmgr/bump.c > > > +BumpBlockIsEmpty(BumpBlock *block) > > > +{ > > > +/* it's empty if the freeptr has not

Re: Proposal: Adjacent B-Tree index

2024-02-19 Thread Matthias van de Meent
y with the same id was removed: The entry could've been removed because (e.g.) test's b column was updated thus inserting a new index entry for the new HOT-chain's TID. > would suffice for this new semantics. With the provided explanation I don't think this is a great idea. Kind regards, Matthias van de Meent.

Re: Experiments with Postgres and SSL

2024-02-19 Thread Matthias van de Meent
at can already limit access to specific ports very efficiently without causing undue load on the database server. Matthias van de Meent Neon (https://neon.tech)

Re: PGC_SIGHUP shared_buffers?

2024-02-18 Thread Matthias van de Meent
On Sun, 18 Feb 2024 at 02:03, Andres Freund wrote: > > Hi, > > On 2024-02-17 23:40:51 +0100, Matthias van de Meent wrote: > > > 5. Re-map the shared_buffers when needed. > > > > > > Between transactions, a backend should not hold any buffer pins. When

Re: automating RangeTblEntry node support

2024-02-17 Thread Matthias van de Meent
serializer would have zero impact on size once the default omission patch is committed. An additional patch with a single function that for this purpose clears junk fields from RTEs that changed kind would be appreciated: it is often hand-coded at those locations the kind changes, but that's more sensitive to programmer error. Kind regards, Matthias van de Meent

Re: PGC_SIGHUP shared_buffers?

2024-02-17 Thread Matthias van de Meent
a different address. This can quite realistically fail to find an unused memory region of sufficient size when the heap is sufficiently fragmented, e.g. through ASLR, which would make it difficult to use this dynamic single-allocation shared_buffers in security-hardened environments. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Revisiting {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2024-02-17 Thread Matthias van de Meent
ility snapshot was taken, but before the current visibility snapshot was taken, and thus definitely included in the changelog index. Because we hold a snapshot, no data in the table that we should see is removed, so we don't have a chance of broken HOT chains. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Reducing output size of nodeToString

2024-02-15 Thread Matthias van de Meent
in this patch set uses this. I think this could all be > much simpler, like (omitting required punctuation) [...] > Not only is this simpler, but it might also have better performance, > because we don't have separate pg_strtok_next() and pg_strtok() calls in > sequence. Good points. I'l

Re: Reducing output size of nodeToString

2024-02-12 Thread Matthias van de Meent
On Mon, 12 Feb 2024 at 20:32, Matthias van de Meent wrote: > > On Mon, 12 Feb 2024 at 19:03, Matthias van de Meent > wrote: > > Attached is patchset v2, which contains the improvements from these patches: > > Attached v3, which fixes an out-of-bounds read in pg_strtoken_nex

Re: Reducing output size of nodeToString

2024-02-12 Thread Matthias van de Meent
On Mon, 12 Feb 2024 at 19:03, Matthias van de Meent wrote: > Attached is patchset v2, which contains the improvements from these patches: Attached v3, which fixes an out-of-bounds read in pg_strtoken_next, detected by asan, that was a likely cause of the problems in CFBot's FreeBSD regress

Re: Reducing output size of nodeToString

2024-02-12 Thread Matthias van de Meent
On Wed, 31 Jan 2024 at 18:47, Robert Haas wrote: > > On Wed, Jan 31, 2024 at 11:17 AM Matthias van de Meent > wrote: > > I was also thinking about smaller per-attribute expression storage, for > > index attribute expressions, table default expressions, and functions. >

Re: Reducing output size of nodeToString

2024-01-31 Thread Matthias van de Meent
On Wed, 31 Jan 2024, 09:16 Peter Eisentraut, wrote: > On 30.01.24 12:26, Matthias van de Meent wrote: > >> Most of the other defaults I'm doubtful about. First, we are colliding > >> here between the goals of minimizing the storage size and making the > >> debug

Re: Reducing output size of nodeToString

2024-01-30 Thread Matthias van de Meent
On Tue, 9 Jan 2024, 09:23 Peter Eisentraut, wrote: > > On 04.01.24 00:23, Matthias van de Meent wrote: > > Something like the attached? It splits out into the following > > 0001: basic 'omit default values' > > /* Write an integer field (anything written as &quo

Re: Add bump memory context type and use it for tuplesorts

2024-01-25 Thread Matthias van de Meent
On Mon, 6 Nov 2023 at 19:54, Matthias van de Meent wrote: > > On Tue, 11 Jul 2023 at 01:51, David Rowley wrote: >> >> On Tue, 27 Jun 2023 at 21:19, David Rowley wrote: >>> I've attached the bump allocator patch and also the script I used to >>> gather the per

Re: btree: implement dynamic prefix truncation (was: Improving btree performance through specializing by key shape, take 2)

2024-01-24 Thread Matthias van de Meent
On Fri, 19 Jan 2024 at 05:55, Dilip Kumar wrote: > > On Wed, Nov 1, 2023 at 2:42 AM Matthias van de Meent > wrote: > > > > Hi, > > > > Currently, nbtree code compares each and every column of an index > > tuple during the binary search on the index page

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2024-01-22 Thread Matthias van de Meent
On Fri, 19 Jan 2024 at 23:42, Peter Geoghegan wrote: > Thank you for your replies so far. > On Thu, Jan 18, 2024 at 11:39 AM Matthias van de Meent > wrote: > > I would agree with you if this was about new APIs and features, but > > here existing APIs are being repurposed

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2024-01-18 Thread Matthias van de Meent
On Tue, 16 Jan 2024 at 03:03, Peter Geoghegan wrote: > > On Mon, Jan 15, 2024 at 2:32 PM Matthias van de Meent > wrote: > > Can you pull these planner changes into their own commit(s)? > > As mentioned upthread, it's a significant change in behavior that > > should

Re: Increasing IndexTupleData.t_info from uint16 to uint32

2024-01-18 Thread Matthias van de Meent
rsion). Next, the only non-internal use of IndexTuple is in IndexOnlyScans. However, here the index may fill the scandesc->xs_hitup with a heap tuple instead, which has a length stored in uint32, too. So, I don't quite see why this would be required for all indexes. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Sequence Access Methods, round two

2024-01-18 Thread Matthias van de Meent
w a sequence to be backed by an Linear Congruential Generator directly, rather than the implementation of our own transactional random_sequence table. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: cleanup patches for incremental backup

2024-01-18 Thread Matthias van de Meent
On Wed, 17 Jan 2024 at 21:10, Robert Haas wrote: > > On Wed, Jan 17, 2024 at 1:42 PM Matthias van de Meent > wrote: > > Sure, added in attached. > > I think this mostly looks good now but I just realized that I think > this needs rephrasing: > > + To restor

Re: cleanup patches for incremental backup

2024-01-17 Thread Matthias van de Meent
On Tue, 16 Jan 2024 at 21:49, Robert Haas wrote: > > On Tue, Jan 16, 2024 at 3:22 PM Matthias van de Meent > wrote: > + A special base > backup > + that for some WAL-logged relations only contains the pages that were > + modified since a previous backup, a

Re: cleanup patches for incremental backup

2024-01-16 Thread Matthias van de Meent
On Tue, 16 Jan 2024 at 16:39, Robert Haas wrote: > > On Mon, Jan 15, 2024 at 3:31 PM Matthias van de Meent > wrote: > > Off-list I was notified that the new WAL summarizer process was not > > yet added to the glossary, so PFA a patch that does that. > > In passing

Re: cleanup patches for incremental backup

2024-01-15 Thread Matthias van de Meent
> Thanks, committed. Off-list I was notified that the new WAL summarizer process was not yet added to the glossary, so PFA a patch that does that. In passing, it also adds "incremental backup" to the glossary, and updates the documented types of backends in monitoring.sgml with the new back

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2024-01-15 Thread Matthias van de Meent
we move the scan backwards we think that the top-level > scan > +-- should terminate, when in reality it should jump backwards to the leaf > page > +-- that we last visited. I notice this adds a complex test case that outputs many rows. Can we do with less rows if we build the index after data insertion, and with a lower (non-default) fillfactor? Note: I did not yet do any in-depth review of the planner changes in indxpath.c/selfuncs.c. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: the s_lock_stuck on perform_spin_delay

2024-01-10 Thread Matthias van de Meent
y spinlocks, and miscadmin.h already holds things related to "System interrupt and critical section handling", which seems quite related. Kind regards, Matthias van de Meent

Re: Revisiting {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2024-01-04 Thread Matthias van de Meent
hase of R/CIC. However, in a worst case it could still require another O(tablesize) of storage. Kind regards, Matthias van de Meent Neon (https://neon.tech) [0] https://www.postgresql.org/message-id/caakru_a+g2oe6ahjcbibftnfiy2aib4e31x9qyj_qkjxzmz...@mail.gmail.com

Re: the s_lock_stuck on perform_spin_delay

2024-01-04 Thread Matthias van de Meent
ose operations - although I consider their existence unlikely I can't rule them out as I've yet to go through all lock-touching code). This would probably work in a similar manner as START_CRIT_SECTION/END_CRIT_SECTION. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Reducing output size of nodeToString

2024-01-03 Thread Matthias van de Meent
On Tue, 2 Jan 2024 at 11:30, Peter Eisentraut wrote: > > On 06.12.23 22:08, Matthias van de Meent wrote: > > PFA a patch that reduces the output size of nodeToString by 50%+ in > > most cases (measured on pg_rewrite), which on my system reduces the > > total size of pg_

Re: Reducing output size of nodeToString

2024-01-03 Thread Matthias van de Meent
On Wed, 3 Jan 2024 at 03:02, David Rowley wrote: > > On Thu, 14 Dec 2023 at 19:21, Matthias van de Meent > wrote: > > > > On Thu, 7 Dec 2023 at 13:09, David Rowley wrote: > > > We could also easily serialize plans to binary format for copying to > > >

Re: Next step towards 64bit XIDs: Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid

2023-12-29 Thread Matthias van de Meent
an 2^31 concurrently created transactions, so we could well add a fxid to each WAL segment header (and checkpoint record?) and calculate the fxid of each record as a relative fxid off of that. Kind regards Matthias van de Meent [0] https://commitfest.postgresql.org/46/4386/

Re: Revisiting {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2023-12-20 Thread Matthias van de Meent
e pruning, afaik - no rules that I know of which are against that - but it would create issues where normal snapshot visibility rules would indicate it visible to both snapshots regardless of whether it actually was visible to the older snapshot when that snapshot was created... Either way, &quo

Re: pg_waldump

2023-12-19 Thread Matthias van de Meent
L, as VACUUM FULL on the catalog tables would change the file numbering of catalog tables... Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Add --check option to pgindent

2023-12-18 Thread Matthias van de Meent
k top right corner, which uses the community account) and then just go on to your prefered page, click edit, and do your modifications. Don't forget to save the modifications - I'm not sure whether the wiki editor's state is locally persisted. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Revisiting {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2023-12-17 Thread Matthias van de Meent
ssue may be caused by the new locking pattern - we > will be required to wait for all transaction started before the ending > of the phase to exit. What do you mean by "new locking pattern"? We already keep an ShareUpdateExclusiveLock on every heap table we're accessing during R/CIC, and that should already prevent any concurrent VACUUM operations, right? Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Revisiting {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2023-12-15 Thread Matthias van de Meent
e time C/RIC is the one backend holding back cleanup of old tuples. Kind regards, Matthias van de Meent Neon (https://neon.tech) > [1]: https://postgr.es/m/20210115133858.GA18931@alvherre.pgsql > [2]: https://postgr.es/m/17485-396609c6925b982d%40postgresql.org > > >

Re: Reducing output size of nodeToString

2023-12-13 Thread Matthias van de Meent
On Thu, 7 Dec 2023 at 13:09, David Rowley wrote: > > On Thu, 7 Dec 2023 at 10:09, Matthias van de Meent > wrote: > > PFA a patch that reduces the output size of nodeToString by 50%+ in > > most cases (measured on pg_rewrite), which on my system reduces the > > tot

Re: initdb caching during tests

2023-12-07 Thread Matthias van de Meent
On Thu, 7 Dec 2023 at 15:06, Daniel Gustafsson wrote: > > > On 7 Dec 2023, at 14:50, Matthias van de Meent > > wrote: > > > Attached a patch that fixes this for both make and meson, by adding > > --no-clean to the initdb template. > > Makes sense. While

Re: initdb caching during tests

2023-12-07 Thread Matthias van de Meent
t; Cool. Pushed that way. Only change I made is to redirect the output of cp > (and/or robocopy) in pg_regress, similar to how that was done for initdb > proper. While working on some things that are prone to breaking initdb, I noticed that this template isn't generated with --no-clean, while pg

Re: Reducing output size of nodeToString

2023-12-07 Thread Matthias van de Meent
On Thu, 7 Dec 2023 at 11:26, Peter Eisentraut wrote: > > On 06.12.23 22:08, Matthias van de Meent wrote: > > PFA a patch that reduces the output size of nodeToString by 50%+ in > > most cases (measured on pg_rewrite), which on my system reduces the > > total size of pg_

  1   2   3   4   5   6   >