Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread Matthias van de Meent
On Mon, 2 May 2022 at 16:09, Pavel Stehule wrote: > > > > po 2. 5. 2022 v 16:02 odesílatel Pavel Stehule > napsal: >> there is just shared buffers changed to 32GB and work_mem to 70MB. >> Unfortunately - it is in production environment with customer data, so I >> cannot to play too much >> >>

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread Matthias van de Meent
On Mon, 2 May 2022 at 11:00, Pavel Stehule wrote: > > Hi > > I found a query that is significantly slower with more memory Which PostgreSQL version did you use? Did you enable assert checking? Do you have an example database setup to work with? > plan 2 > QUERY PLAN > >

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-23 Thread Matthias van de Meent
On Sat, 23 Apr 2022 at 00:51, David Christensen wrote: > > Hi -hackers, > > Enclosed is a patch to allow extraction/saving of FPI from the WAL > stream via pg_waldump. > > Description from the commit: > > Extracts full-page images from the WAL stream into a target directory, > which must be empty

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-04-18 Thread Matthias van de Meent
Seeing that the busiest time for PG15 - the last commitfest before the feature freeze - has passed, could someone take another look at this? The changes that were requested by Heikki and Andres have been merged into patch v3, and I think it would be nice to fix this security issue in the upcoming

Re: Temporary file access API

2022-04-12 Thread Matthias van de Meent
On Mon, 11 Apr 2022 at 10:05, Antonin Houska wrote: > > Robert Haas wrote: > > > On Fri, Apr 8, 2022 at 4:54 AM Antonin Houska wrote: > > > Do you think that the use of a system call is a problem itself (e.g. > > > because > > > the code looks less simple if read/write is used somewhere and >

Re: API stability

2022-04-11 Thread Matthias van de Meent
rted by you as a reply on the original thread, but with the [was: pgsql: ...]-tag dropped and thus considered a new thread). So, this might be the reason Robert overlooked your declaration to volunteer: he was looking for volunteers in the thread "Re: API Stability [was: pgsql: ...]" in the Gmail UI, which didn't show your messages there because of the different subject line. Kind regards, Matthias van de Meent

Re: Improving btree performance through specializing by key shape, take 2

2022-04-10 Thread Matthias van de Meent
On Sun, 10 Apr 2022 at 23:45, Peter Geoghegan wrote: > > On Fri, Apr 8, 2022 at 9:55 AM Matthias van de Meent > wrote: > > Here's generation 2 of this effort. Instead of proceeding to trying to > > shoehorn all types of btree keys into one common code path, this >

Re: Size of pg_rewrite

2022-04-08 Thread Matthias van de Meent
On Fri, 8 Apr 2022 at 17:20, Dagfinn Ilmari Mannsåker wrote: > > Matthias van de Meent writes: > > > But, as text literal concatenations don't seem to get constant folded > > before storing them in the rules table, this rewrite of the views > > would result in long li

Size of pg_rewrite (Was: Report checkpoint progress with pg_stat_progress_checkpoint)

2022-04-08 Thread Matthias van de Meent
On Sat, 19 Mar 2022 at 01:15, Andres Freund wrote: > pg_rewrite without pg_stat_progress_checkpoint: 745472, with: 753664 > > pg_rewrite is the second biggest relation in an empty database already... Yeah, that's not great. Thanks for nerd-sniping me into looking into how views and pg_rewrite

Re: Lowering the ever-growing heap->pd_lower

2022-04-08 Thread Matthias van de Meent
On Fri, 8 Apr 2022 at 01:01, Andres Freund wrote: > > Hi, > > On 2022-04-04 19:24:22 -0700, Peter Geoghegan wrote: > > We should definitely increase MaxHeapTuplesPerPage before too long, > > for a variety of reasons that I have talked about in the past. Its > > current value is 291 on all

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-04-07 Thread Matthias van de Meent
On Thu, 7 Apr 2022 at 21:11, Robert Haas wrote: > > On Thu, Apr 7, 2022 at 2:43 PM Peter Geoghegan wrote: > > But if we were in a green-field situation we'd probably not want to > > use up several bytes for a nonse anyway. You said so yourself. > > I don't know what statement of mine you're

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-04-07 Thread Matthias van de Meent
On Tue, 5 Apr 2022 at 21:45, Robert Haas wrote: > > On Fri, Apr 1, 2022 at 11:12 AM Matthias van de Meent > wrote: > > Here's a new 0001 to keep CFBot happy. > > This seems like it would conflict with the proposal from > http://postgr.es/m/ca+tgmoad8wmn6i1mmuo+4zn

Re: How to generate a WAL record spanning multiple WAL files?

2022-04-05 Thread Matthias van de Meent
On Tue, 5 Apr 2022 at 15:13, Bharath Rupireddy wrote: > > Hi, > > I wanted to have a WAL record spanning multiple WAL files of size, say > 16MB. I'm wondering if the Full Page Images (FPIs) of a TOAST table > would help here. Please let me know if there's any way to generate > such large WAL

Re: Restructure ALTER TABLE notes to clarify table rewrites and verification scans

2022-04-01 Thread Matthias van de Meent
On Fri, 1 Apr 2022 at 16:10, James Coleman wrote: > > On Thu, Mar 31, 2022 at 10:58 AM Matthias van de Meent > wrote: > > > > On Tue, 29 Mar 2022 at 16:20, James Coleman wrote: > > > > > > Over in the "Document atthasmissing default optimization

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-04-01 Thread Matthias van de Meent
On Fri, 1 Apr 2022 at 10:50, Matthias van de Meent wrote: > > On Fri, 1 Apr 2022 at 07:38, Michael Paquier wrote: > > > > On Thu, Mar 31, 2022 at 12:09:35PM +0200, Matthias van de Meent wrote: > > > PageInit MAXALIGNs the size of the special area that it receives a

Re: Is monotonous xid8 is a right way to do?

2022-04-01 Thread Matthias van de Meent
at a rate higher than 5B /sec (and not logged at that rate) we can start considering this to be a problem. A different and more important issue (IMO) is that the xlog record header currently only supports 32-bit xids -- long-running transactions can reasonably see a xid4 wraparound in their lifetime. Enjoy, Matthias van de Meent

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-04-01 Thread Matthias van de Meent
On Fri, 1 Apr 2022 at 07:38, Michael Paquier wrote: > > On Thu, Mar 31, 2022 at 12:09:35PM +0200, Matthias van de Meent wrote: > > PageInit MAXALIGNs the size of the special area that it receives as an > > argument; so any changes to the page header that would misalign the >

Re: Restructure ALTER TABLE notes to clarify table rewrites and verification scans

2022-03-31 Thread Matthias van de Meent
onstraints already imply the new partition constraints. The previous wording is better in that regard ("may need", instead of "requires"), though it could be improved by refering to the sql-altertable-attach-partition section. Kind regards, Matthias van de Meent

Re: Commitfest Update

2022-03-31 Thread Matthias van de Meent
On Thu, 31 Mar 2022 at 12:56, Alvaro Herrera wrote: > > On 2022-Mar-31, Julien Rouhaud wrote: > > > On Wed, Mar 30, 2022 at 02:41:26PM -0400, Greg Stark wrote: > > > > > > Patches that are Waiting on Author and haven't had activity in months > > > -- traditionally they were set to Returned with

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-03-31 Thread Matthias van de Meent
On Thu, 31 Mar 2022 at 09:32, Michael Paquier wrote: > > On Mon, Mar 28, 2022 at 05:09:10PM +0200, Matthias van de Meent wrote: > > Not all clusters have checksums enabled (boo!, but we can't > > realistically fix that), so on-disk corruption could reasonably > &g

Re: CREATE TABLE ( .. STORAGE ..)

2022-03-29 Thread Matthias van de Meent
On Wed, 2 Feb 2022 at 11:13, Teodor Sigaev wrote: > > Hi! > > > Are they both set to name or ColId? Although they are the same. > > > > Thank you, fixed, that was just an oversight. > > > 2 For ColumnDef new member storage_name, did you miss the function > > _copyColumnDef() _equalColumnDef()?

Re: Correct docs re: rewriting indexes when table rewrite is skipped

2022-03-29 Thread Matthias van de Meent
ly warns for index rebuilds, even when the letter of the docs is incorrect: there are indeed cases we don't need to rebuild the indexes; but that would require more elaboration. Kind regards, Matthias van de Meent

Re: TRAP: FailedAssertion("HaveRegisteredOrActiveSnapshot()", File: "toast_internals.c", Line: 670, PID: 19403)

2022-03-29 Thread Matthias van de Meent
On Tue, 29 Mar 2022 at 11:10, Kyotaro Horiguchi wrote: > > At Tue, 29 Mar 2022 17:06:21 +0900 (JST), Kyotaro Horiguchi > wrote in > > At Mon, 28 Mar 2022 18:36:46 +0900 (JST), Kyotaro Horiguchi > > wrote in > > Then, I found that portal->holdSnapshot is that. I came up with the > > attached.

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-03-28 Thread Matthias van de Meent
On Mon, 28 Mar 2022 at 06:33, Michael Paquier wrote: > > On Wed, Feb 16, 2022 at 10:24:58PM +0100, Matthias van de Meent wrote: > > A first good reason to do this is preventing further damage when a > > page is corrupted: if I can somehow overwrite pd_special, > > non-ass

Re: Assert in pageinspect with NULL pages

2022-03-27 Thread Matthias van de Meent
special area. As such, you might be interested in that patch. Note that currently in core postgres a corrupted special area pointer will likely target neighbouring blocks in the buffer pool; resulting in spreading corruption when the special area is updated. This spreading corruption should be lim

Re: Invalid comment in ParallelQueryMain

2022-03-26 Thread Matthias van de Meent
On Sat, 26 Mar 2022 at 17:01, Julien Rouhaud wrote: > > Hi, > > While reading code around I just noticed that I failed to adapt a comment a > couple of lines above a removed line in 0f61727b75b9. Patch attached. +1, seems OK to me.

Re: Hardening heap pruning code (was: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum)

2022-03-20 Thread Matthias van de Meent
On Sun, 20 Mar 2022 at 04:48, Peter Geoghegan wrote: > > Attached is v6, which goes a bit further than v5 in using local state > that we build up-front to describe the state of the page being pruned > (rather than rereading the page itself). I didn't test the code; so these comments are my

Re: Probable CF bot degradation

2022-03-20 Thread Matthias van de Meent
ommits on master won't impact CFbot results as easy? Kind regards, Matthias van de Meent

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-03-15 Thread Matthias van de Meent
Apart from registering this in CF 2022-07 last Friday, I've also just added this issue to the Open Items list for PG15 under "Older bugs affecting stable branches"; as a precaution to not lose track of this issue in the buzz of the upcoming feature freeze. -Matthias

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-03-15 Thread Matthias van de Meent
hich would > allow us to e.g. get away from needing Heap2. Which would aestethically be > pleasing. That would be interesting; though out of scope for this bug I'm trying to fix. > On 2022-03-14 17:57:23 +0100, Matthias van de Meent wrote: > > I'm not sure whether or not to inclu

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-03-14 Thread Matthias van de Meent
Thank you all for the feedback. Please find attached v2 of the patchset, which contains updated comments and applies the suggested changes. On Sat, 12 Mar 2022 at 02:03, Andres Freund wrote: > > Hi, > > On 2022-03-11 22:42:42 +0200, Heikki Linnakangas wrote: > > Have you been able to create a

Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-03-11 Thread Matthias van de Meent
t.c. In this patch, I've also added a comment to the XLogRecord spec to document that xl_tot_len should not be larger than 1GB - 1B; and why that limit exists. Kind regards, Matthias van de Meent From a72121b8fccbbf8c289e71a2c76801a1004f5353 Mon Sep 17 00:00:00 2001 From: Matthias van de Meent Date: Fri, 11

Re: Lowering the ever-growing heap->pd_lower

2022-03-10 Thread Matthias van de Meent
On Wed, 16 Feb 2022 at 21:14, Matthias van de Meent wrote: > > On Wed, 16 Feb 2022 at 20:54, Peter Geoghegan wrote: > > > > On Tue, Feb 15, 2022 at 10:48 AM Matthias van de Meent > > wrote: > > > Peter Geoghegan asked for good arguments for the two change

Re: New Table Access Methods for Multi and Single Inserts

2022-03-07 Thread Matthias van de Meent
On Sun, 6 Mar 2022 at 12:12, Bharath Rupireddy wrote: > > On Fri, Mar 4, 2022 at 8:07 PM Matthias van de Meent > wrote: > > > Another rebase due to conflicts in 0003. Attaching v6 for review. > > > > I recently touched the topic of multi_insert, and I remembere

Re: New Table Access Methods for Multi and Single Inserts

2022-03-04 Thread Matthias van de Meent
On Mon, 19 Apr 2021 at 06:52, Bharath Rupireddy wrote: > > On Mon, Apr 5, 2021 at 9:49 AM Bharath Rupireddy > wrote: > > > > On Wed, Mar 10, 2021 at 10:21 AM Bharath Rupireddy > > wrote: > > > Attaching the v4 patch set. Please review it further. > > > > Attaching v5 patch set after rebasing

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-28 Thread Matthias van de Meent
On Sun, 27 Feb 2022 at 16:14, Bharath Rupireddy wrote: > 3) Why do we need this extra calculation for start_lsn? > Do you ever see a negative LSN or something here? > +('0/0'::pg_lsn + ( > +CASE > +WHEN (s.param3 < 0) THEN pow((2)::numeric, (64)::numeric) > +

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-25 Thread Matthias van de Meent
On Fri, 25 Feb 2022 at 17:35, Julien Rouhaud wrote: > > On Fri, Feb 25, 2022 at 08:53:50PM +0530, Nitin Jadhav wrote: > > > > > > I'm not sure what Matthias meant, but as far as I know there's no > > > fundamental > > > difference between checkpoint with and without the CHECKPOINT_IMMEDIATE > >

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-24 Thread Matthias van de Meent
erally, enum-like values in a stat_progress field are 1-indexed, to differentiate between empty/uninitialized (0) and states that have been set by the progress reporting infrastructure. Kind regards, Matthias van de Meent

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-23 Thread Matthias van de Meent
On Wed, 23 Feb 2022 at 15:24, Nitin Jadhav wrote: > > > At least for pg_stat_progress_checkpoint, storing only a timestamp in > > the pg_stat storage (instead of repeatedly updating the field as a > > duration) seems to provide much more precise measures of 'time > > elapsed' for other sessions

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-22 Thread Matthias van de Meent
On Tue, 22 Feb 2022 at 07:39, Nitin Jadhav wrote: > > > > Thank you for sharing the information. 'triggering backend PID' (int) > > > - can be stored without any problem. 'checkpoint or restartpoint?' > > > (boolean) - can be stored as a integer value like > > >

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-17 Thread Matthias van de Meent
case. As assigning uint64 to > int64 type would be an issue for larger values and can lead to hidden > bugs. Not necessarily - we can (without much trouble) do a bitwise cast from uint64 to int64, and then (in SQL) cast it back to a pg_lsn [1]. Not very elegant, but it works quite well. Kind reg

Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-02-16 Thread Matthias van de Meent
opaques by providing a typed accessor macro similar to what is used in the GIN and (SP-)GIST index methods; improving the legibility of the code and decreasing the churn. Kind regards, Matthias van de Meent. v1-0001-Add-known-size-pre-aligned-special-area-pointer-m.patch Description: Binary data

Re: Lowering the ever-growing heap->pd_lower

2022-02-16 Thread Matthias van de Meent
On Wed, 16 Feb 2022 at 20:54, Peter Geoghegan wrote: > > On Tue, Feb 15, 2022 at 10:48 AM Matthias van de Meent > wrote: > > Peter Geoghegan asked for good arguments for the two changes > > implemented. Below are my arguments detailed, with adversarial loads > >

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-16 Thread Matthias van de Meent
checkpoint operation is requested due to timeout. + The checkpoint was started due to the expiration of a >checkpoint_timeout< interval > + The checkpoint operation is forced even if no XLOG activity has > occurred > + since the last one. + Some operation forced a checkpoint. > + checkpointing CommitTs pages CommitTs -> Commit time stamp Thanks for working on this. Kind regards, Matthias van de Meent

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-16 Thread Matthias van de Meent
essing would be required at the view, but that's not impossible to overcome. Kind regards, Matthias van de Meent

Re: Lowering the ever-growing heap->pd_lower

2022-02-15 Thread Matthias van de Meent
adversarial loads that show the problematic behaviour of the line pointer array that is fixed with the patch. Kind regards, Matthias van de Meent Truncating lp_array to 0 line pointers === On 32-bit pre-patch systems the heap grows without limit; post-patch the rel

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-02-11 Thread Matthias van de Meent
to committable? Kind regards, Matthias van de Meent

Re: generalized conveyor belt storage

2021-12-15 Thread Matthias van de Meent
ation. Is that too on the missing features list, or did I misunderstand what you meant with "relation fork"? Kind regards, Matthias van de Meent

Re: [PATCH] Partial foreign key updates in referential integrity triggers

2021-12-01 Thread Matthias van de Meent
d DDL operations now growing linearly with the amount of tenants in the system. Kind regards, Matthias van de Meent

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-10 Thread Matthias van de Meent
On Wed, 10 Nov 2021 at 11:51, Amit Kapila wrote: > > On Fri, Nov 5, 2021 at 8:16 PM Matthias van de Meent > wrote: > > AFAICU, in the thread referred by you, it seems that the main reported > issue will be resolved by this patch but there is a discussion about > xmin moving

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-05 Thread Matthias van de Meent
, and it correctly fixes the issue of blocking xmin from advancing, and also fixes an issue of retreating the observed *_oldest_nonremovable in XidHorizons through parallel workers. There are still some other soundness issues with xmin handling (see [0]), but that should not prevent this patch from landing in the relevant branches. Kind regards, Matthias van de Meent [0] https://www.postgresql.org/message-id/flat/17257-1e46de26bec11433%40postgresql.org

Re: Next Steps with Hash Indexes

2021-10-27 Thread Matthias van de Meent
happens all the time, I do think it can happen often enough on e.g. an HTAP system to make it a noteworthy test case. Kind regards, Matthias van de Meent

Re: Adding CI to our tree

2021-10-21 Thread Matthias van de Meent
ect to the addition of these config files, but I would appreciate it if a clear policy could be provided on the inclusion of configurations for external tools that are not expected to be used by all users of the repository, such as CI, editors and IDEs. Kind regards, Matthias van de Meent

Re: RFC: compression dictionaries for JSONB

2021-10-13 Thread Matthias van de Meent
On Wed, 13 Oct 2021 at 11:48, Aleksander Alekseev wrote: > > Hi Matthias, > > > Assuming this above is option 1. If I understand correctly, this > > option was 'adapt the data type so that it understands how to handle a > > shared dictionary, decreasing storage requirements'. > > [...] > >

Re: RFC: compression dictionaries for JSONB

2021-10-11 Thread Matthias van de Meent
that an 'universal dictionary encoder' would be useful, but that a data type might also have good reason to implement their replacement methods by themselves for better overall performance (such as maintaining partial detoast support in dictionaried items, or overall lower memory footprint, or ...). As such, I'd really appreciate it if Option 1 is not ruled out by any implementation of Option 2. Kind regards, Matthias van de Meent

Re: RFC: compression dictionaries for JSONB

2021-10-09 Thread Matthias van de Meent
On Fri, 8 Oct 2021 at 21:21, Alvaro Herrera wrote: > > On 2021-Oct-08, Matthias van de Meent wrote: > > > That's a good point, but if we're extending this syntax to allow the > > ability of including other types, then I'd instead extend the syntax > > that

Re: RFC: compression dictionaries for JSONB

2021-10-08 Thread Matthias van de Meent
On Fri, 8 Oct 2021 at 17:19, Alvaro Herrera wrote: > > On 2021-Oct-08, Matthias van de Meent wrote: > > > On Fri, 8 Oct 2021 at 11:47, Aleksander Alekseev > > wrote: > > > > In order to do this, the SQL syntax should be modified. The proposed > > > synta

Re: RFC: compression dictionaries for JSONB

2021-10-08 Thread Matthias van de Meent
On Fri, 8 Oct 2021 at 11:47, Aleksander Alekseev wrote: > This is a follow-up thread to `Add ZSON extension to /contrib/` [1]. > The ZSON extension introduces a new type called ZSON, which is 100% > compatible with JSONB but uses a shared dictionary of strings most > frequently used by given

Re: Map WAL segment files on PMEM as WAL buffers

2021-10-07 Thread Matthias van de Meent
On Wed, 30 Jun 2021 at 06:53, Takashi Menjo wrote: > > Rebased. Thanks for these patches! I recently took a dive into the WAL subsystem, and got to this patchset through the previous ones linked from [0]. This patchset seems straightforward to understand, so thanks! I've looked over the

Re: CLUSTER on partitioned index

2021-09-23 Thread Matthias van de Meent
strpart; I believe that this cleanup should not be fully removed, but moved to before '-- Test CLUSTER with external tuplesorting', as the table is not used after that line. Kind regards, Matthias van de Meent

Re: 64 bit TID?

2021-09-13 Thread Matthias van de Meent
e used for rolled-back and thus dead tuples). > Are there any plans in the works to widen the TID? This was recently discussed here [0] as well, but to the best of my knowledge no material proposal to update the APIs has been suggested as of yet. Kind regards, Matthias van de Meent [0] htt

Re: PostgreSQL <-> Babelfish integration

2021-08-25 Thread Matthias van de Meent
][1][2] also do not appear to have seen an update. The last mention of babelfish in a thread here on -hackers also only seem to date back to late March. Kind regards, Matthias van de Meent [0] https://babelfish-for-postgresql.github.io/babelfish-for-postgresql/ [1] https://aws.amazon.com/rds/a

Re: NAMEDATALEN increase because of non-latin languages

2021-08-20 Thread Matthias van de Meent
On Thu, 19 Aug 2021 at 14:58, Andres Freund wrote: > > Hi, > > On 2021-08-19 14:47:42 +0200, Matthias van de Meent wrote: > > I tried to implement this 'compact attribute access descriptor' a few > > months ago in my effort to improve btree index performance. > > co

Re: NAMEDATALEN increase because of non-latin languages

2021-08-19 Thread Matthias van de Meent
ytes, whilst also being useable in both tuple forming and deforming. I hope this can is useful, otherwise sorry for the noise. Kind regards, Matthias van de Meent From 3dd3f470ab78b8811015c9f374d0bdc44ffef531 Mon Sep 17 00:00:00 2001 From: Matthias van de Meent Date: Wed, 23 Jun 2021 20:38:47 +02

Re: Lowering the ever-growing heap->pd_lower

2021-08-04 Thread Matthias van de Meent
I would argue 'yes, but isn't it better then to also actually fully mark that space as unused'. All in all, I'd just rather remove the distinction between once-used pages and fresh pages completely by truncating the LP-array to 0 than to leave this bloating behaviour in the system. Kind regards, Matthias van de Meent.

Re: Lowering the ever-growing heap->pd_lower

2021-08-04 Thread Matthias van de Meent
e than just this one vacuum run, but the chances on fully clearing the page are indeed very, very slim. Kind regards, Matthias van de Meent

Re: Lowering the ever-growing heap->pd_lower

2021-08-03 Thread Matthias van de Meent
On Tue, 3 Aug 2021 at 20:37, Simon Riggs wrote: > > On Tue, 3 Aug 2021 at 17:15, Matthias van de Meent > wrote: > > > and further future optimizations might include > > > > - Full-page WAL logging of empty pages produced in the checkpointer > > could potent

Re: Lowering the ever-growing heap->pd_lower

2021-08-03 Thread Matthias van de Meent
On Tue, 3 Aug 2021 at 08:57, Simon Riggs wrote: > > On Tue, 18 May 2021 at 20:33, Peter Geoghegan wrote: > > > > On Tue, May 18, 2021 at 12:29 PM Matthias van de Meent > > wrote: > > > PFA the updated version of this patch. Apart from add

Re: ATTACH PARTITION locking documentation for DEFAULT partitions

2021-07-27 Thread Matthias van de Meent
On Tue, 27 Jul 2021 at 08:02, David Rowley wrote:\> > On Tue, 13 Jul 2021 at 02:30, Matthias van de Meent > wrote: > > The algoritm as described in your patch implies that this recursive > > locking is conditional on _only_ the check-constraints of the topmost > > p

Re: Fix comments of heap_prune_chain()

2021-07-12 Thread Matthias van de Meent
is, I'd go for "unexpected result from heap_prune_satisfies_vacuum" as a message instead. Kind regards, Matthias van de Meent

Re: ATTACH PARTITION locking documentation for DEFAULT partitions

2021-07-12 Thread Matthias van de Meent
On Mon, 12 Jul 2021 at 15:28, David Rowley wrote: > > On Tue, 13 Jul 2021 at 00:14, Matthias van de Meent > wrote: > > Sorry for the delay. I think that covers the basics of what I was > > missing in these docs, and although it does not cover the recursive > &g

Re: ATTACH PARTITION locking documentation for DEFAULT partitions

2021-07-12 Thread Matthias van de Meent
y for the delay. I think that covers the basics of what I was missing in these docs, and although it does not cover the recursive 'if the check is implied by constraints don't lock this partition', I'd say that your suggested patch is good enough. Thanks for looking over this. Kind regards, Matthias van de Meent

Re: [PoC] Improve dead tuple storage for lazy vacuum

2021-07-07 Thread Matthias van de Meent
heck what the (performance, memory) impact would be if these proposed structures were to support the maximum MaxHeapTuplesPerPage of 1169 or the full uint16-range of offset numbers that could be supported by our current TID struct? Kind regards, Matthias van de Meent [0] https://www.postgresql.org/message-id/fla

Re: Removing redundant check for transaction in progress in check_safe_enum_use

2021-07-06 Thread Matthias van de Meent
C snapshots (at least its documentation does not warn against non-MVCC snapshots). Kind regards, Matthias van de Meent

Re: Iterating on IndexTuple attributes and nbtree page-level dynamic prefix truncation

2021-06-24 Thread Matthias van de Meent
On Thu, 17 Jun 2021 at 17:14, Matthias van de Meent wrote: > > I'll try to > benchmark the patches in this patchset (both 0001, and 0001+0002) in > the upcoming weekend. Somewhat delayed, benchmark results are attached. These are based on 7 iterations of the attached benc

Re: Iterating on IndexTuple attributes and nbtree page-level dynamic prefix truncation

2021-06-17 Thread Matthias van de Meent
On Fri, 23 Apr 2021 at 12:45, Matthias van de Meent wrote: > > On Sat, 17 Apr 2021 at 01:05, Peter Geoghegan wrote: > > > It would be convenient if the first patch could be treated > > as an independent thing. > > Patch 0002 was the reason for writing 000

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Matthias van de Meent
On Wed, 16 Jun 2021 at 21:22, Andres Freund wrote: > > Hi, > > On 2021-06-16 09:46:07 -0700, Peter Geoghegan wrote: > > On Wed, Jun 16, 2021 at 9:03 AM Peter Geoghegan wrote: > > > On Wed, Jun 16, 2021 at 3:59 AM Matthias van de Meent > > > > So the

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Matthias van de Meent
On Wed, 16 Jun 2021 at 21:12, Andres Freund wrote: > > Hi, > > On 2021-06-16 12:59:33 +0200, Matthias van de Meent wrote: > > PFA my adapted patch that fixes this new-ish issue, and does not > > include the (incorrect) assertions in GlobalVisUpdateApply. I've

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Matthias van de Meent
lieve it correctly handles the case where the vistest non-removable horizon moved backwards, a second pair of eyes would be appreciated. With regards, Matthias van de Meent From f4b8de3c53b9f2ff5c8ac2907dafd328d2ec787a Mon Sep 17 00:00:00 2001 From: Matthias van de Meent Date: Wed

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-14 Thread Matthias van de Meent
On Thu, 10 Jun 2021 at 19:43, Peter Geoghegan wrote: > > On Thu, Jun 10, 2021 at 10:29 AM Matthias van de Meent > wrote: > > I see one exit for HEAPTUPLE_DEAD on a potentially recently committed > > xvac (?), and we might also check against recently committed > > t

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-10 Thread Matthias van de Meent
On Thu, 10 Jun 2021 at 19:07, Peter Geoghegan wrote: > > On Thu, Jun 10, 2021 at 9:57 AM Matthias van de Meent > wrote: > > > By "matches what we expect", I meant "involves a just-aborted > > > transaction". We could defensively verify that the i

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-10 Thread Matthias van de Meent
On Thu, 10 Jun 2021 at 18:03, Peter Geoghegan wrote: > > On Thu, Jun 10, 2021 at 8:49 AM Matthias van de Meent > wrote: > > Could you elaborate on what this "matches what we expect" entails? > > > > Apart from this, I'm also quite certain that the goto-branc

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-10 Thread Matthias van de Meent
also going to change code to update / move forward vacrel->OldestXmin in lazy_scan_prune between the HPSV call and the loop with HTSV. With regards, Matthias van de Meent

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-10 Thread Matthias van de Meent
e, or give me a notice if you want me to split the patch. > > Another approach might be changing GlobalVisTestFor(rel) instead to > > reflect the conditions in GetOldestNonRemovableTransactionId. > > No, that'd not be correct, afaict. Allright, I wasn't sure of that myself. With regards, Matthias van de Meent.

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-09 Thread Matthias van de Meent
destNonRemovableTransactionId. With attached prototype patch, I was unable to reproduce the problematic case in 10 minutes. Without, I got the problematic behaviour in seconds. With regards, Matthias From fe5cb0430a06a44823d5b62f2f909da624505962 Mon Sep 17 00:00:00 2001 From: Matthias van d

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Matthias van de Meent
On Tue, 8 Jun 2021 at 14:11, Justin Pryzby wrote: > > On Tue, Jun 08, 2021 at 01:54:41PM +0200, Matthias van de Meent wrote: > > On Tue, 8 Jun 2021 at 13:03, Justin Pryzby wrote: > > > > > > On Sun, Jun 06, 2021 at 11:00:38AM -0700, Peter Geoghegan wrote: > >

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Matthias van de Meent
currently stuck on lacking information in GlobalVisCatalogRels and the PruneState. One curiosity that I did notice is that the t_xmax of the problematic tuples has been exactly one lower than the OldestXmin. Not weird, but a curiosity. With regards, Matthias van de Meent. PS. Attached

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-06 Thread Matthias van de Meent
on that page (well, the tuple headers with their transactionids and their line pointers), as that would help with determining if my suspicion could be correct. With regards, Matthias van de Meent

Re: pg_stat_progress_create_index vs. parallel index builds

2021-06-02 Thread Matthias van de Meent
On Wed, 2 Jun 2021 at 17:42, Tomas Vondra wrote: > > On 6/2/21 4:54 PM, Matthias van de Meent wrote: > > On Wed, 2 Jun 2021 at 15:23, Tomas Vondra > > wrote: > >> > >> ... > >> > > > > After looking at it a bit more, it seems like

Re: pg_stat_progress_create_index vs. parallel index builds

2021-06-02 Thread Matthias van de Meent
On Wed, 2 Jun 2021 at 15:23, Tomas Vondra wrote: > > > > On 6/2/21 3:03 PM, Matthias van de Meent wrote: > > On Wed, 2 Jun 2021 at 13:57, Tomas Vondra > > wrote: > >> > >> Hi, > >> > >> While experimenting with parallel

Re: pg_stat_progress_create_index vs. parallel index builds

2021-06-02 Thread Matthias van de Meent
he leader. This might mean that the 'sorting' phase is already finished during the 'scanning' phase by waiting for the parallel workers; I haven't looked further if this is the case and whether it could be changed to also produce the sorting metrics, but seeing as it is part of the parallel workers API of tuplesort, I think fixing it in current releases is going to be difficult. With regards, Matthias van de Meent

Re: Add ZSON extension to /contrib/

2021-05-26 Thread Matthias van de Meent
ed for jsonb (yet). The point I tried to make was that introspection of GSON would be even more difficult due to it adding a non-standard compression method which makes introspection effectively impossible (the algorithm can replace things other than the strings it should replace, so it will be difficul

Re: Add ZSON extension to /contrib/

2021-05-25 Thread Matthias van de Meent
ar with the jsonb- and enum-code (which I suspect to be critical for an efficient implementation of such type), although whilst researching I've noticed that it is possible to use most of the JSONB infrastructure / read older jsonb values, as there are still some JEntry type masks available which could flag such substitutions. With regards, Matthias van de Meent

Re: Lowering the ever-growing heap->pd_lower

2021-05-18 Thread Matthias van de Meent
On Mon, 3 May 2021 at 16:39, Matthias van de Meent wrote: > I am planning on fixing this patch sometime > before the next commit fest so that we can truncate the LP array > during hot pruning as well, instead of only doing so in the 2nd VACUUM > pass. PFA the updated version of this

Re: RFC: Logging plan of the running query

2021-05-12 Thread Matthias van de Meent
(cost=0.42..42377.43 rows=100 width=89) > -> Index Only Scan using pgbench_accounts_pkey on > pgbench_accounts b (cost=0.42..25980.42 rows=100 width=4) > > > Attached a PoC patch. > > Any thoughts? Great idea. One feature I'd suggest would be adding a 'format' option as well, if such feature would be feasable. With regards, Matthias van de Meent

Re: PG 14 release notes, first draft

2021-05-10 Thread Matthias van de Meent
On Mon, 10 May 2021 at 19:34, Bruce Momjian wrote: > > On Mon, May 10, 2021 at 01:44:12PM +0200, Matthias van de Meent wrote: > > On Mon, 10 May 2021 at 08:03, Bruce Momjian wrote: > > > > > > I have committed the first draft of the PG 14 release notes. You can >

Re: PG 14 release notes, first draft

2021-05-10 Thread Matthias van de Meent
inserted on pages which have only a small amount of data, regardless of fillfactor. Together they should be able to help significantly in both bloat prevention and bloat reduction. > I plan to work on completing this document this coming week in > preparation for beta next week. Thanks!

Re: Draft back-branch release notes are up

2021-05-07 Thread Matthias van de Meent
; +Branch: master [fe06819f1] 2021-02-19 22:01:25 +0900 Thanks in advance, Matthias van de Meent

Re: MaxOffsetNumber for Table AMs

2021-05-06 Thread Matthias van de Meent
On Thu, 6 May 2021 at 01:22, Peter Geoghegan wrote: > > On Wed, May 5, 2021 at 3:18 PM Matthias van de Meent > wrote: > > I believe that the TID is the unique identifier of that tuple, within > > context. > > > > For normal indexes, the TID as supplied directly

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Matthias van de Meent
On Wed, 5 May 2021 at 22:09, Peter Geoghegan wrote: > > On Wed, May 5, 2021 at 12:43 PM Matthias van de Meent > wrote: > > I believe that it cannot be "just" an additive thing, at least not > > through a normal INCLUDEd column, as you'd get duplicate TIDs in

<    1   2   3   4   5   6   >