Re: Tracking last scan time

2022-08-31 Thread Matthias van de Meent
h I think is a waste of time (heh) even in VDSO-enabled systems. Apart from the above, I don't have any other meaningful opinion on this patch - it might be a good addition, but I don't consume stats often enough to make a good cost / benefit comparison. Kind regards, Matthias van de Meent

Re: effective_multixact_freeze_max_age issue

2022-08-29 Thread Matthias van de Meent
blem we're trying to solve. I understand that the use of < is pre-existing, but since we're touching this code shouldn't we try to get this new code up to current standards? Kind regards, Matthias van de Meent

Re: First draft of the PG 15 release notes

2022-08-27 Thread Matthias van de Meent
aut) > [...] > to match the operating system collation version. DETAILS? > > Kind regards, Matthias van de Meent

Re: Remaining case where reltuples can become distorted across multiple VACUUM operations

2022-08-11 Thread Matthias van de Meent
On Mon, 8 Aug 2022 at 18:48, Peter Geoghegan wrote: > > On Mon, Aug 8, 2022 at 9:17 AM Matthias van de Meent > wrote: > > Because if a subset of the pages of a relation contains more tuples > > than your current total expected tuples in the table, you should > &g

Re: Remaining case where reltuples can become distorted across multiple VACUUM operations

2022-08-08 Thread Matthias van de Meent
On Mon, 8 Aug 2022 at 17:49, Peter Geoghegan wrote: > > On Mon, Aug 8, 2022 at 8:33 AM Matthias van de Meent > wrote: > > For example, if currently the measured 2% of the pages contains more > > than 100% of the previous count of tuples, or with your patch the last > &

Re: Remaining case where reltuples can become distorted across multiple VACUUM operations

2022-08-08 Thread Matthias van de Meent
On Mon, 8 Aug 2022 at 17:26, Peter Geoghegan wrote: > > On Mon, Aug 8, 2022 at 8:14 AM Matthias van de Meent > wrote: > > I do not have intimate knowledge of this code, but shouldn't we also > > add some sefety guarantees like the following in these blocks?

Re: Clarifying Commitfest policies

2022-08-08 Thread Matthias van de Meent
On Thu, 4 Aug 2022 at 20:38, Jacob Champion wrote: > > On Wed, Aug 3, 2022 at 2:05 PM Matthias van de Meent > wrote: > > On Wed, 3 Aug 2022 at 20:04, Jacob Champion wrote: > > > Is that enough, or should we do more? > > > > "The CF Checklist" see

Re: Remaining case where reltuples can become distorted across multiple VACUUM operations

2022-08-08 Thread Matthias van de Meent
t the count is incorrect. if (scanned_tuples > old_rel_tuples) return some_weighted_scanned_tuples; Kind regards, Matthias van de Meent

Re: postgres_fdw: batch inserts vs. before row triggers

2022-08-03 Thread Matthias van de Meent
On Wed, 3 Aug 2022 at 23:57, Tom Lane wrote: > > Matthias van de Meent writes: > > I don't have a current version of the SQL spec, but one preliminary > > version of SQL:2012 I retrieved via the wiki details that all BEFORE > > triggers on INSERT/UPDATE/DELETE state

Re: postgres_fdw: batch inserts vs. before row triggers

2022-08-03 Thread Matthias van de Meent
ered on that statement, regardless of the FOR EACH ROW specifier. The sections for DELETE are 15.7 (6) and 15.7 (7); and for UPDATE 15.13(7) and 15.13(9) respectively. I don't know about the semantics of triggers in the latest SQL standard versions, but based on that sample it seems like we're non-compliant on BEFORE trigger behaviour, and it doesn't seem like it's documented in the trigger documentation. I seem to recall a mail on this topic (changes in trigger execution order with respect to the DML it is triggered by in the newest SQL spec) but I can't seem to find that thread. Kind regards, Matthias van de Meent

Re: Clarifying Commitfest policies

2022-08-03 Thread Matthias van de Meent
f the main PostgreSQL repository receives a decent amount of pull request traffic. When a project has a CONTRIBUTING.md -file at the top level people writing the pull request message will be pointed to those contributing guidelines. This could Thank you for raising this to a topical thread. Kind regards, Matthias van de Meent [0] https://www.postgresql.org/developer/

Re: Maximize page freezing

2022-07-29 Thread Matthias van de Meent
On Fri, 29 Jul 2022 at 16:38, Simon Riggs wrote: > > On Thu, 28 Jul 2022 at 14:55, Matthias van de Meent > wrote: > > Great idea, yet this patch seems to only freeze those tuples that are > > located after the first to-be-frozen tuple. It should probably > > re-v

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-28 Thread Matthias van de Meent
d. See bottom for examples (all 3 builds for different patches). Kind regards, Matthias van de Meent. [1] https://cirrus-ci.com/task/4960990331666432?logs=test_world#L2631-L2662 [2] https://cirrus-ci.com/task/5012678384025600?logs=test_world#L2631-L2662 [3] https://cirrus-ci.com/task/5147001137397760?logs=test_world#L2631-L2662

Re: ci: update to freebsd 13.1 / remove minor versions from image names

2022-07-28 Thread Matthias van de Meent
On Thu, 28 Jul 2022 at 19:31, Andres Freund wrote: > > Hi, > > On July 28, 2022 7:29:43 PM GMT+02:00, Matthias van de Meent > wrote: > >On Thu, 28 Jul 2022 at 11:57, Andres Freund wrote: > >> > >> Hi, > >> > >> Freebsd 13.0, so far used

Re: ci: update to freebsd 13.1 / remove minor versions from image names

2022-07-28 Thread Matthias van de Meent
esting() was not seen. # Looks like your test exited with 29 just after 1. t/033_replay_tsp_drops.pl .... Dubious, test returned 29 (wstat 7424, 0x1d00) All 1 subtests passed --- Kind regards, Matthias van de Meent [0] https://cirrus-ci.com/task/5147001137397760?logs=test_world#L2631-

Re: Maximize page freezing

2022-07-28 Thread Matthias van de Meent
; So +1 for this idea. > > Updated patch attached. Great idea, yet this patch seems to only freeze those tuples that are located after the first to-be-frozen tuple. It should probably re-visit earlier live tuples to potentially freeze those as well. Kind regards, Matthias van de Meent

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-07-27 Thread Matthias van de Meent
On Wed, 27 Jul 2022 at 11:09, Michael Paquier wrote: > > On Tue, Jul 26, 2022 at 06:58:02PM +0200, Matthias van de Meent wrote: > > - Retained the check in XLogRegisterData, so that we check against > > integer overflows in the registerdata code instead of only an assert > &g

Re: [PATCH] Compression dictionaries for JSONB

2022-07-27 Thread Matthias van de Meent
his dictionary compression can instead be added using the proposed extensible toasting mechanism at relatively low overhead. Kind regards, Matthias van de Meent

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-07-26 Thread Matthias van de Meent
ta, so that we check against integer overflows in the registerdata code instead of only an assert in XLogRecordAssemble where it might be too late. - Kept the inline static elog-ing function (as per Andres' suggestion on 2022-03-14; this decreases binary sizes) - Dropped any changes in xlogreader.h/c Kind regards, Matthias van de Meent v8-0001-Add-protections-in-xlog-record-APIs-against-large.patch Description: Binary data

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-07-25 Thread Matthias van de Meent
On Wed, 13 Jul 2022 at 07:54, Michael Paquier wrote: > > On Mon, Jul 11, 2022 at 02:26:46PM +0200, Matthias van de Meent wrote: > > Thanks for reviewing. > > I think that v6 is over-engineered because there should be no need to > add a check in xlogreader.c as long as the ori

Re: Pluggable toaster

2022-07-22 Thread Matthias van de Meent
ch. --- > Currently I'm working on an update to the default Toaster (some internal > optimizations, not affecting functionality) > and readme files explaining Pluggable TOAST. That would be greatly appreciated - 0006 does not cover why we need vtable, nor how it's expect

Re: StrategyAM for IndexAMs

2022-07-22 Thread Matthias van de Meent
27;t > clear). > > Comments welcome on this still WIP patch. I think this is a great step in the right direction, fixing one of the issues with core index AMs, issues I also complained about earlier [0]. Thanks, Matthias van de Meent [0] https://www.postgresql.org/message-id/CAEze2Wg8QhpOnHoqPNB-AaexGX4Zaij%3D4TT0kaMhF_6T5FXxmQ%40mail.gmail.com

Re: Make name optional in CREATE STATISTICS

2022-07-20 Thread Matthias van de Meent
On Wed, 13 Jul 2022 at 08:07, Simon Riggs wrote: > > On Thu, 7 Jul 2022 at 11:58, Matthias van de Meent > wrote: > > A more correct version would be > > > > + *CREATE STATISTICS [ [IF NOT EXISTS] stats_name ] > > [(stat types)] > > There you

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-07-15 Thread Matthias van de Meent
use that space for rmgr-specific flags, as opposed to stealing bits from xl_info? Kind regards, Matthias van de Meent

Re: MERGE and parsing with prepared statements

2022-07-15 Thread Matthias van de Meent
popped up earlier, and I think it's not something we've planned to fix in backbranches. Kind regards, Matthias van de Meent

Re: Commitfest Update

2022-07-11 Thread Matthias van de Meent
On Fri, 8 Jul 2022, 23:41 Jacob Champion, wrote: > > On 3/31/22 07:37, Tom Lane wrote: >> Robert Haas writes: >>> On Thu, Mar 31, 2022 at 10:11 AM Tom Lane wrote: ... Would it be feasible or reasonable to drop reviewers if they've not commented in the thread in X amount of time? >

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-07-11 Thread Matthias van de Meent
reviously, > > "postgres=# SELECT pg_logical_emit_message(false, long, long) FROM > repeat(repeat(' ', 1024), 1024*1023) as l(long); > ERROR: too much WAL data" > > 3) without this v5 patch, the same test will cause the standby crash > like below, and the st

Re: Reducing Memory Consumption (aset and generation)

2022-07-11 Thread Matthias van de Meent
ving them to MEMORY_CONTEXT_CHECKING context. I like this patch over 001 due to allowing less corruption to occur in the memory context code. This allows for detecting some issues in 003, as opposed to none in 001. Kind regards, Matthias van de Meent

Re: Make name optional in CREATE STATISTICS

2022-07-07 Thread Matthias van de Meent
tat types)] > Patch v4 attached Thanks for working on this! Kind regards, Matthias van de Meent

Re: Make name optional in CREATE STATISTICS

2022-07-06 Thread Matthias van de Meent
pdated section that details the expected grammar for CREATE STATISTICS, as you seem to have overlooked that copy of grammar documentation. Apart from these two small issues, this passes tests and seems complete. Kind regards, Matthias van de Meent

Re: doc: Move enum storage commentary to top of section

2022-07-06 Thread Matthias van de Meent
On Thu, 9 Jun 2022 at 18:12, David G. Johnston wrote: > > Per suggestion over on -docs: > > https://www.postgresql.org/message-id/bl0pr06mb4978f6c0b69f3f03aebed0fbb3...@bl0pr06mb4978.namprd06.prod.outlook.com I agree with moving the size of an enum into the top, but I don't think that the label l

Re: [RFC] Improving multi-column filter cardinality estimation using MCVs and HyperLogLog

2022-07-06 Thread Matthias van de Meent
ned bitmap-based statistics would not allow this, because the index of a sampled row will likely shift between runs, even assuming that a row is shared in the sample. Summary: We shouldn't use HLL, (compressed) bitmaps will work fine for an initial implementation of combined sketch-based MCV

Re: Custom tuplesorts for extensions

2022-07-06 Thread Matthias van de Meent
hat it is used for, and how to use it. > +struct Tuplesortstate > +{ > +TuplesortOps ops; This field needs a comment too. 0006: Needs a commit message, but otherwise seems fine. Kind regards, Matthias van de Meent

Re: doc: Fix description of how the default user name is chosen

2022-07-06 Thread Matthias van de Meent
not connect to the database named after the user's username"). The right-ness of the systematic default doesn't matter in this context (that's something to put in the comments of that code or discuss on -hackers), and the right-ness of the contextual default was already proven to be wrong in this configuration of server and client, by the context of failing to connect to that defaulted database. Kind regards, Matthias van de Meent

Re: [PATCH] Compression dictionaries for JSONB

2022-07-04 Thread Matthias van de Meent
tion with different TableAMs, etc), I > seriously doubt that this particular part of "pluggable TOASTer" will > end-up in the core. Yes, and that's why I think that this where this dictionary infrastructure could provide value, as an alternative or extension to the proposed jsonb toaster in the 'pluggable toaster' thread. Kind regards, Matthias van de Meent

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-07-01 Thread Matthias van de Meent
On Tue, 21 Jun 2022 at 03:45, Michael Paquier wrote: > + /* > +* Ensure that xlogreader.c can read the record by ensuring that the > +* data section of the WAL record can be allocated. > +*/ > + if (unlikely(!AllocSizeIsValid(total_len))) > + XLogErrorDataLimitExceeded(); > >

Re: Pluggable toaster

2022-07-01 Thread Matthias van de Meent
he correct set of patches to apply once downloaded. Kind regards, Matthias van de Meent [0] https://www.postgresql.org/message-id/CA%2BTgmoZBgNipyKuQAJzNw2w7C9z%2B2SMC0SAHqCnc_dG1nSLNcw%40mail.gmail.com

Re: making relfilenodes 56 bits

2022-06-29 Thread Matthias van de Meent
On Wed, 29 Jun 2022 at 14:41, Simon Riggs wrote: > > On Tue, 28 Jun 2022 at 19:18, Matthias van de Meent > wrote: > > > I will be the first to admit that it is quite unlikely to be common > > practise, but this workload increases the number of dbOid+spcOid > > co

Re: making relfilenodes 56 bits

2022-06-28 Thread Matthias van de Meent
o be common practise, but this workload increases the number of dbOid+spcOid combinations to 100s (even while using only a single tablespace), which in my opinion requires some more thought than just handwaving it into an smgr array and/or checkpoint records. Kind regards, Matthias van de Meent

Re: Pre-installed index access methods cannot be manually installed.

2022-06-24 Thread Matthias van de Meent
> PS. I noticed this when checking the sortsupport code for some active > patches, and after playing around a bit while trying to run PG > extensions that are not system-registered. The attached test case > fails, where I'd expect it to succeed, or at least I expected it not > to fail at "This AM's

Pre-installed index access methods cannot be manually installed.

2022-06-24 Thread Matthias van de Meent
of the supplied relation's access method, which will not match the expected value when manually installed. Is this expected? Would we accept patches that remove or reduce the impact of these artificial limitations? Kind regards, Matthias van de Meent PS. I noticed this when checking the sortsupport

Re: some aspects of our qsort might not be ideal

2022-06-23 Thread Matthias van de Meent
On Thu, 23 Jun 2022 at 17:04, Peter Geoghegan wrote: > > On Thu, Jun 23, 2022 at 7:51 AM Matthias van de Meent > wrote: > > I think that mostly has to do with reliability / stability of ORDER BY > > in combination with LIMIT and OFFSET, even though right now we cannot >

Re: some aspects of our qsort might not be ideal

2022-06-23 Thread Matthias van de Meent
On Thu, 23 Jun 2022 at 15:52, Robert Haas wrote: > > On Thu, Jun 23, 2022 at 6:13 AM John Naylor > wrote: > > Here is a *rough* first pass at dual-pivot quicksort. I haven't > > changed the regression tests to adjust for qsort being an unstable > > sort, ... > > Hmm. I thought we had some reasons

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-06-20 Thread Matthias van de Meent
On Mon, 20 Jun 2022 at 07:02, Michael Paquier wrote: > > On Sat, Jun 11, 2022 at 09:25:48PM +0200, Matthias van de Meent wrote: > > Did you initiate a new cluster or otherwise skip the invalid record > > you generated when running the instance based on master? It seems to >

Re: CREATE TABLE ( .. STORAGE ..)

2022-06-16 Thread Matthias van de Meent
On Wed, 15 Jun 2022 at 16:51, Aleksander Alekseev wrote: > > Hi hackers, > > I noticed that cfbot is not entirely happy with the patch, so I rebased it. > > > I see that COMPRESSION and STORAGE now are handled slightly > > differently in the grammar. Maybe we could standardize that a bit > > more;

Re: better page-level checksums

2022-06-14 Thread Matthias van de Meent
On Tue, 14 Jun 2022 at 14:56, Robert Haas wrote: > > On Mon, Jun 13, 2022 at 5:14 PM Matthias van de Meent > wrote: > > It's not that I disagree with (or dislike the idea of) increasing the > > resilience of checksums, I just want to be very careful that we do

Re: better page-level checksums

2022-06-13 Thread Matthias van de Meent
On Fri, 10 Jun 2022 at 15:58, Robert Haas wrote: > > On Thu, Jun 9, 2022 at 8:00 PM Matthias van de Meent > wrote: > > Why so? We already dole out per-page space in 4-byte increments > > through pd_linp, and I see no reason why we can't reserve some line > > poin

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-06-11 Thread Matthias van de Meent
On Sat, 11 Jun 2022 at 01:32, David Zhang wrote: > > Hi, > > > > MaxAllocSize is pretty easy: > > > SELECT pg_logical_emit_message(false, long, long) FROM repeat(repeat(' ', > > > 1024), 1024*1023) as l(long); > > > > > > on a standby: > > > > > > 2022-03-11 16:41:59.336 PST [3639744][startup][1/

Re: better page-level checksums

2022-06-09 Thread Matthias van de Meent
ffer manager also touch / reserve space in the special areas seems like a break of abstraction: Quoting from bufpage.h: * AM-generic per-page information is kept in PageHeaderData. * * AM-specific per-page data (if any) is kept in the area marked "special * space"; each AM has an "opaque" structure defined somewhere that is * stored as the page trailer. an access method should always * initialize its pages with PageInit and then set its own opaque * fields. I'd rather we keep this contract: am-generic stuff belongs in PageHeaderData, with the rest of the page fully available for the AM to use (including the special area). Kind regards, Matthias van de Meent

Re: [PATCH] Compression dictionaries for JSONB

2022-06-05 Thread Matthias van de Meent
that would allow for an overall better user experience and better performance due to decreased need for fully decompressed type casting. Kind regards, Matthias van de Meent.

Re: compiler warnings with gcc 4.8 and -Og

2022-06-02 Thread Matthias van de Meent
On Thu, 2 Jun 2022, 07:10 Tom Lane, wrote: > > Justin Pryzby writes: > > forking: <20220302205058.gj15...@telsasoft.com>: Re: Adding CI to our tree > > On Wed, Mar 02, 2022 at 02:50:58PM -0600, Justin Pryzby wrote: > >> BTW (regarding the last patch), I just noticed that -Og optimization can > >

Re: Add SPLIT PARTITION/MERGE PARTITIONS commands

2022-05-31 Thread Matthias van de Meent
) into two partitions with (mod, rem) of (6, 1) and (6, 4) respectively, with the reverse being true for merge operations, right? Kind regards, Matthias van de Meent

Re: Ignoring BRIN for HOT udpates seems broken

2022-05-30 Thread Matthias van de Meent
On Sat, 28 May 2022 at 22:51, Tomas Vondra wrote: > > > > On 5/28/22 21:24, Matthias van de Meent wrote: > > On Sat, 28 May 2022 at 16:51, Tomas Vondra > > wrote: > >> > >> Hi, > >> > >> while working on some BRIN stuff, I realized (m

Re: Ignoring BRIN for HOT udpates seems broken

2022-05-28 Thread Matthias van de Meent
On Sat, 28 May 2022 at 16:51, Tomas Vondra wrote: > > Hi, > > while working on some BRIN stuff, I realized (my) commit 5753d4ee320b > ignoring BRIN indexes for HOT is likely broken. Consider this example: > > -- > CREATE TABLE t (

Re: adding status for COPY progress report

2022-05-25 Thread Matthias van de Meent
On Wed, 25 May 2022 at 16:32, Zhihong Yu wrote: > > On Wed, May 25, 2022 at 3:55 AM Matthias van de Meent > wrote: >> >> On Wed, 25 May 2022 at 10:15, Zhihong Yu wrote: >> > >> > Hi, >> > Patch v3 follows advice from Matthias (status field has be

Re: adding status for COPY progress report

2022-05-25 Thread Matthias van de Meent
for the progress reporting view. Could you please provide some insights on why you think that we should change the progress reporting guts to accomodate something that it was not built for? Kind regards, Matthias van de Meent

Re: adding status for COPY progress report

2022-05-24 Thread Matthias van de Meent
On Tue, 24 May 2022 at 22:12, Zhihong Yu wrote: > > On Tue, May 24, 2022 at 12:37 PM Matthias van de Meent > wrote: >> >> On Tue, 24 May 2022 at 19:13, Zhihong Yu wrote: >> > >> > Hi, >> > Please see attached for enhancement to COPY command p

Re: adding status for COPY progress report

2022-05-24 Thread Matthias van de Meent
d _copy views when VACUUM is running, etc, because you removed the filter on cmdtype. Also, the added fields in CopyToStateData / CopyFromStateData seem useless when a pgstat_progress_update_param in the right place should suffice. Kind regards, Matthias van de Meent

Re: [RFC] Improving multi-column filter cardinality estimation using MCVs and HyperLogLog

2022-05-19 Thread Matthias van de Meent
On Mon, 16 May 2022 at 00:09, Tomas Vondra wrote: > > On 5/15/22 21:55, Matthias van de Meent wrote: > > Note: I am not (currently) planning on implementing this rough idea, > > just putting it up to share and document the idea, on request of Tomas > > (cc-ed). > &g

[RFC] Improving multi-column filter cardinality estimation using MCVs and HyperLogLog

2022-05-15 Thread Matthias van de Meent
nt nTuples distinct values, so low values for default_statistics_target would allow for smaller values for m in the HLL sketches, whilst still providing fairly accurate result estimates. Kind regards, Matthias van de Meent PS: Several later papers correctly point out that HLL can only count up to 2

Re: First draft of the PG 15 release notes

2022-05-11 Thread Matthias van de Meent
On Tue, 10 May 2022 at 17:44, Bruce Momjian wrote: > > I have completed the first draft of the PG 15 release notes and you can > see the results here: > > https://momjian.us/pgsql_docs/release-15.html > > The feature count is similar to recent major releases: > > release-10 195 >

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread Matthias van de Meent
On Mon, 2 May 2022 at 16:09, Pavel Stehule wrote: > > > > po 2. 5. 2022 v 16:02 odesílatel Pavel Stehule > napsal: >> there is just shared buffers changed to 32GB and work_mem to 70MB. >> Unfortunately - it is in production environment with customer data, so I >> cannot to play too much >> >>

Re: strange slow query - lost lot of time somewhere

2022-05-02 Thread Matthias van de Meent
On Mon, 2 May 2022 at 11:00, Pavel Stehule wrote: > > Hi > > I found a query that is significantly slower with more memory Which PostgreSQL version did you use? Did you enable assert checking? Do you have an example database setup to work with? > plan 2 > QUERY PLAN > > Nested

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-23 Thread Matthias van de Meent
On Sat, 23 Apr 2022 at 00:51, David Christensen wrote: > > Hi -hackers, > > Enclosed is a patch to allow extraction/saving of FPI from the WAL > stream via pg_waldump. > > Description from the commit: > > Extracts full-page images from the WAL stream into a target directory, > which must be empty

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-04-18 Thread Matthias van de Meent
Seeing that the busiest time for PG15 - the last commitfest before the feature freeze - has passed, could someone take another look at this? The changes that were requested by Heikki and Andres have been merged into patch v3, and I think it would be nice to fix this security issue in the upcoming

Re: Temporary file access API

2022-04-12 Thread Matthias van de Meent
On Mon, 11 Apr 2022 at 10:05, Antonin Houska wrote: > > Robert Haas wrote: > > > On Fri, Apr 8, 2022 at 4:54 AM Antonin Houska wrote: > > > Do you think that the use of a system call is a problem itself (e.g. > > > because > > > the code looks less simple if read/write is used somewhere and >

Re: API stability

2022-04-11 Thread Matthias van de Meent
.]-tag, and two of which seem to be started by you as a reply on the original thread, but with the [was: pgsql: ...]-tag dropped and thus considered a new thread). So, this might be the reason Robert overlooked your declaration to volunteer: he was looking for volunteers in the thread "Re: API Stability [was: pgsql: ...]" in the Gmail UI, which didn't show your messages there because of the different subject line. Kind regards, Matthias van de Meent

Re: Improving btree performance through specializing by key shape, take 2

2022-04-10 Thread Matthias van de Meent
On Sun, 10 Apr 2022 at 23:45, Peter Geoghegan wrote: > > On Fri, Apr 8, 2022 at 9:55 AM Matthias van de Meent > wrote: > > Here's generation 2 of this effort. Instead of proceeding to trying to > > shoehorn all types of btree keys into one common code path, this >

Re: Size of pg_rewrite

2022-04-08 Thread Matthias van de Meent
On Fri, 8 Apr 2022 at 17:20, Dagfinn Ilmari Mannsåker wrote: > > Matthias van de Meent writes: > > > But, as text literal concatenations don't seem to get constant folded > > before storing them in the rules table, this rewrite of the views > > would result in lo

Size of pg_rewrite (Was: Report checkpoint progress with pg_stat_progress_checkpoint)

2022-04-08 Thread Matthias van de Meent
On Sat, 19 Mar 2022 at 01:15, Andres Freund wrote: > pg_rewrite without pg_stat_progress_checkpoint: 745472, with: 753664 > > pg_rewrite is the second biggest relation in an empty database already... Yeah, that's not great. Thanks for nerd-sniping me into looking into how views and pg_rewrite rul

Re: Lowering the ever-growing heap->pd_lower

2022-04-08 Thread Matthias van de Meent
On Fri, 8 Apr 2022 at 01:01, Andres Freund wrote: > > Hi, > > On 2022-04-04 19:24:22 -0700, Peter Geoghegan wrote: > > We should definitely increase MaxHeapTuplesPerPage before too long, > > for a variety of reasons that I have talked about in the past. Its > > current value is 291 on all mainstre

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-04-07 Thread Matthias van de Meent
On Thu, 7 Apr 2022 at 21:11, Robert Haas wrote: > > On Thu, Apr 7, 2022 at 2:43 PM Peter Geoghegan wrote: > > But if we were in a green-field situation we'd probably not want to > > use up several bytes for a nonse anyway. You said so yourself. > > I don't know what statement of mine you're talki

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-04-07 Thread Matthias van de Meent
On Tue, 5 Apr 2022 at 21:45, Robert Haas wrote: > > On Fri, Apr 1, 2022 at 11:12 AM Matthias van de Meent > wrote: > > Here's a new 0001 to keep CFBot happy. > > This seems like it would conflict with the proposal from > http://postgr.es/m/ca+tgmoad8wmn6i1mmu

Re: How to generate a WAL record spanning multiple WAL files?

2022-04-05 Thread Matthias van de Meent
On Tue, 5 Apr 2022 at 15:13, Bharath Rupireddy wrote: > > Hi, > > I wanted to have a WAL record spanning multiple WAL files of size, say > 16MB. I'm wondering if the Full Page Images (FPIs) of a TOAST table > would help here. Please let me know if there's any way to generate > such large WAL recor

Re: Restructure ALTER TABLE notes to clarify table rewrites and verification scans

2022-04-01 Thread Matthias van de Meent
On Fri, 1 Apr 2022 at 16:10, James Coleman wrote: > > On Thu, Mar 31, 2022 at 10:58 AM Matthias van de Meent > wrote: > > > > On Tue, 29 Mar 2022 at 16:20, James Coleman wrote: > > > > > > Over in the "Document atthasmissing default optimization

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-04-01 Thread Matthias van de Meent
On Fri, 1 Apr 2022 at 10:50, Matthias van de Meent wrote: > > On Fri, 1 Apr 2022 at 07:38, Michael Paquier wrote: > > > > On Thu, Mar 31, 2022 at 12:09:35PM +0200, Matthias van de Meent wrote: > > > PageInit MAXALIGNs the size of the special area that it receives a

Re: Is monotonous xid8 is a right way to do?

2022-04-01 Thread Matthias van de Meent
transaction IDs are reasonably consumed at a rate higher than 5B /sec (and not logged at that rate) we can start considering this to be a problem. A different and more important issue (IMO) is that the xlog record header currently only supports 32-bit xids -- long-running transactions can reasonably see a xid4 wraparound in their lifetime. Enjoy, Matthias van de Meent

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-04-01 Thread Matthias van de Meent
On Fri, 1 Apr 2022 at 07:38, Michael Paquier wrote: > > On Thu, Mar 31, 2022 at 12:09:35PM +0200, Matthias van de Meent wrote: > > PageInit MAXALIGNs the size of the special area that it receives as an > > argument; so any changes to the page header that would misalign the >

Re: Restructure ALTER TABLE notes to clarify table rewrites and verification scans

2022-03-31 Thread Matthias van de Meent
able scan if the table partition's existing constraints already imply the new partition constraints. The previous wording is better in that regard ("may need", instead of "requires"), though it could be improved by refering to the sql-altertable-attach-partition section. Kind regards, Matthias van de Meent

Re: Commitfest Update

2022-03-31 Thread Matthias van de Meent
On Thu, 31 Mar 2022 at 12:56, Alvaro Herrera wrote: > > On 2022-Mar-31, Julien Rouhaud wrote: > > > On Wed, Mar 30, 2022 at 02:41:26PM -0400, Greg Stark wrote: > > > > > > Patches that are Waiting on Author and haven't had activity in months > > > -- traditionally they were set to Returned with Fe

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-03-31 Thread Matthias van de Meent
On Thu, 31 Mar 2022 at 09:32, Michael Paquier wrote: > > On Mon, Mar 28, 2022 at 05:09:10PM +0200, Matthias van de Meent wrote: > > Not all clusters have checksums enabled (boo!, but we can't > > realistically fix that), so on-disk corruption could reasonably > >

Re: CREATE TABLE ( .. STORAGE ..)

2022-03-29 Thread Matthias van de Meent
On Wed, 2 Feb 2022 at 11:13, Teodor Sigaev wrote: > > Hi! > > > Are they both set to name or ColId? Although they are the same. > > > > Thank you, fixed, that was just an oversight. > > > 2 For ColumnDef new member storage_name, did you miss the function > > _copyColumnDef() _equalColumnDef()? >

Re: Correct docs re: rewriting indexes when table rewrite is skipped

2022-03-29 Thread Matthias van de Meent
arns for index rebuilds, even when the letter of the docs is incorrect: there are indeed cases we don't need to rebuild the indexes; but that would require more elaboration. Kind regards, Matthias van de Meent

Re: TRAP: FailedAssertion("HaveRegisteredOrActiveSnapshot()", File: "toast_internals.c", Line: 670, PID: 19403)

2022-03-29 Thread Matthias van de Meent
On Tue, 29 Mar 2022 at 11:10, Kyotaro Horiguchi wrote: > > At Tue, 29 Mar 2022 17:06:21 +0900 (JST), Kyotaro Horiguchi > wrote in > > At Mon, 28 Mar 2022 18:36:46 +0900 (JST), Kyotaro Horiguchi > > wrote in > > Then, I found that portal->holdSnapshot is that. I came up with the > > attached.

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-03-28 Thread Matthias van de Meent
On Mon, 28 Mar 2022 at 06:33, Michael Paquier wrote: > > On Wed, Feb 16, 2022 at 10:24:58PM +0100, Matthias van de Meent wrote: > > A first good reason to do this is preventing further damage when a > > page is corrupted: if I can somehow overwrite pd_special, > > non-ass

Re: Assert in pageinspect with NULL pages

2022-03-27 Thread Matthias van de Meent
ying) the size and location of the special area. As such, you might be interested in that patch. Note that currently in core postgres a corrupted special area pointer will likely target neighbouring blocks in the buffer pool; resulting in spreading corruption when the special area is updated. This

Re: Invalid comment in ParallelQueryMain

2022-03-26 Thread Matthias van de Meent
On Sat, 26 Mar 2022 at 17:01, Julien Rouhaud wrote: > > Hi, > > While reading code around I just noticed that I failed to adapt a comment a > couple of lines above a removed line in 0f61727b75b9. Patch attached. +1, seems OK to me.

Re: Hardening heap pruning code (was: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum)

2022-03-20 Thread Matthias van de Meent
On Sun, 20 Mar 2022 at 04:48, Peter Geoghegan wrote: > > Attached is v6, which goes a bit further than v5 in using local state > that we build up-front to describe the state of the page being pruned > (rather than rereading the page itself). I didn't test the code; so these comments are my genera

Re: Probable CF bot degradation

2022-03-20 Thread Matthias van de Meent
or CF entries, so that "bad" commits on master won't impact CFbot results as easy? Kind regards, Matthias van de Meent

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-03-15 Thread Matthias van de Meent
Apart from registering this in CF 2022-07 last Friday, I've also just added this issue to the Open Items list for PG15 under "Older bugs affecting stable branches"; as a precaution to not lose track of this issue in the buzz of the upcoming feature freeze. -Matthias

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-03-15 Thread Matthias van de Meent
hich would > allow us to e.g. get away from needing Heap2. Which would aestethically be > pleasing. That would be interesting; though out of scope for this bug I'm trying to fix. > On 2022-03-14 17:57:23 +0100, Matthias van de Meent wrote: > > I'm not sure whether or no

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-03-14 Thread Matthias van de Meent
Thank you all for the feedback. Please find attached v2 of the patchset, which contains updated comments and applies the suggested changes. On Sat, 12 Mar 2022 at 02:03, Andres Freund wrote: > > Hi, > > On 2022-03-11 22:42:42 +0200, Heikki Linnakangas wrote: > > Have you been able to create a tes

Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-03-11 Thread Matthias van de Meent
his patch, I've also added a comment to the XLogRecord spec to document that xl_tot_len should not be larger than 1GB - 1B; and why that limit exists. Kind regards, Matthias van de Meent From a72121b8fccbbf8c289e71a2c76801a1004f5353 Mon Sep 17 00:00:00 2001 From: Matthias van de Meent Date: Fri

Re: Lowering the ever-growing heap->pd_lower

2022-03-10 Thread Matthias van de Meent
On Wed, 16 Feb 2022 at 21:14, Matthias van de Meent wrote: > > On Wed, 16 Feb 2022 at 20:54, Peter Geoghegan wrote: > > > > On Tue, Feb 15, 2022 at 10:48 AM Matthias van de Meent > > wrote: > > > Peter Geoghegan asked for good arguments for the two change

Re: New Table Access Methods for Multi and Single Inserts

2022-03-07 Thread Matthias van de Meent
On Sun, 6 Mar 2022 at 12:12, Bharath Rupireddy wrote: > > On Fri, Mar 4, 2022 at 8:07 PM Matthias van de Meent > wrote: > > > Another rebase due to conflicts in 0003. Attaching v6 for review. > > > > I recently touched the topic of multi_insert, and I remembered

Re: New Table Access Methods for Multi and Single Inserts

2022-03-04 Thread Matthias van de Meent
On Mon, 19 Apr 2021 at 06:52, Bharath Rupireddy wrote: > > On Mon, Apr 5, 2021 at 9:49 AM Bharath Rupireddy > wrote: > > > > On Wed, Mar 10, 2021 at 10:21 AM Bharath Rupireddy > > wrote: > > > Attaching the v4 patch set. Please review it further. > > > > Attaching v5 patch set after rebasing ont

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-28 Thread Matthias van de Meent
On Sun, 27 Feb 2022 at 16:14, Bharath Rupireddy wrote: > 3) Why do we need this extra calculation for start_lsn? > Do you ever see a negative LSN or something here? > +('0/0'::pg_lsn + ( > +CASE > +WHEN (s.param3 < 0) THEN pow((2)::numeric, (64)::numeric) > +ELS

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-25 Thread Matthias van de Meent
On Fri, 25 Feb 2022 at 17:35, Julien Rouhaud wrote: > > On Fri, Feb 25, 2022 at 08:53:50PM +0530, Nitin Jadhav wrote: > > > > > > I'm not sure what Matthias meant, but as far as I know there's no > > > fundamental > > > difference between checkpoint with and without the CHECKPOINT_IMMEDIATE > >

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-24 Thread Matthias van de Meent
OGRESS_CHECKPOINT_PHASE_INIT 0 Generally, enum-like values in a stat_progress field are 1-indexed, to differentiate between empty/uninitialized (0) and states that have been set by the progress reporting infrastructure. Kind regards, Matthias van de Meent

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-23 Thread Matthias van de Meent
On Wed, 23 Feb 2022 at 15:24, Nitin Jadhav wrote: > > > At least for pg_stat_progress_checkpoint, storing only a timestamp in > > the pg_stat storage (instead of repeatedly updating the field as a > > duration) seems to provide much more precise measures of 'time > > elapsed' for other sessions if

<    1   2   3   4   5   6   7   >