Re: Change GUC hashtable to use simplehash?

2023-11-17 Thread Andres Freund
ing that makes simplehash always faster, but still doesn't win on memory usage at the upper end - the two pointers in GUCHashEntry make it too big. I think, independent of this patch, it might be worth requiring that hash table lookups applied the transformation before the lookup. A comparison function this expensive is not great... Greetings, Andres Freund

Re: Change GUC hashtable to use simplehash?

2023-11-17 Thread Andres Freund
e same code for highly contended, partitioned, shared memory hashtables and many tiny local memory hashtables. The design goals are just very different. Greetings, Andres Freund

Re: Change GUC hashtable to use simplehash?

2023-11-17 Thread Andres Freund
worth it I don't quite know, but I'd not be against it on principle or such. Greetings, Andres Freund

Re: simplehash: preserve consistency in case of OOM

2023-11-17 Thread Andres Freund
On 2023-11-17 13:00:19 -0800, Jeff Davis wrote: > Please tell me if you think the use of simplehash for a search_path > cache is the wrong tool for the job. No, seems fine. I just was curious - as you said, the older existing users won't ever care about this case.

Re: Lifetime of commit timestamps

2023-11-17 Thread Andres Freund
mechanism to clean up data in order > to avoid creating a new one. I don't really understand this part - independent of the mechanism (i.e. an slru), at some point we need to remove old data, just for space reasons. Greetings, Andres Freund

Re: lazy_scan_heap() should release lock on buffer before vacuuming FSM

2023-11-17 Thread Andres Freund
Hi, On 2023-11-15 16:21:45 -0500, Melanie Plageman wrote: > On Tue, Nov 14, 2023 at 7:15 PM Andres Freund wrote: > > On 2023-11-14 07:46:10 -0500, Melanie Plageman wrote: > > > > FreeSpaceMapVacuumRange()'s comment says: > > > > * As above, but assume that only

Re: ResourceOwner refactoring

2023-11-17 Thread Andres Freund
Hi, On 2023-11-17 12:44:41 -0800, Andres Freund wrote: > On 2023-11-07 13:28:28 +0200, Heikki Linnakangas wrote: > > I feel pretty good about this overall. Barring objections or new cfbot > > failures, I will commit this in the next few days. > > I am working on rebasi

Re: ResourceOwner refactoring

2023-11-17 Thread Andres Freund
memberLock(). It's currently not strictly required, but that seems like it's just leaking an implementation detail out? Greetings, Andres Freund

Re: simplehash: preserve consistency in case of OOM

2023-11-17 Thread Andres Freund
t; return tb; > } Maybe add a comment explaining why it's important to update parameters after allocating? Greetings, Andres Freund

Re: meson documentation build open issues

2023-11-17 Thread Andres Freund
Hi, On 2023-11-14 16:30:24 -0800, Andres Freund wrote: > On 2023-11-14 16:22:31 -0800, Andres Freund wrote: > > > v2-0004-meson-Add-world-target.patch > > > > > > AFAICT, this world target doesn't include the man target. (Again, this > > > would all

Use of backup_label not noted in log

2023-11-16 Thread Andres Freund
age when starting in a state that's "equivalent" to having used the backup_label. Thoughts? Greetings, Andres Freund

Re: lazy_scan_heap() should release lock on buffer before vacuuming FSM

2023-11-16 Thread Andres Freund
Hi, On 2023-11-16 15:29:38 -0500, Robert Haas wrote: > On Wed, Nov 15, 2023 at 6:17 PM Andres Freund wrote: > > Thoughts on whether to backpatch? It'd probably be at least a bit painful, > > there have been a lot of changes in the surrounding code in the last 5 > > years.

Re: lazy_scan_heap() should release lock on buffer before vacuuming FSM

2023-11-15 Thread Andres Freund
Hi, On 2023-11-15 16:32:48 -0500, Robert Haas wrote: > On Mon, Nov 13, 2023 at 8:26 PM Andres Freund wrote: > > I think this undersells the situation a bit. We right now do > > FreeSpaceMapVacuumRange() for 8GB of data (VACUUM_FSM_EVERY_PAGES) in the > > main &g

Re: Some performance degradation in REL_16 vs REL_15

2023-11-15 Thread Andres Freund
increases page faults. Rebuilding an optimized build generates something like 1GB of dirty data. Particularly with ccache, that'll typically not yet be flushed by the time you run a benchmark. That's not nothing, even with a decent NVMe SSD. - many more, unfortunately Greetings, Andres Freund

Re: Some performance degradation in REL_16 vs REL_15

2023-11-15 Thread Andres Freund
ast disk) the CPU would have plenty time to clock down. Benchmarking sucks. Modern hardware realities suck. Greetings, Andres Freund

Re: remaining sql/json patches

2023-11-15 Thread Andres Freund
On 2023-11-15 09:11:19 -0800, Andres Freund wrote: > On 2023-11-15 22:00:41 +0900, Amit Langote wrote: > > > This causes a nontrivial increase in the size of the parser (~5% in an > > > optimized build here), I wonder if we can do better. > > > > Hmm, sorry if

Re: remaining sql/json patches

2023-11-15 Thread Andres Freund
gt; raw-parsing speed. What does "isn't much slowdown" mean in numbers? Greetings, Andres Freund

Potential use-after-free in partion related code

2023-11-15 Thread Andres Freund
quickly enough identify what partConstraint, defPartConstraint, partBoundConstraint are, so I don't don't really know what the fix here is. Greetings, Andres Freund

Re: Explicitly skip TAP tests under Meson if disabled

2023-11-15 Thread Andres Freund
On 2023-11-15 11:02:19 +0100, Peter Eisentraut wrote: > On 04.11.23 01:51, Andres Freund wrote: > > I'd just use a single test() invocation here, and add an argument to > > testwrap > > indicating that it should print out the skipped message. That way we a) > &

Re: Assert failure on 'list_member_ptr(rel->joininfo, restrictinfo)'

2023-11-14 Thread Andres Freund
Hi, On 2023-11-14 14:42:13 +0200, Alexander Korotkov wrote: > It's possibly dumb option, but what about just removing the assert? That's not at all an option - the in-place bms_* functions can free their input. So a dangling pointer to the "old" version is a use-after-free waiting to happen -

Re: Assert failure on 'list_member_ptr(rel->joininfo, restrictinfo)'

2023-11-14 Thread Andres Freund
Hi, On 2023-11-14 19:14:57 +0800, Richard Guo wrote: > While working on BUG #18187 [1], I noticed that we also have issues with > how SJE replaces join clauses involving the removed rel. As an example, > consider the query below, which would trigger an Assert. > > create table t (a int primary

Re: typo in fallback implementation for pg_atomic_test_set_flag()

2023-11-14 Thread Andres Freund
entations that may activate this code for certain > compiler/architecture combinations. Hm. I don't really see how adding a faster atomic-exchange implementation could trigger this implementation being used? Greetings, Andres Freund

Re: meson documentation build open issues

2023-11-14 Thread Andres Freund
Hi, On 2023-11-14 16:22:31 -0800, Andres Freund wrote: > > v2-0004-meson-Add-world-target.patch > > > > AFAICT, this world target doesn't include the man target. (Again, this > > would all work better if we added "man" to "docs".) > > I agree

Re: meson documentation build open issues

2023-11-14 Thread Andres Freund
Hi, On 2023-11-14 11:58:53 +0100, Peter Eisentraut wrote: > On 09.11.23 18:52, Andres Freund wrote: > > I think if we want to do this, we'd need a configure time check for being > > able > > to validate a document with > > > V4.5//EN""http://www.oas

Re: meson documentation build open issues

2023-11-14 Thread Andres Freund
be the right > one. I'm not really seeing why that'd affect pkglibdir being adjustable, besides needing to tweak the description of pkglibdir? > I suggest we leave this patch for a separate discussion. Fair enough. Thanks for the review, Andres Freund

Re: lazy_scan_heap() should release lock on buffer before vacuuming FSM

2023-11-14 Thread Andres Freund
* even looking at the refcount and returns false if not. And writing out a buffer takes a content lock. Which made me realize that "tuples missed: 5848 dead from 89 pages not removed due to cleanup lock contention" is often kinda wrong - the contention is *not* cleanup lock specific. It's o

Re: pgsql: doc: fix wording describing the checkpoint_flush_after GUC

2023-11-14 Thread Andres Freund
Hi, On 2023-11-14 17:49:59 +0100, Alvaro Herrera wrote: > On 2023-Nov-13, Andres Freund wrote: > > On 2023-11-13 12:31:42 +0100, Alvaro Herrera wrote: > > > On 2023-Nov-09, Bruce Momjian wrote: > > > > > > > doc: fix wording describing the checkpoint_flush_

Re: building with meson on windows with ssl

2023-11-13 Thread Andres Freund
st that? The problem could be related to the fact that on windows you (I think) can have binaries with both 32bit and 64bit libraries loaded. Was this with msvc or gcc/mingw? Greetings, Andres Freund

Re: mxid_age() and age(xid) appear undocumented

2023-11-13 Thread Andres Freund
need to provide for the needs of those monitoring queries I > mentioned. I think it'd be a bad idea to "deprecate" existing working queries, with the replacement being a more complicated way to represent the same information. Greetings, Andres Freund

Re: lazy_scan_heap() should release lock on buffer before vacuuming FSM

2023-11-13 Thread Andres Freund
──┴─┴───┴─┴─┘ =# DELETE FROM copytest_0; =# VACUUM (VERBOSE) copytest_0; ... INFO: 0: table "copytest_0": truncated 146264 to 110934 pages ... tuples missed: 5848 dead from 89 pages not removed due to cleanup lock contention ... A bit of debugging later I figured out that this is due to the background writer. If I SIGSTOP bgwriter, the whole relation is truncated... Greetings, Andres Freund

Re: pgsql: doc: fix wording describing the checkpoint_flush_after GUC

2023-11-13 Thread Andres Freund
r a configurable number of bytes. Otherwise, these [...] > I'm not suggesting to revert this change, but rather I'd like to prevent > future changes of this type. Just saying it'd be sad to turn all the > Postgres documentation to using Basic English or whatever. +1 for the general notion. Greetings, Andres Freund

Re: Requiring recovery.signal or standby.signal when recovering with a backup_label

2023-11-13 Thread Andres Freund
Hi, On 2023-11-14 09:13:44 +0900, Michael Paquier wrote: > On Mon, Nov 13, 2023 at 03:41:44PM -0800, Andres Freund wrote: > > On 2023-11-09 12:16:52 +0900, Michael Paquier wrote: > >> On Thu, Nov 09, 2023 at 12:04:19PM +0900, Michael Paquier wrote: > >> > Sure, sorr

Re: [PATCH] pgbench log file headers

2023-11-13 Thread Andres Freund
ther are a wall-clock timestamp. Without differentiating between those, the column headers seem not very useful, because one needs to look in the documentation to understand the fields. I don't think there's all that strong a need for backward compatibility in pgbench. We could just change the columns as I suggest above and then always emit the header in the "main" log file. Greetings, Andres Freund

Re: Requiring recovery.signal or standby.signal when recovering with a backup_label

2023-11-13 Thread Andres Freund
in the label > file because we are missing some stuff in pg_wal/, which is something > that the code is currently able to handle. "able to handle" as in detect and error out? Because that's the only possible sane thing to do, correct? Greetings, Andres Freund

Re: Why do indexes and sorts use the database collation?

2023-11-13 Thread Andres Freund
Hi, On 2023-11-14 00:02:13 +0100, Tomas Vondra wrote: > On 11/13/23 23:12, Andres Freund wrote: > > On 2023-11-13 22:36:24 +0100, Tomas Vondra wrote: > >> ISTM it's about how complex the rules implemented by the collation are, > >> so I agree the cost should

Re: archive modules loose ends

2023-11-13 Thread Andres Freund
+ PG_exception_stack = NULL; > + > + /* Reset our memory context and switch back to the original one > */ > + MemoryContextSwitchTo(oldcontext); > + MemoryContextReset(archive_context); > + } It could be worth setting up an errcontext providing the module and file that's being processed. I personally find that at least as important as setting up a ps string detailing the log file... But I guess that could be a separate patch. It'd be nice to add a comment explaining why pgarch_archiveXlog() is the right place to handle errors. Greetings, Andres Freund

Re: Why do indexes and sorts use the database collation?

2023-11-13 Thread Andres Freund
ch, it structurally seems the sanest place > > to > > locate the cost. > > > > ISTM it's about how complex the rules implemented by the collation are, > so I agree the cost should be a feature of collations not providers. I'm not sure analysing the complexity in detail is worth it. ISTM there's a few "levels" of costliness: 1) memcmp() suffices 2) can safely use strxfrm() (i.e. ICU), possibly limited to when we sort 3) deterministic collations 4) non-deterministic collations I'm sure there are graduations, particularly within 3), but I'm not sure it's realistic / worthwhile to go to that detail. I think a cost model like the above would provide enough detail to make better decisions than today... Greetings, Andres Freund

Re: [PATCHES] Post-special page storage TDE support

2023-11-13 Thread Andres Freund
Hi, On 2023-11-13 14:37:47 -0600, David Christensen wrote: > On Mon, Nov 13, 2023 at 2:27 PM Andres Freund wrote: > > On 2023-11-08 18:47:56 -0800, Peter Geoghegan wrote: > > > On Wed, Nov 8, 2023 at 6:04 AM Stephen Frost wrote: > > > > In conversations with

Re: Split index and table statistics into different types of stats

2023-11-13 Thread Andres Freund
tructs for same relation: */ > + struct PgStat_IndexXactStatus *upper; /* next higher subxact if any */ > + PgStat_IndexStatus *parent; /* per-table status */ > + /* structs of same subxact level are linked here: */ > + struct PgStat_IndexXactStatus *next;/* next of same subxact */ > +} PgStat_IndexXactStatus; I don't think much of this is used? It doesn't look like you're using most of the fields. Which makes sense - there's not really the same transactional behaviour for indexes as there is for tables. Greetings, Andres Freund

Re: [PATCHES] Post-special page storage TDE support

2023-11-13 Thread Andres Freund
header, when we've wanted space in the page header for plenty other things, just doesn't seem a good use of said space. Greetings, Andres Freund

Re: Why do indexes and sorts use the database collation?

2023-11-13 Thread Andres Freund
Hi, On 2023-11-11 23:19:55 -0800, Jeff Davis wrote: > On Fri, 2023-11-10 at 17:19 -0800, Andres Freund wrote: > > I guess you are arguing that the user didn't intend to create an > > index here? > > No, obviously the user should expect an index when a primary key is > c

Re: pg_walfile_name_offset can return inconsistent values

2023-11-13 Thread Andres Freund
t; 00010017 > + 0/1701 | 00010017 | 1 | > 00010017 > +(3 rows) These would break when testing with a different segment size. Today that's not the case... Greetings, Andres Freund

Re: autovectorize page checksum code included elsewhere

2023-11-11 Thread Andres Freund
(0avgtext+0avgdata 278660maxresident)k 82624inputs+4571480outputs (571major+14004898minor)pagefaults 0swaps Greetings, Andres Freund

Re: pgsql: Don't trust unvalidated xl_tot_len.

2023-11-11 Thread Andres Freund
stdout/$header_path"; +open my $header_h, '<', "$fname" or die "could not open $fname: $!"; my @match = undef; while (<$header_h>) would be helpful. I guess we should also apply something like that to our tree - printing a stringified errno without even a path/filename isn't very useful. Greetings, Andres Freund

Re: pg_walfile_name_offset can return inconsistent values

2023-11-10 Thread Andres Freund
file_name); SELECT segment_number, file_offset = :segment_size - 1 FROM pg_walfile_name_offset('0/0'::pg_lsn + :segment_size - 1), pg_split_walfile_name(file_name); Greetings, Andres Freund

Re: locked reads for atomics

2023-11-10 Thread Andres Freund
Hi, On 2023-11-10 20:38:13 -0600, Nathan Bossart wrote: > On Fri, Nov 10, 2023 at 03:11:50PM -0800, Andres Freund wrote: > > On 2023-11-10 14:51:28 -0600, Nathan Bossart wrote: > >> + * This read is guaranteed to read the current value, > > > > It doesn't

Re: Adding facility for injection points (or probe points?) for more advanced tests

2023-11-10 Thread Andres Freund
e "modified" code, and do something else in the test script. But as-is a decent amount of C code would need to be written to write such a test, from what I can tell? Greetings, Andres Freund

Re: Why do indexes and sorts use the database collation?

2023-11-10 Thread Andres Freund
request (e.g. DISTINCT/GROUP BY, merge joins). I think we should also explain in our docs that C can be considerably faster - I couldn't find anything in a quick look. Greetings, Andres Freund

Re: Force the old transactions logs cleanup even if checkpoint is skipped

2023-11-10 Thread Andres Freund
if that doesn't happen - do we really want to wait till a checkpoint finishes to free up space? What if we instead made archiver delete WAL files after archiving, if they're old enough? Some care would be needed to avoid checkpointer and archiver trampling on each other, but that doesn't seem too hard. Greetings, Andres Freund

Re: ResourceOwner refactoring

2023-11-10 Thread Andres Freund
() or just call dsa_get_address() on an > existing dsa_pointer, you wouldn't expect the current resource owner to > matter. I think dsa_create/attach() should store the current resource owner > in the dsa_area, for use in subsequent operations on the DSA, per attached > patch (0002-Fix-dsa.c-with-different-resource-owners.patch). Yea, that seems the right direction from here. Greetings, Andres Freund

Re: locked reads for atomics

2023-11-10 Thread Andres Freund
And the fence specification is just about unreadable (although I think they did fix some of the worst issues). Greetings, Andres Freund

Re: locked reads for atomics

2023-11-10 Thread Andres Freund
Hi, On 2023-11-10 14:51:28 -0600, Nathan Bossart wrote: > Moving this to a new thread and adding it to the January commitfest. > > On Thu, Nov 09, 2023 at 03:27:33PM -0600, Nathan Bossart wrote: > > On Tue, Nov 07, 2023 at 04:58:16PM -0800, Andres Freund wrote: > >>

Re: Failure during Building Postgres in Windows with Meson

2023-11-10 Thread Andres Freund
Hi, On November 10, 2023 10:53:12 AM PST, Tristan Partin wrote: >On Thu Nov 9, 2023 at 9:31 AM CST, Nazir Bilal Yavuz wrote: >> Hi, >> >> On Thu, 9 Nov 2023 at 18:27, Tristan Partin wrote: >> > >> > Can you try with Meson v1.2.3? >> >> I tried with Meson v1.2.3 and upstream, both failed with

Re: AdvanceXLInsertBuffers() vs wal_sync_method=open_datasync

2023-11-10 Thread Andres Freund
Hi, On 2023-11-10 17:16:35 +0200, Heikki Linnakangas wrote: > On 10/11/2023 05:54, Andres Freund wrote: > > In this case I had used wal_sync_method=open_datasync - it's often faster > > and > > if we want to scale WAL writes more we'll have to use it more widely (you &g

Re: Add new option 'all' to pg_stat_reset_shared()

2023-11-09 Thread Andres Freund
Hi, On November 8, 2023 11:28:08 PM PST, Michael Paquier wrote: >On Thu, Nov 09, 2023 at 01:50:34PM +0900, torikoshia wrote: >> PGSTAT_KIND_SLRU cannot be reset by pg_stat_reset_shared(), so I feel >> uncomfortable to delete it all together. >> It might be better after pg_stat_reset_shared()

AdvanceXLInsertBuffers() vs wal_sync_method=open_datasync

2023-11-09 Thread Andres Freund
ere's much point in not also being somewhat aggressive in GetXLogBuffer()->AdvanceXLInsertBuffer(). Greetings, Andres Freund

Re: Failure during Building Postgres in Windows with Meson

2023-11-09 Thread Andres Freund
Hi, On 2023-11-10 08:17:48 +0530, vignesh C wrote: > On Thu, 9 Nov 2023 at 21:12, Tristan Partin wrote: > > > > On Thu Nov 9, 2023 at 9:31 AM CST, Nazir Bilal Yavuz wrote: > > > Hi, > > > > > > On Thu, 9 Nov 2023 at 18:27, Tristan Partin wrote: > > > > > > > > Can you try with Meson v1.2.3? > >

Re: pg_upgrade failing for 200+ million Large Objects

2023-11-09 Thread Andres Freund
Hi, On November 9, 2023 10:41:01 AM PST, Tom Lane wrote: >Also, pg_upgrade is often invoked indirectly via scripts, so I do >not especially buy the idea that we're going to get useful control >input from some human somewhere. I think we'd be better off to >assume that pg_upgrade is on its own

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2023-11-09 Thread Andres Freund
already valid, I don't think it's ok to just read/parse WAL data that's concurrently being modified. We can never allow WAL being read that's past XLogBytePosToRecPtr(XLogCtl->Insert->CurrBytePos) as it does not exist. And if the to-be-read LSN is between XLogCtl->LogwrtResult->Write and XLogBytePosToRecPtr(Insert->CurrBytePos) we need to call WaitXLogInsertionsToFinish() before copying the data. Greetings, Andres Freund

Re: meson documentation build open issues

2023-11-09 Thread Andres Freund
Hi, On 2023-11-09 15:32:39 +0100, Peter Eisentraut wrote: > On 09.11.23 01:59, Andres Freund wrote: > > > I think we could build doc/src/sgml/postgres-full.xml by default. That > > > takes less than 0.5 seconds here and it's an intermediate target for html > > >

Re: make pg_ctl more friendly

2023-11-08 Thread Andres Freund
Hi, On 2023-11-09 09:29:32 +0800, Crisp Lee wrote: > How to judge from 'DB_SHUTDOWNED' that PITR ends normally? 'DB_SHUTDOWNED' > is just a state, it could not give more meaning, so I reuse the > recovery.done. DB_SHUTDOWNED cannot be encountered while recovery is ongoing. If there was a hard

Re: Add new option 'all' to pg_stat_reset_shared()

2023-11-08 Thread Andres Freund
e the individual kind of stats is reset. Greetings, Andres Freund

Re: meson documentation build open issues

2023-11-08 Thread Andres Freund
fault, otherwise we'll cause errors on plenty systems that don't get them today. The docbook dts aren't a huge dependency, but still. Some OSs might not have a particularly install source for them, e.g. windows. I don't think that'd detect the missing ids? Greetings, Andres Freund

Re: meson documentation build open issues

2023-11-08 Thread Andres Freund
ould still offer > alternatives that exclude some areas. If it's the other way round, > some areas will never be checked widely. The 'test' target (generated by meson, otherwise I'd have named it check), runs all enabled tests. You obviously can run a subset if you so desire. Greetings, Andres Freund

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2023-11-08 Thread Andres Freund
Hi, On 2023-11-08 19:00:01 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2023-11-08 13:18:39 -0500, Tom Lane wrote: > >> I think an actually usable feature of this sort would involve > >> copying all the failed lines to some alternate output medium,

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2023-11-08 Thread Andres Freund
iling rows and also to abort if the number of errors or the type of errors gets to be too big. Greetings, Andres Freund

Re: meson documentation build open issues

2023-11-08 Thread Andres Freund
mit. But I don't think that hurts much. Greetings, Andres Freund >From 9b0b5cd952880ecebbd157c05698125755bc53ed Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Wed, 8 Nov 2023 09:29:38 -0800 Subject: [PATCH v2 1/7] meson: Change default of 'selinux' feature option to auto There is rea

Re: Syncrep and improving latency due to WAL throttling

2023-11-08 Thread Andres Freund
k 100 bytes, wait for the data be transmitted, and then wake up a process, there are periods where the link in one direction is largely idle. But in case of a 10kB packet on the 1gbit network, yes, we are bandwidth limited for ~80us (or perhaps more interestingly, we are bandwidth limited for 0.8ms when sending 100kB). Greetings, Andres Freund

Re: meson documentation build open issues

2023-11-08 Thread Andres Freund
Uniformity seems useful and it's low risk. > I played around with this a bit and noticed some files missing or in the > wrong place. See two attached patches (plus e9f075f9a1 already committed). Make sense. Greetings, Andres Freund

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-11-08 Thread Andres Freund
Hi, On 2023-11-07 15:55:48 -0500, Stephen Frost wrote: > * Andres Freund (and...@anarazel.de) wrote: > > On 2023-11-06 13:02:50 -0500, Stephen Frost wrote: > > > > > The max_total_memory limit is checked whenever the global counters are > > > > > upda

Re: Syncrep and improving latency due to WAL throttling

2023-11-08 Thread Andres Freund
ll messages don't need to get acknowledged individually. Sending more data at once is good for CPU efficiency (reducing syscall and network device overhead), but doesn't do much for throughput. Sending 4kB of data in each send() in a bandwidth oriented test already gets to ~9.3gbit/s in my network. That's close to the maximum atainable with normal framing. If I change the mtu back to 9000 I get 9.89 gbit/s, again very close to the theoretical max. Greetings, Andres Freund

Re: Syncrep and improving latency due to WAL throttling

2023-11-07 Thread Andres Freund
198070.20 638205 17 4.781 100 100 22 361166.12 1302368 24 2.568 No meaningful difference before getting to 100k. But it's pretty easy to lag by 100k on a longer distance link... Greetings, Andres Freund

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-11-07 Thread Andres Freund
On 2023-10-30 08:25:17 +0900, Michael Paquier wrote: > On Fri, Oct 27, 2023 at 09:45:52AM +0200, Drouvot, Bertrand wrote: > > Done in V8 attached (pgindent has been run on pgstatfuncs.c and > > pgstat_relation.c). > > And applied that after editing a bit the comments. Thank you both!

Re: Add new option 'all' to pg_stat_reset_shared()

2023-11-07 Thread Andres Freund
types of stats, not more. The proposed new code is far longer than the current pg_stat_reset(), despite doing something conceptually simpler. Greetings, Andres Freund

Re: Add new option 'all' to pg_stat_reset_shared()

2023-11-07 Thread Andres Freund
t; {reset specified stats kind} like the pg_stat_reset_slru()? It's not like oids are a precious resource. It's a more confusing API to have to have to specify a NULL as an argument than not having to do so. If we really want to avoid a separate oid, a more sensible path would be to add a default argument to pg_stat_reset_slru() (by doing a CREATE OR REPLACE in system_functions.sql). Greetings, Andres Freund

Re: Show WAL write and fsync stats in pg_stat_io

2023-11-07 Thread Andres Freund
sion related to the handling of PendingWalStats added in > pgstat_io.c, for one. Another approach would be to fetch the relevant columns from pg_stat_io in the pg_stat_wal view. That'd avoid double accounting and breaking existing monitoring. Greetings, Andres Freund

Re: Atomic ops for unlogged LSN

2023-11-07 Thread Andres Freund
o get rid of the memory barrier around reading unloggedLSN in CreateCheckPoint(), I'd do the conversion in a separate commit. Greetings, Andres Freund

Re: Atomic ops for unlogged LSN

2023-11-07 Thread Andres Freund
g gone through a bunch of code that implies memory barriers (spinlocks, lwlocks). However, even if there's likely some other implied memory barrier that we could piggyback on, the patch much simpler to understand if it doesn't change coherency rules. There's no way the overhead could matter. Greetings, Andres Freund

Re: Moving forward with TDE [PATCH v3]

2023-11-07 Thread Andres Freund
Hi, On 2023-11-06 11:26:44 +0100, Matthias van de Meent wrote: > On Sat, 4 Nov 2023 at 03:38, Andres Freund wrote: > > On 2023-11-02 22:09:40 +0100, Matthias van de Meent wrote: > > > I'm quite surprised at the significant number of changes being made > > > outside the

Re: [PATCHES] Post-special page storage TDE support

2023-11-07 Thread Andres Freund
t of features enabled in a cluster with a bitmask? That pretty much precludes utilizing extra page space in extensions. We could instead just have an extra cluster-wide file that defines a mapping of offset to feature. Greetings, Andres Freund

Re: Moving forward with TDE [PATCH v3]

2023-11-07 Thread Andres Freund
Hi, On 2023-11-06 09:56:37 -0500, Stephen Frost wrote: > * Andres Freund (and...@anarazel.de) wrote: > > I still am quite quite unconvinced that using the LSN as a nonce is a good > > design decision. > > This is a really important part of the overall path to moving this &

Re: Show WAL write and fsync stats in pg_stat_io

2023-11-07 Thread Andres Freund
n pg_stat_io. For some things we might want to also have more detailed information in other views (e.g. it doesn't make sense to track FPIs in pg_stat_io, but does make sense in pg_stat_wal) - but that should be in addition, not instead of. Greetings, Andres Freund

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-11-07 Thread Andres Freund
w for this will be confusing for users, without really much to show for. Excluding the current query would be useful for other cases as well, why don't we provide a way to do that with pg_backend_memory_contexts? Greetings, Andres Freund

Re: meson documentation build open issues

2023-11-07 Thread Andres Freund
Hi, On 2023-11-06 10:45:27 +0100, Christoph Berg wrote: > Re: Andres Freund > > > > The reason for that is simply that the docs take too long to build. > > > > > > That why I'd prefer to be able to separate arch:all and arch:any > > > builds, yes. > &

Re: meson documentation build open issues

2023-11-07 Thread Andres Freund
Hi, On November 7, 2023 7:55:37 AM PST, Peter Eisentraut wrote: >On 03.11.23 19:19, Christoph Berg wrote: > You can control this with the "docs" option for meson, as of recently. I've been looking into switching the Debian PG 17 build to meson, but I'm running into several

Re: Explicitly skip TAP tests under Meson if disabled

2023-11-06 Thread Andres Freund
Hi, On 2023-11-06 17:46:23 +0100, Peter Eisentraut wrote: > On 04.11.23 01:51, Andres Freund wrote: > > I'd just use a single test() invocation here, and add an argument to > > testwrap > > indicating that it should print out the skipped message. That way we a) > &

Re: Guiding principle for dropping LLVM versions?

2023-11-04 Thread Andres Freund
sid| > xenodermus | x86_64 | 6.0.1| Debian | sid| Would you want me to do this now or just before you apply the patch? I think I should stand up a few more replacement animals to cover older llvm versions with assertions enabled... Greetings, Andres Freund

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-11-03 Thread Andres Freund
---+--- > + 9890 | pg_stat_get_memory_reservation | | > {23,23,20,20,20,20,20,20} | {i,o,o,o,o,o,o,o} > + 9891 | pg_get_backend_memory_allocation | | > {23,23,20,20,20,20,20} | {i,o,o,o,o,o,o} > +(2 rows) This indicates that your pg_proc entries are broken, they need to fixed rather than allowed here. Greetings, Andres Freund

Re: Moving forward with TDE [PATCH v3]

2023-11-03 Thread Andres Freund
On 2023-11-02 19:32:28 -0700, Andres Freund wrote: > > From 327e86d52be1df8de9c3a324cb06b85ba5db9604 Mon Sep 17 00:00:00 2001 > > From: David Christensen > > Date: Fri, 29 Sep 2023 15:16:00 -0400 > > Subject: [PATCH v3 5/5] Add encrypted/authenticated WAL > > >

Re: Moving forward with TDE [PATCH v3]

2023-11-03 Thread Andres Freund
e used without a real IV, with some downsides), but we'd pretty much prevent ourselves from ever implementing authenticated encryption. Greetings, Andres Freund

Re: Something seems weird inside tts_virtual_copyslot()

2023-11-03 Thread Andres Freund
nt this in ExecCopySlot(), as that's what "users" of ExecCopySlot() would presumably would look at. I dug a bit in the history, and we used to say The caller must ensure the slots have compatible tupdescs. whatever that precisely means. > Is the Assert() in tts_virtual_copyslot() wrong? Yes, it's inverted. Greetings, Andres Freund

Re: make pg_ctl more friendly

2023-11-03 Thread Andres Freund
WNED b) get_control_dbstate() at the end is DB_SHUTDOWNED Greetings, Andres Freund

Re: Add new option 'all' to pg_stat_reset_shared()

2023-11-03 Thread Andres Freund
function call instead of callers defining in a loop didn't draw > much interest. That's not analoguous - in your example the caller by definition knows the set of wait events it wants to create. Introducing a batch API wouldn't change that. But in case of resetting all stats the caller does *not* inherently know the set of stats types. Greetings, Andres Freund

Re: Inconsistent use of "volatile" when accessing shared memory?

2023-11-03 Thread Andres Freund
Hi, On 2023-11-03 17:44:44 -0700, Jeff Davis wrote: > On Fri, 2023-11-03 at 15:59 -0700, Andres Freund wrote: > > I don't think so. We used to use volatile for most shared memory > > accesses, but > > volatile doesn't provide particularly useful semantics - and > > ge

Re: Explicitly skip TAP tests under Meson if disabled

2023-11-03 Thread Andres Freund
on here, and add an argument to testwrap indicating that it should print out the skipped message. That way we a) don't need two test() invocations, b) could still see the test name etc in the test invocation. Greetings, Andres Freund

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2023-11-03 Thread Andres Freund
of the time reading from the > file/avoided pread system calls: > https://www.postgresql.org/message-id/CALj2ACXKKK%3DwbiG5_t6dGao5GoecMwRkhr7GjVBM_jg54%2BNa%3DQ%40mail.gmail.com > > This commit also benefits when direct IO is enabled for WAL. > Reading WAL from WAL buffers puts back the p

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2023-11-03 Thread Andres Freund
gReaderState so that the callers can set it if they want this > facility via XLogReaderAllocate()? That seems wrong architecturally - why should xlogreader itself know about any of this? What would it mean in frontend code if read_from_wal_buffers were set? IMO this is something that should happen purely within the read function. Greetings, Andres Freund

Re: Inconsistent use of "volatile" when accessing shared memory?

2023-11-03 Thread Andres Freund
n lead to better code to use volatile *on the individual access* compared to using pg_compiler_barrier(), because it allows the compiler to keep other variables in registers. Greetings, Andres Freund

Re: meson documentation build open issues

2023-11-03 Thread Andres Freund
Hi, On 2023-11-03 20:19:18 +0100, Christoph Berg wrote: > Re: Andres Freund > > The reason for that is simply that the docs take too long to build. > > That why I'd prefer to be able to separate arch:all and arch:any > builds, yes. What's stopping you from doing that? I think

Re: meson documentation build open issues

2023-11-03 Thread Andres Freund
Hi, On 2023-11-03 19:19:17 +0100, Christoph Berg wrote: > Re: Andres Freund > > > > You can control this with the "docs" option for meson, as of recently. > > > > > > I've been looking into switching the Debian PG 17 build to meson,

<    1   2   3   4   5   6   7   8   9   10   >