Re: Proposal: Removing 32 bit support starting from PG17++

2023-05-24 Thread Andres Freund
Hi, On 2023-05-24 17:44:36 -0400, Tom Lane wrote: > Andres Freund writes: > > Dropping CPUs without native atomic operations / without a way to do > > tear-free > > 8 byte reads would make several substantial performance improvements easier, > > while not really dro

Re: Atomic ops for unlogged LSN

2023-05-24 Thread Andres Freund
ecause it doesn't feel like > it would be easy to measure the benefit of this patch in isolation. > That's not a reason to reject it, though, just something that makes me > curious. I doubt it's a meaningful, if even measurable win. But removing atomic ops and reducing shared memory space isn't a bad thing, even if there's no immediate benefit... Greetings, Andres Freund

Re: Proposal: Removing 32 bit support starting from PG17++

2023-05-24 Thread Andres Freund
8 byte reads would make several substantial performance improvements easier, while not really dropping any relevant platform. Etc. Greetings, Andres Freund

Re: SyncRepWaitForLSN waits for XLogFlush?

2023-05-24 Thread Andres Freund
t yet been flushed. So a backend with such a spurious SyncRepWaitForLSN() would just wait until the LSN is actually flushed and *then* replicated. Any specific reason for that question? Greetings, Andres Freund

Re: PostgreSQL 16 Beta 1 release announcement draft

2023-05-24 Thread Andres Freund
Q-CONNECT-LOAD-BALANCE-HOSTS) > to libpq, the client library for PostgreSQL. I think the relation extension improvements ought to be mentioned here as well? Up to 3x faster concurrent data load with COPY seems practically relevant. Greetings, Andres Freund

testing dist tarballs

2023-05-24 Thread Andres Freund
Hi, On 2023-05-23 14:51:03 -0400, Tom Lane wrote: > Andres Freund writes: > > I guess I need to go and check how long the "release" tarball generation > > takes... > > It's quick except for the documentation-generating steps. Maybe > we could test that

Re: walsender performance regression due to logical decoding on standby changes

2023-05-24 Thread Andres Freund
Hi, On 2023-05-24 05:53:51 +, Zhijie Hou (Fujitsu) wrote: > On Tuesday, May 23, 2023 1:53 AM Andres Freund wrote: > > On 2023-05-22 12:15:07 +, Zhijie Hou (Fujitsu) wrote: > > > About "a backend doing logical decoding", do you mean the case when a > &

Re: memory leak in trigger handling (since PG12)

2023-05-24 Thread Andres Freund
lived memory context would be more elegant, and > >> more in line with how we do things. But it's not clear to me why we'd > >> end up with the same problem in a few years with what the patch does. > > > > Because it sets up the pattern of manual memory ma

Re: memory leak in trigger handling (since PG12)

2023-05-24 Thread Andres Freund
pping back a bit, why are we doing the work in ExecGetAllUpdatedCols() over and over? Unless I am missing something, the result doesn't change across rows. And it doesn't look that cheap to compute, leaving aside the allocation that bms_union() does. It's visible in profiles, not as a top entry, but still. Perhaps the easiest to backpatch fix is to just avoid recomputing the value? But perhaps it'd be just as problmeatic, because callers might modify ExecGetAllUpdatedCols()'s return value in place... Greetings, Andres Freund

Re: memory leak in trigger handling (since PG12)

2023-05-24 Thread Andres Freund
modifiedCols = bms_union(ExecGetInsertedCols(rootrel, estate), ExecGetUpdatedCols(rootrel, estate)); } Once in ExecPartitionCheckEmitError(), *twice* in ExecConstraints(), ExecWithCheckOptions(). Some of the partitioning stuff has been added in a really myopic way. Greetings, Andres Freund

Re: memory leak in trigger handling (since PG12)

2023-05-24 Thread Andres Freund
ea, that's a valid concern. I think it might be reasonable to use something like ExecGetAllUpdatedColsCtx() in the backbranches, and switch to a short-lived context for the trigger invocations in >= 16. Greetings, Andres Freund

Re: memory leak in trigger handling (since PG12)

2023-05-24 Thread Andres Freund
Hi, On 2023-05-23 23:26:42 +0200, Tomas Vondra wrote: > On 5/23/23 19:14, Andres Freund wrote: > > Hi, > > > > On 2023-05-23 18:23:00 +0200, Tomas Vondra wrote: > >> This means that for an UPDATE with triggers, we may end up calling this > >> for each ro

Re: could not extend file "base/5/3501" with FileFallocate(): Interrupted system call

2023-05-23 Thread Andres Freund
Hi, On 2023-05-24 10:56:28 +0900, Kyotaro Horiguchi wrote: > At Wed, 26 Apr 2023 11:37:55 +1200, Thomas Munro > wrote in > > On Tue, Apr 25, 2023 at 12:16 PM Andres Freund wrote: > > > On 2023-04-24 15:32:25 -0700, Andres Freund wrote: > > > > We

Re: unnecessary #include "pg_getopt.h"?

2023-05-23 Thread Andres Freund
ouched in the patch use optarg, opterr etc. - which are declared in pg_getopt.h. Making it reasonable to directly include pg_getopt.h. I don't really see a need to change anything here? Greetings, Andres Freund

Re: pgsql: Move snowball_create.sql creation into perl file

2023-05-23 Thread Andres Freund
e of those. I guess I need to go and check how long the "release" tarball generation takes... Greetings, Andres Freund

Re: memory leak in trigger handling (since PG12)

2023-05-23 Thread Andres Freund
Hi, On 2023-05-23 13:28:30 -0400, Tom Lane wrote: > Andres Freund writes: > > Could it help to have a mode where the executor shutdown hook checks how > > much > > memory is allocated in ExecutorState and warns if its too much? > > It'd be very hard to set a li

Re: memory leak in trigger handling (since PG12)

2023-05-23 Thread Andres Freund
y to use a shorter lived memory context instead? Otherwise we'll just end up with the same problem in a few years. Greetings, Andres Freund

Re: PG 16 draft release notes ready

2023-05-22 Thread Andres Freund
Hi, On 2023-05-21 22:52:09 -0400, Bruce Momjian wrote: > On Sun, May 21, 2023 at 10:13:41AM -0700, Andres Freund wrote: > > Hi, > > > > Thanks for the release notes! > > > > > > > > > > > > > > > > > Allow more e

Re: PG 16 draft release notes ready

2023-05-22 Thread Andres Freund
ge. > I can see this item as a big win, but I don't know how to describe it in a way > that is helpful for the user to know. In doubt the subject of the commit would just work IMO. Greetings, Andres Freund

Re: walsender performance regression due to logical decoding on standby changes

2023-05-22 Thread Andres Freund
eased immediately. I forgot the details, but IIRC there's some outfuncs (enum?) that intentionally delay releasing locks till transaction commit. Greetings, Andres Freund

Re: pg_stat_io for the startup process

2023-05-21 Thread Andres Freund
case the stats are also not accessible, so that's not a problem. It's not the prettiest solution, but I think the simplicity is worth a lot. Greetings, Andres Freund

Re: PG 16 draft release notes ready

2023-05-21 Thread Andres Freund
Hi, On May 21, 2023 11:46:56 AM PDT, "Jonathan S. Katz" wrote: >On 5/21/23 1:13 PM, Andres Freund wrote: > >> >> Looking through the release notes, I didn't see an entry for >> >> commit c6e0fe1f2a08505544c410f613839664eea9eb21 >> Author:

Re: PG 16 draft release notes ready

2023-05-21 Thread Andres Freund
Hi, Thanks for the release notes! > > > > > Allow more efficient addition of multiple heap and index pages (Andres Freund) > > While the case of extending by multiple pages improved the most, even extending by a single page at a time got a good bit more scalable

Re: walsender performance regression due to logical decoding on standby changes

2023-05-21 Thread Andres Freund
I also did some minor comment polishing. Greetings, Andres Freund

Re: walsender performance regression due to logical decoding on standby changes

2023-05-21 Thread Andres Freund
ess might be blocked waiting for a lock, held by a backend doing logical decoding - which can't progress until the startup process wakes the backend up. So I don't think this is promising avenue in the near term. Greetings, Andres Freund

Re: benchmark results comparing versions 15.2 and 16

2023-05-19 Thread Andres Freund
od > news. My testing scripts have been improved so I should be able to finish > the next round of tests in less time. With "yet to see any significant changes" do you mean that the runs are comparable with earlier runs, showing the same regression? Or that the regression vanished? Or ...? Greetings, Andres Freund

Re: Add operator for dividing interval by an interval

2023-05-19 Thread Andres Freund
Hi, On 2023-05-18 17:03:24 -0400, Tom Lane wrote: > Andres Freund writes: > > What about an interval / interval -> double operator that errors out > > whenever > > month is non-zero? As far as I can tell that would always be deterministic. > > We have month

Add operator for dividing interval by an interval

2023-05-18 Thread Andres Freund
taken for some portion of that (e.g. time spent waiting for IO). What about an interval / interval -> double operator that errors out whenever month is non-zero? As far as I can tell that would always be deterministic. Greetings, Andres Freund

Re: Introduce WAIT_EVENT_EXTENSION and WAIT_EVENT_BUFFER_PIN

2023-05-17 Thread Andres Freund
On 2023-05-17 09:22:19 -0400, Robert Haas wrote: > Adding pgstat_get_wait_extension adds runtime cost for no corresponding > benefit. Having a special case in the code to avoid that seems worthwhile. I don't think that should ever be used in a path where performance is relevant?

Re: issue with meson builds on msys2

2023-05-17 Thread Andres Freund
Hi, On May 17, 2023 2:51:41 PM PDT, Andrew Dunstan wrote: > >On 2023-05-16 Tu 17:52, Andres Freund wrote: >> >>> I suppose the alternative would be to change the way the buildfarm calls >>> pg_ctl stop. Do you have a concrete suggestion for that? >> The easi

Re: walsender performance regression due to logical decoding on standby changes

2023-05-17 Thread Andres Freund
Hi, On 2023-05-17 12:53:15 -0700, Andres Freund wrote: > I'll try to come up with a benchmark without the issues I pointed out in > https://postgr.es/m/20230517194331.ficfy5brpfq5lrmz%40awork3.anarazel.de Here we go: setup: create primary SELECT pg_create_physical_replication_sl

Re: walsender performance regression due to logical decoding on standby changes

2023-05-17 Thread Andres Freund
) callers' life easy. > + * > + * XXX: When available, WaitEventSetWait() can be replaced with its CV's > + * counterpart. I don't really understand that XXX - the potential bright future would be to add support for CVs into wait event sets, not to replace WES with a CV? Greetings, Andres Freund

Re: walsender performance regression due to logical decoding on standby changes

2023-05-17 Thread Andres Freund
gher than the "original" cost of the function. gprof style instrumentation is good for things like code coverage, but for performance evaluation it's normally better to use a sampling profiler like perf. That also causes slowdowns, but largely only in places that already take up substantial execution time. Greetings, Andres Freund

Re: walsender performance regression due to logical decoding on standby changes

2023-05-17 Thread Andres Freund
Hi, On 2023-05-10 08:39:08 +0200, Drouvot, Bertrand wrote: > On 5/9/23 11:00 PM, Andres Freund wrote: > > Hi, > > > > On 2023-05-09 13:38:24 -0700, Jeff Davis wrote: > > > On Tue, 2023-05-09 at 12:02 -0700, Andres Freund wrote: > > > > I don'

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-05-17 Thread Andres Freund
, it sounds like it's pretty close to being ready. Thanks for the nudge. I just pushed the changes, with some very minor changes (a newline, slight changes in commit messages). I'll go and mark the item as closed. Greetings, Andres Freund

Re: smgrzeroextend clarification

2023-05-16 Thread Andres Freund
Hi, On 2023-05-16 20:40:01 +0200, Peter Eisentraut wrote: > On 10.05.23 20:10, Andres Freund wrote: > > > So if you want to understand what smgrzeroextend() does, you need to > > > mentally combine the documentation of three different functions. Could we > > >

Re: WL_SOCKET_ACCEPT fairness on Windows

2023-05-16 Thread Andres Freund
Hi, On 2023-05-17 08:41:24 +1200, Thomas Munro wrote: > Yeah. No one seems to think this is worth worrying about (please > speak up if you do). +1 - we have much bigger fish to fry IMO. Greetings, Andres Freund

Re: issue with meson builds on msys2

2023-05-16 Thread Andres Freund
y want to spend a lot more energy on it. > I suppose the alternative would be to change the way the buildfarm calls > pg_ctl stop. Do you have a concrete suggestion for that? The easiest fix is to redirect stdin to /dev/null (or some file, if that's easier to do portably) - that should fix the problem entirely, without needing IPC::Run. Greetings, Andres Freund

Re: Introduce WAIT_EVENT_EXTENSION and WAIT_EVENT_BUFFER_PIN

2023-05-15 Thread Andres Freund
Hi, On 2023-05-16 09:38:54 +0900, Michael Paquier wrote: > On Mon, May 15, 2023 at 05:17:16PM -0700, Andres Freund wrote: > > IMO the submission should include why automating requires these changes > > (yours > > doesn't really either). I can probably figure it o

Re: benchmark results comparing versions 15.2 and 16

2023-05-15 Thread Andres Freund
better performance when forcing things to be colocated. Greetings, Andres Freund

Re: Introduce WAIT_EVENT_EXTENSION and WAIT_EVENT_BUFFER_PIN

2023-05-15 Thread Andres Freund
Hi, On 2023-05-16 07:30:54 +0900, Michael Paquier wrote: > On Mon, May 15, 2023 at 11:29:56AM -0700, Andres Freund wrote: > > Without an explanation for why this change is needed for [1], it's hard to > > give useful feedback... > > The point is to integrate the wait ev

Re: benchmark results comparing versions 15.2 and 16

2023-05-15 Thread Andres Freund
Hi, On 2023-05-15 14:20:24 +0900, Michael Paquier wrote: > On Thu, May 11, 2023 at 01:28:40PM +0900, Michael Paquier wrote: > > On Tue, May 09, 2023 at 09:48:24AM -0700, Andres Freund wrote: > >> On 2023-05-08 12:11:17 -0700, Andres Freund wrote: > >>> I can reproduc

Re: issue with meson builds on msys2

2023-05-15 Thread Andres Freund
Hi, On 2023-05-15 15:30:28 -0700, Andres Freund wrote: > As soon as either the pg_ctl for the start, or the whole bash invocation, has > stdin redirected, the problem vanishes. For a moment I thought this could be related to InheritStdHandles() - but no, it doesn't make a differenc

Re: issue with meson builds on msys2

2023-05-15 Thread Andres Freund
Hi, On 2023-05-15 13:13:26 -0700, Andres Freund wrote: > It wouldn't really - the echo $? inside the system() would report the > error. Which it doesn't - note the "0" in the second output. Ah. Interesting. Part of the issue is perl (or msys?) swalling some error

Re: issue with meson builds on msys2

2023-05-15 Thread Andres Freund
Hi, On 2023-05-15 16:01:39 -0400, Andrew Dunstan wrote: > On 2023-05-15 Mo 15:38, Andres Freund wrote: > > Hi, > > > > On 2023-05-05 07:08:39 -0400, Andrew Dunstan wrote: > > > If you want to play I can arrange access. > > Andrew did - thanks! > > >

Re: issue with meson builds on msys2

2023-05-15 Thread Andres Freund
ractions with the stuff that InheritstdHandles() does? Andrew, is it ok if modify pg_ctl.c and rebuild? I don't know how "detached" from the actual buildfarm animal the system you gave me access to is... Greetings, Andres Freund

Re: Introduce WAIT_EVENT_EXTENSION and WAIT_EVENT_BUFFER_PIN

2023-05-15 Thread Andres Freund
ww.postgresql.org/message-id/flat/77a86b3a-c4a8-5f5d-69b9-d70bbf2e9...@gmail.com > > Looking forward to your feedback, Without an explanation for why this change is needed for [1], it's hard to give useful feedback... Greetings, Andres Freund

Re: smgrzeroextend clarification

2023-05-12 Thread Andres Freund
Hi, On May 12, 2023 11:36:23 AM PDT, Thomas Munro wrote: >Just a thought: should RelationCopyStorageUsingBuffer(), the new code >used by CREATE DATABASE with the default strategy WAL_LOG, use the >newer interface so that it creates fully allocated files instead of >sparse ones? I played with th

Re: smgrzeroextend clarification

2023-05-12 Thread Andres Freund
Hi, On May 11, 2023 2:37:00 AM PDT, Peter Eisentraut wrote: >On 10.05.23 20:10, Andres Freund wrote: >>> Moreover, the text "except the relation can be extended by multiple blocks >>> at once and the added blocks will be filled with zeroes" doesn't make

Re: smgrzeroextend clarification

2023-05-10 Thread Andres Freund
nd() does #2 - it just does it by writing data. The FileFallocate() path in smgrzeroextend() tries to avoid writing data if extending by sufficient blocks - not having dirty data in the kernel page cache can substantially reduce the IO usage. Whereas the FileZero() path just optimizes the number of syscalls (and cache misses etc). Greetings, Andres Freund

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-05-10 Thread Andres Freund
Hi, On 2023-05-10 17:44:07 +0200, Peter Eisentraut wrote: > On 12.03.23 00:41, Andres Freund wrote: > > Hi, > > > > On 2023-03-11 15:34:55 -0800, Mark Dilger wrote: > > > > On Mar 11, 2023, at 3:22 PM, Andres Freund wrote: > > > > > > > &

Re: walsender performance regression due to logical decoding on standby changes

2023-05-09 Thread Andres Freund
Hi, On 2023-05-09 13:38:24 -0700, Jeff Davis wrote: > On Tue, 2023-05-09 at 12:02 -0700, Andres Freund wrote: > > I don't think the approach of not having any sort of "registry" of > > whether > > anybody is waiting for the replay position to be updated

Re: Feature: Add reloption support for table access method

2023-05-09 Thread Andres Freund
.) > 2. ALTER TABLE ... SET ... > 3. ALTER TABLE ... RESET ... > The reason why the context matters is that some reloptions are disallowed to > change after creating > the table, while some reloptions are allowed. What kind of reloption are you thinking of here? Greetings, Andres Freund

walsender performance regression due to logical decoding on standby changes

2023-05-09 Thread Andres Freund
Hi, Unfortunately I have found the following commit to have caused a performance regression: commit e101dfac3a53c20bfbf1ca85d30a368c2954facf Author: Andres Freund Date: 2023-04-08 00:24:24 -0700 For cascading replication, wake physical and logical walsenders separately Physical

Re: benchmark results comparing versions 15.2 and 16

2023-05-09 Thread Andres Freund
Hi, On 2023-05-08 12:11:17 -0700, Andres Freund wrote: > Hi, > > On 2023-05-08 16:00:01 +0300, Alexander Lakhin wrote: > > This difference is confirmed by multiple test runs. `git bisect` for this > > regression pointed at f193883fc. > > I can reproduce a significant

Re: DROP DATABASE is interruptible

2023-05-08 Thread Andres Freund
using (perhaps it'd make more sense if 'invalid because creating' and > 'invalid because dropping' were distinguished). I'm bit hesitant to do so for now. Once it's a bit more settled, maybe? Although I wonder if there's something better suited to the task than autovacuum. Greetings, Andres Freund

Re: pg_stat_io for the startup process

2023-05-08 Thread Andres Freund
r a > subset of it) right before invoking WaitLatch and at segment > boundaries. I've pondered that as well. But I don't think it's great - it's not exactly intuitive that stats reporting gets far less common if you use a 1GB wal_segment_size. Greetings, Andres Freund

Re: base backup vs. concurrent truncation

2023-05-08 Thread Andres Freund
Hi, On 2023-04-25 10:28:58 -0700, Andres Freund wrote: > On 2023-04-25 11:42:43 -0400, Robert Haas wrote: > > On Mon, Apr 24, 2023 at 8:03 PM Andres Freund wrote: > > > What we've discussed somewhere in the past is to always truncate N+1 when > > > creating the fi

Re: base backup vs. concurrent truncation

2023-05-08 Thread Andres Freund
will determine the relation size to figure out what segments to remove. Because is < SEGSIZE it'll only truncate , not .N. And boom, a disconnected segment. (I'll post a separate email about an evolved proposal about fixing this set of issues) Greetings, Andres Freund

Re: issue with meson builds on msys2

2023-05-08 Thread Andres Freund
Hi, On 2023-05-05 07:08:39 -0400, Andrew Dunstan wrote: > On 2023-05-04 Th 19:54, Andres Freund wrote: > > Hm. I can't reproduce this in my test win10 VM, unfortunately. What OS / OS > > version is the host? Any chance to get systeminfo.exe output or something > > li

Re: [PATCH] Add native windows on arm64 support

2023-05-08 Thread Andres Freund
don't really have feelings either way - but haven't we gone further and even backpatched things like spinlock support for new arches in the past? Greetings, Andres Freund

Re: benchmark results comparing versions 15.2 and 16

2023-05-08 Thread Andres Freund
of something dedicated. We also now need to do permission checking etc. I don't think that's the entirety of the regression... Greetings, Andres Freund

Re: benchmark results comparing versions 15.2 and 16

2023-05-05 Thread Andres Freund
lt at 70s, 15 then has one low, but 16 has two low results. Greetings, Andres Freund

Re: issue with meson builds on msys2

2023-05-04 Thread Andres Freund
Hi, On 2023-05-03 09:20:28 -0400, Andrew Dunstan wrote: > On 2023-04-27 Th 18:18, Andres Freund wrote: > > On 2023-04-26 09:59:05 -0400, Andrew Dunstan wrote: > > > Still running into this, and I am rather stumped. This is a blocker for > > > buildfarm support for

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-05-04 Thread Andres Freund
a backend type basis. I think the startup process will use the normal backend logic, and thus also trigger writebacks? Greetings, Andres Freund

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-05-04 Thread Andres Freund
houldn't be too hard. I'll try to look more tomorrow. Greetings, Andres Freund

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-05-04 Thread Andres Freund
ack(&BackendWritebackContext, > > &buf_hdr->tag); > } What about passing the io_context to ScheduleBufferTagForWriteback instead? > --- a/src/test/regress/sql/stats.sql > +++ b/src/test/regress/sql/stats.sql Hm. Could we add a test for this? While it's not implemented everywhere, we still issue the smgrwriteback() afaics. The default for the _flush_after GUCs changes, but backend_flush_after is USERSET, so we could just change it for a single command. Greetings, Andres Freund

Re: refactoring relation extension and BufferAlloc(), faster COPY

2023-05-03 Thread Andres Freund
r COPY copytest_0 FROM '/tmp/copytest_data_binary.copy'; > Also what are the 'seconds' and 'tbl-MBs' metrics that were reported. The total time for inserting N (1024 for the small files, 64 for the larger ones). "tbl-MBs" is size of the resulting table, divided by time. I.e. a measure of throughput. Greetings, Andres Freund

Re: refactoring relation extension and BufferAlloc(), faster COPY

2023-05-03 Thread Andres Freund
'/tmp/copytest_data_text.copy' WITH (FORMAT test); > [2] COPY (SELECT repeat(random()::text, 5) FROM generate_series(1, 6*10)) > TO '/tmp/copytest_data_text.copy' WITH (FORMAT text); Greetings, Andres Freund

Re: issue with meson builds on msys2

2023-05-03 Thread Andres Freund
Hi, On 2023-05-03 09:20:28 -0400, Andrew Dunstan wrote: > On 2023-04-27 Th 18:18, Andres Freund wrote: > > Hi, > > > > On 2023-04-26 09:59:05 -0400, Andrew Dunstan wrote: > > > Still running into this, and I am rather stumped. This is a blocker for >

Re: issue with meson builds on msys2

2023-04-27 Thread Andres Freund
case I had some circumstantial evidence that the problem was dependent on the windows runtime version. Downthread you mention that the issue doesn't happen with IPC::Run - the biggest difference I can see is that IPC::Run would IIRC not use a shell? Does the problem "re-appear" if you make IPC::Run use a shell? Greetings, Andres Freund

Re: pg_stat_io for the startup process

2023-04-27 Thread Andres Freund
/* _STARTUP_H */ > diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h > index e561a1cde9..a8d360e255 100644 > --- a/src/include/utils/timeout.h > +++ b/src/include/utils/timeout.h > @@ -35,6 +35,7 @@ typedef enum TimeoutId > IDLE_STATS_UPDATE_TIMEOUT, > CLIENT_CONNECTION_CHECK_TIMEOUT, > STARTUP_PROGRESS_TIMEOUT, > + STARTUP_STAT_FLUSH_TIMEOUT, > /* First user-definable timeout reason */ > USER_TIMEOUT, > /* Maximum number of timeout reasons */ I think we could just reuse IDLE_STATS_UPDATE_TIMEOUT? Greetings, Andres Freund

Re: pg_stat_io for the startup process

2023-04-25 Thread Andres Freund
Hi, On 2023-04-25 16:00:24 -0400, Robert Haas wrote: > On Tue, Apr 25, 2023 at 2:39 PM Andres Freund wrote: > > I'm mildly inclined to not consider it a bug, given that this looks to have > > been true for other stats for quite a while? But it does still seem worth > >

Re: pg_stat_io for the startup process

2023-04-25 Thread Andres Freund
artup process on the standby. > Elsewhere on the internet where you originally raised this, I mentioned > that I hacked a pgstat_flush_io() into the redo apply loop in > PerformWalRecovery() but that I wasn't sure that this was affordable. > Andres Freund replied saying that it wou

Re: base backup vs. concurrent truncation

2023-04-25 Thread Andres Freund
Hi, On 2023-04-25 11:42:43 -0400, Robert Haas wrote: > On Mon, Apr 24, 2023 at 8:03 PM Andres Freund wrote: > > What we've discussed somewhere in the past is to always truncate N+1 when > > creating the first page in N. I.e. if we extend into 23456.1, we truncate > > 23

Re: could not extend file "base/5/3501" with FileFallocate(): Interrupted system call

2023-04-24 Thread Andres Freund
Hi, On 2023-04-24 15:32:25 -0700, Andres Freund wrote: > On 2023-04-24 10:53:35 +0200, Christoph Berg wrote: > > I'm often seeing PG16 builds erroring out in the pgbench tests: > > I don't think the disk is full since it's always hitting that same > > spot,

Re: base backup vs. concurrent truncation

2023-04-24 Thread Andres Freund
ward also doesn't sound particularly > cheap. Nonetheless I don't have another idea right now. What we've discussed somewhere in the past is to always truncate N+1 when creating the first page in N. I.e. if we extend into 23456.1, we truncate 23456.2 to 0 blocks. As far as I can tell, that'd solve this issue? Greetings, Andres Freund

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-04-24 Thread Andres Freund
Hi, On 2023-04-24 18:36:24 -0400, Melanie Plageman wrote: > On Mon, Apr 24, 2023 at 6:13 PM Andres Freund wrote: > > > Also, it seems like this (given the current code) is only reachable for > > > permanent relations (i.e. not for IO object temp relation). If other > &g

Re: could not extend file "base/5/3501" with FileFallocate(): Interrupted system call

2023-04-24 Thread Andres Freund
fd.c having code to retry: - FileWriteback() / pg_flush_data() - FileSync() / pg_fsync() - FileFallocate() - FileTruncate() With the first two there's the added complication that it's not entirely obvious whether it'd be better to handle this in File* or pg_*. I'd argue the latter is a bit more sensible? Greetings, Andres Freund

Re: Remove io prefix from pg_stat_io columns

2023-04-24 Thread Andres Freund
On 2023-04-21 07:38:01 +0900, Michael Paquier wrote: > On Thu, Apr 20, 2023 at 11:38:42AM +0900, Michael Paquier wrote: > > No worries, committers should take care of that. > > Done as of 0ecb87e, as I can keep an eye on the buildfarm today, with > a catversion bump. Thanks!

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-04-24 Thread Andres Freund
an get > this in PG16 or not. If this is just adding a new column and a few > existing stats update calls then it should be okay to get in but if > this requires some more complex work then we can probably update the > docs. I suspect it should really just be adding a few stats calls. The only possible complication that I can see is that we might need to pass a bit more context down in a place or two. Greetings, Andres Freund

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-04-24 Thread Andres Freund
Hi, On 2023-04-24 17:37:48 -0400, Melanie Plageman wrote: > On Mon, Apr 24, 2023 at 02:14:32PM -0700, Andres Freund wrote: > > It starts blocking once "enough" IO is in flight. For things like an > > immediate > > checkpoint, that can happen fairly quickly,

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-04-24 Thread Andres Freund
Hi, On 2023-04-24 16:39:36 -0400, Melanie Plageman wrote: > On Wed, Apr 19, 2023 at 10:23:26AM -0700, Andres Freund wrote: > > Hi, > > > > I noticed that the numbers in pg_stat_io dont't quite add up to what I > > expected in write heavy workloads. Particularly fo

Re: Should we remove vacuum_defer_cleanup_age?

2023-04-24 Thread Andres Freund
Hi, On 2023-04-24 14:36:36 +0200, Alvaro Herrera wrote: > On 2023-Apr-22, Andres Freund wrote: > > I'm afraid we'll need TransactionIdRetreatSafely() again, when we convert > > more > > things to 64bit xids (lest they end up with the same bug as fixed by > >

Re: Should we remove vacuum_defer_cleanup_age?

2023-04-22 Thread Andres Freund
Hi, On 2023-04-13 13:18:38 +0200, Alvaro Herrera wrote: > On 2023-Apr-11, Andres Freund wrote: > > > Updated patch attached. I think we should either apply something like that > > patch, or at least add a to the docs. > > I gave this patch a look. The

Re: LLVM strip -x fails

2023-04-20 Thread Andres Freund
Hi, On 2023-04-20 12:43:48 -0400, Tom Lane wrote: > Andres Freund writes: > > Afaict the only safe thing to use when stripping static libs is > > -g/--strip-debug. > > The previous complaint about this [1] suggested we use --strip-unneeded > for all cases with GNU strip,

Re: LLVM strip -x fails

2023-04-20 Thread Andres Freund
Date: 2002-04-10 16:45:25 + Add make install-strip target. Afaict the only safe thing to use when stripping static libs is -g/--strip-debug. Greetings, Andres Freund

Re: Wrong results from Parallel Hash Full Join

2023-04-19 Thread Andres Freund
ependant in > various ways (which is why the tests use "explain analyze so rarely). I think with sufficient options it's not machine specific. We have a bunch of EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF) .. in our tests. Greetings, Andres Freund

Re: Direct I/O

2023-04-19 Thread Andres Freund
Hi, On 2023-04-19 13:16:54 -0400, Robert Haas wrote: > On Wed, Apr 19, 2023 at 12:54 PM Andres Freund wrote: > > Interestingly, I haven't seen that as much in more recent benchmarks as it > > used to. Partially I think because common s_b settings have gotten bigger, I > &

pg_stat_io not tracking smgrwriteback() is confusing

2023-04-19 Thread Andres Freund
seems pretty clear that we should track writeback as well. I wonder if it's worth doing so for 16? It'd give a more complete picture that way. The counter-argument I see is that we didn't track the time for it in existing stats either, and that nobody complained - but I suspect that&#x

Re: Direct I/O

2023-04-19 Thread Andres Freund
e relevant options like DIO is much more likely to cause problems, due to the drastically different performance characteristics you get. I can see us making it configurable to try using DIO though, but I am not convinced it's worth bothering with that. But we'll see. Greetings, Andres Freund

Re: Direct I/O

2023-04-19 Thread Andres Freund
they ever did in their bug tracker. Agreed. If we ever find such issues in a postmordial compiler, we'll just need to beef up our configure test to detect that it doesn't actually fully support specifying alignment. Greetings, Andres Freund

Re: Direct I/O

2023-04-19 Thread Andres Freund
sizing aren't particularly hard, I think. Instead of actually changing the s_b shmem allocation - which would be very hard in a process based model - we can tell the kernel that some parts of that memory aren't currently in use with madvise(MADV_REMOVE). It's not quite as trivial as it sounds, because we'd have to free in multiple of huge_page_size. Greetings, Andres Freund

Re: v16dev: TRAP: failed Assert("size > SizeOfXLogRecord"), File: "xlog.c", Line: 1055, PID: 13564

2023-04-17 Thread Andres Freund
uce further back? > > Maybe the state where all the pages are new didn't happen before? > > Bingo: bisecting shows the failure started at > > commit 3d6a98457d8e21d85bed86cfd3e1d1df1b260721 > Author: Andres Freund > Date: Wed Apr 5 08:19:39 2023 -0700 > >

Re: v16dev: TRAP: failed Assert("size > SizeOfXLogRecord"), File: "xlog.c", Line: 1055, PID: 13564

2023-04-17 Thread Andres Freund
o bulk extend if we know that we'll insert multiple pages (when coming from heap_multi_insert(), with sufficient data). Greetings, Andres Freund

Re: Direct I/O

2023-04-14 Thread Andres Freund
Hi, On 2023-04-14 15:21:18 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2023-04-14 13:21:33 -0400, Tom Lane wrote: > >> ... I'm not sure why only those two animals > >> are unhappy, but I think they have a point: typical ABIs don't > >> g

Re: Direct I/O

2023-04-14 Thread Andres Freund
whether the compiler is OK with that, and disable direct IO support if not? Greetings, Andres Freund

Re: Assertion being hit during WAL replay

2023-04-14 Thread Andres Freund
Hi, On 2023-04-11 15:03:02 -0700, Andres Freund wrote: > On 2023-04-11 16:54:53 -0400, Tom Lane wrote: > > Here's something related to what I hit that time: > > > > diff --git a/src/backend/optimizer/plan/subselect.c > > b/src/backend/optimizer/plan/subselect.c

Re: refactoring relation extension and BufferAlloc(), faster COPY

2023-04-14 Thread Andres Freund
Hi, On 2023-04-12 08:00:00 +0300, Alexander Lakhin wrote: > 12.04.2023 02:21, Andres Freund wrote: > > Hi, > > > > On 2023-04-11 22:00:00 +0300, Alexander Lakhin wrote: > > > A few days later I've found a new defect introduced with 31966b151. > > That

Re: Temporary tables versus wraparound... again

2023-04-14 Thread Andres Freund
etical to the "otherwise", which doesn't make a whole lot of sense. How about: * If rel is not NULL the horizon may be considerably more recent (i.e. * allowing more tuples to be removed) than otherwise. In the NULL case a * horizon that is correct (but not optimal) for all relations will be * returned. Thus, if possible, a relation should be provided. Greetings, Andres Freund

<    7   8   9   10   11   12   13   14   15   16   >