Re: killing perl2host

2022-02-20 Thread Thomas Munro
On Sun, Feb 20, 2022 at 10:51 AM Andrew Dunstan wrote: > On 2/19/22 16:34, Andres Freund wrote: > >> The first > >> removes perl2host completely and adjusts all the places that use it. > >> The second removes almost all the remaining msys special processing in > >> TAP tests. Very nice improveme

Re: Missed condition-variable wakeups on FreeBSD

2022-02-27 Thread Thomas Munro
On Sun, Feb 27, 2022 at 8:07 AM Tom Lane wrote: > I don't know much about how gdb interacts with kernel calls on > FreeBSD, but I speculate that the poll(2) call returns with EINTR > after gdb releases the process, and then things resume fine, Yeah, at least FreeBSD and macOS interrupt system cal

Re: Checkpointer sync queue fills up / loops around pg_usleep() are bad

2022-02-27 Thread Thomas Munro
L); + WaitLatch(NULL, WL_EXIT_ON_PM_DEATH | WL_TIMEOUT, 10, WAIT_EVENT_SYNC_REQUEST); But... really, this should be waiting on a condition variable that the checkpointer broadcasts on when the queue goes from full to not full, no? Perhaps for master only? From 23abdf95dea74b914dc56a46739

Re: Missed condition-variable wakeups on FreeBSD

2022-02-28 Thread Thomas Munro
On Sun, Feb 27, 2022 at 11:18 AM Melanie Plageman wrote: > How could it be that worker 2 is waiting on the build barrier in > PHJ_BUILD_HASHING_INNER and worker 1 and the leader are waiting on it > with it supposedly in PHJ_BUILD_HASHING_OUTER? That'd be consistent with a wakeup going missing, so

Re: Missed condition-variable wakeups on FreeBSD

2022-02-28 Thread Thomas Munro
On Sun, Feb 27, 2022 at 9:44 AM Andres Freund wrote: > > (gdb) p debug_query_string > > $1 = 0x21873090 "select count(*) from simple r join simple s using (id);" > > (gdb) bt > > #0 _poll () at _poll.S:4 > > #1 0x21701361 in __thr_poll (fds=0x219dc170, nfds=2, timeout=-1) at > > /usr/src/lib/li

Re: Checkpointer sync queue fills up / loops around pg_usleep() are bad

2022-03-01 Thread Thomas Munro
On Mon, Feb 28, 2022 at 2:36 PM Andres Freund wrote: > On February 27, 2022 4:19:21 PM PST, Thomas Munro > wrote: > >It seems a little strange to introduce a new wait event that will very > >often appear into a stable branch, but ... it is actually telling the > >truth,

Re: pg_tablespace_location() failure with allow_in_place_tablespaces

2022-03-04 Thread Thomas Munro
y break something on Windows? From 6f001ec46c5e5f6ffa8e103f2b0d711e6904b763 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Fri, 4 Mar 2022 23:07:46 +1300 Subject: [PATCH] Fix warning in basebackup of in-place tablespaces. While building the tablespace map for a base backup, don't call readlink() on direct

Re: Regression tests failures on Windows Server 2019 - on master at commit # d816f366b

2022-03-04 Thread Thomas Munro
On Fri, Mar 4, 2022 at 10:37 PM Bharath Rupireddy wrote: > It looks like regression tests are failing on Windows Server 2019 [1]. > Ignore if it's reported elsewhere. It's broken since 46ab07ff "Clean up assorted failures under clang's -fsanitize=undefined checks.": https://github.com/postgres/p

Re: Regression tests failures on Windows Server 2019 - on master at commit # d816f366b

2022-03-04 Thread Thomas Munro
On Sat, Mar 5, 2022 at 7:10 AM Andres Freund wrote: > Or perhaps pg_ctl ought to pass CREATE_NEW_PROCESS_GROUP to CreateProcess()? > The lack of a process group would explain why we're getting signalled on > ctrl-c... I thought that sounded promising, given that the recent Cirrus agent commit you

Re: Regression tests failures on Windows Server 2019 - on master at commit # d816f366b

2022-03-04 Thread Thomas Munro
On Sat, Mar 5, 2022 at 10:56 AM Andres Freund wrote: > I suspect one also needs the console detach thing. I tried adding DETACHED_PROCESS (which should be like calling FreeConsole() in child process) and then I tried CREATE_NEW_CONSOLE instead, on top of CREATE_NEW_PROCESS_GROUP. Neither helped,

Re: Regression tests failures on Windows Server 2019 - on master at commit # d816f366b

2022-03-04 Thread Thomas Munro
On Sat, Mar 5, 2022 at 4:19 PM Andres Freund wrote: > https://github.com/cirruslabs/cirrus-ci-agent/issues/218#issuecomment-1059657781 Oof. Nice detective work. > As indicated in the message above it, there's a workaround. Not sure if worth > committing, if they were to fix it in a few days? c

Re: pg_tablespace_location() failure with allow_in_place_tablespaces

2022-03-07 Thread Thomas Munro
On Tue, Mar 8, 2022 at 12:58 AM Michael Paquier wrote: > On Fri, Mar 04, 2022 at 11:26:43PM +1300, Thomas Munro wrote: > > + /* Skip in-place tablespaces (testing use only) */ > > + if (get_dirent_type(fullpath, de, false, ERROR) == PGFILETYPE_DIR) > > +

Re: pg_tablespace_location() failure with allow_in_place_tablespaces

2022-03-07 Thread Thomas Munro
On Tue, Mar 8, 2022 at 10:39 AM Thomas Munro wrote: > Test on a Win10 VM. Erm, "Tested" (as in, I tested), I meant to write...

Re: Adding CI to our tree

2022-03-09 Thread Thomas Munro
On Thu, Mar 10, 2022 at 9:37 AM Justin Pryzby wrote: > On Wed, Mar 09, 2022 at 10:12:54AM -0800, Andres Freund wrote: > > On 2022-03-09 11:47:23 -0600, Justin Pryzby wrote: > > > I'm curious what you think of this patch. > > > > > > It makes check-world on freebsd over 30% faster - saving 5min. >

Re: Adding CI to our tree

2022-03-09 Thread Thomas Munro
On Thu, Mar 10, 2022 at 4:33 PM Andres Freund wrote: > On 2022-03-10 15:43:16 +1300, Thomas Munro wrote: > > I'm confused. > > The "terrible IO wait" thing was before we reduced the number of CPUs and > concurrent jobs. It makes sense to me that with just two CPUs

Re: WIP: WAL prefetch (another approach)

2022-03-10 Thread Thomas Munro
On Fri, Mar 11, 2022 at 6:31 PM Thomas Munro wrote: > Thanks for your review of 0001! It gave me a few things to think > about and some good improvements. And just in case it's useful, here's what changed between v21 and v22.. diff --git a/src/backend/access/transam/xlogreader.

Re: pg_tablespace_location() failure with allow_in_place_tablespaces

2022-03-14 Thread Thomas Munro
On Tue, Mar 8, 2022 at 4:01 PM Kyotaro Horiguchi wrote: > At Tue, 8 Mar 2022 10:28:46 +0900, Michael Paquier > wrote in > > On Tue, Mar 08, 2022 at 10:06:50AM +0900, Kyotaro Horiguchi wrote: > > > At Tue, 8 Mar 2022 10:39:06 +1300, Thomas Munro > > > wrote

Re: pg_tablespace_location() failure with allow_in_place_tablespaces

2022-03-14 Thread Thomas Munro
On Tue, Mar 15, 2022 at 2:33 PM Thomas Munro wrote: > As for the complaint about pg_tablespace_location() failing, would it > be better to return an empty string? That's what was passed in as > LOCATION. Something like the attached. (Hrrmm, the contract for pgwin32_is_junction

Re: pg_tablespace_location() failure with allow_in_place_tablespaces

2022-03-14 Thread Thomas Munro
On Tue, Mar 15, 2022 at 2:50 PM Michael Paquier wrote: > On Tue, Mar 15, 2022 at 02:33:17PM +1300, Thomas Munro wrote: > > As for the complaint about pg_tablespace_location() failing, would it > > be better to return an empty string? That's what was passed in as > > LOC

Re: pg_tablespace_location() failure with allow_in_place_tablespaces

2022-03-15 Thread Thomas Munro
On Tue, Mar 15, 2022 at 10:30 PM Michael Paquier wrote: > So, which one of a relative path or an absolute path do you think > would be better for the user? My preference tends toward the relative > path, as we know that all those tablespaces stay in pg_tblspc/ so one > can make the difference wit

Re: Checkpointer sync queue fills up / loops around pg_usleep() are bad

2022-03-15 Thread Thomas Munro
On Wed, Mar 2, 2022 at 10:58 AM Andres Freund wrote: > On 2022-03-02 06:46:23 +1300, Thomas Munro wrote: > > From a9344bb2fb2a363bec4be526f87560cb212ca10b Mon Sep 17 00:00:00 2001 > > From: Thomas Munro > > Date: Mon, 28 Feb 2022 11:27:05 +1300 > > Subject: [PATCH v2

Re: USE_BARRIER_SMGRRELEASE on Linux?

2022-03-15 Thread Thomas Munro
On Sat, Feb 19, 2022 at 7:01 AM Nathan Bossart wrote: > Here is a new patch. This is essentially the same as v1, but I've spruced > up the comments and the commit message. Hi Nathan, Pushed and back-patched (it's slightly different before 12). Thanks!

Re: pg_tablespace_location() failure with allow_in_place_tablespaces

2022-03-16 Thread Thomas Munro
On Thu, Mar 17, 2022 at 3:53 PM Michael Paquier wrote: > > On Wed, Mar 16, 2022 at 05:15:58PM +0900, Kyotaro Horiguchi wrote: > > I'm not sure that the "of the symbolic link in pg_tblspc/" is > > needed. And allow_in_place_tablespaces alone doesn't create in-place > > tablespace. So this might nee

Re: pg_tablespace_location() failure with allow_in_place_tablespaces

2022-03-16 Thread Thomas Munro
On Thu, Mar 17, 2022 at 7:18 PM Michael Paquier wrote: > On Thu, Mar 17, 2022 at 04:34:30PM +1300, Thomas Munro wrote: > > I think what Horiguchi-san was pointing out above is that you need to > > enable the GUC *and* say LOCATION '', which the new paragraph doesn&#x

Re: WIP: WAL prefetch (another approach)

2022-03-17 Thread Thomas Munro
On Mon, Mar 14, 2022 at 8:17 PM Julien Rouhaud wrote: > Great! I'm happy with 0001 and I think it's good to go! I'll push 0001 today to let the build farm chew on it for a few days before moving to 0002.

Re: Declare PG_HAVE_8BYTE_SINGLE_COPY_ATOMICITY for aarch64

2022-03-17 Thread Thomas Munro
On Thu, Mar 17, 2022 at 1:32 AM Yura Sokolov wrote: > So I believe it is safe to define PG_HAVE_8BYTE_SINGLE_COPY_ATOMICITY > for aarch64 Agreed, and pushed. There was another thread that stalled, so I added a reference and a reviewer from that to your commit message. This should probably also

Re: Probable CF bot degradation

2022-03-18 Thread Thomas Munro
On Sat, Mar 19, 2022 at 5:07 AM Julien Rouhaud wrote: > On Fri, Mar 18, 2022 at 07:43:47PM +0400, Pavel Borisov wrote: > > Hi, hackers! > > I've noticed that CF bot hasn't been running active branches from yesterday: > > https://github.com/postgresql-cfbot/postgresql/branches/active > > > > Also,

Re: [PATCH] Add native windows on arm64 support

2022-03-18 Thread Thomas Munro
On Wed, Feb 23, 2022 at 11:09 PM Niyas Sait wrote: > I have created a patch that adds support for building Postgres for the > windows/arm64 platform using the MSVC toolchain. Following changes have been > included > > 1. Extend MSVC scripts to handle ARM64 platform. > 2. Add arm64 definition of

Re: Probable CF bot degradation

2022-03-18 Thread Thomas Munro
On Sat, Mar 19, 2022 at 9:41 AM Pavel Borisov wrote: >> >> remote: Internal Server Error >> To github.com:postgresql-cfbot/postgresql.git >> ! [remote rejected] commitfest/37/3489 -> commitfest/37/3489 >> (Internal Server Error) >> error: failed to push some refs to >> 'github.com:postgres

Re: A test for replay of regression tests

2022-03-19 Thread Thomas Munro
Another failure under 027_stream_regress.pl: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2022-03-16%2005%3A58%3A05 vacuum ... FAILED 3463 ms I'll try to come up with the perl needed to see the regression.diffs next time...

Re: WIP: WAL prefetch (another approach)

2022-03-19 Thread Thomas Munro
On Fri, Mar 18, 2022 at 9:59 AM Thomas Munro wrote: > I'll push 0001 today to let the build farm chew on it for a few days > before moving to 0002. Clearly 018_wal_optimize.pl is flapping and causing recoveryCheck to fail occasionally, but that predates the above commit. I didn&#

Re: WIP: WAL prefetch (another approach)

2022-03-20 Thread Thomas Munro
On Sun, Mar 20, 2022 at 5:36 PM Thomas Munro wrote: > Clearly 018_wal_optimize.pl is flapping Correction, 019_replslot_limit.pl, discussed at https://www.postgresql.org/message-id/flat/83b46e5f-2a52-86aa-fa6c-8174908174b8%40iki.fi .

Re: A test for replay of regression tests

2022-03-20 Thread Thomas Munro
On Sun, Mar 20, 2022 at 5:20 PM Thomas Munro wrote: > Another failure under 027_stream_regress.pl: > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2022-03-16%2005%3A58%3A05 > > vacuum ... FAILED 3463 ms > > I'l

Re: A test for replay of regression tests

2022-03-20 Thread Thomas Munro
On Mon, Mar 21, 2022 at 2:34 AM Andrew Dunstan wrote: > On 3/20/22 05:36, Thomas Munro wrote: > > On Sun, Mar 20, 2022 at 5:20 PM Thomas Munro wrote: > >> I'll try to come up with the perl needed to see the regression.diffs > >> next time... > > Here'

Re: Probable CF bot degradation

2022-03-20 Thread Thomas Munro
On Mon, Mar 21, 2022 at 1:58 AM Matthias van de Meent wrote: > Would you know how long the expected bitrot re-check period for CF > entries that haven't been updated is, or could the bitrot-checking > queue be displayed somewhere to indicate the position of a patch in > this queue? I see that you

Re: Probable CF bot degradation

2022-03-20 Thread Thomas Munro
On Mon, Mar 21, 2022 at 12:23 PM Thomas Munro wrote: > On Mon, Mar 21, 2022 at 1:58 AM Matthias van de Meent > wrote: > > Would you know how long the expected bitrot re-check period for CF > > entries that haven't been updated is, or could the bitrot-checking > > qu

Re: Probable CF bot degradation

2022-03-20 Thread Thomas Munro
On Mon, Mar 21, 2022 at 1:41 PM Peter Geoghegan wrote: > BTW, I think that the usability of the CFBot website would be improved > if there was a better visual indicator of what each "green tick inside > a circle" link actually indicates -- what are we testing for each > green tick/red X shown? > >

Re: Probable CF bot degradation

2022-03-20 Thread Thomas Munro
On Mon, Mar 21, 2022 at 3:11 PM Andres Freund wrote: > On 2022-03-21 14:44:55 +1300, Thomas Munro wrote: > > Those tooltips come from the "name" elements of the .cirrus.yml file > > where tasks are defined, with Cirrus's task status appended. If we > > had a s

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-03-20 Thread Thomas Munro
ation option to be specified"); + goto bad_argument; + } + if ((optind + 2) < argc) { pg_log_error("too many command-line arguments (first is \"%s\")", @@ -1148,6 +1261,19 @@ main(int argc, char **argv) config.filter_by_xid != record->xl_xid) conti

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-03-20 Thread Thomas Munro
On Mon, Mar 21, 2022 at 4:36 PM Thomas Munro wrote: > On Sat, Feb 26, 2022 at 7:58 AM David Christensen > wrote: > > Attached is V2 with additional feedback from this email, as well as the > > specification of the > > ForkNumber and FPW as specifiable options. > &

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-03-21 Thread Thomas Munro
waldump. This feature allows you to only output records that are touch a specific RelFileNode and optionally BlockNumber and ForkNum in this relation. We also add the independent ability to filter Full Page Write records. Author: David Christensen Reviewed-by: Peter Geoghegan Reviewed-by:

Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

2022-03-21 Thread Thomas Munro
I have a new socket abstraction patch that should address the known Windows socket/event bugs, but it's a little bigger than I thought it would be, not quite ready, and now too late to expect people to review for 15, so I think it should go into the next cycle. I've bounced https://commitfest.post

Broken make dependency somewhere near win32ver.o?

2022-03-21 Thread Thomas Munro
Hi, Here's a strange one-off failure seen on CI[1], in the CompilerWarnings task where we check that mingw cross-compile works: [10:48:29.045] time make -s -j${BUILD_JOBS} world-bin [10:48:38.705] x86_64-w64-mingw32-gcc: error: win32ver.o: No such file or directory [10:48:38.705] make[3]: *** [Ma

Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

2022-03-21 Thread Thomas Munro
On Tue, Mar 22, 2022 at 4:13 PM Tom Lane wrote: > Thomas Munro writes: > > I have a new socket abstraction patch that should address the known > > Windows socket/event bugs, but it's a little bigger than I thought it > > would be, not quite ready, and now too late

Re: A test for replay of regression tests

2022-03-21 Thread Thomas Munro
Well, at least the first one. Here's a patch like that. From ba05f03c202bf66c7692787ec24ece13b193a897 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Tue, 22 Mar 2022 16:54:42 +1300 Subject: [PATCH] Try to stabilize vacuum test. As commits b700f96c and 3414099c did for the reloptions te

Re: New Object Access Type hooks

2022-03-21 Thread Thomas Munro
On Fri, Mar 18, 2022 at 4:22 PM Mark Dilger wrote: > (FYI, I got a test failure from src/test/recovery/t/013_crash_restart.pl when > testing v1-0001. I'm not sure yet what that is about.) Doesn't look like 0001 has anything to do with that... Are you on a Mac? Did it look like this recent fa

Re: Broken make dependency somewhere near win32ver.o?

2022-03-21 Thread Thomas Munro
On Tue, Mar 22, 2022 at 4:14 PM Andres Freund wrote: > The problem looks to be that pg_dumpall doesn't have a dependency on OBJs, > which in turn is what contains the dependency on WIN32RES, which in turn > contains win32ver.o. So the build succeeds if pg_dump/restores's dependencies > are built f

Re: MDAM techniques and Index Skip Scan patch

2022-03-22 Thread Thomas Munro
On Tue, Mar 22, 2022 at 2:34 PM Andres Freund wrote: > IMO it's pretty clear that having "duelling" patches below one CF entry is a > bad idea. I think they should be split, with inactive approaches marked as > returned with feeback or whatnot. I have the impression that this thread is getting so

Re: Probable CF bot degradation

2022-03-22 Thread Thomas Munro
On Mon, Mar 21, 2022 at 12:46 PM Thomas Munro wrote: > On Mon, Mar 21, 2022 at 12:23 PM Thomas Munro wrote: > > On Mon, Mar 21, 2022 at 1:58 AM Matthias van de Meent > > wrote: > > > Additionally, are there plans to validate commits of the main branch > > > b

Re: cpluspluscheck vs ICU

2022-03-22 Thread Thomas Munro
On Wed, Mar 23, 2022 at 3:23 PM Andres Freund wrote: > On 2022-03-22 17:20:24 -0700, Andres Freund wrote: > > I was about to propose adding headerscheck / cpluspluscheck to the CI file > > so > > that cfbot can catch future issues. > > The attached patch does so, with ICU disabled to avoid the pr

Re: [PATCH] Add native windows on arm64 support

2022-03-22 Thread Thomas Munro
On Tue, Mar 22, 2022 at 11:30 PM Julien Rouhaud wrote: > On Tue, Mar 22, 2022 at 09:37:46AM +, Niyas Sait wrote: > > Yes, we could look into providing a build machine. Do you have any > > reference to what the CI system looks like now for PostgresSQL and how to > > add new workers etc.? > > It

Re: Adding CI to our tree

2022-03-23 Thread Thomas Munro
On Thu, Mar 10, 2022 at 9:37 AM Justin Pryzby wrote: > -Og Adding this to CXXFLAGS caused a torrent of warnings from g++ about LLVM headers, which I also see on my local system for LLVM 11 and LLVM 14: [19:47:11.047] /usr/lib/llvm-11/include/llvm/ADT/Twine.h: In member function ‘llvm::CallInst*

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-03-23 Thread Thomas Munro
On Thu, Mar 24, 2022 at 9:59 AM Peter Geoghegan wrote: > On Wed, Mar 23, 2022 at 1:53 PM Robert Haas wrote: > > And therefore I favor defining it to mean that we don't > > skip any work at all. > > But even today DISABLE_PAGE_SKIPPING won't do pruning when we cannot > acquire a cleanup lock on a

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-03-23 Thread Thomas Munro
On Thu, Mar 24, 2022 at 9:53 AM Peter Eisentraut wrote: > On 21.03.22 05:55, Thomas Munro wrote: > > [04:30:50.630] pg_waldump.c:963:26: error: format ‘%u’ expects > > argument of type ‘unsigned int *’, but argument 3 has type ‘ForkNumber > > *’ [-Werror=format=] > &

Re: Avoid erroring out when unable to remove or parse logical rewrite files to save checkpoint work

2022-03-23 Thread Thomas Munro
On Wed, Feb 16, 2022 at 12:11 PM Nathan Bossart wrote: > On Tue, Feb 15, 2022 at 10:37:58AM -0800, Nathan Bossart wrote: > > On Tue, Feb 15, 2022 at 10:10:34AM -0800, Andres Freund wrote: > >> I generally think it'd be a good exercise to go through an use > >> get_dirent_type() in nearly all ReadD

Checking pgwin32_is_junction() errors

2022-03-23 Thread Thomas Munro
to work around that problem, but I don't like it, and would need to come up with something better... Sketch code attached. From 8e0e51359c0191cf673c3ddc87a3262b13f530a4 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Wed, 16 Mar 2022 23:05:39 +1300 Subject: [PATCH] Raise errors in pgwin32_i

Re: BUG #16663: DROP INDEX did not free up disk space: idle connection hold file marked as deleted

2020-11-30 Thread Thomas Munro
On Mon, Nov 30, 2020 at 6:59 PM Thomas Munro wrote: > On Wed, Nov 25, 2020 at 8:00 AM Pavel Borisov wrote: > > The new status of this patch is: Ready for Committer > ... That's the version I plan to commit tomorrow, unless > there are further comments or objections.

Re: BUG #16663: DROP INDEX did not free up disk space: idle connection hold file marked as deleted

2020-11-30 Thread Thomas Munro
On Tue, Dec 1, 2020 at 3:55 PM Michael Paquier wrote: > On Mon, Nov 30, 2020 at 06:59:40PM +1300, Thomas Munro wrote: > > So I fixed that, by adding a return value to do_truncate() and > > checking it. That's the version I plan to commit tomorrow, unless > > the

PG vs LLVM 12 on seawasp, next round

2020-11-30 Thread Thomas Munro
Hi Since seawasp's bleeding edge LLVM installation moved to "trunk 20201114 c8f4e06b 12.0.0" ~16 days ago, it has been red. Further updates didn't help it and it's now on "trunk 20201127 6ee22ca6 12.0.0". I wonder if there is something in Fabien's scripting that needs to be tweaked, perhaps a sy

Re: Recent eelpout failures on 9.x branches

2020-12-01 Thread Thomas Munro
On Wed, Dec 2, 2020 at 11:36 AM Tom Lane wrote: > Perhaps this is just a question of the machine being too slow to complete > the test, in which case we ought to raise wal_sender_timeout. But it's > weird that it would've started to fail just now, because I don't really > see any changes in those

Re: Recent eelpout failures on 9.x branches

2020-12-01 Thread Thomas Munro
On Wed, Dec 2, 2020 at 12:07 PM Tom Lane wrote: > I'm also wondering a bit why the issue isn't affecting the newer > branches. It's certainly not because we made the test shorter ... I looked at htop while it was building the 9.x branches and saw pg_basebackup sitting in D state waiting for glac

Re: Commitfest 2020-11 is closed

2020-12-02 Thread Thomas Munro
On Thu, Dec 3, 2020 at 10:00 AM Tom Lane wrote: > This is actually a bit problematic, because now the cfbot is ignoring > those patches (or if it's not, I don't know where it's displaying the > results). Please go ahead and move the remaining open patches, or > else re-open the CF if that's possi

Re: Commitfest 2020-11 is closed

2020-12-02 Thread Thomas Munro
On Thu, Dec 3, 2020 at 10:51 AM David Steele wrote: > On 12/2/20 4:15 PM, Thomas Munro wrote: > > On Thu, Dec 3, 2020 at 10:00 AM Tom Lane wrote: > >> This is actually a bit problematic, because now the cfbot is ignoring > >> those patches (or if it's not, I don&

Github Actions (CI)

2020-12-02 Thread Thomas Munro
rhn8bH836OB%2B3SboiaeEcgOtrJS58Bki4%3D5yeVqToxgw%40mail.gmail.com From 424bcae7f1fcb1ada0e7046bfc5e0c5254c6f439 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Thu, 3 Dec 2020 10:58:16 +1300 Subject: [PATCH] Github CI WIP --- .github/workflows/ci-linux.yml | 43 +

Re: Commitfest 2020-11 is closed

2020-12-02 Thread Thomas Munro
On Thu, Dec 3, 2020 at 12:02 PM Josef Šimánek wrote: > st 2. 12. 2020 v 23:36 odesílatel Andrew Dunstan napsal: > > On 12/2/20 5:13 PM, Thomas Munro wrote: > > > I'm experimenting with Github's built in CI. All other ideas welcome. > > > > I'd lo

Re: Github Actions (CI)

2020-12-03 Thread Thomas Munro
On Fri, Dec 4, 2020 at 2:55 AM Josef Šimánek wrote: > Any chance to also share links to failing/passing testing builds? https://github.com/macdice/postgres/runs/1490727809 https://github.com/macdice/postgres/runs/1490727838 However, looking these in a clean/cookieless browser, I see that it won'

Re: Index Skip Scan (new UniqueKeys)

2020-12-05 Thread Thomas Munro
On Wed, Dec 2, 2020 at 9:59 AM Heikki Linnakangas wrote: > On 01/12/2020 22:21, Dmitry Dolgov wrote: > >> On Mon, Nov 30, 2020 at 04:42:20PM +0200, Heikki Linnakangas wrote: > >> - I'm surprised you need a new index AM function (amskip) for this. Can't > >> you just restart the scan with index_res

Re: Blocking I/O, async I/O and io_uring

2020-12-07 Thread Thomas Munro
On Tue, Dec 8, 2020 at 3:56 PM Craig Ringer wrote: > I thought I'd start the discussion on this and see where we can go with it. > What incremental steps can be done to move us toward parallelisable I/O > without having to redesign everything? > > I'm thinking that redo is probably a good first

Re: Commitfest 2020-11 is closed

2020-12-08 Thread Thomas Munro
On Fri, Dec 4, 2020 at 1:29 AM Peter Eisentraut wrote: > On 2020-12-02 23:13, Thomas Munro wrote: > > I'm experimenting with Github's built in CI. All other ideas welcome. > > You can run Linux builds on AppVeyor, too. Yeah. This would be the easiest change to mak

Re: Autovacuum worker doesn't immediately exit on postmaster death

2020-12-10 Thread Thomas Munro
On Fri, Dec 11, 2020 at 8:34 AM Robert Haas wrote: > On Thu, Oct 29, 2020 at 5:36 PM Alvaro Herrera > wrote: > > Maybe instead of thinking specifically in terms of vacuum, we could > > count buffer accesses (read from kernel) and check the latch once every > > 1000th such, or something like that

Re: Cache relation sizes?

2020-12-16 Thread Thomas Munro
Hi Andy, On Thu, Dec 17, 2020 at 7:29 PM Andy Fan wrote: > I spent one day studying the patch and I want to talk about one question for > now. > What is the purpose of calling smgrimmedsync to evict a DIRTY sr (what will > happen > if we remove it and the SR_SYNCING and SR_JUST_DIRTIED flags)?

pg_preadv() and pg_pwritev()

2020-12-19 Thread Thomas Munro
63cb0 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Thu, 26 Nov 2020 15:48:31 +1300 Subject: [PATCH 07/11] Add pg_preadv() and pg_pwritev(). Provide synchronous scatter/gather I/O routines. These map to preadv(), pwritev() with various fallbacks for systems that don't have them. -

Re: pg_preadv() and pg_pwritev()

2020-12-19 Thread Thomas Munro
On Sun, Dec 20, 2020 at 12:34 PM Tom Lane wrote: > Thomas Munro writes: > > I want to be able to do synchronous vectored file I/O, so I made > > wrapper macros for preadv() and pwritev() with fallbacks for systems > > that don't have them. Following the prece

Re: pg_preadv() and pg_pwritev()

2020-12-20 Thread Thomas Munro
On Sun, Dec 20, 2020 at 8:07 PM Tom Lane wrote: > One minor thought is that in > > + struct iovec iov[Min(IOV_MAX, 1024)]; /* cap stack space */ > > it seems like pretty much every use of IOV_MAX would want some > similar cap. Should we centralize that idea with, say, > > #define

Re: pg_preadv() and pg_pwritev()

2020-12-21 Thread Thomas Munro
On Mon, Dec 21, 2020 at 8:25 PM Jakub Wartak wrote: > > > I'm drawing a blank on trivial candidate uses for preadv(), without > > > infrastructure from later patches. > > > > Can't immediately think of something either. > > This might be not that trivial , but maybe acquire_sample_rows() from > a

Re: pg_preadv() and pg_pwritev()

2020-12-22 Thread Thomas Munro
On Mon, Dec 21, 2020 at 11:40 AM Andres Freund wrote: > On 2020-12-20 16:26:42 +1300, Thomas Munro wrote: > > > 1. port.h cannot assume that has already been included; > > > nor do I want to fix that by including there. Do we > > > really need to define a fal

Re: Cache relation sizes?

2020-12-23 Thread Thomas Munro
On Thu, Dec 17, 2020 at 10:22 PM Andy Fan wrote: > Let me try to understand your point. Suppose process 1 extends a file to > 2 blocks from 1 block, and fsync is not called, then a). the lseek *may* still > return 1 based on the comments in the ReadBuffer_common ("because > of buggy Linux kernels

Re: Re: Cache relation sizes?

2020-12-23 Thread Thomas Munro
On Wed, Dec 23, 2020 at 1:31 AM 陈佳昕(步真) wrote: > I studied your patch these days and found there might be a problem. > When execute 'drop database', the smgr shared pool will not be removed > because of no call 'smgr_drop_sr'. Function 'dropdb' in dbcommands.c remove > the buffer from bufferpool

Re: doc review for v14

2020-12-28 Thread Thomas Munro
On Thu, Dec 24, 2020 at 9:12 PM Michael Paquier wrote: > On Mon, Dec 21, 2020 at 10:11:53PM -0600, Justin Pryzby wrote: > > Specifies the amount of memory that should be allocated at server > > -startup time for use by parallel queries. When this memory region > > is > > +

Re: Parallel Full Hash Join

2020-12-28 Thread Thomas Munro
n true if this worker is elected to do the unmatched inner scan + */ +bool +ExecParallelPrepHashTableForUnmatched(HashJoinState *hjstate) Comment name doesn't match function name. From 9199bfcfa84acbcfeb9a8d3c21962096c7ff645c Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Thu, 5 Nov 20

Re: Let's start using setenv()

2020-12-28 Thread Thomas Munro
On Tue, Dec 29, 2020 at 4:21 PM Tom Lane wrote: > Back in 2001 we decided to prefer putenv() over setenv() because the > latter wasn't in POSIX (SUSv2) at the time. That decision has been > overtaken by events: more recent editions of POSIX not only provide > setenv() but recommend it over putenv

Re: Let's start using setenv()

2020-12-29 Thread Thomas Munro
On Wed, Dec 30, 2020 at 5:45 AM Tom Lane wrote: > Since the cfbot seems happy with v1, here's a v2 that runs around > and converts all putenv() uses to setenv(). In some places there's > no notational savings, but it seems to me that standardizing > on just one way to do it is beneficial. +

Re: Cache relation sizes?

2020-12-29 Thread Thomas Munro
On Mon, Dec 28, 2020 at 5:24 PM Andy Fan wrote: > lseek(..., SEEK_END) = 9670656 > write(...) = 8192 > lseek(..., SEEK_END) = 9678848 > fsync(...) = -1 > lseek(..., SEEK_END) = 9670656 > > I got 2 information from above. a) before the fsync, the lseek(fd, 0, > SEEK_END) > returns a correct valu

Re: Re: Re: Cache relation sizes?

2020-12-29 Thread Thomas Munro
be better to increment sr->generation. That was already done in the "eviction" code path, where other processes might still have references to the SR object, and I don't think it's possible for anyone to access a dropped relation, but I suppose it's more consistent

Re: Re: Re: Cache relation sizes?

2020-12-29 Thread Thomas Munro
On Wed, Dec 30, 2020 at 5:52 PM Thomas Munro wrote: > and requires it on retry s/requires/reacquires/

Re: pgbench: option delaying queries till connections establishment?

2020-12-31 Thread Thomas Munro
On Sun, Nov 15, 2020 at 4:53 AM Fabien COELHO wrote: > In the attached version, I just comment out the call and add an > explanation about why it is commented out. If pg overall version > requirements are changed on windows, then it could be reinstated. It looks like macOS doesn't have pthread ba

Re: new heapcheck contrib module

2020-12-31 Thread Thomas Munro
On Tue, Oct 27, 2020 at 5:12 AM Mark Dilger wrote: > The v20 patches 0002, 0003, and 0005 still apply cleanly, but 0004 required a > rebase. (0001 was already committed last week.) > > Here is a rebased set of 4 patches, numbered 0002..0005 to be consistent with > the previous naming. There ar

Re: WIP: BRIN multi-range indexes

2021-01-01 Thread Thomas Munro
On Sun, Dec 20, 2020 at 1:16 PM Tomas Vondra wrote: > Attached is an updated version of the patch series, rebased on current > master, and results for benchmark comparing the various bloom variants. Perhaps src/include/utils/inet.h needs to include , because FreeBSD says: brin_minmax_multi.c:169

Re: psql \df choose functions by their arguments

2021-01-01 Thread Thomas Munro
On Thu, Dec 31, 2020 at 7:01 AM Greg Sabino Mullane wrote: > Attached is the latest patch against HEAD - basically fixes a few typos. Hi Greg, It looks like there is a collation dependency here that causes the test to fail on some systems: === ./src/test/regress/regression.diffs === diff -U3 /t

Re: Context diffs

2021-01-04 Thread Thomas Munro
On Tue, Jan 5, 2021 at 8:07 AM Bruce Momjian wrote: > * "git apply" and "git am" can't process context diffs (they throw an >error once a context-like section of the diff is hit; simple >adding/removing lines in a block works) > > * the commit-fest doesn't recognized context diff attachm

Re: Handing off SLRU fsyncs to the checkpointer

2021-01-04 Thread Thomas Munro
On Mon, Jan 4, 2021 at 3:35 AM Tomas Vondra wrote: > Seems this commit left behind a couple unnecessary prototypes in a bunch > of header files. In particular, it removed these functions > > - ShutdownCLOG(); > - ShutdownCommitTs(); > - ShutdownSUBTRANS(); > - ShutdownMultiXact(); Thanks. Fixed.

Re: language cleanups in code and docs

2021-01-04 Thread Thomas Munro
On Wed, Nov 4, 2020 at 4:10 AM Magnus Hagander wrote: > On Wed, Oct 21, 2020 at 11:23 PM Thomas Munro wrote: > > Hmm, can we find a more descriptive name for this mechanism? What > > about calling it the "uncommitted enum table"? See attached. > > Thank

Re: language cleanups in code and docs

2021-01-04 Thread Thomas Munro
On Tue, Jan 5, 2021 at 1:12 PM Dagfinn Ilmari Mannsåker wrote: > Magnus Hagander writes: > > In looking at this I realize we also have exactly one thing referred to as > > "blacklist" in our codebase, which is the "enum blacklist" (and then a > > small internal variable in pgindent). > > Here's a

Re: language cleanups in code and docs

2021-01-04 Thread Thomas Munro
On Tue, Jan 5, 2021 at 1:44 PM Dagfinn Ilmari Mannsåker wrote: > Thanks! Just after sending that, I thought to grep for "white\W*list" > as well, and found a few more occurrences that were trivially reworded, > per the attached patch. Pushed.

Cirrus CI (Windows help wanted)

2021-01-04 Thread Thomas Munro
Hi, My new favourite CI is Cirrus CI, because it has 4 operating systems, generous enough quotas to handle 250+ branches in a single account, and public build/test log URLs. I flipped cfbot.cputube.org (mostly) over to that and it seems to work well so far -- fingers crossed. I've also been using

Re: Reducing WaitEventSet syscall churn

2021-01-04 Thread Thomas Munro
On Fri, Jul 31, 2020 at 9:42 AM Thomas Munro wrote: > On Thu, Jul 30, 2020 at 5:50 PM Thomas Munro wrote: > > I pushed those three patches, but will wait for more discussion on the rest. > > And here's a rebase, to make cfbot happy. And again. To restate the two goals of t

Re: Moving other hex functions to /common

2021-01-04 Thread Thomas Munro
On Tue, Jan 5, 2021 at 4:47 PM Bruce Momjian wrote: > ... let's see how it likes this version. cfbot ideally processes a new patch fairly quickly but I didn't think of ".diff.gz" when writing the regexp to recognise patch files. I just tweaked the relevant regexp and it's building your patch now

Re: Context diffs

2021-01-05 Thread Thomas Munro
On Wed, Jan 6, 2021 at 6:58 AM Bruce Momjian wrote: > On Tue, Jan 5, 2021 at 11:21:07AM +1300, Thomas Munro wrote: > > For the record, cfbot just uses plain old GNU patch, because that > > seems to accept nearly everything that anyone posts here (after a step > > that tri

Re: Cirrus CI (Windows help wanted)

2021-01-05 Thread Thomas Munro
On Wed, Jan 6, 2021 at 11:48 AM Andrew Dunstan wrote: > OK, I have dug into this quite a bit. The way cirrus works is in fact > somewhat fragile. Anyway, here are the highlights: Thanks! > However, sadly the vctools package above isn't installed with all its > optional packages, so some crucial

Re: Terminate the idle sessions

2021-01-06 Thread Thomas Munro
On Thu, Jan 7, 2021 at 12:55 PM Tom Lane wrote: > * Thomas' patch for improving timeout.c seems like a great idea, but > it did indeed have a race condition, and I felt the comments could do > with more work. Oops, and thanks! Very happy to see this one in the tree. > * I'm not entirely comfort

  1   2   3   4   5   6   7   8   9   10   >