Re: Avoiding smgrimmedsync() during nbtree index builds

2021-12-09 Thread Andres Freund
> /* This needs documentation. The whole comment above isn't accurate anymore afaict? > diff --git a/src/backend/access/transam/xlog.c > b/src/backend/access/transam/xlog.c > index 1616448368..63fd212787 100644 > --- a/src/backend/access/transam/xlog.c > +++ b/src/backend/access/transam/xlog.c > @@ -8704,6 +8704,20 @@ GetRedoRecPtr(void) > return RedoRecPtr; > } > > +bool > +RedoRecPtrChanged(XLogRecPtr comparator_ptr) > +{ > + XLogRecPtr ptr; > + > + SpinLockAcquire(&XLogCtl->info_lck); > + ptr = XLogCtl->RedoRecPtr; > + SpinLockRelease(&XLogCtl->info_lck); > + > + if (RedoRecPtr < ptr) > + RedoRecPtr = ptr; > + return RedoRecPtr != comparator_ptr; > +} What's the deal with the < comparison? Greetings, Andres Freund

Re: A test for replay of regression tests

2021-12-09 Thread Andres Freund
by the name, will also contain other things seems likely to cause confusion. > I'm slightly worried that some bright spark will discover it and think > it's a good idea for a production setup. It'd not really be worse than the current situation of accidentally corrupting a local replica or such :/. Greetings, Andres Freund

Re: A test for replay of regression tests

2021-12-09 Thread Andres Freund
c', > + '-p', $node_standby_1->port, 'regression' ], > + "dump standby server"); > +command_ok( > + [ "diff", $outputdir . '/primary.dump', $outputdir . '/standby.dump' ], > + "compare primary and standby dumps"); > + Absurd nitpick: What's the deal with using "" for one part, and '' for the rest? Separately: I think the case of seeing diffs will be too hard to debug like this, as the difference isn't shown afaict? Greetings, Andres Freund

Re: do only critical work during single-user vacuum?

2021-12-09 Thread Andres Freund
pdate it before doing truncation (that can be quite expensive). But we probably should do it even before the final lazy_cleanup_all_indexes() pass - often that'll be the only pass, and there's really no reason to delay relfrozenxid advancement till after that. Greetings, Andres Freund

Re: do only critical work during single-user vacuum?

2021-12-09 Thread Andres Freund
che will be cold, there's no walwriter, etc, making the vacuum slower. Making vacuum not consume an xid seems like a lot more promising - and quite doable. Then there's no need to restart at all. Greetings, Andres Freund

Re: A test for replay of regression tests

2021-12-10 Thread Andres Freund
, so for > example if you comment out the bit that deals with SEQUENCE caching > you'll see: Ah, ok. Not sure what I thought there... > On Fri, Dec 10, 2021 at 10:35 AM Thomas Munro wrote: > > On Fri, Dec 10, 2021 at 8:38 AM Andres Freund wrote: > > > Personally I

isolationtester: add session name to application name

2021-12-10 Thread Andres Freund
hen establishing connections to deal with that. As attached this appends "control connection" for the control connection, but perhaps we should just not append anything for that? Greetings, Andres Freund >From fce722b66ae2a727c0300b3ece843e49f61e0359 Mon Sep 17 00:00:00 2

Re: Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2021-12-10 Thread Andres Freund
orig_rel_pages, > new_rel_pages))); > orig_rel_pages = new_rel_pages; > } while (new_rel_pages > vacrel->nonempty_pages && > lock_waiter_detected); > } These imo are useful. Perhaps we could just make them part of some log message that autovac logging includes as well? Greetings, Andres Freund

Re: Add client connection check during the execution of the query

2021-12-10 Thread Andres Freund
ics this still changes when we'd e.g. detect postmaster death. Am I misunderstanding code or comment, or is the comment saying that it's ok to clobber the wes, but then we actually unclobber it? Greetings, Andres Freund

Re: parallel vacuum comments

2021-12-10 Thread Andres Freund
Hi, On 2021-10-30 14:21:01 -0700, Andres Freund wrote: > Due to bug #17245: [1] I spent a considerably amount of time looking at vacuum > related code. And I found a few things that I think could stand improvement: While working on the fix for #17255 (more specifically some cleanup that

Re: Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2021-12-11 Thread Andres Freund
On 2021-12-11 09:52:29 -0800, Peter Geoghegan wrote: > On Fri, Dec 10, 2021 at 8:30 PM Andres Freund wrote: > > I think some actually ended up being omitted compared to the previous > > state. E.g. "aggressively vacuuming ...", but I think others as well. > > The &

Re: Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2021-12-11 Thread Andres Freund
Hi, On 2021-12-11 13:13:56 -0800, Peter Geoghegan wrote: > On Sat, Dec 11, 2021 at 12:24 PM Andres Freund wrote: > > But the ereport is inside an if (verbose), no? > > Yes -- in order to report aggressiveness in VACUUM VERBOSE. But the > autovacuum case still reports verbose

Re: [PATCH] pg_stat_toast

2021-12-12 Thread Andres Freund
re refining this further. I'm worried about the additional overhead this might impose. For some workload it'll substantially increase the amount of stats traffic. Have you tried to qualify the overheads? Both in stats size and in stats management overhead? Greetings, Andres Freund

Re: [PATCH] pg_stat_toast

2021-12-12 Thread Andres Freund
ring 1.000.000 INSERTs (2 TOASTable columns each) with and > without "pgstat_track_toast" resulted in 12792.882 ms vs. 12810.557 ms. So > at least the call overhead seems to be neglectible. Yea, you'd probably need a few more tables and a few more connections for it to have a chance of mattering meaningfully. Greetings, Andres Freund

Re: isolationtester: add session name to application name

2021-12-13 Thread Andres Freund
Hi, On 2021-12-13 19:46:34 +0900, Michael Paquier wrote: > On Fri, Dec 10, 2021 at 05:20:52PM -0800, Andres Freund wrote: > > These days isolationtester.c already prefixes log output with the session > > name. How about doing the same for application_name? It's a *tad* more

Re: Add client connection check during the execution of the query

2021-12-13 Thread Andres Freund
Hi, On 2021-12-13 17:51:00 +1300, Thomas Munro wrote: > On Sat, Dec 11, 2021 at 7:09 PM Thomas Munro wrote: > > On Sat, Dec 11, 2021 at 6:11 PM Andres Freund wrote: > > > Yuck. Is there really no better way to deal with this? What kind of > > > errors is

Re: port conflicts when running tests concurrently on windows.

2021-12-13 Thread Andres Freund
Hi, On 2021-12-10 10:22:13 +0100, Peter Eisentraut wrote: > On 09.12.21 19:41, Andres Freund wrote: > > Withhttps://github.com/anarazel/postgres/commit/046203741803da863f6129739fd215f8a32ec357 > > all tests pass. pg_regress requires PG_REGRESS_SOCK_DIR because it checks > >

Re: isolationtester: add session name to application name

2021-12-13 Thread Andres Freund
Hi, On 2021-12-13 13:57:52 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2021-12-13 19:46:34 +0900, Michael Paquier wrote: > >> +1 for the idea. Maybe it could be backpatched? > > > Not entirely trivially - the changes have some dependencies on other changes &

Re: Adding CI to our tree

2021-12-13 Thread Andres Freund
gh: One I explicitly brought up before: On 2021-10-01 15:27:52 -0700, Andres Freund wrote: > One policy discussion that we'd have to have is who should control the images > used for CI. Right now that's on my personal google cloud account - which I am > happy to do, but medium - lon

Re: Adding CI to our tree

2021-12-13 Thread Andres Freund
Hi, On 2021-12-13 16:02:50 -0600, Justin Pryzby wrote: > On Mon, Dec 13, 2021 at 01:12:23PM -0800, Andres Freund wrote: > > Hi, > > > > Attached is an updated version of the CI patches. An example of a test run > > with the attached version of this >

Re: Adding CI to our tree

2021-12-13 Thread Andres Freund
Hi, On 2021-12-13 18:14:52 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2021-12-13 16:02:50 -0600, Justin Pryzby wrote: > >> sudo is used exactly twice; maybe it's not needed at all ? > > > The macos one is needed, but the freebsd one indeed isn't

Re: Rewriting the test of pg_upgrade as a TAP test - take three - remastered set

2021-12-13 Thread Andres Freund
fine with a test doing nothing. We'd not loose coverage with non-updated BF animals unless they have tap tests disabled. Just the cross-version test would need timely work by buildfarm operators - but I think Andrew could deal with that. Greetings, Andres Freund

Re: Adding CI to our tree

2021-12-13 Thread Andres Freund
omplete Window install directory as an artefact > so a Windows user without a dev environment could try out a proposed > change/CF entry/... I think the size of these artifacts would make this not something to enable by default. But perhaps a manually triggered task would make sense? > I hope we can get ccache working on Windows. They did merge a number of the other required changes for that over the weekend. I'll try once they released... Thanks! Andres Freund

Re: Rewriting the test of pg_upgrade as a TAP test - take three - remastered set

2021-12-13 Thread Andres Freund
On 2021-12-14 14:31:24 +0900, Michael Paquier wrote: > On Mon, Dec 13, 2021 at 06:08:24PM -0800, Andres Freund wrote: > > Seems like we might get away with making make -C contrib/pg_upgrade check > > and > > vcregress.pl upgradecheck do nothing? > > You mean #contrib

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-12-16 Thread Andres Freund
vent that we don't get around to adding more bgwriter stats right > away). How about keeping old bgwriter values in place in the view , but generated from the new stats stuff? > I noticed after changing the docs on the "bgwriter" target for > pg_stat_reset_shared to say "checkpointer", that it still said "bgwriter" in > src/backend/po/ko.po > src/backend/po/it.po > ... > I presume these are automatically updated with some incantation, but I wasn't > sure what it was nor could I find documentation on this. Yes, they are - and often some languages lag updating things. There's a bit of docs at https://www.postgresql.org/docs/devel/nls.html Greetings, Andres Freund

Re: Adding CI to our tree

2021-12-17 Thread Andres Freund
Hi, On 2021-12-17 12:34:36 +0100, Peter Eisentraut wrote: > On 13.12.21 22:12, Andres Freund wrote: > > Attached is an updated version of the CI patches. An example of a test run > > with the attached version of this > > https://cirrus-ci.com/build/65019985214

Re: Adding CI to our tree

2021-12-17 Thread Andres Freund
the windows runs are a lot slower than the others is just that visual studio isn't yet supported by ccache, and that there doesn't seem to be good other tools. The ccache maintainers merged more of the msvc support last weekend! So I have quite a bit of hope of being able to use ccache there as well. Greetings, Andres Freund

Re: Adding CI to our tree

2021-12-17 Thread Andres Freund
On 2021-12-17 11:31:59 -0800, Andres Freund wrote: > > Don't we need the ulimit call for FreeBSD? > > I think the default core limits were different, I will check. Yep, freebsd has -c unlimited by default: https://cirrus-ci.com/task/6199382239346688?logs=sysinfo#L23 vs https://c

Re: Adding CI to our tree

2021-12-20 Thread Andres Freund
steps - Address Thomas' feeback - Try to address Peter's feedback Regards, Andres >From 264c0797033dd0325032e89fee0095bf75ea7fc1 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Mon, 15 Mar 2021 09:25:15 -0700 Subject: [PATCH v4 1/2] ci: Add CI for FreeBSD, Linux, MacOS and W

Re: do only critical work during single-user vacuum?

2021-12-20 Thread Andres Freund
this patch would regress) and not every emergency making single user mode useful is related to wraparound. Greetings, Andres Freund

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-29 Thread Andres Freund
falling back further because XLogFlush() is being throttled). I also don't think it's a sane thing to add hooks to these places. It's complicated enough as-is, adding the chance for random other things to happen during such crucial operations will make it even harder to maintain. Greetings, Andres Freund

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-29 Thread Andres Freund
Hi, On 2021-12-29 11:34:53 -0800, SATYANARAYANA NARLAPURAM wrote: > On Wed, Dec 29, 2021 at 11:31 AM Andres Freund wrote: > Andres, thanks for the comments. Agreed on this based on the previous > discussions on this thread. Could you please share your thoughts on adding &

Re: WIP: WAL prefetch (another approach)

2021-12-29 Thread Andres Freund
in the tree, rather than coming up with them locally over and over / across multiple people working on a branch. Greetings, Andres Freund

Re: Adding CI to our tree

2021-12-29 Thread Andres Freund
Hi, On 2021-12-20 11:21:05 -0800, Andres Freund wrote: > Attached is v4 of the CI patch. I'd like to push this - any objections? It's not disruptive to anything but cfbot, so we can incrementally improve it further. I'll try to sync pushing with Thomas, so that he can adjus

Re: Add Boolean node

2021-12-29 Thread Andres Freund
On 2021-12-27 09:53:32 -0500, Tom Lane wrote: > Didn't really read the patch in any detail, but I did have one idea: > I think that the different things-that-used-to-be-Value-nodes ought to > use different field names, say ival, rval, bval, sval not just "val". > That makes it more likely that you'

Re: Add Boolean node

2021-12-29 Thread Andres Freund
8bef 100644 > --- a/src/test/isolation/expected/ri-trigger.out > +++ b/src/test/isolation/expected/ri-trigger.out > @@ -4,9 +4,9 @@ starting permutation: wxry1 c1 r2 wyrx2 c2 > step wxry1: INSERT INTO child (parent_id) VALUES (0); > step c1: COMMIT; > step r2: SELECT TRUE; > -bool > - > -t > +?column? > + > +t > (1 row) This doesn't seem great. It might be more consistent ("SELECT 1" doesn't end up with 'integer' as column name), but this still seems like an unnecessarily large user-visible change for an internal data-representation change? Greetings, Andres Freund

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-30 Thread Andres Freund
ose applying to WAL throttling to a significant degree, particularly not when it's on a transaction level. Greetings, Andres Freund

Re: Apple's ranlib warns about protocol_openssl.c

2021-12-30 Thread Andres Freund
openssl.c. Perhaps we should just move it to protocol_openssl.c? Greetings, Andres Freund

Re: Adding CI to our tree

2021-12-30 Thread Andres Freund
ge...) unless I hear protests. Greetings, Andres Freund >From 124ec8dea062e95931d5d2f80633e025aa7733ad Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Mon, 15 Mar 2021 09:25:15 -0700 Subject: [PATCH v5] ci: Add continuous integration for github repositories via cirrus-ci. Currently FreeBSD

Re: Adding CI to our tree

2021-12-30 Thread Andres Freund
Hi, On 2021-12-30 20:28:46 -0600, Justin Pryzby wrote: > [ language fixes] Thanks! > script uses a pseudo-tty, which do support locking. > => which does This didn't seem right either way - it's pseudo-ttys that don't support locking, so plural seemed appropriate? I changed it to "script uses ps

Re: Adding CI to our tree

2021-12-30 Thread Andres Freund
On 2021-12-30 17:46:52 -0800, Andres Freund wrote: > I plan to push this after another cycle of tests passing (and driving > over the bay bridge...) unless I hear protests. Pushed. Marked CF entry as committed.

Re: Patch to avoid orphaned dependencies

2021-12-30 Thread Andres Freund
the right place to add them, is it?) This fails on windows w/ msvc: https://cirrus-ci.com/task/5368174125252608?logs=configure#L102 https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.157904#L12 Greetings, Andres Freund

slowest tap tests - split or accelerate?

2021-12-31 Thread Andres Freund
en if they take the same time, because we can use prove level concurrency on windows to gain some test parallelism. As a nice side-effect it makes it also quicker to run a split test isolated during development. Greetings, Andres Freund [1] https://cirrus-ci.com/task/5207126145499136 [2] https://cir

Re: O(n) tasks cause lengthy startups and checkpoints

2022-01-02 Thread Andres Freund
rate what an auxiliary process approach might look like. This generates a compiler warning: https://cirrus-ci.com/task/5740581082103808?logs=mingw_cross_warning#L378 Greetings, Andres Freund

Re: [PATCH] Tracking statements entry timestamp in pg_stat_statements

2022-01-02 Thread Andres Freund
Hi, On 2021-12-03 17:03:46 +0300, Andrei Zubkov wrote: > I've attached a new version of a patch. This fails with an assertion failure: https://cirrus-ci.com/task/5567540742062080?logs=cores#L55 Greetings, Andres Freund

Re: [PATCH] Accept IP addresses in server certificate SANs

2022-01-02 Thread Andres Freund
ror LNK2019: unresolved external symbol pg_inet_net_pton referenced in function network_in [c:\cirrus\postgres.vcxproj] Greetings, Andres Freund

Re: daitch_mokotoff module

2022-01-02 Thread Andres Freund
us-ci.com/task/5941863248035840?logs=test_world#L572 https://api.cirrus-ci.com/v1/artifact/task/5941863248035840/regress_diffs/contrib/fuzzystrmatch/regression.diffs Greetings, Andres Freund

Re: Add jsonlog log_destination for JSON server logs

2022-01-02 Thread Andres Freund
by zero could not open "c:/cirrus/src/bin/pg_ctl/tmp_check/t_004_logrotate_primary_data/pgdata/current_logfiles": The system cannot find the file specified at t/004_logrotate.pl line 87. Greetings, Andres Freund

Re: daitch_mokotoff module

2022-01-03 Thread Andres Freund
bot creates, rather than the test run itself: https://github.com/postgresql-cfbot/postgresql/commit/d5b4ec87cfd65dc08d26e1b789bd254405c90a66#diff-388d4bb360a3b24c425e29a85899315dc02f9c1dd9b9bc9aaa828876bdfea50aR56 Greetings, Andres Freund

Re: using extended statistics to improve join estimates

2022-01-04 Thread Andres Freund
6:27: error: ‘index’ may be used uninitialized in this function [-Werror=maybe-uninitialized] [18:52:22.697] 2766 | if (mcv->items[i].isnull[index]) [18:52:22.697] | ^ Greetings, Andres Freund

Re: PoC: using sampling to estimate joins / complex conditions

2022-01-04 Thread Andres Freund
th improving this, possible even in PG15. The tests in this patch fail: https://cirrus-ci.com/task/5304621299138560 https://api.cirrus-ci.com/v1/artifact/task/5304621299138560/regress_diffs/src/test/regress/regression.diffs Looks like the regression test output hasn't been updated? Greetings, Andres Freund

Re: In-placre persistance change of a relation

2022-01-04 Thread Andres Freund
p/src/test/recovery/tmp_check/log/regress_log_027_persistence_change Greetings, Andres Freund

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2022-01-05 Thread Andres Freund
Hi, On 2021-12-29 11:31:51 -0800, Andres Freund wrote: > That's pretty much the same - XLogInsert() can trigger an > XLogWrite()/Flush(). > > I think it's a complete no-go to add throttling to these places. It's quite > possible that it'd cause new deadlocks,

Re: [EXTERNAL] Re: PQcancel does not use tcp_user_timeout, connect_timeout and keepalive settings

2022-01-05 Thread Andres Freund
Code = (MsgType) pg_hton32(CANCEL_REQUEST_CODE); > - crp.cp.backendPID = pg_hton32(be_pid); > - crp.cp.cancelAuthCode = pg_hton32(be_key); > + crp.cp.backendPID = pg_hton32(cancel->be_pid); > + crp.cp.cancelAuthCode = pg_hton32(cancel->be_key); Others might differ, but I'd separate changing the type passed to internal_cancel() into its own commit. Greetings, Andres Freund

Re: In-placre persistance change of a relation

2022-01-05 Thread Andres Freund
Hi, On January 5, 2022 8:30:17 PM PST, Kyotaro Horiguchi wrote: >At Tue, 4 Jan 2022 16:05:08 -0800, Andres Freund wrote in >> The tap tests seems to fail on all platforms. See >> https://cirrus-ci.com/build/4911549314760704 >> >> E.g. the linux failure is >

Re: Make relfile tombstone files conditional on WAL level

2022-01-06 Thread Andres Freund
On 2022-01-06 08:52:01 -0500, Robert Haas wrote: > On Thu, Jan 6, 2022 at 3:47 AM Thomas Munro wrote: > > Another problem is that relfilenodes are normally allocated with > > GetNewOidWithIndex(), and initially match a relation's OID. We'd need > > a new allocator, and they won't be able to match

Re: Add spin_delay() implementation for Arm in s_lock.h

2022-01-06 Thread Andres Freund
so I might also just have misunderstood what you were doing... Greetings, Andres Freund

Re: Add spin_delay() implementation for Arm in s_lock.h

2022-01-06 Thread Andres Freund
Hi, On 2022-01-06 21:39:57 -0500, Tom Lane wrote: > Andres Freund writes: > > I wonder if this will show the full set of spinlock contention issues - > > isn't > > this only causing contention for one spinlock between two processes? > > I don't think so

Re: Add spin_delay() implementation for Arm in s_lock.h

2022-01-06 Thread Andres Freund
y contain > spinlocks; they all use atomic ops. They contain spinlocks too, and the naming is similar enough that I got confused: struct shm_mq { slock_t mq_mutex; We don't use them for all that much anymore though... Greetings, Andres Freund

Re: Disallow quorum uncommitted (with synchronous standbys) txns in logical replication subscribers

2022-01-07 Thread Andres Freund
the newly promoted primary. To me this just sounds like trying to shoehorn something into syncrep that it's not made for. Greetings, Andres Freund

Re: Disallow quorum uncommitted (with synchronous standbys) txns in logical replication subscribers

2022-01-07 Thread Andres Freund
Hi, On 2022-01-07 14:36:46 -0800, Jeff Davis wrote: > On Fri, 2022-01-07 at 12:22 -0800, Andres Freund wrote: > > I don't see how it can *not* cause a major performance / latency > > gotcha. You're deliberately delaying replication after all? > > Are there use cas

Re: Adding CI to our tree

2022-01-09 Thread Andres Freund
't really seem like a scalable path forward - duplicating configuration in more places doesn't seem sane. It seems it'd make more sense to teach vcregress.pl to run NO_INSTALLCHECK targets properly? ISTM that changing the options passed to pg_regress based on fetchTests() return value wouldn't be too hard? Greetings, Andres Freund

Re: Windows crash / abort handling

2022-01-09 Thread Andres Freund
Hi, On 2021-10-06 14:11:51 +0800, Craig Ringer wrote: > On Wed, 6 Oct 2021, 03:30 Andres Freund, wrote: > > > Hi, > > > > > > My first attempt was to try to use the existing crashdump stuff in > > pgwin32_install_crashdump_handler(). That's not real

Re: Adding CI to our tree

2022-01-09 Thread Andres Freund
Hi, On 2021-10-02 12:59:09 -0700, Andres Freund wrote: > On 2021-10-02 11:05:20 -0400, Tom Lane wrote: > > I don't know enough about Windows to evaluate 0001, but I'm a little > > worried about it because it looks like it's changing our *production* > > error

Re: pg_dump/restore --no-tableam

2022-01-09 Thread Andres Freund
01.356] [02:43:01.356]^ > + > + Yup... Greetings, Andres Freund

Re: TAP test to cover "EndOfLogTLI != replayTLI" case

2022-01-09 Thread Andres Freund
isn't quite reliable. It occasionally fails on freebsd, macos, linux and always on windows (starting with the new CI stuff, before the test wasn't run). See https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest/36/3427 Greetings, Andres Freund

pg_upgrade verbosity when redirecting output to log file

2022-01-09 Thread Andres Freund
huge cost compared to actually copying a file, but it's not entirely free either. Greetings, Andres Freund [1] called from transfer_all_new_tablespaces() -> parallel_transfer_all_new_dbs() -> transfer_all_new_dbs() -> transfer_single_new_db() -> transfer_relfile() ->

Re: pg_upgrade verbosity when redirecting output to log file

2022-01-09 Thread Andres Freund
Hi, On 2022-01-09 20:28:40 -0800, Andres Freund wrote: > Turns out that it only happens when the output is not a tty. And I notice it > whenever I redirect the log output to a file, pipe, or such. Ah. More precisely, it happens when doing make -s -Otarget -j32 check-world, but *not

Re: pg_upgrade verbosity when redirecting output to log file

2022-01-09 Thread Andres Freund
Hi, On 2022-01-10 01:14:32 -0500, Tom Lane wrote: > Andres Freund writes: > Fun! That seems to me to be a strong argument for not letting > the behavior vary depending on isatty(). Yea. > I think I'd vote for just nuking that output altogether. > It seems of very dubi

Re: Use -fvisibility=hidden for shared libraries

2022-01-10 Thread Andres Freund
Hi, On 2022-01-10 19:01:36 -0500, Tom Lane wrote: > Andres Freund writes: > > [ v2-0001-Use-hidden-visibility-for-shared-libraries-where-.patch ] > > This seems like a good idea, but it's failing to apply right now, > mainly because of some cleanup I did in e04a8059a.

Re: Use -fvisibility=hidden for shared libraries

2022-01-10 Thread Andres Freund
Hi, On 2022-01-10 21:11:07 -0600, Justin Pryzby wrote: > On Mon, Jan 10, 2022 at 07:01:36PM -0500, Tom Lane wrote: > > Andres Freund writes: > > > [ v2-0001-Use-hidden-visibility-for-shared-libraries-where-.patch ] > > > > This seems like a good idea, but i

Re: Use -fvisibility=hidden for shared libraries

2022-01-10 Thread Andres Freund
Hi, On 2022-01-10 23:44:06 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2022-01-10 21:11:07 -0600, Justin Pryzby wrote: > >> Without the patch, it fails under windows like: > > > Ah, yea. It's about matching the declarations in ltree.h with t

Re: Windows crash / abort handling

2022-01-10 Thread Andres Freund
Hi, On 2022-01-10 10:57:00 -0500, Andrew Dunstan wrote: > On 10/5/21 15:30, Andres Freund wrote > > The above ends up dumping all crashes into a single file, but that can > > probably be improved. But cdb is so gnarly that I wanted to stop looking > > once > > I got t

Re: Windows crash / abort handling

2022-01-11 Thread Andres Freund
Hi, On 2022-01-11 12:01:42 -0500, Andrew Dunstan wrote: > On 1/11/22 02:51, Andres Freund wrote: > > It'd be a bit of a fight with cdb's awfully documented and quirky > > scripting [1], but the best solution would probably be to just use an > > environment vari

Re: Windows vs recovery tests

2022-01-12 Thread Andres Freund
n\timeout.exe" -v --kill-after=35m 30m perl path/to/vcregress.pl ... Perhaps worth re-adding? Greetings, Andres Freund

Re: Stream replication test fails of cfbot/windows server 2019

2022-01-12 Thread Andres Freund
it's relevant that the .cirrus file uses "unix sockets" on windows as well, to avoid port conflicts (otherwise I saw frequent spurious test failures due to port conflicts). # Avoids port conflicts between concurrent tap test runs PG_TEST_USE_UNIX_SOCKETS: 1 It's not particularly hard to imagine that either our "windows unix socket" support still has some bugs, or that windows' implementation of unix sockets is borked. Greetings, Andres Freund

Re: Add non-blocking version of PQcancel

2022-01-12 Thread Andres Freund
ke I've done for TLS reads and unencrypted reads. Since in > case of a cancel connection an EOF is actually expected, and should not > be treated as an error. The failures do not seem related to this. Greetings, Andres Freund

Re: Windows vs recovery tests

2022-01-12 Thread Andres Freund
Hi, On 2022-01-12 15:58:26 -0800, Andres Freund wrote: > On 2022-01-12 14:34:00 -0500, Andrew Dunstan wrote: > > For some considerable time the recovery tests have been at best flaky on > > Windows, and at worst disastrous (i.e. they can hang rather than just > > fail). It

Re: Windows vs recovery tests

2022-01-12 Thread Andres Freund
Hi, On 2022-01-12 18:25:26 -0800, Andres Freund wrote: > There's one interesting patch that also times out just on windows, albeit in > another test group: > https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest/36/2096 > > This IMO looks likely to be a bug i

Re: Windows vs recovery tests

2022-01-12 Thread Andres Freund
Hi, On 2022-01-12 20:03:14 -0800, Andres Freund wrote: > So it looks like psql is hanging somewhere after that. I assume with an error > popup that nobody can click on :/. Not quite. Psql is actually just logging output in an endless loop. I connected with cdb.exe. kP: `00

Re: psql - add SHOW_ALL_RESULTS option

2022-01-12 Thread Andres Freund
tatus == PGRES_COPY_OUT || > result_status == PGRES_COPY_IN) > > > So we called ClearOrSaveResult() with did a PQclear(), and then we go and call > PQresultStatus(). Greetings, Andres Freund

Re: Time to drop plpython2?

2022-01-13 Thread Andres Freund
or ~1 release, so things could mopped up without blocking everyone. > I think what I wrote in the message I'm responding to is the best way > forward, although maybe with less than 60 days now. Already happened, but I'm a belated +1 ;) Greetings, Andres Freund

Re: Adding CI to our tree

2022-01-13 Thread Andres Freund
Hi, On 2021-12-30 17:46:52 -0800, Andres Freund wrote: > I plan to push this after another cycle of tests passing (and driving > over the bay bridge...) unless I hear protests. I noticed that it's harder to see compile warnings on msvc in CI than it was at an earlier point. There u

Re: Adding CI to our tree

2022-01-13 Thread Andres Freund
Hi, On 2022-01-10 16:07:48 -0600, Justin Pryzby wrote: > On Sun, Jan 09, 2022 at 11:57:44AM -0800, Andres Freund wrote: > > On 2022-01-09 13:16:50 -0600, Justin Pryzby wrote: > > > diff --git a/contrib/test_decoding/Makefile > > > b/contrib/test_decoding/Makef

Re: Adding CI to our tree

2022-01-13 Thread Andres Freund
> chown -R postgres:postgres ${CCACHE_DIR} > echo '* - memlock 134217728' > /etc/security/limits.d/postgres.conf > -su postgres -c "ulimit -l -H && ulimit -l -S" > +su postgres -c "ulimit -l -H && ulimit -l -S" # XXX Hm? Greetings, Andres Freund

Re: Avoid erroring out when unable to remove or parse logical rewrite files to save checkpoint work

2022-01-13 Thread Andres Freund
This seems like it'd make failures to remove the files practically invisible. Which'd have it's own set of problems? What motivated proposing this change? Greetings, Andres Freund

Re: [PATCH] Disable bgworkers during servers start in pg_upgrade

2022-01-13 Thread Andres Freund
Hi, On 2022-01-12 17:54:31 +0800, Julien Rouhaud wrote: > On Sat, Aug 28, 2021 at 10:40:42AM +0900, Michael Paquier wrote: > > On Fri, Aug 27, 2021 at 12:28:42PM -0700, Andres Freund wrote: > > > Isn't that just going to end up with extension code erroring out and/or >

Re: Add client connection check during the execution of the query

2022-01-13 Thread Andres Freund
it would have to be at the bottom or top, not between if (work_to_do) and WaitLatch(). Greetings, Andres Freund

Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

2022-01-14 Thread Andres Freund
to remove a lot of the messy workarounds we have in socket.c etc. But: It wouldn't do much good here, because the socket is not a socket created by socket.c but by libpq :(. Greetings, Andres Freund

Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

2022-01-14 Thread Andres Freund
Hi, On 2022-01-14 12:28:48 -0800, Andres Freund wrote: > But once all the data is read, walsender.c will do another > WaitLatchOrSocket(), which does WSAEventSelect(), clearing the "internal event > record" and loosing the FD_CLOSE. Walreceiver only started using WES in 20

Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

2022-01-14 Thread Andres Freund
Hi, On 2022-01-15 10:59:00 +1300, Thomas Munro wrote: > On Sat, Jan 15, 2022 at 9:28 AM Andres Freund wrote: > > I think it doesn't even need to touch socket.c to cause breakage. Using two > > different WaitEventSets is enough. > > Right. I was interested in your obser

Re: Adding CI to our tree

2022-01-14 Thread Andres Freund
dfarm doesn't start to run tests multiple times... Greetings, Andres Freund

Re: A test for replay of regression tests

2022-01-14 Thread Andres Freund
fix... But if you're talking about fairywren's failure, it's more than not running some tests, it's that we do not test windows mingw outside of cross compilation. Greetings, Andres Freund

Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

2022-01-14 Thread Andres Freund
Hi, On 2022-01-15 13:19:42 +1300, Thomas Munro wrote: > On Sat, Jan 15, 2022 at 11:44 AM Andres Freund wrote: > > > The patch Alexander tested most recently uses a tri-state eof flag [...] > > > > What about instead giving WalReceiverConn an internal WaitEventSet,

pg_basebackup WAL streamer shutdown is bogus - leading to slow tests

2022-01-16 Thread Andres Freund
Hi, I was wondering in [1] what we could do about the slowest tests on windows. On 2021-12-31 11:25:28 -0800, Andres Freund wrote: > Picking a random successful cfbot run [1] I see the following tap tests taking > more than 20 seconds: > > 67188 ms pg_basebackup t/010_pg_basebackup.p

Re: pg_basebackup WAL streamer shutdown is bogus - leading to slow tests

2022-01-16 Thread Andres Freund
Hi, On 2022-01-16 17:39:11 +0100, Magnus Hagander wrote: > On Sun, Jan 16, 2022 at 5:36 PM Magnus Hagander wrote: > > > > On Sun, Jan 16, 2022 at 5:34 PM Tom Lane wrote: > > > > > > Andres Freund writes: > > > > I don't immediately see a solu

Re: pg_basebackup WAL streamer shutdown is bogus - leading to slow tests

2022-01-16 Thread Andres Freund
Hi, On 2022-01-16 15:28:00 -0800, Andres Freund wrote: > I hacked that up last night. And a fix or two later, it seems to be > working. What I'd missed at first is that the event needs to be reset in > reached_end_position(), otherwise we'll busy loop. > > I wonder if

Re: Getting better results from valgrind leak tracking

2021-03-16 Thread Andres Freund
Hi, On Tue, Mar 16, 2021, at 20:01, Tom Lane wrote: > Andres Freund writes: > > On 2021-03-16 19:36:10 -0400, Tom Lane wrote: > >> It doesn't appear to me that we leak much at all, at least not if you > >> are willing to take "still reachable" blocks a

Re: Getting better results from valgrind leak tracking

2021-03-16 Thread Andres Freund
Hi, On 2021-03-16 20:50:17 -0700, Andres Freund wrote: > What I meant was that I didn't understand how there's not a leak > danger when compilation fails halfway through, given that the context > in question is below TopMemoryContext and that I didn't see a relevant > T

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Andres Freund
Hi, On Wed, Mar 17, 2021, at 07:16, Tom Lane wrote: > Andres Freund writes: > > On 2021-03-16 20:50:17 -0700, Andres Freund wrote: > Meanwhile, I'm still trying to understand why valgrind is whining > about the rd_indexcxt identifier strings. AFAICS it shouldn't. I fou

<    1   2   3   4   5   6   7   8   9   10   >