Re: [PATCH] Identify LWLocks in tracepoints

2021-04-13 Thread Andres Freund
Hi, On 2021-04-13 14:25:23 -0400, Robert Haas wrote: > On Mon, Apr 12, 2021 at 11:06 PM Andres Freund wrote: > You could identify every lock by a tranche ID + an array offset + a > "tranche instance ID". But where would you store the tranche instance > ID to make it rea

Re: xl_heap_header alignment?

2020-07-21 Thread Andres Freund
Hi, On July 21, 2020 10:45:37 AM PDT, Antonin Houska wrote: >I don't quite understand this part of the comment of the xl_heap_header >structure: > >* NOTE: t_hoff could be recomputed, but we may as well store it because > * it will come for free due to alignment considerations. > >What are the a

'with' regression tests fails rarely (and spuriously)

2020-07-23 Thread Andres Freund
learly not a common occurance on my machine. Greetings, Andres Freund

Re: 'with' regression tests fails rarely (and spuriously)

2020-07-23 Thread Andres Freund
Hi, On 2020-07-23 13:05:32 -0400, Tom Lane wrote: > Andres Freund writes: > > I've twice seen the below failure when running tests in a loop (to > > verify another rare issue in a patch is fixed): > > Weird. It sort of looks like autovacuum came along and changed th

HOT vs freezing issue causing "cannot freeze committed xmax"

2020-07-23 Thread Andres Freund
whether it's possible today, it seems extremely fragile. ISTM we should at least have a bunch of additional error checks in the HOT branch for HEAPTUPLE_DEAD. Greetings, Andres Freund

Re: Making CASE error handling less surprising

2020-07-23 Thread Andres Freund
rhead to be worthwhile? That'd require using a subtransaction for expression simplification. That'd be way too high overhead. Given how often we've had a need to call functions while handling errors, I do wonder if it'd be worthwhile and feasible to mark functions as being safe to call without subtransactions, or mark them as not erroring out (e.g. comparators would usually be safe). Greetings, Andres Freund

heap_abort_speculative() sets xmin to Invalid* without HEAP_XMIN_INVALID

2020-07-23 Thread Andres Freund
ing tuples without it - therefore we will have to forever take care about encountering that combination :(. Perhaps we should instead or additionally make HeapTupleHeaderXminInvalid() explicitly check for InvalidTransactionId? Greetings, Andres Freund

Re: Making CASE error handling less surprising

2020-07-23 Thread Andres Freund
Hi, On 2020-07-23 15:43:44 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2020-07-23 18:50:32 +0100, Dagfinn Ilmari Mannsåker wrote: > >> Would it be feasible to set up an exception handler when constant- > >> folding cases that might not be reached, and leave t

Re: Making CASE error handling less surprising

2020-07-23 Thread Andres Freund
lky because we'd need to track more state across recursive expression_tree_mutator() calls. Greetings, Andres Freund

Re: Making CASE error handling less surprising

2020-07-23 Thread Andres Freund
Hi, On 2020-07-23 16:34:25 -0400, Tom Lane wrote: > Andres Freund writes: > > Is there an argument to continue simplifying expressions within case > > when only involving "true" constants even with not leakproof functions, > > but only simplify "pseudo"

Re: Making CASE error handling less surprising

2020-07-23 Thread Andres Freund
Hi, On 2020-07-23 13:42:08 -0700, Andres Freund wrote: > On 2020-07-23 16:34:25 -0400, Tom Lane wrote: > > Andres Freund writes: > > > It doesn't seem like it'd be too hard to implement that, but that it'd > > > probably be fairly bulky be

Re: Making CASE error handling less surprising

2020-07-23 Thread Andres Freund
Hi, On 2020-07-23 16:56:44 -0400, Tom Lane wrote: > Andres Freund writes: > > Hm. Would SQL function inlining be a problem? It looks like that just > > substitutes parameters. Before calling > > eval_const_expressions_mutator(). So we'd not know not to evaluate such &g

Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64

2020-07-23 Thread Andres Freund
pretty high contention on a few lwlocks. Could you post the numbers? Greetings, Andres Freund

Re: [Patch] ALTER SYSTEM READ ONLY

2020-07-23 Thread Andres Freund
/vacuum.h" > #include "lib/integerset.h" > #include "miscadmin.h" The number of places that now need this new header - pretty much the same set of files that do XLogInsert, already requiring an xlog* header to be included - drives me further towards the conclusion that it's not a good idea to have it separate. > extern void ProcessInterrupts(void); > > +#ifdef USE_ASSERT_CHECKING > +typedef enum > +{ > + WALPERMIT_UNCHECKED, > + WALPERMIT_CHECKED, > + WALPERMIT_CHECKED_AND_USED > +} WALPermitCheckState; > + > +/* in access/walprohibit.c */ > +extern WALPermitCheckState walpermit_checked_state; > + > +/* > + * Reset walpermit_checked flag when no longer in the critical section. > + * Otherwise, marked checked and used. > + */ > +#define RESET_WALPERMIT_CHECKED_STATE() \ > +do { \ > + walpermit_checked_state = CritSectionCount ? \ > + WALPERMIT_CHECKED_AND_USED : WALPERMIT_UNCHECKED; \ > +} while(0) > +#else > +#define RESET_WALPERMIT_CHECKED_STATE() ((void) 0) > +#endif > + Why are these in headers? And why is this tied to CritSectionCount? Greetings, Andres Freund

Re: Any objections to implementing LogicalDecodeMessageCB for pgoutput?

2020-07-24 Thread Andres Freund
gt; postgres. > > Implementing LogicalDecodeMessageCB provides some synchronization facility > that is not easily replicated. It's definitely useful. Probably needs to be parameter that signals whether they should be sent out? Greetings, Andres Freund

Re: Making CASE error handling less surprising

2020-07-24 Thread Andres Freund
while making the > behavior in more complex cases harder to understand is not necessarily > a win. Well, if we formalize the desired behaviour it's probably a lot easier to work towards implementing it in additional cases (like subselects). It doesn't seem to hard to keep track of whether a specific subquery can be evaluate constants in a certain way, if that's what we need. Greetings, Andres Freund

Re: HOT vs freezing issue causing "cannot freeze committed xmax"

2020-07-24 Thread Andres Freund
Hi, On 2020-07-24 11:06:58 -0400, Robert Haas wrote: > On Thu, Jul 23, 2020 at 2:10 PM Andres Freund wrote: > > In the case the HOT logic triggers, we'll call > > heap_prepare_freeze_tuple() even when the tuple is dead. > > I think this is very bad. I've a

Re: Making CASE error handling less surprising

2020-07-24 Thread Andres Freund
Hi, On 2020-07-24 19:03:30 +0200, Pavel Stehule wrote: > pá 24. 7. 2020 v 18:49 odesílatel Andres Freund napsal: > > Wouldn't the rule that I proposed earlier, namely that sub-expressions > > that involve only "proper" constants continue to get evaluated even &g

Re: Improving connection scalability: GetSnapshotData()

2020-07-24 Thread Andres Freund
s no call to GetOldestXmin() in pgstatapprox and pg_visibility after patch 0002? And similarly, the PROCARRAY_* references are also removed in the same patch? Greetings, Andres Freund

Re: Making CASE error handling less surprising

2020-07-24 Thread Andres Freund
Hi, On 2020-07-23 22:34:53 -0400, Tom Lane wrote: > Andres Freund writes: > > I'm a bit worried about a case like: > > > CREATE FUNCTION yell(int, int) > > RETURNS int > > IMMUTABLE > > LANGUAGE SQL AS $$ > >SELECT CASE WHEN $1 != 0 THEN 17 /

Re: Making CASE error handling less surprising

2020-07-24 Thread Andres Freund
On July 24, 2020 10:30:37 AM PDT, Pavel Stehule wrote: >pá 24. 7. 2020 v 19:13 odesílatel Andres Freund >napsal: >> Your earlier example of a WHEN ... THEN upper('constant') ... would >> still have the upper('constant') be evaluated, because it doesn&#

Re: hashagg slowdown due to spill changes

2020-07-24 Thread Andres Freund
Hi, On 2020-06-12 14:37:15 -0700, Andres Freund wrote: > On 2020-06-11 11:14:02 -0700, Jeff Davis wrote: > > On Thu, 2020-06-11 at 10:45 -0700, Andres Freund wrote: > > > Did you run any performance tests? > > > > Yes, I reproduced your ~12% regression f

Re: Improving connection scalability: GetSnapshotData()

2020-07-24 Thread Andres Freund
On 2020-07-24 18:15:15 -0300, Ranier Vilela wrote: > Em sex., 24 de jul. de 2020 às 14:16, Andres Freund > escreveu: > > > On 2020-07-24 14:05:04 -0300, Ranier Vilela wrote: > > > Latest Postgres > > > Windows 64 bits > > > msvc 2019 64 bits > &

Re: Threading in BGWorkers (!)

2020-07-30 Thread Andres Freund
ve to figure out how to do that on our > supported systems. Couldn't this be driven by --disable-thread-safety? Greetings, Andres Freund

Re: Threading in BGWorkers (!)

2020-07-30 Thread Andres Freund
get it. Why not? Our answer to threading inside functions has been, for quite a while, that it's kinda ok if the threads never call into postgres and can never escape the lifetime of a function. But that's not actually really safe due to the signal handler issue. Whether it's a normal backend or a bgworker doesn't really play a role here, no? Greetings, Andres Freund

Re: new heapcheck contrib module

2020-07-30 Thread Andres Freund
ust ignore the problem, or try to work around it by holding XactTruncationLock. Greetings, Andres Freund

Re: Why is pq_begintypsend so slow?

2020-07-31 Thread Andres Freund
rein. > In appendStringInfoChar, why do we need to cast to restrict twice? Can > we not just do something like this: > > char *pg_restrict ep = str->data+str->len; > ep[0] = ch; > ep[1] = '\0'; I don't think that'd tell the compiler that this couldn't overlap with str itself? A single 'restrict' can never (?) help, you need *two* things that are marked as not overlapping in any way. Greetings, Andres Freund

Re: Ought to use heap_multi_insert() for pg_attribute/depend insertions?

2020-07-31 Thread Andres Freund
t. > > > > Thanks. Attached is a polished version of the patch that I intend to > > commit for pg_attribute and pg_shdepend. Let's look again at > > pg_depend later, as there are also links with the handling of > > dependencies for ALTER TABLE mainly. > > Looks good, thanks. Nice work! Greetings, Andres Freund

Re: new heapcheck contrib module

2020-07-31 Thread Andres Freund
. With 'announcing' I mean using the normal mechanism for avoiding the clog being truncated for values one might look up. Which is announcing the oldest xid one may look up in PGXACT->xmin. Greetings, Andres Freund

Re: refactoring basebackup.c

2020-07-31 Thread Andres Freund
Seems odd to have to send payload messages with 0 payload just because we want to update progress (in case of uploading to e.g. S3). And I think it'd be nice if we could have a more extensible progress measurement approach than a fixed length prefix. E.g. it might be nice to allow it to report both the overall progress, as well as a per archive progress. Or we might want to send progress when uploading to S3, even when not having pre-calculated the total size of the data directory. Greetings, Andres Freund

Re: Why is pq_begintypsend so slow?

2020-07-31 Thread Andres Freund
Hi, On 2020-07-31 12:28:04 -0400, Robert Haas wrote: > On Fri, Jul 31, 2020 at 11:50 AM Andres Freund wrote: > > I hope the above makes this make sene now? It's about subsequent uses of > > the StringInfo, rather than the body of resetStringInfo itself. > > That do

Re: new heapcheck contrib module

2020-07-31 Thread Andres Freund
Hi, On 2020-07-31 12:42:51 -0400, Robert Haas wrote: > On Fri, Jul 31, 2020 at 12:33 PM Andres Freund wrote: > > I'm not sure what I was thinking "back then", but right now I'd argue > > that the best lock against vacuum isn't a SUE, but announcing the >

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-07-31 Thread Andres Freund
licit hashtable probes for all pages iff the size of the relation is < s_b / 500 or so. That'll address the case where small tables are frequently dropped - and dropping large relations is more expensive from the OS and data loading perspective, so it's not gonna happen as often. Greetings, Andres Freund

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-07-31 Thread Andres Freund
Hi, On 2020-07-31 15:50:04 -0400, Tom Lane wrote: > Andres Freund writes: > > Indeed. The buffer mapping hashtable already is visible as a major > > bottleneck in a number of workloads. Even in readonly pgbench if s_b is > > large enough (so the hashtable is larger than the

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-08-05 Thread Andres Freund
y end up reloading tb->data a bunch of times, because we do the access to ->data as SH_ELEMENT_TYPE *entry = &tb->data[curelem]; Think we should instead store tb->data in a local variable. Greetings, Andres Freund

Re: PROC_IN_ANALYZE stillborn 13 years ago

2020-08-05 Thread Andres Freund
without disabled cost limiting for analyze (or slow IO), the snapshot that analyze holds is often by far the transaction with the oldest xmin. It's not entirely trivial to fix (just ignoring it could lead to detoasting issues), but also not that. Only mildly against because it'd not be hard to reintroduce once we need it. Greetings, Andres Freund

Re: PROC_IN_ANALYZE stillborn 13 years ago

2020-08-06 Thread Andres Freund
o hold a snapshot to ensure that we don't get failures looking up toasted datums. IIRC there were some other similar issues that I can't quite recall right now. Greetings, Andres Freund

Re: PROC_IN_ANALYZE stillborn 13 years ago

2020-08-06 Thread Andres Freund
y abuse fields inside SnapshotData to return data that can't be returned through the normal API. It'd probably be better to have more explicit APIs for these, rather than going through snapshot. > It's not clear to me that it would even be correct to categorize those > somewhat-different results as "less accurate." Tuples that are > invisible to a query often have performance consequences very similar > to visible tuples, in terms of the query run time. +1 Greetings, Andres Freund

Re: Issue with cancel_before_shmem_exit while searching to remove a particular registered exit callbacks

2020-08-07 Thread Andres Freund
, i.e. basically this would only be usable for PG_ENSURE_ERROR_CLEANUP and nothing else. And even there it seems like it's pretty easy to get into a situation where it's not safe. Greetings, Andres Freund

Re: PROC_IN_ANALYZE stillborn 13 years ago

2020-08-07 Thread Andres Freund
Hi, On 2020-08-06 18:02:26 -0400, Tom Lane wrote: > Andres Freund writes: > > In fact using conceptually like a new snapshot for each sample tuple > > actually seems like it'd be somewhat of an improvement over using a > > single snapshot. > > Dunno, that feels l

Re: Issue with cancel_before_shmem_exit while searching to remove a particular registered exit callbacks

2020-08-10 Thread Andres Freund
st. There's no other uses of cancel_before_shmem_exit afaict. I guess alternatively we at some point might just need a more complex callback system, where one can specify where in relation to another callback a callback needs to be registered etc. Greetings, Andres Freund

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-08-10 Thread Andres Freund
Hi, On 2020-07-09 10:25:14 +1200, David Rowley wrote: > On Thu, 9 Jul 2020 at 04:53, Andres Freund wrote: > > > > On 2020-05-20 23:44:27 +1200, David Rowley wrote: > > > I've attached a patch which implements this. The new node type is > > > called "

Re: Switch to multi-inserts for pg_depend

2020-08-10 Thread Andres Freund
number of dependencies to be inserted, and then go ahead and create the right number of slots? > From fcc0a11e9fc94d2fedc71dd10ba2a23713225963 Mon Sep 17 00:00:00 2001 > From: Michael Paquier > Date: Fri, 7 Aug 2020 15:14:51 +0900 > Subject: [PATCH 2/2] Switch to multi-insert dependenci

Re: walsender waiting_for_ping spuriously set

2020-08-10 Thread Andres Freund
On 2020-08-06 18:55:58 -0400, Alvaro Herrera wrote: > Ashutosh Bapat noticed that WalSndWaitForWal() is setting > waiting_for_ping_response after sending a keepalive that does *not* > request a reply. The bad consequence is that other callers that do > require a reply end up in not sending a keepa

Re: Add LWLock blocker(s) information

2020-08-10 Thread Andres Freund
per-common or super-difficult to figure out. Most of the cases where this kind of information really is interesting seem to benefit a lot from having stack information available. That obviously has overhead, so we don't want the cost all the time. The script at https://postgr.es/m/20170622210845.d2hsbqv6rxu2tiye%40alap3.anarazel.de can give you results like e.g. https://anarazel.de/t/2017-06-22/pgsemwait_64_async.svg Greetings, Andres Freund

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-08-10 Thread Andres Freund
Hi, On 2020-08-11 17:23:42 +1200, David Rowley wrote: > On Tue, 11 Aug 2020 at 12:21, Andres Freund wrote: > > > > On 2020-07-09 10:25:14 +1200, David Rowley wrote: > > > On Thu, 9 Jul 2020 at 04:53, Andres Freund wrote: > > > > I'm not convinced

Re: Can I test Extended Query in core test framework

2020-08-11 Thread Andres Freund
it. I am not a perl person by any stretch (even though that's where I started...). But we can (and do iirc) have tests that just use libpq, so it should be possible to test things like this at a bit higher cost. Greetings, Andres Freund

Re: Improving connection scalability: GetSnapshotData()

2020-08-11 Thread Andres Freund
s than "relative". > > The best I've got is "anchor" xid. It is an xid that is known to > limit nextFullXid's range while the receiving function runs. Thinking about it, I think that relative is a good descriptor. It's just that 'via' is weird. How about: FullXidRelativeTo? Greetings, Andres Freund

Re: Dependencies for partitioned indexes are still a mess

2020-08-12 Thread Andres Freund
them to be the same between primary / standby, without hiding bugs. Greetings, Andres Freund --- /tmp/first.sql 2020-08-12 15:01:11.810862861 -0700 +++ /tmp/second.sql 2020-08-12 15:02:05.877709572 -0700 @@ -15392,9 +15392,9 @@ -- CREATE TABLE public.test_type_diff2_c1 ( +int_two smallin

run pgindent on a regular basis / scripted manner

2020-08-12 Thread Andres Freund
nt on a more regular basis, that should be much less the case. It'd also help newer developers who we sometimes tell to use pgindent - which doesn't really work. Greetings, Andres Freund [1] ./src/tools/pgindent/pgindent $(git diff-tree --no-commit-id --name-only -r upstream/master.

Re: Dependencies for partitioned indexes are still a mess

2020-08-12 Thread Andres Freund
Hi, On 2020-08-12 18:29:16 -0400, Tom Lane wrote: > Andres Freund writes: > > I've attached the diff between first.sql and second.sql. Here's the > > highlights: > > As I recall, the differences in b_star etc are expected, because > pg_dump reorders that table&

Re: run pgindent on a regular basis / scripted manner

2020-08-12 Thread Andres Freund
Hi, On 2020-08-12 16:08:50 -0700, Jesse Zhang wrote: > On Wed, Aug 12, 2020 at 3:34 PM Andres Freund wrote: > > > > Hi, > > > > When developing patches I find it fairly painful that I cannot re-indent > > patches with pgindent without also seeing a lot of inde

Re: Add LWLock blocker(s) information

2020-08-12 Thread Andres Freund
Hi, On 2020-08-12 16:47:13 -0700, Peter Geoghegan wrote: > On Mon, Aug 10, 2020 at 5:41 PM Andres Freund wrote: > > Most of the cases where this kind of information really is interesting > > seem to benefit a lot from having stack information available. That > > obviousl

Re: [HACKERS] Dynamic instrumentation of lwlock wait times (lwlock flamegraphs)

2020-08-12 Thread Andres Freund
Hi, On 2017-06-22 14:08:45 -0700, Andres Freund wrote: > At pgcon some people were talking about the difficulty of instrumenting > the time actually spent waiting for lwlocks and related measurements. > I'd mentioned that linux these days provides infrastructure to measure &g

Re: run pgindent on a regular basis / scripted manner

2020-08-13 Thread Andres Freund
of typedefs.list updates etc. So I'm against that, even if we had a patchprep target that didn't need external tools. Greetings, Andres Freund

Re: pg_dump from v13 is slow

2020-08-13 Thread Andres Freund
ely idle. If it's close to 100% busy for a significant part of that time, it'd be useful to get a perf profile. If it's largely queries to the server that are the issue, logging those would be relevant. Greetings, Andres Freund

Re: Improving connection scalability: GetSnapshotData()

2020-08-15 Thread Andres Freund
er than a real > test-and-elog. Assert doesn't seem like an appropriate way to check for > plausible data corruption cases.) Robert, and to a lesser degree you, gave me quite a bit of grief over converting nearby asserts to elogs. I agree it'd be better if it were an assert, but ... Greetings, Andres Freund

Re: run pgindent on a regular basis / scripted manner

2020-08-15 Thread Andres Freund
Hi, On 2020-08-15 13:47:41 +0200, Peter Eisentraut wrote: > On 2020-08-13 00:34, Andres Freund wrote: > > I e.g. just re-indented patch 0001 of my GetSnapshotData() series and > > most of the hunks were entirely unrelated. Despite the development > > window for 14 having onl

Re: Improving connection scalability: GetSnapshotData()

2020-08-16 Thread Andres Freund
Hi, On 2020-08-15 09:42:00 -0700, Andres Freund wrote: > On 2020-08-15 11:10:51 -0400, Tom Lane wrote: > > We have two essentially identical buildfarm failures since these patches > > went in: > > > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=damselfly&a

Re: Improving connection scalability: GetSnapshotData()

2020-08-16 Thread Andres Freund
On 2020-08-16 14:30:24 -0400, Tom Lane wrote: > Andres Freund writes: > > 690 successful runs later, it didn't trigger for me :(. Seems pretty > > clear that there's another variable than pure chance, otherwise it seems > > like that number of runs should have hit

Re: Improving connection scalability: GetSnapshotData()

2020-08-16 Thread Andres Freund
Hi, On 2020-08-16 16:17:23 -0400, Tom Lane wrote: > I wrote: > > It seems entirely likely that there's a timing component in this, for > > instance autovacuum coming along at just the right time. > > D'oh. The attached seems to make it 100% reproducible. Great! It interestingly didn't work as

Re: Improving connection scalability: GetSnapshotData()

2020-08-16 Thread Andres Freund
Hi, On 2020-08-16 13:31:53 -0700, Andres Freund wrote: > I now luckily have a rr trace of the problem, so I hope I can narrow it > down to the original problem fairly quickly. Gna, I think I see the problem. In at least one place I wrongly accessed the 'dense' array of in-pro

Re: Improving connection scalability: GetSnapshotData()

2020-08-16 Thread Andres Freund
Hi, On 2020-08-16 13:52:58 -0700, Andres Freund wrote: > On 2020-08-16 13:31:53 -0700, Andres Freund wrote: > > I now luckily have a rr trace of the problem, so I hope I can narrow it > > down to the original problem fairly quickly. > > Gna, I think I see the problem. In

Re: Improving connection scalability: GetSnapshotData()

2020-08-16 Thread Andres Freund
Hi, On 2020-08-16 14:11:46 -0700, Andres Freund wrote: > On 2020-08-16 13:52:58 -0700, Andres Freund wrote: > > On 2020-08-16 13:31:53 -0700, Andres Freund wrote: > > Gna, I think I see the problem. In at least one place I wrongly > > accessed the 'dense' arra

Re: Improving connection scalability: GetSnapshotData()

2020-08-16 Thread Andres Freund
Hi, On 2020-08-16 17:28:46 -0400, Tom Lane wrote: > Andres Freund writes: > > For the first, one issue is that there's no obviously good candidate for > > an uninitialized xid. We could use something like FrozenTransactionId, > > which may never be in the procarray. B

Re: run pgindent on a regular basis / scripted manner

2020-08-17 Thread Andres Freund
Hi, On 2020-08-17 13:54:15 -0400, Bruce Momjian wrote: > On Sat, Aug 15, 2020 at 01:44:34PM -0400, Tom Lane wrote: > > Andres Freund writes: > > > Without a properly indented baseline that's hard to do, because it'll > > > cause damage all over. So I don&#

Re: run pgindent on a regular basis / scripted manner

2020-08-17 Thread Andres Freund
Hi, On 2020-08-15 13:44:34 -0400, Tom Lane wrote: > Andres Freund writes: > > One thing is that some here are actively against manually adding entries > > to typedefs.list. > > I've been of the opinion that it's pointless to do so under the current > regime w

Re: Corruption during WAL replay

2020-08-17 Thread Andres Freund
e > GetVirtualXIDsDelayingChkpt() function call in CreateCheckPoint(). We could > have a similar mechanism to wait out the truncations before *completing* a > checkpoint. What I outlined earlier *is* essentially a way to do so, by preventing checkpointing from finishing the buffer scan while a dangerous state exists. Greetings, Andres Freund

prepared transaction isolation tests

2020-08-18 Thread Andres Freund
Hi, This thread started on committers, at https://www.postgresql.org/message-id/20200818234532.uiafo5br5lo6zhya%40alap3.anarazel.de In it I wanted to add a isolation test around prepared transactions: On 2020-08-18 16:45:32 -0700, Andres Freund wrote: > I think it's worth adding an i

Re: prepared transaction isolation tests

2020-08-18 Thread Andres Freund
Hi, On 2020-08-18 22:24:20 -0400, Tom Lane wrote: > Andres Freund writes: > > ISTM we should just add an alternative output for disabled prepared > > xacts, and re-add the test? > > I believe the buildfarm runs the isolation step with "make installcheck", > so

Re: Creating a function for exposing memory usage of backend process

2020-08-19 Thread Andres Freund
I'm afraid. Many many users don't have access to the server log. > If it stays, I'd like to see restrictions on who can read the view. As long as access is grantable rather than needing a security definer wrapper I'd be fine with that. Greetings, Andres Freund

Re: Creating a function for exposing memory usage of backend process

2020-08-19 Thread Andres Freund
Hi, On 2020-08-19 21:29:06 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2020-08-19 11:01:37 -0400, Tom Lane wrote: > >> I agree with that, but I think this patch has a bigger problem: > >> why bother at all? It seems like a waste of code space and fut

Re: Fix a couple of typos in JIT

2020-08-20 Thread Andres Freund
Hi, On 2020-08-20 15:59:26 +0530, Abhijit Menon-Sen wrote: > The original sentence may not be the most shining example of > sentence-ry, but it is correct, and removing the "That" breaks it. That made me laugh ;) David, sounds good, after adapting to Abhijit's concerns

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-08-24 Thread Andres Freund
ll predictable branch. But it's also easy enough to just have a different Exec* function for the caching and non-caching case, should that turn out to be a problem. > And also, as the JIT saga continues, aren't we going to get plan trees > recompiled too, at which point it won't matter much? That's a fair bit out, I think. And even then it'll only help for queries that run long enough (eventually also often enough, if we get prepared statement JITing) to be worth JITing. Greetings, Andres Freund

Re: Continuing instability in insert-conflict-specconflict test

2020-08-24 Thread Andres Freund
am not sure how we better can achieve test coverage for this stuff. And there've been bugs, so it's worth testing. Greetings, Andres Freund

Re: Continuing instability in insert-conflict-specconflict test

2020-08-24 Thread Andres Freund
On 2020-08-24 13:42:35 -0700, Andres Freund wrote: > Hi, > > On 2020-08-23 22:53:18 -0400, Tom Lane wrote: > > We've seen repeated failures in the tests added by commit 43e084197: > > ... > > I dug into this a bit today, and found that I can reproduce the failur

LWLockAcquire and LockBuffer mode argument

2020-08-24 Thread Andres Freund
ufferDescriptorGetContentLock() available in bufmgr.h I think that's not worth it. So I think we'd be best off having LockBufferExclusive() be a static inline wrapper doing the BufferIsLocal() check and then calling LockBufferExclusiveImpl which'd do the LWLockAcquireExclusive(). Thoughts? Greetings, Andres Freund

Re: Avoid unnecessary ReplicationSlotControl lwlock acquistion

2020-08-24 Thread Andres Freund
Hi, On August 24, 2020 7:38:39 PM PDT, Masahiko Sawada wrote: >Hi all, > >While testing with DTrace, I realized we acquire >ReplicationSlotControl lwlock at some places even when >max_replication_slots is set to 0. For instance, we call >ReplicationSlotCleanup() within PostgresMian() when an er

Re: Avoid unnecessary ReplicationSlotControl lwlock acquistion

2020-08-24 Thread Andres Freund
Hi, On 2020-08-25 12:00:47 +0900, Masahiko Sawada wrote: > On Tue, 25 Aug 2020 at 11:42, Andres Freund wrote: > > > > Hi, > > > > On August 24, 2020 7:38:39 PM PDT, Masahiko Sawada > > wrote: > > >Hi all, > > > > > >While testing wi

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-08-25 Thread Andres Freund
Hi, On 2020-08-25 20:48:37 +1200, David Rowley wrote: > On Tue, 25 Aug 2020 at 08:26, Andres Freund wrote: > > While I'm against introducing a separate node for the caching, I'm *not* > > against displaying a different node type when caching is > > present. E.g. it

Re: LWLockAcquire and LockBuffer mode argument

2020-08-25 Thread Andres Freund
Hi, On 2020-08-25 13:59:35 -0400, Robert Haas wrote: > On Mon, Aug 24, 2020 at 6:35 PM Andres Freund wrote: > > Thoughts? > > This is likely to cause a certain amount of annoyance to many > PostgreSQL developers, but if you have evidence that it will improve > performance s

Re: LWLockAcquire and LockBuffer mode argument

2020-08-25 Thread Andres Freund
Hi, On 2020-08-25 14:22:28 -0400, Robert Haas wrote: > On Tue, Aug 25, 2020 at 2:17 PM Andres Freund wrote: > > It seems easy enough to slap a compiler "enforced" deprecation warning > > on the new compat version, in master only. Seems unnecessary to make > &

Re: Handing off SLRU fsyncs to the checkpointer

2020-08-25 Thread Andres Freund
. Another approach could be to not perform any sorting during recovery, instead including enough information in the WAL record to avoid doing a full blown PageRepairFragmentation during recovery. Greetings, Andres Freund

Re: [EXTERNAL] Re: WIP: WAL prefetch (another approach)

2020-08-27 Thread Andres Freund
Hi, On August 27, 2020 11:26:42 AM PDT, Stephen Frost wrote: >Is WAL FPW compression enabled..? I'm trying to figure out how, given >what's been shared here, that replaying 25GB of WAL is being helped out >by 2.5x thanks to prefetch in the SSD case. That prefetch is hurting >in >the HDD case e

Re: Autovac cancellation is broken in v14

2020-08-27 Thread Andres Freund
Hi, On 2020-08-27 16:20:30 -0400, Jeff Janes wrote: > On Thu, Aug 27, 2020 at 3:10 PM Jeff Janes wrote: > > > If I create a large table with "CREATE TABLE ... AS SELECT ... from > > generate_series(1,3e7)" with no explicit transactions, then once it is done > > I wait for autovac to kick in, the

Re: Allow ERROR from heap_prepare_freeze_tuple to be downgraded to WARNING

2020-08-28 Thread Andres Freund
Hi, On 2020-08-28 12:37:17 -0400, Robert Haas wrote: > On Mon, Jul 20, 2020 at 4:30 PM Andres Freund wrote: > > If we really were to do something like this the option would need to be > > called vacuum_allow_making_corruption_worse or such. Its need to be > > *exceedingl

Re: LogwrtResult contended spinlock

2020-08-31 Thread Andres Freund
Hi, On August 31, 2020 11:21:56 AM PDT, Alvaro Herrera wrote: >Jaime Casanova recently reported a situation where pglogical >replicating >from 64 POS sites to a single central (64-core) node, each with two >replication sets, causes XLog's info_lck to become highly contended >because of frequent

Re: LogwrtResult contended spinlock

2020-08-31 Thread Andres Freund
Hi, On August 31, 2020 11:34:45 AM PDT, Alvaro Herrera wrote: >On 2020-Aug-31, Andres Freund wrote: > >> Hi, >> >> On August 31, 2020 11:21:56 AM PDT, Alvaro Herrera > wrote: > >> >At first I wanted to make the new LWLock cover only LogwrtResult >

Re: Clang UndefinedBehaviorSanitize (Postgres14) Detected undefined-behavior

2020-08-31 Thread Andres Freund
Hi, On August 31, 2020 11:08:49 AM PDT, Ranier Vilela wrote: >Em seg., 31 de ago. de 2020 às 14:43, Ranier Vilela > >escreveu: > >> Em seg., 31 de ago. de 2020 às 14:00, Alvaro Herrera < >> alvhe...@2ndquadrant.com> escreveu: >> >>> On 2020-Aug-31, Ranier Vilela wrote: >>> >>> > More troubles wi

Re: Clang UndefinedBehaviorSanitize (Postgres14) Detected undefined-behavior

2020-08-31 Thread Andres Freund
Hi, On 2020-08-31 12:38:51 -0700, Peter Geoghegan wrote: > On Mon, Aug 31, 2020 at 11:42 AM Andres Freund wrote: > > Unsigned integer overflow is well defined in the standard. So I don't > > understand what this is purporting to warn about. > > Presumably it'

Re: Clang UndefinedBehaviorSanitize (Postgres14) Detected undefined-behavior

2020-08-31 Thread Andres Freund
Hi, On 2020-08-31 17:35:14 -0300, Ranier Vilela wrote: > Em seg., 31 de ago. de 2020 às 17:05, Andres Freund > escreveu: > > So it seems Rainier needs to turn this test off, because it actually is > > intentional. > > > No problem. > If intentional, the code at

Re: Get memory contexts of an arbitrary backend process

2020-08-31 Thread Andres Freund
]; > + chardumpfile[MAXPGPATH]; > + > + snprintf(tmpfile, sizeof(tmpfile), "pg_memusage/%d.tmp", MyProcPid); > + snprintf(dumpfile, sizeof(dumpfile), "pg_memusage/%d", MyProcPid); > + > + /* > + * Open a temp file to dump the current memory context. > + */ > + fpout = AllocateFile(tmpfile, PG_BINARY_W); > + if (fpout == NULL) > + { > + ereport(LOG, > + (errcode_for_file_access(), > + errmsg("could not write temporary memory > context file \"%s\": %m", > + tmpfile))); > + return; > + } Probably should be opened with O_CREAT | O_TRUNC? Greetings, Andres Freund

Re: SyncRepLock acquired exclusively in default configuration

2020-09-01 Thread Andres Freund
On 2020-09-02 10:58:58 +0900, Fujii Masao wrote: > Asim and Sawada-san, thanks for the review! I pushed the patch. Thanks for all your combined work!

builtin functions, parameter names and psql's \df

2020-09-01 Thread Andres Freund
missing something? Greetings, Andres Freund [1] https://www.postgresql.org/docs/current/functions-string.html#FUNCTIONS-STRING-OTHER [2] https://www.postgresql.org/docs/current/functions-matching.html

Re: LogwrtResult contended spinlock

2020-09-04 Thread Andres Freund
me. Also, > are there better ways to go about this? Hm, I was thinking that we'd first go for reading it without a spinlock, but continuing to write it as we currently do. But yea, I can't see an issue with what you propose here. I personally find do {} while () weird and avoid it when not explicitly useful, but that's extremely minor, obviously. Greetings, Andres Freund

Re: LogwrtResult contended spinlock

2020-09-04 Thread Andres Freund
Hi, On 2020-09-04 10:05:45 -0700, Andres Freund wrote: > On 2020-09-03 14:34:52 -0400, Alvaro Herrera wrote: > > Looking at patterns like this > > > > if (XLogCtl->LogwrtRqst.Write < EndPos) > > XLogCtl->LogwrtRqst.Write = EndPos; > >

Re: Improving connection scalability: GetSnapshotData()

2020-09-04 Thread Andres Freund
Hi, On 2020-09-03 17:18:29 +0900, Michael Paquier wrote: > On Sun, Aug 16, 2020 at 02:26:57PM -0700, Andres Freund wrote: > > So we get some builfarm results while thinking about this. > > Andres, there is an entry in the CF for this thread: > https://commitfest.postgresql.o

Re: Improving connection scalability: GetSnapshotData()

2020-09-04 Thread Andres Freund
#x27;s hyperv virtualized, with all the "lost" cores dedicated to the hypervisor. But I did reproduce the speedups on my unvirtualized workstation (2x Xeon Gold 5215 CPUs) - the ceiling is lower, obviously. > May be it is because of more complex architecture of my server? Think we'll need profiles to know... Greetings, Andres Freund

Re: Improving connection scalability: GetSnapshotData()

2020-09-04 Thread Andres Freund
On 2020-09-04 11:53:04 -0700, Andres Freund wrote: > There's a seperate benchmark that I found to be quite revealing that's > far less dependent on scheduler behaviour. Run two pgbench instances: > > 1) With a very simply script '\sleep 1s' or such, and many conn

Re: A micro-optimisation for walkdir()

2020-09-04 Thread Andres Freund
s already is there. But here you're doing a separate lookup with GetFileAttributes()? What am I missing? Greetings, Andres Freund

<    3   4   5   6   7   8   9   10   11   12   >