Re: Parallelize correlated subqueries that execute within each worker

2023-01-21 Thread James Coleman
On Wed, Jan 18, 2023 at 9:34 PM James Coleman wrote: > > On Wed, Jan 18, 2023 at 2:09 PM Tomas Vondra > wrote: > > > > Hi, > > > > This patch hasn't been updated since September, and it got broken by > > 4a29eabd1d91c5484426bc5836e0a7143b064f5a which

Re: Parallelize correlated subqueries that execute within each worker

2023-01-18 Thread James Coleman
t; Workers Planned: 2 > -> Unique (cost=18582710.39..18613960.39 rows=1 ...) >-> Sort (cost=18582710.39..18593127.06 ...) > Sort Key: t.unique1, ((SubPlan 1)) > ... > > which probably makes sense,

Re: Commit fest 2022-11

2022-11-14 Thread James Coleman
's something the app should do for us in this situation. Without that though the patch authors are left to wade through unrelated discussion, and, probably more importantly, the patch discussion thread doesn't show the current state (I think bumping there is more likely to prompt activity as well). James Coleman

Re: cirrus-ci cross-build interactions?

2022-09-26 Thread James Coleman
On Mon, Sep 26, 2022 at 10:48 PM Andres Freund wrote: > > Hi, > > On 2022-09-26 22:36:24 -0400, James Coleman wrote: > > I had a build on Cirrus CI fail tonight in what I have to assume was > > either a problem with caching across builds or some such similar > > fl

Re: Parallelize correlated subqueries that execute within each worker

2022-09-26 Thread James Coleman
On Mon, Mar 21, 2022 at 8:48 PM Andres Freund wrote: > > Hi, > > On 2022-01-22 20:25:19 -0500, James Coleman wrote: > > On the other hand this is a dramatically simpler patch series. > > Assuming the approach is sound, it should much easier to maintain than > > the

Re: cirrus-ci cross-build interactions?

2022-09-26 Thread James Coleman
On Mon, Sep 26, 2022 at 10:36 PM James Coleman wrote: > > I had a build on Cirrus CI fail tonight in what I have to assume was > either a problem with caching across builds or some such similar > flakiness. In the Debian task [1] I received this error: > > su postgres -c "

cirrus-ci cross-build interactions?

2022-09-26 Thread James Coleman
pushing the build [2] seems to be fine. I've double-checked there are no differences between the commits on the two builds (git diff shows no output). Is it possible we're missing some kind of necessary build isolation in the Cirrus CI scripting? Thanks, James Coleman 1: https://cirru

Add hint about downloadable logs to CI README

2022-09-26 Thread James Coleman
I was wondering about how to debug failed builds on Cirrus CI, and after poking at the interface I realized we helpfully upload the logs from CI runs for user download. In an effort to save the next person a few minutes I thought the attached minor patch would help. Thanks, James Coleman v1

Re: Allow foreign keys to reference a superset of unique columns

2022-09-26 Thread James Coleman
On Mon, Sep 26, 2022 at 10:04 AM Wolfgang Walther wrote: > > James Coleman: > > As I was reading through the email chain I had this thought: could you > > get the same benefit (or 90% of it anyway) by instead allowing the > > creation of a uniqueness constraint that co

Re: Allow foreign keys to reference a superset of unique columns

2022-09-26 Thread James Coleman
On Mon, Sep 26, 2022 at 9:59 AM Wolfgang Walther wrote: > > James Coleman: > > So the broader point I'm trying to make is that, as I understand it, > > indexes backing foreign key constraints is an implementation detail. > > The SQL standard details the behavi

Re: Consider parallel for lateral subqueries with limit

2022-09-26 Thread James Coleman
On Mon, Sep 26, 2022 at 10:37 AM Robert Haas wrote: > > On Thu, Sep 22, 2022 at 5:19 PM James Coleman wrote: > > > Your sample query gets a plan like this: > > > > > > Nested Loop Left Join (cost=0.00..1700245.00 rows=1 width=8) > > >-> S

Re: Allow foreign keys to reference a superset of unique columns

2022-09-26 Thread James Coleman
On Mon, Sep 26, 2022 at 2:28 AM Wolfgang Walther wrote: > > James Coleman: > > If we have a declared constraint on x,y where x is unique based on an > > index including on x I do not think we should have that fk constraint > > work differently than a constraint on x,y

Re: Allow foreign keys to reference a superset of unique columns

2022-09-25 Thread James Coleman
ang Walther had commented similarly, but it appears that that idea got lost (or at least not responded to). I'd be happy to sign up to review an updated patch if you're interested in continuing this effort. If so, could you register the patch in the CF app (if not there already)? Thanks, James Coleman

Re: Allow foreign keys to reference a superset of unique columns

2022-09-25 Thread James Coleman
On Sun, Sep 25, 2022 at 4:49 AM Wolfgang Walther wrote: > > James Coleman: > > If I'm following properly this sounds like an overengineered EAV > > schema, and neither of those things inspires me to think "this is a > > use case I want to support". > &g

Re: Allow foreign keys to reference a superset of unique columns

2022-09-24 Thread James Coleman
In the example above then an update to bar.x is going to update the rows in foo that match bar.x = foo.a and bar.y = foo.b *using the old values of bar.x and bar.y* to be the new values. You seem to be suggesting that instead it should look for other rows that already match the *new value* of only one of the columns in the constraint. If I'm understanding the example correctly, that seems like a *very* bad idea. James Coleman

Re: Consider parallel for lateral subqueries with limit

2022-09-24 Thread James Coleman
On Thu, Sep 22, 2022 at 5:19 PM James Coleman wrote: > > On Mon, Sep 19, 2022 at 4:29 PM Robert Haas wrote: > > > > On Mon, Sep 19, 2022 at 3:58 PM James Coleman wrote: > > > But in the case where there's correlation via LATERAL we already don't > > &g

Re: Consider parallel for lateral subqueries with limit

2022-09-22 Thread James Coleman
On Mon, Sep 19, 2022 at 4:29 PM Robert Haas wrote: > > On Mon, Sep 19, 2022 at 3:58 PM James Coleman wrote: > > But in the case where there's correlation via LATERAL we already don't > > guarantee unique executions for a given set of params into the lateral > &

Re: Support pg_attribute_aligned and noreturn in MSVC

2022-09-21 Thread James Coleman
On Tue, Sep 20, 2022 at 9:18 PM Michael Paquier wrote: > > On Tue, Sep 20, 2022 at 08:01:20AM -0400, James Coleman wrote: > > I don't have access to a Windows machine for testing, but re-reading > > the documentation it looks like the issue is that our noreturn macr

Re: Auto explain after query timeout

2022-09-20 Thread James Coleman
On Tue, Sep 20, 2022 at 3:06 PM Robert Haas wrote: > > On Tue, Sep 20, 2022 at 2:35 PM James Coleman wrote: > > Either I'm missing something (and/or this was fixed in a later PG > > version), but I don't think this is how it works. We have this >

Re: Auto explain after query timeout

2022-09-20 Thread James Coleman
On Tue, Sep 20, 2022 at 2:12 PM Gurjeet wrote: > > On Tue Sep 20, 2022 at 10:34 AM PDT, James Coleman wrote: > > Hopefully I'm not missing something obvious, but as far as I know > > there's no way to configure auto explain to work fire > > statement_timeout f

Auto explain after query timeout

2022-09-20 Thread James Coleman
impediment to doing so? Thanks, James Coleman

Re: Support pg_attribute_aligned and noreturn in MSVC

2022-09-20 Thread James Coleman
On Mon, Sep 19, 2022 at 11:21 PM Michael Paquier wrote: > > On Mon, Sep 19, 2022 at 08:51:37PM -0400, James Coleman wrote: > > Yes, fixed. > > The CF bot is failing compilation on Windows: > http://commitfest.cputube.org/james-coleman.html > https://api.cirrus-ci.com/v1/ta

Re: Support pg_attribute_aligned and noreturn in MSVC

2022-09-19 Thread James Coleman
On Mon, Sep 19, 2022 at 8:21 PM Michael Paquier wrote: > > On Mon, Sep 19, 2022 at 06:21:58PM -0400, James Coleman wrote: > > It turns out that MSVC supports both noreturn [2] [3] and alignment > > [4] [5] attributes, so this patch adds support for those. MSVC also > > su

Support pg_attribute_aligned and noreturn in MSVC

2022-09-19 Thread James Coleman
I opted not to implement that attribute. James Coleman 1: https://www.postgresql.org/message-id/Yk6UgCGlZKuxRr4n%40paquier.xyz 2: 2008+ https://learn.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2008/k6ktzx3s(v=vs.90) 3. 2015+ https://learn.microsoft.com/en-us/cpp/c-language

Re: Consider parallel for lateral subqueries with limit

2022-09-19 Thread James Coleman
execution, then I believe your suggestion here is orthogonal and would expand the use cases even more. For example, if we were able to guarantee a unique result set (including order), then we could allow parallelizing subqueries even if they're not lateral and correlated. James Coleman

Fix comment in convert_saop_to_hashed_saop

2022-09-14 Thread James Coleman
In 29f45e29 we added support for executing NOT IN(values) with a hashtable, however this comment still claims that we only do so for cases where the ScalarArrayOpExpr's useOr flag is true. See attached for fix. Thanks, James Coleman v1-0001-Fix-convert_saop_to_hashed_saop-comment.

Re: Fix broken link to FreeBSD DocProj in docs

2022-09-13 Thread James Coleman
On Tue, Sep 13, 2022 at 4:43 AM Daniel Gustafsson wrote: > > > On 12 Sep 2022, at 20:46, Daniel Gustafsson wrote: > > > >> On 12 Sep 2022, at 18:13, James Coleman wrote: > > > >> See attached simple patch to fix $SUBJECT; the old link generates

Fix broken link to FreeBSD DocProj in docs

2022-09-12 Thread James Coleman
See attached simple patch to fix $SUBJECT; the old link generates a Not Found. Thanks, James Coleman v1-0001-Fix-FreeBSD-DocProj-link.patch Description: Binary data

Re: [Commitfest 2022-07] Patch Triage: Waiting on Author

2022-07-26 Thread James Coleman
m this CF, but I've been sufficiently busy that that hasn't happened. I'd like to just move these to the next CF. Thanks, James Coleman

Re: pg_rewind: warn when checkpoint hasn't happened after promotion

2022-07-05 Thread James Coleman
On Tue, Jul 5, 2022 at 2:39 PM Robert Haas wrote: > > On Sat, Jun 4, 2022 at 8:59 AM James Coleman wrote: > > A quick background refresher: after promoting a standby rewinding the > > former primary requires that a checkpoint have been completed on the > > new primary

Re: PG 15 (and to a smaller degree 14) regression due to ExprEvalStep size

2022-06-17 Thread James Coleman
here's maybe even some gains due to the smaller step > size. I didn't see that comment when working on this (it's quite a long unioned struct; I concur on adding an assert to catch it). This patch looks very reasonable to me though. James Coleman

Re: pg_rewind: warn when checkpoint hasn't happened after promotion

2022-06-06 Thread James Coleman
On Mon, Jun 6, 2022 at 1:26 AM Kyotaro Horiguchi wrote: > > At Sat, 4 Jun 2022 19:09:41 +0530, Bharath Rupireddy > wrote in > > On Sat, Jun 4, 2022 at 6:29 PM James Coleman wrote: > > > > > > A few weeks back I sent a bug report [1] directly to the -bugs maili

Re: pg_rewind: warn when checkpoint hasn't happened after promotion

2022-06-06 Thread James Coleman
On Sat, Jun 4, 2022 at 9:39 AM Bharath Rupireddy wrote: > > On Sat, Jun 4, 2022 at 6:29 PM James Coleman wrote: > > > > A few weeks back I sent a bug report [1] directly to the -bugs mailing > > list, and I haven't seen any activity on it (maybe this is because I &g

pg_rewind: warn when checkpoint hasn't happened after promotion

2022-06-04 Thread James Coleman
etects this condition and reports it as an error to the user. In the spirit of the new-ish "ensure shutdown" functionality I could imagine extending this to automatically issue a checkpoint when this situation is detected. I haven't started to code that up, however, wanting to first ge

Re: Restructure ALTER TABLE notes to clarify table rewrites and verification scans

2022-04-01 Thread James Coleman
On Thu, Mar 31, 2022 at 10:58 AM Matthias van de Meent wrote: > > On Tue, 29 Mar 2022 at 16:20, James Coleman wrote: > > > > Over in the "Document atthasmissing default optimization avoids > > verification table scan" thread David Johnston (who I've cc&

Re: Correct docs re: rewriting indexes when table rewrite is skipped

2022-04-01 Thread James Coleman
On Fri, Apr 1, 2022 at 8:58 AM Robert Haas wrote: > > On Thu, Mar 31, 2022 at 4:19 PM James Coleman wrote: > > On Thu, Mar 31, 2022 at 3:25 PM Robert Haas wrote: > > > On Thu, Mar 31, 2022 at 10:51 AM James Coleman wrote: > > > > Updated. > > > >

Re: Correct docs re: rewriting indexes when table rewrite is skipped

2022-03-31 Thread James Coleman
On Thu, Mar 31, 2022 at 3:25 PM Robert Haas wrote: > > On Thu, Mar 31, 2022 at 10:51 AM James Coleman wrote: > > Updated. > > This version looks fine to me. If nobody objects I will commit it and > credit myself as a co-author. Sounds great; thanks again for the review. James Coleman

Re: Correct docs re: rewriting indexes when table rewrite is skipped

2022-03-31 Thread James Coleman
On Thu, Mar 31, 2022 at 10:29 AM Robert Haas wrote: > > On Thu, Mar 31, 2022 at 10:14 AM James Coleman wrote: > > Is the attached more along the lines of what you were thinking? > > Yeah. Maybe this would be a little clearer: "For example, if the > collation for a c

Re: Correct docs re: rewriting indexes when table rewrite is skipped

2022-03-31 Thread James Coleman
On Thu, Mar 31, 2022 at 9:43 AM Robert Haas wrote: > > On Thu, Mar 31, 2022 at 9:17 AM James Coleman wrote: > > All right, thanks for feedback. Attached is v2 with such a change. > > I've not included examples, and I'm about 50/50 on doing so. What are > > your

Re: Correct docs re: rewriting indexes when table rewrite is skipped

2022-03-31 Thread James Coleman
On Wed, Mar 30, 2022 at 5:41 PM Robert Haas wrote: > > On Wed, Mar 30, 2022 at 4:33 PM James Coleman wrote: > > Hmm, having it match the way it works makes sense. Would you feel > > comfortable with an intermediate step (queueing up that as a larger > > change) changing

Re: Correct docs re: rewriting indexes when table rewrite is skipped

2022-03-30 Thread James Coleman
On Wed, Mar 30, 2022 at 11:41 AM Robert Haas wrote: > > On Wed, Mar 30, 2022 at 10:04 AM James Coleman wrote: > > Admittedly I hadn't thought of that case. But isn't it already covered > > in the existing docs by the phrase "or an unconstrained domain over &g

Re: Correct docs re: rewriting indexes when table rewrite is skipped

2022-03-30 Thread James Coleman
On Tue, Mar 29, 2022 at 11:29 AM Matthias van de Meent wrote: > > On Tue, 29 Mar 2022 at 16:04, James Coleman wrote: > > > > Back in 367bc42 (for 9.2!) we "avoid[ed] index rebuild[ing] for > > no-rewrite ALTER TABLE > > .. ALTER TYPE." However the d

Restructure ALTER TABLE notes to clarify table rewrites and verification scans

2022-03-29 Thread James Coleman
ted the slight correction in "Correct docs re: rewriting indexes when table rewrite is skipped" [2] here, and will rebase this patch if that gets committed. Thanks, James Coleman 1: https://www.postgresql.org/message-id/CAKFQuwZyBaJjNepdTM3kO8PLaCpRdRd8%2BmtLT8QdE73oAsGv8Q%40mail.gmail.com 2

Correct docs re: rewriting indexes when table rewrite is skipped

2022-03-29 Thread James Coleman
ate the docs to match the current behavior. Thanks, James Coleman From f6515a5f5f39d728b4cad837480c3ca953ed4623 Mon Sep 17 00:00:00 2001 From: jcoleman Date: Tue, 29 Mar 2022 13:56:39 + Subject: [PATCH v1] Docs: When table rewriting is skipped indexes are not rebuilt In 367bc42 (for 9.2!) we a

Re: Document atthasmissing default optimization avoids verification table scan

2022-03-29 Thread James Coleman
On Sun, Mar 27, 2022 at 11:12 PM David G. Johnston wrote: > > On Sun, Mar 27, 2022 at 11:17 AM James Coleman wrote: >> >> Hmm, I didn't realize that was project policy, > > > Guideline/Rule of Thumb is probably a better concept. Ah, OK, thanks. >> >&g

Re: Document atthasmissing default optimization avoids verification table scan

2022-03-28 Thread James Coleman
On Mon, Mar 28, 2022 at 9:30 AM Robert Haas wrote: > > On Sun, Mar 27, 2022 at 1:00 PM James Coleman wrote: > > So "undocumented concept" is just not accurate, and so I don't see it > > as a valid reason to reject the patch. > > I mean, I think it's

Re: Document atthasmissing default optimization avoids verification table scan

2022-03-27 Thread James Coleman
On Sun, Mar 27, 2022 at 1:46 PM David G. Johnston wrote: > > On Sun, Mar 27, 2022 at 10:00 AM James Coleman wrote: >> >> As shown above, table scans (and specifically table scans used to >> validate constraints, which is what this patch is about) are clearly >> d

Re: Document atthasmissing default optimization avoids verification table scan

2022-03-27 Thread James Coleman
On Sun, Mar 27, 2022 at 11:43 AM Robert Haas wrote: > > On Sat, Mar 26, 2022 at 6:25 PM James Coleman wrote: > > I simply do not accept the claim that this is not a reasonable concern > > to have nor that this isn't worth documenting. > > I don't think I said

Re: Document atthasmissing default optimization avoids verification table scan

2022-03-26 Thread James Coleman
already is] pretty fragmented" is there a place for a simpler improvement (adding a short explanation of this particular hazard) in the meantime? I don't mean this specific v4 patch -- just in general (since the patch can be revised of course). Thanks, James Coleman

Re: Document atthasmissing default optimization avoids verification table scan

2022-03-26 Thread James Coleman
On Fri, Mar 25, 2022 at 4:40 PM Robert Haas wrote: > > On Tue, Jan 25, 2022 at 8:49 AM James Coleman wrote: > > Here's a version that looks like that. I'm not convinced it's an > > improvement over the previous version: again, I expect more advanced > > u

Re: Synchronizing slots from primary to standby

2022-02-23 Thread James Coleman
ccur without any attempt to trigger the catalog xmin problem. Given this error seems pretty significant in terms of indicating fundamental lack of test coverage (the primary stated benefit of the patch is physical failover), and it currently is a blocker to testing more deeply. Thanks, Jame

Re: Synchronizing slots from primary to standby

2022-02-18 Thread James Coleman
quot;localy" -> "locally"). This patch would be a significant improvement for us; I'm hoping we can see some activity on it. I'm also hoping to try to do some testing next week and see if I can poke any holes in the functionality (with the goal of verifying Andres's concerns about the safety without the minimal logical decoding on a replica patch). Thanks, James Coleman

Re: Synchronizing slots from primary to standby

2022-02-18 Thread James Coleman
review this patch, and last time I checked I noticed it didn't seem to apply cleanly to master anymore. Would you be able to send a rebased version? Thanks, James Coleman

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-29 Thread James Coleman
On Fri, Jan 28, 2022 at 7:47 PM Andres Freund wrote: > > On 2022-01-28 16:36:32 -0800, Andres Freund wrote: > > On 2022-01-28 18:43:57 -0500, James Coleman wrote: > > > Alternatively I see pg_attribute_aligned, but that's not defined > > > (AFAICT) on clang, for

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-28 Thread James Coleman
On Thu, Jan 20, 2022 at 8:15 AM James Coleman wrote: > > On Wed, Jan 19, 2022 at 10:12 PM Andres Freund wrote: > > > > Hi, > > > > On 2022-01-19 21:23:12 -0500, James Coleman wrote: > > > { oid => '3537', descr => 'get identification o

Re: Document atthasmissing default optimization avoids verification table scan

2022-01-25 Thread James Coleman
On Sat, Jan 22, 2022 at 10:28 AM David G. Johnston wrote: > > > > On Saturday, January 22, 2022, James Coleman wrote: >> >> On Sat, Jan 22, 2022 at 12:35 AM David G. Johnston >> wrote: >> > >> > On Fri, Jan 21, 2022 at 5:14 PM James Coleman w

Re: Parallelize correlated subqueries that execute within each worker

2022-01-22 Thread James Coleman
On Fri, Jan 21, 2022 at 3:20 PM Robert Haas wrote: > > On Fri, Jan 14, 2022 at 2:25 PM James Coleman wrote: > > I've been chewing on this a bit, and I was about to go re-read the > > code and see how easy it'd be to do exactly what you're suggesting in > >

Re: Document atthasmissing default optimization avoids verification table scan

2022-01-22 Thread James Coleman
On Sat, Jan 22, 2022 at 12:35 AM David G. Johnston wrote: > > On Fri, Jan 21, 2022 at 5:14 PM James Coleman wrote: >> >> >> > Really? That's horrid, because that's directly useful advice. >> >> Remedied, but rewritten a bit to better fit with th

Re: Document atthasmissing default optimization avoids verification table scan

2022-01-21 Thread James Coleman
d then add any desired default as > > described > > - below. > > > has now been completely removed from the documentation. > > Really? That's horrid, because that's directly useful advice. Remedied, but rewritten a bit to better fit with the new style/goal of

Re: Document atthasmissing default optimization avoids verification table scan

2022-01-21 Thread James Coleman
On Fri, Jan 21, 2022 at 4:08 PM Andrew Dunstan wrote: > > > On 1/21/22 13:55, James Coleman wrote: > > On Thu, Jan 20, 2022 at 3:43 PM James Coleman wrote: > >> As noted earlier I expect to be posting an updated patch soon. > > Here's the updated series. I

Re: Document atthasmissing default optimization avoids verification table scan

2022-01-21 Thread James Coleman
On Thu, Jan 20, 2022 at 3:43 PM James Coleman wrote: > > As noted earlier I expect to be posting an updated patch soon. Here's the updated series. In 0001 I've moved the documentation tweak into the ALTER TABLE notes section. In 0002 I've taken David J's suggestion of s

Re: Document atthasmissing default optimization avoids verification table scan

2022-01-20 Thread James Coleman
On Thu, Jan 20, 2022 at 3:31 PM Andrew Dunstan wrote: > > > On 1/20/22 12:25, Bossart, Nathan wrote: > > On 1/19/22, 5:15 PM, "James Coleman" wrote: > >> I'm open to the idea of wordsmithing here, of course, but I strongly > >> disagree t

Re: Document atthasmissing default optimization avoids verification table scan

2022-01-20 Thread James Coleman
On Wed, Jan 19, 2022 at 9:34 PM David G. Johnston wrote: > > On Wed, Jan 19, 2022 at 6:14 PM James Coleman wrote: >> >> I'm open to the idea of wordsmithing here, of course, but I strongly >> disagree that this is irrelevant data. > > > Ok, but wording aside

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-20 Thread James Coleman
On Wed, Jan 19, 2022 at 10:12 PM Andres Freund wrote: > > Hi, > > On 2022-01-19 21:23:12 -0500, James Coleman wrote: > > { oid => '3537', descr => 'get identification of SQL object', > > diff --git a/src/include/storage/proc.h b/src/include/stor

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-19 Thread James Coleman
SN */ > > }; > > We do not rely on 64bit integers to be read/written atomically, just 32bit > ones. To make this work for older platforms you'd have to use a > pg_atomic_uint64. On new-ish platforms pg_atomic_read_u64/pg_atomic_write_u64 > end up as plain read/writes, but o

Re: Document atthasmissing default optimization avoids verification table scan

2022-01-19 Thread James Coleman
On Wed, Jan 19, 2022 at 7:51 PM David G. Johnston wrote: > > On Wed, Jan 19, 2022 at 5:08 PM Bossart, Nathan wrote: >> >> On 9/24/21, 7:30 AM, "James Coleman" wrote: >> > When PG11 added the ability for ALTER TABLE ADD COLUMN to set a constant >> >

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
_xact() currently finds its home there. I think we need a shared ProcArrayLock to read the array, correct? We also need to do the global updating under lock, but given it's when a proc is removed, that shouldn't be a performance issue if I'm following what you are saying. Thanks, Jame

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
On Tue, Jan 18, 2022 at 8:05 PM Andres Freund wrote: > > Hi, > > On 2022-01-18 18:31:42 -0500, James Coleman wrote: > > One other question on this: if we went with this would you expect a > > new function to parallel pg_last_committed_xact()? > > I don't think I

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
be present if track_commit_timestamps isn't on)? Or would you expect the current xid and timestamp use the new infrastructure also? Thanks, James Coleman

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
e that. We'd also > need to maintain a value for all disconnected backends, but that's also not a > hot > path. I expect most monitoring setups default to around something like checking anywhere from every single digit seconds to minutes. If I read between the lines I imagine you'd see even e.g. every 2s as not that big of a deal here, right? Thanks, James Coleman

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
On Tue, Jan 18, 2022 at 1:52 PM Alvaro Herrera wrote: > > On 2022-Jan-18, James Coleman wrote: > > > Reading the code it seems the only usage (besides > > the boolean activation status also stored there) is in > > TransactionIdGetCommitTsData, and the only consumers of

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
On Tue, Jan 18, 2022 at 12:50 PM Alvaro Herrera wrote: > > On 2022-Jan-17, James Coleman wrote: > > > I'd be happy to make it a separate GUC, though it seems adding an > > additional atomic access is worse (assuming we can convince ourselves > > putting

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-18 Thread James Coleman
On Tue, Jan 18, 2022 at 9:25 AM Robert Haas wrote: > > On Mon, Jan 17, 2022 at 8:39 PM James Coleman wrote: > > I wondered about that, but commit_ts already does more than commit > > timestamps by recording the xid of the last commit. > > Well, if you're maintaining

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-17 Thread James Coleman
On Mon, Jan 17, 2022 at 4:34 PM Alvaro Herrera wrote: > > On 2022-Jan-14, James Coleman wrote: > > > The logical slot can't flush past the > > last commit, so even if there's 100s of megabytes of unflushed WAL on > > the slot there may be zero lag (in te

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-17 Thread James Coleman
On Mon, Jan 17, 2022 at 4:20 PM Robert Haas wrote: > > On Fri, Jan 14, 2022 at 7:42 PM James Coleman wrote: > > I've attached a simple patch (sans tests and documentation) to get > > feedback early. After poking around this afternoon it seemed to me > > that the

Add last commit LSN to pg_last_committed_xact()

2022-01-14 Thread James Coleman
on't write it out to disk). That the downside of making this feature dependent on "track_commit_timestamps = on", but that seems reasonable: 1. Getting the xid of the last commit is similarly dependent on commit timestamps infrastructure. 2. It's a simple place to hook into

Re: Parallelize correlated subqueries that execute within each worker

2022-01-14 Thread James Coleman
On Mon, Nov 15, 2021 at 10:01 AM Robert Haas wrote: > > On Wed, Nov 3, 2021 at 1:34 PM James Coleman wrote: > > As I understand the current code, parallel plans are largely chosen > > based not on where it's safe to insert a Gather node but rather by > > determinin

Re: Parallelize correlated subqueries that execute within each worker

2022-01-14 Thread James Coleman
On Fri, Dec 3, 2021 at 2:35 AM Michael Paquier wrote: > > On Mon, Nov 15, 2021 at 10:01:37AM -0500, Robert Haas wrote: > > On Wed, Nov 3, 2021 at 1:34 PM James Coleman wrote: > >> As I understand the current code, parallel plans are largely chosen > >> based not

Re: Consider parallel for lateral subqueries with limit

2022-01-13 Thread James Coleman
On Tue, Jan 4, 2022 at 9:59 PM James Coleman wrote: > > On Tue, Jan 4, 2022 at 5:31 PM Tom Lane wrote: > > > > Greg Nancarrow writes: > > > The patch LGTM. > > > I have set the status to "Ready for Committer". > > > > I don't really

Re: Consider parallel for lateral subqueries with limit

2022-01-04 Thread James Coleman
when it's a lateral reference. That limit/scan (if lateral) has to be being executed per tuple in the outer scan, right? And if it's a unique execution per tuple, then consistency across tuples (that are in different workers) can't be a concern. Is there a scenario I'm missing where lateral can currently be executed in that way in that structure (or a different one)? Thanks, James Coleman

Re: Parallelize correlated subqueries that execute within each worker

2021-11-03 Thread James Coleman
er than anyone. > > On Fri, May 7, 2021 at 12:30 PM James Coleman wrote: > > The basic idea is that we need to track (both on nodes and relations) > > not only whether that node or rel is parallel safe but also whether > > it's parallel safe assuming params are rechecke

Re: Consider parallel for lateral subqueries with limit

2021-11-03 Thread James Coleman
On Fri, Jul 16, 2021 at 3:16 PM James Coleman wrote: > > On Thu, May 27, 2021 at 9:01 PM Greg Nancarrow wrote: > > > > On Tue, Dec 8, 2020 at 10:46 AM James Coleman wrote: > > > > > > While I haven't actually tracked down to guarantee this is handled &

Re: Parallelize correlated subqueries that execute within each worker

2021-11-03 Thread James Coleman
On Wed, Sep 8, 2021 at 8:47 AM James Coleman wrote: > See updated patch series attached. Jaime, I noticed on 3-October you moved this into "waiting on author"; I don't see anything waiting in this thread, however. Am I missing something? I'm planning to change i

Re: Document spaces in .pgpass need to be escaped

2021-09-30 Thread James Coleman
On Wed, Sep 29, 2021 at 12:13 PM Tom Lane wrote: > > James Coleman writes: > > A coworker has a space in a Postgres password and noticed .pgpass > > didn't work; escaping it fixed the issue. That requirement wasn't > > documented (despite other escaping

Document spaces in .pgpass need to be escaped

2021-09-28 Thread James Coleman
A coworker has a space in a Postgres password and noticed .pgpass didn't work; escaping it fixed the issue. That requirement wasn't documented (despite other escaping requirements being documented), so I've attached a patch to add that comment. Thanks, James Coleman v1-0001-Doc

Document atthasmissing default optimization avoids verification table scan

2021-09-24 Thread James Coleman
null. That scan happens under an exclusive lock on the table, so it can have a meaningful impact on database "accessible uptime". I've attached a patch to document that the new mechanism also precludes that scan. Thanks, James Coleman v1-0001-Document-atthasmissing

Re: Parallelize correlated subqueries that execute within each worker

2021-09-08 Thread James Coleman
On Tue, Sep 7, 2021 at 11:06 AM Zhihong Yu wrote: > > > > On Tue, Sep 7, 2021 at 6:17 AM James Coleman wrote: >> >> On Wed, Sep 1, 2021 at 7:06 AM Daniel Gustafsson wrote: >> > >> > > On 7 May 2021, at 18:30, James Coleman wrote: >> > &

Re: Parallelize correlated subqueries that execute within each worker

2021-09-07 Thread James Coleman
On Wed, Sep 1, 2021 at 7:06 AM Daniel Gustafsson wrote: > > > On 7 May 2021, at 18:30, James Coleman wrote: > > > ..here we are now, and I finally have this patch cleaned up > > enough to share. > > This patch no longer applies to HEAD, can you please submit a

Re: [PATCH] Use optimized single-datum tuplesort in ExecSort

2021-07-20 Thread James Coleman
On Tue, Jul 20, 2021 at 4:35 AM David Rowley wrote: > > On Tue, 20 Jul 2021 at 01:10, James Coleman wrote: > > To be clear up front: I'm in favor of the patch, and I don't want to > > put unnecessary stumbling blocks up for it getting committed. So if we > > de

Re: [PATCH] Use optimized single-datum tuplesort in ExecSort

2021-07-19 Thread James Coleman
On Sat, Jul 17, 2021 at 4:36 AM David Rowley wrote: > > On Sat, 17 Jul 2021 at 01:14, James Coleman wrote: > > The only remaining question I have is whether or not costing needs to > > change, given the very significant speedup for datum sort. > > I'm looking a

Re: Consider parallel for lateral subqueries with limit

2021-07-16 Thread James Coleman
On Thu, May 27, 2021 at 9:01 PM Greg Nancarrow wrote: > > On Tue, Dec 8, 2020 at 10:46 AM James Coleman wrote: > > > > While I haven't actually tracked down to guarantee this is handled > > elsewhere, a thought experiment -- I think -- shows it must be so. > >

Re: Early Sort/Group resjunk column elimination.

2021-07-16 Thread James Coleman
d the other examples you gave compelling also. Of course I haven't seen code yet, but my first intuition is to try to avoid adding extra nodes and teach the (hopefully few) relevant nodes to remove the resjunk entries themselves. Presumably in this case that would mostly be the sort nodes (including gather merge). One thing to pay attention to here is that we can't necessarily remove resjunk entries every time in a sort node since, for example, in parallel mode the gather merge node above it will need those entries to complete the sort. I'm interested to see what you're working on with a patch. Thanks, James Coleman

Re: [PATCH] Use optimized single-datum tuplesort in ExecSort

2021-07-16 Thread James Coleman
On Thu, Jul 15, 2021 at 11:45 PM David Rowley wrote: > > On Fri, 16 Jul 2021 at 02:53, Ronan Dunklau wrote: > > Please find attached a v9 just moving the flag setting to ExecInitSort, and > > my > > apologies if I misunderstood your point. > > I took this and adjusted a few things and ended up w

Re: [PATCH] Use optimized single-datum tuplesort in ExecSort

2021-07-15 Thread James Coleman
On Thu, Jul 15, 2021 at 10:19 AM David Rowley wrote: > > On Fri, 16 Jul 2021 at 01:44, James Coleman wrote: > > > > On Wed, Jul 14, 2021 at 9:22 PM David Rowley wrote: > > > > > > On Thu, 15 Jul 2021 at 12:30, Ranier Vilela wrote: > > > > >

Re: [PATCH] Use optimized single-datum tuplesort in ExecSort

2021-07-15 Thread James Coleman
On Wed, Jul 14, 2021 at 9:22 PM David Rowley wrote: > > On Thu, 15 Jul 2021 at 12:30, Ranier Vilela wrote: > > > > Em qua., 14 de jul. de 2021 às 21:21, David Rowley > > escreveu: > >> But, in v8 there is no additional branch, so no branch to mispredict. > >> I don't really see how your explana

Re: [PATCH] Use optimized single-datum tuplesort in ExecSort

2021-07-06 Thread James Coleman
On Tue, Jul 6, 2021 at 11:03 AM Ronan Dunklau wrote: > > Thank you for the review, I will address those shortly, but will answer some > questions in the meantime. > > > > First, the changes are lacking any explanatory comments. Probably we > > > should follow how nodeAgg does this and add both com

Re: [PATCH] Use optimized single-datum tuplesort in ExecSort

2021-07-06 Thread James Coleman
Adding David since this patch is likely a precondition for [1]. On Tue, Jul 6, 2021 at 2:15 AM Ronan Dunklau wrote: > > Hello, > > While testing the patch "Add proper planner support for ORDER BY / DISTINCT > aggregates" [0] I discovered the performance penalty from adding a sort node > essential

Re: Add proper planner support for ORDER BY / DISTINCT aggregates

2021-07-05 Thread James Coleman
On Mon, Jul 5, 2021 at 8:08 AM Ronan Dunklau wrote: > > > Ok, I reproduced that case, just not using a group by: by adding the group > > by a sort node is added in both cases (master and your patch), except that > > with your patch we sort on both keys and that doesn't really incur a > > performan

Re: Add proper planner support for ORDER BY / DISTINCT aggregates

2021-07-05 Thread James Coleman
On Sat, Jun 12, 2021 at 11:07 AM David Rowley wrote: > > A few years ago I wrote a patch to implement the missing aggregate > combine functions for array_agg and string_agg [1]. In the end, the > patch was rejected due to some concern [2] that if we allow these > aggregates to run in parallel the

Minor typo in generate_useful_gather_paths comment

2021-07-05 Thread James Coleman
While re-reading this code I found a small typo and fixed it (making the comment more explicit at the same time). Thanks, James v1-0001-Fix-typo-in-comment.patch Description: Binary data

<    1   2   3   4   5   6   >