Re: Teach predtest about IS [NOT] proofs

2024-04-05 Thread James Coleman
On Mon, Apr 1, 2024 at 8:06 AM James Coleman wrote: > > On Mon, Mar 25, 2024 at 5:53 PM Tom Lane wrote: > > > > James Coleman writes: > > > [ v6 patchset ] > > > > I went ahead and committed 0001 after one more round of review > > >

Re: Teach predtest about IS [NOT] proofs

2024-04-01 Thread James Coleman
On Mon, Mar 25, 2024 at 5:53 PM Tom Lane wrote: > > James Coleman writes: > > [ v6 patchset ] > > I went ahead and committed 0001 after one more round of review > > statements; my bad). I also added the changes in test_predtest.c from > 0002. I attach a rebased versi

Re: Teach predtest about IS [NOT] proofs

2024-04-01 Thread James Coleman
ant, but perhaps future readers of the archives will > be confused. I was wondering myself :) so thanks for clarifying. Regards, James Coleman

Re: [DOCS] HOT - correct claim about indexes not referencing old line pointers

2024-03-23 Thread James Coleman
On Thu, Mar 21, 2024 at 1:09 PM Robert Haas wrote: > > On Thu, Mar 14, 2024 at 9:07 PM James Coleman wrote: > > If the goal here is the most minimal patch possible, then please > > commit what you proposed. I am interested in improving the document > > further, bu

Re: [DOCS] HOT - correct claim about indexes not referencing old line pointers

2024-03-23 Thread James Coleman
On Wed, Mar 20, 2024 at 2:15 PM Robert Haas wrote: > > On Thu, Mar 14, 2024 at 9:07 PM James Coleman wrote: > > Obviously I have reasons for the other changes I made: for example, > > "no longer visible" improves the correctness, since being an old > > v

Re: [DOCS] HOT - correct claim about indexes not referencing old line pointers

2024-03-14 Thread James Coleman
On Thu, Mar 14, 2024 at 10:28 AM Robert Haas wrote: > > On Wed, Oct 4, 2023 at 9:12 PM James Coleman wrote: > > All right, attached is a v3 which attempts to fix the wrong > > information with an economy of words. I may at some point submit a > > separate patch tha

Re: RFC: Logging plan of the running query

2024-03-02 Thread James Coleman
> If this is correctly implemented, the overhead in the case where the > feature isn't used should be essentially zero, I believe. If I can rephrase this idea: it's basically "delay this interrupt until inline to the next ExecProcNode execution". That seems pretty promising to me as well. Regards, James Coleman

Re: RFC: Logging plan of the running query

2024-02-24 Thread James Coleman
On Fri, Feb 23, 2024 at 10:23 AM Robert Haas wrote: > > On Fri, Feb 23, 2024 at 7:50 PM Julien Rouhaud wrote: > > On Fri, Feb 23, 2024 at 10:22:32AM +0530, Robert Haas wrote: > > > On Thu, Feb 22, 2024 at 6:25 AM James Coleman wrote: > > > > This is potent

Re: RFC: Logging plan of the running query

2024-02-21 Thread James Coleman
able as we want > PostgreSQL to be. This is potentially a bit of a wild idea, but I wonder if having some kind of argument to CHECK_FOR_INTERRUPTS() signifying we're in "normal" as opposed to "critical" (using that word differently than the existing critical sections) would be worth it. Regards, James Coleman

Re: Question about behavior of deletes with REPLICA IDENTITY NOTHING

2024-02-08 Thread James Coleman
NG > > Records no information about the old row. This is equivalent to having > no replica identity. This is the default for system tables. This is the simplest change, and it does solve the confusion, so I'd be happy with it also. The other proposals have the benefit of having all the information necessary on the publications page rather than requiring the user to refer to the ALTER TABLE REPLICA IDENTITY page to understand what's meant. Regards, James Coleman

Re: Question about behavior of deletes with REPLICA IDENTITY NOTHING

2024-02-08 Thread James Coleman
index that doesn't exist) is added ... I think that would work also. I was reading the initial suggestion as "(or with replica identity behavior the same as..." as defining what "without a replica identity" meant, which would avoid the confusion. But your proposal is more explicit and more succinct, so I think it's the better option of the two. Regards, James Coleman

Re: Question about behavior of deletes with REPLICA IDENTITY NOTHING

2024-02-07 Thread James Coleman
On Wed, Feb 7, 2024 at 6:04 PM Peter Smith wrote: > > On Thu, Feb 8, 2024 at 9:04 AM James Coleman wrote: > > > > On Wed, Feb 7, 2024 at 3:22 PM Laurenz Albe > > wrote: > > > > > > On Wed, 2024-02-07 at 15:12 -0500, James Coleman wrote: > >

Re: Question about behavior of deletes with REPLICA IDENTITY NOTHING

2024-02-07 Thread James Coleman
On Wed, Feb 7, 2024 at 3:22 PM Laurenz Albe wrote: > > On Wed, 2024-02-07 at 15:12 -0500, James Coleman wrote: > > We recently noticed some behavior that seems reasonable but also > > surprised our engineers based on the docs. > > > > If we have this setup

Question about behavior of deletes with REPLICA IDENTITY NOTHING

2024-02-07 Thread James Coleman
table. I'm wondering if this might be a surprise to anyone else, and if so, is there a minor docs tweak that might avoid the confusion? Thanks, James Coleman 1: https://www.postgresql.org/docs/current/sql-altertable.html#SQL-ALTERTABLE-REPLICA-IDENTITY 2: https://www.postgresql.org/docs/current/logical-replication-publication.html

Re: set_cheapest without checking pathlist

2024-02-01 Thread James Coleman
On Thu, Feb 1, 2024 at 1:36 AM Richard Guo wrote: > > > On Thu, Feb 1, 2024 at 11:37 AM David Rowley wrote: >> >> On Thu, 1 Feb 2024 at 16:29, Richard Guo wrote: >> > On Thu, Feb 1, 2024 at 10:04 AM James Coleman wrote: >> >> I don't see any

Re: Parallelize correlated subqueries that execute within each worker

2024-01-31 Thread James Coleman
On Wed, Jan 31, 2024 at 3:18 PM Robert Haas wrote: > > On Tue, Jan 30, 2024 at 9:56 PM James Coleman wrote: > > I don't follow the "Idle since July" since it just hasn't received > > review since then, so there's been nothing to reply to. > > It wasn't clear t

set_cheapest without checking pathlist

2024-01-31 Thread James Coleman
ely. That being said, on master I don't have a case showing this is necessary. Thanks, James Coleman 1: https://www.postgresql.org/message-id/flat/CAAaqYe-Aq6oNf9NPZnpPE7SgRLomXXWJA1Gz9L9ndi_Nv%3D94Yw%40mail.gmail.com#e0b1a803d0fdb97189ce493f15f99c14 v1-0001-Guard-set_cheapest-with-path

Re: Parallelize correlated subqueries that execute within each worker

2024-01-31 Thread James Coleman
On Tue, Jan 30, 2024 at 10:34 PM Tom Lane wrote: > > James Coleman writes: > > I've finally had a chance to look at this, and I don't believe there's > > any real failure here, merely drift of how the planner works on master > > resulting in this query now being eligi

Re: Parallelize correlated subqueries that execute within each worker

2024-01-30 Thread James Coleman
to reply to. That being said, Vignesh's note in January about a now-failing test is relevant activity, and I've just today responded to that, so I'm changing the status back from Waiting on Author to Needs Review. Regards, James Coleman

Re: Parallelize correlated subqueries that execute within each worker

2024-01-30 Thread James Coleman
at the diff in the patch at that point (v10) that particular test query formed a different plan shape (there were two gather nodes being created, and params crossing between them). But in the current revision of master with the current patch applied that's no longer true: we have a Gather node, and t

Re: Opportunistically pruning page before update

2024-01-29 Thread James Coleman
On Fri, Jan 26, 2024 at 8:33 PM James Coleman wrote: > > On Tue, Jan 23, 2024 at 2:46 AM Dilip Kumar wrote: > > > > On Tue, Jan 23, 2024 at 7:18 AM James Coleman wrote: > > > > > > On Mon, Jan 22, 2024 at 8:21 PM James Coleman wrote: > > > > &

Re: Opportunistically pruning page before update

2024-01-26 Thread James Coleman
On Tue, Jan 23, 2024 at 2:46 AM Dilip Kumar wrote: > > On Tue, Jan 23, 2024 at 7:18 AM James Coleman wrote: > > > > On Mon, Jan 22, 2024 at 8:21 PM James Coleman wrote: > > > > > > See rebased patch attached. > > > > I just realized I left a c

Re: Opportunistically pruning page before update

2024-01-22 Thread James Coleman
On Mon, Jan 22, 2024 at 8:21 PM James Coleman wrote: > > See rebased patch attached. I just realized I left a change in during the rebase that wasn't necessary. v4 attached. Regards, James Coleman v4-0002-Opportunistically-prune-to-avoid-building-a-new-p.patch Description: Binary dat

Re: Opportunistically pruning page before update

2024-01-22 Thread James Coleman
f necessary. > > == > [1] https://commitfest.postgresql.org/46/4384// > [2] > https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest/46/4384 See rebased patch attached. Thanks, James Coleman v3-0001-Allow-getting-lock-before-calling-heap_page_prune.patch Descripti

Re: Add last_commit_lsn to pg_stat_database

2024-01-22 Thread James Coleman
f necessary. > > == > [1] > https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest/46/4355 > > Kind Regards, > Peter Smith. Updated patch attached, Thanks, James Coleman v3-0001-Add-last-commit-s-LSN-to-pg_stat_database.patch Description: Binary data

Re: PG12 change to DO UPDATE SET column references

2024-01-20 Thread James Coleman
On Sat, Jan 20, 2024 at 5:57 PM Tom Lane wrote: > > James Coleman writes: > > On Sat, Jan 20, 2024 at 12:59 PM Tom Lane wrote: > >> A HINT if the bogus column name (1) matches the relation name and > >> (2) is field-qualified seems plausible to me. Then it's p

Re: PG12 change to DO UPDATE SET column references

2024-01-20 Thread James Coleman
On Sat, Jan 20, 2024 at 12:59 PM Tom Lane wrote: > > James Coleman writes: > > I do wonder if it's plausible (and sufficiently easy) to improve the > > error message here. "column 'foo' of relation 'foo'" makes one thing > > that you've written foo.foo, (in my

Re: PG12 change to DO UPDATE SET column references

2024-01-20 Thread James Coleman
On Sat, Jan 20, 2024 at 11:12 AM Tom Lane wrote: > > James Coleman writes: > > Suppose I have this table: > > create table foo (id int primary key); > > > On PG11 this works: > > postgres=# insert into foo (id) values (1) on conflict (id) do update > >

Re: PG12 change to DO UPDATE SET column references

2024-01-20 Thread James Coleman
On Fri, Jan 19, 2024 at 1:53 PM David G. Johnston wrote: > > On Fri, Jan 19, 2024 at 10:01 AM James Coleman wrote: >> >> Making this more confusing is the fact that if I want to do something >> like "SET bar = foo.bar + 1" the table qualification cannot b

PG12 change to DO UPDATE SET column references

2024-01-19 Thread James Coleman
the read column the error is more understandable: ERROR: column reference "bar" is ambiguous It seems to me that it'd be desirable to either allow the unnecessary qualification or give an error that's more easily understood. Regards, James Coleman

Re: Add last_commit_lsn to pg_stat_database

2024-01-17 Thread James Coleman
On Sun, Jan 14, 2024 at 6:01 AM vignesh C wrote: > > On Sat, 10 Jun 2023 at 07:57, James Coleman wrote: > > > > I've previously noted in "Add last commit LSN to > > pg_last_committed_xact()" [1] that it's not possible to monitor how > > many bytes

Re: Add last_commit_lsn to pg_stat_database

2024-01-17 Thread James Coleman
which is > probably correct. But I would appreciate a second opinion on this. Sounds good. > - Wouldn't it be appropriate to add a test or two? Added. > - `if (!XLogRecPtrIsInvalid(commit_lsn))` - I suggest adding > XLogRecPtrIsValid() macro for better readability We have 36 usages of !XLogRecPt

Re: Teach predtest about IS [NOT] proofs

2024-01-17 Thread James Coleman
On Fri, Dec 22, 2023 at 2:48 PM Tom Lane wrote: > > James Coleman writes: > > I've not yet applied all of your feedback, but I wanted to get an > > initial read on your thoughts on how using switch statements ends up > > looking. Attached is a single (pure refac

Re: Teach predtest about IS [NOT] proofs

2023-12-22 Thread James Coleman
On Thu, Dec 14, 2023 at 4:38 PM Tom Lane wrote: > > James Coleman writes: > > On Wed, Dec 13, 2023 at 1:36 PM Tom Lane wrote: > >> I don't have an objection in principle to adding more smarts to > >> predtest.c. However, we should be wary of slowing down ca

Re: Teach predtest about IS [NOT] proofs

2023-12-13 Thread James Coleman
Thanks for taking a look! On Wed, Dec 13, 2023 at 1:36 PM Tom Lane wrote: > > James Coleman writes: > > Attached is a patch that solves that issue. It also teaches predtest about > > quite a few more cases involving BooleanTest expressions (e.g., how they > > relate

Teach predtest about IS [NOT] proofs

2023-12-11 Thread James Coleman
;x, y" case as well as the "y, x" case with a single call so as to eliminate a lot of repetition in clause/expression test cases. If reviewers agree that's desirable, then I could do that as a precursor. Regards, James Coleman v1-0001-Teach-predtest-about-IS-NOT-bool-proofs.patch Description: Binary data

Re: RFC: Logging plan of the running query

2023-10-18 Thread James Coleman
tates may leak depending upon when this > >>> function gets called. > >>> 3. Building features on top as James envisions will be easier. In my view the fact that auto_explain is itself not part of core is a mistake, and I know there are more prominent hackers than myself who hold that view. While that decision as regards auto_explain has long since been made (and there would be work to undo it), I don't believe that we should repeat that choice here. I'm -10 on moving this into auto_explain. However perhaps there is still an opportunity for moving some of the auto_explain code into core so as to enable deduplicating the code. Regards, James Coleman

Re: RFC: Logging plan of the running query

2023-10-06 Thread James Coleman
On Fri, Oct 6, 2023 at 8:58 AM torikoshia wrote: > > On 2023-10-04 03:00, James Coleman wrote: > > On Thu, Sep 7, 2023 at 2:09 AM torikoshia > > wrote: > >> > >> On 2023-09-06 11:17, James Coleman wrote: > >> > >> >> > I've

Re: Opportunistically pruning page before update

2023-10-06 Thread James Coleman
Hi, Thanks for taking a look! On Fri, Oct 6, 2023 at 1:18 AM Dilip Kumar wrote: > > On Thu, Oct 5, 2023 at 2:35 AM James Coleman wrote: > > > > I talked to Andres and Peter again today, and out of that conversation > > I have some observations and ideas for future

Re: [DOCS] HOT - correct claim about indexes not referencing old line pointers

2023-10-04 Thread James Coleman
On Wed, Oct 4, 2023 at 9:42 AM Robert Haas wrote: > > On Wed, Oct 4, 2023 at 9:36 AM James Coleman wrote: > > Are you thinking we should simply elide the fact that there is pruning > > that happens outside of HOT? Or add that information onto the HOT > > page, even though

Re: Opportunistically pruning page before update

2023-10-04 Thread James Coleman
On Tue, Sep 26, 2023 at 8:30 AM James Coleman wrote: > > On Tue, Sep 5, 2023 at 1:40 PM Melanie Plageman > wrote: > > > > On Wed, Jun 21, 2023 at 8:51 AM James Coleman wrote: > > > While at PGCon I was chatting with Andres (and I think Peter G. and a > &g

Re: [DOCS] HOT - correct claim about indexes not referencing old line pointers

2023-10-04 Thread James Coleman
On Wed, Oct 4, 2023 at 9:18 AM Robert Haas wrote: > > On Tue, Oct 3, 2023 at 3:35 PM James Coleman wrote: > > I like your changes. Reading through this several times, and noting > > Peter's comments about pruning being more than just HOT, I'm thinking > > that ra

Re: [DOCS] HOT - correct claim about indexes not referencing old line pointers

2023-10-03 Thread James Coleman
On Mon, Oct 2, 2023 at 2:55 PM Robert Haas wrote: > > On Sat, Sep 30, 2023 at 1:05 AM Peter Geoghegan wrote: > > > This is why I discovered it: it says "indexes do not reference their > > > page item identifiers", which is manifestly not true when talking > > > about the root item, and in fact

Re: RFC: Logging plan of the running query

2023-10-03 Thread James Coleman
On Thu, Sep 7, 2023 at 2:09 AM torikoshia wrote: > > On 2023-09-06 11:17, James Coleman wrote: > > >> > I've never been able to reproduce it (haven't tested the new version, > >> > but v28 at least) on my M1 Mac; where I've reproduced it is on Debian >

Re: [DOCS] HOT - correct claim about indexes not referencing old line pointers

2023-09-29 Thread James Coleman
On Fri, Sep 29, 2023 at 4:06 PM Peter Geoghegan wrote: > > On Fri, Sep 29, 2023 at 11:45 AM James Coleman > wrote:my reading the issue is that "old versions" doesn't say > > anything about "old HOT versions; it seems to be describing what > > happens general

Re: [DOCS] HOT - correct claim about indexes not referencing old line pointers

2023-09-29 Thread James Coleman
e the first time a heap-only tuple is written. And when it's the first heap-only tuple the "old version" would be the original version, which would not be a heap-only tuple. I can work up a tweaked version of the patch that shows there are two paths here (original tuple is being updated versus an intermediate heap-only tuple is being updated); would you be willing to consider that? Thanks, James Coleman

Re: Fix incorrect comment reference

2023-09-29 Thread James Coleman
On Fri, Sep 29, 2023 at 2:26 PM Bruce Momjian wrote: > > On Mon, Jan 23, 2023 at 06:42:45PM -0500, James Coleman wrote: > > On Mon, Jan 23, 2023 at 4:07 PM James Coleman wrote: > > > > > > On Mon, Jan 23, 2023 at 3:41 PM Robert Haas wrote: > > > > >

[DOCS] HOT - correct claim about indexes not referencing old line pointers

2023-09-29 Thread James Coleman
is correctly, attached is a patch correcting the description. Thanks, James Coleman v1-0001-Correct-HOT-docs-to-account-for-LP_REDIRECT.patch Description: Binary data

Re: Opportunistically pruning page before update

2023-09-26 Thread James Coleman
On Tue, Sep 5, 2023 at 1:40 PM Melanie Plageman wrote: > > On Wed, Jun 21, 2023 at 8:51 AM James Coleman wrote: > > While at PGCon I was chatting with Andres (and I think Peter G. and a > > few others who I can't remember at the moment, apologies) and Andres >

Re: RFC: Logging plan of the running query

2023-09-05 Thread James Coleman
On Tue, Sep 5, 2023 at 9:59 AM torikoshia wrote: > > On 2023-08-28 22:47, James Coleman wrote: > > On Mon, Aug 28, 2023 at 3:01 AM torikoshia > > wrote: > >> > >> On 2023-08-26 21:03, James Coleman wrote: > >> > On Fri, Aug 25, 2023 at 7:43 AM Jame

Re: RFC: Logging plan of the running query

2023-08-28 Thread James Coleman
On Mon, Aug 28, 2023 at 3:01 AM torikoshia wrote: > > On 2023-08-26 21:03, James Coleman wrote: > > On Fri, Aug 25, 2023 at 7:43 AM James Coleman wrote: > >> > >> On Thu, Aug 17, 2023 at 10:02 AM torikoshia > >> wrote: > >> > > >> &

Re: RFC: Logging plan of the running query

2023-08-26 Thread James Coleman
On Fri, Aug 25, 2023 at 7:43 AM James Coleman wrote: > > On Thu, Aug 17, 2023 at 10:02 AM torikoshia > wrote: > > > > On 2023-06-16 01:34, James Coleman wrote: > > > Attached is v28 > > > which sets ProcessLogQueryPlanInterruptActive to false in err

Re: RFC: Logging plan of the running query

2023-08-25 Thread James Coleman
On Thu, Aug 17, 2023 at 10:02 AM torikoshia wrote: > > On 2023-06-16 01:34, James Coleman wrote: > > Attached is v28 > > which sets ProcessLogQueryPlanInterruptActive to false in errfinish > > when necessary. Once built with those two patches I'm simply running > > `

Re: pgindent (probably my missing something obvious)

2023-07-03 Thread James Coleman
On Mon, Jul 3, 2023 at 9:20 PM Tom Lane wrote: > > James Coleman writes: > > This is the first time I've run pgindent on my current machine, and it > > doesn't seem to be making any modifications to source files. For > > example this command: > > > ./src/to

Re: Parallelize correlated subqueries that execute within each worker

2023-07-03 Thread James Coleman
On Sun, Jun 11, 2023 at 10:23 PM James Coleman wrote: > > ... > > And while trying the v9 patch I came across a crash with the query > > below. > > > > set min_parallel_table_scan_size to 0; > > set parallel_setup_cost to 0; > > set parallel_tuple_cost to

pgindent (probably my missing something obvious)

2023-07-03 Thread James Coleman
very long function calls. I've downloaded the latest typedefs list, but I haven't added any types anyway. What obvious thing am I missing? Thanks, James Coleman

Re: Analyze on table creation?

2023-06-27 Thread James Coleman
On Mon, Jun 26, 2023 at 4:16 PM James Coleman wrote: > > On Mon, Jun 26, 2023 at 4:00 PM Andres Freund wrote: > > > > Hi, > > > > On 2023-06-26 13:40:49 -0400, James Coleman wrote: > > > Have we ever discussed running an analyze immediately after creati

Re: Analyze on table creation?

2023-06-26 Thread James Coleman
On Mon, Jun 26, 2023 at 4:00 PM Andres Freund wrote: > > Hi, > > On 2023-06-26 13:40:49 -0400, James Coleman wrote: > > Have we ever discussed running an analyze immediately after creating a > > table? > > That doesn't make a whole lot of sense to me - we could jus

Re: Analyze on table creation?

2023-06-26 Thread James Coleman
cc'ing Tom because I'm curious if he's willing to provide some greater context on the commit in question. On Mon, Jun 26, 2023 at 2:16 PM Pavel Stehule wrote: > > > > po 26. 6. 2023 v 19:48 odesílatel James Coleman napsal: >> >> On Mon, Jun 26, 2023 at 1:45 PM

Re: Analyze on table creation?

2023-06-26 Thread James Coleman
On Mon, Jun 26, 2023 at 1:45 PM Pavel Stehule wrote: > > > > po 26. 6. 2023 v 19:43 odesílatel Pavel Stehule > napsal: >> >> Hi >> >> po 26. 6. 2023 v 19:41 odesílatel James Coleman napsal: >>> >>> Hello, >>> >>> Have w

Analyze on table creation?

2023-06-26 Thread James Coleman
, and so I assume people have considered it before. If so, I'd like to understand why the conclusion was not to do it, or, alternatively if it's a lack of tuits. Regards, James Coleman

Re: Stampede of the JIT compilers

2023-06-25 Thread James Coleman
n't > have a Google Cloud SQL or RDS instance running right to verify their > settings. I do seem to remember that they did as well though, at least a > while back. > > > Michael I believe it's off by default in Aurora Postgres also. Regards, James Coleman

Re: Stampede of the JIT compilers

2023-06-24 Thread James Coleman
On Sat, Jun 24, 2023 at 8:14 PM David Rowley wrote: > > On Sun, 25 Jun 2023 at 05:54, Tom Lane wrote: > > > > James Coleman writes: > > > On Sat, Jun 24, 2023 at 7:40 AM Tomas Vondra > > > wrote: > > >> On 6/24/23 02:33, David Rowley wrote: >

Re: Stampede of the JIT compilers

2023-06-24 Thread James Coleman
On Sat, Jun 24, 2023 at 1:54 PM Tom Lane wrote: > > James Coleman writes: > > In that context capping the number of backends compiling, particularly > > where plans (and JIT?) might be cached, could well save us (depending > > on workload). > > TBH I do not

Re: Stampede of the JIT compilers

2023-06-24 Thread James Coleman
On Sat, Jun 24, 2023 at 7:40 AM Tomas Vondra wrote: > > > > On 6/24/23 02:33, David Rowley wrote: > > On Sat, 24 Jun 2023 at 02:28, James Coleman wrote: > >> There are a couple of issues here. I'm sure it's been discussed > >> before, and it's not the

Stampede of the JIT compilers

2023-06-23 Thread James Coleman
like "max_concurrent_jit_compilations" to cap the number of backends that may be compiling a query at any given point so that we avoid an optimization from running amok and consuming all of a servers resources? Regards, James Coleman

Re: Memory leak in incremental sort re-scan

2023-06-21 Thread James Coleman
d_keys). That avoids unnecessary recreation of the sort states, but it also fixes the problem Tom noted as well: the call to preparePresortedCols() is already guarded by a test on fullsort_state being NULL, so with this change we also won't unnecessarily redo that work. Regards, James Coleman v2-0001-Fix-memory-leak-in-incremental-sort-rescan.patch Description: Binary data

Re: Use of additional index columns in rows filtering

2023-06-21 Thread James Coleman
On Wed, Jun 21, 2023 at 11:28 AM Tomas Vondra wrote: > > > > On 6/21/23 14:45, James Coleman wrote: > > Hello, > > > > I've cc'd Jeff Davis on this due to a conversation we had at PGCon > > about applying filters on index tuples during index scans. > >

Opportunistically pruning page before update

2023-06-21 Thread James Coleman
properly to that, so I'm wondering if there's anyone who might be interested in collaborating on that part. Other TODOs: - Audit other callers of RelationSetTargetBlock() to ensure they don't hold pointers into the page. Regards, James Coleman v1-0001-Allow-getting-lock-before-calling

Re: Use of additional index columns in rows filtering

2023-06-21 Thread James Coleman
: (b = 4) > Buffers: shared hit=544 > Planning Time: 0.105 ms > Execution Time: 13.690 ms > (10 rows) > > ... I did also confirm that this properly identifies cases Jeff had mentioned to me like "Index Filter: (((a * 2) > 50) AND ((b % 10) = 4))". I noticed also you still had questions/TODOs about handling index scans for join clauses. Regards, James Coleman 1: https://www.postgresql.org/message-id/20230609000600.syqy447e6metnvyj%40awork3.anarazel.de

Re: path->param_info only set for lateral?

2023-06-20 Thread James Coleman
On Sun, Jun 18, 2023 at 10:57 PM Tom Lane wrote: > > James Coleman writes: > > Over in "Parallelize correlated subqueries that execute within each > > worker" [1} Richard Guo found a bug in the current version of my patch > > in that thread. While debugging

path->param_info only set for lateral?

2023-06-18 Thread James Coleman
're not going to get any ParamPathInfo added to the path or the rel. Is there a reason why we don't track the required relids providing the PARAM_EXEC params in this case? Thanks, James Coleman 1: https://www.postgresql.org/message-id/CAMbWs4_evjcMzN8Gw78bHfhfo2FKJThqhEjRJRmoMZx%3DNXcJ7w%40mail.gmail.com

Re: RFC: Logging plan of the running query

2023-06-15 Thread James Coleman
On Thu, Jun 15, 2023 at 9:00 AM torikoshia wrote: > > On 2023-06-15 01:48, James Coleman wrote: > > On Tue, Jun 13, 2023 at 11:53 AM James Coleman > > wrote: > >> > >> ... > >> I'm going to re-run tests with my patch version + resetting the flag

Re: RFC: Logging plan of the running query

2023-06-14 Thread James Coleman
On Tue, Jun 13, 2023 at 11:53 AM James Coleman wrote: > > ... > I'm going to re-run tests with my patch version + resetting the flag > on SIGINT (and any other error condition) to be certain that the issue > you uncovered (where backends get stuck after a SIGINT not responding >

Re: RFC: Logging plan of the running query

2023-06-13 Thread James Coleman
On Tue, Jun 13, 2023 at 11:22 AM torikoshia wrote: > > On 2023-06-13 00:52, James Coleman wrote: > >> > >> > I've attached v27. The important change here in 0001 is that it > >> > guarantees the interrupt handler is re-entrant, since that was a bug > >

Re: RFC: Logging plan of the running query

2023-06-12 Thread James Coleman
On Sun, Jun 11, 2023 at 11:07 PM torikoshia wrote: > > On 2023-06-06 03:26, James Coleman wrote: > > On Mon, Jun 5, 2023 at 4:30 AM torikoshia > > wrote: > >> > >> On 2023-06-03 02:51, James Coleman wrote: > >> > Hello, > >> > >

Re: Parallelize correlated subqueries that execute within each worker

2023-06-11 Thread James Coleman
On Tue, Jun 6, 2023 at 4:36 AM Richard Guo wrote: > > > On Mon, Jan 23, 2023 at 10:00 PM James Coleman wrote: >> >> Which this patch we do in fact now see (as expected) rels with >> non-empty lateral_relids showing up in generate_[useful_]gather_paths. >> And t

Add last_commit_lsn to pg_stat_database

2023-06-09 Thread James Coleman
ght of exposing it in pg_stat_wal, but that's per-cluster rather than per-database (indeed, this is a flaw I hadn't considered in the original patch), so I think pg_stat_database is the correct location. I've attached a patch to track the latest commit's LSN in pg_stat_database. Regards, James Col

Re: RFC: Logging plan of the running query

2023-06-05 Thread James Coleman
On Mon, Jun 5, 2023 at 4:30 AM torikoshia wrote: > > On 2023-06-03 02:51, James Coleman wrote: > > Hello, > > > > Thanks for working on this patch! Sure thing! I'm *very interested* in seeing this available, and I think it paves the way for some additional features later

Re: RFC: Logging plan of the running query

2023-06-02 Thread James Coleman
ening this patch? I'd be happy to provide further review and help to try to push this along. I've rebased the patch and attached as v26. Thanks, James Coleman v26-0001-Add-function-to-log-the-plan-of-the-query.patch Description: Binary data

Re: An inefficient query caused by unnecessary PlaceHolderVar

2023-06-01 Thread James Coleman
On Wed, May 31, 2023 at 10:30 PM Richard Guo wrote: > > > On Wed, May 31, 2023 at 1:27 AM James Coleman wrote: >> >> This looks good to me. > > > Thanks for the review! Sure thing! >> >> A few small tweaks suggested to comment wording: >> &g

Re: An inefficient query caused by unnecessary PlaceHolderVar

2023-05-30 Thread James Coleman
* join. I think this is clearer: "references something outside the subquery being pulled up and is not under the same lowest outer join." One other thing: it would be helpful to have the test query output be stable between HEAD and this patch; perhaps add: order by 1, 2, 3, 4, 5, 6, 7 to ensure stability? Thanks, James Coleman

Re: pg_rewind: warn when checkpoint hasn't happened after promotion

2023-02-28 Thread James Coleman
On Mon, Feb 27, 2023 at 2:33 AM Heikki Linnakangas wrote: > > On 16/11/2022 07:17, kuroda.keis...@nttcom.co.jp wrote: > >> The issue here is pg_rewind looks into control file to determine the > >> soruce timeline, because the control file is not updated until the > >> first checkpoint ends after

Re: Parallelize correlated subqueries that execute within each worker

2023-02-08 Thread James Coleman
On Mon, Feb 6, 2023 at 11:39 AM Antonin Houska wrote: > > James Coleman wrote: > > Which this patch we do in fact now see (as expected) rels with > > non-empty lateral_relids showing up in generate_[useful_]gather_paths. > > And the partial paths can now have non-empty

Re: Fix incorrect comment reference

2023-01-23 Thread James Coleman
On Mon, Jan 23, 2023 at 4:07 PM James Coleman wrote: > > On Mon, Jan 23, 2023 at 3:41 PM Robert Haas wrote: > > > > On Mon, Jan 23, 2023 at 3:19 PM James Coleman wrote: > > > On Mon, Jan 23, 2023 at 1:26 PM Robert Haas wrote: > > > > On Mon, Jan 2

Re: Fix incorrect comment reference

2023-01-23 Thread James Coleman
On Mon, Jan 23, 2023 at 3:41 PM Robert Haas wrote: > > On Mon, Jan 23, 2023 at 3:19 PM James Coleman wrote: > > On Mon, Jan 23, 2023 at 1:26 PM Robert Haas wrote: > > > On Mon, Jan 23, 2023 at 8:31 AM James Coleman wrote: > > > > See the attached for a sim

Re: Fix incorrect comment reference

2023-01-23 Thread James Coleman
On Mon, Jan 23, 2023 at 1:26 PM Robert Haas wrote: > > On Mon, Jan 23, 2023 at 8:31 AM James Coleman wrote: > > See the attached for a simple comment fix -- the referenced > > generate_useful_gather_paths call isn't in grouping_planner it's in > > apply_scanjoin_target_to

Re: Parallelize correlated subqueries that execute within each worker

2023-01-23 Thread James Coleman
On Sat, Jan 21, 2023 at 10:07 PM James Coleman wrote: > ... > While working through Tomas's comment about a conditional in the > max_parallel_hazard_waker being guaranteed true I realized that in the > current version of the patch the safe_param_ids tracking in > is_parallel_safe

Fix incorrect comment reference

2023-01-23 Thread James Coleman
Hello, See the attached for a simple comment fix -- the referenced generate_useful_gather_paths call isn't in grouping_planner it's in apply_scanjoin_target_to_paths. Thanks, James Coleman v1-0001-Fixup-incorrect-comment.patch Description: Binary data

Re: Parallelize correlated subqueries that execute within each worker

2023-01-21 Thread James Coleman
On Wed, Jan 18, 2023 at 9:34 PM James Coleman wrote: > > On Wed, Jan 18, 2023 at 2:09 PM Tomas Vondra > wrote: > > > > Hi, > > > > This patch hasn't been updated since September, and it got broken by > > 4a29eabd1d91c5484426bc5836e0a7143b064f5a which the

Re: Parallelize correlated subqueries that execute within each worker

2023-01-18 Thread James Coleman
Unique (cost=18582710.39..18613960.39 rows=1 ...) >-> Sort (cost=18582710.39..18593127.06 ...) > Sort Key: t.unique1, ((SubPlan 1)) >... > > which probably makes sense, as the cost estimate decreases a bit. Off the cuf

Re: Commit fest 2022-11

2022-11-14 Thread James Coleman
us in this situation. Without that though the patch authors are left to wade through unrelated discussion, and, probably more importantly, the patch discussion thread doesn't show the current state (I think bumping there is more likely to prompt activity as well). James Coleman

Re: cirrus-ci cross-build interactions?

2022-09-26 Thread James Coleman
On Mon, Sep 26, 2022 at 10:48 PM Andres Freund wrote: > > Hi, > > On 2022-09-26 22:36:24 -0400, James Coleman wrote: > > I had a build on Cirrus CI fail tonight in what I have to assume was > > either a problem with caching across builds or some such similar > > fl

Re: Parallelize correlated subqueries that execute within each worker

2022-09-26 Thread James Coleman
On Mon, Mar 21, 2022 at 8:48 PM Andres Freund wrote: > > Hi, > > On 2022-01-22 20:25:19 -0500, James Coleman wrote: > > On the other hand this is a dramatically simpler patch series. > > Assuming the approach is sound, it should much easier to maintain than

Re: cirrus-ci cross-build interactions?

2022-09-26 Thread James Coleman
On Mon, Sep 26, 2022 at 10:36 PM James Coleman wrote: > > I had a build on Cirrus CI fail tonight in what I have to assume was > either a problem with caching across builds or some such similar > flakiness. In the Debian task [1] I received this error: > > su postgres -c "

cirrus-ci cross-build interactions?

2022-09-26 Thread James Coleman
ild [2] seems to be fine. I've double-checked there are no differences between the commits on the two builds (git diff shows no output). Is it possible we're missing some kind of necessary build isolation in the Cirrus CI scripting? Thanks, James Coleman 1: https://cirrus-ci.com/task/61415592582

Add hint about downloadable logs to CI README

2022-09-26 Thread James Coleman
I was wondering about how to debug failed builds on Cirrus CI, and after poking at the interface I realized we helpfully upload the logs from CI runs for user download. In an effort to save the next person a few minutes I thought the attached minor patch would help. Thanks, James Coleman v1

Re: Allow foreign keys to reference a superset of unique columns

2022-09-26 Thread James Coleman
On Mon, Sep 26, 2022 at 10:04 AM Wolfgang Walther wrote: > > James Coleman: > > As I was reading through the email chain I had this thought: could you > > get the same benefit (or 90% of it anyway) by instead allowing the > > creation of a uniqueness constraint that co

Re: Allow foreign keys to reference a superset of unique columns

2022-09-26 Thread James Coleman
On Mon, Sep 26, 2022 at 9:59 AM Wolfgang Walther wrote: > > James Coleman: > > So the broader point I'm trying to make is that, as I understand it, > > indexes backing foreign key constraints is an implementation detail. > > The SQL standard details the behavior of

Re: Consider parallel for lateral subqueries with limit

2022-09-26 Thread James Coleman
On Mon, Sep 26, 2022 at 10:37 AM Robert Haas wrote: > > On Thu, Sep 22, 2022 at 5:19 PM James Coleman wrote: > > > Your sample query gets a plan like this: > > > > > > Nested Loop Left Join (cost=0.00..1700245.00 rows=1 width=8) > > >-> S

  1   2   3   4   5   6   >