Re: BitmapHeapScan streaming read user and prelim refactoring

2024-04-26 Thread Melanie Plageman
On Thu, Apr 25, 2024 at 7:57 PM Tom Lane wrote: > > I wrote: > > Hmm, is that actually true? There's no more reason to think a tuple > > in a temp table is old enough to be visible to all other sessions > > than one in any other table. It could be all right if we had a > > special-case rule for

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-04-25 Thread Melanie Plageman
On Wed, Apr 24, 2024 at 4:46 PM Melanie Plageman wrote: > > On Tue, Apr 23, 2024 at 6:43 PM Tomas Vondra > wrote: > > > > On 4/23/24 18:05, Melanie Plageman wrote: > > > One other note: there is some concurrency effect in the parallel > > > schedule group

Re: New GUC autovacuum_max_threshold ?

2024-04-25 Thread Melanie Plageman
On Thu, Apr 25, 2024 at 2:52 AM Frédéric Yhuel wrote: > > Le 24/04/2024 à 21:10, Melanie Plageman a écrit : > > On Wed, Apr 24, 2024 at 8:08 AM Frédéric Yhuel > > wrote: > >> > >> Hello, > >> > >> I would like to suggest a new parameter, a

Re: broken reading on standby (PostgreSQL 16.2)

2024-04-25 Thread Melanie Plageman
On Thu, Apr 25, 2024 at 2:13 AM Pavel Stehule wrote: > > Hi > > yesterday, I had to fix strange issue on standby server > > The query to freshly updated data fails > > select * from seller_success_rate where create_time::date = '2024-04-23'; > ERROR: 58P01: could not access status of transaction

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-04-24 Thread Melanie Plageman
On Tue, Apr 23, 2024 at 6:43 PM Tomas Vondra wrote: > > On 4/23/24 18:05, Melanie Plageman wrote: > > The patch with a fix is attached. I put the test in > > src/test/regress/sql/join.sql. It isn't the perfect location because > > it is testing something exercisable with

Re: New GUC autovacuum_max_threshold ?

2024-04-24 Thread Melanie Plageman
On Wed, Apr 24, 2024 at 8:08 AM Frédéric Yhuel wrote: > > Hello, > > I would like to suggest a new parameter, autovacuum_max_threshold, which > would set an upper limit on the number of tuples to delete/update/insert > prior to vacuum/analyze. Hi Frédéric, thanks for the proposal! You are

Re: soliciting patches to review

2024-04-23 Thread Melanie Plageman
On Tue, Apr 23, 2024 at 1:27 PM Robert Haas wrote: > > Hi, > > Just a quick update. We have so far had 8 suggested patches from 6 > people, if I haven't missed anything. I'm fairly certain that not all > of those patches are going to be good candidates for this session, so > it would be great if

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-04-23 Thread Melanie Plageman
On Mon, Apr 22, 2024 at 1:01 PM Melanie Plageman wrote: > > On Thu, Apr 18, 2024 at 5:39 AM Tomas Vondra > wrote: > > > > On 4/18/24 09:10, Michael Paquier wrote: > > > On Sun, Apr 07, 2024 at 10:54:56AM -0400, Melanie Plageman wrote: > > >> I've added

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-04-22 Thread Melanie Plageman
On Thu, Apr 18, 2024 at 5:39 AM Tomas Vondra wrote: > > On 4/18/24 09:10, Michael Paquier wrote: > > On Sun, Apr 07, 2024 at 10:54:56AM -0400, Melanie Plageman wrote: > >> I've added an open item [1], because what's one open item when you can > >> have two? (me) >

Re: Table AM Interface Enhancements

2024-04-12 Thread Melanie Plageman
On Thu, Apr 11, 2024 at 6:04 PM Alexander Korotkov wrote: > > On Fri, Apr 12, 2024 at 12:04 AM Andres Freund wrote: > > On 2024-04-11 20:46:02 +0300, Alexander Korotkov wrote: > > > I hope this work is targeting pg18. > > > > I think anything of the scope discussed by Melanie would be very

Re: Table AM Interface Enhancements

2024-04-11 Thread Melanie Plageman
On Thu, Apr 11, 2024 at 12:19 PM Melanie Plageman wrote: > > On Wed, Apr 10, 2024 at 5:21 PM Andres Freund wrote: > > > > Hi, > > > > On 2024-04-10 16:50:44 -0400, Melanie Plageman wrote: > > > This brings up a question about the prefetching. We

Re: Table AM Interface Enhancements

2024-04-11 Thread Melanie Plageman
On Wed, Apr 10, 2024 at 5:21 PM Andres Freund wrote: > > Hi, > > On 2024-04-10 16:50:44 -0400, Melanie Plageman wrote: > > This brings up a question about the prefetching. We never had to have > > this discussion for sequential scan streaming read because it didn't &g

Re: Table AM Interface Enhancements

2024-04-10 Thread Melanie Plageman
On Wed, Apr 10, 2024 at 4:33 PM Andres Freund wrote: > > Hi, > > On 2024-04-10 16:24:40 -0400, Melanie Plageman wrote: > > This thread has been moving pretty fast, so could someone point out > > which version of the patch has the modifications to > > acquire_sample

Re: Table AM Interface Enhancements

2024-04-10 Thread Melanie Plageman
On Wed, Apr 10, 2024 at 4:03 PM Andres Freund wrote: > > Hi, > > On 2024-04-10 15:19:47 +0300, Alexander Korotkov wrote: > > On Mon, Apr 8, 2024 at 9:54 PM Robert Haas wrote: > > > On Mon, Apr 8, 2024 at 12:33 PM Alexander Korotkov > > > wrote: > > > > Yes, it was my mistake. I got rushing

Re: PostgreSQL 17 Release Management Team & Feature Freeze

2024-04-08 Thread Melanie Plageman
On Mon, Apr 8, 2024 at 10:41 AM Robert Treat wrote: > > On Mon, Apr 8, 2024 at 10:27 AM Melanie Plageman > wrote: > > On Mon, Apr 8, 2024 at 9:26 AM Robert Haas wrote: > > > On Sun, Apr 7, 2024 at 6:50 PM Michael Paquier > > > wrote: > > > > And,

Re: PostgreSQL 17 Release Management Team & Feature Freeze

2024-04-08 Thread Melanie Plageman
On Mon, Apr 8, 2024 at 9:26 AM Robert Haas wrote: > > On Sun, Apr 7, 2024 at 6:50 PM Michael Paquier wrote: > > And, as of the moment of typing this email, I get: > > =# select '2024-04-08 00:00-12:00' - now() as time_remaining; > > time_remaining > > - > > 13:10:35.688134 > >

Re: Extension Enhancement: Buffer Invalidation in pg_buffercache

2024-04-07 Thread Melanie Plageman
On Sat, Apr 6, 2024 at 7:08 PM Thomas Munro wrote: > > On second thoughts, I think the original "invalidate" terminology was > fine, no need to invent a new term. > > I thought of a better name for the bufmgr.c function though: > InvalidateUnpinnedBuffer(). That name seemed better to me after I

Re: Use streaming read API in ANALYZE

2024-04-07 Thread Melanie Plageman
On Sun, Apr 07, 2024 at 03:00:00PM -0700, Andres Freund wrote: > Hi, > > On 2024-04-07 16:59:26 -0400, Melanie Plageman wrote: > > From 1dc2343661f3edb3b1bc4307afb0e956397eb76c Mon Sep 17 00:00:00 2001 > > From: Melanie Plageman > > Date: Sun, 7 Apr 2024 14:55:22 -040

Re: Use streaming read API in ANALYZE

2024-04-07 Thread Melanie Plageman
On Sun, Apr 7, 2024 at 3:57 PM Melanie Plageman wrote: > > On Thu, Apr 04, 2024 at 02:03:30PM +0300, Nazir Bilal Yavuz wrote: > > > > On Wed, 3 Apr 2024 at 23:44, Melanie Plageman > > wrote: > > > > > > I don't see the point of BlockSampler_HasMore() a

Re: Use streaming read API in ANALYZE

2024-04-07 Thread Melanie Plageman
On Thu, Apr 04, 2024 at 02:03:30PM +0300, Nazir Bilal Yavuz wrote: > > On Wed, 3 Apr 2024 at 23:44, Melanie Plageman > wrote: > > > > I don't see the point of BlockSampler_HasMore() anymore. I removed it in > > the attached and made BlockSampler_Next() return I

Re: Streaming I/O, vectored I/O (WIP)

2024-04-07 Thread Melanie Plageman
On Sun, Apr 7, 2024 at 1:33 PM Nazir Bilal Yavuz wrote: > > Hi, > > On Tue, 2 Apr 2024 at 11:40, Thomas Munro wrote: > > > > I had been planning to commit v14 this morning but got cold feet with > > the BMR-based interface. Heikki didn't like it much, and in the end, > > neither did I. I have

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-04-07 Thread Melanie Plageman
On Sun, Apr 7, 2024 at 10:42 AM Tomas Vondra wrote: > > > > On 4/7/24 16:24, Melanie Plageman wrote: > >>> Thanks! I thought about it a bit more, and I got worried about the > >>> > >>> Assert(scan->rs_empty_tuples_pending =

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-04-07 Thread Melanie Plageman
On Sun, Apr 7, 2024 at 10:10 AM Tomas Vondra wrote: > > On 4/7/24 15:11, Melanie Plageman wrote: > > > Also, the iterators in the TableScanDescData might be something I > > could live with in the source code for a couple months before we make > > the rest of the changes

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-04-07 Thread Melanie Plageman
On Sun, Apr 7, 2024 at 7:38 AM Tomas Vondra wrote: > > > > On 4/7/24 06:17, Melanie Plageman wrote: > > On Sun, Apr 07, 2024 at 02:27:43AM +0200, Tomas Vondra wrote: > >> On 4/6/24 23:34, Melanie Plageman wrote: > >>> ... > >>>> > >&

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-04-07 Thread Melanie Plageman
On Sun, Apr 7, 2024 at 7:38 AM Tomas Vondra wrote: > > On 4/7/24 06:17, Melanie Plageman wrote: > >> What bothers me on 0006-0008 is that the justification in the commit > >> messages is "future commit will do something". I think it's fine to have > >>

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-04-06 Thread Melanie Plageman
On Sun, Apr 07, 2024 at 02:27:43AM +0200, Tomas Vondra wrote: > On 4/6/24 23:34, Melanie Plageman wrote: > > ... > >> > >> I realized it makes more sense to add a FIXME (I used XXX. I'm not when > >> to use what) with a link to the message where Andres desc

Re: Streaming read-ready sequential scan code

2024-04-06 Thread Melanie Plageman
k, heap sequential scans and TID range scans now use the streaming read API introduced in b5a9b18cd0. Author: Melanie Plageman Reviewed-by: Andres Freund Discussion: https://postgr.es/m/flat/CAAKRu_YtXJiYKQvb5JsA2SkwrsizYLugs4sSOZh3EAjKUg%3DgEQ%40mail.gmail.com ---

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-04-06 Thread Melanie Plageman
On Sat, Apr 06, 2024 at 04:57:51PM +0200, Tomas Vondra wrote: > On 4/6/24 15:40, Melanie Plageman wrote: > > On Sat, Apr 06, 2024 at 02:51:45AM +0200, Tomas Vondra wrote: > >> > >> > >> On 4/6/24 01:53, Melanie Plageman wrote: > >>> On Fri, Apr 05,

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-04-06 Thread Melanie Plageman
On Sat, Apr 06, 2024 at 02:51:45AM +0200, Tomas Vondra wrote: > > > On 4/6/24 01:53, Melanie Plageman wrote: > > On Fri, Apr 05, 2024 at 04:06:34AM -0400, Melanie Plageman wrote: > >> On Thu, Apr 04, 2024 at 04:35:45PM +0200, Tomas Vondra wrote: > >>> On

Re: Streaming read-ready sequential scan code

2024-04-05 Thread Melanie Plageman
On Fri, Apr 5, 2024 at 7:28 PM Thomas Munro wrote: > > On Sat, Apr 6, 2024 at 6:55 AM Melanie Plageman > wrote: > > I haven't looked into or reviewed the memory prefetching part. > > > > While reviewing 0002, I realized that I don't quite see how > > rea

Re: Streaming read-ready sequential scan code

2024-04-05 Thread Melanie Plageman
9cbf5b202a Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Fri, 5 Apr 2024 12:08:24 +1300 Subject: [PATCH v11 2/4] Improve read_stream.c's fast path. Unfortunately the "fast path" for cached scans that don't do any I/O was coded in a way that could be used by pg_prewarm, but not the p

Re: Streaming read-ready sequential scan code

2024-04-05 Thread Melanie Plageman
On Thu, Apr 4, 2024 at 12:39 PM Andres Freund wrote: > > On 2024-04-04 22:37:39 +1300, Thomas Munro wrote: > > On Thu, Apr 4, 2024 at 10:31 PM Thomas Munro wrote: > > > Alright what about this? > > I think it's probably worth adding a bit more of the commit message to the > function comment.

Re: Streaming read-ready sequential scan code

2024-04-04 Thread Melanie Plageman
On Thu, Apr 4, 2024 at 7:45 AM Thomas Munro wrote: > > On Thu, Apr 4, 2024 at 8:02 PM David Rowley wrote: > > 3a4a3537a > > latency average = 34.497 ms > > latency average = 34.538 ms > > > > 3a4a3537a + read_stream_for_seqscans.patch > > latency average = 40.923 ms > > latency average = 41.415

Re: Streaming read-ready sequential scan code

2024-04-04 Thread Melanie Plageman
On Thu, Apr 4, 2024 at 3:02 AM David Rowley wrote: > > On Thu, 4 Apr 2024 at 16:45, David Rowley wrote: > > I've pushed the v9-0001 with that rename done. > > I've now just pushed the 0002 patch with some revisions: Thanks! > 1. The function declarations you added for

Re: Streaming read-ready sequential scan code

2024-04-03 Thread Melanie Plageman
On Wed, Apr 3, 2024 at 9:08 PM David Rowley wrote: > > On Thu, 4 Apr 2024 at 06:03, Melanie Plageman > wrote: > > Attached v8 is rebased over current master (which now has the > > streaming read API). > > I've looked at the v8-0001 patch. Thanks for taking a

Re: Streaming read-ready sequential scan code

2024-04-03 Thread Melanie Plageman
On Wed, Apr 3, 2024 at 4:28 PM Thomas Munro wrote: > > On Thu, Apr 4, 2024 at 6:03 AM Melanie Plageman > wrote: > > On the topic of BAS_BULKREAD buffer access strategy, I think the least > > we could do is add an assert like this to read_stream_begin_relation()

Re: Use streaming read API in ANALYZE

2024-04-03 Thread Melanie Plageman
ot;storage/shm_toc.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -388,9 +389,8 @@ extern bool HeapTupleIsSurelyDead(HeapTuple htup, struct GlobalVisState *vistest); /* in heap/heapam_handler.c*/ -extern void heapam_scan_analyze_next_block(Tabl

Re: Opportunistically pruning page before update

2024-04-03 Thread Melanie Plageman
On Fri, Jan 26, 2024 at 8:33 PM James Coleman wrote: > > On Tue, Jan 23, 2024 at 2:46 AM Dilip Kumar wrote: > > > > On Tue, Jan 23, 2024 at 7:18 AM James Coleman wrote: > > > > > > On Mon, Jan 22, 2024 at 8:21 PM James Coleman wrote: > > > > > > > > See rebased patch attached. > > > > > > I

Re: Combine Prune and Freeze records emitted by vacuum

2024-04-03 Thread Melanie Plageman
On Wed, Apr 3, 2024 at 12:34 PM Heikki Linnakangas wrote: > > On 03/04/2024 17:18, Melanie Plageman wrote: > > I noticed you didn't make the comment updates I suggested in my > > version 13 here [1]. A few of them are outdated references to > > heap_page_prune() and one t

Re: Streaming read-ready sequential scan code

2024-04-03 Thread Melanie Plageman
On Tue, Apr 2, 2024 at 1:10 PM Heikki Linnakangas wrote: > > On 01/04/2024 22:58, Melanie Plageman wrote: > > Attached v7 has version 14 of the streaming read API as well as a few > > small tweaks to comments and code. > > I saw benchmarks in this thread to show that th

Re: Combine Prune and Freeze records emitted by vacuum

2024-04-03 Thread Melanie Plageman
On Wed, Apr 3, 2024 at 8:39 AM Heikki Linnakangas wrote: > > On 02/04/2024 16:11, Heikki Linnakangas wrote: > > On 01/04/2024 20:22, Melanie Plageman wrote: > >> Review for 0003-0006 (I didn't have any new thoughts on 0002). I know > >> you didn't modify them m

Re: Streaming I/O, vectored I/O (WIP)

2024-04-03 Thread Melanie Plageman
On Tue, Apr 2, 2024 at 8:32 PM Thomas Munro wrote: > > Here are the remaining patches discussed in this thread. They give > tablespace-specific io_combine_limit, effective_io_readahead_window > (is this useful?), and up-to-1MB io_combine_limit (is this useful?). > I think the first two would

Re: Combine Prune and Freeze records emitted by vacuum

2024-04-02 Thread Melanie Plageman
On Tue, Apr 02, 2024 at 01:24:27PM -0400, Melanie Plageman wrote: > On Tue, Apr 2, 2024 at 9:11 AM Heikki Linnakangas wrote: > > > > On 01/04/2024 20:22, Melanie Plageman wrote: > > > Review for 0003-0006 (I didn't have any new thoughts on 0002). I know > > > y

Re: Combine Prune and Freeze records emitted by vacuum

2024-04-02 Thread Melanie Plageman
On Tue, Apr 2, 2024 at 9:11 AM Heikki Linnakangas wrote: > > On 01/04/2024 20:22, Melanie Plageman wrote: > > Review for 0003-0006 (I didn't have any new thoughts on 0002). I know > > you didn't modify them much/at all, but I noticed some things in my code > > that could b

Re: Streaming read-ready sequential scan code

2024-04-01 Thread Melanie Plageman
On Wed, Mar 27, 2024 at 08:47:03PM -0400, Melanie Plageman wrote: > On Fri, Mar 8, 2024 at 4:56 PM Melanie Plageman > wrote: > > > > On Sat, Mar 02, 2024 at 06:07:48PM -0500, Melanie Plageman wrote: > > > On Wed, Feb 28, 2024 at 12:30 PM Melanie Plageman > > >

Re: Combine Prune and Freeze records emitted by vacuum

2024-04-01 Thread Melanie Plageman
On Mon, Apr 1, 2024 at 1:37 PM Heikki Linnakangas wrote: > > Committed the first of the remaining patches with those changes. And > also this, which is worth calling out: > > > if (prstate.htsv[offnum] == HEAPTUPLE_DEAD) > > { > > ItemId

Re: Combine Prune and Freeze records emitted by vacuum

2024-04-01 Thread Melanie Plageman
On Mon, Apr 01, 2024 at 05:17:51PM +0300, Heikki Linnakangas wrote: > On 30/03/2024 07:57, Melanie Plageman wrote: > > > The final state of the code could definitely use more cleanup. I've been > > staring at it for awhile, so I could use some thoughts/ideas about what

Re: Combine Prune and Freeze records emitted by vacuum

2024-04-01 Thread Melanie Plageman
On Mon, Apr 01, 2024 at 05:17:51PM +0300, Heikki Linnakangas wrote: > On 30/03/2024 07:57, Melanie Plageman wrote: > > On Fri, Mar 29, 2024 at 12:32:21PM -0400, Melanie Plageman wrote: > > > On Fri, Mar 29, 2024 at 12:00 PM Heikki Linnakangas > > > wrote:

Re: Combine Prune and Freeze records emitted by vacuum

2024-03-30 Thread Melanie Plageman
On Sat, Mar 30, 2024 at 8:00 AM Robert Haas wrote: > > On Sat, Mar 30, 2024 at 1:57 AM Melanie Plageman > wrote: > > I think that we are actually successfully removing more RECENTLY_DEAD > > HOT tuples than in master with heap_page_prune()'s new approach, and I > > t

Re: Combine Prune and Freeze records emitted by vacuum

2024-03-29 Thread Melanie Plageman
On Fri, Mar 29, 2024 at 12:00 PM Heikki Linnakangas wrote: > > On 29/03/2024 07:04, Melanie Plageman wrote: > > On Thu, Mar 28, 2024 at 11:07:10AM -0400, Melanie Plageman wrote: > >> These comments could use another pass. I had added some extra > >> (probably redun

Re: Combine Prune and Freeze records emitted by vacuum

2024-03-28 Thread Melanie Plageman
On Thu, Mar 28, 2024 at 4:49 AM Heikki Linnakangas wrote: > > On 28/03/2024 03:53, Melanie Plageman wrote: > > On Thu, Mar 28, 2024 at 01:04:04AM +0200, Heikki Linnakangas wrote: > >> One change with this is that live_tuples and many of the other fields are > &g

Re: Combine Prune and Freeze records emitted by vacuum

2024-03-27 Thread Melanie Plageman
On Thu, Mar 28, 2024 at 01:04:04AM +0200, Heikki Linnakangas wrote: > On 27/03/2024 20:26, Melanie Plageman wrote: > > On Wed, Mar 27, 2024 at 12:18 PM Heikki Linnakangas wrote: > > > > > > On 27/03/2024 17:18, Melanie Plageman wrote: > > > > I n

Re: Streaming read-ready sequential scan code

2024-03-27 Thread Melanie Plageman
On Fri, Mar 8, 2024 at 4:56 PM Melanie Plageman wrote: > > On Sat, Mar 02, 2024 at 06:07:48PM -0500, Melanie Plageman wrote: > > On Wed, Feb 28, 2024 at 12:30 PM Melanie Plageman > > wrote: > > > > > > On Mon, Feb 26, 2024 at 03:56:57PM -0500, Melanie Plageman

Re: Streaming I/O, vectored I/O (WIP)

2024-03-27 Thread Melanie Plageman
On Wed, Mar 27, 2024 at 10:11 AM Thomas Munro wrote: > > I got rid of "finished" (now represented by distance == 0, I was > removing branches and variables). I got rid of "started", which can > now be deduced (used for suppressing advice when you're calling > _next() because you need a block and

Re: Use streaming read API in ANALYZE

2024-03-27 Thread Melanie Plageman
On Tue, Mar 26, 2024 at 02:51:27PM +0300, Nazir Bilal Yavuz wrote: > Hi, > > On Wed, 28 Feb 2024 at 14:42, Nazir Bilal Yavuz wrote: > > > > > > The new version of the streaming read API [1] is posted. I updated the > > streaming read API changes patch (0001), using the streaming read API > > in

Re: Combine Prune and Freeze records emitted by vacuum

2024-03-27 Thread Melanie Plageman
On Wed, Mar 27, 2024 at 2:26 PM Melanie Plageman wrote: > > On Wed, Mar 27, 2024 at 12:18 PM Heikki Linnakangas wrote: > > > > On 27/03/2024 17:18, Melanie Plageman wrote: > > > I need some way to modify the control flow or accounting such that I > > > know

Re: Combine Prune and Freeze records emitted by vacuum

2024-03-27 Thread Melanie Plageman
On Wed, Mar 27, 2024 at 12:18 PM Heikki Linnakangas wrote: > > On 27/03/2024 17:18, Melanie Plageman wrote: > > I need some way to modify the control flow or accounting such that I > > know which HEAPTUPLE_RECENTLY_DEAD tuples will not be marked LP_DEAD. > > And a way to

Re: Combine Prune and Freeze records emitted by vacuum

2024-03-27 Thread Melanie Plageman
On Tue, Mar 26, 2024 at 5:46 PM Melanie Plageman wrote: > > On Mon, Mar 25, 2024 at 09:33:38PM +0200, Heikki Linnakangas wrote: > > On 24/03/2024 18:32, Melanie Plageman wrote: > > > On Thu, Mar 21, 2024 at 9:28 AM Heikki Linnakangas > > > wrote: > > >

Re: Combine Prune and Freeze records emitted by vacuum

2024-03-25 Thread Melanie Plageman
Thanks for committing the new WAL format! On Mon, Mar 25, 2024 at 3:33 PM Heikki Linnakangas wrote: > > On 24/03/2024 18:32, Melanie Plageman wrote: > > On Thu, Mar 21, 2024 at 9:28 AM Heikki Linnakangas wrote: > >> > >> In heap_page_prune_and_freeze(), we now d

Re: Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE

2024-03-25 Thread Melanie Plageman
On Mon, Mar 25, 2024 at 2:29 AM Donghang Lin wrote: > > > > On Sat, Feb 17, 2024 at 2:31 PM Tomas Vondra > > wrote: > > 2) Leader vs. worker counters > > > > It seems to me this does nothing to add the per-worker values from "Heap > > Blocks" into the leader, which means we get stuff like this:

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-24 Thread Melanie Plageman
On Sun, Mar 24, 2024 at 5:59 PM Tomas Vondra wrote: > > BTW when you say "up to 'Make table_scan_bitmap_next_block() async > friendly'" do you mean including that patch, or that this is the first > patch that is not one of the independently useful patches. I think the code is easier to

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-24 Thread Melanie Plageman
On Sun, Mar 24, 2024 at 2:22 PM Tomas Vondra wrote: > > On 3/24/24 18:38, Melanie Plageman wrote: > > I haven't had a chance yet to reproduce the regressions you saw in the > > streaming read user patch or to look closely at the performance results. > > So you tried to rep

Re: Combine Prune and Freeze records emitted by vacuum

2024-03-24 Thread Melanie Plageman
On Sat, Mar 23, 2024 at 01:09:30AM +0200, Heikki Linnakangas wrote: > On 20/03/2024 21:17, Melanie Plageman wrote: > > Attached patch changes-for-0001.patch has a bunch of updated comments -- > > especially for heapam_xlog.h (including my promised diagram). And a few > >

Re: Streaming I/O, vectored I/O (WIP)

2024-03-24 Thread Melanie Plageman
On Sun, Mar 24, 2024 at 9:02 AM Thomas Munro wrote: > > On Wed, Mar 20, 2024 at 4:04 AM Heikki Linnakangas wrote: > > On 12/03/2024 15:02, Thomas Munro wrote: > > > src/backend/storage/aio/streaming_read.c > > > src/include/storage/streaming_read.h > > > > Standard file header comments missing.

Re: Combine Prune and Freeze records emitted by vacuum

2024-03-24 Thread Melanie Plageman
On Thu, Mar 21, 2024 at 9:28 AM Heikki Linnakangas wrote: > > In heap_page_prune_and_freeze(), we now do some extra work on each live > tuple, to set the all_visible_except_removable correctly. And also to > update live_tuples, recently_dead_tuples and hastup. When we're not > freezing, that's a

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-22 Thread Melanie Plageman
On Tue, Mar 19, 2024 at 02:33:35PM +0200, Heikki Linnakangas wrote: > On 18/03/2024 17:19, Melanie Plageman wrote: > > I've attached v7 rebased over this commit. > > If we delayed table_beginscan_bm() call further, after starting the TBM > iterator, we could skip it altogether

Re: Combine Prune and Freeze records emitted by vacuum

2024-03-20 Thread Melanie Plageman
On Wed, Mar 20, 2024 at 03:15:49PM +0200, Heikki Linnakangas wrote: > > 0009-: The rest of the v4 patches, rebased over the WAL format changes. I > also added a few small commits for little cleanups that caught my eye, let > me know if you disagree with those. This review is just of the patches

Re: Combine Prune and Freeze records emitted by vacuum

2024-03-20 Thread Melanie Plageman
On Wed, Mar 20, 2024 at 4:04 PM Peter Geoghegan wrote: > > On Wed, Mar 20, 2024 at 9:15 AM Heikki Linnakangas wrote: > > > I made it its own sub-record (xlhp_conflict_horizon) less to help with > > > alignment (though we can use all the help we can get there) and more to > > > keep it from

Re: Test 031_recovery_conflict.pl is not immune to autovacuum

2024-03-20 Thread Melanie Plageman
On Wed, Mar 20, 2024 at 10:00 AM Alexander Lakhin wrote: > > Hello Melanie, > > 20.03.2024 16:15, Melanie Plageman wrote: > > Seems like we could just add autovacuum_enabled=false to the table like > > this: > > diff --git a/src/test/recovery/t/031_recovery_conflic

Re: Combine Prune and Freeze records emitted by vacuum

2024-03-20 Thread Melanie Plageman
On Wed, Mar 20, 2024 at 03:15:49PM +0200, Heikki Linnakangas wrote: > On 20/03/2024 03:36, Melanie Plageman wrote: > > On Mon, Mar 18, 2024 at 01:15:21AM +0200, Heikki Linnakangas wrote: > > > On 15/03/2024 02:56, Melanie Plageman wrote: > > > > Okay, so I was goin

Re: Test 031_recovery_conflict.pl is not immune to autovacuum

2024-03-20 Thread Melanie Plageman
On Wed, Mar 20, 2024 at 9:00 AM Alexander Lakhin wrote: > > Hello hackers, > > Among many recoveryCheck (more concretely, 027_stream_regress) failures > occurred on a number of buildfarm animals after switching to meson, which > can be explained by timeouts, I saw a different failure on adder: >

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-18 Thread Melanie Plageman
On Sun, Mar 17, 2024 at 3:21 PM Tomas Vondra wrote: > > On 3/14/24 22:39, Melanie Plageman wrote: > > On Thu, Mar 14, 2024 at 5:26 PM Tomas Vondra > > wrote: > >> > >> On 3/14/24 19:16, Melanie Plageman wrote: > >>> On Thu, Mar 14, 20

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-15 Thread Melanie Plageman
On Fri, Mar 15, 2024 at 5:14 PM Andres Freund wrote: > > Hi, > > On 2024-03-14 17:39:30 -0400, Melanie Plageman wrote: > > I will soon send out a summary of what we investigated off-list about > > 0010 (though we didn't end up concluding anyt

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-14 Thread Melanie Plageman
On Thu, Mar 14, 2024 at 5:26 PM Tomas Vondra wrote: > > On 3/14/24 19:16, Melanie Plageman wrote: > > On Thu, Mar 14, 2024 at 03:32:04PM +0200, Heikki Linnakangas wrote: > >> ... > >> > >> Ok, committed that for now. Thanks for looking! > > > > A

Re: Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE

2024-03-14 Thread Melanie Plageman
On Thu, Mar 14, 2024 at 05:30:30PM +0200, Heikki Linnakangas wrote: > On 18/02/2024 00:31, Tomas Vondra wrote: > > Do you plan to work continue working on this patch? I did take a look, > > and on the whole it looks reasonable - it modifies the right places etc. > > +1 > > > I think there are

Re: Confine vacuum skip logic to lazy_scan_skip

2024-03-11 Thread Melanie Plageman
On Sun, Mar 10, 2024 at 11:01 PM Thomas Munro wrote: > > On Mon, Mar 11, 2024 at 5:31 AM Melanie Plageman > wrote: > > I have investigated the interaction between > > maintenance_io_concurrency, streaming reads, and the vacuum buffer > > access strategy (BAS_VACUUM).

Re: Confine vacuum skip logic to lazy_scan_skip

2024-03-11 Thread Melanie Plageman
On Mon, Mar 11, 2024 at 2:47 PM Heikki Linnakangas wrote: > > > > Otherwise LGTM > > Ok, pushed! Thank you, this is much more understandable now! Cool, thanks!

Re: Confine vacuum skip logic to lazy_scan_skip

2024-03-11 Thread Melanie Plageman
adability inside heap_vac_scan_next_block(), move the logic of > finding the next unskippable block to separate function, and add some > comments. > > This refactoring will also help future patches to switch to using a > streaming read interface, and eventually AIO > (https://pos

Re: Confine vacuum skip logic to lazy_scan_skip

2024-03-10 Thread Melanie Plageman
On Wed, Mar 6, 2024 at 6:47 PM Melanie Plageman wrote: > > Performance results: > > The TL;DR of my performance results is that streaming read vacuum is > faster. However there is an issue with the interaction of the streaming > read code and the vacuum buffer access s

Re: Combine Prune and Freeze records emitted by vacuum

2024-03-09 Thread Melanie Plageman
Thanks so much for the review! On Wed, Mar 6, 2024 at 7:59 AM Heikki Linnakangas wrote: > > On 25/01/2024 00:49, Melanie Plageman wrote: > > > The attached patch set is broken up into many separate commits for > > ease of review. Each patch does a single thing which can be

Re: Streaming read-ready sequential scan code

2024-03-08 Thread Melanie Plageman
On Sat, Mar 02, 2024 at 06:07:48PM -0500, Melanie Plageman wrote: > On Wed, Feb 28, 2024 at 12:30 PM Melanie Plageman > wrote: > > > > On Mon, Feb 26, 2024 at 03:56:57PM -0500, Melanie Plageman wrote: > > > On Mon, Feb 19, 2024 at 6:05 PM Melanie Plageman > > >

Re: Confine vacuum skip logic to lazy_scan_skip

2024-03-08 Thread Melanie Plageman
On Fri, Mar 8, 2024 at 12:34 PM Melanie Plageman wrote: > > On Fri, Mar 08, 2024 at 06:07:33PM +0200, Heikki Linnakangas wrote: > > On 08/03/2024 02:46, Melanie Plageman wrote: > > > On Wed, Mar 06, 2024 at 10:00:23PM -0500, Melanie Plageman wrote: > > > > On Wed,

Re: Confine vacuum skip logic to lazy_scan_skip

2024-03-08 Thread Melanie Plageman
On Fri, Mar 08, 2024 at 06:07:33PM +0200, Heikki Linnakangas wrote: > On 08/03/2024 02:46, Melanie Plageman wrote: > > On Wed, Mar 06, 2024 at 10:00:23PM -0500, Melanie Plageman wrote: > > > On Wed, Mar 06, 2024 at 09:55:21PM +0200, Heikki Linnakangas wrote: > >

Re: Confine vacuum skip logic to lazy_scan_skip

2024-03-08 Thread Melanie Plageman
On Fri, Mar 8, 2024 at 11:31 AM Melanie Plageman wrote: > > On Fri, Mar 8, 2024 at 11:00 AM Peter Geoghegan wrote: > > > > On Fri, Mar 8, 2024 at 10:48 AM Melanie Plageman > > wrote: > > > Not that it will be fun to maintain another special case in the VM >

Re: Confine vacuum skip logic to lazy_scan_skip

2024-03-08 Thread Melanie Plageman
On Fri, Mar 8, 2024 at 11:00 AM Peter Geoghegan wrote: > > On Fri, Mar 8, 2024 at 10:48 AM Melanie Plageman > wrote: > > Not that it will be fun to maintain another special case in the VM > > update code in lazy_scan_prune(), but we could have a special

Re: Confine vacuum skip logic to lazy_scan_skip

2024-03-08 Thread Melanie Plageman
On Fri, Mar 8, 2024 at 10:41 AM Peter Geoghegan wrote: > > On Fri, Mar 8, 2024 at 8:49 AM Heikki Linnakangas wrote: > > ISTM we should revert the above hunk, and backpatch it to v16. I'm a > > little wary because I don't understand why that change was made in the > > first place, though. I think

Re: Confine vacuum skip logic to lazy_scan_skip

2024-03-08 Thread Melanie Plageman
On Fri, Mar 8, 2024 at 8:49 AM Heikki Linnakangas wrote: > > On 08/03/2024 02:46, Melanie Plageman wrote: > > On Wed, Mar 06, 2024 at 10:00:23PM -0500, Melanie Plageman wrote: > >>> I feel heap_vac_scan_get_next_block() function could use some love. Maybe > >>>

Re: Confine vacuum skip logic to lazy_scan_skip

2024-03-07 Thread Melanie Plageman
On Wed, Mar 06, 2024 at 10:00:23PM -0500, Melanie Plageman wrote: > On Wed, Mar 06, 2024 at 09:55:21PM +0200, Heikki Linnakangas wrote: > > I made some further changes. I kept them as separate commits for easier > > review, see the commit messages for details. Any thoughts o

Re: Confine vacuum skip logic to lazy_scan_skip

2024-03-06 Thread Melanie Plageman
On Wed, Mar 06, 2024 at 09:55:21PM +0200, Heikki Linnakangas wrote: > On 27/02/2024 21:47, Melanie Plageman wrote: > > The attached v5 has some simplifications when compared to v4 but takes > > largely the same approach. > > > > 0001-0004 are refactoring > > I'

Re: Confine vacuum skip logic to lazy_scan_skip

2024-03-06 Thread Melanie Plageman
On Tue, Feb 27, 2024 at 02:47:03PM -0500, Melanie Plageman wrote: > On Mon, Jan 29, 2024 at 8:18 PM Melanie Plageman > wrote: > > > > On Fri, Jan 26, 2024 at 8:28 AM vignesh C wrote: > > > > > > CFBot shows that the patch does not apply anymore as in [1]: >

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-02 Thread Melanie Plageman
On Sat, Mar 2, 2024 at 6:59 PM Tomas Vondra wrote: > > > > On 3/1/24 17:51, Melanie Plageman wrote: > > On Fri, Mar 1, 2024 at 9:05 AM Tomas Vondra > > wrote: > >> > >> On 3/1/24 02:18, Melanie Plageman wrote: > >>> On Thu, Feb 29, 2024 at 6:

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-02 Thread Melanie Plageman
On Sat, Mar 2, 2024 at 5:51 PM Tomas Vondra wrote: > > On 3/2/24 23:11, Melanie Plageman wrote: > > On Fri, Mar 1, 2024 at 2:31 PM Melanie Plageman > > wrote: > >> > >> ... > >> > >> Hold the phone on this one. I realiz

Re: Streaming read-ready sequential scan code

2024-03-02 Thread Melanie Plageman
On Wed, Feb 28, 2024 at 12:30 PM Melanie Plageman wrote: > > On Mon, Feb 26, 2024 at 03:56:57PM -0500, Melanie Plageman wrote: > > On Mon, Feb 19, 2024 at 6:05 PM Melanie Plageman > > wrote: > > > > > > On Mon, Jan 29, 2024 at 4

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-02 Thread Melanie Plageman
On Sat, Mar 2, 2024 at 5:41 PM Tomas Vondra wrote: > > > > On 3/2/24 23:28, Melanie Plageman wrote: > > On Sat, Mar 2, 2024 at 10:05 AM Tomas Vondra > > wrote: > >> > >> Here's a PDF with charts for a dataset where the row selectivity is more &g

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-02 Thread Melanie Plageman
On Sat, Mar 2, 2024 at 10:05 AM Tomas Vondra wrote: > > Here's a PDF with charts for a dataset where the row selectivity is more > correlated to selectivity of pages. I'm attaching the updated script, > with the SQL generating the data set. But the short story is all rows on > a single page have

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-02 Thread Melanie Plageman
On Fri, Mar 1, 2024 at 2:31 PM Melanie Plageman wrote: > > On Thu, Feb 29, 2024 at 7:29 PM Melanie Plageman > wrote: > > > > On Thu, Feb 29, 2024 at 5:44 PM Tomas Vondra > > wrote: > > > > > > > > > > > > On 2/29/24 22:19, Melani

Re: CF entries for 17 to be reviewed

2024-03-02 Thread Melanie Plageman
On Sat, Mar 2, 2024 at 1:32 PM Andrey M. Borodin wrote: > > Hi hackers! > > In this thread, I want to promote entries from CommitFest that require > review. I have scanned through the bugs, clients, and documentation sections, > and here is my take on the current situation. All of these threads

Re: Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE

2024-03-01 Thread Melanie Plageman
On Sat, Feb 17, 2024 at 5:31 PM Tomas Vondra wrote: > > Hi David, > > Do you plan to work continue working on this patch? I did take a look, > and on the whole it looks reasonable - it modifies the right places etc. I haven't started reviewing this patch yet, but I just ran into the behavior

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-01 Thread Melanie Plageman
On Thu, Feb 29, 2024 at 7:29 PM Melanie Plageman wrote: > > On Thu, Feb 29, 2024 at 5:44 PM Tomas Vondra > wrote: > > > > > > > > On 2/29/24 22:19, Melanie Plageman wrote: > > > On Thu, Feb 29, 2024 at 7:54 AM Tomas Vondra > > > wrote: &g

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-01 Thread Melanie Plageman
On Fri, Mar 1, 2024 at 9:05 AM Tomas Vondra wrote: > > On 3/1/24 02:18, Melanie Plageman wrote: > > On Thu, Feb 29, 2024 at 6:44 PM Tomas Vondra > > wrote: > >> > >> On 2/29/24 23:44, Tomas Vondra wrote: > >> 1) On master there's c

  1   2   3   4   5   6   7   >