Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-05-17 Thread Jonathan S. Katz
On 5/17/23 3:19 PM, Andres Freund wrote: Hi, On 2023-05-16 10:30:27 -0400, Jonathan S. Katz wrote: On 5/6/23 1:30 PM, Melanie Plageman wrote: I've done that in the attached v5. [RMT hat] RMT nudge on this thread, as we're approaching the Beta 1 cutoff. From the above discussion, it sounds

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-05-17 Thread Andres Freund
Hi, On 2023-05-16 10:30:27 -0400, Jonathan S. Katz wrote: > On 5/6/23 1:30 PM, Melanie Plageman wrote: > > > I've done that in the attached v5. > > [RMT hat] > > RMT nudge on this thread, as we're approaching the Beta 1 cutoff. From the > above discussion, it sounds like it's pretty close to be

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-05-16 Thread Jonathan S. Katz
On 5/6/23 1:30 PM, Melanie Plageman wrote: I've done that in the attached v5. [RMT hat] RMT nudge on this thread, as we're approaching the Beta 1 cutoff. From the above discussion, it sounds like it's pretty close to being ready. Thanks, Jonathan OpenPGP_signature Description: OpenPGP d

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-05-06 Thread Melanie Plageman
v5 attached. On Thu, May 4, 2023 at 12:44 PM Andres Freund wrote: > On 2023-04-27 11:36:49 -0400, Melanie Plageman wrote: > > > > /* and finally tell the kernel to write the data to > > > > storage */ > > > > reln = smgropen(currlocator, InvalidBackendId); > > > >

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-05-04 Thread Jonathan S. Katz
On 5/4/23 12:46 PM, Andres Freund wrote: Hi, On 2023-05-03 11:36:10 -0400, Jonathan S. Katz wrote: It'd be good if we can get this into Beta 1 if everyone is comfortable with the patch. I think we need one more iteration, then I think it can be committed. The changes are docs phrasing and pol

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-05-04 Thread Andres Freund
Hi, On 2023-04-24 21:29:48 -0400, Melanie Plageman wrote: > 1) Does it make sense for writebacks to count the number of blocks for > which writeback was requested or the number of calls to smgrwriteback() or > the number of actual syscalls made? We don't actually know from outside > of mdwriteback

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-05-04 Thread Andres Freund
Hi, On 2023-05-03 11:36:10 -0400, Jonathan S. Katz wrote: > It'd be good if we can get this into Beta 1 if everyone is comfortable with > the patch. I think we need one more iteration, then I think it can be committed. The changes are docs phrasing and polishing the API a bit, which shouldn't be

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-05-04 Thread Andres Freund
Hi, On 2023-04-27 11:36:49 -0400, Melanie Plageman wrote: > > > /* and finally tell the kernel to write the data to storage > > > */ > > > reln = smgropen(currlocator, InvalidBackendId); > > > smgrwriteback(reln, BufTagGetForkNum(&tag), tag.blockNum, > >

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-05-03 Thread Jonathan S. Katz
On 4/27/23 11:36 AM, Melanie Plageman wrote: Thanks for the review! On Wed, Apr 26, 2023 at 10:22 PM Kyotaro Horiguchi wrote: At Wed, 26 Apr 2023 17:08:14 -0400, Melanie Plageman wrote in On Mon, Apr 24, 2023 at 9:29 PM Melanie Plageman wrote: I've yet to cook up a client backend test ca

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-04-27 Thread Melanie Plageman
Thanks for the review! On Wed, Apr 26, 2023 at 10:22 PM Kyotaro Horiguchi wrote: > > At Wed, 26 Apr 2023 17:08:14 -0400, Melanie Plageman > wrote in > > On Mon, Apr 24, 2023 at 9:29 PM Melanie Plageman > > wrote: > > > I've yet to cook up a client backend test case (e.g. with COPY). I've > >

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-04-26 Thread Kyotaro Horiguchi
At Wed, 26 Apr 2023 17:08:14 -0400, Melanie Plageman wrote in > On Mon, Apr 24, 2023 at 9:29 PM Melanie Plageman > wrote: > > I've yet to cook up a client backend test case (e.g. with COPY). I've taken > > that as a todo. > > It was trivial to see client backend writebacks in almost any scenar

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-04-26 Thread Melanie Plageman
On Mon, Apr 24, 2023 at 9:29 PM Melanie Plageman wrote: > I've yet to cook up a client backend test case (e.g. with COPY). I've taken > that as a todo. It was trivial to see client backend writebacks in almost any scenario once I set backend_flush_after. I wonder if it is worth mentioning the var

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-04-24 Thread Melanie Plageman
On Mon, Apr 24, 2023 at 7:02 PM Melanie Plageman wrote: > On Mon, Apr 24, 2023 at 03:56:54PM -0700, Andres Freund wrote: > > > > I was thinking we'd track writeback separately from the write, rather > than > > attributing the writeback to "write". Otherwise it looks good, based on > a > > quick

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-04-24 Thread Jonathan S. Katz
On 4/24/23 6:14 PM, Andres Freund wrote: Hi, On 2023-04-24 10:52:15 +0530, Amit Kapila wrote: On Sun, Apr 23, 2023 at 12:55 AM Jonathan S. Katz wrote: I wonder if it's worth doing so for 16? It'd give a more complete picture that way. The counter-argument I see is that we didn't track the

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-04-24 Thread Melanie Plageman
On Mon, Apr 24, 2023 at 03:56:54PM -0700, Andres Freund wrote: > Hi, > > On 2023-04-24 18:36:24 -0400, Melanie Plageman wrote: > > On Mon, Apr 24, 2023 at 6:13 PM Andres Freund wrote: > > > > Also, it seems like this (given the current code) is only reachable for > > > > permanent relations (i.e.

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-04-24 Thread Andres Freund
Hi, On 2023-04-24 18:36:24 -0400, Melanie Plageman wrote: > On Mon, Apr 24, 2023 at 6:13 PM Andres Freund wrote: > > > Also, it seems like this (given the current code) is only reachable for > > > permanent relations (i.e. not for IO object temp relation). If other > > backend > > > types than ch

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-04-24 Thread Melanie Plageman
On Mon, Apr 24, 2023 at 6:13 PM Andres Freund wrote: > Hi, > > On 2023-04-24 17:37:48 -0400, Melanie Plageman wrote: > > On Mon, Apr 24, 2023 at 02:14:32PM -0700, Andres Freund wrote: > > > It starts blocking once "enough" IO is in flight. For things like an > immediate > > > checkpoint, that can

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-04-24 Thread Andres Freund
Hi, On 2023-04-24 10:52:15 +0530, Amit Kapila wrote: > On Sun, Apr 23, 2023 at 12:55 AM Jonathan S. Katz > wrote: > > > I wonder if it's > > > worth doing so for 16? It'd give a more complete picture that way. The > > > counter-argument I see is that we didn't track the time for it in existing >

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-04-24 Thread Andres Freund
Hi, On 2023-04-24 17:37:48 -0400, Melanie Plageman wrote: > On Mon, Apr 24, 2023 at 02:14:32PM -0700, Andres Freund wrote: > > It starts blocking once "enough" IO is in flight. For things like an > > immediate > > checkpoint, that can happen fairly quickly, unless you have a very fast IO > > subs

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-04-24 Thread Melanie Plageman
On Mon, Apr 24, 2023 at 02:14:32PM -0700, Andres Freund wrote: > Hi, > > On 2023-04-24 16:39:36 -0400, Melanie Plageman wrote: > > On Wed, Apr 19, 2023 at 10:23:26AM -0700, Andres Freund wrote: > > > Hi, > > > > > > I noticed that the numbers in pg_stat_io dont't quite add up to what I > > > expe

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-04-24 Thread Andres Freund
Hi, On 2023-04-24 16:39:36 -0400, Melanie Plageman wrote: > On Wed, Apr 19, 2023 at 10:23:26AM -0700, Andres Freund wrote: > > Hi, > > > > I noticed that the numbers in pg_stat_io dont't quite add up to what I > > expected in write heavy workloads. Particularly for checkpointer, the > > numbers

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-04-24 Thread Melanie Plageman
On Wed, Apr 19, 2023 at 10:23:26AM -0700, Andres Freund wrote: > Hi, > > I noticed that the numbers in pg_stat_io dont't quite add up to what I > expected in write heavy workloads. Particularly for checkpointer, the numbers > for "write" in log_checkpoints output are larger than what is visible in

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-04-23 Thread Amit Kapila
On Sun, Apr 23, 2023 at 12:55 AM Jonathan S. Katz wrote: > > On 4/19/23 1:23 PM, Andres Freund wrote: > > Hi, > > > > I noticed that the numbers in pg_stat_io dont't quite add up to what I > > expected in write heavy workloads. Particularly for checkpointer, the > > numbers > > for "write" in log

Re: pg_stat_io not tracking smgrwriteback() is confusing

2023-04-22 Thread Jonathan S. Katz
On 4/19/23 1:23 PM, Andres Freund wrote: Hi, I noticed that the numbers in pg_stat_io dont't quite add up to what I expected in write heavy workloads. Particularly for checkpointer, the numbers for "write" in log_checkpoints output are larger than what is visible in pg_stat_io. That partially i

pg_stat_io not tracking smgrwriteback() is confusing

2023-04-19 Thread Andres Freund
Hi, I noticed that the numbers in pg_stat_io dont't quite add up to what I expected in write heavy workloads. Particularly for checkpointer, the numbers for "write" in log_checkpoints output are larger than what is visible in pg_stat_io. That partially is because log_checkpoints' "write" covers w