On Wed, Apr 1, 2020 at 5:01 PM Amit Kapila wrote:
>
> On Wed, Apr 1, 2020 at 4:29 PM Amit Kapila wrote:
> >
> > v9-0005-Keep-track-of-WAL-usage-in-pg_stat_statements
> >
>
> One more comment related to this patch.
> +
> + snprintf(buf, sizeof buf, UINT64_FORMAT, tmp.wal_bytes);
> +
> + /*
On Wed, Apr 1, 2020 at 12:01 PM Amit Kapila wrote:
>
> On Wed, Apr 1, 2020 at 8:51 AM Dilip Kumar wrote:
> >
> > > Agreed. I've attached the updated patch.
> > >
> > > Thank you for testing, Dilip!
> >
> > Thanks! One hunk is failing on the latest head. And, I have rebased
> > the patch for my
On Wed, Apr 1, 2020 at 4:29 PM Amit Kapila wrote:
>
> v9-0005-Keep-track-of-WAL-usage-in-pg_stat_statements
>
One more comment related to this patch.
+
+ snprintf(buf, sizeof buf, UINT64_FORMAT, tmp.wal_bytes);
+
+ /* Convert to numeric. */
+ wal_bytes = DirectFunctionCall3(numeric_in,
+
On Wed, Apr 1, 2020 at 1:32 PM Julien Rouhaud wrote:
>
> So here's a v9, rebased on top of the latest versions of Sawada-san's bug
> fixes
> (Amit's v6 for vacuum and Sawada-san's v2 for create index), with all
> previously mentionned changes.
>
Few other comments:
So here's a v9, rebased on top of the latest versions of Sawada-san's bug fixes
(Amit's v6 for vacuum and Sawada-san's v2 for create index), with all
previously mentionned changes.
Note that I'm only attaching those patches for convenience and to make sure
that cfbot is happy.
>From
On Wed, Apr 1, 2020 at 8:51 AM Dilip Kumar wrote:
>
> On Wed, Apr 1, 2020 at 8:26 AM Masahiko Sawada
> wrote:
> >
> > On Wed, 1 Apr 2020 at 11:46, Amit Kapila wrote:
> > >
> > > On Tue, Mar 31, 2020 at 7:32 PM Dilip Kumar wrote:
> > > >
> > > > While testing I have found one issue. Basically,
On Wed, Apr 1, 2020 at 8:51 AM Dilip Kumar wrote:
>
> > Agreed. I've attached the updated patch.
> >
> > Thank you for testing, Dilip!
>
> Thanks! One hunk is failing on the latest head. And, I have rebased
> the patch for my testing so posting the same. I have done some more
> testing to test
On Wed, Apr 1, 2020 at 8:26 AM Masahiko Sawada
wrote:
>
> On Wed, 1 Apr 2020 at 11:46, Amit Kapila wrote:
> >
> > On Tue, Mar 31, 2020 at 7:32 PM Dilip Kumar wrote:
> > >
> > > While testing I have found one issue. Basically, during a parallel
> > > vacuum, it was showing more number of
> > >
On Wed, 1 Apr 2020 at 11:46, Amit Kapila wrote:
>
> On Tue, Mar 31, 2020 at 7:32 PM Dilip Kumar wrote:
> >
> > While testing I have found one issue. Basically, during a parallel
> > vacuum, it was showing more number of
> > shared_blk_hits+shared_blks_read. After, some investigation, I found
>
On Wed, Apr 1, 2020 at 8:16 AM Amit Kapila wrote:
>
> On Tue, Mar 31, 2020 at 7:32 PM Dilip Kumar wrote:
> >
> > While testing I have found one issue. Basically, during a parallel
> > vacuum, it was showing more number of
> > shared_blk_hits+shared_blks_read. After, some investigation, I found
On Tue, Mar 31, 2020 at 7:32 PM Dilip Kumar wrote:
>
> While testing I have found one issue. Basically, during a parallel
> vacuum, it was showing more number of
> shared_blk_hits+shared_blks_read. After, some investigation, I found
> that during the cleanup phase nworkers are -1, and because
On Tue, Mar 31, 2020 at 7:39 PM Julien Rouhaud wrote:
>
> On Tue, Mar 31, 2020 at 12:21 PM Kuntal Ghosh
> wrote:
> >
> > On Mon, Mar 30, 2020 at 6:14 PM Julien Rouhaud wrote:
> > >
> > @@ -448,6 +449,7 @@ XLogInsert(RmgrId rmid, uint8 info)
> > bool doPageWrites;
> > XLogRecPtr fpw_lsn;
> >
On Tue, Mar 31, 2020 at 12:21 PM Kuntal Ghosh
wrote:
>
> On Mon, Mar 30, 2020 at 6:14 PM Julien Rouhaud wrote:
> >
> @@ -448,6 +449,7 @@ XLogInsert(RmgrId rmid, uint8 info)
> bool doPageWrites;
> XLogRecPtr fpw_lsn;
> XLogRecData *rdt;
> + int num_fpw = 0;
>
> /*
> * Get values needed
On Tue, Mar 31, 2020 at 12:20 PM Dilip Kumar wrote:
>
> On Tue, Mar 31, 2020 at 10:44 AM Masahiko Sawada
> wrote:
> >
> > On Tue, 31 Mar 2020 at 12:58, Amit Kapila wrote:
> > >
> > > On Mon, Mar 30, 2020 at 12:31 PM Masahiko Sawada
> > > wrote:
> > > >
> > > > The patch for vacuum conflicts
On Tue, Mar 31, 2020 at 12:17 PM Amit Kapila wrote:
>
> It is possible that both of us are having different meanings for below
> two variables:
> +typedef struct WalUsage
> +{
> + long wal_records; /* # of WAL records produced */
> + long wal_fpw_records; /* # of full page write WAL records
> + *
On Tue, Mar 31, 2020 at 11:21 AM Dilip Kumar wrote:
>
> I have started reviewing this patch and I have some comments/questions.
Thanks a lot!
>
> 1.
> @@ -22,6 +22,10 @@ static BufferUsage save_pgBufferUsage;
>
> static void BufferUsageAdd(BufferUsage *dst, const BufferUsage *add);
>
>
On Mon, Mar 30, 2020 at 6:14 PM Julien Rouhaud wrote:
>
@@ -448,6 +449,7 @@ XLogInsert(RmgrId rmid, uint8 info)
bool doPageWrites;
XLogRecPtr fpw_lsn;
XLogRecData *rdt;
+ int num_fpw = 0;
/*
* Get values needed to decide whether to do full-page writes. Since
@@ -457,9 +459,9 @@
On Tue, Mar 31, 2020 at 2:39 PM Julien Rouhaud wrote:
>
> On Tue, Mar 31, 2020 at 8:53 AM Amit Kapila wrote:
> >
> > On Mon, Mar 30, 2020 at 6:14 PM Julien Rouhaud wrote:
> > >
> > > On Mon, Mar 30, 2020 at 03:52:38PM +0530, Amit Kapila wrote:
> > > >
> > > > I think the right place to compute
On Tue, Mar 31, 2020 at 2:51 PM Dilip Kumar wrote:
>
> 4. Currently, we are combining all full-page write
> force/normal/consistency checks in one category. I am not sure
> whether it will be good information to know how many are force_fpw and
> how many are normal_fpw?
>
We can do it if we
On Tue, Mar 31, 2020 at 12:23 PM Amit Kapila wrote:
>
> On Mon, Mar 30, 2020 at 6:14 PM Julien Rouhaud wrote:
> >
> > On Mon, Mar 30, 2020 at 03:52:38PM +0530, Amit Kapila wrote:
> > >
> > > I think the right place to compute this information is
> > > XLogRecordAssemble even though we update it
On Tue, Mar 31, 2020 at 8:53 AM Amit Kapila wrote:
>
> On Mon, Mar 30, 2020 at 6:14 PM Julien Rouhaud wrote:
> >
> > On Mon, Mar 30, 2020 at 03:52:38PM +0530, Amit Kapila wrote:
> > >
> > > I think the right place to compute this information is
> > > XLogRecordAssemble even though we update it
On Mon, Mar 30, 2020 at 6:14 PM Julien Rouhaud wrote:
>
> On Mon, Mar 30, 2020 at 03:52:38PM +0530, Amit Kapila wrote:
> >
> > I think the right place to compute this information is
> > XLogRecordAssemble even though we update it at the place where you
> > have it in the patch. You can probably
On Tue, Mar 31, 2020 at 10:44 AM Masahiko Sawada
wrote:
>
> On Tue, 31 Mar 2020 at 12:58, Amit Kapila wrote:
> >
> > On Mon, Mar 30, 2020 at 12:31 PM Masahiko Sawada
> > wrote:
> > >
> > > The patch for vacuum conflicts with recent changes in vacuum. So I've
> > > attached rebased one.
> > >
>
On Tue, 31 Mar 2020 at 12:58, Amit Kapila wrote:
>
> On Mon, Mar 30, 2020 at 12:31 PM Masahiko Sawada
> wrote:
> >
> > The patch for vacuum conflicts with recent changes in vacuum. So I've
> > attached rebased one.
> >
>
> + /*
> + * Next, accumulate buffer usage. (This must wait for the
On Mon, Mar 30, 2020 at 12:31 PM Masahiko Sawada
wrote:
>
> The patch for vacuum conflicts with recent changes in vacuum. So I've
> attached rebased one.
>
+ /*
+ * Next, accumulate buffer usage. (This must wait for the workers to
+ * finish, or we might get incomplete data.)
+ */
+ for (i = 0;
On Mon, Mar 30, 2020 at 03:52:38PM +0530, Amit Kapila wrote:
> On Sun, Mar 29, 2020 at 5:49 PM Julien Rouhaud wrote:
> >
>
> @@ -1249,6 +1250,16 @@ XLogInsertRecord(XLogRecData *rdata,
> ProcLastRecPtr = StartPos;
> XactLastRecEnd = EndPos;
>
> + /* Provide WAL update data to the
On Sun, Mar 29, 2020 at 5:49 PM Julien Rouhaud wrote:
>
@@ -1249,6 +1250,16 @@ XLogInsertRecord(XLogRecData *rdata,
ProcLastRecPtr = StartPos;
XactLastRecEnd = EndPos;
+ /* Provide WAL update data to the instrumentation */
+ if (inserted)
+ {
+ pgWalUsage.wal_bytes += rechdr->xl_tot_len;
+
On Mon, Mar 30, 2020 at 04:01:18PM +0900, Masahiko Sawada wrote:
> On Mon, 30 Mar 2020 at 15:46, Masahiko Sawada
> wrote:
> >
> > On Sun, 29 Mar 2020 at 20:44, Masahiko Sawada
> > wrote:
> > >
> > > > > > I think we need to change parallel maintenance commands so that they
> > > > > > report
On Mon, 30 Mar 2020 at 15:46, Masahiko Sawada
wrote:
>
> On Sun, 29 Mar 2020 at 20:44, Masahiko Sawada
> wrote:
> >
> > On Sun, 29 Mar 2020 at 20:15, Amit Kapila wrote:
> > >
> > > On Sun, Mar 29, 2020 at 1:44 PM Julien Rouhaud wrote:
> > > >
> > > > On Sun, Mar 29, 2020 at 9:52 AM Masahiko
On Sun, 29 Mar 2020 at 20:44, Masahiko Sawada
wrote:
>
> On Sun, 29 Mar 2020 at 20:15, Amit Kapila wrote:
> >
> > On Sun, Mar 29, 2020 at 1:44 PM Julien Rouhaud wrote:
> > >
> > > On Sun, Mar 29, 2020 at 9:52 AM Masahiko Sawada
> > > wrote:
> > > >
> > > > I've run vacuum with/without parallel
Hi Amit,
Sorry I just noticed your mail.
On Sun, Mar 29, 2020 at 05:12:16PM +0530, Amit Kapila wrote:
> On Sun, Mar 29, 2020 at 1:26 PM Julien Rouhaud wrote:
> >
> > I'm not sure that I get your point. I'm assuming that you meant
> > parallel-read-only queries, but surely buffer usage
On Mon, Mar 23, 2020 at 11:24:50PM +0900, Fujii Masao wrote:
>
> > Here are the comments for 0001 patch.
> >
> > + /*
> > + * Report a full page image constructed for the WAL record
> > + */
> > + pgWalUsage.wal_fp_records++;
> >
> > Isn't it better
On Sun, 29 Mar 2020 at 20:15, Amit Kapila wrote:
>
> On Sun, Mar 29, 2020 at 1:44 PM Julien Rouhaud wrote:
> >
> > On Sun, Mar 29, 2020 at 9:52 AM Masahiko Sawada
> > wrote:
> > >
> > > I've run vacuum with/without parallel workers on the table having 5
> > > indexes. The vacuum reads all
On Sun, Mar 29, 2020 at 1:26 PM Julien Rouhaud wrote:
>
> I'm not sure that I get your point. I'm assuming that you meant
> parallel-read-only queries, but surely buffer usage infrastructure for
> parallel query relies on the same approach as non-parallel one (each node
> computes the
On Sun, Mar 29, 2020 at 1:44 PM Julien Rouhaud wrote:
>
> On Sun, Mar 29, 2020 at 9:52 AM Masahiko Sawada
> wrote:
> >
> > I've run vacuum with/without parallel workers on the table having 5
> > indexes. The vacuum reads all blocks of table and indexes.
> >
> > * VACUUM command with no parallel
On Sun, Mar 29, 2020 at 9:52 AM Masahiko Sawada
wrote:
>
> On Sun, 29 Mar 2020 at 15:19, Masahiko Sawada
> wrote:
> >
> > On Sun, 29 Mar 2020 at 14:23, Amit Kapila wrote:
> > >
> > > On Sat, Mar 28, 2020 at 8:47 PM Julien Rouhaud wrote:
> > > >
> > > > On Sat, Mar 28, 2020 at 02:38:27PM +0100,
On Sun, Mar 29, 2020 at 11:03:50AM +0530, Amit Kapila wrote:
> On Sat, Mar 28, 2020 at 7:08 PM Julien Rouhaud wrote:
> >
> > On Sat, Mar 28, 2020 at 04:14:04PM +0530, Amit Kapila wrote:
> > >
> > > Basically,
> > > I don't know changes done in ExecInitParallelPlan and friends allow us
> > > to
On Sun, 29 Mar 2020 at 15:19, Masahiko Sawada
wrote:
>
> On Sun, 29 Mar 2020 at 14:23, Amit Kapila wrote:
> >
> > On Sat, Mar 28, 2020 at 8:47 PM Julien Rouhaud wrote:
> > >
> > > On Sat, Mar 28, 2020 at 02:38:27PM +0100, Julien Rouhaud wrote:
> > > > On Sat, Mar 28, 2020 at 04:14:04PM +0530,
On Sun, 29 Mar 2020 at 14:23, Amit Kapila wrote:
>
> On Sat, Mar 28, 2020 at 8:47 PM Julien Rouhaud wrote:
> >
> > On Sat, Mar 28, 2020 at 02:38:27PM +0100, Julien Rouhaud wrote:
> > > On Sat, Mar 28, 2020 at 04:14:04PM +0530, Amit Kapila wrote:
> > > >
> > > > I see some basic problems with the
On Sat, Mar 28, 2020 at 7:08 PM Julien Rouhaud wrote:
>
> On Sat, Mar 28, 2020 at 04:14:04PM +0530, Amit Kapila wrote:
> >
> > Basically,
> > I don't know changes done in ExecInitParallelPlan and friends allow us
> > to compute WAL for parallel operations. Those will primarily cover
> > parallel
On Sat, Mar 28, 2020 at 8:47 PM Julien Rouhaud wrote:
>
> On Sat, Mar 28, 2020 at 02:38:27PM +0100, Julien Rouhaud wrote:
> > On Sat, Mar 28, 2020 at 04:14:04PM +0530, Amit Kapila wrote:
> > >
> > > I see some basic problems with the patch. The way it tries to compute
> > > WAL usage for
On Sat, Mar 28, 2020 at 02:38:27PM +0100, Julien Rouhaud wrote:
> On Sat, Mar 28, 2020 at 04:14:04PM +0530, Amit Kapila wrote:
> >
> > I see some basic problems with the patch. The way it tries to compute
> > WAL usage for parallel stuff doesn't seem right to me. Can you share
> > or point me
On Sat, Mar 28, 2020 at 04:14:04PM +0530, Amit Kapila wrote:
>
> I see some basic problems with the patch. The way it tries to compute
> WAL usage for parallel stuff doesn't seem right to me. Can you share
> or point me to any test done where we have computed WAL for parallel
> operations like
On Sat, Mar 28, 2020 at 12:54 AM Julien Rouhaud wrote:
>
> On Fri, Mar 27, 2020 at 8:21 PM Kirill Bychik wrote:
> >
> >
> > All these are really valuable objections. Unfortunately, I won't be
> > able to get all sorted out soon, due to total lack of time. I would be
> > very glad if somebody
On Fri, Mar 27, 2020 at 8:21 PM Kirill Bychik wrote:
>
> > > > I'm attaching a v5 with fp records only for temp tables, so there's
> > > > no risk of
> > > > instability. As I previously said I'm fine with your two patches,
> > > > so unless
> > > > you have objections on
> > > I'm attaching a v5 with fp records only for temp tables, so there's
> > > no risk of
> > > instability. As I previously said I'm fine with your two patches, so
> > > unless
> > > you have objections on the fpi test for temp tables or the
> > > documentation
> >
On Mon, Mar 23, 2020 at 3:24 PM Fujii Masao wrote:
>
> On 2020/03/23 21:01, Fujii Masao wrote:
> >
> >
> > On 2020/03/23 7:32, Kirill Bychik wrote:
> > I'm attaching a v5 with fp records only for temp tables, so there's no
> > risk of
> > instability. As I previously said I'm fine
On 2020/03/23 21:01, Fujii Masao wrote:
On 2020/03/23 7:32, Kirill Bychik wrote:
I'm attaching a v5 with fp records only for temp tables, so there's no risk of
instability. As I previously said I'm fine with your two patches, so unless
you have objections on the fpi test for temp tables
On 2020/03/23 7:32, Kirill Bychik wrote:
I'm attaching a v5 with fp records only for temp tables, so there's no risk of
instability. As I previously said I'm fine with your two patches, so unless
you have objections on the fpi test for temp tables or the documentation
changes, I believe
> > > I'm attaching a v5 with fp records only for temp tables, so there's no
> > > risk of
> > > instability. As I previously said I'm fine with your two patches, so
> > > unless
> > > you have objections on the fpi test for temp tables or the documentation
> > > changes, I believe those should
On Thu, Mar 19, 2020 at 09:03:02PM +0900, Fujii Masao wrote:
>
> On 2020/03/19 2:19, Julien Rouhaud wrote:
> >
> > I'm attaching a v5 with fp records only for temp tables, so there's no risk
> > of
> > instability. As I previously said I'm fine with your two patches, so unless
> > you have
On 2020/03/19 2:19, Julien Rouhaud wrote:
On Wed, Mar 18, 2020 at 09:02:58AM +0300, Kirill Bychik wrote:
There is a higher-level Instrumentation API that can be used with
INSTRUMENT_WAL flag to collect the wal usage information. I believe
the instrumentation is widely used in the executor
On Wed, Mar 18, 2020 at 08:48:17PM +0300, Kirill Bychik wrote:
> > I'm attaching a v5 with fp records only for temp tables, so there's no risk
> > of
> > instability. As I previously said I'm fine with your two patches, so unless
> > you have objections on the fpi test for temp tables or the
> > There is a higher-level Instrumentation API that can be used with
> > INSTRUMENT_WAL flag to collect the wal usage information. I believe
> > the instrumentation is widely used in the executor code, so it should
> > not be a problem to colelct instrumentation information on autovacuum
> >
On Wed, Mar 18, 2020 at 09:02:58AM +0300, Kirill Bychik wrote:
>
> There is a higher-level Instrumentation API that can be used with
> INSTRUMENT_WAL flag to collect the wal usage information. I believe
> the instrumentation is widely used in the executor code, so it should
> not be a problem to
> > > > Please feel free to work on any extension of this patch idea. I lack
> > > > both time and knowledge to do it all by myself.
> > >
> > > I'm adding a 3rd patch on top of yours to expose the new WAL counters in
> > > pg_stat_database, for vacuum and autovacuum. I'm not really
> > >
On Tue, Mar 17, 2020 at 10:27:05PM +0300, Kirill Bychik wrote:
> > > Please feel free to work on any extension of this patch idea. I lack
> > > both time and knowledge to do it all by myself.
> >
> > I'm adding a 3rd patch on top of yours to expose the new WAL counters in
> > pg_stat_database, for
> > Please feel free to work on any extension of this patch idea. I lack
> > both time and knowledge to do it all by myself.
>
>
> I'm adding a 3rd patch on top of yours to expose the new WAL counters in
> pg_stat_database, for vacuum and autovacuum. I'm not really enthiusiastic
> with
> this
On Sun, Mar 15, 2020 at 09:52:18PM +0300, Kirill Bychik wrote:
> > > > On Thu, Mar 5, 2020 at 8:55 PM Kirill Bychik
> > > > wrote:
> After extensive thinking and some code diving, I did not manage to
> come up with a sane idea on how to expose data about autovacuum WAL
> usage. Must be the flu.
> > > On Thu, Mar 5, 2020 at 8:55 PM Kirill Bychik
> > > wrote:
> > > > I wanted to keep the patch small and simple, and fit to practical
> > > > needs. This patch is supposed to provide tuning assistance, catching
> > > > an io heavy query in commit-bound situation.
> > > > Total WAL usage per
On Fri, Mar 6, 2020 at 6:59 PM Kirill Bychik wrote:
>
> пт, 6 мар. 2020 г. в 20:14, Julien Rouhaud :
> >
> > On Thu, Mar 5, 2020 at 8:55 PM Kirill Bychik
> > wrote:
> > > I wanted to keep the patch small and simple, and fit to practical
> > > needs. This patch is supposed to provide tuning
пт, 6 мар. 2020 г. в 20:14, Julien Rouhaud :
>
> On Thu, Mar 5, 2020 at 8:55 PM Kirill Bychik wrote:
> >
> > > While at it, did you consider adding a full-page image counter in the
> > > WalUsage?
> > > That's something I'd really like to have and it doesn't seem hard to
> > > integrate.
> >
>
On Thu, Mar 5, 2020 at 8:55 PM Kirill Bychik wrote:
>
> > While at it, did you consider adding a full-page image counter in the
> > WalUsage?
> > That's something I'd really like to have and it doesn't seem hard to
> > integrate.
>
> Well, not sure I understand you 100%, being new to Postgres
On Wed, Mar 04, 2020 at 05:02:25PM +0100, Julien Rouhaud wrote:
> I'm quite worried about the stability of those counters for regression tests.
> Wouldn't a checkpoint happening during the test change them?
Yep. One way to go through that would be to test if this output is
non-zero still I
On Thu, Feb 20, 2020 at 06:56:27PM +0300, Kirill Bychik wrote:
> > вт, 18 февр. 2020 г. в 06:23, Thomas Munro :
> > > On Mon, Feb 10, 2020 at 8:20 PM Craig Ringer
> > > wrote:
> > > > On Wed, 5 Feb 2020 at 21:36, Kirill Bychik
> > > > wrote:
> > > > > Patch is separated in two parts: core
> вт, 18 февр. 2020 г. в 06:23, Thomas Munro :
> > On Mon, Feb 10, 2020 at 8:20 PM Craig Ringer wrote:
> > > On Wed, 5 Feb 2020 at 21:36, Kirill Bychik
> > > wrote:
> > > > Patch is separated in two parts: core changes and pg_stat_statements
> > > > additions. Essentially the extension has its
вт, 18 февр. 2020 г. в 06:23, Thomas Munro :
> On Mon, Feb 10, 2020 at 8:20 PM Craig Ringer wrote:
> > On Wed, 5 Feb 2020 at 21:36, Kirill Bychik wrote:
> > > Patch is separated in two parts: core changes and pg_stat_statements
> > > additions. Essentially the extension has its schema updated to
On Mon, Feb 10, 2020 at 8:20 PM Craig Ringer wrote:
> On Wed, 5 Feb 2020 at 21:36, Kirill Bychik wrote:
> > Patch is separated in two parts: core changes and pg_stat_statements
> > additions. Essentially the extension has its schema updated to allow
> > two more fields, docs updated to reflect
On Wed, 5 Feb 2020 at 21:36, Kirill Bychik wrote:
>
> Hello pgsql-hackers,
>
> Submitting a patch that would enable gathering of per-statement WAL
> generation statistics, similar to how it is done for buffer usage.
> Collected is the number of records added to WAL and number of WAL
> bytes
101 - 169 of 169 matches
Mail list logo
