Re: [HACKERS] pg_stat directory and pg_stat_statements
Fujii-san, I found the right description in REL9_3_STABLE branch, thanks for pointing out the commit. Sorry for noise. 2014-06-18 12:39 GMT+09:00 Fujii Masao : > On Tue, Jun 17, 2014 at 2:11 PM, Shigeru Hanada > wrote: >> Fujii-san, >> >> I agree not to backpatch, but I noticed that the 9.3 document about >> stats collector doesn't mention $PGDATA/pg_stat. >> >> http://www.postgresql.org/docs/current/static/monitoring-stats.html >> >> It just says: >>> When the server shuts down, a permanent copy of the statistics data is >>> stored in the global subdirectory, so that statistics can be retained >>> across server restarts. >> >> I'm not sure whether we should modify the 9.3 document at the moment, >> but actually the description made me confused a bit. > > Could you check 8dc90b9c4c45fa30a8f59313e21d294529ef7182 in REL9_3_STABLE > branch? You can see that I added the description of pg_stat directory > into the document > in 9.3. > > Regards, > > -- > Fujii Masao -- Shigeru HANADA -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_stat directory and pg_stat_statements
On Tue, Jun 17, 2014 at 2:11 PM, Shigeru Hanada wrote: > Fujii-san, > > I agree not to backpatch, but I noticed that the 9.3 document about > stats collector doesn't mention $PGDATA/pg_stat. > > http://www.postgresql.org/docs/current/static/monitoring-stats.html > > It just says: >> When the server shuts down, a permanent copy of the statistics data is >> stored in the global subdirectory, so that statistics can be retained across >> server restarts. > > I'm not sure whether we should modify the 9.3 document at the moment, > but actually the description made me confused a bit. Could you check 8dc90b9c4c45fa30a8f59313e21d294529ef7182 in REL9_3_STABLE branch? You can see that I added the description of pg_stat directory into the document in 9.3. Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_stat directory and pg_stat_statements
Fujii-san, I agree not to backpatch, but I noticed that the 9.3 document about stats collector doesn't mention $PGDATA/pg_stat. http://www.postgresql.org/docs/current/static/monitoring-stats.html It just says: > When the server shuts down, a permanent copy of the statistics data is stored > in the global subdirectory, so that statistics can be retained across server > restarts. I'm not sure whether we should modify the 9.3 document at the moment, but actually the description made me confused a bit. 2014-05-29 12:22 GMT+09:00 Fujii Masao : > On Thu, May 29, 2014 at 4:55 AM, Tomas Vondra wrote: >> On 28.5.2014 19:52, Fujii Masao wrote: >>> On Thu, May 29, 2014 at 12:37 AM, Peter Geoghegan wrote: On Wed, May 28, 2014 at 7:01 AM, Fujii Masao wrote: > But pg_stat_statements file is saved under $PGDATA/global yet. > Is this intentional or just oversight? I think it's an oversight. >>> >>> OK, patch attached. >>> >>> I'm afraid that it's not okay to change the file layout in $PGDATA at this >>> beta1 >>> stage because that change basically seems to need initdb. Otherwise >>> something >>> like "no such file or directory" error can happen. But in this case what we >>> need >>> to change is only the location of the pg_stat_statements permanent stats >>> file. >>> So, without initdb, the server will not be able to find the >>> pg_stat_statements stats >>> file, but this is not so harmful. Only the problem is that the >>> pg_stat_statements >>> stats which were collected in past would disappear. OTOH, the server can >>> keep >>> running successfully from then and no critical data will not >>> disappear. Therefore >>> I think we can commit this patch even at beta1. Thought? >> >> For HEAD, probably yes. But what about backpatching 9.3? > > I think No. So we should not backpatch this to 9.3. > > Regards, > > -- > Fujii Masao > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers -- Shigeru HANADA -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_stat directory and pg_stat_statements
On Tue, Jun 3, 2014 at 6:38 PM, Magnus Hagander wrote: > On Mon, Jun 2, 2014 at 4:07 PM, Andres Freund > wrote: >> >> On 2014-06-02 22:59:55 +0900, Fujii Masao wrote: >> > On Thu, May 29, 2014 at 3:02 PM, Peter Geoghegan wrote: >> > > On Wed, May 28, 2014 at 10:49 PM, Fujii Masao >> > > wrote: >> > >> You're concerned about the scenario using pg_upgrade? I'm not sure >> > >> the detail >> > >> of pg_upgrade. But if it doesn't work properly, we should have gotten >> > >> the trouble >> > > >> > > I'm not worried about pg_upgrade, because by design pg_stat_statements >> > > will discard stats files that originated in earlier versions. However, >> > > I don't see a need to change pg_stat_statements to serialize its >> > > statistics to disk in the pg_stat directory before we branch off 9.4. >> > > As you mentioned, it's harmless. >> > >> > Yeah, that's an idea. OTOH, there is no *strong* reason to postpone >> > the fix to 9.5. So I just feel inclined to apply the fix now... >> >> +1 for fixing it now. >> > > +1 for fixing it for 9.4 before the next beta, but *not* backpatching it to > 9.3 - it *is* a behaviour change after all.. Yep, I just applied the patch only to HEAD. Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_stat directory and pg_stat_statements
On Mon, Jun 2, 2014 at 4:07 PM, Andres Freund wrote: > On 2014-06-02 22:59:55 +0900, Fujii Masao wrote: > > On Thu, May 29, 2014 at 3:02 PM, Peter Geoghegan wrote: > > > On Wed, May 28, 2014 at 10:49 PM, Fujii Masao > wrote: > > >> You're concerned about the scenario using pg_upgrade? I'm not sure > the detail > > >> of pg_upgrade. But if it doesn't work properly, we should have gotten > > >> the trouble > > > > > > I'm not worried about pg_upgrade, because by design pg_stat_statements > > > will discard stats files that originated in earlier versions. However, > > > I don't see a need to change pg_stat_statements to serialize its > > > statistics to disk in the pg_stat directory before we branch off 9.4. > > > As you mentioned, it's harmless. > > > > Yeah, that's an idea. OTOH, there is no *strong* reason to postpone > > the fix to 9.5. So I just feel inclined to apply the fix now... > > +1 for fixing it now. > > +1 for fixing it for 9.4 before the next beta, but *not* backpatching it to 9.3 - it *is* a behaviour change after all.. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/
Re: [HACKERS] pg_stat directory and pg_stat_statements
On Mon, Jun 2, 2014 at 11:07 PM, Andres Freund wrote: > On 2014-06-02 22:59:55 +0900, Fujii Masao wrote: >> On Thu, May 29, 2014 at 3:02 PM, Peter Geoghegan wrote: >> > On Wed, May 28, 2014 at 10:49 PM, Fujii Masao >> > wrote: >> >> You're concerned about the scenario using pg_upgrade? I'm not sure the >> >> detail >> >> of pg_upgrade. But if it doesn't work properly, we should have gotten >> >> the trouble >> > >> > I'm not worried about pg_upgrade, because by design pg_stat_statements >> > will discard stats files that originated in earlier versions. However, >> > I don't see a need to change pg_stat_statements to serialize its >> > statistics to disk in the pg_stat directory before we branch off 9.4. >> > As you mentioned, it's harmless. >> >> Yeah, that's an idea. OTOH, there is no *strong* reason to postpone >> the fix to 9.5. So I just feel inclined to apply the fix now... > > +1 for fixing it now. +1. A beta is here for that as well. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_stat directory and pg_stat_statements
On 2014-06-02 22:59:55 +0900, Fujii Masao wrote: > On Thu, May 29, 2014 at 3:02 PM, Peter Geoghegan wrote: > > On Wed, May 28, 2014 at 10:49 PM, Fujii Masao wrote: > >> You're concerned about the scenario using pg_upgrade? I'm not sure the > >> detail > >> of pg_upgrade. But if it doesn't work properly, we should have gotten > >> the trouble > > > > I'm not worried about pg_upgrade, because by design pg_stat_statements > > will discard stats files that originated in earlier versions. However, > > I don't see a need to change pg_stat_statements to serialize its > > statistics to disk in the pg_stat directory before we branch off 9.4. > > As you mentioned, it's harmless. > > Yeah, that's an idea. OTOH, there is no *strong* reason to postpone > the fix to 9.5. So I just feel inclined to apply the fix now... +1 for fixing it now. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_stat directory and pg_stat_statements
On Thu, May 29, 2014 at 3:02 PM, Peter Geoghegan wrote: > On Wed, May 28, 2014 at 10:49 PM, Fujii Masao wrote: >> You're concerned about the scenario using pg_upgrade? I'm not sure the detail >> of pg_upgrade. But if it doesn't work properly, we should have gotten >> the trouble > > I'm not worried about pg_upgrade, because by design pg_stat_statements > will discard stats files that originated in earlier versions. However, > I don't see a need to change pg_stat_statements to serialize its > statistics to disk in the pg_stat directory before we branch off 9.4. > As you mentioned, it's harmless. Yeah, that's an idea. OTOH, there is no *strong* reason to postpone the fix to 9.5. So I just feel inclined to apply the fix now... Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_stat directory and pg_stat_statements
On Thu, May 29, 2014 at 11:32 AM, Peter Geoghegan wrote: > On Wed, May 28, 2014 at 10:49 PM, Fujii Masao > wrote: > > You're concerned about the scenario using pg_upgrade? > Yeah - I was. > I'm not sure the detail > > of pg_upgrade. But if it doesn't work properly, we should have gotten > > the trouble > > I'm not worried about pg_upgrade, because by design pg_stat_statements > will discard stats files that originated in earlier versions. However, > I don't see a need to change pg_stat_statements to serialize its > statistics to disk in the pg_stat directory before we branch off 9.4. > As you mentioned, it's harmless. > K. I was just curious about the scenario. If it was discarding the stats files that originated in earlier version, It should be ok. -- Thanks & Regards, Ashesh Vashi > > -- > Peter Geoghegan >
Re: [HACKERS] pg_stat directory and pg_stat_statements
On Wed, May 28, 2014 at 10:49 PM, Fujii Masao wrote: > You're concerned about the scenario using pg_upgrade? I'm not sure the detail > of pg_upgrade. But if it doesn't work properly, we should have gotten > the trouble I'm not worried about pg_upgrade, because by design pg_stat_statements will discard stats files that originated in earlier versions. However, I don't see a need to change pg_stat_statements to serialize its statistics to disk in the pg_stat directory before we branch off 9.4. As you mentioned, it's harmless. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_stat directory and pg_stat_statements
On Thu, May 29, 2014 at 1:01 PM, Ashesh Vashi wrote: > On Thu, May 29, 2014 at 8:52 AM, Fujii Masao wrote: >> >> On Thu, May 29, 2014 at 4:55 AM, Tomas Vondra wrote: >> > On 28.5.2014 19:52, Fujii Masao wrote: >> >> On Thu, May 29, 2014 at 12:37 AM, Peter Geoghegan >> >> wrote: >> >>> On Wed, May 28, 2014 at 7:01 AM, Fujii Masao >> >>> wrote: >> But pg_stat_statements file is saved under $PGDATA/global yet. >> Is this intentional or just oversight? >> >>> >> >>> >> >>> I think it's an oversight. >> >> >> >> OK, patch attached. >> >> >> >> I'm afraid that it's not okay to change the file layout in $PGDATA at >> >> this beta1 >> >> stage because that change basically seems to need initdb. Otherwise >> >> something >> >> like "no such file or directory" error can happen. But in this case >> >> what we need >> >> to change is only the location of the pg_stat_statements permanent >> >> stats file. >> >> So, without initdb, the server will not be able to find the >> >> pg_stat_statements stats >> >> file, but this is not so harmful. Only the problem is that the >> >> pg_stat_statements >> >> stats which were collected in past would disappear. OTOH, the server >> >> can keep >> >> running successfully from then and no critical data will not >> >> disappear. Therefore >> >> I think we can commit this patch even at beta1. Thought? >> > >> > For HEAD, probably yes. But what about backpatching 9.3? >> >> I think No. So we should not backpatch this to 9.3. > > Just curious. > Will it work in upgrade scenario? You're concerned about the scenario using pg_upgrade? I'm not sure the detail of pg_upgrade. But if it doesn't work properly, we should have gotten the trouble when 9.3 (which pg_stat directory was introduced) was released. But I've not heard such trouble Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_stat directory and pg_stat_statements
On Thu, May 29, 2014 at 8:52 AM, Fujii Masao wrote: > On Thu, May 29, 2014 at 4:55 AM, Tomas Vondra wrote: > > On 28.5.2014 19:52, Fujii Masao wrote: > >> On Thu, May 29, 2014 at 12:37 AM, Peter Geoghegan > wrote: > >>> On Wed, May 28, 2014 at 7:01 AM, Fujii Masao > wrote: > But pg_stat_statements file is saved under $PGDATA/global yet. > Is this intentional or just oversight? > >>> > >>> > >>> I think it's an oversight. > >> > >> OK, patch attached. > >> > >> I'm afraid that it's not okay to change the file layout in $PGDATA at > this beta1 > >> stage because that change basically seems to need initdb. Otherwise > something > >> like "no such file or directory" error can happen. But in this case > what we need > >> to change is only the location of the pg_stat_statements permanent > stats file. > >> So, without initdb, the server will not be able to find the > >> pg_stat_statements stats > >> file, but this is not so harmful. Only the problem is that the > >> pg_stat_statements > >> stats which were collected in past would disappear. OTOH, the server > can keep > >> running successfully from then and no critical data will not > >> disappear. Therefore > >> I think we can commit this patch even at beta1. Thought? > > > > For HEAD, probably yes. But what about backpatching 9.3? > > I think No. So we should not backpatch this to 9.3. > Just curious. Will it work in upgrade scenario? -- Thanks & Regards, Ashesh Vashi > > Regards, > > -- > Fujii Masao > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers >
Re: [HACKERS] pg_stat directory and pg_stat_statements
On Thu, May 29, 2014 at 4:55 AM, Tomas Vondra wrote: > On 28.5.2014 19:52, Fujii Masao wrote: >> On Thu, May 29, 2014 at 12:37 AM, Peter Geoghegan wrote: >>> On Wed, May 28, 2014 at 7:01 AM, Fujii Masao wrote: But pg_stat_statements file is saved under $PGDATA/global yet. Is this intentional or just oversight? >>> >>> >>> I think it's an oversight. >> >> OK, patch attached. >> >> I'm afraid that it's not okay to change the file layout in $PGDATA at this >> beta1 >> stage because that change basically seems to need initdb. Otherwise something >> like "no such file or directory" error can happen. But in this case what we >> need >> to change is only the location of the pg_stat_statements permanent stats >> file. >> So, without initdb, the server will not be able to find the >> pg_stat_statements stats >> file, but this is not so harmful. Only the problem is that the >> pg_stat_statements >> stats which were collected in past would disappear. OTOH, the server can keep >> running successfully from then and no critical data will not >> disappear. Therefore >> I think we can commit this patch even at beta1. Thought? > > For HEAD, probably yes. But what about backpatching 9.3? I think No. So we should not backpatch this to 9.3. Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_stat directory and pg_stat_statements
On 28.5.2014 19:52, Fujii Masao wrote: > On Thu, May 29, 2014 at 12:37 AM, Peter Geoghegan wrote: >> On Wed, May 28, 2014 at 7:01 AM, Fujii Masao wrote: >>> But pg_stat_statements file is saved under $PGDATA/global yet. >>> Is this intentional or just oversight? >> >> >> I think it's an oversight. > > OK, patch attached. > > I'm afraid that it's not okay to change the file layout in $PGDATA at this > beta1 > stage because that change basically seems to need initdb. Otherwise something > like "no such file or directory" error can happen. But in this case what we > need > to change is only the location of the pg_stat_statements permanent stats file. > So, without initdb, the server will not be able to find the > pg_stat_statements stats > file, but this is not so harmful. Only the problem is that the > pg_stat_statements > stats which were collected in past would disappear. OTOH, the server can keep > running successfully from then and no critical data will not > disappear. Therefore > I think we can commit this patch even at beta1. Thought? For HEAD, probably yes. But what about backpatching 9.3? Tomas -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_stat directory and pg_stat_statements
On Thu, May 29, 2014 at 12:37 AM, Peter Geoghegan wrote: > On Wed, May 28, 2014 at 7:01 AM, Fujii Masao wrote: >> But pg_stat_statements file is saved under $PGDATA/global yet. >> Is this intentional or just oversight? > > > I think it's an oversight. OK, patch attached. I'm afraid that it's not okay to change the file layout in $PGDATA at this beta1 stage because that change basically seems to need initdb. Otherwise something like "no such file or directory" error can happen. But in this case what we need to change is only the location of the pg_stat_statements permanent stats file. So, without initdb, the server will not be able to find the pg_stat_statements stats file, but this is not so harmful. Only the problem is that the pg_stat_statements stats which were collected in past would disappear. OTOH, the server can keep running successfully from then and no critical data will not disappear. Therefore I think we can commit this patch even at beta1. Thought? Regards, -- Fujii Masao From 8f3281566399bbb088b826b8fe9c3baa5027e5da Mon Sep 17 00:00:00 2001 From: MasaoFujii Date: Thu, 29 May 2014 02:14:34 +0900 Subject: [PATCH] Save pg_stat_statements statistics file into $PGDATA/pg_stat directory at shutdown. 187492b6c2e8cafc5b39063ca3b67846e8155d24 changed pgstat.c so that the stats files were saved into $PGDATA/pg_stat directory when the server was shutdowned. But it accidentally forgot to change the location of pg_stat_statements permanent stats file. This commit fixes pg_stat_statements so that its stats file is also saved into $PGDATA/pg_stat at shutdown. --- contrib/pg_stat_statements/pg_stat_statements.c | 2 +- src/backend/postmaster/pgstat.c | 8 src/include/pgstat.h| 8 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/contrib/pg_stat_statements/pg_stat_statements.c b/contrib/pg_stat_statements/pg_stat_statements.c index 32d16cc..a3e8c59 100644 --- a/contrib/pg_stat_statements/pg_stat_statements.c +++ b/contrib/pg_stat_statements/pg_stat_statements.c @@ -80,7 +80,7 @@ PG_MODULE_MAGIC; /* Location of permanent stats file (valid when database is shut down) */ -#define PGSS_DUMP_FILE "global/pg_stat_statements.stat" +#define PGSS_DUMP_FILE PGSTAT_STAT_PERMANENT_DIRECTORY "/pg_stat_statements.stat" /* * Location of external query text file. We don't keep it in the core diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c index f864816..3ab1428 100644 --- a/src/backend/postmaster/pgstat.c +++ b/src/backend/postmaster/pgstat.c @@ -68,14 +68,6 @@ /* -- - * Paths for the statistics files (relative to installation's $PGDATA). - * -- - */ -#define PGSTAT_STAT_PERMANENT_DIRECTORY "pg_stat" -#define PGSTAT_STAT_PERMANENT_FILENAME "pg_stat/global.stat" -#define PGSTAT_STAT_PERMANENT_TMPFILE "pg_stat/global.tmp" - -/* -- * Timer definitions. * -- */ diff --git a/src/include/pgstat.h b/src/include/pgstat.h index d9de09f..0892533 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -20,6 +20,14 @@ #include "utils/relcache.h" +/* -- + * Paths for the statistics files (relative to installation's $PGDATA). + * -- + */ +#define PGSTAT_STAT_PERMANENT_DIRECTORY "pg_stat" +#define PGSTAT_STAT_PERMANENT_FILENAME "pg_stat/global.stat" +#define PGSTAT_STAT_PERMANENT_TMPFILE "pg_stat/global.tmp" + /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" -- 1.7.12.4 (Apple Git-37) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_stat directory and pg_stat_statements
On Wed, May 28, 2014 at 7:01 AM, Fujii Masao wrote: > But pg_stat_statements file is saved under $PGDATA/global yet. > Is this intentional or just oversight? I think it's an oversight. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pg_stat directory and pg_stat_statements
Hi, Thanks to 187492b6c2e8cafc5b39063ca3b67846e8155d24, pgstat files are now saved to $PGDATA/pg_stat directory at shutdown. But pg_stat_statements file is saved under $PGDATA/global yet. Is this intentional or just oversight? Saving that file to global is harmless, though. Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
