Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Wed, Mar 28, 2012 at 10:08 AM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Mar 28, 2012 at 12:30 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Fujii Masao's message of mar mar 27 06:40:34 -0300 2012: Anyway, should I add this patch into the next CF? Or is anyone planning to commit the patch for 9.2? I think the correct thing to do here is add to next CF, and if some committer has enough interest in getting it quickly it can be grabbed from there and committed into 9.2. Yep! I've added it to next CF. Attached is the revised version of the patch. Regards, -- Fujii Masao pg_controldata_walfilename_v4.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Mon, Mar 26, 2012 at 11:18 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Mar 26, 2012 at 2:50 AM, Magnus Hagander mag...@hagander.net wrote: s/segment/file/g? We're already using file to mean something different *internally*, don't we? And since pg_controldata shows fairly internal information, I'm not sure this is the best idea. Maybe compromise and call it segment file - that is both easier to understand than segment, and not actually using a term that means something else... It's also kind of wordy. I think file is fine. +1 for file. I think the internal usage of file to mean roughly 4GB worth of WAL is going to go away soon anyway, as there won't be much reason to worry about the concept once LSN arithmetic is 64-bit. Agreed. This would mean that the following lots of log messages need to be changed after 64-bit LSN will have been committed. errmsg(could not fdatasync log file %u, segment %u: %m, log, seg))); Anyway, should I add this patch into the next CF? Or is anyone planning to commit the patch for 9.2? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
Excerpts from Fujii Masao's message of mar mar 27 06:40:34 -0300 2012: Anyway, should I add this patch into the next CF? Or is anyone planning to commit the patch for 9.2? I think the correct thing to do here is add to next CF, and if some committer has enough interest in getting it quickly it can be grabbed from there and committed into 9.2. -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Wed, Mar 28, 2012 at 12:30 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Fujii Masao's message of mar mar 27 06:40:34 -0300 2012: Anyway, should I add this patch into the next CF? Or is anyone planning to commit the patch for 9.2? I think the correct thing to do here is add to next CF, and if some committer has enough interest in getting it quickly it can be grabbed from there and committed into 9.2. Yep! I've added it to next CF. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Fri, Mar 23, 2012 at 18:05, Fujii Masao masao.fu...@gmail.com wrote: On Sat, Mar 24, 2012 at 1:49 AM, Robert Haas robertmh...@gmail.com wrote: On Fri, Mar 23, 2012 at 12:42 PM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Mar 23, 2012 at 9:41 PM, Robert Haas robertmh...@gmail.com wrote: On Fri, Mar 23, 2012 at 6:13 AM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Mar 23, 2012 at 5:56 PM, Magnus Hagander mag...@hagander.net wrote: Might it be a good idea to put it on it's own row instead of changing the format of an existing row, in order not to break scripts and programs that are parsing the previous output? Good idea! What row name should we use for the WAL file containing REDO record? Latest checkpoint's REDO file? Sounds good to me. I like the idea, too. The status quo is an unnecessary nuisance, so this will be a nice usability improvement. Attached patch adds new row Latest checkpoint's REDO WAL segment: into the result of pg_controldata. I used the term WAL segment for the row name instead of file because WAL segment is used in another row Bytes per WAL segment:. But better name? s/segment/file/g? Yep, file might be more intuitive for a user than segment. Attached is the file version of the patch. We're already using file to mean something different *internally*, don't we? And since pg_controldata shows fairly internal information, I'm not sure this is the best idea. Maybe compromise and call it segment file - that is both easier to understand than segment, and not actually using a term that means something else... -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Mon, Mar 26, 2012 at 2:50 AM, Magnus Hagander mag...@hagander.net wrote: s/segment/file/g? Yep, file might be more intuitive for a user than segment. Attached is the file version of the patch. We're already using file to mean something different *internally*, don't we? And since pg_controldata shows fairly internal information, I'm not sure this is the best idea. Maybe compromise and call it segment file - that is both easier to understand than segment, and not actually using a term that means something else... It's also kind of wordy. I think file is fine. There are a few references to xlogid indicating a file number, but the actual field name is just xlogid. We also use the term file to mean the other thing, as in XLOGfileslop, and I have a hard time believing anyone's really going to get confused about what is meant here. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
Robert Haas robertmh...@gmail.com writes: On Mon, Mar 26, 2012 at 2:50 AM, Magnus Hagander mag...@hagander.net wrote: s/segment/file/g? We're already using file to mean something different *internally*, don't we? And since pg_controldata shows fairly internal information, I'm not sure this is the best idea. Maybe compromise and call it segment file - that is both easier to understand than segment, and not actually using a term that means something else... It's also kind of wordy. I think file is fine. +1 for file. I think the internal usage of file to mean roughly 4GB worth of WAL is going to go away soon anyway, as there won't be much reason to worry about the concept once LSN arithmetic is 64-bit. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Fri, Mar 23, 2012 at 05:06, Fujii Masao masao.fu...@gmail.com wrote: Hi, I'd like to propose to change pg_controldata so that it reports the name of WAL file containing the latest checkpoint's REDO record, as follows: $ pg_controldata $PGDATA ... Latest checkpoint's REDO location: 0/16D6ACC (file 00010001) Latest checkpoint's TimeLineID: 1 ... This simplifies very much the way to calculate the archive file cutoff point because the reported WAL file is just cutoff point itself. If the file name is not reported, we have to calculate the cutoff point from the reported location and timeline, which is complicated calculation. We can use pg_xlogfile_name function to calculate that, but it cannot be executed in the standby. Another problem is that pg_xlogfile_name always uses current timeline for the calculation, so if the reported timeline is not the same as current one, pg_xlogfile_name cannot return the correct WAL file name. Making pg_controldata report that WAL file name gets rid of such a complexity. You may think that archive_cleanup_command is usable for that purpose. That's true. But it's not usable simply for the case where there are more than one standby servers. In this case, the archive file cutoff point needs to be calculated from each standby's REDO location and timeline. Attached patch changes pg_controldata as above. Thought? Might it be a good idea to put it on it's own row instead of changing the format of an existing row, in order not to break scripts and programs that are parsing the previous output? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Fri, Mar 23, 2012 at 2:06 PM, Jaime Casanova ja...@2ndquadrant.com wrote: On Thu, Mar 22, 2012 at 11:06 PM, Fujii Masao masao.fu...@gmail.com wrote: We can use pg_xlogfile_name function to calculate that, but it cannot be executed in the standby. Another problem is that pg_xlogfile_name always uses current timeline for the calculation, so if the reported timeline is not the same as current one, pg_xlogfile_name cannot return the correct WAL file name. Making pg_controldata report that WAL file name gets rid of such a complexity. i would think that pg_xlogfile_name() is not allowed in the standby because ThisTimelineId is not very well defined in recovery but if you extend pg_xlogfile_name() to also receive a timelineid as you suggested in [1] then that version of the function could be allowed in the standby. or there is something else i'm missing? is that enough for you to solve your problem? Yes, we can more easily calculate the cutoff point by using that extended pg_xlogfile_name(). But if pg_controldata reports the WAL file name, we can calculate the cutoff point without starting the server. So I think that it's worth changing pg_controldata that way even if we extend pg_xlogfile_name(). Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Fri, Mar 23, 2012 at 10:51, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Mar 23, 2012 at 2:06 PM, Jaime Casanova ja...@2ndquadrant.com wrote: On Thu, Mar 22, 2012 at 11:06 PM, Fujii Masao masao.fu...@gmail.com wrote: We can use pg_xlogfile_name function to calculate that, but it cannot be executed in the standby. Another problem is that pg_xlogfile_name always uses current timeline for the calculation, so if the reported timeline is not the same as current one, pg_xlogfile_name cannot return the correct WAL file name. Making pg_controldata report that WAL file name gets rid of such a complexity. i would think that pg_xlogfile_name() is not allowed in the standby because ThisTimelineId is not very well defined in recovery but if you extend pg_xlogfile_name() to also receive a timelineid as you suggested in [1] then that version of the function could be allowed in the standby. or there is something else i'm missing? is that enough for you to solve your problem? Yes, we can more easily calculate the cutoff point by using that extended pg_xlogfile_name(). But if pg_controldata reports the WAL file name, we can calculate the cutoff point without starting the server. So I think that it's worth changing pg_controldata that way even if we extend pg_xlogfile_name(). +1 - I think they're both useful things, each on it's own. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Fri, Mar 23, 2012 at 5:56 PM, Magnus Hagander mag...@hagander.net wrote: Might it be a good idea to put it on it's own row instead of changing the format of an existing row, in order not to break scripts and programs that are parsing the previous output? Good idea! What row name should we use for the WAL file containing REDO record? Latest checkpoint's REDO file? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Fri, Mar 23, 2012 at 6:13 AM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Mar 23, 2012 at 5:56 PM, Magnus Hagander mag...@hagander.net wrote: Might it be a good idea to put it on it's own row instead of changing the format of an existing row, in order not to break scripts and programs that are parsing the previous output? Good idea! What row name should we use for the WAL file containing REDO record? Latest checkpoint's REDO file? Sounds good to me. I like the idea, too. The status quo is an unnecessary nuisance, so this will be a nice usability improvement. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Fri, Mar 23, 2012 at 9:41 PM, Robert Haas robertmh...@gmail.com wrote: On Fri, Mar 23, 2012 at 6:13 AM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Mar 23, 2012 at 5:56 PM, Magnus Hagander mag...@hagander.net wrote: Might it be a good idea to put it on it's own row instead of changing the format of an existing row, in order not to break scripts and programs that are parsing the previous output? Good idea! What row name should we use for the WAL file containing REDO record? Latest checkpoint's REDO file? Sounds good to me. I like the idea, too. The status quo is an unnecessary nuisance, so this will be a nice usability improvement. Attached patch adds new row Latest checkpoint's REDO WAL segment: into the result of pg_controldata. I used the term WAL segment for the row name instead of file because WAL segment is used in another row Bytes per WAL segment:. But better name? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center pg_controldata_walfilename_v2.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Fri, Mar 23, 2012 at 12:42 PM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Mar 23, 2012 at 9:41 PM, Robert Haas robertmh...@gmail.com wrote: On Fri, Mar 23, 2012 at 6:13 AM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Mar 23, 2012 at 5:56 PM, Magnus Hagander mag...@hagander.net wrote: Might it be a good idea to put it on it's own row instead of changing the format of an existing row, in order not to break scripts and programs that are parsing the previous output? Good idea! What row name should we use for the WAL file containing REDO record? Latest checkpoint's REDO file? Sounds good to me. I like the idea, too. The status quo is an unnecessary nuisance, so this will be a nice usability improvement. Attached patch adds new row Latest checkpoint's REDO WAL segment: into the result of pg_controldata. I used the term WAL segment for the row name instead of file because WAL segment is used in another row Bytes per WAL segment:. But better name? s/segment/file/g? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Sat, Mar 24, 2012 at 1:49 AM, Robert Haas robertmh...@gmail.com wrote: On Fri, Mar 23, 2012 at 12:42 PM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Mar 23, 2012 at 9:41 PM, Robert Haas robertmh...@gmail.com wrote: On Fri, Mar 23, 2012 at 6:13 AM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Mar 23, 2012 at 5:56 PM, Magnus Hagander mag...@hagander.net wrote: Might it be a good idea to put it on it's own row instead of changing the format of an existing row, in order not to break scripts and programs that are parsing the previous output? Good idea! What row name should we use for the WAL file containing REDO record? Latest checkpoint's REDO file? Sounds good to me. I like the idea, too. The status quo is an unnecessary nuisance, so this will be a nice usability improvement. Attached patch adds new row Latest checkpoint's REDO WAL segment: into the result of pg_controldata. I used the term WAL segment for the row name instead of file because WAL segment is used in another row Bytes per WAL segment:. But better name? s/segment/file/g? Yep, file might be more intuitive for a user than segment. Attached is the file version of the patch. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center pg_controldata_walfilename_v3.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
Hi, I'd like to propose to change pg_controldata so that it reports the name of WAL file containing the latest checkpoint's REDO record, as follows: $ pg_controldata $PGDATA ... Latest checkpoint's REDO location:0/16D6ACC (file 00010001) Latest checkpoint's TimeLineID: 1 ... This simplifies very much the way to calculate the archive file cutoff point because the reported WAL file is just cutoff point itself. If the file name is not reported, we have to calculate the cutoff point from the reported location and timeline, which is complicated calculation. We can use pg_xlogfile_name function to calculate that, but it cannot be executed in the standby. Another problem is that pg_xlogfile_name always uses current timeline for the calculation, so if the reported timeline is not the same as current one, pg_xlogfile_name cannot return the correct WAL file name. Making pg_controldata report that WAL file name gets rid of such a complexity. You may think that archive_cleanup_command is usable for that purpose. That's true. But it's not usable simply for the case where there are more than one standby servers. In this case, the archive file cutoff point needs to be calculated from each standby's REDO location and timeline. Attached patch changes pg_controldata as above. Thought? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center *** a/src/bin/pg_controldata/pg_controldata.c --- b/src/bin/pg_controldata/pg_controldata.c *** *** 24,29 --- 24,30 #include fcntl.h #include access/xlog.h + #include access/xlog_internal.h #include catalog/pg_control.h *** *** 101,106 main(int argc, char *argv[]) --- 102,110 char sysident_str[32]; const char *strftime_fmt = %c; const char *progname; + uint32 log; + uint32 seg; + char xlogfilename[MAXFNAMELEN]; set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN(pg_controldata)); *** *** 177,182 main(int argc, char *argv[]) --- 181,193 localtime(time_tmp)); /* + * Calculate the WAL file name containing the latest checkpoint's REDO + * record. + */ + XLByteToSeg(ControlFile.checkPointCopy.redo, log, seg); + XLogFileName(xlogfilename, ControlFile.checkPointCopy.ThisTimeLineID, log, seg); + + /* * Format system_identifier separately to keep platform-dependent format * code out of the translatable message string. */ *** *** 204,212 main(int argc, char *argv[]) printf(_(Prior checkpoint location:%X/%X\n), ControlFile.prevCheckPoint.xlogid, ControlFile.prevCheckPoint.xrecoff); ! printf(_(Latest checkpoint's REDO location:%X/%X\n), ControlFile.checkPointCopy.redo.xlogid, ! ControlFile.checkPointCopy.redo.xrecoff); printf(_(Latest checkpoint's TimeLineID: %u\n), ControlFile.checkPointCopy.ThisTimeLineID); printf(_(Latest checkpoint's full_page_writes: %s\n), --- 215,224 printf(_(Prior checkpoint location:%X/%X\n), ControlFile.prevCheckPoint.xlogid, ControlFile.prevCheckPoint.xrecoff); ! printf(_(Latest checkpoint's REDO location:%X/%X (file %s)\n), ControlFile.checkPointCopy.redo.xlogid, ! ControlFile.checkPointCopy.redo.xrecoff, ! xlogfilename); printf(_(Latest checkpoint's TimeLineID: %u\n), ControlFile.checkPointCopy.ThisTimeLineID); printf(_(Latest checkpoint's full_page_writes: %s\n), -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Thu, Mar 22, 2012 at 11:06 PM, Fujii Masao masao.fu...@gmail.com wrote: We can use pg_xlogfile_name function to calculate that, but it cannot be executed in the standby. Another problem is that pg_xlogfile_name always uses current timeline for the calculation, so if the reported timeline is not the same as current one, pg_xlogfile_name cannot return the correct WAL file name. Making pg_controldata report that WAL file name gets rid of such a complexity. i would think that pg_xlogfile_name() is not allowed in the standby because ThisTimelineId is not very well defined in recovery but if you extend pg_xlogfile_name() to also receive a timelineid as you suggested in [1] then that version of the function could be allowed in the standby. or there is something else i'm missing? is that enough for you to solve your problem? [1] http://archives.postgresql.org/message-id/cahgqgwhwqjgeksmp2oteru8p0sj6x7ypqyh5qqeqcbqsxaa...@mail.gmail.com -- Jaime Casanova www.2ndQuadrant.com Professional PostgreSQL: Soporte 24x7 y capacitación -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
