Re: [PATCHES] Fix for erroneous warning on Shutdown
On Fri, 2004-06-11 at 19:25, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Minor patch to correct erroneous warning in cvs tip, believed to be a very minor regression. This patch is wrong; it effectively disables the warning altogether. When a shutdown was requested within CHECKPOINT_SECONDS of a checkpoint, the shutdown code in the bgwriter (which has does checkpointing) would issue the erroneous message: I don't think so ... the shutdown path doesn't go through this code. Could you take another look at the cause of what you saw? Hmmm...perhaps I should submit this as a *low priority* bug then, rather than just a patch? As of now, (i.e. even including the new bgwriter shutdown) if you: 1. start postmaster 2. do some work that writes xlog 3. shutdown within some few seconds of startup you get a WARNING suggesting you increase CHECKPOINT_SEGMENTS, which is clearly erroneous since no checkpoint has taken place since startup. Not exactly a common circumstance, I grant you, but it does seem to be a regression nonetheless. My thinking was that the circumstance was not limited to this edge case, but actually any shutdown within 30s of a checkpoint, but the latter is just speculation. Only reason I spotted it is I've been doing a lot of startup/shutdown work recently... Best Regards, Simon Riggs ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [PATCHES] Fix for erroneous warning on Shutdown
Simon Riggs [EMAIL PROTECTED] writes: As of now, (i.e. even including the new bgwriter shutdown) if you: 1. start postmaster 2. do some work that writes xlog 3. shutdown within some few seconds of startup you get a WARNING suggesting you increase CHECKPOINT_SEGMENTS, which is clearly erroneous since no checkpoint has taken place since startup. [ scratches head... ] Not for me. Are you using any nondefault settings in postgresql.conf? regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] Fix for erroneous warning on Shutdown
On Tue, 2004-06-15 at 19:33, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: As of now, (i.e. even including the new bgwriter shutdown) if you: 1. start postmaster 2. do some work that writes xlog 3. shutdown within some few seconds of startup you get a WARNING suggesting you increase CHECKPOINT_SEGMENTS, which is clearly erroneous since no checkpoint has taken place since startup. [ scratches head... ] Not for me. Are you using any nondefault settings in postgresql.conf? Hmmm...I did a complete re-download of CVS, then a vanilla configure/make/make install with .conf straight from can. I wanted to be absolutely certain I had got it clean. Shutdown was via CTRL-Cmake a difference? Damn I hate the wierd ones...if only because of the 90% chance its something wrong that I've done... I'll do it again more slowly and report back... Best Regards, Simon Riggs ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [PATCHES] Fix for erroneous warning on Shutdown
Simon Riggs [EMAIL PROTECTED] writes: Shutdown was via CTRL-Cmake a difference? Wouldn't think so. I can force the message to appear if I do a *manual* CHECKPOINT command within thirty seconds of startup. I'm not sure if that should be considered wrong or not, but in any case it doesn't seem to be what you're describing. regards, tom lane ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [PATCHES] Fix for erroneous warning on Shutdown
Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Shutdown was via CTRL-Cmake a difference? Wouldn't think so. I can force the message to appear if I do a *manual* CHECKPOINT command within thirty seconds of startup. I'm not sure if that should be considered wrong or not, but in any case it doesn't seem to be what you're describing. Just to confirm that Simon is not suffering this uniquely, I saw this the other day on Windows, I believe - meant to report it but it got away from me. cheers andrew ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] Fix for erroneous warning on Shutdown
Andrew Dunstan [EMAIL PROTECTED] writes: Just to confirm that Simon is not suffering this uniquely, I saw this the other day on Windows, I believe - meant to report it but it got away from me. Oh, I bet I know what's going on --- are you guys launching the postmaster in a console window and then typing ^C to shut it down? Depending on your shell that might result in SIGINT being delivered not only to the postmaster but to all its children. Backends, if any, will take that as a query-cancel while the bgwriter will take it as a checkpoint request. Which could easily result in the too-frequent-checkpoints message. The only real fix I can see for this is to use some other signal than SIGINT to transmit checkpoint requests to the bgwriter. Maybe we could piggyback all bgwriter requests onto SIGUSR2, comparable to the way that signals up to the postmaster are now handled. Not sure how important it is though. regards, tom lane ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [PATCHES] Fix for erroneous warning on Shutdown
On Tue, 2004-06-15 at 21:04, Tom Lane wrote: Andrew Dunstan [EMAIL PROTECTED] writes: Just to confirm that Simon is not suffering this uniquely, I saw this the other day on Windows, I believe - meant to report it but it got away from me. Oh, I bet I know what's going on --- are you guys launching the postmaster in a console window and then typing ^C to shut it down? Depending on your shell that might result in SIGINT being delivered not only to the postmaster but to all its children. Backends, if any, will take that as a query-cancel while the bgwriter will take it as a checkpoint request. Which could easily result in the too-frequent-checkpoints message. The only real fix I can see for this is to use some other signal than SIGINT to transmit checkpoint requests to the bgwriter. Maybe we could piggyback all bgwriter requests onto SIGUSR2, comparable to the way that signals up to the postmaster are now handled. Not sure how important it is though. Sounds like the cause to me...that would explain why it has just started happening... Its minor then... I would only do that for testing... Regards, Simon ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] Fix for erroneous warning on Shutdown
Bruce Momjian [EMAIL PROTECTED] writes: I just did CHECKPOINT;CHECKPOINT and got the warning in the logs. This needs to be fixed. See code: /* * Ideally we should only warn if this checkpoint was * requested due to running out of segment files, and not * if it was manually requested. However we can't tell the * difference with the current signalling mechanism. */ I could argue that a client-driven process that issues CHECKPOINT every few seconds is equally deserving of a warning. The only thing wrong is that the HINT is inapplicable ... but that's why it's a HINT and not part of the main message. regards, tom lane ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [PATCHES] Fix for erroneous warning on Shutdown
Bruce Momjian [EMAIL PROTECTED] writes: Tom Lane wrote: I could argue that a client-driven process that issues CHECKPOINT every few seconds is equally deserving of a warning. The only thing wrong is that the HINT is inapplicable ... but that's why it's a HINT and not part of the main message. Also consider they could have issued a checkpoint right after the system did one. Yuck. When I added the warning I hoped to only have it happen for full logs and not CHECKPOINT, but I guess I couldn't and someone else realized that and added that clearer comment, or originally I could do that, but since it has been moved into the bgwriter, it can't anymore. I believe the original implementation in the postmaster had a somewhat different set of bugs ;-). IIRC it did not react to manual checkpoints but it did confuse WAL checkpoints with timeout-driven checkpoints. The present bgwriter can distinguish the third but not the first two. If we were willing to take the time to generalize the backend-to-bgwriter signaling mechanism then we could distinguish WAL-driven checkpoints from manually issued checkpoints. I'm sort of intending to do that anyway. The question stands though: why isn't it appropriate to warn of overly-frequently-issued manual checkpoints? regards, tom lane ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [PATCHES] Fix for erroneous warning on Shutdown
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Tom Lane wrote: I could argue that a client-driven process that issues CHECKPOINT every few seconds is equally deserving of a warning. The only thing wrong is that the HINT is inapplicable ... but that's why it's a HINT and not part of the main message. Also consider they could have issued a checkpoint right after the system did one. Yuck. When I added the warning I hoped to only have it happen for full logs and not CHECKPOINT, but I guess I couldn't and someone else realized that and added that clearer comment, or originally I could do that, but since it has been moved into the bgwriter, it can't anymore. I believe the original implementation in the postmaster had a somewhat different set of bugs ;-). IIRC it did not react to manual checkpoints but it did confuse WAL checkpoints with timeout-driven checkpoints. The present bgwriter can distinguish the third but not the first two. If we were willing to take the time to generalize the backend-to-bgwriter signaling mechanism then we could distinguish WAL-driven checkpoints from manually issued checkpoints. I'm sort of intending to do that anyway. The question stands though: why isn't it appropriate to warn of overly-frequently-issued manual checkpoints? Imagine pgbench issuing a checkpoint, which it does. That could trigger the warning when the parameter really shouldn't be increased. To me, the warning is for cases when you are filling up the WAL logs too quickly and checkpoints are happening too frequently. If a user is doing checkpoints, it isn't anything increasing the checkpoint segments is going to help. If you can't fix the code to distinguish between manual and wal-full checkpoints, we can adjust the warning logic to warn when there are 3 checkpoints in a short period, rather than just two. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [PATCHES] Fix for erroneous warning on Shutdown
Bruce Momjian [EMAIL PROTECTED] writes: Tom Lane wrote: ... The question stands though: why isn't it appropriate to warn of overly-frequently-issued manual checkpoints? ... the warning is for cases when you are filling up the WAL logs too quickly and checkpoints are happening too frequently. If a user is doing checkpoints, it isn't anything increasing the checkpoint segments is going to help. No, I think the warning is for when checkpoints are happening too frequently, period. An overly small checkpoint_segments setting is one possible cause of that, but the performance penalty from too many checkpoints is just as bad no matter what's causing it. (Remember that a checkpoint not only forces I/O in itself, but significantly increases subsequent WAL traffic because of needing to dump whole page images into WAL.) How do you feel about improving the signaling mechanism but using it just to vary the HINT? LOG: checkpoints are occurring too frequently (nn seconds apart) HINT: Consider increasing the configuration parameter checkpoint_segments. LOG: checkpoints are occurring too frequently (nn seconds apart) HINT: Issuing explicit CHECKPOINTs so often is really expensive. and so on. regards, tom lane ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] Fix for erroneous warning on Shutdown
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Tom Lane wrote: ... The question stands though: why isn't it appropriate to warn of overly-frequently-issued manual checkpoints? ... the warning is for cases when you are filling up the WAL logs too quickly and checkpoints are happening too frequently. If a user is doing checkpoints, it isn't anything increasing the checkpoint segments is going to help. No, I think the warning is for when checkpoints are happening too frequently, period. An overly small checkpoint_segments setting is one possible cause of that, but the performance penalty from too many checkpoints is just as bad no matter what's causing it. (Remember that a checkpoint not only forces I/O in itself, but significantly increases subsequent WAL traffic because of needing to dump whole page images into WAL.) How do you feel about improving the signaling mechanism but using it just to vary the HINT? LOG: checkpoints are occurring too frequently (nn seconds apart) HINT: Consider increasing the configuration parameter checkpoint_segments. LOG: checkpoints are occurring too frequently (nn seconds apart) HINT: Issuing explicit CHECKPOINTs so often is really expensive. Sure, fine by me. My only point is that we need something to tell people they need to increase their checkpoint_segments. If we add other warnings, that is fine too. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [PATCHES] Fix for erroneous warning on Shutdown
Simon Riggs [EMAIL PROTECTED] writes: Minor patch to correct erroneous warning in cvs tip, believed to be a very minor regression. This patch is wrong; it effectively disables the warning altogether. When a shutdown was requested within CHECKPOINT_SECONDS of a checkpoint, the shutdown code in the bgwriter (which has does checkpointing) would issue the erroneous message: I don't think so ... the shutdown path doesn't go through this code. Could you take another look at the cause of what you saw? regards, tom lane ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [PATCHES] Fix for erroneous warning on Shutdown
Patch rejected, asking for more research. --- Simon Riggs wrote: Minor patch to correct erroneous warning in cvs tip, believed to be a very minor regression. When a shutdown was requested within CHECKPOINT_SECONDS of a checkpoint, the shutdown code in the bgwriter (which has does checkpointing) would issue the erroneous message: LOG: checkpoints are occurring too frequently (%d seconds apart) HINT: Consider increasing the configuration parameter checkpoint_segments. Clearly, this should only occur when specific checkpoint requests have been made, shutdown checkpoints should not be included in the warning. Best regards, Simon Riggs [ Attachment, skipping... ] ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
