Re: [ceph-users] Very chatty MON logs: Is this normal?
On 06/19/2015 11:16 AM, Daniel Schneller wrote: On 2015-06-18 09:53:54 +, Joao Eduardo Luis said: Setting 'mon debug = 0/5' should be okay. Unless you see that setting '/5' impacts your performance and/or memory consumption, you should leave that be. '0/5' means 'output only debug 0 or lower to the logs; keep the last 1000 debug level 5 or lower in memory in case of a crash'. Your logs will not be as heavily populated but, if for some reason the daemon crashes, you get quite a few of debug information to help track down the source of the problem. Great, will do. Just for my understanding re/ memory: If this is a ring buffer for the last 1 events, shouldn't that be a somewhat fixed amount of memory? How would it negatively affect the MON's consumption? Assuming it works that way, once they have been running for a few days or weeks, these buffers would be full of events anyway, just more aged ones if the memory level was lower? Daniel From briefly taking a peak at 'src/log/*', this looks like it is a linked list rather than a buffer ring. So, given it will always be capped at 10k events, there's a fixed amount of memory it will consume in the worst case (when you have 10k events). But if you have bare minimum activity in the logs, said memory consumption should be lower, or at most slowly growing as the queue grows. Although I was not obvious, my initial thought was that someone with debug levels set at 0/0 would certainly be surprised if, after setting 0/5, the daemon's memory consumption started to grow. In retrospect, 10k log messages should not take more than a handful of MBs, and should not have any impact at all as long as you're not provisioning your monitor's memory in the dozens of MBs. -Joao ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Very chatty MON logs: Is this normal?
On 2015-06-18 09:53:54 +, Joao Eduardo Luis said: Setting 'mon debug = 0/5' should be okay. Unless you see that setting '/5' impacts your performance and/or memory consumption, you should leave that be. '0/5' means 'output only debug 0 or lower to the logs; keep the last 1000 debug level 5 or lower in memory in case of a crash'. Your logs will not be as heavily populated but, if for some reason the daemon crashes, you get quite a few of debug information to help track down the source of the problem. Great, will do. Just for my understanding re/ memory: If this is a ring buffer for the last 1 events, shouldn't that be a somewhat fixed amount of memory? How would it negatively affect the MON's consumption? Assuming it works that way, once they have been running for a few days or weeks, these buffers would be full of events anyway, just more aged ones if the memory level was lower? Daniel ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Very chatty MON logs: Is this normal?
On 06/17/2015 08:30 PM, Somnath Roy wrote: However, I'd rather not set the level to 0/0, as that would disable all logging from the MONs I don't think so. All the error scenarios and stack trace (in case of crash) are supposed to be logged with log level 0. But, generally, we need the highest log level (say 20) to get all the information when something to debug. So, I doubt how beneficial it will be to enable logging for some intermediate levels. Probably, there is no guideline for these log level too which developer should follow strictly. I don't think this is documented anywhere, but for a while now we've been using roughly this approach to debug levels: -1 - errors. 0 - info you really want in the log each time it happens. 1 - info that should be outputted by default should be stuff that don't happen often and is quite important to get to the logs when it happens. 5 - important but happens a bit often not to output as 1 10 - gross majority of debug messages in the monitor 20 - debug that could impact monitor performance severely (e.g., debug from inside a loop) 30 - debug that you should not need unless you're really looking for it It is fairly common a developer will ask you for 'debug mon = 10' in order to catch all debug messages at levels 5 and 10, because those are the ones that usually pay off when tracking down issues. But given this is left pretty much to the developer's criteria, different services may use different levels of verbosity for different things, and you may need a higher debug level to get info out of some parts of the code than others. In this particular case, the message that is being outputted should, imo, be on debug level 5 instead of 1. We used to output a lot of stuff on debug level 1, but have been moving away from that; there are still artifacts though, and this is one of them. Setting 'mon debug = 0/5' should be okay. Unless you see that setting '/5' impacts your performance and/or memory consumption, you should leave that be. '0/5' means 'output only debug 0 or lower to the logs; keep the last 1000 debug level 5 or lower in memory in case of a crash'. Your logs will not be as heavily populated but, if for some reason the daemon crashes, you get quite a few of debug information to help track down the source of the problem. HTH, -Joao Thanks Regards Somnath -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Daniel Schneller Sent: Wednesday, June 17, 2015 12:11 PM To: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Very chatty MON logs: Is this normal? On 2015-06-17 18:52:51 +, Somnath Roy said: This is presently written from log level 1 onwards :-) So, only log level 0 will not log this.. Try, 'debug_mon = 0/0' in the conf file.. Yeah, once I had sent the mail I realized that 1 in the log line was the level. Had overlooked that before. However, I'd rather not set the level to 0/0, as that would disable all logging from the MONs. Now, I don't have enough knowledge on that part to say whether it is important enough to log at log level 1 , sorry :-( That would indeed be an interesting to know. Judging from the sheer amount, at least I have my doubts, because the cluster seems to be running without any issues. So I figure at least it isn't indicative of an immediate issue. Anyone with a little more definitve knowledge around? Should I create a bug ticket for this? Cheers, Daniel ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Very chatty MON logs: Is this normal?
However, I'd rather not set the level to 0/0, as that would disable all logging from the MONs I don't think so. All the error scenarios and stack trace (in case of crash) are supposed to be logged with log level 0. But, generally, we need the highest log level (say 20) to get all the information when something to debug. So, I doubt how beneficial it will be to enable logging for some intermediate levels. Probably, there is no guideline for these log level too which developer should follow strictly. Thanks Regards Somnath -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Daniel Schneller Sent: Wednesday, June 17, 2015 12:11 PM To: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Very chatty MON logs: Is this normal? On 2015-06-17 18:52:51 +, Somnath Roy said: This is presently written from log level 1 onwards :-) So, only log level 0 will not log this.. Try, 'debug_mon = 0/0' in the conf file.. Yeah, once I had sent the mail I realized that 1 in the log line was the level. Had overlooked that before. However, I'd rather not set the level to 0/0, as that would disable all logging from the MONs. Now, I don't have enough knowledge on that part to say whether it is important enough to log at log level 1 , sorry :-( That would indeed be an interesting to know. Judging from the sheer amount, at least I have my doubts, because the cluster seems to be running without any issues. So I figure at least it isn't indicative of an immediate issue. Anyone with a little more definitve knowledge around? Should I create a bug ticket for this? Cheers, Daniel ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Very chatty MON logs: Is this normal?
On 2015-06-17 18:52:51 +, Somnath Roy said: This is presently written from log level 1 onwards :-) So, only log level 0 will not log this.. Try, 'debug_mon = 0/0' in the conf file.. Yeah, once I had sent the mail I realized that 1 in the log line was the level. Had overlooked that before. However, I'd rather not set the level to 0/0, as that would disable all logging from the MONs. Now, I don't have enough knowledge on that part to say whether it is important enough to log at log level 1 , sorry :-( That would indeed be an interesting to know. Judging from the sheer amount, at least I have my doubts, because the cluster seems to be running without any issues. So I figure at least it isn't indicative of an immediate issue. Anyone with a little more definitve knowledge around? Should I create a bug ticket for this? Cheers, Daniel ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
