Re: [Openais] [Cluster-devel] cluster/logging settings
On Thu, 2008-11-06 at 19:42 -0700, Steven Dake wrote: On Tue, 2008-11-04 at 14:58 -0600, David Teigland wrote: On Thu, Oct 30, 2008 at 11:26:14PM -0700, Steven Dake wrote: There are two types of messages. Those intended for users/admins and those intended for developers. Both of these message types should always be recorded *somewhere*. The entire concept of LOG_LEVEL_DEBUG is dubious to me. If you want to stick with that symanetic and definition that is fine, but really a LOG_LEVEL_DEBUG means this message is for the developer. These messages should be recorded and stored when a process segfaults, aborts due to assertion, or at administrative request. Since the frequency of these messages is high there is no other option for recording them since they must _always_ be recorded for the purposes of debugging a field failure. Recording to disk or syslog has significant performance impact. The only solution for these types of messages is to record them into a flight recorder buffer which can be dumped: 1) at segv 2) at sigabt 3) at administrative request This is a fundamental difference in how we have approached logging debugging messages in the past but will lead to the ability to ensure we _always_ have debug trace data available instead of telling the user/admin Go turn on debug and hope you can reproduce that error and btw since 10k messages are logged your disk will fill up with irrelevant debug messages and your system will perform like mud. Logging these in memory is the only solution that I see as suitable and in all cases they should be filtered from any output source such as stderr, file, or syslog. There's a difference between high volume trace debug data stored in memory, and low volume informational debug data that can be easily written to a file. Both kinds of data can be useful. My programs are simple enough that low volume informational debug data is enough for me to identify and fix a problem. So, low volume informational data is all I produce. It can be useful to write this data to a file. Your program is complex enough that high volume trace debug data is usually needed for you to identify and fix a problem. So, high volume trace data is all you produce. This is too much data to write to a file (by the running program). So, we're using DEBUG to refer to different things. We need to define two different levels (just for clarity in this discussion): . DEBUGLO is low volume informational data like I use . DEBUGHI is high volume trace data like you use DEBUGHI messages wouldn't ever be logged to files by the program while running. DEBUGLO messages could be, though, if the user configured it. So, circling back around, how should a user configure DEBUGLO messages to appear in syslog or a logfile? In particular, what would they enter in the cluster.conf logging/ section? My suggestion is: syslog_level=foo logfile_level=bar where foo and bar are one of the standard priority names in syslog.h. So, if a user wanted DEBUGLO messages to appear in daemon.log, they'd set logging/daemon/logfile_level=debug and if they wanted DEBUGLO messages to appear in /var/log/messages, logging/daemon/syslog_level=debug (Note that debug means DEBUGLO here because DEBUGHI messages are only saved in memory, not to files by a running program.) There's another separate question I have about corosync, and that's whether you could identify some limited number of messages that would be appropriate for DEBUGLO? They would be used by non-experts to do some rough debugging of problems, and by experts to narrow down a problem before digging into the high volume trace data. I'd suggest that a good starting point for DEBUGLO would be the data that openais has historically put in /var/log/messages. Data that helps you quickly triage a problem (or verify that things are happening correctly) without stepping through all the trace data. I thought about this a few days and this seems to make sense. The priority (ie: specifically LOG_LEVEL_DEBUG messages) would be selectable per output medium (syslog, file, or stderr) and the high performance tracing would use the trace macros or log_rec and only ever go to memory. Each output medium if configured would filter based upon a filtering priority which is LOG_LEVEL_DEBUG or others. This does require further API changes that Fabio would have to manage with. Ok good that we are heading towards the same goal. Before you fully dive into this code, let's finalize the config options for logging across the board. The API change at this point is no problem since cluster.git didn't move to the logsys v2 API yet. I'll send a summary of the different thread in another email to the different mailing lists. Users might have different ideas as well.
Re: [Openais] [Cluster-devel] cluster/logging settings
On Thu, Oct 30, 2008 at 11:26:14PM -0700, Steven Dake wrote: There are two types of messages. Those intended for users/admins and those intended for developers. Both of these message types should always be recorded *somewhere*. The entire concept of LOG_LEVEL_DEBUG is dubious to me. If you want to stick with that symanetic and definition that is fine, but really a LOG_LEVEL_DEBUG means this message is for the developer. These messages should be recorded and stored when a process segfaults, aborts due to assertion, or at administrative request. Since the frequency of these messages is high there is no other option for recording them since they must _always_ be recorded for the purposes of debugging a field failure. Recording to disk or syslog has significant performance impact. The only solution for these types of messages is to record them into a flight recorder buffer which can be dumped: 1) at segv 2) at sigabt 3) at administrative request This is a fundamental difference in how we have approached logging debugging messages in the past but will lead to the ability to ensure we _always_ have debug trace data available instead of telling the user/admin Go turn on debug and hope you can reproduce that error and btw since 10k messages are logged your disk will fill up with irrelevant debug messages and your system will perform like mud. Logging these in memory is the only solution that I see as suitable and in all cases they should be filtered from any output source such as stderr, file, or syslog. There's a difference between high volume trace debug data stored in memory, and low volume informational debug data that can be easily written to a file. Both kinds of data can be useful. My programs are simple enough that low volume informational debug data is enough for me to identify and fix a problem. So, low volume informational data is all I produce. It can be useful to write this data to a file. Your program is complex enough that high volume trace debug data is usually needed for you to identify and fix a problem. So, high volume trace data is all you produce. This is too much data to write to a file (by the running program). So, we're using DEBUG to refer to different things. We need to define two different levels (just for clarity in this discussion): . DEBUGLO is low volume informational data like I use . DEBUGHI is high volume trace data like you use DEBUGHI messages wouldn't ever be logged to files by the program while running. DEBUGLO messages could be, though, if the user configured it. So, circling back around, how should a user configure DEBUGLO messages to appear in syslog or a logfile? In particular, what would they enter in the cluster.conf logging/ section? My suggestion is: syslog_level=foo logfile_level=bar where foo and bar are one of the standard priority names in syslog.h. So, if a user wanted DEBUGLO messages to appear in daemon.log, they'd set logging/daemon/logfile_level=debug and if they wanted DEBUGLO messages to appear in /var/log/messages, logging/daemon/syslog_level=debug (Note that debug means DEBUGLO here because DEBUGHI messages are only saved in memory, not to files by a running program.) There's another separate question I have about corosync, and that's whether you could identify some limited number of messages that would be appropriate for DEBUGLO? They would be used by non-experts to do some rough debugging of problems, and by experts to narrow down a problem before digging into the high volume trace data. I'd suggest that a good starting point for DEBUGLO would be the data that openais has historically put in /var/log/messages. Data that helps you quickly triage a problem (or verify that things are happening correctly) without stepping through all the trace data. ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] [Cluster-devel] cluster/logging settings
On Tue, Nov 04, 2008 at 02:58:47PM -0600, David Teigland wrote: the cluster.conf logging/ section? My suggestion is: syslog_level=foo logfile_level=bar FWIW, I'm not set on this if someone has a better suggestion. I just want something unambiguous. debug=on has been shown to mean something different to everyone. Dave ___ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
Re: [Openais] [Cluster-devel] cluster/logging settings
On Fri, 2008-10-31 at 06:28 +0100, Fabio M. Di Nitto wrote:
On Thu, 30 Oct 2008, David Teigland wrote:
If I set log_level to LOG_INFO, I do expect to see both critblabla and
infoblabla on all selected outputs.
Note that you've said log_level, but it's actually syslog_level, which
might lead people to different conclusions about what the option does.
I meant syslog_level. You are right.
void logt_print(int level, char *fmt, ...)
{
va_list ap;
if (level logt_priority)
return;
we send a logt_print(LOG_DEBUG..
but if syslog_level is set to LOG_INFO, that message will not make it
anywhere, even if debug=on.
If I set log_level to LOG_DEBUG, I clearly expect to see debugblabla
passing throgh as well.
(except if filtering to syslog is enabled, but we already agreed on this
as required feature so I won't mention it as special case anylonger).
I often expect that enabling LOG_DEBUG as priority, it will also enable
debugging code in general and viceversa. If I set debug=on, I want to be
able to catch LOG_DEBUG automatically, because i assume that most of the
LOG_DEBUG is wrapped between if(debug) { } statements. I don't feel the
need to set two options to enable full debugging.
You don't need to. Under my scheme, you set debug=on, and full debugging
appears in the log file where most people want it, and it doesn't appear
in syslog where most people wouldn't want it.
Not really.. see below the code snippet...
If someone *does* want all
debugging to appear in syslog, then they set debug=on and
syslog_level=debug to get it... (and once they see the result, they'll
change it back, because it's really not nice to see all that in
/var/log/messages.)
We all agreed on this already that's why we have the
LOG_MODE_FILTER_DEBUG_FROM_SYSLOG that we should probably make a config
option in cluster.conf or simply avoid by default.
Under your scheme, syslog_level and debug change each other in confusing
and redundant ways.
It's not really confusing.. debug enables LOG_DEBUG and LOG_DEBUG enables
debug. The only thing that might be confusing is that you need to take
into account global debug, (sub)system debug setting and priority.
By setting debug=on you automatically get all
debugging in syslog (in addition to the logfile usually), which is not
where we wanted it... So we added yet another complication:
LOG_MODE_FILTER_DEBUG_FROM_SYSLOG, which counteracts the bad effects of
the debug/syslog_level interaction which we didn't really want in the
first place. See how complicated this gets when you have one option
changing another one, but not quite the way you want, so you add another
flag to work around the unintended effects of one setting implicitly
changing another? It's bad all around: just keep them independent and all
the pain goes away.
(We wouldn't need FILTER_DEBUG_FROM_SYSLOG in my scheme because you
control each source and destination point explicitly; you say exactly what
you want, and get it.)
Unless I am misreading your code:
void logt_print(int level, char *fmt, ...)
{
va_list ap;
if (level logt_priority)
return;
we send a logt_print(LOG_DEBUG..
if syslog_level is set to LOG_INFO, that message will not make it
anywhere, even if debug=on.
maybe debug=on should really set syslog_level=LOG_DEBUG and
LOG_MODE_FILTER_DEBUG_FROM_SYSLOG to make this really happens
Here is my take on how logging should work and does work in logsysv2.
There are two types of messages. Those intended for users/admins and
those intended for developers.
Both of these message types should always be recorded *somewhere*. The
entire concept of LOG_LEVEL_DEBUG is dubious to me. If you want to
stick with that symanetic and definition that is fine, but really a
LOG_LEVEL_DEBUG means this message is for the developer. These
messages should be recorded and stored when a process segfaults, aborts
due to assertion, or at administrative request. Since the frequency of
these messages is high there is no other option for recording them since
they must _always_ be recorded for the purposes of debugging a field
failure. Recording to disk or syslog has significant performance
impact.
The only solution for these types of messages is to record them into a
flight recorder buffer which can be dumped:
1) at segv
2) at sigabt
3) at administrative request
This is a fundamental difference in how we have approached logging
debugging messages in the past but will lead to the ability to ensure we
_always_ have debug trace data available instead of telling the
user/admin Go turn on debug and hope you can reproduce that error and
btw since 10k messages are logged your disk will fill up with
irrelevant debug messages and your system will perform like mud.
Logging these in memory is the only solution that I see as suitable and
in all
