On Thu, Oct 30, 2008 at 11:26:14PM -0700, Steven Dake wrote: > There are two types of messages. Those intended for users/admins and > those intended for developers. > > Both of these message types should always be recorded *somewhere*. The > entire concept of "LOG_LEVEL_DEBUG" is dubious to me. If you want to > stick with that symanetic and definition that is fine, but really a > LOG_LEVEL_DEBUG means "this message is for the developer". These > messages should be recorded and stored when a process segfaults, aborts > due to assertion, or at administrative request. Since the frequency of > these messages is high there is no other option for recording them since > they must _always_ be recorded for the purposes of debugging a field > failure. Recording to disk or syslog has significant performance > impact. > > The only solution for these types of messages is to record them into a > flight recorder buffer which can be dumped: > 1) at segv > 2) at sigabt > 3) at administrative request > > This is a fundamental difference in how we have approached logging > debugging messages in the past but will lead to the ability to ensure we > _always_ have debug trace data available instead of telling the > user/admin "Go turn on debug and hope you can reproduce that error and > btw since 100000k messages are logged your disk will fill up with > irrelevant debug messages and your system will perform like mud". > > Logging these in memory is the only solution that I see as suitable and > in all cases they should be filtered from any output source such as > stderr, file, or syslog.
There's a difference between high volume trace debug data stored in memory, and low volume informational debug data that can be easily written to a file. Both kinds of data can be useful. My programs are simple enough that low volume informational debug data is enough for me to identify and fix a problem. So, low volume informational data is all I produce. It can be useful to write this data to a file. Your program is complex enough that high volume trace debug data is usually needed for you to identify and fix a problem. So, high volume trace data is all you produce. This is too much data to write to a file (by the running program). So, we're using "DEBUG" to refer to different things. We need to define two different levels (just for clarity in this discussion): . DEBUGLO is low volume informational data like I use . DEBUGHI is high volume trace data like you use DEBUGHI messages wouldn't ever be logged to files by the program while running. DEBUGLO messages could be, though, if the user configured it. So, circling back around, how should a user configure DEBUGLO messages to appear in syslog or a logfile? In particular, what would they enter in the cluster.conf <logging/> section? My suggestion is: syslog_level=foo logfile_level=bar where foo and bar are one of the standard priority names in syslog.h. So, if a user wanted DEBUGLO messages to appear in daemon.log, they'd set logging/<daemon>/logfile_level=debug and if they wanted DEBUGLO messages to appear in /var/log/messages, logging/<daemon>/syslog_level=debug (Note that "debug" means DEBUGLO here because DEBUGHI messages are only saved in memory, not to files by a running program.) There's another separate question I have about corosync, and that's whether you could identify some limited number of messages that would be appropriate for DEBUGLO? They would be used by non-experts to do some rough debugging of problems, and by experts to narrow down a problem before digging into the high volume trace data. I'd suggest that a good starting point for DEBUGLO would be the data that openais has historically put in /var/log/messages. Data that helps you quickly triage a problem (or verify that things are happening correctly) without stepping through all the trace data. _______________________________________________ Openais mailing list Openais@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/openais
