OK, unless anyone has anything else to add I'm going to assume that the 
project maintainers are OK with message IDs and I'll start the process.

To summarize:
1. I'm going to do some minor surgery to the logger so that the log 
format is pinned an not user modifiable.  This is necessary to ensure 
that message IDs can be substituted into the string.
2. Message IDs will have the following format: VDSM#####
3. Message IDs will just be a simple up-counter across all of VDSM.
4. Existing strings will be converted to translatable strings.  The 
string itself won't be changed it will just be wrapped by _(...) so that 
gettext will work.
5. Message IDs may be documented, somewhere not sure where yet, with a 
brief explanation (if an explanation is appropriate).  This part might 
take some and; hopefully, the explanations will evolve.  I expect  many 
of the IDs to not have explanations right away though and I definitely 
don't want to put any bad explanations up or ones that haven't been vetted.


On 10/06/2011 03:29 PM, Keith Robertson wrote:
> On 10/06/2011 03:11 PM, Ayal Baron wrote:
>>>>>>>>> I would like to propose enhancing the VDSM source to emit
>>>>>>>>> message
>>>>>>>>> IDs in
>>>>>>>>> log messages with levels greater than DEBUG (i.e. INFO, ERROR,
>>>>>>>>> WARN,
>>>>>>>>> EXCEPTION, and CRITICAL). Messages IDs have been used
>>>>>>>>> successfully in
>>>>>>>>> many products and they can aid in debugging, documentation,
>>>>>>>>> root
>>>>>>>>> cause
>>>>>>>>> analysis, and internationalization.
>>>>>>>>> I have scanned the source and have determined that there are
>>>>>>>>> roughly 525
>>>>>>>>> calls to log.[error|warn|info|critical|exception] and I am
>>>>>>>>> volunteering
>>>>>>>>> to go through and add IDs to them all but, before I do I want
>>>>>>>>> to get
>>>>>>>>> buy-in from the project maintainers. While adding IDs to the
>> wrt i18n, I'm not entirely convinced.
> Yeah, i18n is one reason to do it but it is only relevant if you take 
> the time to create lang-packs for different nationalities (which can 
> be expensive). However, it is a cakewalk to code for in Python 
> _("string here").  So you can code for it ... and if someone ever 
> wanted to take the time create a lang-pack you'd be set.
>>   Every line we log has the module name and line (never localized) 
>> which logged it so effectively we already have this info.
>> However, the ID would remain constant where lines will change place 
>> between versions so it makes it easier.
>> I also agree about the lookup and being able to write simpler parsers.
>> So I'm in.
> Cool.
>>>>>>>>> messages I
>>>>>>>>> would also enable them for i18n.
>>>>>>>>> I have written a rough plan for adding messages IDs to VDSM
>>>>>>>>> below
>>>>>>>>> and
>>>>>>>>> I've provided some examples of products/projects that use
>>>>>>>>> message
>>>>>>>>> IDs.
>>>>>>>>> Examples of Projects that Use Message IDs:
>>>>>>>>> - JBoss 7 is adopting Message IDs for all of its logging. See
>>>>>>>>> the
>>>>>>>>> annotations (e.g. @Message(id = 12000, ) in the following
>>>>>>>>> piece of
>>>>>>>>> source:
>>>>>>>>> https://github.com/jbossas/jboss-as/blob/master/process-controller/src/main/java/org/jboss/as/process/ProcessLogger.java
>>>>>>>>> - Almost all IBM products use message IDs and they do a fairly
>>>>>>>>> good
>>>>>>>>> job
>>>>>>>>> of documenting them. Google has indexed them all and you can
>>>>>>>>> simply
>>>>>>>>> type
>>>>>>>>> them in to a search and get a description of the problem that
>>>>>>>>> is
>>>>>>>>> associated with the ID.
>>>>>>>>> http://publib.boulder.ibm.com/infocenter/tivihelp/v42r1/topic/com.ibm.omegamon.stor.doc/GC23-9705-0484.htm
>>>>>>>>> Example of a Log Statement with a Message ID in Python:
>>>>>>>>> FORMAT = '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s):
>>>>>>>>> %(message)s'
>>>>>>>>> logging.basicConfig(format=FORMAT)
>>>>>>>>> logger = logging.getLogger()
>>>>>>>>> logger.warning('NFS problem. Unable to mount.', extra={
>>>>>>>>> 'messageid' :
>>>>>>>>> 'VDSM1000'})
>>>>>>>>> // Produces
>>>>>>>>> 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem.
>>>>>>>>> Unable
>>>>>>>>> to mount.
>>>>>>>>> Example of a Log Statement with a Message ID in Python with
>>>>>>>>> i18n:
>>>>>>>>> logger.warning(_('NFS problem. Unable to mount.'), extra={
>>>>>>>>> 'messageid' :
>>>>>>>>> 'VDSM1000'})
>>>>>>>>> Example of an i18n Log Message:
>>>>>>>>> Chinese: 2011年10月5日13:24:43,18011警告(VDSM1000):NFS问 
>>>>>>>>> 题。无法
>>>>>>>>> 安 装。
>>>>>>>>> English: 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS
>>>>>>>>> problem.
>>>>>>>>> Unable to mount.
>>>>>>>>> A Rough Plan for adding Message IDs to VDSM:
>>>>>>>>> Step 0: Decide on a logging format of the ID itself:
>>>>>>>>> - JBoss is using "JBAS#####". I am thinking that VDSM could
>>>>>>>>> probably do
>>>>>>>>> something similar "VDSM####".
>>>>>>>>> Step 1: Decide where to list and document all of the IDs.
>>>>>>>>> - It could be as simple as a page on VDSM's fedorahosted wiki.
>>>>>>>>> Step 2: Pin the format of the log messages:
>>>>>>>>> - Currently VDSM allows a user configurable Python log format.
>>>>>>>>> We
>>>>>>>>> would
>>>>>>>>> need to pin the format so that it includes a substitution
>>>>>>>>> variable for
>>>>>>>>> the message ID.
>>>>>>>>> Example: '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s):
>>>>>>>>> %(message)s'
>>>>>>>>> Step 4: Plow through the VDSM source and add message IDs.
>>>>>>>>> Cheers,
>>>>>>>>> Keith
>>>>>>>>> _______________________________________________
>>>>>>>>> vdsm-devel mailing list
>>>>>>>>> vdsm-devel@lists.fedorahosted.org
>>>>>>>>> https://fedorahosted.org/mailman/listinfo/vdsm-devel
>>>>>>>> Seems like a nice idea. But how does it affect developers. How
>>>>>>>> do
>>>>>>>> they choose mIDs? How do you prevent collisions? How are they
>>>>>>>> different from simple gettext?
>>>>>> As I never used this myself so I can't really imagine the all
>>>>>> benefits
>>>>>> of using such a system, at least enough to warrant all the
>>>>>> tedious
>>>>>> work. If you could just list out some of the use cases it'll be
>>>>>> most
>>>>>> helpful.
>>>>>> For instance I assume grepping for an ID is easier then grepping
>>>>>> for a
>>>>>> message especially in multiple versions where the message might
>>>>>> have
>>>>>> changed.
>>>>> Sure.  As far as the tedious part, yes, adding message IDs to an
>>>>> existing source tree is a bit of a PITA but the 525 invocations in
>>>>> VDSM
>>>>> isn't all that bad.  I've seen much worse.  Plus, once it's done
>>>>> the
>>>>> effort is much smaller (new lines of code only).
>>>>> As for the value (I've tried to itemize a few common reasons for
>>>>> doing it):
>>>>> Value: Message IDs allow you to use gettext globalize your
>>>>> messages. If
>>>>> you just used gettext and pulled the strings out of your code for
>>>>> globalization then you might end up with something like this in
>>>>> vdsm.log...
>>>>>       错误:问题安装驱动器
>>>>> the English translation (according to google) is:
>>>>>       ERROR: Problem mounting drive
>>>>> Unless you can read Chinese, then the message above is completely
>>>>> unhelpful to a support engineer.  The converse is true also.
>>>>>   Consider a
>>>>> non-english speaking person trying to decipher "ERROR: Problem
>>>>> mounting
>>>>> drive".  With a message ID; however, you get this...
>>>>>       错误(VDSM1000):安装驱动问题
>>>> Is it possible to just hash the main string before
>>>> internationalization?
>>>> So basically:
>>>> fprintf(f, "[%s] ERROR(VDSM%04x) %s", date_str, crc32(message),
>>>> message);
>>> Yes, a hash of the string is possible.  However, would make it hard
>>> for
>>> someone to document the ID because they would have to hash the string
>>> to
>>> get the ID.
>>> One of the features of message ID/codes is that they are
>>> documentable.
>>>> Or the equivalent Python.
>>>>> Value: Message IDs allow other applications to perform actions
>>>>> based on
>>>>> what is in your logs.  On the mainframe, this is extremely common.
>>>>> You'll have one program watching the logs for a particular ID(s).
>>>>>   When
>>>>> the program sees the ID it will look in a dictionary and decide
>>>>> what (if
>>>>> anything) needs to be done.  On linux, logwatch, is frequently
>>>>> used for
>>>>> this purpose and message IDs make this process much simpler
>>>>> because the
>>>>> program doesn't have to write complicated REGEXs to look for
>>>>> arbitrary
>>>>> strings it can just scan for an ID.
>>>>> Value: Message IDs generally have the same meaning across releases
>>>>> of
>>>>> the product.  This means that the developer can change the text of
>>>>> the
>>>>> log message without worrying about breaking someone's automation.
>>>>> Example:
>>>>>       Version 1 of the product- ERROR(VDSM1000): Problem mounting
>>>>>       drive.
>>>>>       Version 2 of the product- ERROR(VDSM1000): unable to contact
>>>>> for mounting. Try blah blah.
>>>> Does this mean that logging messages are an ABI that needs to be
>>>> maintained over time?
>>> The degree to which message IDs need to be maintained is generally a
>>> function of how dynamic the code is and how much you want to document
>>> what the ID actually means.  If, on the WIKI wherein you define
>>> messages, you elect to say [1] then this "ABI" doesn't need any
>>> maintenance whatsoever.
>>> [1] VDSM1000 == "Call support or post to the forum to figure out what
>>> the heck the program just did"
>>>> That's probably very difficult to do right.
>>> Lots of other software programs have been successful at it [1], [2].
>>>   I
>>> would prefer to think of it in terms of how easy we want to make it
>>> for
>>> people to self diagnose.  Also, without message IDs supporting source
>>> that is i18n is nearly impossible unless everyone is multilingual.
>>> [1]
>>> http://www.cisco.com/en/US/products/sw/netmgtsw/ps1982/products_command_reference_chapter09186a0080087439.html
>>> [2] http://mail.google.com/support/bin/answer.py?answer=12336
>>>> Regards,
>>>> Anthony Liguori
>>>>> Value: If you do it right search engines will index your message
>>>>> IDs and
>>>>> take users to a page where they can hopefully get more detail on
>>>>> the
>>>>> issue and potentially "self solve" the problem without calling
>>>>> support.
>>>>> Put this into a search engine... "KDFAP0020S"
>>>>> Value: Message IDs allow you to easily describe flows.  You might
>>>>> be
>>>>> able to say... "if you saw ID123 then look farther back for ID124
>>>>> and if
>>>>> you see ID124 then the problem is _probably_ XYZ."
>>>>> I am probably not doing enough justice to the value of message IDs
>>>>> but,
>>>>> I will say that lots of very large companies use them in complex
>>>>> software applications to solve these and other issues.
>>>>> Cheers,
>>>>> Keith
>>>>> _______________________________________________
>>>>> vdsm-devel mailing list
>>>>> vdsm-devel@lists.fedorahosted.org
>>>>> https://fedorahosted.org/mailman/listinfo/vdsm-devel
>>> _______________________________________________
>>> vdsm-devel mailing list
>>> vdsm-devel@lists.fedorahosted.org
>>> https://fedorahosted.org/mailman/listinfo/vdsm-devel
>> _______________________________________________
>> vdsm-devel mailing list
>> vdsm-devel@lists.fedorahosted.org
>> https://fedorahosted.org/mailman/listinfo/vdsm-devel

vdsm-devel mailing list

Reply via email to