RE: [Syslog] New direction and proposed charter
Andrew, That's exactly our experience. 100% same story... Rainer -Original Message- From: Andrew Ross [mailto:[EMAIL PROTECTED] Sent: Thursday, November 24, 2005 11:20 PM To: Rainer Gerhards Cc: [EMAIL PROTECTED] Subject: RE: [Syslog] New direction and proposed charter Rainer, I agree that 3164 is only really valid with respect to the PRI. When we implemented it in Kiwi Syslog we found no device actually used the 3164 format exactly. Sometimes the hostname was there, sometimes not. Having to write parsing code to work out if a hostname was actually a TAG or not was a huge processing overhead. We now disable 3164 parsing by default. Cheers Andrew ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] New direction and proposed charter
Glenn, very interesting approach with the timestamp. I think your ideas can be the key to maintaining a lot of backwards compatibility by still retaining new functionality. First some bad news: I am not sure if by BSD syslog you are refering to RFC 3164 or a specific distribution of BSD. I have created a small script to test out your recommendation. I used FreeBSD stock syslogd as the receiver. It did NOT work as expected. There are two reasons a) (that) BSD syslogd takes the sender always from the system that send it b) even worse, when relaying, it puts Forwarded from hostname: into the hostname part (yes, with all that spaces) So while the idea sounds excellent, it does not work with stock FreeBSD syslogd. I am not sure about other BSD variants, nor have I checked the sysklogd package. I believe it will have less issues in this regard. This was the message I sent (via perl script): 148Oct 11 22:14:15 mymachine.example.com 1 ID47 2003T.003Z 'su root' failed for lonvick on /dev/pts/8 this was the raw message received after being relayed once by FreeBSD stock syslogd: 148Oct 11 22:14:15 Forwarded from 172.19.2.7: mymachine.example.com 1 ID47 2003T.003Z 'su root' failed for lonvick on /dev/pts/8 As you can see, the message is somewhat distorted - definitely enough for digital signatures to be broken. [Implementor's side-note: This can be fixed on a syslog-application layer level, far beyond the IETF scope. It's straightforward and easy to do, so it will probably happen.] Even though this actual sample seems not to work, it paves the way to a very elegant compatibility solution. The key is to add the extra information (e.g. Timestamp) in a different place. At least I was so focussed on fields at whole that I did not notice this possibility. I have experiemented a bit more with Glenn's proposal, shuffeling some fields. The result was this: PRIBSD_syslog_timestamp FQDN TAG @#VERSION MSGID Remainder_Timestamp [SD-ID]s MSG or in an actual sample: 148Oct 11 22:14:15 mymachine.example.com su[4711]: @#1 ID47 2003T.003Z [SD-IDs] 'su root' failed for lonvick on /dev/pts/8 I have used the BSD timestamp and FAQN as Glenn suggested. Then, I have added the TAG again. If we think in the spirit of my mail this morning on syslog non-IETF standards, it would not really hurt if we standardize TAG instead of two fields. If I would like to retain the APP-NAME and PROCESSID, I could do the following ABNF: TAG = APP-NAME [[ PROCESSID ]] : The side-effect of this is that almost all syslog-messages currently emited comply with that format. So I suggest that we strongly consider joining these two again. Out of the sudden, we have the old header, but it is parsable by a hyptothetical new syslogd. Next, I have used a trick from syslog-sign. I have changed the VERSION from a number to a number plus a cookie. The version would now be @#1. I do not care if the cookie is @# or something else. The key point is that it, together with the version should be very unlikely to exist at that place in old-style syslog. That would allow a new receiver to differentiate between old and new style syslog messages. The rest of the message is just applying Glenn's proposal again: it has the MSG, the missing parts of the timestamp, SD-IDs and MSG. The Remainder_Timestamp looks strange. We might like it, we might like something else. That's easy to change and discuss. It's the concept that matters right now, not the exact format. If we take the outlined route, we would be able to extend the syslog protocol with as much backward compatibility as is possible in a not-yet-standardized world. I find this very desirable. I think we even have good chances that many existing old syslogds would relay such messages without changing them, thus keeping digital signatures intact. The required text changes for syslog-protocol should be moderate. I strongly propose we go in that direction. Rainer -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Glenn Mansfield Keeni Sent: Wednesday, November 23, 2005 1:39 PM To: [EMAIL PROTECTED] Subject: Re: [Syslog] New direction and proposed charter Chris/Rainer, we continue to use PRI... at the start of syslog messages. This will allow current receivers to continue to receive messages and put them in the right bins. Does anyone disagree with this? Complete agreement. The WG has agreed to use the timestamp Rainer has in the current syslog-protocol. In principle I agree with the timestamp format. It is good. I may have missed the discussion on this matter, in that case please accept my apologies and ignore the rest of the mail. To get existing BSD syslog devices specifically relays into the compatibility fold it WILL be good idea to keep the timestamp in two parts RainerTimestamp = BSD_syslog_timestamp + RemainderTimestamp One possibility would be to assemble a syslog message as: PRITIMESTAMP
RE: [Syslog] New direction and proposed charter
WG, I have completed the promised testing. I used various syslogds on Linux, BSD and Windows platforms. The list obviously is not complete, but I think I got a fair enough sample of what is deployed. The good news is that by putting the PRI part in front of the message all of them were able to put the otherwise syslog-protocol-15 formatted message into the right bins. The format recorded was also acceptable, the non-recognized hostname was replaced by the sender address and the local date was added. This is within the typical current user experience when different syslogds are being used. Please note that some (e.g. BSD syslogd) do never pull the hostname from the message but always use the sender's address. The only culpit that I came along is associated with NUL octets. Many syslogds can not handle them. The message is only recorded up to the first NUL inside the message, no further interpretation happens. We have had discussion on this topic previously. As a reminder, it was said that excluding the NUL would be a crappy little rule that could open the door for many more CLRs. I still tend to think this is true and the problem exposed is acceptable. I try to sum up yesterday's discussion and my current position on it: Once again, I think David's comments on the charter are in the right direction (http://www.mail-archive.com/syslog%40lists.ietf.org/msg00143.html). It calls for some compatibility but puts emphasis on newer development. I suggest we accept the wording. We have had several comments on the field order in syslog-protocol. Based on them, I propose the following format: PRIVERSION TIMESTAMP HOSTNAME APP-NAME PROCID [SD-ID]s MSG With (optional) SD-IDs for - Extended Facility - TRUNCATE - MSGID - [Language Identification] Please note that I moved MSGID to the optional SD-ID part and instead re-introduced APP-NAME and PROCID as formal fields. I have done this because these two were designed as a replacement for TAG, whereas MSGID was meant to be something totally different. If we would prefer to have one field, only, I recommend to name it TAG and use the same semantics RFC 3164 describes. I support Chris proposal to leave the current size limitations exactly as they are in syslog-protocol. Everything else would cause the n-th re-iteration of the message size issue (side note: exactly the same discussion seems to have started on the NETCONF WG for event notifications right now, so this is an ever-popular topic). I have included an Extended Facility to provide finer-grain facility control within the message. This was brought up by Anton and others and does not seem to hurt anything if in SD-ID. I have included the Language Identification. I don't object it, but I question its usefulness. If we follow the message defition above, we can probably recycle most of the text in syslog-protocol, just shuffle it a little around. This has two advantages: - we do not loose what we already have discussed - the work can progress rather quickly Also, syslog-transport-udp most probably does not need any change at all, at most in a minor way. Modifying existing syslog receivers should not be very hard with the new definitions. The only major issue I see from the implementors point of view is UTF-8 decoding. But that is more of a storage problem. It is of course possible to receive the message and store it as UTF-8. I do not think this would cause non-compliance to the spec - or would it? If it would, UTF-8 would most probably be a major drawback when it comes to implementor acceptance. I am also a bit concerned about the NUL character, which can only be handled in one of two ways with existing syslog code base: - implementing a byte-counted string class and do not use the C RTL - replace NUL with an escape sequence upon reception (e.g. 00) I guess most implementations would take the later route. If we consider this to be acceptable, the majority of syslogds should be fairly easy to upgrade. If we can agree on these points, I would volunteer to implement the resulting document in C code so that we might see if there were any hidden problems. I would try to apply as minimal changes as possible. Suggestion for progressing: - we need more comments from other list members! - we should re-charter - we should reach rough consensus on the new format - once done, I can update syslog-protocol to it - I (and others?) can do test implementations in parallel to the review - discussion can show if (hopefully minor) adjustments need to be made - the goal should still be to finish this work (including AD approval) by the next IETF meeting Rainer -Original Message- From: Chris Lonvick [mailto:[EMAIL PROTECTED] Sent: Monday, November 21, 2005 9:58 PM To: Rainer Gerhards Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: RE: [Syslog] New direction and proposed charter Hi Rainer and all, On Mon, 21 Nov 2005, Rainer Gerhards wrote: Chris WG From the meeting, it sounds like
RE: [Syslog] New direction and proposed charter
Darren: WG, PRIVERSION TIMESTAMP HOSTNAME APP-NAME PROCID [SD-ID]s MSG I would put the SD-IDs after the message. The SD-IDs and detailed bits of meaning to the MSG and without the MSG, are irrelevant. The exception being a language marker. I would prefer SD-ID where it is in example. I would also re-iterate suggestion of having MSGID in the header, which a number of people supported. Those two combined are arguably more important than the MSG part itself. For example (in abbreviated syntax: host.domain.com MyApp proc1234 STARTED_UP [ip=1.1.1.1]: The applications has started I could live without the MSG here if it got truncated. The MSGID and SD-ID are much more important in this case. BTW, the possible truncation of text and its variability (possible substitutable variables) is another reason why MSGID is so useful. It makes it easier for intelligent receivers to do things like event correlation. Thanks, Anton. - replace NUL with an escape sequence upon reception (e.g. 00) Why not \0 ? Darren ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
RE: [Syslog] New direction and proposed charter
If we go for framing, we must use byte-couting, because we have not outruled any sequence. If we go for octet-stuffing, we must define an escape mechanism. Any of this would be helpful for plain tcp syslog, but that is definitely a big departure from current syslog. Please note that currently many syslogds do octet-stuffing and the message TRAILER is LF. That's unfortunate :( I agree, but that's the way it is in current (non-standard) implementations. In nearly all IETF protocols, the message trailer or EOL marker is CR-LF. If we go for a very simplistic tcp transport, there is nothing that hinders us in chaning it to CR-LF. That would also be compatible to existing receivers, as LF thankfully comes after CR... Rainer ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
Re: [Syslog] New direction and proposed charter
WG, PRIVERSION TIMESTAMP HOSTNAME APP-NAME PROCID [SD-ID]s MSG I would put the SD-IDs after the message. This raises the question of what terminates the MSG part ;) Using the above syntax, how do you distinguish between [] at the start of the message from actualy SD-ID data? I think what's missing from the above, is a ':' and the syntax should be: PRIVERSION TIMESTAMP HOSTNAME APP-NAME PROCID [SD-ID]: MSG The protocol document needs to outlaw ':' being in any field before the MSG. If you mark VERSION, PROCID and SD-ID data as all being optional then the format comes back to being very close to what's in use today. That would mean we would need to introduce byte-counting, at least I think so. Well, without the ':' to say where the MSG starts, I'd have argued How do you tell where SD-ID ends and MSG starts? vs there just being a string of bad SD-ID data following some good SD-ID data. As for but the SD has important information and the MSD does not, that's simply a matter of how you structure the message. - replace NUL with an escape sequence upon reception (e.g. 00) Why not \0 ? That's another good choice. It's also how data gets escaped, in general, in Internet stuff. That was my main message. Is it better to live with that or introduce a CLR on not allowing NUL? I'd like to see NUL outlawed from messages. Darren ___ Syslog mailing list Syslog@lists.ietf.org https://www1.ietf.org/mailman/listinfo/syslog
