RE: RFC 3339 UTC offset
I have gone through all of the good posts on this issue. I have the feeling that we currently have the following status: - some people argue that a synced/I know my TZ indicator for the timestmap has advantages in the real world - some people argue that a synced/I know my TZ indicator for the timestamp has no advantege in the real world, because nobody can automatically assert the timestamp - among those who would like to have a timestamp, there are some that would favour chaging the RFC3339 stamp and others who not like this idea. All of those who do not like the indicator do also not like changing the timestamp in a non-standard way. - if an indicator would be provided, I think it is concensus that this indicator should take up only very limited message space in the usual [?;)] case of synced time and known TZ - there is more than one issue, effectively, it is o am I synced to a reliable source? o do I know my TZ? o how good is my timesync (precision issue) - in the real world, there we can't demand timesync is in place, so we can't really follow RFC 3339 in this regard This is my honest try to sum it up - PLEASE correct me if you think I am wrong. I've tried to be neutral, but as I have my own opinion, it might have influence the summary. In the light of this sum-up, I propose the following compromise: - we will continue to use the rfc 3339 timestamp in its unmodified way - we will ignore that RFC 3339 calls for timesync (because we can't ensure it) - there will be NO header field indicating the reliability of the time flag - there will be an optional structured data element which allows to specify the timestamp reliability in all three cases. If it is absent, it is assumed that the timestamp is correct. If an implementor has a good indication this may not be the case, his implementation SHOULD write the respective element indicating this - the ID in general and/or its security considerations should list this imperfection I know this probably is not the ideal form. But given the diversity of good (!) thoughts on this issue, I think it is an agreeable compromise ... or may be not ;) Rainer -Original Message- From: Anton Okmianski [mailto:[EMAIL PROTECTED] Sent: Thursday, February 12, 2004 11:39 PM To: Rainer Gerhards; [EMAIL PROTECTED] Subject: RE: RFC 3339 UTC offset Rainer: #1 is more a spec-stunt than a real solution ;) In 4.4, RFC 3339 says ### o Use another host in the same local time zone as a gateway to the Internet. This host MUST correct unqualified local times that are transmitted to other hosts. ### Definitely a stunt. :) This requirement is constraining deployment options. #2 is to tweak the format a little bit. Honestly, I don't like this messing with a standard, and this is very far to the edge... All special sequences (like -00:00) are already taken up. 3339, however, specifies that the Z UTC indicator can be either upper or lower case. I have choosen to allow only upper case for simplicity. We *could* say that if the device does not know its ZT and does not know if time is properly synced, it should use z (lower case) as the TZ indicator. This would still be a valid 3339 indicator, but definitely not be in the spirit of 3339. I think it is okay, although the distinction is not as clear as I'd prefer. I like - for offset because this removes ambiguity. I know it is not consistent with RFC3339, but RFC3339 format is ONLY for situations when you know the offset, which won't always be the case for syslog clients. So, I guess in some cases RFC3339 is not appropriate. #3 is to actually add a syslog header field I know my TZ info or I am synced (Y/N) in -protcol, eventually adjecent to the timestamp field. Honestly, this, too, does not smell really good to me. I think the I am sync'ed maybe be different than I know my TZ. For example, you maybe no be sync'ed but know your time zone. This brings up another question that maybe other people raised before. I guess there are cases when you know the time is definitely NOT synced. But when do we consider it synced? What's the precision of sync'ed? Does it mean it is sync'ed with NTP or set manually yesterday? I guess for some cases set manually yesterday is fine, but for other cases (say when correlating messages from multiple systems based on precise time) this is not good enough because clocks drift and manually set time can't be precise. So, maybe this is a deployment-specific decision how the authoritative-time flag is interpreted? Anton.
RE: RFC 3339 UTC offset
Rainer: In the light of this sum-up, I propose the following compromise: - we will continue to use the rfc 3339 timestamp in its unmodified way - we will ignore that RFC 3339 calls for timesync (because we can't ensure it) - there will be NO header field indicating the reliability of the time flag - there will be an optional structured data element which allows to specify the timestamp reliability in all three cases. If it is absent, it is assumed that the timestamp is correct. If an implementor has a good indication this may not be the case, his implementation SHOULD write the respective element indicating this This could be an ok compromise. But some clarification is still needed. 1. What is timestamp is correct? Left to implementation to decide? I am fine with that, if we can wordsmith it. 2. I am concerned about the SHOULD in the above. I think when implementation does have good indication that time is wrong it MUST provide a structured element to indicate this. It is still kinda voluntary because the implementation may not have a good indication or may have its own definition of wrong time. 3. What do devices which have no time put in the timestamp? Anything? I understand they will provide the structured element (MUST), but why not recommend what they should put in the timestamp instead of leaving it up to them to invent a convention? 4. Do we want to make a distinction between non-authoritative time and unknown time? Separate structured element? I know this has been a big thread. But there is a lot of intricate issues to deal with here. The biggest problem, like you indicated, is that our protocol is designed for a an environment which is not RFC 3339 compliant. Anton.
Re: RFC 3339 UTC offset
On Wed, Feb 11, 2004 at 06:35:28PM +0100, Rainer Gerhards wrote: Anton, this is a tough one ;) Well, I think demanding that capability is more about host implementation. I think it is better to focus on the protocol and what the client/sender must put into the UTC offset field if one is not set on the system. This is right. And I have to admit it is an important issue. We always have nightmares about non-timesynced devices, as many customers have. In fact, the default of our product is to use time of reception, not the time from the message timestamp. I guess this shows it actually *is* a real world issue... [ snip ] The problem that I have with these solutions is that they require the syslog daemon to know if the time and timezone on the machine are valid. I can't think of any way of doing that. I'd would be perfectly happy with a recomendadtion that the hosts should hove the correct time/timezone information. However, ultimatly it's the sys admin's problem if their machines have incorrect time/timezone. And the sys admin which will suffer for it. This isn't meat to trivialize the issue, but I don't think that it's a problem which we should try to solve in the syslog protocol. You can only do so much to protect people from them selves. My $0.02 -- Devin Kowatch [EMAIL PROTECTED]
RE: RFC 3339 UTC offset
The problem that I have with these solutions is that they require the syslog daemon to know if the time and timezone on the machine are valid. I can't think of any way of doing that. Actually, I think this is the easy part. Its a trivial solution, but I think it works. I think we can require that a syslog implementation - by default - sets a I am not sure about my time flag. This is changed only after the operator configures it to be differently. I'd would be perfectly happy with a recomendadtion that the hosts should hove the correct time/timezone information. However, ultimatly it's the sys admin's problem if their machines have incorrect time/timezone. And the sys admin which will suffer for it. This isn't meat to trivialize the issue, but I don't think that it's a problem which we should try to solve in the syslog protocol. You can only do so much to protect people from them selves. I think this time its worth doing it. I see a real-world issue here. I see a fairly simple solution for most of the part. I am still missing a solution for this in the message. But I begin to recommend that we actually put this reliable time info flag in a one-byte header field. Looks ugly, but solves an issue... Rainer
RE: RFC 3339 UTC offset
Actually, I think this is the easy part. Its a trivial solution, but I think it works. I think we can require that a syslog implementation - by default - sets a I am not sure about my time flag. This is changed only after the operator configures it to be differently. This isn't really solving the prbolem it is doging the problem. Using a configuration variable for this parameter does not add any useful information to the syslog message. If the admin set the parameter in the first place then they know their time is correct so why should enver syslog message tell them that? Also it just lets the admin lie about how correct their time is, so whats the use. I am just responding to this because I think the other comments boild down to this one. My point is that some things my simply be overlooked. OK, we may not want to make it foolproof. But does it really hurt? IF the admin goes to the box and says hey, you are synced than at least the admin has put some manual effort and taken over the responsibility. If the box is just plugged and not properly configured, it will use the default which says I can't be fully trusted. In a larger enterprise, a higher level admin (or a co-worker) may detect this failure which otherwise may be undetected. As I said, this is a real-world issue. We have changed the default in our products so that they DO NOT use the syslog message timestamp, simply because this caused too much trouble. Once we told customers to switch to replace it with the syslogd's local reception time, this trouble ends. I think this flag can adress the need. But maybe I am going overboard. Anyone else with comments? Rainer
Re: RFC 3339 UTC offset
On Thu, Feb 12, 2004 at 06:24:12PM +0100, Rainer Gerhards wrote: The problem that I have with these solutions is that they require the syslog daemon to know if the time and timezone on the machine are valid. I can't think of any way of doing that. Actually, I think this is the easy part. Its a trivial solution, but I think it works. I think we can require that a syslog implementation - by default - sets a I am not sure about my time flag. This is changed only after the operator configures it to be differently. This isn't really solving the prbolem it is doging the problem. Using a configuration variable for this parameter does not add any useful information to the syslog message. If the admin set the parameter in the first place then they know their time is correct so why should enver syslog message tell them that? Also it just lets the admin lie about how correct their time is, so whats the use. I'd would be perfectly happy with a recomendadtion that the hosts should hove the correct time/timezone information. However, ultimatly it's the sys admin's problem if their machines have incorrect time/timezone. And the sys admin which will suffer for it. This isn't meat to trivialize the issue, but I don't think that it's a problem which we should try to solve in the syslog protocol. You can only do so much to protect people from them selves. I think this time its worth doing it. I see a real-world issue here. I see a fairly simple solution for most of the part. I am still missing a solution for this in the message. But I begin to recommend that we actually put this reliable time info flag in a one-byte header field. Looks ugly, but solves an issue... Again why. Syslog is generally used internal to an organization. It's is the problem of each organization to determine how to get correct time on their machines and if they care. I agree that having correct time on a box is important, but I don't think that we should add extra complexit to syslog just to tell the sys admin that their time is incorrect. I guess the biggest problem I have with this is that if the reliable time info flag is not reliable then what's the point? If you are dead set on having some indication of time correctness, try adding a timestamp for each relay that the message passes through (and the final server, but that is a file format issue), or use RFC3195 with SYSLOG/COOKED and path elements. -- Devin Kowatch [EMAIL PROTECTED]
RE: RFC 3339 UTC offset
Rainer: #1 is more a spec-stunt than a real solution ;) In 4.4, RFC 3339 says ### o Use another host in the same local time zone as a gateway to the Internet. This host MUST correct unqualified local times that are transmitted to other hosts. ### Definitely a stunt. :) This requirement is constraining deployment options. #2 is to tweak the format a little bit. Honestly, I don't like this messing with a standard, and this is very far to the edge... All special sequences (like -00:00) are already taken up. 3339, however, specifies that the Z UTC indicator can be either upper or lower case. I have choosen to allow only upper case for simplicity. We *could* say that if the device does not know its ZT and does not know if time is properly synced, it should use z (lower case) as the TZ indicator. This would still be a valid 3339 indicator, but definitely not be in the spirit of 3339. I think it is okay, although the distinction is not as clear as I'd prefer. I like - for offset because this removes ambiguity. I know it is not consistent with RFC3339, but RFC3339 format is ONLY for situations when you know the offset, which won't always be the case for syslog clients. So, I guess in some cases RFC3339 is not appropriate. #3 is to actually add a syslog header field I know my TZ info or I am synced (Y/N) in -protcol, eventually adjecent to the timestamp field. Honestly, this, too, does not smell really good to me. I think the I am sync'ed maybe be different than I know my TZ. For example, you maybe no be sync'ed but know your time zone. This brings up another question that maybe other people raised before. I guess there are cases when you know the time is definitely NOT synced. But when do we consider it synced? What's the precision of sync'ed? Does it mean it is sync'ed with NTP or set manually yesterday? I guess for some cases set manually yesterday is fine, but for other cases (say when correlating messages from multiple systems based on precise time) this is not good enough because clocks drift and manually set time can't be precise. So, maybe this is a deployment-specific decision how the authoritative-time flag is interpreted? Anton.
RE: RFC 3339 UTC offset
Hi, I agree that we cannot demand mechanisms for correct synchronization. We can provide a means to set the timezone, and recommend it be set automatically via NTP. Following the recommendations of draft-jones-opsec-03.txt, we should also allow an admin to set an object that contains the offset from UTC. These are two objects that could be defined in the syslog mib. Since other protocols may also want to have such a capability, and a standard mib for time-sync could be done at some time, I recommend developing syslog objects for this, which other protocols could reuse of desired, but also design them such that they could rfelect the values set via other mechanisms, such as NTP. dbh -Original Message- From: Rainer Gerhards [mailto:[EMAIL PROTECTED] Sent: Wednesday, February 11, 2004 8:48 AM To: Anton Okmianski; [EMAIL PROTECTED] Subject: RE: RFC 3339 UTC offset Just noticed this in RFC 3339 (Section 4.4): Systems that are configured with a local time, are unaware of the corresponding UTC offset, and depend on time synchronization with other Internet systems, MUST use a mechanism that ensures correct synchronization with UTC. Since -protocol is a distributed protocol where multiple hosts can generate syslog messages to one syslog server, one can say that syslog depends on hosts synchronization with other Internet systems. Does this mean that syslog clients and servers MUST be aware of their UTC offset? Are we going to require this in -protocol? Besides, I don't think RFC 3339 specifies any designation for unknown offset, so what will such client provide? Actually, I think we can't ensure this. The real world is against us. You are right, I should specify that a syslog sender MUST offer a way learn its time zone, but I am very hesitant to demand time sync for syslog. I guess everybody in this WG knows how important time sync is, yet this is a real-life issue. Admin's don't use NTP, they don't configure time zones and device clocks are often set to wild values. So I think we should demand the capability to set the time zone in any syslog sender - but not go any further. Anton. PS: BTW, just as an FYI, I searched rfc-editor.org web site and came across only one other RFC which uses RFC 3339 format (RFC3340). I think if we are worried about acceptability of -protocol, this could be yet another slight concern. I think RFC 3339 time stamp is unlike anything that is in wide use and people may have some initial reservations about it until it becomes more accepted. I like it, accept for T and Z characters. I, too, disliked this when I came across it the first time. But when you begin to actually develop applications, the T is a great, quick shortcut to detect that it is actually an ISO date. Unfortunately, ANSI and ISO standards are worse. I think this is the most important argument why I selected 3339 - I don't like to invent yet another timestamp, the older timestamps either lack the year or the time zone or resolution or ... and the other standards are worse. Plus, I like to stick refering to RFCs, because these are available without purchasing (expensive) books [which also means RFCs are instantly available when you need them...]. Rainer
RE: RFC 3339 UTC offset
Hi, Who makes the assertion that you can trust my timestamp? An application? As Devin points out, an application has no way to determine whether a timestsamp is valid, and thus cannot be trusted to make a valid assertion on its own. The best you could do is allow an administrator to set a value (maybe via a mib object) that asserts the timestamps used by this syslog engine are reasonably valid, and it would be up to the administrator to make sure the assertion is reasonably valid. dbh -Original Message- From: Rainer Gerhards [mailto:[EMAIL PROTECTED] Sent: Thursday, February 12, 2004 12:26 PM To: Devin Kowatch Cc: Anton Okmianski; [EMAIL PROTECTED] Subject: RE: RFC 3339 UTC offset one more thing... I just got a - I think - good idea. We could say that by default, we don't fully trust the timestamp. Then, we could define an optional structured data element (hey, we've got those!) that says yes, you can trust my time. This could also be extended for use by timestamper services. Out of the options so far, I like this one most... Rainer -Original Message- From: Devin Kowatch [mailto:[EMAIL PROTECTED] Sent: Thursday, February 12, 2004 6:19 PM To: Rainer Gerhards Cc: Anton Okmianski; [EMAIL PROTECTED] Subject: Re: RFC 3339 UTC offset On Wed, Feb 11, 2004 at 06:35:28PM +0100, Rainer Gerhards wrote: Anton, this is a tough one ;) Well, I think demanding that capability is more about host implementation. I think it is better to focus on the protocol and what the client/sender must put into the UTC offset field if one is not set on the system. This is right. And I have to admit it is an important issue. We always have nightmares about non-timesynced devices, as many customers have. In fact, the default of our product is to use time of reception, not the time from the message timestamp. I guess this shows it actually *is* a real world issue... [ snip ] The problem that I have with these solutions is that they require the syslog daemon to know if the time and timezone on the machine are valid. I can't think of any way of doing that. I'd would be perfectly happy with a recomendadtion that the hosts should hove the correct time/timezone information. However, ultimatly it's the sys admin's problem if their machines have incorrect time/timezone. And the sys admin which will suffer for it. This isn't meat to trivialize the issue, but I don't think that it's a problem which we should try to solve in the syslog protocol. You can only do so much to protect people from them selves. My $0.02 -- Devin Kowatch [EMAIL PROTECTED]
RE: RFC 3339 UTC offset
Hi! I will reply to various emails in one. Who makes the assertion that you can trust my timestamp? An application? As Devin points out, an application has no way to determine whether a timestsamp is valid, and thus cannot be trusted to make a valid assertion on its own. Application can know if NTP is configured. Or administrator can set it. In fact, Cisco devices follow a similar convention in logging (including through syslog). I am not advocating doing anything just because Cisco does it and I work there :), but it just shows that some vendors find some convention for time sync indication useful. Cisco devices prefix the timestamp with * or . when they know that time is not authoritative, i.e. NTP is not setup or is not working properly (i.e network connection lost). NTP can be integrated into whatever application I think. I like the general idea of some indication since we can't require time synchronization AND what's important won't always be possible. What if you want to report an NTP client error? From: Devin Kowatch ... I guess the biggest problem I have with this is that if the reliable time info flag is not reliable then what's the point? Well, what part of syslog is reliable? Security is a different issue IMO. Sure people can lie, but that does not mean we should not give good admins a useful option to simplify their administration of a large farm of servers. How people decide to use or not use this indication will be different in different deployments I think. So, I'd go for something that does not take to much space. From: Rainer Gerhards [mailto:[EMAIL PROTECTED] ... one more thing... I just got a - I think - good idea. We could say that by default, we don't fully trust the timestamp. Then, we could define an optional structured data element (hey, we've got those!) that says yes, you can trust my time. This could also be extended for use by timestamper services. I just don't like punishing the good client which has synchronized time by making it send more data in a structured element. I think the overhead, if any should be on the clients which don't know if time is authoritative. IMO we should assume that people should setup time sync in most cases. -Original Message- From: Devin Kowatch Sent: Thursday, February 12, 2004 12:39 PM ... If you are dead set on having some indication of time correctness, try adding a timestamp for each relay that the message passes through (and the final server, but that is a file format issue), I like this idea, but I don't think it solves the problem. Time of origination is what is used to establish message order for one and multiple hosts. Time of reception can be different including order. I think time of reception is a good thing to keep (for cases where origin time is unreliable), and I think it would be nice if first relay could add this information (but not overwrite original data). Maybe add a structured element? BTW, is relay prohibited from adding information to the message? I also talked about source IP as potentially another thing you may want relay to add. or use RFC3195 with SYSLOG/COOKED and path elements. But what about UDP transport? -- We jumped to time synchronization issue (which is fine), while my original issue got forgotten. I was actually raising the issue of UTC offset. Some systems may know their time, but have no idea about their offset from UTC. What should they put as UTC offset value in the timestamp? Random value? I think this is an issue we have to address even if we provide for some flag that is incorrect. I propose that they put - instead of an offset or we come up with another suffix like U for unknown. -- Still on the same subject of timestamps, but yet another issue. Do we want to come up with some timestamp format that devices can put when they don't know their time? For example: T---Z? It seems we have scoped it out, but it is a problem. Cisco has a number of devices that maybe unaware of their clock. Cisco currently sends messages with uptime (like 8d5h3m) instead of timestamp when this is the case. If Cisco where to adopt -protocol, they will need to deal with this one way or the other. How? Unless -protocol specifies this, they will have to invent something, then Nortel will invent something, Juniper, etc... Not good. I am not speaking for Cisco, but from experience of trying to standardize logging within Cisco. When I tried to scope out devices that have no knowledge of time, I got a pushback we had to add support for unknown date/time. I think uptime can be left out of scope and be a vendor structured parameter for now. Thanks, Anton.
RE: RFC 3339 UTC offset
Anton, this is a tough one ;) Well, I think demanding that capability is more about host implementation. I think it is better to focus on the protocol and what the client/sender must put into the UTC offset field if one is not set on the system. This is right. And I have to admit it is an important issue. We always have nightmares about non-timesynced devices, as many customers have. In fact, the default of our product is to use time of reception, not the time from the message timestamp. I guess this shows it actually *is* a real world issue... We can't tell them to put -00:00 or +00:00 as this has special meaning in the RFC. I would say that UTC offset if a MUST, while time sync is a SHOULD. If offset is not a MUST, then we MUST specify what they put as offset to indicate it is unknown. Yaks. Crappy little rule? :) I spent some time reading and re-reading RFC 3339 and thinking how this could be handled... I think all clean options are already used up. I see three approaches: #1 is more a spec-stunt than a real solution ;) In 4.4, RFC 3339 says ### o Use another host in the same local time zone as a gateway to the Internet. This host MUST correct unqualified local times that are transmitted to other hosts. ### If I look at syslog, that means we can require all systems that don't know their TZ to talk to a syslog relay which does. That, of course means we leave everything as imperfect as it is (but it is within the specs ;)). OK, now to the solutions... #2 is to tweak the format a little bit. Honestly, I don't like this messing with a standard, and this is very far to the edge... All special sequences (like -00:00) are already taken up. 3339, however, specifies that the Z UTC indicator can be either upper or lower case. I have choosen to allow only upper case for simplicity. We *could* say that if the device does not know its ZT and does not know if time is properly synced, it should use z (lower case) as the TZ indicator. This would still be a valid 3339 indicator, but definitely not be in the spirit of 3339. #3 is to actually add a syslog header field I know my TZ info or I am synced (Y/N) in -protcol, eventually adjecent to the timestamp field. Honestly, this, too, does not smell really good to me. In any case, I think we have a real-world problem and it would be good to find a solution. Which options does the WG prefer? Or is there another one I have not seen so far? Thanks, Rainer PS: I have listed this as issue 12 now.
