Hi WG,
I am preparing a large edit on -international which will include all
those good comments into the draft. I will probably raise a number of
questions just so that I can verify my understanding of the discussion.
This is the first of those messages.
I think we have consensus that there are some (rare) occasions where
oversized messages need to be transmitted via a standard-compliant way,
thus we need to define something how this can be done. On the other
hand, the vast majority of messages will NOT be oversized, so we should
add as little overhead as possible to the ID to support this. As a
general guideline, the burden should be placed on those applications
that need to deal with oversized messages while all others should not
need to pay attention to it.
syslog-international-00 introduced a very rough way to implement this.
Here is a quote:
####
MORE and SEQNO provide support for syslog messages larger than the
allowed syslog packet size. This is introduced to allow transmittal
of "oversized" message, which may be the result of some character
sets and encodings. These messages will be fragmented by the syslog
client and reconstructed by the collector. Relays will pass them
through unmodifed.
Message fragmentation MAY be used if the underlying transport
provides reliable and in-order delivery (for example RFC 3195 [8]).
It the underlying transport is unreliable or its reliability is not
known, fragmentation MUST NOT be used.
More specifies wether this is the final fragment of the message or
not. An asterisk ("*") means that at least one more fragement will
follow. A period (".") means that this is the final (or only)
fragment.
SEQNO specifies the sequence number of fragments. It MUST start by 0
for the first fragment and MUST be incremented by 1 for each
following fragment. SEQNO MUST restart at 0 for each new full
message. A new full message begins after the last message that had
"." in MORE.
If fragementation is not used, all messages contain ". 0" as the MORE
SEQNO sequence.
####
There are a number of issues, especially when it comes to out of order
delivery. I would like to revise this and would appreciate feedback on
the following suggestion:
The modified ABNF would look like this (just the fragementation-related
parts):
HDR-i18n = COOKIE SP ENCODING SP CHARSET SP LANGUAGE SP FRAG-INFO SP
FRAG-INFO = "." /
("*" SP REBOOTID SP MSGNBR SP TOTALLEN SP INDEX SP
FRAGLEN)
REBOOTID = 1*10DIGIT ; never repeat or decrease
TOTALLEN = 1*7DIGIT
INDEX = 1*7DIGIT
FRAGLEN = 1*3DIGIT
FRAG-INFO is the important part I am talking about in this mail. It will
describe if a message is fragmented.
In the typical case (normal size message), it would just be ".". So for
most applications, nothing would need to be changed. IF we have a syslog
implementation that intends to support fragemented oversize package,
THEN (and only then) it needs to provide the following fields. As you
will probably see, I have borrowed from -sign, which hopefully will make
it easier to implement:
REBOOTID is the exact same reboot session ID as in -sign. An aplication
implementing both -sign and -international SHOULD use the same value as
in the -sign messages.
TOTALLEN is the total size of the message in bytes. It probably is
allowed to grow a bit large, but I think it doesn't hurt to support
idiot-size messages. If there is violent opposition, I would like to
limit this to 1 MB, but not less. I have seen *really* large events >
100KB once in a year and would like to see these covered (comparable to
-sign 4.3.5). BUT... I am aware of the fact that a malicious user could
use this to create a DoS on collectors / log reviewers / log analysers
that re-assemble the full oversize message.
INDEX is the position, in bytes, into the oversize message where this
fragment starts (as is e.g. the fragment index in -sign 4.3.6)
FRAGLEN is the length, in bytes, of this message. Byte counting starts
with the byte after the HDR-i18n (that is the byte AFTER the SP after
FRAGLEN). It can be a maximum of 3 digits because, after subtracting the
overhead of the headers, no wellformed message (fragement) can contain
more than 999 bytes.
If an oversize message is fragmented, the individual fragments are sent
as syslog messages in their own right, the fragmentation is just
indicated by the proper HDR-i18n fields. All fragments MUST be send with
the same TAG and HOSTNAME and SHOULD be send over the same interface.
The TIMESTAMP MUST be set to either the timestamp of the first fragement
or be increasing (never decreasing). The actual time the message
fragment was scheduled for transmittal SHOULD be placed inside the
TIMESTAMP of each fragment.
A device supporting oversize messages SHOULD place the most important
facts early in the message, and SHOULD try to place vital information
into the first fragment. The reason for this is that a collector or
offline analyzer may decide to discard the message after some supported
byte count.
A relay MUST forward the individual (fragmented) messages unaltered. A
collector SHOULD store them as separate entities. This is necessary to
verify -sign signatures. Either the collector or an offline log review
tool can re-assemble the messages. It SHOULD do so before displaying
them to a human or using them for an (automatted) analysis process.
If fragementation is used, messages SHOULD be sent over a reliable
transport like RFC 3195. No redundancy is provided for unreliable links,
as this could potentially eat up a lot of bandwidth.
I think this approach fits the goals. Most importantly, it allows for
oversize messages without the need to modify any existing applications,
which also means oversize messages can travel over existing relays, be
signed with -sign (the individual fragment will be signed) and also be
used over RFC 3195.
Is there any opposition against this approach? I am specifically
interested on feedback on:
- if this is considered overkill compared to the initial approach
- if the approach with using just "." for normal-size messages rings
some bells
- if someone sees some interoperability issues with existing syslog
- if there should be some redundancy be added for unreliable transport
- as a side note, should the reboot session ID be limited
to the 32 bit unsigned range or left as in -sign?
- if we should include some wording that a collector MUST support
e.g. a message size of 5 KB but MAY discard anything beyond
this. That would resolve the issue with the potential DoS (at least
most parts of it...).
I appreciate all feedback!
Rainer
