On 24 July 2018 at 11:46, Guy Harris <ghar...@sonic.net> wrote:
> On Jul 23, 2018, at 6:36 PM, David Mirabito <dav...@metamako.com> wrote:
>
> > We'd like to request a new DLT_ value for Metamako's packet capture
> trailer
> > that is generated by our MetaWatch network application.
> >
> > https://www.metamako.com/applications/metawatch-app.html
> >
> > It is a variable length data trailer, appended immediately after an
> > original Ethernet Payload. It is backwards-compatible with DLT_EN10MB but
> > adds additional fields after the original frame payload until the
> > end-of-frame marker.
> >
> > We were thinking something like this
> >
> > LINKTYPE_METAMAKO_TRAILER --> DLT_METAMAKO_TRAILER
> >
> > Is there anything in particular that you require?
>
> A detailed description of the format of the trailer.
>
> Presumably, for Ethernet packets with a type field, the trailer can't be
> found until the payload is dissected, because there's no other way to
> determine how large the payload is.
>
> Also, presumably, the trailer comes after any padding, and there's no FCS
> in these packets.
Hi Guy,
I have attached a more detailed description as a text file (lest email
mangle the ASCII-art), but in short, a packet would look on the wire like:
* SOF
* Destination address
* Source Address
* EtherType/Length
* Payload
* FCS . <--- everything up to, and including, here is unmodified by
MetaWatch. Typically our appending a trailer may allow this FCS to be
captured by NICs which would otherwise have stripped it
* Optional Extension TLVs
* Timestamp: Seconds
* Timestamp: Nanoseconds
* Flags (currently whether FCS was correct at ingress, and presence of any
extension TLVs)
* Device ID (by default unique portion of the device's serial number)
* Port ID
* NewFCS (calculated so that metawatch emits a valid frame, not always
captured by capturing NICs)
* EOF
The intent when parsing is that the base trailer is always the last 12
bytes so given a complete capture one may simply skip directly to "wlen -
12", so there is no need to interpret EtherLen, or do a full dissection to
locate the trailer. I agree that the case where caplen < wirelen may be
problematic, but then you're guaranteed a corrupted or missing trailer
anyway so it might not be worth even trying.
Our current Wireshark integration does indeed register as a heuristic
Ethernet Trailer, and as such does rely on a correct prior dissection to
identify any trailer and give us a chance to match. We tend ot make use of
the "Fixed ethernet trailer length" feature in the cases where prior
dissection fails to identify any trailer.
The main motivation to request a DLT that this allows us first-chance look
at each frame in capture known to contain trailers, in part to eliminate
such reliance. We then may scrape the trailer off the end, and then pass
the remainder down the usual dissector stack. To me it seemed similar to
the pseudo-header notion, except working backwards from the buffer's end.
It also allows us to poke our timestamp in as "the" time used for display
and any measurements (i.e RTT) done by sub-dissectors when opening such a
trace file.
Regarding padding:
* In a compliant frame, the original sender would have padded and then
added an FCS (covering the padding) on the wire. To us this looks like any
other 64-byte frame and we still immediately stick the trailer immediately
after this original FCS
* In the case we are presented with an undersized frame, we will add enough
padding ourselves such that "whatever came in + padding + tailer" will be a
legal frame. Presumably in this case the original FCS would not have been
correct, but the one we apply at the end will be correct, regardless.
Either way this would preserve the parsing scheme. The "original FCS" may
be flagged as incorrect in the case of runts or other bit errors prior to
our ingress, but we still "wrap" in a trailer + good new-FCS allowing such
frames to be captured on commodity HW.
I hope this helps, please let us know any more info might be required.
Thanks,
David M
Metamako Trailer.
================
This describes a potentially variable length data trailer, appended
immediately after an original Ethernet Payload.
Being a trailer, it is backwards-compatible with DLT_EN10MB: an unaware decoder
will simply see extra bytes between the end of the original frame payload and
the end-of-frame marker. These should be safely ignored as in the case of padded
frames anyway.
The trailer is intended to be walked backwards from the End of Frame (EOF).
Unless otherwise specified, all multi-byte values are in network-byte-order (Big
Endian).
Base Trailer
------------
At a minimum, all Ethernet frames are grown by 16 bytes with the addition of the
trailer structure immediately after the FCS of the timestamped frame. Due to the
varying length of the Ethernet payload, the end of the OriginalFCS (and hence
the start of any subsequent trailer data) may no be 32bit aligned. Trailer data
always immediately follows the FCS and not padding is inserted.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------------------------------------------------------+
| Seconds |
+---------------------------------------------------------------+
| Nanoseconds |
+---------------------------------------------------------------+
| Reserved |X|V| DeviceID | PortID |
+---------------------------------------------------------------+
| NewFCS |
+---------------------------------------------------------------+
Seconds:
Time in seconds (UTC) since the Jan 1st, 1970 Epoch.
Nanoseconds:
Nanosecond portion of time. Will be less than 1,000,000,000
Reserved:
Reserved for future use, to be written as zero and ignored if set
eXtensions present
1: If optional trailer extensions are present.
0: If no extensions are present.
FCS Valid:
1: If the original FCS of the frame was correct.
0: If the original FCS was incorrect.
DeviceID:
A 16bit identifier for the device which captured this frame.
Defaults to the unique portion of the device's serial number but may be
configured to any value.
PortID:
An 8bit identifier for the port which captured this frame.
Defaults to the number of the ingress AP port, but may be
configured to any value.
NewFCS:
A new FCS is calculated and appended so that the annotated Ethernet
frame + original FCS + [extensions] + trailer is still valid on the wire.
NB: Not all capturing devices will record this field, presumably those would
also not have recorded the FCS value of the un-trailered Ethernet frame
either.
Trailer Extensions
==================
When present, trailers are found immediately preceding the 'Seconds' field of
the base trailer, in TLV (Tag, Length, Value) format. Currently only some tag
types are defined, decoders should use the length field to ignore and skip over
any unknown extension types.
There are two tag formats, 'primary' and 'secondary' which offer a tradeoff
between compactness and the ability to select from more available tags or
longer payload lengths.
Primary
-------
Primary tags are the most compact, allowing up to 24bits of payload to be
specified in the Tag header. Up to 3 additional 32bit words of payload may also
be specified.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------------------------------------------------------+
| Data |Len|F| Tag |
+---------------------------------------------------------------+
Tag:
Specified the type of information stored in the Data section.
Allowable values are 0x00-0x1E (0-30).
Tag value 0x1F (31) is used to note a Secondary extension type
Final:
1: If this is the Final extension present.
0: If there is another extension preceding this one's data.
Trailers and extensions are processed walking backwards from the
EOF, so the first extension received on the wire will be last one processed
and hence have this bit set.
Only one extension header may have the Final bit set.
Length:
The number of additional 32bit words of payload in this extension.
Note that a value of zero is allowed, as the inline 24bits may be
sufficient and no additional words may be required.
Data:
Either 24, 56, 88 or 120 bits of Tag-specific data, depending on the Length
value.
Secondary
---------
Secondary tags allow more possible Type values and longer possible lengths buy
not having any data payload inline in the extension header.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------------------------------------------------------+
| Data |
+---------------------------------------------------------------+
| Tag2 | Len |F| 11111 |
+---------------------------------------------------------------+
Tag:
Always 0x1F (32) in secondary tags
Final:
Same meaning as Primary extensions
Len:
Value is N-1, specifying how many additional 32bit words of Data payload
preceed the extension header
0 --> 1 word (32bits)
1 --> 2 words (64 bits)
...
1023 --> 1024 words (32768 bits)
Tag2:
Specifies the meaning of the Data payload.
Defined extension tags
=======================
Some extension types have already been defined:
Sequence number:
----------------
* Primary Extension; Tag = 1
Sequence Numbers are unsigned 16bit numbers, maintained per-port and
incremented on ingress. Gaps in a port's numbers may indicate drops.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------------------------------------------------------+
| Reserved | SequenceNo |Len|F| 00000 |
+---------------------------------------------------------------+
Fractional Nanoseconds:
-----------------------
* Primary Extension; Tag = 2
Additional timestamp precision may be available beyond the nanoseconds-level
found in the base trailer. Value is in 1/2^24 of 1ns.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------------------------------------------------------+
| FractionalNS |Len|F| 00001 |
+---------------------------------------------------------------+
ASCII String:
------------
* Secondary Extension; Tag2 = 00
Allows for arbitrary strings up to 4k in length. Strings need not be
zero-terminated if they end on a 4-byte boundary, and unused bytes should be
padded with zero.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------------------------------------------------------+
| |
+ ASCII String +
| |
+---------------------------------------------------------------+
| 0000000000000000 | Len |F| 11111 |
+---------------------------------------------------------------+
Example
=======
Here we show a full frame on the wire:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DestinationAddress |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
| SourceAddress |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| EtherType | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
| |
+ +
| |
+ Payload +
...
+ (46-1500bytes) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| OrigFCS |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0x064dd3 | 00|1| 00001 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0x00 | 0xffb6 | 00|0| 00000 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0x5b32cdda |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0x2a2b1a86 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 000000 |1|1| 0x1dc0 | 0x09 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NewFCS |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
The decoding process is as follows:
1. The NewFCS may or may not by physically captured. Regardless, we can assume
is correct by virtue of successfully capturing the frame.
2. Flags, DeviceID and PortID fields are always in a fixed location relative to
the end of frame. We determine:
* The original FCS was good. (i.e running CRC32 over from DestinationAddr to
the end of Ethernet payload would yield OrigFCS)
* There is at least one Trailer Extension present.
* Packet was timestamped by device 7617 on Application Port #9.
3. Similarly, Seconds:Nanoseconds fields are also fixed relative to EOF:
* The timestamp is 1530056154.707467910 UTC --> Jun 27, 2018
09:35:54.707467910 AEST
4. An extension is present so the 32bits prior to Seconds field is the header of
the first extension.
* Type is 0 --> is sequence number #65462
* Final is not set --> there is at least one more trailer extension present
* Length is 0, no more words for this tag. SequenceNumbers are defined to
have a length of 0, but unaware implementations may use the Length field to
skip to the next extension header.
5. The next extension header in in the preceding 32bit word
* Type is 1 --> Is a Fractional Nanosecond additional timestamp precision
* Final is set --> no more extensions to process.
* Length is 0 --> no more data to read for this tag.
* Data is 0x064dd3, so subnanos prevision is float(0x064dd3) / 2**24 =
0.024625003337860107
~= 24.6
picoseconds
Thus the full timestamp may be interpreted as: 1530056154.707467910024625
6. No more extensions --> end of processing.
Note:
Packet diagrams created with https://github.com/luismartingarcia/protocol
(perhaps with somehand editing)
Source for diagrams:
mmtrailer;
Seconds:32,Nanoseconds:32,Reserved:6,HasExtension:1,FcsValid:1,DeviceID:16,PortID:8,NewFCS:32
mmtlv1; Data:24,Len:2,Final:1,Tag:5
mmSequenceNumber; Reserved:8,Sequence:16,Len:2,Final:1,00000:5
mmFracNano; FractionalNS:24,Len:2,Final:1,00001:5
mmlvl2 Data:32,Tag2:16,len:10,Last:1,11111:5
mmstring ASCII
String:64,0000000000000000:16,Len:10,Last:1,11111:5
mmexample
DestinationAddress:48,SourceAddress:48,EtherType:16,Payload:144,OrigFCS:32,0x064dd3:24,00:2,1:1,00001:5,0x00:8,0xffb6:16,00:2,0:1,00000:5,0x5b32cdda:32,0x2a2b1a86:32,000000:6,1:1,1:1,0x1dc0:16,0x09:8,NewFCS:32
_______________________________________________
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
