Hello,

I want to start a discussion on the proper support for the packet metadata that 
can be collected on Darwin platforms using `tcpdump -k` flag. For disclosure, I 
work at Apple at the team that is responsible for all Darwin networking, 
including the tcpdump, but also all the transport protocols (kernel and user 
space), as well as routing etc. Recently I got a permission from Apple 
management to contribute to Wireshark. I believe this combination gives me the 
opportunity to fix something that has been broken for a long time.


Context:

For the background, the `-k` flag will cause Darwin’s  `tcpdump` command to 
collect process metadata: the name, the id, and the uuid of both the associated 
process, and the effective process.  The metadata can be viewed by `tcpdump`, 
for example:

   tcpdump -n -k -r ~/tmp/tcpdump_k_pktap.pcapng -c 2 
   reading from PCAP-NG file /Users/oesh/tmp/tcpdump_k_pktap.pcapng 
   16:30:58.065667 (en0, proc mDNSResponder:428, eproc :428, svc BE, in, so) IP 
17.230.173.255.5353 > 224.0.0.251.5353: 0 TXT (QU)? TyM-bM-^@M-^Ys MacBook 
Pro._companion-link._tcp.local. (63)
   16:30:58.065680 (en0, proc mDNSResponder:428, eproc :428, svc BE, in, so) 
IP6 fe80::cb8:176a:f22c:66f9.5353 > ff02::fb.5353: 0 TXT (QU)? TyM-bM-^@M-^Ys 
MacBook Pro._companion-link._tcp.local. (63)

Support for the process metadata has been added around 2012-2013. This metadata 
is quite valuable when analyzing and optimizing traffic on mobile devices, 
where waking the antenna uses a lot of energy. 


The Challenge:

Unfortunately, when the support for the process metadata was added, the team 
missed the opportunity to do the right thing and to use CUSTOM block for the 
metadata, and instead used the LOCAL block id (with the MSB set). This was 
before my time, and as far as I can tell,  this decision was not made in spite, 
but due to the lack of context and the general sense of urgency.

It seems that the right thing to do is to get Darwin `tcpdump` to encode the 
process information in a standards-conforming fashion. I am fully supportive of 
this idea, and will work to make this happen. 

There is, however, an issue with the ossification of the current process 
metadata encoding. During the 12-13 years since the introduction of the process 
metadata, there are hundreds of thousands of existing pcap files that contain 
the metadata in the LOCAL block, and this is just in a single company. It is 
not realistic to expect the engineers to re-encode the “legacy” pcap files.

Because of that, I believe that while it is possible to “seal” the structure of 
the current process metadata block, there is no alternative but to continue 
supporting this, indefinitely.

I would like to mention that I am not super proud of this situation. I cannot 
change the past, but at least right now I am in the position where I can try to 
fix the situation, which is why I am making this effort.


Suggested Path Forward:

From the perspective of today, I would like to get to a situation where:

1. Darwin tcpdump uses the standards-conforming representation for the metadata 
when writing new files.
2. It is possible to gradually extend the metadata in a standards-conforming 
way.
3. Developers can use Wireshark to filter the traffic by the process metadata.
4. Developers can use Wireshark with both the “legacy” files, and the new files.

Since there are more than billion of devices running Darwin in some shape or 
form, and since there were more than ten years to collect potentially valuable 
pcap files on those devices, it is important to preserve the ability to dissect 
the “legacy” files. It unrealistic to expect that all the legacy files can be 
converted to a new representation; there is just too much of that legacy.

My understanding is that to support the “legacy” representation of Darwin 
metadata, Wireshark will need to start treating `0x80000001` as an ossified 
quasi-standard block type, and to at least attempt to decode this as Darwin 
process metadata.

Moreover, due to the quantity of the “legacy” pcap files, it might be a 
pragmatic idea to mention the block 0x80000001 as an “exception” in 
https://datatracker.ietf.org/doc/draft-ietf-opsawg-pcapng/ , so that future 
developers would skip this. 


Discussion

I would like to hear the opinions of  the core developers on the above. Again, 
I am not super proud that mistakes have happened, an d I am trying to fix it 
going forward, but please keep in mind the backwards compatibility. 
_______________________________________________
Wireshark-dev mailing list -- wireshark-dev@wireshark.org
To unsubscribe send an email to wireshark-dev-le...@wireshark.org

Reply via email to