Open MPI is generally structured in layers, but adjacent layers don't necessarily have any knowledge of each other.
For example, the PML (point-to-point messaging layer) is the first layer behind MPI point-to-point functions such as MPI_SEND and MPI_RECV. Different PMLs do not have the same packet layouts, and may, themselves, be layered. For example, the OB1 PML is mostly a high-level protocol engine that uses BTLs for low-level sending to and receiving from peers. The BTLs, therefore, are responsible for their outermost packet formats, and encapsulate the OB1 payload (which, itself, has a header and a payload). If you want to look at the TCP BTL (over the OB1 PML), you can look at the various structs in opal/mca/btl/tcp. Additionally, you'll need to look at the OB1 PML structs in ompi/mca/pml/ob1. It's conceivable that you could make a Wireshark plugin for this use case, for example. The TCP BTL and OB1 PML structs are subject to change at any time -- it's not like they're published standards -- but they have been pretty stable for years. For other use cases -- e.g., OS-bypass networks -- you'll need to sniff the packets from the network itself (because, by definition, the OS won't have visibility of the packets). Regardless, all of those structs are defined in their BTL / MTL / PML / etc. components. We don't have formal documentation of any of them, sorry! -- Jeff Squyres jsquy...@cisco.com ________________________________________ From: victor sv <victo...@gmail.com> Sent: Monday, May 16, 2022 1:17 PM To: Jeff Squyres (jsquyres) Cc: users@lists.open-mpi.org Subject: Re: [OMPI users] Network traffic packets documentation Hi Jeff, Ok, maybe "packet headers" are not the right words. What I would like to know is how MPI application data is structured inside each packet in order to dissect and caracterize the messages. As a first step I would like to start with TCP over Ethernet (MCA BTL TCP, I think). How can I figure out how the application data structure looks like inside network packets? In the future I would like to extend it to other network and transport combinations. What do you think? Has it sense? Thanks in advance. Víctor El lun, 16 may 2022 a las 15:45, Jeff Squyres (jsquyres) (<jsquy...@cisco.com<mailto:jsquy...@cisco.com>>) escribió: Open MPI doesn't proscribe a specific network protocol for anything. Indeed, each network transport uses their own protocols, headers, etc. It's basically a "each Open MPI plugin needs to be able to talk to itself", and therefore no commonality is needed (or desired). Which network and Open MPI transport are you looking to sniff? -- Jeff Squyres jsquy...@cisco.com<mailto:jsquy...@cisco.com> ________________________________________ From: users <users-boun...@lists.open-mpi.org<mailto:users-boun...@lists.open-mpi.org>> on behalf of victor sv via users <users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>> Sent: Sunday, May 15, 2022 3:55 PM To: users@lists.open-mpi.org<mailto:users@lists.open-mpi.org> Cc: victor sv Subject: [OMPI users] Network traffic packets documentation Hello, I would like to sniff the OMPI network traffic from outside the MPI application. I was traversing the OpenMPI code and documentation, but I have not found any central point explaining MPI communications from the network point of view. Please, is there any official documentation, or paper, or presentation or picture about MPI packet headers? Sorry if this a basic question or if it was already answered. Thanks in advance for your help! BR, Víctor.
