Hi Tianran, Tom's note already includes the hint: You'll add IOAM data to the protocol/layer that you're interested in monitoring. Again using Geneve over IPv6 as an example: * If you're interested in the overlay, i.e. Geneve (e.g. timestamping the packet when it enters and exists the tunnel) - you'd add IOAM data to Geneve * If you're interested in the underlay, i.e. IPv6 (e.g. you'd like to understand which path packets take in the v6 network) - you'd add IOAM data to IPv6 * If you're interested in both, then you'd add IOAM data to Geneve and IPv6
Draft draft-ietf-ippm-ioam-data-02 already mentions layering (see section 3): "Layering: If several encapsulation protocols (e.g., in case of tunneling) are stacked on top of each other, IOAM data-records could be present at every layer. The behavior follows the ships-in-the-night model." Given the discussion here, we'll add some additional text in the next revision to make things crisper (e.g. adding an example might help). Frank -----Original Message----- From: Tianran Zhou <zhoutian...@huawei.com> Sent: Dienstag, 17. April 2018 03:18 To: Tom Herbert <t...@herbertland.com> Cc: Shwetha Bhandari (shwethab) <shwet...@cisco.com>; Frank Brockners (fbrockne) <fbroc...@cisco.com>; Mickey Spiegel <mspie...@barefootnetworks.com>; NVO3 <nvo3@ietf.org>; Service Function Chaining IETF list <s...@ietf.org>; IETF IPPM WG <i...@ietf.org> Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various protocols - follow up from WG discussion in London I think it's better that Frank or Shwetha can explain the multi-layer use case in detail. Tianran > -----Original Message----- > From: Tom Herbert [mailto:t...@herbertland.com] > Sent: Monday, April 16, 2018 10:40 PM > To: Tianran Zhou <zhoutian...@huawei.com> > Cc: Shwetha Bhandari (shwethab) <shwet...@cisco.com>; Frank Brockners > (fbrockne) <fbroc...@cisco.com>; Mickey Spiegel > <mspie...@barefootnetworks.com>; NVO3 <nvo3@ietf.org>; int-area > <int-a...@ietf.org>; Service Function Chaining IETF list > <s...@ietf.org>; IETF IPPM WG <i...@ietf.org> > Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various > protocols - follow up from WG discussion in London > > On Mon, Apr 16, 2018 at 6:31 AM, Tianran Zhou <zhoutian...@huawei.com> wrote: > > Hi Shwetha, > > > > You are talking about the outer encapsution. It is straight forward > > for the underlay to record by the header. But what about the > > overlay, i.e., inner encapsulation(e.g. geneve)? Without special > > configuration, intermediate node will not read the inner header, > > hence not be able to process IOAM.e > > Hi Tianran, > > I believe that is also not protocol conformant. Intermediate nodes > should not be processing transport layer data as this can lead to > misinterpretation and possibly silent data corruption. > > For instance, Geneve is a UDP encapsulation protocol with assigned port 6081. > In order for an intermediate device to process the Geneve > encapsulation header it would need to look for packets with > destination port of 6081 since that is the only possible > discriminator. However, transport port numbers do not have global > meaning and hosts may use port numbers for other purposes (RFC7605 > describes this). So a packet to port 6081 might be something other > than Geneve and may be misinterpreted. If a misinterpreted packet is changed > (like ippm data is written) then that would be systematic silent data > corruption. > > As far as I know, hop-by-hop options is the only protocol confirming > mechanism that allows an intermediate note to change data of packet in flight. > Encpasulation is the only conforming mechanism that allows an > intermediate node to add data (like extension headers) to a packet in flight. > > Tom > > > Maybe we are not synced by this overlay/underlay use case. :-) > > > > Tianran > > > > > > > > ________________________________ > > Sent from WeLink > > > > 发件人： Shwetha Bhandari (shwethab) > > 收件人： Tianran Zhou<zhoutian...@huawei.com>;Frank Brockners > > (fbrockne)<fbroc...@cisco.com>;Mickey > > Spiegel<mspie...@barefootnetworks.com>;Tom > > Herbert<t...@herbertland.com> > > 抄送： NVO3<nvo3@ietf.org>;int-area<int-a...@ietf.org>;Service Function > > Chaining IETF list<s...@ietf.org>;IETF IPPM WG<i...@ietf.org> > > 主题： Re: [ippm] [Int-area] encapsulation of IOAM data in various > > protocols - follow up from WG discussion in London > > 时间： 2018-04-16 18:17:01 > > > > Hi Tianran, > > > >> If I recall right, it is not written in the ioam data draft. > > > > Data draft is defining the data to be carried in IOAM in an > > encapsulation agnostic way, it does not specify how the > > encapsulation protocol is configured. > > > > > > > >> Yes, node by node configuration is an easy way. > > > > While it is, it does not have to be a node by node configuration. It > > can be part of the encapsulation definition. > > > > For e.g. If the encapsulation is IPv6 and if we define the data to > > be carried as HbH options, then based on the Option Type with > > highest order 2 bits set to 00 then the v6 nodes that implement IOAM > > will process the option and others will skip over. > > > > > > > > > > > > Thanks, > > > > Shwetha > > > > > > > > From: ippm <ippm-boun...@ietf.org> on behalf of Tianran Zhou > > <zhoutian...@huawei.com> > > Date: Monday, April 16, 2018 at 2:36 PM > > To: "Frank Brockners (fbrockne)" <fbroc...@cisco.com>, Mickey > > Spiegel <mspie...@barefootnetworks.com>, Tom Herbert > > <t...@herbertland.com> > > Cc: NVO3 <nvo3@ietf.org>, "int-a...@ietf.org" <int-a...@ietf.org>, > > Service Function Chaining IETF list <s...@ietf.org>, IETF IPPM WG > > <i...@ietf.org> > > Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various > > protocols - follow up from WG discussion in London > > > > > > > > Hi Frank, > > > > > > > > If I recall right, it is not written in the ioam data draft. > > > > Yes, node by node configuration is an easy way. In the > > draft-zhou-ippm-ioam-yang, we have the “protocol-type” to indicate > > the layering. > > > > +--rw ioam > > > > +--rw ioam-profiles > > > > +--rw enabled? boolean > > > > +--rw ioam-profile* [profile-name] > > > > +--rw profile-name string > > > > +--rw filter > > > > | +--rw filter-type? ioam-filter-type > > > > | +--rw acl-name? -> /acl:acls/acl/name > > > > +--rw protocol-type? ioam-protocol-type > > > > +--rw incremental-tracing-profile {incremental-trace}? > > > > | ... > > > > +--rw preallocated-tracing-profile {preallocated-trace}? > > > > | ... > > > > +--rw pot-profile {proof-of-transit}? > > > > | ... > > > > +--rw e2e-profile {edge-to-edge}? > > > > ... > > > > > > > > > > > > Tianran > > > > From: Frank Brockners (fbrockne) [mailto:fbroc...@cisco.com] > > Sent: Monday, April 16, 2018 4:51 PM > > To: Tianran Zhou <zhoutian...@huawei.com>; Mickey Spiegel > > <mspie...@barefootnetworks.com>; Tom Herbert <t...@herbertland.com> > > Cc: NVO3 <nvo3@ietf.org>; int-a...@ietf.org; Service Function > > Chaining IETF list <s...@ietf.org>; IETF IPPM WG <i...@ietf.org> > > Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various > > protocols - follow up from WG discussion in London > > > > > > > > Hi Tianran, > > > > > > > > IOAM is a domain specific feature (see also > > draft-ietf-ippm-ioam-data-02 sections 3 and 4), which allows an > > operator to control by means of configuration where and for which > > traffic IOAM data fields are added/updated/removed from the customer > > traffic. Using your example of Geneve over IPv6 – with IOAM data in > > both the Geneve and the IPv6 protocol, one would expect that the > > operator configures the endpoints of the Geneve tunnel to operate on > > the IOAM data in Geneve, and the IPv6 routers that the Geneve tunnel > traverses to operate on the IOAM data in IPv6. > > > > > > > > Frank > > > > > > > > From: Tianran Zhou <zhoutian...@huawei.com> > > Sent: Montag, 16. April 2018 10:37 > > To: Frank Brockners (fbrockne) <fbroc...@cisco.com>; Mickey Spiegel > > <mspie...@barefootnetworks.com>; Tom Herbert <t...@herbertland.com> > > Cc: NVO3 <nvo3@ietf.org>; int-a...@ietf.org; Service Function > > Chaining IETF list <s...@ietf.org>; IETF IPPM WG <i...@ietf.org> > > Subject: RE: [ippm] [Int-area] encapsulation of IOAM data in various > > protocols - follow up from WG discussion in London > > > > > > > > Hi Frank, > > > > > > > > How does a forwarder know when and where to insert the data? > > > > In the case of Geneve over IPv6, do you mean the device need to scan > > all the protocol stack? Or just the outer encapsulation? > > > > > > > > Tianran > > > > > > > > From: ippm [mailto:ippm-boun...@ietf.org] On Behalf Of Frank > > Brockners > > (fbrockne) > > Sent: Monday, April 16, 2018 3:08 PM > > To: Mickey Spiegel <mspie...@barefootnetworks.com>; Tom Herbert > > <t...@herbertland.com> > > Cc: NVO3 <nvo3@ietf.org>; int-a...@ietf.org; Service Function > > Chaining IETF list <s...@ietf.org>; IETF IPPM WG <i...@ietf.org> > > Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various > > protocols - follow up from WG discussion in London > > > > > > > > > > > > Tom, > > > > > > > > a quick addition to what Mickey mentioned below: What you seem to > > have in mind is what draft-ietf-ippm-ioam-data-02 refers to as “layering” > > (see section 3.), i.e. if you’re running for example Geneve over > > IPv6, then IOAM data could be encapsulated in both protocols, Geneve > > and > > IPv6 – giving you visibility into the “underlay” (IPv6) and the “overlay” > (Geneve). > > > > > > > > Frank > > > > > > > > From: ippm <ippm-boun...@ietf.org> On Behalf Of Mickey Spiegel > > Sent: Freitag, 13. April 2018 20:22 > > To: Tom Herbert <t...@herbertland.com> > > Cc: NVO3 <nvo3@ietf.org>; int-a...@ietf.org; Service Function > > Chaining IETF list <s...@ietf.org>; IETF IPPM WG <i...@ietf.org> > > Subject: Re: [ippm] [Int-area] encapsulation of IOAM data in various > > protocols - follow up from WG discussion in London > > > > > > > > Tom, > > > > > > > > On Thu, Apr 12, 2018 at 10:17 PM, Tom Herbert <t...@herbertland.com> wrote: > > > > Mickey, > > > > Looking at these ippm drafts more closely, I have a much more > > fundamental concern. > > > > In draft-brockners-ippm-ioam-geneve-00 for instance, there is the > > text in the introduction: > > > > "In-situ OAM (IOAM) records OAM information within the packet while > > the packet traverses a particular network domain. The term "in-situ" > > refers to the fact that the IOAM data fields are added to the data > > packets rather than is being sent within packets specifically > > dedicated to OAM. This document defines how IOAM data fields are > > transported as part of the Geneve [I-D.ietf-nvo3-geneve] > > encapsulation." > > > > I assume this means that as packets with Geneve encapsulation > > traverse the network they are interpreted by intermediate nodes as > > being Geneve. Since Geneve is a UDP encapsulation, then the > > destination UDP port number would be used to identify packets as > > being Geneve. So an intermediate device might be looking for UDP > > packets destined to port > > 6081 (the assigned UDP port for Geneve). If my understanding is > > correct, then this is a problem. > > > > UDP port numbers do not have global meaning. An intermediate device > > may very well see UDP packets destined to port 6081 that are not > > actually Geneve. This scenario is discussed in RFC7605: > > > > "...intermediate device interprets traffic based on the port number. > > It is important to recognize that any interpretation of port numbers > > -- except at the endpoints -- may be incorrect, because port numbers > > are meaningful only at the endpoints." > > > > If the UDP data is modified, as the draft would imply, then > > misinterpretation may also mean silent data corruption of packets. A > > protocol that would allow this seems pretty incorrect! Note that > > this would be true also for any UDP encapsulation that the network > > tries to interpret. > > > > > > > > The intention is to allow for multiple nodes that a packet traverses > > > > to be able to insert IOAM node information in the same trace option, > > > > but leave some flexibility regarding which nodes actually do the > > > > IOAM processing and the node information. This may vary > > > > depending on the transport. > > > > > > > > In case of a tunneled encapsulation such as Geneve or VXLAN, > > > > there may still be multiple hops. For example a network may use > > > > Geneve or VXLAN, but only do L2 processing at ToRs, with L3 > > > > processing done at aggregation or core switches. In this case > > > > many packets would do 2 Geneve or VXLAN hops, so the packet > > > > would contain IOAM node information from two nodes. > > > > > > > > Another example is service function chaining using Geneve or > > > > VXLAN rather than NSH. > > > > > > > > > > I am also wondering if hop-by-hop options been considered for this > > application? Their interpretation in the network is unabiguous and > > they also have the advantage that the work with any IP protocol or > > encapsulation. > > > > > > > > IPv6 hop-by-hop options has been considered. See > > > > draft-brockners-inband-oam-transport-05. This has not yet been > > > > broken out into a separate draft. > > > > > > > > Mickey > > > > > > > > > > Thanks, > > Tom > > > > > > On Thu, Apr 12, 2018 at 3:31 PM, Mickey Spiegel > > <mspie...@barefootnetworks.com> wrote: > > > >> Tom, > >> > >> On Thu, Apr 12, 2018 at 2:46 PM, Tom Herbert <t...@herbertland.com> wrote: > >>> > >>> On Thu, Apr 12, 2018 at 9:54 AM, Greg Mirsky > >>> <gregimir...@gmail.com> > >>> wrote: > >>> > Hi Frank, > >>> > thank you for sharing your points. Please find my notes in-line > >>> > and tagged > >>> > GIM>>. I believe that this is very much relevant to work of > >>> > GIM>>other > >>> > working > >>> > groups that directly work on the overlay encapsulations in the > >>> > center of the discussion and hence I've added them to the list. > >>> > Hope we'll have more opinions to reach the conclusion that is > >>> > acceptable to all. > >>> > > >>> > Regards, > >>> > Greg > >>> > > >>> > On Wed, Apr 11, 2018 at 12:02 PM, Frank Brockners (fbrockne) > >>> > <fbroc...@cisco.com> wrote: > >>> >> > >>> >> Back at the IPPM meeting in London, we discussed several drafts > >>> >> dealing with the encapsulation of IOAM data in various > >>> >> protocols (draft-brockners-ippm-ioam-vxlan-gpe-00, > >>> >> draft-brockners-ippm-ioam-geneve-00, > >>> >> draft-weis-ippm-ioam-gre-00). One discussion topic that we > >>> >> decided to take to the list was the question on whether > >>> >> draft-ooamdt-rtgwg-ooam-header could be leveraged.. After > >>> >> carefully considering draft-ooamdt-rtgwg-ooam-header, I came to > >>> >> the conclusion that the “OOAM header” does not meet the needs > >>> >> of > >>> >> IOAM: > >>> >> > >>> >> * Efficiency: IOAM adds data to live user traffic. As such, an > >>> >> encapsulation needs to be as efficient as possible. The “OOAM header” > >>> >> is 8 > >>> >> bytes long. The approach for IOAM data encapsulation in the > >>> >> above mentioned drafts only requires 4 bytes. Using the OOAM > >>> >> header approach would add an unnecessary overhead of 4 bytes – > >>> >> which is significant. > >>> Greg, > >>> > >>> I'm missing something here. I looked at the drafts you referenced > >>> and each of them looks like the overhead for OAM is greater that > >>> four bytes. In each there is some overhead equivalent to > >>> type/length, for instance in Geneve four bytes are needed for > >>> option class, type, and length. Unless the the OAM data is zero > >>> length, I don't see how this adds up to only four bytes of overhead. > >> > >> > >> The four versus eight bytes just refers to the fields in the four > >> bytes of IOAM info, that is common to all IOAM options. Beyond > >> that, there are IOAM option specific fields. For example if doing > >> one of the IOAM trace options, there are four bytes of trace option > >> header, including the IOAM-trace-type, NodeLen, Flags, and > >> RemainingLen fields. These are followed by the node data list > >> containing the per hop IOAM information. > >> > >> In looking at the OOAM header content, it has nothing to do with > >> any of the IOAM information after the first four bytes. It contains > >> another variant of the information in the first four bytes of IOAM > >> info, spread out over eight bytes. > >> > >>> > >>> Tom > >>> > >>> > > >>> > GIM>> The difference in four octets is because OOAM Header: > >>> > > >>> > provides more flexibility, e.g. Flags field and Reserved fields; > >> > >> > >> The flags field only has one defined flag at the moment, for a > >> timestamp block. For IOAM trace we need per hop timestamps, which > >> the timestamp block cannot address, i.e. the timestamp block is > >> redundant for > IOAM. > >> > >>> > >>> > supports larger OAM packets than iOAM header; > >> > >> > >> For IOAM purposes, 1020 octets is more than enough. > >> > >>> > >>> > is future proof by supporting versioning (Version field). > >> > >> > >> IMO, taking the first two bits of the IOAM-Type to define a Version > >> field would be a good thing. This does not require adding four more > >> bytes of overhead. 64 IOAM-Types is more than enough. > >> > >>> > >>> >> > >>> >> * Maturity: IOAM has several implementations, which were also > >>> >> shown at recent IETF hackathons – and we’re expecting > >>> >> additional implementations to be publicized soon. Interoperable > >>> >> implementations need timely specifications. Despite the > >>> >> question being asked, the recent thread on OOAM in the NVO3 > >>> >> list hasn’t revealed any implementation of the OOAM header. > >>> >> In > >>> >> addition, the thread revealed that several fundamental > >>> >> questions about the OOAM header are still open, such as whether > >>> >> or how active OAM mechanisms within protocols such as Geneve > >>> >> would apply to the OOAM header. This ultimately means that we > >>> >> won’t get to a timely specification. > >>> > > >>> > GIM>> May I ask which encapsulations supported by the > >>> > GIM>> implementations > >>> > you > >>> > refer to. Until very recently all iOAM proposals were to use > >>> > meta-data TLV in, e.g. Geneve and NSH. And if these or some of > >>> > these implementations already updated to the newly proposed iOAM > >>> > shim, I don't see problem in making them use OOAM Header. Would > >>> > you agree? > >>> > > >>> >> > >>> >> * Scope: It isn’t entirely clear to which protocols the OOAM > >>> >> header would ultimately apply to. The way the OOAM header is > >>> >> defined, OOAM uses a 8-bit field for “Next Prot”, the next > >>> >> protocol. Some protocols that IOAM data needs to be > >>> >> encapsulated into use 16-bits for their next protocol code points. See > >>> >> e.g. > >>> >> the GRE encapsulation – as specified in > >>> >> draft-weis-ippm-ioam-gre-00. > >>> > > >>> > GIM>> The first paragraph of the Introduction section states: > >>> > New protocols that support overlay networks like VxLAN-GPE > >>> > [I-D.ietf-nvo3-vxlan-gpe], GUE [I-D.ietf-nvo3-gue], Geneve > >>> > [I-D.ietf-nvo3-geneve], BIER > >>> > [I-D.ietf-bier-mpls-encapsulation], > and > >>> > NSH [I-D.ietf-sfc-nsh] support multi-protocol payload, e.g. > >>> > Ethernet, IPv4/IPv6, and recognize Operations, Administration, and > >>> > Maintenance (OAM) as one of distinct types. That ensures that > >>> > Overlay OAM (OOAM)packets are sharing fate with Overlay data packet > >>> > traversing the underlay. > >>> > I'm updating the OOAM Header draft and along with cleaning nits > >>> > will update reference to GUE. I think that the list and the > >>> > statemnt are quite clear in identifying the scope of networks > >>> > that may benefit from using not only common OOAM Header but > >>> > common OOAM mechanisms, e.g. Echo Request/Reply. > >>> > > >>> >> With the above in mind, I’d suggest that the WG moves forward > >>> >> with specific definitions for encapsulating IOAM data into > >>> >> protocols – per the above mentioned drafts. > >>> >> > >>> >> > >>> >> > >>> >> Regards, Frank > >>> >> > >>> >> > >>> >> _______________________________________________ > >>> >> ippm mailing list > >>> >> i...@ietf.org > >>> >> https://www.ietf.org/mailman/listinfo/ippm > >>> >> > >>> > > >>> > > >>> > _______________________________________________ > >>> > Int-area mailing list > >>> > int-a...@ietf.org > >>> > https://www.ietf.org/mailman/listinfo/int-area > >>> > > >>> > >>> _______________________________________________ > >>> ippm mailing list > >>> i...@ietf.org > >>> https://www.ietf.org/mailman/listinfo/ippm > >> > >> > > > > _______________________________________________ nvo3 mailing list nvo3@ietf.org https://www.ietf.org/mailman/listinfo/nvo3
