Mike, thank you for taking the time to so thoroughly understand what is going on. See below for my responses marked as “>>”:
Fred From: C. M. Heard <he...@pobox.com> Sent: Monday, November 18, 2024 3:13 PM To: Templin (US), Fred L <fred.l.temp...@boeing.com> Cc: Gorry Fairhurst <go...@erg.abdn.ac.uk>; Joe Touch <to...@strayalpha.com>; Tom Herbert <t...@herbertland.com>; int-area <int-area@ietf.org>; 6man <i...@ietf.org>; TSVWG <ts...@ietf.org> Subject: Re: [tsvwg] Re: UDP options [was IP Parcels and Advanced Jumbos (AJs)] Fred, I did some homework in the last few days, and I would describe your characterization of my comments as "under-informed" as being overly charitable :-( My apologies for wasting your time and everyone else's. This tutorial<https://www.sipanda.io/post/segmentation-offload-and-protocols-let-s-be-friends> from Tom Herbert, along socket API documentation for Windows<https://learn.microsoft.com/en-us/windows/win32/winsock/ipproto-udp-socket-options> and Linux<https://man7.org/linux/man-pages/man7/udp.7.html>, made it amply clear what is going on: GSO and GRO are specialized implementation techniques. In particular, the UDP variants, which are sometimes called USO and URO, allow an application to send or receive, in a single API call, a block of J datagrams, the first J-1 of which are all the same size L, with the Jth being no larger than the others. The sequence of packets that appear on the wire is not any different than what would have been sent had the application used J separate send API calls. USO and URO can be used independently by the two endpoints. When URO is used, the segment size L is attached to the packet's metadata when it is passed to the application so that upper layers can deduce the boundaries of the original UDP packets. The usage is sketched in Appendix D of draft-templin-6man-parcels2. >> This is all correct so far. If I (finally) understand this all correctly, the IP Parcels specification provides a means to package the sequence of UDP datagrams that would be created by the USO API inside of a single IPv4 or IPv6 packet. If that IPv4 or IPv6 packet needs to be forwarded to a node that does not understand IP Parcels, it is converted into the same series of separately transmitted UDP datagrams that would be generated on the wire by USO today, except that a Parcel Parameters Option is appended to each of those UDP datagrams. The intent is that a receiving node that does not understand parcels or the Parcel Parameters Option would process those datagrams as it does today. URO should also work as it does today, provided that the presence of an options trailer does not interfere with URO as presently implemented (which as far as I know is an open question). >> An important clarification – parcels may only be sent over paths in which >> all initial forwarding >> nodes (possibly up to and including the final destination) recognize the >> parcel format. This >> means that the path needs to be probed in advance to qualify it for parcel >> service before any >> parcels can be sent. If the probe reaches an intermediate forwarding node >> that understands >> parcels but is aware that the next hop does not, that forwarding node >> becomes the terminal >> point for parcel transmissions and must transform any parcels into >> individual UDP packets >> via packetization. With that being said, I do have some comments about the treatment of UDP options in IP Parcels, plus some more general ones: 1.) The plan of placing the UDP options common to all UDP datagrams in the parcel in a trailer will surely work; I understand that the two bytes immediately preceding the end of the parcel, as indicated by the Parcel Payload Length, contain the option field length, and it is not part of the options. But it is not necessary to do this. The UDP options could just as easily be placed before or after the parcel integrity block. The only reasons UDP options go in a trailer in regular UDP datagrams is because there is no other place to put them and still be backward compatible with implementations that are unaware of UDP options. IP Parcels amount to a new protocol, and the layout of a parcel is not constrained by backward compatibility. >> I get what you are saying, but I don’t like the idea of having a >> variable-length UDP options field immediately >> following the Parcel Integrity Block. I would rather keep UDP options as a >> trailer the same as for ordinary >> UDP/IP packets. Plus, the 2-octet UDP Option Length field is only necessary >> for parcels and AJs that >> exceed 64KB, where adding an extra 2-octet trailer would impart very little >> overhead. 2.) If the intent is that IP Parcels for UDP will use the same API as USO, there seems to be an implicit assumption that there will be an extension of that API that will allow for specification of a set of UDP options common to all datagrams in the parcel. As Tom Herbert noted in an earlier message, whether or not things will play out this way is an open question. Indeed, as far as I can tell, the current UDP options draft (now in AD review) defines only one option -- TIME -- that would actually be useful in this context, and that option is not one of the ones that are mandatory to implement. >> Yes, the UDP options would have to be either common to all segments >> (datagrams) in the parcel >> or specific to only one of the segments. So, if we say for example that the >> options are specific to >> the first segment in the parcel that still seems like a reasonable >> application of UDP options. If >> other segments of the parcel really need to include their own specific UDP >> options, then they >> can be packaged as the first segments of separate parcels. 3.) Perhaps I missed it, and I apologize if so, but I didn't see in draft-templin-6man-parcels2 what Next Header value would be placed in the IPv6 extension header immediately preceding the parcel payload area (typically the extension header would be the Hop-by-Hop options header containing the IPv6 Parcel Payload Option, but it could also be a Destination Options header if one followed the HbH Options header). That Next Header value should NOT be 6 (TCP) or 17 (UDP), because the IPv6 Payload Length is non-zero and the parcel payload is not laid out like a conventional TCP or UDP packet but is something else altogether, despite containing a notional TCP or UDP header. It is NOT sufficient to rely on the IPv6 Parcel Payload Option to let receivers know the difference, for a parcel-unaware receiver may well be configured to ignore and skip over any HbH Options Header or, if it is configured to process the HbH Options header, may not recognize the option and skip over it. Having a unique Next Header value (or Protocol Number) to identify a parcel payload would ensure that a parcel-unaware receiver would not misinterpret a parcel as something else. I would strongly recommend that the draft be modified to specify this and to include an appropriate request to the IANA. Granted, it would be necessary to add something to the Parcel Payload Option or to the parcel payload itself to identify the protocol it carries. Note that the RFC 2675 Jumbo Payload Option did not need this because the IPv6 Payload Length in that spec is zero, and receivers who do not implement that option would see an empty IPv6 payload and drop the packet. >> I think there is an important point of clarification here – parcels and AJs >> may only be sent over >> paths that have been pre-qualified to ensure that all forwarding nodes in >> the path recognize >> and correctly process the constructs. There will never be a case where an >> intermediate system >> or final destination is parcel-unaware and skips over the HBH options >> header. This allows the >> HBH option Next Header value to encode TCP or UDP. 4.) Note that comments (1) and (3) above also apply to Advanced Parcels. >> I think you meant to say “Advanced Jumbos (AJs)”? 5.) Given that the TCP or UDP Parcel Parameters Option is expendable -- that is, correctness will not be sacrificed if it is ignored or omitted -- I need to question its utility. Noting that it is different in every packet, Tom's tutorial<https://www.sipanda.io/post/segmentation-offload-and-protocols-let-s-be-friends> suggests that it would interfere with existing TCP GRO implementations. For UDP GRO the question is open whether or not the presence of a UDP options trailer (aka non-empty surplus area) of any kind (constant or not) would interfere with packet coalescing, but it's a possibility. Should either of these be true, we may be better off to omit the Parcel Parameters Option altogether. >> This is a fair point – existing GRO implementations are already able to >> coalesce the UDP >> datagrams sent by GSO in some fashion and deliver a multi-segment buffer to >> the receive >> side socket buffer. But, there is currently nothing in the datagrams sent by >> GRO to indicate >> an identification for the original multi-segment buffer nor the relative >> segment offset in the >> original multi-segment buffer. This means that GRO could potentially deliver >> segments out >> of their original order, or possibly even deliver multi-segment buffers >> that include a different >> packaging of segments than was originally intended by the GSO sender. So, >> the Parcel >> Parameters Option addresses these two issues when the GRO node recognizes and >> correctly processes the option. Tom Herbert raised the question of compatibility with checksum offload, and I shall deal with that in another message. Thanks and regards. Mike Heard On Fri, Nov 15, 2024 at 11:25 AM Templin (US), Fred L <fred.l.temp...@boeing.com<mailto:fred.l.temp...@boeing.com>> wrote: Mike, your statement about not respecting datagram boundaries is incorrect and under-informed – that is not what is happening. If you want documentary evidence for GSO/GRO, it is found at www.kernel.org<http://www.kernel.org>. From: C. M. Heard <he...@pobox.com<mailto:he...@pobox.com>> Sent: Friday, November 15, 2024 10:48 AM To: Templin (US), Fred L <fred.l.temp...@boeing.com<mailto:fred.l.temp...@boeing.com>> Cc: Gorry Fairhurst <go...@erg.abdn.ac.uk<mailto:go...@erg.abdn.ac.uk>>; Joe Touch <to...@strayalpha.com<mailto:to...@strayalpha.com>>; Tom Herbert <t...@herbertland.com<mailto:t...@herbertland.com>>; int-area <int-area@ietf.org<mailto:int-area@ietf.org>>; 6man <i...@ietf.org<mailto:i...@ietf.org>>; TSVWG <ts...@ietf.org<mailto:ts...@ietf.org>> Subject: [EXTERNAL] Re: [tsvwg] Re: UDP options [was IP Parcels and Advanced Jumbos (AJs)] Fred, Please provide documentary evidence that GSO/GRO implementations send UDP packets across the Internet without respecting the datagram boundaries set by the original sender and without breaking the applications layered on top of them. Note the word "UDP" in the above. Mike On Fri, Nov 15, 2024 at 10:36 AM Templin (US), Fred L <fred.l.temp...@boeing.com<mailto:fred.l.temp...@boeing.com>> wrote: GSO/GRO for UDP are widely deployed and widely used in modern Internetworking. There is no published RFC for them (yet), but popular implementations invoke them all the time in actual practice. So, the “reference” would be the source code of the popular implementations. Fred From: C. M. Heard <he...@pobox.com<mailto:he...@pobox.com>> Sent: Friday, November 15, 2024 10:20 AM To: Templin (US), Fred L <fred.l.temp...@boeing.com<mailto:fred.l.temp...@boeing.com>> Cc: Gorry Fairhurst <go...@erg.abdn.ac.uk<mailto:go...@erg.abdn.ac.uk>>; Joe Touch <to...@strayalpha.com<mailto:to...@strayalpha.com>>; int-area <int-area@ietf.org<mailto:int-area@ietf.org>>; 6man <i...@ietf.org<mailto:i...@ietf.org>>; TSVWG <ts...@ietf.org<mailto:ts...@ietf.org>> Subject: Re: [tsvwg] Re: UDP options [was IP Parcels and Advanced Jumbos (AJs)] Fred, Please provide a reference with evidence for that statement as it applies to UDP. Mike On Fri, Nov 15, 2024 at 10:00 AM Templin (US), Fred L <fred.l.temp...@boeing.com<mailto:fred.l.temp...@boeing.com>> wrote: Hi Mike, This is exactly what happens with Generic Segment Offload (GSO) and Generic Receive Offload (GRO) in real networks today. After a multi-segment GSO buffer is broken into individual IP packets during packetization, the receiver applies GRO to restore the multi-segment buffer if possible; otherwise, it delivers the individual IP packets to the upper layer. Each individual IP packet is an atomic unit that will be understood by upper layers even if it is not restored into the original multi-segment buffer originally prepared by GSO. This is the way GSO/GRO work today, and IP parcels does not change that. Fred From: C. M. Heard <he...@pobox.com<mailto:he...@pobox.com>> Sent: Friday, November 15, 2024 9:36 AM To: Templin (US), Fred L <fred.l.temp...@boeing.com<mailto:fred.l.temp...@boeing.com>> Cc: Gorry Fairhurst <go...@erg.abdn.ac.uk<mailto:go...@erg.abdn.ac.uk>>; Joe Touch <to...@strayalpha.com<mailto:to...@strayalpha.com>>; int-area <int-area@ietf.org<mailto:int-area@ietf.org>>; 6man <i...@ietf.org<mailto:i...@ietf.org>>; TSVWG <ts...@ietf.org<mailto:ts...@ietf.org>> Subject: [EXTERNAL] Re: [tsvwg] Re: UDP options [was IP Parcels and Advanced Jumbos (AJs)] EXT email: be mindful of links/attachments. Fred, I very strongly disagree with your statement that it is not an error to "it will instead deliver each of the individual IP packets to upper layers without restoring the parce." That amounts to delivering PIECES of the UDP packet sent by the originator to the upper layer. This would be exactly equivalent to allowing a receiver that does not understand the UDP FRAG option to deliver the payload of each individual fragment to the upper layer. The UDP options draft goes to considerable effort to ensure that this does not happen. There is long-standing precedent that the boundaries of UDP datagrams are preserved during transmission across the Internet. Many if not all UDP-based protocols depend on that, explicitly or implicitly. I do not understand why this point is even considered open for discussion. Mike On Fri, Nov 15, 2024 at 8:26 AM Templin (US), Fred L <fred.l.temp...@boeing.com<mailto:fred.l.temp...@boeing.com>> wrote: Hi Mike, Thank you for looking and commenting. A UDP/IP parcel containing N segments that undergoes packetization somewhere along the path will arrive at the final destination as N individual IP packets, each containing a Parcel Parameters UDP option. If the destination recognizes the option, it will restore the N segment parcel within the kernel before delivering the entire parcel as a single unit to upper layers. If the destination does not recognize the option, it will instead deliver each of the individual IP packets to upper layers without restoring the parcel which is not an error. Very significantly, each of the individual IP packets can be dealt with as standalone units once packetization has been applied; it is just that if the receive-side does not apply restoration (i.e., GRO) the performance optimization will be lost at that end. So, I don’t think there is any problem to worry about. Fred Templin From: C. M. Heard <he...@pobox.com<mailto:he...@pobox.com>> Sent: Thursday, November 14, 2024 4:55 PM To: Templin (US), Fred L <fred.l.temp...@boeing.com<mailto:fred.l.temp...@boeing.com>> Cc: Gorry Fairhurst <go...@erg.abdn.ac.uk<mailto:go...@erg.abdn.ac.uk>>; Joe Touch <to...@strayalpha.com<mailto:to...@strayalpha.com>>; int-area <int-area@ietf.org<mailto:int-area@ietf.org>>; 6man <i...@ietf.org<mailto:i...@ietf.org>>; TSVWG <ts...@ietf.org<mailto:ts...@ietf.org>> Subject: Re: [tsvwg] Re: UDP options [was IP Parcels and Advanced Jumbos (AJs)] Fred and WG participants, I have finally freed up some cycles to read draft-templin-6man-parcels2-14<https://datatracker.ietf.org/doc/html/draft-templin-6man-parcels2-14>, and I have found some issues with it that need to be addressed with respect to its handling of UDP. The big one -- if I have correctly understood what I have read -- is that it's possible for a single parcel of a parcellated UDP packet to be turned into a stand-alone UDP packet (see Section 7.1<https://datatracker.ietf.org/doc/html/draft-templin-6man-parcels2#name-packetization-over-non-parc>) and delivered to an end system as such (see Section 7.4<https://datatracker.ietf.org/doc/html/draft-templin-6man-parcels2#name-final-destination-restorati>). That packet would contain a Parcel Parameters UDP Option to tell the endpoint host that the packet is a parcel and not a complete UDP datagram, but the option kind is taken from the SAFE option space (KIND = 127; see draft-ietf-tsvwg-udp-options#section-10<https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-udp-options#section-10>). Legacy endpoints that do not understand UDP options will ignore that SAFE option and will deliver the parcel as if it were a complete UDP datagram. That, to my mind, is completely unacceptable. Unlike TCP, which is a byte-stream protocol in which segment boundaries have no meaning for the upper layer, UDP is a datagram protocol in which message boundaries are meaningful to the upper layer. The protocol has a contract with the upper layer to deliver a message as it was submitted or not at all. Delivering a parcel in a manner that can be misinterpreted as a complete datagram violates that contract. It is possible to repair this defect by making the Parcel Parameters Option, or something with equivalent functionality, into an UNSAFE option. My suggestion would be to define an UNSAFE version of the existing FRAG option (see draft-ietf-tsvwg-udp-options#section-11.4<https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-udp-options#section-11.4>) -- let's call it UFRAG -- that would allow for packet sizes greater than 65,535 bytes. The same option could be used to send singleton advanced jumbo packets as atomic fragments. This would avoid any need to modify the base UDP and UDP Options specifications. Additionally: during the review of draft-ietf-tsvwg-udp-options<https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-udp-options>, Joe Touch correctly pointed out that RFC 2675<https://datatracker.ietf.org/doc/html/rfc2675> (and its predecessor RFC 2147<https://datatracker.ietf.org/doc/html/rfc2147>) failed to note that it updated RFC 768<https://datatracker.ietf.org/doc/html/rfc768>. Similar concerns apply to TCP. If this draft foes forward, it should note that it updates the UDP and TCP specifications, and it should get buy-in from TSVWG and TCPM. Thanks and regards, Mike Heard On Sun, Sep 29, 2024 at 3:51 PM C. M. Heard <he...@pobox.com<mailto:he...@pobox.com>> wrote: Fred, I currently hold the editing pen for the changes to the UDP Options draft that have been requested prior to the shepherd report, and my intention is to remain silent about how, if at all, IP Parcels and Advanced Jumbos (AJs) will support UDP Options. I'll provide comments on the IP Parcels and Advanced Jumbos work at a later date, when I have spare intellectual cycles to fully comprehend the contents of draft-templin-6man-parcels2. At this point I must confess that, like Brian, I do not understand how a receiver will locate the options trailer in the case of an IP Payload Length exceeding 65535. Like Joe, I think it would be better to put the options just after the UDP header and make a new UNSAFE option to delimit the position where the options end and the user data begins.. But that discussion (and the corresponding update to the UDP options draft) can occur when it is ripe; IMO that is not the case at this time. Respectfully, Mike Heard On Sun, Sep 29, 2024 at 3:07 PM Templin (US), Fred L <Fred.L.Templin=40boeing....@dmarc.ietf.org<mailto:40boeing....@dmarc.ietf.org>> wrote: IP parcels and Advanced Jumbos (AJs) of all sizes ranging from 1 to 2^32 are now eligible for using UDP options. This is just one way in which they offer a better service than RFC2675 Jumbograms, but there are also many others. Joe, you can either note this in your draft or just leave it be and let my draft do an “updates UDP options”. Thank you - Fred From: Brian Carpenter <brian.e.carpen...@gmail.com<mailto:brian.e.carpen...@gmail.com>> Sent: Saturday, September 28, 2024 2:20 AM To: Gorry (erg) <go...@erg.abdn.ac.uk<mailto:go...@erg.abdn.ac.uk>> Cc: Joe Touch <to...@strayalpha.com<mailto:to...@strayalpha.com>>; Templin (US), Fred L <fred.l.temp...@boeing.com<mailto:fred.l.temp...@boeing.com>>; Tim Chown <tim.ch...@jisc.ac.uk<mailto:tim.ch...@jisc.ac.uk>>; Internet Area <Int-area@ietf.org<mailto:Int-area@ietf.org>>; IPv6 List <i...@ietf.org<mailto:i...@ietf.org>>; tsvwg IETF list <ts...@ietf.org<mailto:ts...@ietf.org>> Subject: [EXTERNAL] Re: [tsvwg] Re: UDP options [was IP Parcels and Advanced Jumbos (AJs)] EXT email: be mindful of links/attachments. That works for me.. (via tiny screen & keyboard) Regards, Brian Carpenter On Sat, 28 Sept 2024, 19:08 Gorry (erg), <go...@erg.abdn.ac.uk<mailto:go...@erg.abdn.ac.uk>> wrote: See below > On 28 Sep 2024, at 04:05, Brian E Carpenter > <brian.e.carpen...@gmail.com<mailto:brian.e.carpen...@gmail.com>> wrote: > > Joe, > On 28-Sep-24 03:13, to...@strayalpha.com<mailto:to...@strayalpha.com> wrote: >>>> On Sep 27, 2024, at 7:58 AM, Templin (US), Fred L >>>> <Fred.L.Templin=40boeing....@dmarc.ietf.org<mailto:40boeing....@dmarc.ietf.org>> >>>> wrote: >>> >>>> Indeed. But if sendmsg() and recvmsg() can and do generate RFC2675 >>>> packets, it means that any discussion of obsoleting RFC2675 should be >>>> off the table. >>> >>> No one that I know of has suggested obsoleting RFC2675 - my documents do >>> not say "obsoletes" (nor even "updates”). >> That approach to UDP jumbo grams is incompatible with UDP options. >> And yes, there was a proposal to move that RFC to historic: >> Jones, T., G. Fairhurst, "Change Status of RFC 2675 to Historic," >> draft-jones-6man-historic-rfc2675, May 2019. >> We COULD have a new option with a longer length, but that’s not in our >> baseline draft. > > Wouldn't that be tricky, because the options follow the whole payload as I > understand it? So a JumboUDPgram has to be received in full, however big it > is, before the option saying that it's a jumbo can be received and > interpreted. > > Where the udp-options draft says: > >>> The technique has been proposed for deprecation [Jo19]. > > I think you'd better change it to something like: > > The technique is known to be in active use in special situations, so cannot > reasonably be deprecated. However, users of this technique cannot > simultaneously use UDP options. > > Brian > I do not think the I-D needs to say anything about the deployment status of jumbograms, that another topic. I suggest if people wish, we just say that users of this technique can or cannot simultaneously use UDP options. Gorry > >
_______________________________________________ Int-area mailing list -- int-area@ietf.org To unsubscribe send an email to int-area-le...@ietf.org
