Re: [Int-area] Jumbo frame side meeting at IETF118 - notes
The question is not necessarily one of hardware vs. software; but it is one of end-to-end vs. the middle. From: Int-area On Behalf Of Robinson, Herbie Sent: Tuesday, December 19, 2023 3:49 PM To: Templin (US), Fred L ; Paul Vixie ; Tom Herbert ; Christian Huitema Cc: Gorry (erg) ; int-area Subject: Re: [Int-area] Re: Jumbo frame side meeting at IETF118 - notes At some point, it becomes ridiculous to keep band-aiding 40+ year old hardware designs with ever increasing software complexity. The hard thing to know is when you hit that tipping point. From: Templin (US), Fred L mailto:Fred.L.Templin=40boeing@dmarc.ietf.org>> Sent: Tuesday, December 19, 2023 4:44 PM To: Robinson, Herbie mailto:herbie.robin...@stratus.com>>; Paul Vixie mailto:p...@redbarn.org>>; Tom Herbert mailto:t...@herbertland.com>>; Christian Huitema mailto:huit...@huitema.net>> Cc: Gorry (erg) mailto:go...@erg.abdn.ac.uk>>; int-area mailto:int-area@ietf.org>> Subject: RE: [Int-area] [EXTERNAL] Re: Jumbo frame side meeting at IETF118 - notes [EXTERNAL SENDER: This email originated from outside of Stratus Technologies. Do not click links or open attachments unless you recognize the sender and know the content is safe.] Herbie, an alternative to trying to force advanced integrity checks into hardware is for the source to include a higher fidelity integrity check along with the jumbo, then the destination gets to verify the integrity after the jumbo has traversed the path. That way, there is still good integrity coverage even if all links in the path are still only using CRC32. Fred From: Int-area mailto:int-area-boun...@ietf.org>> On Behalf Of Robinson, Herbie Sent: Monday, December 18, 2023 5:31 PM To: Paul Vixie mailto:paul=40redbarn@dmarc.ietf.org>>; Tom Herbert mailto:tom=40herbertland@dmarc.ietf.org>>; Christian Huitema mailto:huit...@huitema.net>> Cc: Gorry (erg) mailto:go...@erg.abdn.ac.uk>>; int-area mailto:int-area@ietf.org>> Subject: Re: [Int-area] [EXTERNAL] Re: Jumbo frame side meeting at IETF118 - notes If we are going much beyond 9K, the hardware has to change, because a 32 bit CRC doesn’t cut it for really large packets. If the hardware has to change, we can push MTU negotiation into the hardware. And completely bypass the momentum involved with getting every existing implementation on the planet to actually implement PTB. ___ Int-area mailing list Int-area@ietf.org https://www.ietf.org/mailman/listinfo/int-area
Re: [Int-area] Jumbo frame side meeting at IETF118 - notes
Tim and Gorry, thank you for this meeting report-out. I have a draft out right now that I think addresses most if not all of the points regarding larger packet sizes. The draft was aligned with intarea for a long time but was recently re-aligned with 6man and refactored to focus on IPv6. I did leave behind an intarea-aligned draft on IPv4 dependencies also though. The draft is called: "IPv6 Parcels and Advanced Jumbos" and brings a comprehensive portfolio of techniques necessary to deal with these larger sizes in both the Internet and other generalized Internetworking scenarios. It is worth a look if you haven't already seen it, and worth another look if you already have. Thank you - Fred > -Original Message- > From: Int-area On Behalf Of Tim Chown > Sent: Monday, December 18, 2023 8:15 AM > To: int-area@ietf.org > Cc: Gorry (erg) > Subject: [Int-area] Jumbo frame side meeting at IETF118 - notes > > Hi, > > Apologies for the delay in posting these notes. Gorry and I held a side > meeting in Prague on the topic of (lack of) use of jumbo frames, and > what topics might lie within the IETF’s remit to help promote greater use. > > After talking to an AD it was suggested we raise the topic on the int-area > list to gauge interest, rather than set up a new list at this stage. > > So, all thoughts and comments welcome... > > -- > > Jumbo frame side meeting, IETF118, 2-3pm Thu 9 November > > Convened by Tim Chown (Jisc) and Gorry Fairhurst (Univ Aberdeen) > > The meeting had no set agenda. The aim was to gather those interested in more > widespread use of jumbo frames to gather and discuss > what actions might be taken in or by the IETF and its WGs towards that goal. > > Comments: > • There is no standard for Ethernet for frame sizes above 1500 bytes > • Would it be useful to work towards a “certified jumbo” interoperability > test? > • NICs at 1Gbit/s+ should all use phase-locked loop (PLL). > • What tools should we use to identify issues or errors in transmission > at various MTU sizes? > • Tim noted that Jisc’s 100G perfSONAR node at London showed no errors on > its 9000 MTU interface – stats can be seen under the > interface details section at https://ps-london-bw.perf.ja.net/toolkit/ > • We should consider the relevance of MTU in respective IETF areas – INT, > TSV and OPS > • Jen Linkova has talked about networks with multiple sizes of MTU > • There are providers who offer 9000 MTU networks, end-to-end, such as > Hurrican Electric > • Tim reported that many PBs of data are moved by the CERN experiments > and a proportion of that is using 9000 MTU. Single stream TCP > performance can be 2-3x better, depending on RTT and other factors. > • What issues might there be in specific technologies, e.g. ND, BGP, > ECMP, multipath TCP, …? > • There is a perception that IXPs find 9000 MTU problematic > • There are previous IETF I-Ds on MTU use, e.g. in IXPs – we should look > at old drafts or any RFCs > • There may be relevant presentations from *NOG and RIR member meetings > • Improvements to host stacks can make the performance gains of jumbo > frames less important, e.g. various offloading technologiesCan > we get current measurements and data, e.g., via MAPRG? > • We should look at hyperscalers; there is support there for 9000 MTU > • IPsec, and any encapsulation that benefits from avoiding fragmentation, > can work better with jumbo frames > • We could look a Globus transfer logs to detect MD5 errors for evidence > of issues in the application data not picked up at lower layers > • There are other non-Ethernet technologies used in DCs with large frames > • Does QUIC break offload due to its encryption? In practice QUIC uses a > Max Datagram Packet size less than 1500. Might larger MTUs > be useful for QUIC > • Post-quantum scenarios were mentioned. > • What about MTU discovery? There is anecdotal evidence of issues; Tim > has seen this at a UK university where ICMPv6 PTB was being > dropped. > • PLPMTUD is specified by QUIC; useful when there’s no path back to a > sender for receipt of an ICMP PTB message. > > Agreed actions: > • Tim will ask Eric Vyncke (INT area AD) for support to create a > “jumbo-discuss” IETF mail list > • We will seek to collectively document the status of jumbo frames, > focusing on what works (success stories), opportunities, gaps > (potential work items in the IETF and elsewhere) and other open issues. > • Tim will ask Eric Vyncke for a side meeting at a future IETF. > • We will seek to present relevant parts of the above documented status > in the INT, TSV and OPS area open meetings at t
Re: [Int-area] Jumbo frame side meeting at IETF118 - notes
We've got to teach the system how to negotiate and/or discover this. 9000 was about right for fast Ethernet but it's small at gigabit Ethernet and above. Petabit is probably coming within our lifetimes. 9000 would be a great starting point and 64k after that but like the ibmpc 640k memory threshold the lesson is that hard limits don't last. I've been happy with 9000 in my various campus networks. But more would be better. Max speed changing by no more than 6x isn't the issue. Complexity of work arounds, power utilization, cost, and future proofing are the drivers here. p vixie On Dec 18, 2023 12:52, Tom Herbert wrote: On Mon, Dec 18, 2023 at 11:24 AM Christian Huitema wrote: > > On 12/18/2023 9:15 AM, Kyle Rose wrote: > > Right, I should have said*at best* a 6x improvement. The point I'm trying > > to get to is: how much sense does it make to try to make the public > > internet safe for jumbo frames? I honestly don't know, and since I wasn't > > at the meeting, I don't know much much this was even a focus. > > It is certainly less that 6x, especially for encrypted transports. There > is a fixed cost per packet, but is corresponds more or less to the > encryption of a per packet header and checksum, so maybe 32 to 64 bytes. > After that, the cost of encryption is linear with the size of the message. > > That does not mean that we should not do it. How many of us remember > mocking ATM and its 48 byte packet size? The max speed of ATM circuits > then was maybe 150Mbps, and 48 bytes meant 2.6us. Guess what, at 10Gbps, > 1500 bytes means 1.2us... Christian, Right, and at 1Tbps 1500 bytes means 12ns! With respect to Internet protocols there's no reason to artificially limit MTUs to 1500 bytes, or for that matter even 9000 bytes or 64K bytes with IPv6. Tom > > -- Christian Huitema ___ Int-area mailing list Int-area@ietf.org https://www.ietf.org/mailman/listinfo/int-area ___ Int-area mailing list Int-area@ietf.org https://www.ietf.org/mailman/listinfo/int-area
Re: [Int-area] Jumbo frame side meeting at IETF118 - notes
On Mon, Dec 18, 2023 at 11:24 AM Christian Huitema wrote: > > On 12/18/2023 9:15 AM, Kyle Rose wrote: > > Right, I should have said*at best* a 6x improvement. The point I'm trying > > to get to is: how much sense does it make to try to make the public > > internet safe for jumbo frames? I honestly don't know, and since I wasn't > > at the meeting, I don't know much much this was even a focus. > > It is certainly less that 6x, especially for encrypted transports. There > is a fixed cost per packet, but is corresponds more or less to the > encryption of a per packet header and checksum, so maybe 32 to 64 bytes. > After that, the cost of encryption is linear with the size of the message. > > That does not mean that we should not do it. How many of us remember > mocking ATM and its 48 byte packet size? The max speed of ATM circuits > then was maybe 150Mbps, and 48 bytes meant 2.6us. Guess what, at 10Gbps, > 1500 bytes means 1.2us... Christian, Right, and at 1Tbps 1500 bytes means 12ns! With respect to Internet protocols there's no reason to artificially limit MTUs to 1500 bytes, or for that matter even 9000 bytes or 64K bytes with IPv6. Tom > > -- Christian Huitema ___ Int-area mailing list Int-area@ietf.org https://www.ietf.org/mailman/listinfo/int-area
Re: [Int-area] Jumbo frame side meeting at IETF118 - notes
On 12/18/2023 9:15 AM, Kyle Rose wrote: Right, I should have said*at best* a 6x improvement. The point I'm trying to get to is: how much sense does it make to try to make the public internet safe for jumbo frames? I honestly don't know, and since I wasn't at the meeting, I don't know much much this was even a focus. It is certainly less that 6x, especially for encrypted transports. There is a fixed cost per packet, but is corresponds more or less to the encryption of a per packet header and checksum, so maybe 32 to 64 bytes. After that, the cost of encryption is linear with the size of the message. That does not mean that we should not do it. How many of us remember mocking ATM and its 48 byte packet size? The max speed of ATM circuits then was maybe 150Mbps, and 48 bytes meant 2.6us. Guess what, at 10Gbps, 1500 bytes means 1.2us... -- Christian Huitema ___ Int-area mailing list Int-area@ietf.org https://www.ietf.org/mailman/listinfo/int-area
Re: [Int-area] Jumbo frame side meeting at IETF118 - notes
Right, I should have said *at best* a 6x improvement. The point I'm trying to get to is: how much sense does it make to try to make the public internet safe for jumbo frames? I honestly don't know, and since I wasn't at the meeting, I don't know much much this was even a focus. Thanks, Kyle On Mon, Dec 18, 2023, 11:58 AM Tom Herbert wrote: > On Mon, Dec 18, 2023 at 8:38 AM Kyle Rose > wrote: > > > > Apologies for not being able to make the meeting. Had I been able to > attend, the question I was going to ask was: with respect to overhead, > there's a constant factor 6x improvement when moving from 1500->9000 > octets. How quickly do hardware performance improvements typically reach 6x > packet-per-second throughput at ~the same cost (capex, power, etc.)? > > Kyle, > > It's not really constant. A larger MTU is opportunistic optimization, > it's only useful when the host is sending large amounts of data and > path MTU allows a larger MSS (for instance, we'll still see forty byte > pure ACKs being sent). There should be a reduction in packets but > probably much less than 6x I would think. > > Tom > > > > > Kyle > > > > On Mon, Dec 18, 2023 at 11:15 AM Tim Chown 40jisc.ac...@dmarc.ietf.org> wrote: > >> > >> Hi, > >> > >> Apologies for the delay in posting these notes. Gorry and I held a side > meeting in Prague on the topic of (lack of) use of jumbo frames, and what > topics might lie within the IETF’s remit to help promote greater use. > >> > >> After talking to an AD it was suggested we raise the topic on the > int-area list to gauge interest, rather than set up a new list at this > stage. > >> > >> So, all thoughts and comments welcome... > >> > >> -- > >> > >> Jumbo frame side meeting, IETF118, 2-3pm Thu 9 November > >> > >> Convened by Tim Chown (Jisc) and Gorry Fairhurst (Univ Aberdeen) > >> > >> The meeting had no set agenda. The aim was to gather those interested > in more widespread use of jumbo frames to gather and discuss what actions > might be taken in or by the IETF and its WGs towards that goal. > >> > >> Comments: > >> • There is no standard for Ethernet for frame sizes above 1500 bytes > >> • Would it be useful to work towards a “certified jumbo” > interoperability test? > >> • NICs at 1Gbit/s+ should all use phase-locked loop (PLL). > >> • What tools should we use to identify issues or errors in > transmission at various MTU sizes? > >> • Tim noted that Jisc’s 100G perfSONAR node at London showed no > errors on its 9000 MTU interface – stats can be seen under the interface > details section at https://ps-london-bw.perf.ja.net/toolkit/ > >> • We should consider the relevance of MTU in respective IETF areas > – INT, TSV and OPS > >> • Jen Linkova has talked about networks with multiple sizes of MTU > >> • There are providers who offer 9000 MTU networks, end-to-end, such > as Hurrican Electric > >> • Tim reported that many PBs of data are moved by the CERN > experiments and a proportion of that is using 9000 MTU. Single stream TCP > performance can be 2-3x better, depending on RTT and other factors. > >> • What issues might there be in specific technologies, e.g. ND, > BGP, ECMP, multipath TCP, …? > >> • There is a perception that IXPs find 9000 MTU problematic > >> • There are previous IETF I-Ds on MTU use, e.g. in IXPs – we should > look at old drafts or any RFCs > >> • There may be relevant presentations from *NOG and RIR member > meetings > >> • Improvements to host stacks can make the performance gains of > jumbo frames less important, e.g. various offloading technologiesCan we get > current measurements and data, e.g., via MAPRG? > >> • We should look at hyperscalers; there is support there for 9000 > MTU > >> • IPsec, and any encapsulation that benefits from avoiding > fragmentation, can work better with jumbo frames > >> • We could look a Globus transfer logs to detect MD5 errors for > evidence of issues in the application data not picked up at lower layers > >> • There are other non-Ethernet technologies used in DCs with large > frames > >> • Does QUIC break offload due to its encryption? In practice QUIC > uses a Max Datagram Packet size less than 1500. Might larger MTUs be > useful for QUIC > >> • Post-quantum scenarios were mentioned. > >> • What about MTU discovery? There is anecdotal evidence of issues; > Tim has seen this at a UK university where ICMPv6 PTB was being dropped. > >> • PLPMTUD is specified by QUIC; useful when there’s no path back to > a sender for receipt of an ICMP PTB message. > >> > >> Agreed actions: > >> • Tim will ask Eric Vyncke (INT area AD) for support to create a > “jumbo-discuss” IETF mail list > >> • We will seek to collectively document the status of jumbo frames, > focusing on what works (success stories), opportunities, gaps (potential > work items in the IETF and elsewhere) and other open issues. > >> • Tim will ask Eric Vyncke for a side
Re: [Int-area] Jumbo frame side meeting at IETF118 - notes
On Mon, Dec 18, 2023 at 8:38 AM Kyle Rose wrote: > > Apologies for not being able to make the meeting. Had I been able to attend, > the question I was going to ask was: with respect to overhead, there's a > constant factor 6x improvement when moving from 1500->9000 octets. How > quickly do hardware performance improvements typically reach 6x > packet-per-second throughput at ~the same cost (capex, power, etc.)? Kyle, It's not really constant. A larger MTU is opportunistic optimization, it's only useful when the host is sending large amounts of data and path MTU allows a larger MSS (for instance, we'll still see forty byte pure ACKs being sent). There should be a reduction in packets but probably much less than 6x I would think. Tom > > Kyle > > On Mon, Dec 18, 2023 at 11:15 AM Tim Chown > wrote: >> >> Hi, >> >> Apologies for the delay in posting these notes. Gorry and I held a side >> meeting in Prague on the topic of (lack of) use of jumbo frames, and what >> topics might lie within the IETF’s remit to help promote greater use. >> >> After talking to an AD it was suggested we raise the topic on the int-area >> list to gauge interest, rather than set up a new list at this stage. >> >> So, all thoughts and comments welcome... >> >> -- >> >> Jumbo frame side meeting, IETF118, 2-3pm Thu 9 November >> >> Convened by Tim Chown (Jisc) and Gorry Fairhurst (Univ Aberdeen) >> >> The meeting had no set agenda. The aim was to gather those interested in >> more widespread use of jumbo frames to gather and discuss what actions might >> be taken in or by the IETF and its WGs towards that goal. >> >> Comments: >> • There is no standard for Ethernet for frame sizes above 1500 bytes >> • Would it be useful to work towards a “certified jumbo” >> interoperability test? >> • NICs at 1Gbit/s+ should all use phase-locked loop (PLL). >> • What tools should we use to identify issues or errors in transmission >> at various MTU sizes? >> • Tim noted that Jisc’s 100G perfSONAR node at London showed no errors >> on its 9000 MTU interface – stats can be seen under the interface details >> section at https://ps-london-bw.perf.ja.net/toolkit/ >> • We should consider the relevance of MTU in respective IETF areas – >> INT, TSV and OPS >> • Jen Linkova has talked about networks with multiple sizes of MTU >> • There are providers who offer 9000 MTU networks, end-to-end, such as >> Hurrican Electric >> • Tim reported that many PBs of data are moved by the CERN experiments >> and a proportion of that is using 9000 MTU. Single stream TCP performance >> can be 2-3x better, depending on RTT and other factors. >> • What issues might there be in specific technologies, e.g. ND, BGP, >> ECMP, multipath TCP, …? >> • There is a perception that IXPs find 9000 MTU problematic >> • There are previous IETF I-Ds on MTU use, e.g. in IXPs – we should look >> at old drafts or any RFCs >> • There may be relevant presentations from *NOG and RIR member meetings >> • Improvements to host stacks can make the performance gains of jumbo >> frames less important, e.g. various offloading technologiesCan we get >> current measurements and data, e.g., via MAPRG? >> • We should look at hyperscalers; there is support there for 9000 MTU >> • IPsec, and any encapsulation that benefits from avoiding >> fragmentation, can work better with jumbo frames >> • We could look a Globus transfer logs to detect MD5 errors for evidence >> of issues in the application data not picked up at lower layers >> • There are other non-Ethernet technologies used in DCs with large frames >> • Does QUIC break offload due to its encryption? In practice QUIC uses >> a Max Datagram Packet size less than 1500. Might larger MTUs be useful for >> QUIC >> • Post-quantum scenarios were mentioned. >> • What about MTU discovery? There is anecdotal evidence of issues; Tim >> has seen this at a UK university where ICMPv6 PTB was being dropped. >> • PLPMTUD is specified by QUIC; useful when there’s no path back to a >> sender for receipt of an ICMP PTB message. >> >> Agreed actions: >> • Tim will ask Eric Vyncke (INT area AD) for support to create a >> “jumbo-discuss” IETF mail list >> • We will seek to collectively document the status of jumbo frames, >> focusing on what works (success stories), opportunities, gaps (potential >> work items in the IETF and elsewhere) and other open issues. >> • Tim will ask Eric Vyncke for a side meeting at a future IETF. >> • We will seek to present relevant parts of the above documented status >> in the INT, TSV and OPS area open meetings at the next IETF meeting. >> • Tim will email the 118attendees list with the meeting notes >> >> — >> >> Tim >> ___ >> Int-area mailing list >> Int-area@ietf.org >> https://www.ietf.org/mailman/listinfo/int-area > >
Re: [Int-area] Jumbo frame side meeting at IETF118 - notes
On Mon, Dec 18, 2023 at 8:15 AM Tim Chown wrote: > > Hi, > > Apologies for the delay in posting these notes. Gorry and I held a side > meeting in Prague on the topic of (lack of) use of jumbo frames, and what > topics might lie within the IETF’s remit to help promote greater use. > > After talking to an AD it was suggested we raise the topic on the int-area > list to gauge interest, rather than set up a new list at this stage. > > So, all thoughts and comments welcome... Sorry I missed the meeting, I have a couple of comments below... > > -- > > Jumbo frame side meeting, IETF118, 2-3pm Thu 9 November > > Convened by Tim Chown (Jisc) and Gorry Fairhurst (Univ Aberdeen) > > The meeting had no set agenda. The aim was to gather those interested in more > widespread use of jumbo frames to gather and discuss what actions might be > taken in or by the IETF and its WGs towards that goal. > > Comments: > • There is no standard for Ethernet for frame sizes above 1500 bytes > • Would it be useful to work towards a “certified jumbo” interoperability > test? > • NICs at 1Gbit/s+ should all use phase-locked loop (PLL). > • What tools should we use to identify issues or errors in transmission > at various MTU sizes? > • Tim noted that Jisc’s 100G perfSONAR node at London showed no errors on > its 9000 MTU interface – stats can be seen under the interface details > section at https://ps-london-bw.perf.ja.net/toolkit/ > • We should consider the relevance of MTU in respective IETF areas – INT, > TSV and OPS > • Jen Linkova has talked about networks with multiple sizes of MTU > • There are providers who offer 9000 MTU networks, end-to-end, such as > Hurrican Electric > • Tim reported that many PBs of data are moved by the CERN experiments > and a proportion of that is using 9000 MTU. Single stream TCP performance > can be 2-3x better, depending on RTT and other factors. This is going to be very dependent on implementation as well, particularly how effective GSO and GRO are and whether they are offloaded to hardware in TSO and LRO. > • What issues might there be in specific technologies, e.g. ND, BGP, > ECMP, multipath TCP, …? > • There is a perception that IXPs find 9000 MTU problematic > • There are previous IETF I-Ds on MTU use, e.g. in IXPs – we should look > at old drafts or any RFCs > • There may be relevant presentations from *NOG and RIR member meetings > • Improvements to host stacks can make the performance gains of jumbo > frames less important, e.g. various offloading technologiesCan we get current > measurements and data, e.g., via MAPRG? To a degree techniques like GSO and GRO help. But these are dependent on specific transport protocols. > • We should look at hyperscalers; there is support there for 9000 MTU I believe at least Google is using 9K MTUs in the datacenter for a while. This allows two 4K pages to be sent in TCP which maps to the OS concept of pages. When combined with header data split, this allows zero copy receive to user space via page flipping thereby eliminating expensive copies from the kernel to user space. Interestingly, there is a deployed use case of IPv6 jumbograms; this described in https://lwn.net/Articles/884104/ https://netdevconf.info/0x15/slides/35/BIG%20TCP.pdf. The basic idea is that the host stack can create TCP segments larger than 64K. GSO or TSO is always used to break up the jumbo segments into MTU sized segments for sending. In this way the jumbogram is never sent on the wire, but we get the performance advantages since it saves processing in the TCP stack. > • IPsec, and any encapsulation that benefits from avoiding fragmentation, > can work better with jumbo frames > • We could look a Globus transfer logs to detect MD5 errors for evidence > of issues in the application data not picked up at lower layers > • There are other non-Ethernet technologies used in DCs with large frames > • Does QUIC break offload due to its encryption? In practice QUIC uses a > Max Datagram Packet size less than 1500. Might larger MTUs be useful for QUIC > • Post-quantum scenarios were mentioned. > • What about MTU discovery? There is anecdotal evidence of issues; Tim > has seen this at a UK university where ICMPv6 PTB was being dropped. It's generally assumed that ICMP, including PTB, is unreliable. > • PLPMTUD is specified by QUIC; useful when there’s no path back to a > sender for receipt of an ICMP PTB message. > > Agreed actions: > • Tim will ask Eric Vyncke (INT area AD) for support to create a > “jumbo-discuss” IETF mail list > • We will seek to collectively document the status of jumbo frames, > focusing on what works (success stories), opportunities, gaps (potential work > items in the IETF and elsewhere) and other open issues. > • Tim will ask Eric Vyncke for a side meeting at a future IETF. > • We will seek to present relevant parts of the
Re: [Int-area] Jumbo frame side meeting at IETF118 - notes
Apologies for not being able to make the meeting. Had I been able to attend, the question I was going to ask was: with respect to overhead, there's a constant factor 6x improvement when moving from 1500->9000 octets. How quickly do hardware performance improvements typically reach 6x packet-per-second throughput at ~the same cost (capex, power, etc.)? Kyle On Mon, Dec 18, 2023 at 11:15 AM Tim Chown wrote: > Hi, > > Apologies for the delay in posting these notes. Gorry and I held a side > meeting in Prague on the topic of (lack of) use of jumbo frames, and what > topics might lie within the IETF’s remit to help promote greater use. > > After talking to an AD it was suggested we raise the topic on the int-area > list to gauge interest, rather than set up a new list at this stage. > > So, all thoughts and comments welcome... > > -- > > Jumbo frame side meeting, IETF118, 2-3pm Thu 9 November > > Convened by Tim Chown (Jisc) and Gorry Fairhurst (Univ Aberdeen) > > The meeting had no set agenda. The aim was to gather those interested in > more widespread use of jumbo frames to gather and discuss what actions > might be taken in or by the IETF and its WGs towards that goal. > > Comments: > • There is no standard for Ethernet for frame sizes above 1500 bytes > • Would it be useful to work towards a “certified jumbo” > interoperability test? > • NICs at 1Gbit/s+ should all use phase-locked loop (PLL). > • What tools should we use to identify issues or errors in > transmission at various MTU sizes? > • Tim noted that Jisc’s 100G perfSONAR node at London showed no errors > on its 9000 MTU interface – stats can be seen under the interface details > section at https://ps-london-bw.perf.ja.net/toolkit/ > • We should consider the relevance of MTU in respective IETF areas – > INT, TSV and OPS > • Jen Linkova has talked about networks with multiple sizes of MTU > • There are providers who offer 9000 MTU networks, end-to-end, such as > Hurrican Electric > • Tim reported that many PBs of data are moved by the CERN experiments > and a proportion of that is using 9000 MTU. Single stream TCP performance > can be 2-3x better, depending on RTT and other factors. > • What issues might there be in specific technologies, e.g. ND, BGP, > ECMP, multipath TCP, …? > • There is a perception that IXPs find 9000 MTU problematic > • There are previous IETF I-Ds on MTU use, e.g. in IXPs – we should > look at old drafts or any RFCs > • There may be relevant presentations from *NOG and RIR member meetings > • Improvements to host stacks can make the performance gains of jumbo > frames less important, e.g. various offloading technologiesCan we get > current measurements and data, e.g., via MAPRG? > • We should look at hyperscalers; there is support there for 9000 MTU > • IPsec, and any encapsulation that benefits from avoiding > fragmentation, can work better with jumbo frames > • We could look a Globus transfer logs to detect MD5 errors for > evidence of issues in the application data not picked up at lower layers > • There are other non-Ethernet technologies used in DCs with large > frames > • Does QUIC break offload due to its encryption? In practice QUIC > uses a Max Datagram Packet size less than 1500. Might larger MTUs be > useful for QUIC > • Post-quantum scenarios were mentioned. > • What about MTU discovery? There is anecdotal evidence of issues; > Tim has seen this at a UK university where ICMPv6 PTB was being dropped. > • PLPMTUD is specified by QUIC; useful when there’s no path back to a > sender for receipt of an ICMP PTB message. > > Agreed actions: > • Tim will ask Eric Vyncke (INT area AD) for support to create a > “jumbo-discuss” IETF mail list > • We will seek to collectively document the status of jumbo frames, > focusing on what works (success stories), opportunities, gaps (potential > work items in the IETF and elsewhere) and other open issues. > • Tim will ask Eric Vyncke for a side meeting at a future IETF. > • We will seek to present relevant parts of the above documented > status in the INT, TSV and OPS area open meetings at the next IETF meeting. > • Tim will email the 118attendees list with the meeting notes > > — > > Tim > ___ > Int-area mailing list > Int-area@ietf.org > https://www.ietf.org/mailman/listinfo/int-area > ___ Int-area mailing list Int-area@ietf.org https://www.ietf.org/mailman/listinfo/int-area
[Int-area] Jumbo frame side meeting at IETF118 - notes
Hi, Apologies for the delay in posting these notes. Gorry and I held a side meeting in Prague on the topic of (lack of) use of jumbo frames, and what topics might lie within the IETF’s remit to help promote greater use. After talking to an AD it was suggested we raise the topic on the int-area list to gauge interest, rather than set up a new list at this stage. So, all thoughts and comments welcome... -- Jumbo frame side meeting, IETF118, 2-3pm Thu 9 November Convened by Tim Chown (Jisc) and Gorry Fairhurst (Univ Aberdeen) The meeting had no set agenda. The aim was to gather those interested in more widespread use of jumbo frames to gather and discuss what actions might be taken in or by the IETF and its WGs towards that goal. Comments: • There is no standard for Ethernet for frame sizes above 1500 bytes • Would it be useful to work towards a “certified jumbo” interoperability test? • NICs at 1Gbit/s+ should all use phase-locked loop (PLL). • What tools should we use to identify issues or errors in transmission at various MTU sizes? • Tim noted that Jisc’s 100G perfSONAR node at London showed no errors on its 9000 MTU interface – stats can be seen under the interface details section at https://ps-london-bw.perf.ja.net/toolkit/ • We should consider the relevance of MTU in respective IETF areas – INT, TSV and OPS • Jen Linkova has talked about networks with multiple sizes of MTU • There are providers who offer 9000 MTU networks, end-to-end, such as Hurrican Electric • Tim reported that many PBs of data are moved by the CERN experiments and a proportion of that is using 9000 MTU. Single stream TCP performance can be 2-3x better, depending on RTT and other factors. • What issues might there be in specific technologies, e.g. ND, BGP, ECMP, multipath TCP, …? • There is a perception that IXPs find 9000 MTU problematic • There are previous IETF I-Ds on MTU use, e.g. in IXPs – we should look at old drafts or any RFCs • There may be relevant presentations from *NOG and RIR member meetings • Improvements to host stacks can make the performance gains of jumbo frames less important, e.g. various offloading technologiesCan we get current measurements and data, e.g., via MAPRG? • We should look at hyperscalers; there is support there for 9000 MTU • IPsec, and any encapsulation that benefits from avoiding fragmentation, can work better with jumbo frames • We could look a Globus transfer logs to detect MD5 errors for evidence of issues in the application data not picked up at lower layers • There are other non-Ethernet technologies used in DCs with large frames • Does QUIC break offload due to its encryption? In practice QUIC uses a Max Datagram Packet size less than 1500. Might larger MTUs be useful for QUIC • Post-quantum scenarios were mentioned. • What about MTU discovery? There is anecdotal evidence of issues; Tim has seen this at a UK university where ICMPv6 PTB was being dropped. • PLPMTUD is specified by QUIC; useful when there’s no path back to a sender for receipt of an ICMP PTB message. Agreed actions: • Tim will ask Eric Vyncke (INT area AD) for support to create a “jumbo-discuss” IETF mail list • We will seek to collectively document the status of jumbo frames, focusing on what works (success stories), opportunities, gaps (potential work items in the IETF and elsewhere) and other open issues. • Tim will ask Eric Vyncke for a side meeting at a future IETF. • We will seek to present relevant parts of the above documented status in the INT, TSV and OPS area open meetings at the next IETF meeting. • Tim will email the 118attendees list with the meeting notes — Tim ___ Int-area mailing list Int-area@ietf.org https://www.ietf.org/mailman/listinfo/int-area
