Re: [Int-area] IP parcels
On Thu, Jan 27, 2022 at 3:43 PM to...@strayalpha.com wrote: > > Hi, Tom, > > > On Jan 27, 2022, at 2:46 PM, Tom Herbert wrote: > > > > On Thu, Jan 27, 2022 at 2:17 PM to...@strayalpha.com > > wrote: > >> > >> FWIW, GRO/GSO give no end of headaches to the idea of new TCP options, > >> esp. the current ones to extend option space after the SYN > >> (draft-ietf-tcpm-tcp-edo). > > > > GRO and GSO are software implementations and in most deployments > > Ubiquitously deployed Linux kernel software at one point had a bug that > failed to lock the inode structure during modification. Uniquity didn’t make > that magically correct. > > >> Although I appreciate their zeal for optimization, implementers of GRO/GSO > >> still need to play by the rules of TCP and UDP. It’s not clear they are > >> (e.g., coalescing packets with different unknown options), and when they > >> don’t, I want to be clear that it is they that are the problem. > >> > > Joe, > > > > GRO and GSO are software implementations in kernel networking stacks > > and in most cases these are open source projects of Linux or FreeBSD. > > If they have flaws or there's areas for improvement, then by all means > > submit patches to the respective project-- that, after all, is the > > whole premise of an open source project. > > That’s not how open source works. > > The onus is on those who currently maintain the code to ensure it complies > with current protocols. I noted that I and others have experience that it > doesn’t. > > It is not the responsibility of the user community to fix their bugs or > ensure that their approach remains viable. > Actually it is. We have users fixing bugs all the time on Linux. There's no entry fee or membership, and anyone is free to submit patches. It is similar to IETF where anyone can submit I-Ds. Of course in both cases, acceptance often comes down to how much effort and perseverance the submitter is willing to put into it. Tom > > The hardware cognates, TSO > > and LRO, do tend to be more closed and for this reason they haven't > > gotten much traction-- TSO has seen a some amount of deployment, but > > LRO hasn't because of the problems of putting fairly complex > > algorithms in hardware. That problem is addressed once we have > > sufficiently programmable hardware so the stack can program things > > like GSO and GRO in hardware as easily in hardware and of course this > > should be done in ubiquitous open source that works across all > > hardware vendors. > > None of that has anything to do with the issue I raised. > > Both hardware and software implementations of these optimizations MUST > strictly comply with protocol specs. When they encounter options they don’t > know, it’s not their prerogative to do anything beyond “disable” for that > stream. That’s not our experience, though. > > Joe > ___ Int-area mailing list Int-area@ietf.org https://www.ietf.org/mailman/listinfo/int-area
Re: [Int-area] IP parcels
Ease up, Joe - just ease up. > -Original Message- > From: Int-area [mailto:int-area-boun...@ietf.org] On Behalf Of > to...@strayalpha.com > Sent: Thursday, January 27, 2022 3:44 PM > To: Tom Herbert > Cc: int-area@ietf.org > Subject: [EXTERNAL] Re: [Int-area] IP parcels > > EXT email: be mindful of links/attachments. > > > > Hi, Tom, > > > On Jan 27, 2022, at 2:46 PM, Tom Herbert wrote: > > > > On Thu, Jan 27, 2022 at 2:17 PM to...@strayalpha.com > > wrote: > >> > >> FWIW, GRO/GSO give no end of headaches to the idea of new TCP options, > >> esp. the current ones to extend option space after the SYN > (draft-ietf-tcpm-tcp-edo). > > > > GRO and GSO are software implementations and in most deployments > > Ubiquitously deployed Linux kernel software at one point had a bug that > failed to lock the inode structure during modification. Uniquity > didn’t make that magically correct. > > >> Although I appreciate their zeal for optimization, implementers of GRO/GSO > >> still need to play by the rules of TCP and UDP. It’s not clear > they are (e.g., coalescing packets with different unknown options), and when > they don’t, I want to be clear that it is they that are the > problem. > >> > > Joe, > > > > GRO and GSO are software implementations in kernel networking stacks > > and in most cases these are open source projects of Linux or FreeBSD. > > If they have flaws or there's areas for improvement, then by all means > > submit patches to the respective project-- that, after all, is the > > whole premise of an open source project. > > That’s not how open source works. > > The onus is on those who currently maintain the code to ensure it complies > with current protocols. I noted that I and others have > experience that it doesn’t. > > It is not the responsibility of the user community to fix their bugs or > ensure that their approach remains viable. > > > The hardware cognates, TSO > > and LRO, do tend to be more closed and for this reason they haven't > > gotten much traction-- TSO has seen a some amount of deployment, but > > LRO hasn't because of the problems of putting fairly complex > > algorithms in hardware. That problem is addressed once we have > > sufficiently programmable hardware so the stack can program things > > like GSO and GRO in hardware as easily in hardware and of course this > > should be done in ubiquitous open source that works across all > > hardware vendors. > > None of that has anything to do with the issue I raised. > > Both hardware and software implementations of these optimizations MUST > strictly comply with protocol specs. When they encounter > options they don’t know, it’s not their prerogative to do anything beyond > “disable” for that stream. That’s not our experience, though. > > Joe > > ___ > Int-area mailing list > Int-area@ietf.org > https://www.ietf.org/mailman/listinfo/int-area ___ Int-area mailing list Int-area@ietf.org https://www.ietf.org/mailman/listinfo/int-area
Re: [Int-area] IP parcels
Hi, Tom, > On Jan 27, 2022, at 2:46 PM, Tom Herbert wrote: > > On Thu, Jan 27, 2022 at 2:17 PM to...@strayalpha.com > wrote: >> >> FWIW, GRO/GSO give no end of headaches to the idea of new TCP options, esp. >> the current ones to extend option space after the SYN >> (draft-ietf-tcpm-tcp-edo). > > GRO and GSO are software implementations and in most deployments Ubiquitously deployed Linux kernel software at one point had a bug that failed to lock the inode structure during modification. Uniquity didn’t make that magically correct. >> Although I appreciate their zeal for optimization, implementers of GRO/GSO >> still need to play by the rules of TCP and UDP. It’s not clear they are >> (e.g., coalescing packets with different unknown options), and when they >> don’t, I want to be clear that it is they that are the problem. >> > Joe, > > GRO and GSO are software implementations in kernel networking stacks > and in most cases these are open source projects of Linux or FreeBSD. > If they have flaws or there's areas for improvement, then by all means > submit patches to the respective project-- that, after all, is the > whole premise of an open source project. That’s not how open source works. The onus is on those who currently maintain the code to ensure it complies with current protocols. I noted that I and others have experience that it doesn’t. It is not the responsibility of the user community to fix their bugs or ensure that their approach remains viable. > The hardware cognates, TSO > and LRO, do tend to be more closed and for this reason they haven't > gotten much traction-- TSO has seen a some amount of deployment, but > LRO hasn't because of the problems of putting fairly complex > algorithms in hardware. That problem is addressed once we have > sufficiently programmable hardware so the stack can program things > like GSO and GRO in hardware as easily in hardware and of course this > should be done in ubiquitous open source that works across all > hardware vendors. None of that has anything to do with the issue I raised. Both hardware and software implementations of these optimizations MUST strictly comply with protocol specs. When they encounter options they don’t know, it’s not their prerogative to do anything beyond “disable” for that stream. That’s not our experience, though. Joe ___ Int-area mailing list Int-area@ietf.org https://www.ietf.org/mailman/listinfo/int-area
Re: [Int-area] IP parcels
You know, this all stems back to the "qe reset" bug - do you all remember that? Tom might, as a former Sun guy - Bob Hinden maybe too. Or maybe that was even before your time? I think a guy named Rusty Young helped us diagnose it. Anyone remember him? ___ Int-area mailing list Int-area@ietf.org https://www.ietf.org/mailman/listinfo/int-area
Re: [Int-area] IP parcels
On Thu, Jan 27, 2022 at 2:17 PM to...@strayalpha.com wrote: > > FWIW, GRO/GSO give no end of headaches to the idea of new TCP options, esp. > the current ones to extend option space after the SYN > (draft-ietf-tcpm-tcp-edo). GRO and GSO are software implementations and in most deployments > > Although I appreciate their zeal for optimization, implementers of GRO/GSO > still need to play by the rules of TCP and UDP. It’s not clear they are > (e.g., coalescing packets with different unknown options), and when they > don’t, I want to be clear that it is they that are the problem. > Joe, GRO and GSO are software implementations in kernel networking stacks and in most cases these are open source projects of Linux or FreeBSD. If they have flaws or there's areas for improvement, then by all means submit patches to the respective project-- that, after all, is the whole premise of an open source project. The hardware cognates, TSO and LRO, do tend to be more closed and for this reason they haven't gotten much traction-- TSO has seen a some amount of deployment, but LRO hasn't because of the problems of putting fairly complex algorithms in hardware. That problem is addressed once we have sufficiently programmable hardware so the stack can program things like GSO and GRO in hardware as easily in hardware and of course this should be done in ubiquitous open source that works across all hardware vendors. Tom > I agree that the common Unix socket interface has performance flaws, but > perhaps it’s that interface that needs to evolve. > > Joe > > — > Joe Touch, temporal epistemologist > www.strayalpha.com > > On Jan 27, 2022, at 1:29 PM, Behcet Sarikaya wrote: > > Hi Folks, > > Thanks Christian for explaining how GSO/GRO are used by Quic implementations. > So the use is not mandated in Quic RFCs but rather used in implementations. > > > ___ > Int-area mailing list > Int-area@ietf.org > https://www.ietf.org/mailman/listinfo/int-area ___ Int-area mailing list Int-area@ietf.org https://www.ietf.org/mailman/listinfo/int-area
Re: [Int-area] IP parcels
FWIW, GRO/GSO give no end of headaches to the idea of new TCP options, esp. the current ones to extend option space after the SYN (draft-ietf-tcpm-tcp-edo). Although I appreciate their zeal for optimization, implementers of GRO/GSO still need to play by the rules of TCP and UDP. It’s not clear they are (e.g., coalescing packets with different unknown options), and when they don’t, I want to be clear that it is they that are the problem. I agree that the common Unix socket interface has performance flaws, but perhaps it’s that interface that needs to evolve. Joe — Joe Touch, temporal epistemologist www.strayalpha.com > On Jan 27, 2022, at 1:29 PM, Behcet Sarikaya wrote: > > Hi Folks, > > Thanks Christian for explaining how GSO/GRO are used by Quic implementations. > So the use is not mandated in Quic RFCs but rather used in implementations. ___ Int-area mailing list Int-area@ietf.org https://www.ietf.org/mailman/listinfo/int-area
Re: [Int-area] IP parcels
Hi Folks, Thanks Christian for explaining how GSO/GRO are used by Quic implementations. So the use is not mandated in Quic RFCs but rather used in implementations. I found this presentation by Intel: https://www.dpdk.org/wp-content/uploads/sites/35/2018/06/GRO-GSO-Libraries-Bring-Significant-Performance-Gains-to-DPDK-based-Applications.pdf way back in 2017 quite useful. So what I understand is GSO/GRO were originally used by TCP/UDP and Quic implementations use UDP GSO/GRO because of Quic is UDP based. Behcet On Thu, Jan 27, 2022 at 1:24 PM Christian Huitema wrote: > On 12/20/2021 10:03 AM, Templin (US), Fred L wrote: > > > Tom, sorry I will try to use my words more carefully; I am using GSO/GRO > also for > > a UDP-based transport protocol – not QUIC but something similar. I like > GSO/GRO > > very much; I am glad the service is available and I want to see it > continue. But, my > > understanding of the services is that they leverage the IP ID field in > whole IPv4 > > packets that are not eligible for fragmentation and those are > limitations I am > > seeking to improve on. > > Lots of QUIC implementation use UDP GSO/GRO. They do that to achieve > good performance. Without GSO, we have observed that almost 2/3rd of the > CPU time was spent in socket calls, mostly sendmsg on servers. Using GSO > to send batches of packets results in big gains. For example, the same > single-core implementation that achieves 500 Mbps without GSO gets to > 4-5 Gbps while sending batches of packets of about 64K using GSO. > > Note however that this is *UDP* GSO. The application prepares a batch of > UDP packets, all sharing the same source and destination address and > ports, and all but the last having the same length. The GSO call > provides these header parameters, the common payload length, and a set > of UDP payloads, concatenated in a large buffer. Each payload is a fully > formed QUIC packet, encrypted and sealed using AEAD. There is no > expectation that these packets would be somehow re-segmented or > re-assembled in the network -- if the network did that, decryption would > fail. There is also no expectation that the batch of packets will be > delivered as a single batch to the receiver. It might happen by chance, > but mostly it does not because GRO and GSO are not synchronized. On the > other hand, the handling of the small packets by the QUIC stack is > rarely considered a problem. > > > I want to enable a facility similar to GSO/GRO that works for both IPv4 > and IPv6 > > packets and allows for lower layers to fragment if necessary. And, I > want to use > > a well-behaved 32-bit IPv6 ID instead of the 16-bit IPv4 one where the > use is not > > well defined when DF=1. > > Consider encryption. QUIC certainly expects that packets sent will be > received unchanged, or they would fail decryption. Also consider packet > loss. In the GSO example, the sender might send a batch of 42 packets. > If any of them is lost, it will be resent individually. QUIC could of > course send a 64KB packet if the network allowed. But if a 64KB parcel > is segmented in 42 segments, any packet loss will cause the loss of the > whole parcel, and that would cause a significant drop in performance > compared to GSO. Which is indeed the core of the "segmentation > considered harmful" argument. > > -- Christian Huitema > > ___ > Int-area mailing list > Int-area@ietf.org > https://www.ietf.org/mailman/listinfo/int-area > ___ Int-area mailing list Int-area@ietf.org https://www.ietf.org/mailman/listinfo/int-area
Re: [Int-area] IP parcels
On 12/20/2021 10:03 AM, Templin (US), Fred L wrote: Tom, sorry I will try to use my words more carefully; I am using GSO/GRO also for a UDP-based transport protocol – not QUIC but something similar. I like GSO/GRO very much; I am glad the service is available and I want to see it continue. But, my understanding of the services is that they leverage the IP ID field in whole IPv4 packets that are not eligible for fragmentation and those are limitations I am seeking to improve on. Lots of QUIC implementation use UDP GSO/GRO. They do that to achieve good performance. Without GSO, we have observed that almost 2/3rd of the CPU time was spent in socket calls, mostly sendmsg on servers. Using GSO to send batches of packets results in big gains. For example, the same single-core implementation that achieves 500 Mbps without GSO gets to 4-5 Gbps while sending batches of packets of about 64K using GSO. Note however that this is *UDP* GSO. The application prepares a batch of UDP packets, all sharing the same source and destination address and ports, and all but the last having the same length. The GSO call provides these header parameters, the common payload length, and a set of UDP payloads, concatenated in a large buffer. Each payload is a fully formed QUIC packet, encrypted and sealed using AEAD. There is no expectation that these packets would be somehow re-segmented or re-assembled in the network -- if the network did that, decryption would fail. There is also no expectation that the batch of packets will be delivered as a single batch to the receiver. It might happen by chance, but mostly it does not because GRO and GSO are not synchronized. On the other hand, the handling of the small packets by the QUIC stack is rarely considered a problem. I want to enable a facility similar to GSO/GRO that works for both IPv4 and IPv6 packets and allows for lower layers to fragment if necessary. And, I want to use a well-behaved 32-bit IPv6 ID instead of the 16-bit IPv4 one where the use is not well defined when DF=1. Consider encryption. QUIC certainly expects that packets sent will be received unchanged, or they would fail decryption. Also consider packet loss. In the GSO example, the sender might send a batch of 42 packets. If any of them is lost, it will be resent individually. QUIC could of course send a 64KB packet if the network allowed. But if a 64KB parcel is segmented in 42 segments, any packet loss will cause the loss of the whole parcel, and that would cause a significant drop in performance compared to GSO. Which is indeed the core of the "segmentation considered harmful" argument. -- Christian Huitema ___ Int-area mailing list Int-area@ietf.org https://www.ietf.org/mailman/listinfo/int-area
Re: [Int-area] [arch-d] Continuing the addressing discussion: what is an address anyway?
Hi Geoff, Dirk, I wanted to add a couple of comments on your excellent points, see inline. (trimmed a little bit of the text) Ciao L. > On 27 Jan 2022, at 08:51, Dirk Trossen > wrote: > [snip] > > There is also a second factor, that of sunk cost. Nobody wants to pay to > upgrade existing infrastructure. Nobody. So those who want to change things > build around, over and tunnel through existing infrastructure. In a > deregulated world where piecemeal uncoordinated actions predominate, the > level of coordination and orchestration required to uplift common shared > infrastructure is simply impossible. We say to ourselves that we outgrew flag > days on the Internet many decades ago, and thats true, but at times we appear > not to understand precisely what that implies about today. We have built an > application-based superstructure of encapsulated tunnels in the network > (QUIC) that neatly circumvents the entire question of infrastructure renewal. > Whereas IPv6 has been head-butting against the sunk cost of existing > infrastructure for more than two decades then dramatic rise of QUIC, BBR, > SVCB and HTTPS, and similar application-level technologies attests to the > application world’s extreme distaste to engage with the existing > infrastructure. > This last point is tackled in the current set of documents and also touched in the previous thread (but can certainly be improved upon this thread). I think that we are at the stage that _no_ there is no strong reason yet to upgrade the existing infrastructure (we do not even know to what to upgrade). However, the innovation is there and is happening, raising the question: - Up to what point we can “build on top” without risking to jeopardize the whole? In the long run, up to what point we can continue the way we are solving problems today? Seeing this as a possible risk, I would prefer to discuss now about what can be done (including the “is it worthwhile?” question). Up to us to answer this tough question and engineer an improved architecture that provide sufficient benefits to be adopted in less time that IPv6 adoption. [snip] > > Then, as now, the issue was NOT “What’s the answer?” but “Whats the right > question to ask?” - at least for me. Isn’t what we are trying to do here? ;-) I think that the discussions happening in the INTArea mailing list are very fruitful. We just need to capture the current understanding in the documents, trying to formalize the problem we face. As other already mentioned there is an architectural discussion to be made (and happy that the architecture discuss mailing list is in this thread), but we need to start somewhere and addressing is a good starting point. What we need is to gather the experience each and everyone of us can provide in order to shape something meaningful to move forward….. starting from the addresses ;-) ___ Int-area mailing list Int-area@ietf.org https://www.ietf.org/mailman/listinfo/int-area
