Re: Lossy cogent p2p experiences?
Masataka Ohta wrote on 03/09/2023 14:32: See, for example, the famous paper of "Sizing Router Buffers". With thousands of TCP connections at the backbone recognized by the paper, buffers with thousands of packets won't cause packet reordering. What you said reminds me of the old saying: in theory, there's no difference between theory and practice, but in practice there is. In theory, you can always fabricate unrealistic counter examples against theories by ignoring essential assumptions of the theories. In this case, "Without buffer bloat" is an essential assumption. I can see how this conclusion could potentially be reached in specific styles of lab configs, but the real world is more complicated and the assumptions you've made don't hold there, especially the implicit ones. Buffer bloat will make this problem worse, but small buffers won't eliminate the problem. That isn't to say that packet / cell spray arrangements can't work. There are some situations where they can work reasonably well, given specific constraints, e.g. limited distance transmission path and path congruence with far-side reassembly (!), but these are the exception. Usually this only happens inside network devices rather than between devices, but occasionally you see products on the market which support this between devices with varying degrees of success. Generally in real world situations on the internet, packet reordering will happen if you use round robin, and this will impact performance for higher speed flows. There are several reasons for this, but mostly they boil down to a lack of control over the exact profile of the packets that the devices are expected to transmit, and no guarantee that the individual bearer channels have identical transmission characteristics. Then multiply that across the N load-balanced hops that each flow will take between source and destination. It's true that per-hash load balancing is a nuisance, but it works better in practice on larger heterogeneous networks than RR. Nick
Re: Lossy cogent p2p experiences?
On Thu, Aug 31, 2023 at 2:42 PM David Hubbard wrote: > any new TCP flow is subject to numerous dropped packets at establishment and > then ongoing loss every five to ten seconds. Hi David, That sounds like normal TCP behavior over a long fat pipe. After establishment, TCP sends a burst of 10 packets at wire speed. There's a long delay and then they basically get acked all at once so it sends another burst of 20 packets this time. This doubling burst repeats itself until one of the bursts overwhelms the buffers of a mid-path device, causing one or a bunch of them to be lost. That kicks it out of "slow start" so that it stops trying to double the window size every time. Depending on how aggressive your congestion control algorithm is, it then slightly increases the window size until it loses packets, and then falls back to a smaller size. It actually takes quite a while for the packets to spread out over the whole round trip time. They like to stay bunched up in bursts. If those bursts align with other users' traffic and overwhelm a midpoint buffer again, well, there you go. I have a hypothesis that TCP performance could be improved by intentionally spreading out the early packets. Essentially, upon receiving an ack to the first packet that contained data, start a rate limiter that allows only one packet per 1/20th of the round trip time to be sent for the next 20 packets. I left the job where I was looking at that and haven't been back to it. Regards, Bill Herrin -- William Herrin b...@herrin.us https://bill.herrin.us/
Re: Lossy cogent p2p experiences?
On 9/3/23 15:01, Masataka Ohta wrote: Why, do you think, you can rely on existence of flows? You have not quite answered my question - but I will assume you are in favour of per-packet load balancing. I have deployed per-packet load balancing before, ironically, trying to deal with large EoMPLS flows in a LAG more than a decade ago. I won't be doing that again... OoO packets is nasty at scale. And nothing beyond, of course. No serious operator polices in the core. ECMP, surely, is a too abstract concept to properly manage/operate simple situations with equal speed multi parallel point to point links. I must have been doing something wrong for the last 25 years. Mark.
Re: Lossy cogent p2p experiences?
On 9/3/23 15:01, Masataka Ohta wrote: Why, do you think, you can rely on existence of flows? You have not quite answered my question - but I will assume you are in favour of per-packet load balancing. I have deployed per-packet load balancing before, ironically, trying to deal with large EoMPLS flows in a LAG more than a decade ago. I won't be doing that again... OoO packets is nasty at scale. And nothing beyond, of course. No serious operators polices in the core. ECMP, surely, is a too abstract concept to properly manage/operate simple situations with equal speed multi parallel point to point links. I must have been doing something wrong for the last 25 years. Mark.
Re: Lossy cogent p2p experiences?
Nick Hilliard wrote: the proper thing to do is to use the links with round robin fashion without hashing. Without buffer bloat, packet reordering probability within each TCP connection is negligible. Can you provide some real world data to back this position up? See, for example, the famous paper of "Sizing Router Buffers". With thousands of TCP connections at the backbone recognized by the paper, buffers with thousands of packets won't cause packet reordering. What you said reminds me of the old saying: in theory, there's no difference between theory and practice, but in practice there is. In theory, you can always fabricate unrealistic counter examples against theories by ignoring essential assumptions of the theories. In this case, "Without buffer bloat" is an essential assumption. Masataka Ohta
Re: Lossy cogent p2p experiences?
Mark Tinka wrote: So you mean, what... per-packet load balancing, in lieu of per-flow load balancing? Why, do you think, you can rely on existence of flows? So, if you internally have 10 parallel 1G circuits expecting perfect hashing over them, it is not "non-rate-limited 10gig". It is understood in the operator space that "rate limiting" generally refers to policing at the edge/access. And nothing beyond, of course. The core is always abstracted, and that is just capacity planning and management by the operator. ECMP, surely, is a too abstract concept to properly manage/operate simple situations with equal speed multi parallel point to point links. Masataka Ohta
Re: Lossy cogent p2p experiences?
Masataka Ohta wrote on 03/09/2023 08:59: the proper thing to do is to use the links with round robin fashion without hashing. Without buffer bloat, packet reordering probability within each TCP connection is negligible. Can you provide some real world data to back this position up? What you said reminds me of the old saying: in theory, there's no difference between theory and practice, but in practice there is. Nick
Re: it's mailman time again
You didn't lose your /. account because of a mailing list config.You lost it due to the bad practices or knowledge at the time.\o/-- J. HellenthalThe fact that there's a highway to Hell but only a stairway to Heaven says a lot about anticipated traffic volume.On Sep 2, 2023, at 17:08, Richard Porter wrote: Pouring kerosine on fire? *flame me back if warranted* Voice networks have no POTS left in them? *mostly?* …. Get Outlook for iOS From: NANOG on behalf of Randy Bush Sent: Saturday, September 2, 2023 4:30:07 PM To: Jim Popovitch via NANOG Subject: Re: it's mailman time again > Mail in transit is mostly TLS transport these days, yep. mostly. opsec folk are not fond of 'mostly.' > BUT mail in storage and idle state isn't always secured. I'm sure > that most any of us could find a public s3 bucket with an mbox file on > it if we cared to look. sigh randy
Re: Lossy cogent p2p experiences?
On 9/3/23 09:59, Masataka Ohta wrote: If you have multiple parallel links over which many slow TCP connections are running, which should be your assumption, the proper thing to do is to use the links with round robin fashion without hashing. Without buffer bloat, packet reordering probability within each TCP connection is negligible. So you mean, what... per-packet load balancing, in lieu of per-flow load balancing? So, if you internally have 10 parallel 1G circuits expecting perfect hashing over them, it is not "non-rate-limited 10gig". It is understood in the operator space that "rate limiting" generally refers to policing at the edge/access. The core is always abstracted, and that is just capacity planning and management by the operator. Mark.
Re: Lossy cogent p2p experiences?
Mark Tinka wrote: Wrong. It can be performed only at the edges by policing total incoming traffic without detecting flows. I am not talking about policing in the core, I am talking about detection in the core. I'm not talking about detection at all. Policing at the edge is pretty standard. You can police a 50Gbps EoMPLS flow coming in from a customer port in the edge. If you've got N x 10Gbps links in the core and the core is unable to detect that flow in depth to hash it across all those 10Gbps links, you can end up putting all or a good chunk of that 50Gbps of EoMPLS traffic into a single 10Gbps link in the core, despite all other 10Gbps links having ample capacity available. Relying on hash is a poor way to offer wide bandwidth. If you have multiple parallel links over which many slow TCP connections are running, which should be your assumption, the proper thing to do is to use the links with round robin fashion without hashing. Without buffer bloat, packet reordering probability within each TCP connection is negligible. Faster TCP may suffer from packet reordering during slight congestion, but the effect is like that of RED. Anyway, in this case, the situation is: :Moreover, as David Hubbard wrote: :> I've got a non-rate-limited 10gig circuit So, if you internally have 10 parallel 1G circuits expecting perfect hashing over them, it is not "non-rate-limited 10gig". Masataka Ohta
