Even if it is unicasted, performance is destroyed due to reordering. All modern TCP stacks use cubic for congestion control, which considers reorder a packet loss.
However technically speaking QoS is of least concern, because if you change QoS you expect reordering. Of course this excludes ECN bits, only 6 most significant can be considered part of hash legitimately. On Tue, 2 Dec 2025 at 21:44, Chris Woodfield via NANOG <[email protected]> wrote: > > These types of issues are endemic to TCP anycast, unfortunately. If your > services depend on anycast, you really want to make sure that your hashing > algo network-wide uses the TCP/IP five-tuple, and the five-tuple *only*; I’m > assuming that’s what changing to the the “original” load-sharing CEF > algorithm accomplished in your case. > > My personal war story involved connections to a database server that would > change DSCP values between the TCP handshake and subsequent packets, > resulting in RSVP-TE putting the packets on different paths... to a PE router > that incorporated the inbound destination MAC (read: inbound interface) into > the hash algo, with no option to disable. Different inbound interfaces = > different dest MAC = different anycast destinations = immediate SEV1 impact > as all our database connections broke at once. > > The solution*? Reconfigure each L3 interface on the router to the same MAC > address. :P > > -Chris > > * An alternate solution involved screaming into the vendor support void, > which eventually did get results... a year or so later. > ** As I’m typing this, I’ve realized that an iptables rule on the server to > undo the DSCP change would have fixed; putting that thought onto the shelf if > needed later. > > > On Dec 2, 2025, at 09:12, Tim Durack via NANOG <[email protected]> > > wrote: > > > > In case it is useful for anyone else, underlying issue looks to be this: > > > > Cisco CSCws27022: ECN bits being included as part of ECMP hash on IPv6 TCP > > flows (Workaround: Do not use ECMP) > > > > Appears to be platform specific, affecting Cisco Catalyst C9K UADP ASIC > > (C9500-32C) > > > > Another work-around might be to configure "ip cef load-sharing algorithm > > original" > > > > Tim:> > > > > On Tue, Mar 25, 2025 at 4:33 PM Tim Durack <[email protected]> wrote: > > > >> Very helpful, thanks! Will post my own short story once complete... > >> > >> On Tue, Mar 25, 2025 at 4:24 PM Toke Høiland-Jørgensen <[email protected]> > >> wrote: > >> > >>> Tim Durack <[email protected]> writes: > >>> > >>>> Toke, > >>>> > >>>> Resurrecting an old thread, did you ever write this one up? > >>> > >>> Hi Tim > >>> > >>> Thank you for the reminder! No, I never did get around to writing > >>> anything at the time. However, now that you reminded me, I collected my > >>> old notes and posted this: > >>> > >>> > >>> https://blog.tohojo.dk/2025/03/ecn-ecmp-and-anycast-a-cocktail-of-broken-connections.html > >>> > >>>> I believe I have a customer reporting a similar problem with IPv6 TCP > >>> ECN > >>>> probably ECMP resulting in RST coming back from anycast services > >>>> (Cloudflare). > >>>> > >>>> Tricky one to debug, looking for similar reports... > >>> > >>> Hoping the above is helpful :) > >>> > >>> -Toke > >>> > >> > >> > >> -- > >> Tim:> > >> > > > > > > -- > > Tim:> > > > > > > -- > > Tim:> > > _______________________________________________ > > NANOG mailing list > > https://lists.nanog.org/archives/list/[email protected]/message/KSVJBYJYTIEXCHF66JBWR3WBLJT7QX5J/ > > _______________________________________________ > NANOG mailing list > https://lists.nanog.org/archives/list/[email protected]/message/UW4PU2VVAJVUPAMHKVDM4ON5TKMIWZ2W/ -- ++ytti _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/[email protected]/message/3X6YVG3INX5H6W3ZNVXMYCS7KDZX3FJB/
