On 10/13, Willem Ferguson wrote: > This series lets a listener encode a small per-connection identifier in > the low bits of the TCP timestamp values it sends. The peer echoes TSval > back in TSecr, so every segment the peer sends after the handshake > carries the identifier in its TSecr. > > In Anycast networks, mid-stream path changes caused by BGP route flapping > or ECMP shifts frequently deliver active TCP packets to an ingress Point > of Presence (PoP) that lacks the flow's socket state, resulting in abrupt > connection resets (RST). While protocols like QUIC avoid this using explicit > Connection IDs, TCP lacks an in-band routing token to help ingress > nodes identify where a flow actually lives. To address this problem, > we propose embedding an N-bit data center ID into the lower-order bits > of the TCP Timestamp option (TSval, RFC 7323), allowing edge load balancers > to recognize misdirected packets and steer them to the correct origin data > center holding the active session. > > How it works: > > - tcp_ts_id_bits (per netns, 0..8, default 0) sets how many low TSval > bits are reserved. The millisecond clock is shifted up by that many > bits. > - A sock_ops program opts a listener in with > BPF_SOCK_OPS_TCP_TS_ID_CB_FLAG and is called with > BPF_SOCK_OPS_TCP_TS_ID_CB for each SYN, before the SYN-ACK is sent. > It sees the SYN and optionally returns the identifier > - The identifier and width are stored in the request socket, then in > the full and timewait sockets. Incoming TSecr values are decoded > before RTT estimation and PAWS use them. > - TCP_TS_ID reads back the identifier, and /proc/net/netstat gains > TCPTSIdAssigned, TCPTSIdDeclined, TCPTSIdSkipped and > TCPTSIdEchoMismatch. > > RTT estimation on the encoding host sees the same decoded timestamps > as before, so they are unchanged. On the peer, PAWS keeps working too; > the only effects are a shorter idle limit and a narrower tolerance for > reordered segments, both described below. > > Scope and limitations: > > - Passive opens only. SYN cookies, microsecond timestamps and peers > without timestamps are left unencoded. > - The peer sees a clock running 2^bits times faster. That narrows the > real-time span of its PAWS window, so the width should be no larger > than the identifiers need. > - Peers that keep ts_recent for the RFC 7323 24 days (e.g. FreeBSD, > macOS) stall connections that stay idle longer than 24.8/2^bits days > (about 37 hours at 4 bits). Linux peers stop enforcing PAWS after > TCP_PAWS_WRAP and are unaffected. > - TCP_REPAIR cannot restore the identifier, so hosts that migrate > established connections should leave the feature off. > > All of this is documented under tcp_ts_id_bits in ip-sysctl.rst. With > the sysctl at 0 (the default) the only new cost on the established fast > path is one likely() branch when parsing TSecr. > > Questions for reviewers: > > 1. Is encoding an identifier in the low bits of TSval acceptable in > principle, given the trade-offs above? > 2. If so, what feedback do you have on the implementation?
Don't see any feedback yet, so from the top of my head: looks like you control two sides of the communication, so why not a custom (bpf) tcp option? Would you need that on every packet and it makes it expensive? Or any other field you can abuse maybe? IPv6 Flow label? Implementation-wise, the sock_ops is going away in favor of bpf_tcp_ops, so BPF_SOCK_OPS_TCP_TS_ID_CB_FLAG is a no go.
