Hello, I hope this email finds you well. I am writing to inquire about the status of an issue I have encountered regarding the handling of QUIC connections when Nginx is reloaded.
Recently, I delved into the Nginx eBPF implementation, specifically examining how QUIC connection packets are handled, especially during Nginx reloads. My focus was on ensuring that existing QUIC connection packets are correctly routed to the appropriate worker after a reload, and the Nginx eBPF prog have done this job perfectly. However, I observed that during a reload, new QUIC connections might be directed to the old worker. The underlying problem stems from the fact that new QUIC connections fail to match the eBPF reuseport socket map. The kernel default logic then routes UDP packets based on the hash UDP 4-tuple in the reuseport group socket array. Since the old worker's listen FDs persist in the reuseport group socket array (reuse->socks), there is a possibility that the old worker may still be tasked with handling new QUIC connections. Given that the old worker should not process new requests, it results in the old worker not responding to the QUIC new connection. Consequently, clients have to endure the handshake timeout and retry the connection, potentially encountering the old worker again, leading to an ineffective cycle. In my investigation, I came across a thread in the nginx-devel mailing list [https://www.mail-archive.com/nginx-devel@nginx.org/msg10627.html], where it was mentioned that there would be some work addressing this issue. Considering my limited experience with eBPF, I propose a potential solution. The eBPF program could maintain another eBPF map containing only the listen sockets of the new worker. When the eBPF program calls `bpf_sk_select_reuseport` and receives `-ENOENT`, it could utilize this new eBPF map with the hash UDP 4-tuple to route the new QUIC connection to the new worker. This approach aims to circumvent the kernel logic routing the packet to the shutting down worker since removing the old worker's listen socket from the reuseport group socket array seems unfeasible. Not sure about whether this solution is a good idea and I also wonder if there are other solutions for this. I would appreciate any insights or updates you could provide on this matter. Thank you for your time and consideration. Best regards, Zhenzhong
_______________________________________________ nginx-devel mailing list nginx-devel@nginx.org https://mailman.nginx.org/mailman/listinfo/nginx-devel
