Hi, On Sun, Feb 25, 2024 at 03:53:23PM +0800, 上勾拳 wrote: > Hello, > > I hope this email finds you well. I am writing to inquire about the status > of an issue I have encountered regarding the handling of QUIC connections > when Nginx is reloaded. > > Recently, I delved into the Nginx eBPF implementation, specifically > examining how QUIC connection packets are handled, especially during Nginx > reloads. My focus was on ensuring that existing QUIC connection packets are > correctly routed to the appropriate worker after a reload, and the Nginx > eBPF prog have done this job perfectly. > > However, I observed that during a reload, new QUIC connections might be > directed to the old worker. The underlying problem stems from the fact that > new QUIC connections fail to match the eBPF reuseport socket map. The > kernel default logic then routes UDP packets based on the hash UDP 4-tuple > in the reuseport group socket array. Since the old worker's listen FDs > persist in the reuseport group socket array (reuse->socks), there is a > possibility that the old worker may still be tasked with handling new QUIC > connections. > > Given that the old worker should not process new requests, it results in > the old worker not responding to the QUIC new connection. Consequently, > clients have to endure the handshake timeout and retry the connection, > potentially encountering the old worker again, leading to an ineffective > cycle. > > In my investigation, I came across a thread in the nginx-devel mailing list > [https://www.mail-archive.com/nginx-devel@nginx.org/msg10627.html], where > it was mentioned that there would be some work addressing this issue. > > Considering my limited experience with eBPF, I propose a potential > solution. The eBPF program could maintain another eBPF map containing only > the listen sockets of the new worker. When the eBPF program calls > `bpf_sk_select_reuseport` and receives `-ENOENT`, it could utilize this new > eBPF map with the hash UDP 4-tuple to route the new QUIC connection to the > new worker. This approach aims to circumvent the kernel logic routing the > packet to the shutting down worker since removing the old worker's listen > socket from the reuseport group socket array seems unfeasible. Not sure > about whether this solution is a good idea and I also wonder if there are > other solutions for this. I would appreciate any insights or updates you > could provide on this matter. Thank you for your time and consideration.
It is true QUIC in nginx does not handle reloads well. This is a known issue and we are working on improving it. I made an effort a while back to address QUIC reloads in nginx: https://mailman.nginx.org/pipermail/nginx-devel/2023-January/thread.html#16073 Here patch #3 implements ePBF-based solution and patch #4 implements client sockets-based solution. The client sockets require extra worker process privileges to bind to listen port, which is a serious problem for a typical nginx configuration. The ePBF solution does not seem to have any problems, but we still need more feedback to proceed with this. If you apply all 4 patches, make sure you disable client sockets using --without-quic_client_sockets. Otherwise just apply the first 3 patches. Here's a relevant trac ticket: https://trac.nginx.org/nginx/ticket/2528 -- Roman Arutyunyan _______________________________________________ nginx-devel mailing list nginx-devel@nginx.org https://mailman.nginx.org/mailman/listinfo/nginx-devel
