Hi Roman, Thanks a bunch for your quick response, it really made a difference for me.
I went through Patch 3, and it's pretty cool! And about Patch 4, which also addresses the reload route issue, I would like to share an experience from utilizing this solution in a production environment. Unfortunately, I encountered a significant challenge that surpasses the race condition between bind and connect. Specifically, this solution led to a notable performance degradation in the kernel's UDP packet lookup under high concurrency scenarios. The excessive number of client sockets caused the UDP hash table lookup performance degrade into a linked list, because the udp hashtable is based on target ip and target port hash. To address this lookup performance issue, we implemented a proprietary kernel patch that introduces a 4-tuple hash table for UDP socket lookup. Although effective, it appears that the eBPF solution is more versatile and universal. Big thanks again for your attention! Best Regards, Zhenzhong <nginx-devel-requ...@nginx.org> 于2024年2月26日周一 20:00写道： > Send nginx-devel mailing list submissions to > nginx-devel@nginx.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mailman.nginx.org/mailman/listinfo/nginx-devel > or, via email, send a message with subject or body 'help' to > nginx-devel-requ...@nginx.org > > You can reach the person managing the list at > nginx-devel-ow...@nginx.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of nginx-devel digest..." > > > Today's Topics: > > 1. Re: Inquiry Regarding Handling of QUIC Connections During > Nginx Reload (Roman Arutyunyan) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 26 Feb 2024 15:49:30 +0400 > From: Roman Arutyunyan <a...@nginx.com> > To: nginx-devel@nginx.org > Subject: Re: Inquiry Regarding Handling of QUIC Connections During > Nginx Reload > Message-ID: <20240226114930.izp2quxwsp2usnvg@N00W24XTQX> > Content-Type: text/plain; charset=utf-8 > > Hi, > > On Sun, Feb 25, 2024 at 03:53:23PM +0800, 上勾拳 wrote: > > Hello, > > > > I hope this email finds you well. I am writing to inquire about the > status > > of an issue I have encountered regarding the handling of QUIC connections > > when Nginx is reloaded. > > > > Recently, I delved into the Nginx eBPF implementation, specifically > > examining how QUIC connection packets are handled, especially during > Nginx > > reloads. My focus was on ensuring that existing QUIC connection packets > are > > correctly routed to the appropriate worker after a reload, and the Nginx > > eBPF prog have done this job perfectly. > > > > However, I observed that during a reload, new QUIC connections might be > > directed to the old worker. The underlying problem stems from the fact > that > > new QUIC connections fail to match the eBPF reuseport socket map. The > > kernel default logic then routes UDP packets based on the hash UDP > 4-tuple > > in the reuseport group socket array. Since the old worker's listen FDs > > persist in the reuseport group socket array (reuse->socks), there is a > > possibility that the old worker may still be tasked with handling new > QUIC > > connections. > > > > Given that the old worker should not process new requests, it results in > > the old worker not responding to the QUIC new connection. Consequently, > > clients have to endure the handshake timeout and retry the connection, > > potentially encountering the old worker again, leading to an ineffective > > cycle. > > > > In my investigation, I came across a thread in the nginx-devel mailing > list > > [https://www.mail-archive.com/nginx-devel@nginx.org/msg10627.html], > where > > it was mentioned that there would be some work addressing this issue. > > > > Considering my limited experience with eBPF, I propose a potential > > solution. The eBPF program could maintain another eBPF map containing > only > > the listen sockets of the new worker. When the eBPF program calls > > `bpf_sk_select_reuseport` and receives `-ENOENT`, it could utilize this > new > > eBPF map with the hash UDP 4-tuple to route the new QUIC connection to > the > > new worker. This approach aims to circumvent the kernel logic routing the > > packet to the shutting down worker since removing the old worker's listen > > socket from the reuseport group socket array seems unfeasible. Not sure > > about whether this solution is a good idea and I also wonder if there are > > other solutions for this. I would appreciate any insights or updates you > > could provide on this matter. Thank you for your time and consideration. > > It is true QUIC in nginx does not handle reloads well. This is a known > issue > and we are working on improving it. I made an effort a while back to > address > QUIC reloads in nginx: > > > https://mailman.nginx.org/pipermail/nginx-devel/2023-January/thread.html#16073 > > Here patch #3 implements ePBF-based solution and patch #4 implements > client sockets-based solution. The client sockets require extra worker > process > privileges to bind to listen port, which is a serious problem for a typical > nginx configuration. The ePBF solution does not seem to have any problems, > but we still need more feedback to proceed with this. If you apply all 4 > patches, make sure you disable client sockets using > --without-quic_client_sockets. Otherwise just apply the first 3 patches. > > Here's a relevant trac ticket: > > https://trac.nginx.org/nginx/ticket/2528 > > -- > Roman Arutyunyan > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > nginx-devel mailing list > nginx-devel@nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx-devel > > > ------------------------------ > > End of nginx-devel Digest, Vol 162, Issue 26 > ******************************************** >
_______________________________________________ nginx-devel mailing list nginx-devel@nginx.org https://mailman.nginx.org/mailman/listinfo/nginx-devel
