Gargi-jais11 opened a new pull request, #11798: URL: https://github.com/apache/ozone/pull/11798
## What changes were proposed in this pull request? When `accept() `itself fails in `XceiverServerDomainSocket.run`, the loop logs a **WARN** and calls accept() again right away. For example, accept(2) fails with `"Too many open files"` when the datanode is out of file descriptors. On Linux, the queued connection keeps failing, so the accept thread spins on a CPU core and logs thousands of WARN stack traces per second until descriptors are free again. **Changes done:** This change adds a **1-second pause** before retrying when accept() itself fails, which is when no peer has been created yet. Rejections of accepted connections, such as at the xceiver limit, and idle accept timeouts behave as before. Shutdown isn't delayed, because stop() interrupts the pause. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-17436 ## How was this patch tested? added` testAcceptFailureIsRetriedAfterPause` to **TestXceiverServerDomainSocket**. Without the fix it fails with about 14,600 accept() calls in 500 ms; with the fix it passes. **Without fix:** (added few lines from test runs) ``` 2026-10-27 11:02:22,680 [627313eb-83af-4e95-97d1-b2a5308e15c2-XceiverServerDomainSocket] WARN server.XceiverServerDomainSocket (XceiverServerDomainSocket.java:run(231)) - Got an exception. Peer null java.net.SocketException: accept(2) error: Too many open files at org.apache.hadoop.hdds.TestXceiverServerDomainSocket.lambda$testAcceptFailureIsRetriedAfterPause$1(TestXceiverServerDomainSocket.java:647) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:42) at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:103) at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29) at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:34) at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:82) at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.handle(MockMethodAdvice.java:134) at org.apache.hadoop.net.unix.DomainSocket.accept(DomainSocket.java:234) at org.apache.hadoop.ozone.container.common.transport.server.XceiverServerDomainSocket.run(XceiverServerDomainSocket.java:210) at java.base/java.lang.Thread.run(Thread.java:1583) 2026-10-27 11:02:22,680 [627313eb-83af-4e95-97d1-b2a5308e15c2-XceiverServerDomainSocket] WARN server.XceiverServerDomainSocket (XceiverServerDomainSocket.java:run(231)) - Got an exception. Peer null java.net.SocketException: accept(2) error: Too many open files at org.apache.hadoop.hdds.TestXceiverServerDomainSocket.lambda$testAcceptFailureIsRetriedAfterPause$1(TestXceiverServerDomainSocket.java:647) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:42) at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:103) at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29) at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:34) at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:82) at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.handle(MockMethodAdvice.java:134) at org.apache.hadoop.net.unix.DomainSocket.accept(DomainSocket.java:234) at org.apache.hadoop.ozone.container.common.transport.server.XceiverServerDomainSocket.run(XceiverServerDomainSocket.java:210) at java.base/java.lang.Thread.run(Thread.java:1583) 2026-10-27 11:02:22,680 [627313eb-83af-4e95-97d1-b2a5308e15c2-XceiverServerDomainSocket] WARN server.XceiverServerDomainSocket (XceiverServerDomainSocket.java:run(231)) - Got an exception. Peer null java.net.SocketException: accept(2) error: Too many open files . . . org.opentest4j.AssertionFailedError: accept() was called 19061 times ==> Expected :true Actual :false ``` **After fix:** ``` 2026-10-27 10:51:26,895 [b03e6da0-e68c-47b9-ae32-6ce2bda71c99-XceiverServerDomainSocket] WARN server.XceiverServerDomainSocket (XceiverServerDomainSocket.java:run(231)) - Got an exception. Peer null java.net.SocketException: accept(2) error: Too many open files at org.apache.hadoop.hdds.TestXceiverServerDomainSocket.lambda$testAcceptFailureIsRetriedAfterPause$1(TestXceiverServerDomainSocket.java:647) at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:42) at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:103) at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29) at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:34) at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:82) at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.handle(MockMethodAdvice.java:134) at org.apache.hadoop.net.unix.DomainSocket.accept(DomainSocket.java:234) at org.apache.hadoop.ozone.container.common.transport.server.XceiverServerDomainSocket.run(XceiverServerDomainSocket.java:210) at java.base/java.lang.Thread.run(Thread.java:1583) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
