[
https://issues.apache.org/jira/browse/HDDS-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gargi Jaiswal reassigned HDDS-17436:
------------------------------------
Assignee: Gargi Jaiswal
> XceiverServerDomainSocket retries a failed accept() at once and spins while
> the datanode is out of file descriptors
> -------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17436
> URL: https://issues.apache.org/jira/browse/HDDS-17436
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Assignee: Gargi Jaiswal
> Priority: Minor
> Attachments:
> CR-3-pause-before-retrying-failed-domain-socket-accept.patch,
> TestBugCR3AcceptFailureRetry.java
>
>
> h3. Mechanism
> {{XceiverServerDomainSocket.run}} catches every {{IOException}} other than a
> timeout or an asynchronous close in one branch, which logs a WARN with the
> stack trace, closes the peer and calls {{accept()}} again right away. The
> branch was written for the rejection of a single connection at the xceiver
> limit (its comment says "usually when the xceiver count limit is hit"). When
> {{accept()}} itself fails, nothing changes before the next call, so a
> persistent failure turns the loop into a busy spin. Only the
> {{OutOfMemoryError}} branch waits before retrying. The loop follows HDFS
> {{DataXceiverServer.run}}, which has no pause in that branch either. The
> generic {{catch (Throwable)}} branch logs "XceiverServerDomainSocket:
> Exiting." but keeps looping without a pause; it is not changed here.
> h3. Trigger
> # Short-circuit read is enabled and the datanode process has reached its open
> file limit, for whatever reason outside this server.
> # A short-circuit client connects. On Linux the connection stays queued and
> each {{accept()}} fails at once with {{accept(2) error: Too many open files}}.
> The loop spins for as long as both last. Run with an injected accept failure,
> see Reproduction.
> h3. Impact
> * The accept thread uses most of a core and produces one WARN log event with
> a stack trace per attempt: about 470k events per second in the Linux run
> described under Reproduction, about 60k per second with the mocked failure on
> macOS. This happens during an incident in which the datanode is already short
> of resources, and the log volume can rotate away the earlier log lines of
> that incident.
> * It stops once descriptors are free again. No data is affected.
> h3. Reproduction
> PASS, deterministic, unmodified source. [^TestBugCR3AcceptFailureRetry.java]
> ({{hadoop-ozone/integration-test}}, no native library needed) starts the real
> server on a mocked listening socket whose {{accept()}} always throws the
> EMFILE {{SocketException}} and counts attempts and WARN events over one
> second: 58112 attempts and 58113 WARN events on macOS. A passing test means
> the defect is present.
> The real descriptor shortage was run separately on Linux (aarch64 container,
> Zulu 17, RLIMIT_NOFILE 10240, a separate client process waiting in the
> backlog, unmodified source): the accept thread used 82 to 84% of a core and
> produced about 470k WARN events per second, counted in the test rather than
> written to a log file, and both fell to zero once descriptors were released.
> That run was not repeated with the patch; the patch was shown with the mocked
> failing {{accept()}} on macOS. On macOS the real shortage does not produce
> the spin, because the kernel takes the pending connection off the queue when
> no descriptor is free, so {{accept()}} fails only once.
> h3. Patch
> [^CR-3-pause-before-retrying-failed-domain-socket-accept.patch], against
> 69d77beff60b9106e2517c36a7a1b2cc726af568. It also applies to master at
> 1acf315d546.
> When the {{IOException}} came from {{accept()}} itself (no peer was created
> yet), the loop sleeps one second before it calls {{accept()}} again, as the
> {{OutOfMemoryError}} branch already does for 30 seconds. A connection that is
> accepted and then rejected or fails is handled as before, without a pause. An
> interrupt during the pause is ignored like in that branch, because {{stop()}}
> clears the running flag before it interrupts the thread, so shutdown still
> ends the loop at once. The interrupt status is not restored, since a set
> status would make every later pause return at once and bring the spin back.
> Covered by
> {{TestXceiverServerDomainSocket.testAcceptFailureIsRetriedAfterPause}}, which
> drives the loop with the same mocked failure and allows at most 2 attempts in
> 500 ms. Without the change it fails with 17767 attempts. With the patch
> {{TestXceiverServerDomainSocket}} (15 tests) and
> {{TestLocalChunkInputStream}} pass, checkstyle is clean, and the attached
> reproduction fails because at most one attempt is made within its one second
> window.
> Found by code review of the datanode short-circuit read paths, as part of the
> TLA+ verification effort under HDDS-15926, on commit
> 69d77beff60b9106e2517c36a7a1b2cc726af568. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not a
> duplicate: HDDS-11627 (introduced the loop). The attached patch is a proposal
> for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
