Siyao Meng created HDDS-17442:
---------------------------------
Summary: Domain socket requests run on the read executor shared
with gRPC, where a response write that waits for its reader holds a thread
Key: HDDS-17442
URL: https://issues.apache.org/jira/browse/HDDS-17442
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-4-process-domain-socket-requests-on-connection-thread.patch
h3. Mechanism
{{OzoneContainer}} creates one fixed size read executor
({{hdds.datanode.read.chunk.threads.per.volume}} threads per volume, 10 by
default, unbounded queue) and passes it to both {{XceiverServerGrpc}} and
{{XceiverServerDomainSocket}}. The {{Receiver}} thread of each domain socket
connection reads requests and submits one {{Receiver.ProcessRequestTask}} per
request to that executor, with no limit on how many tasks of one connection are
queued or running.
Each task dispatches its request and then calls {{Receiver.sendResponse}},
which writes the response while holding a per connection lock. A write on a
connection whose responses are not being read waits until the socket send
timeout ({{ozone.client.write.timeout}}, 30s by default). Meanwhile every other
task of the same connection that a pool thread picks up waits on the lock.
The tasks of one connection can therefore hold every thread of the executor,
and gRPC requests and the requests of other domain socket connections wait in
the queue behind them.
h3. Trigger
The executor holds at least as many tasks of one domain socket connection as it
has threads while response writes on that connection wait for the reader (for
example a paused client). Short-circuit read must be enabled on the datanode
({{ozone.client.read.short-circuit}}, off by default).
h3. Impact
* While that state lasts, requests to the datanode's gRPC container server,
which runs on the same executor, and short-circuit GetBlock requests of other
connections on that datanode get no response, so their clients time out.
* The executor is released when the connection is closed or its responses are
read. Only the datanode that holds the connection is affected.
* With the default executor size I saw the tasks of one such connection hold
all 10 threads (one in the write, nine waiting on the lock) and a gRPC GetBlock
on the same datanode get no response within 5s until the connection was closed
(one datanode MiniOzoneCluster, macOS). Not run on Linux, where the socket
buffer sizes differ, or in secure mode.
h3. Reproduction
No standalone reproduction is attached to this issue. The regression test in
the attached patch covers it and fails on unmodified source:
{{TestXceiverServerDomainSocket.testUnreadResponsesDoNotDelayOtherConnections}}
leaves the responses of one connection unread, then requires that a GetBlock on
a second connection is answered and that the first connection receives both
responses, in order, once it reads them. On unmodified source (with the suite's
one thread executor) the GetBlock on the second connection gets no response
within the 5s client read timeout and the test fails with
{{SocketTimeoutException: read(2) error: Resource temporarily unavailable}}.
h3. Patch
[^CR-4-process-domain-socket-requests-on-connection-thread.patch], against
69d77beff60b9106e2517c36a7a1b2cc726af568. It also applies to master at
1acf315d546.
{{Receiver.run}} runs each request on the connection's own thread instead of
submitting it to the shared executor. A response write that waits for its
reader then holds only that connection's thread, and gRPC requests and other
connections never queue behind it. {{Receiver}} and
{{XceiverServerDomainSocket}} no longer take the executor, and
{{OzoneContainer}} no longer passes it. The gRPC server is unchanged.
Behaviour change for clients: the requests of one connection are now processed
one at a time in arrival order. Their responses were already written one at a
time, but before the change up to pool size requests of one connection could be
dispatched in parallel, so concurrent GetBlock requests that share one
{{XceiverClientShortCircuit}} are now serialized on the datanode. I expect no
visible throughput change, because only GetBlock uses this connection, one per
block stream, and the block data is then read locally through the received file
descriptor, but I did not measure it. The number of domain socket requests
processed at once is now bounded by the number of connections
({{maxXceiverCount}}, five times the pool size) instead of the pool size. The
local op in queue latency metric ({{localInQueueLatencyNs<op>}}) of domain
socket requests now reads about 0, since there is no queue. There is no
configuration or protocol change.
Alternatives considered: a cap on the requests one connection may have on the
shared executor keeps parallelism per connection, but several such connections
still hold the whole executor and it needs a new limit; a separate executor for
the domain socket server protects gRPC but not the other domain socket clients;
dispatching each request on the shared executor and writing its response on the
connection's thread keeps the executor's bound on concurrent work, but it
serializes the requests of a connection just the same, adds a handoff, keeps
the executor wiring, and keeps domain socket latency tied to the gRPC queue.
With the patch {{TestXceiverServerDomainSocket}} (15 tests),
{{TestXceiverClientManagerSC}}, {{TestLocalChunkInputStream}},
{{TestDomainSocketFactory}}, {{TestOzoneContainer}} and {{TestKeyValueHandler}}
pass, and checkstyle on {{hadoop-hdds/container-service}} and
{{hadoop-ozone/integration-test}} is clean.
Overlap with other open patches: this patch removes the read executor argument
of the {{XceiverServerDomainSocket}} constructor, which the new tests in the
patches of HDDS-17436, HDDS-17437 and HDDS-17441 pass, and it conflicts
textually with the patches of HDDS-17436 (test imports) and HDDS-17437 (the
{{Receiver.create}} call in {{XceiverServerDomainSocket.run}}). Whichever lands
second drops that argument or rebases.
Found by code review of the datanode short-circuit read paths, as part of the
TLA+ verification effort under HDDS-15926, on commit
69d77beff60b9106e2517c36a7a1b2cc726af568. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-11627 (introduced the domain socket GetBlock channel),
HDDS-16918 (datanode logging of domain socket failures). The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
