[ 
https://issues.apache.org/jira/browse/HDDS-17442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDDS-17442:
------------------------------
    Description: 
h3. Mechanism
{{OzoneContainer}} creates one fixed size read executor 
({{hdds.datanode.read.chunk.threads.per.volume}} threads per volume, 10 by 
default, unbounded queue) and passes it to both {{XceiverServerGrpc}} and 
{{XceiverServerDomainSocket}}. The {{Receiver}} thread of each domain socket 
connection reads requests and submits one {{Receiver.ProcessRequestTask}} per 
request to that executor, with no limit on how many tasks of one connection are 
queued or running.

Each task dispatches its request and then calls {{Receiver.sendResponse}}, 
which writes the response while holding a per connection lock. A write on a 
connection whose responses are not being read waits until the socket send 
timeout ({{ozone.client.write.timeout}}, 30s by default). Meanwhile every other 
task of the same connection that a pool thread picks up waits on the lock.

The tasks of one connection can therefore hold every thread of the executor, 
and gRPC requests and the requests of other domain socket connections wait in 
the queue behind them.

h3. Trigger
The executor holds at least as many tasks of one domain socket connection as it 
has threads while response writes on that connection wait for the reader (for 
example a paused client). Short-circuit read must be enabled on the datanode 
({{ozone.client.read.short-circuit}}, off by default).

h3. Impact
* While that state lasts, requests to the datanode's gRPC container server, 
which runs on the same executor, and short-circuit GetBlock requests of other 
connections on that datanode get no response, so their clients time out.
* The executor is released when the connection is closed or its responses are 
read. Only the datanode that holds the connection is affected.
* With the default executor size I saw the tasks of one such connection hold 
all 10 threads (one in the write, nine waiting on the lock) and a gRPC GetBlock 
on the same datanode get no response within 5s until the connection was closed 
(one datanode MiniOzoneCluster, macOS). Not run on Linux, where the socket 
buffer sizes differ, or in secure mode.

h3. Reproduction
No standalone reproduction is attached to this issue. The regression test in 
the attached patch covers it and fails on unmodified source: 
{{TestXceiverServerDomainSocket.testUnreadResponsesDoNotDelayOtherConnections}} 
leaves the responses of one connection unread, then requires that a GetBlock on 
a second connection is answered and that the first connection receives both 
responses once it reads them. On unmodified source (with the suite's one thread 
executor) the GetBlock on the second connection gets no response within the 5s 
client read timeout and the test fails with {{SocketTimeoutException: read(2) 
error: Resource temporarily unavailable}}.

h3. Patch
[^HDDS-17442-dispatch-on-executor-write-on-connection-thread.patch], against 
master at a5fea7c5f3b; it also applies to master at 4531bb94439. The same 
change is in the pull request. It replaces the earlier attachment 
[^CR-4-process-domain-socket-requests-on-connection-thread.patch], which ran 
each request on the connection's own thread: a client sends the GetBlock 
requests of all its block streams on one connection per pipeline, so that 
version dispatched them one after another, and a random read benchmark in the 
pull request showed a large throughput drop.

The patch keeps dispatching on the shared executor and takes only the response 
write off it:
* Each domain socket connection gets a writer thread. A task dispatches its 
request and queues the response for that thread, so a write that waits for its 
reader holds only the connection's own writer thread. The per connection lock 
in {{Receiver.sendResponse}} is removed, since only the writer writes.
* A connection may have at most as many requests on the executor or waiting to 
be written as the executor has threads, so one connection keeps the same 
parallelism as before. All connections together may have at most twice that. 
When a limit is reached, the {{Receiver}} stops reading requests until a 
response is written. The server wide limit bounds the block files and responses 
held for clients that do not read their responses; one such connection can take 
at most half of it.
* When the connection closes, the block files of responses that are still 
waiting are closed, including a response whose dispatch finishes after the 
connection is gone. If the writer thread fails while the connection is open, it 
closes the connection.

The gRPC server is unchanged. There is no configuration or protocol change. 
Responses are written in the order their requests finish, as before. Each 
connection now has two threads instead of one; the number of connections is 
already limited to five times the executor size. Clients that stop reading can 
still delay other domain socket clients once they hold the whole server wide 
limit, until their writes time out ({{ozone.client.write.timeout}}). A GetBlock 
that times out meanwhile is retried over gRPC by {{BlockInputStream}}, and gRPC 
is no longer affected.

Alternatives considered: a per connection cap without moving the write (a few 
connections whose responses are not read still hold the whole executor); a 
separate executor for the domain socket server (protects gRPC but not other 
domain socket clients); a shorter send timeout (shortens the wait but does not 
remove it).

New tests in {{TestXceiverServerDomainSocket}}: 
{{testUnreadResponsesDoNotDelayOtherConnections}} (above), 
{{testRequestsOfOneConnectionAreDispatchedInParallel}} (fails with the earlier 
attachment), {{testResponsesHeldForUnreadConnectionsAreBounded}}, 
{{testQueuedBlockFileClosedWithConnection}} and 
{{testBlockFileClosedWhenDispatchOutlivesConnection}}. With the patch 
{{TestXceiverServerDomainSocket}} (25 tests) and {{TestLocalChunkInputStream}} 
pass on macOS and checkstyle is clean. Not run on Linux; throughput not 
measured.

Found by code review of the datanode short-circuit read paths, as part of the 
TLA+ verification effort under HDDS-15926, on commit 
69d77beff60b9106e2517c36a7a1b2cc726af568. Checked against HDDS issues and 
apache/ozone pull requests for duplicates before filing. Related but not 
duplicates: HDDS-11627 (introduced the domain socket GetBlock channel), 
HDDS-16918 (datanode logging of domain socket failures). The attached patch is 
a proposal for review. Generated with Specula (Claude Opus 5.5).

  was:
h3. Mechanism
{{OzoneContainer}} creates one fixed size read executor 
({{hdds.datanode.read.chunk.threads.per.volume}} threads per volume, 10 by 
default, unbounded queue) and passes it to both {{XceiverServerGrpc}} and 
{{XceiverServerDomainSocket}}. The {{Receiver}} thread of each domain socket 
connection reads requests and submits one {{Receiver.ProcessRequestTask}} per 
request to that executor, with no limit on how many tasks of one connection are 
queued or running.

Each task dispatches its request and then calls {{Receiver.sendResponse}}, 
which writes the response while holding a per connection lock. A write on a 
connection whose responses are not being read waits until the socket send 
timeout ({{ozone.client.write.timeout}}, 30s by default). Meanwhile every other 
task of the same connection that a pool thread picks up waits on the lock.

The tasks of one connection can therefore hold every thread of the executor, 
and gRPC requests and the requests of other domain socket connections wait in 
the queue behind them.

h3. Trigger
The executor holds at least as many tasks of one domain socket connection as it 
has threads while response writes on that connection wait for the reader (for 
example a paused client). Short-circuit read must be enabled on the datanode 
({{ozone.client.read.short-circuit}}, off by default).

h3. Impact
* While that state lasts, requests to the datanode's gRPC container server, 
which runs on the same executor, and short-circuit GetBlock requests of other 
connections on that datanode get no response, so their clients time out.
* The executor is released when the connection is closed or its responses are 
read. Only the datanode that holds the connection is affected.
* With the default executor size I saw the tasks of one such connection hold 
all 10 threads (one in the write, nine waiting on the lock) and a gRPC GetBlock 
on the same datanode get no response within 5s until the connection was closed 
(one datanode MiniOzoneCluster, macOS). Not run on Linux, where the socket 
buffer sizes differ, or in secure mode.

h3. Reproduction
No standalone reproduction is attached to this issue. The regression test in 
the attached patch covers it and fails on unmodified source: 
{{TestXceiverServerDomainSocket.testUnreadResponsesDoNotDelayOtherConnections}} 
leaves the responses of one connection unread, then requires that a GetBlock on 
a second connection is answered and that the first connection receives both 
responses, in order, once it reads them. On unmodified source (with the suite's 
one thread executor) the GetBlock on the second connection gets no response 
within the 5s client read timeout and the test fails with 
{{SocketTimeoutException: read(2) error: Resource temporarily unavailable}}.

h3. Patch
[^CR-4-process-domain-socket-requests-on-connection-thread.patch], against 
69d77beff60b9106e2517c36a7a1b2cc726af568. It also applies to master at 
1acf315d546.

{{Receiver.run}} runs each request on the connection's own thread instead of 
submitting it to the shared executor. A response write that waits for its 
reader then holds only that connection's thread, and gRPC requests and other 
connections never queue behind it. {{Receiver}} and 
{{XceiverServerDomainSocket}} no longer take the executor, and 
{{OzoneContainer}} no longer passes it. The gRPC server is unchanged.

Behaviour change for clients: the requests of one connection are now processed 
one at a time in arrival order. Their responses were already written one at a 
time, but before the change up to pool size requests of one connection could be 
dispatched in parallel, so concurrent GetBlock requests that share one 
{{XceiverClientShortCircuit}} are now serialized on the datanode. A client 
keeps one such connection per pipeline and sends the GetBlock requests of all 
its block streams on it. GetBlock only looks up the block metadata and opens 
the block file, and the block data is then read locally through the received 
file descriptor, so I expect the effect to be small, but I did not measure it. 
The number of domain socket requests processed at once is now bounded by the 
number of connections ({{maxXceiverCount}}, five times the pool size) instead 
of the pool size. The local op in queue latency metric 
({{localInQueueLatencyNs<op>}}) of domain socket requests now reads about 0, 
since there is no queue. There is no configuration or protocol change.

Alternatives considered: a cap on the requests one connection may have on the 
shared executor keeps parallelism per connection, but several such connections 
still hold the whole executor and it needs a new limit; a separate executor for 
the domain socket server protects gRPC but not the other domain socket clients; 
dispatching each request on the shared executor and writing its response on the 
connection's thread keeps the executor's bound on concurrent work, but it 
serializes the requests of a connection just the same, adds a handoff, keeps 
the executor wiring, and keeps domain socket latency tied to the gRPC queue.

With the patch {{TestXceiverServerDomainSocket}} (15 tests), 
{{TestXceiverClientManagerSC}}, {{TestLocalChunkInputStream}}, 
{{TestDomainSocketFactory}}, {{TestOzoneContainer}} and {{TestKeyValueHandler}} 
pass, and checkstyle on {{hadoop-hdds/container-service}} and 
{{hadoop-ozone/integration-test}} is clean.

Overlap with other open patches: this patch removes the read executor argument 
of the {{XceiverServerDomainSocket}} constructor, which the new tests in the 
patches of HDDS-17436, HDDS-17437 and HDDS-17441 pass, and it conflicts 
textually with the patches of HDDS-17436 (test imports) and HDDS-17437 (the 
{{Receiver.create}} call in {{XceiverServerDomainSocket.run}}). Whichever lands 
second drops that argument or rebases.

Found by code review of the datanode short-circuit read paths, as part of the 
TLA+ verification effort under HDDS-15926, on commit 
69d77beff60b9106e2517c36a7a1b2cc726af568. Checked against HDDS issues and 
apache/ozone pull requests for duplicates before filing. Related but not 
duplicates: HDDS-11627 (introduced the domain socket GetBlock channel), 
HDDS-16918 (datanode logging of domain socket failures). The attached patch is 
a proposal for review. Generated with Specula (Claude Opus 5.5).


> Domain socket requests run on the read executor shared with gRPC, where a 
> response write that waits for its reader holds a thread
> ---------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17442
>                 URL: https://issues.apache.org/jira/browse/HDDS-17442
>             Project: Apache Ozone
>          Issue Type: Bug
>    Affects Versions: 2.3.0
>            Reporter: Siyao Meng
>            Assignee: Siyao Meng
>            Priority: Blocker
>              Labels: pull-request-available
>         Attachments: 
> CR-4-process-domain-socket-requests-on-connection-thread.patch, 
> HDDS-17442-dispatch-on-executor-write-on-connection-thread.patch
>
>
> h3. Mechanism
> {{OzoneContainer}} creates one fixed size read executor 
> ({{hdds.datanode.read.chunk.threads.per.volume}} threads per volume, 10 by 
> default, unbounded queue) and passes it to both {{XceiverServerGrpc}} and 
> {{XceiverServerDomainSocket}}. The {{Receiver}} thread of each domain socket 
> connection reads requests and submits one {{Receiver.ProcessRequestTask}} per 
> request to that executor, with no limit on how many tasks of one connection 
> are queued or running.
> Each task dispatches its request and then calls {{Receiver.sendResponse}}, 
> which writes the response while holding a per connection lock. A write on a 
> connection whose responses are not being read waits until the socket send 
> timeout ({{ozone.client.write.timeout}}, 30s by default). Meanwhile every 
> other task of the same connection that a pool thread picks up waits on the 
> lock.
> The tasks of one connection can therefore hold every thread of the executor, 
> and gRPC requests and the requests of other domain socket connections wait in 
> the queue behind them.
> h3. Trigger
> The executor holds at least as many tasks of one domain socket connection as 
> it has threads while response writes on that connection wait for the reader 
> (for example a paused client). Short-circuit read must be enabled on the 
> datanode ({{ozone.client.read.short-circuit}}, off by default).
> h3. Impact
> * While that state lasts, requests to the datanode's gRPC container server, 
> which runs on the same executor, and short-circuit GetBlock requests of other 
> connections on that datanode get no response, so their clients time out.
> * The executor is released when the connection is closed or its responses are 
> read. Only the datanode that holds the connection is affected.
> * With the default executor size I saw the tasks of one such connection hold 
> all 10 threads (one in the write, nine waiting on the lock) and a gRPC 
> GetBlock on the same datanode get no response within 5s until the connection 
> was closed (one datanode MiniOzoneCluster, macOS). Not run on Linux, where 
> the socket buffer sizes differ, or in secure mode.
> h3. Reproduction
> No standalone reproduction is attached to this issue. The regression test in 
> the attached patch covers it and fails on unmodified source: 
> {{TestXceiverServerDomainSocket.testUnreadResponsesDoNotDelayOtherConnections}}
>  leaves the responses of one connection unread, then requires that a GetBlock 
> on a second connection is answered and that the first connection receives 
> both responses once it reads them. On unmodified source (with the suite's one 
> thread executor) the GetBlock on the second connection gets no response 
> within the 5s client read timeout and the test fails with 
> {{SocketTimeoutException: read(2) error: Resource temporarily unavailable}}.
> h3. Patch
> [^HDDS-17442-dispatch-on-executor-write-on-connection-thread.patch], against 
> master at a5fea7c5f3b; it also applies to master at 4531bb94439. The same 
> change is in the pull request. It replaces the earlier attachment 
> [^CR-4-process-domain-socket-requests-on-connection-thread.patch], which ran 
> each request on the connection's own thread: a client sends the GetBlock 
> requests of all its block streams on one connection per pipeline, so that 
> version dispatched them one after another, and a random read benchmark in the 
> pull request showed a large throughput drop.
> The patch keeps dispatching on the shared executor and takes only the 
> response write off it:
> * Each domain socket connection gets a writer thread. A task dispatches its 
> request and queues the response for that thread, so a write that waits for 
> its reader holds only the connection's own writer thread. The per connection 
> lock in {{Receiver.sendResponse}} is removed, since only the writer writes.
> * A connection may have at most as many requests on the executor or waiting 
> to be written as the executor has threads, so one connection keeps the same 
> parallelism as before. All connections together may have at most twice that. 
> When a limit is reached, the {{Receiver}} stops reading requests until a 
> response is written. The server wide limit bounds the block files and 
> responses held for clients that do not read their responses; one such 
> connection can take at most half of it.
> * When the connection closes, the block files of responses that are still 
> waiting are closed, including a response whose dispatch finishes after the 
> connection is gone. If the writer thread fails while the connection is open, 
> it closes the connection.
> The gRPC server is unchanged. There is no configuration or protocol change. 
> Responses are written in the order their requests finish, as before. Each 
> connection now has two threads instead of one; the number of connections is 
> already limited to five times the executor size. Clients that stop reading 
> can still delay other domain socket clients once they hold the whole server 
> wide limit, until their writes time out ({{ozone.client.write.timeout}}). A 
> GetBlock that times out meanwhile is retried over gRPC by 
> {{BlockInputStream}}, and gRPC is no longer affected.
> Alternatives considered: a per connection cap without moving the write (a few 
> connections whose responses are not read still hold the whole executor); a 
> separate executor for the domain socket server (protects gRPC but not other 
> domain socket clients); a shorter send timeout (shortens the wait but does 
> not remove it).
> New tests in {{TestXceiverServerDomainSocket}}: 
> {{testUnreadResponsesDoNotDelayOtherConnections}} (above), 
> {{testRequestsOfOneConnectionAreDispatchedInParallel}} (fails with the 
> earlier attachment), {{testResponsesHeldForUnreadConnectionsAreBounded}}, 
> {{testQueuedBlockFileClosedWithConnection}} and 
> {{testBlockFileClosedWhenDispatchOutlivesConnection}}. With the patch 
> {{TestXceiverServerDomainSocket}} (25 tests) and 
> {{TestLocalChunkInputStream}} pass on macOS and checkstyle is clean. Not run 
> on Linux; throughput not measured.
> Found by code review of the datanode short-circuit read paths, as part of the 
> TLA+ verification effort under HDDS-15926, on commit 
> 69d77beff60b9106e2517c36a7a1b2cc726af568. Checked against HDDS issues and 
> apache/ozone pull requests for duplicates before filing. Related but not 
> duplicates: HDDS-11627 (introduced the domain socket GetBlock channel), 
> HDDS-16918 (datanode logging of domain socket failures). The attached patch 
> is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to