Sergey Soldatov created HDDS-17561:
--------------------------------------
Summary: Deliver blocking XceiverClientGrpc replies on the waiting
thread
Key: HDDS-17561
URL: https://issues.apache.org/jira/browse/HDDS-17561
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Datanode
Affects Versions: 2.2.1
Reporter: Sergey Soldatov
Assignee: Sergey Soldatov
h3. Problem
A blocking {{XceiverClientGrpc#sendCommand}} (ReadChunk, GetBlock and the other
read-path commands) is built on top of the async path. For every request it
opens a new bi-directional gRPC stream, sends one request, half-closes the
stream, and waits on a {{CompletableFuture}}. The reply then travels:
# gRPC transport (Netty event loop) ->
# gRPC's default executor, where the {{StreamObserver}} parses the response and
completes the future ->
# the caller thread, woken up from {{get()}}.
So each blocking request pays two thread hand-offs plus the setup and teardown
of an HTTP/2 stream on both the client and the datanode, even though the caller
is just sitting idle waiting for the reply.
h3. Proposal
* *Reply on the waiting thread.* New {{CallerRunsReplyFuture}}: a
{{CompletableFuture}} that is also the call's {{Executor}}. While a thread
waits in {{get()}} / {{join()}}, the gRPC callbacks of the call are queued and
run by that thread, so the response is parsed and the future completed on the
caller (the same technique gRPC blocking stubs use). Before the caller starts
waiting and after it returns, callbacks run on a shared daemon executor, never
on the transport thread.
* *Pooled streams for untraced blocking calls.* A blocking call without a trace
context (the production default) leases an idle stream of the channel, or opens
one, and sends exactly one request on it at a time. The datanode serves the
requests of a stream in order, so the next reply always belongs to the current
request and no request IDs are needed. After the reply the stream goes back to
the pool. The number of streams is bounded by the client semaphore (max
outstanding requests).
* *Client-side deadline.* A pooled stream has no gRPC deadline, so the request
deadline is enforced by the waiter (and by a timer when nobody waits). On
timeout the request fails with {{DEADLINE_EXCEEDED}}, as before, and the stream
is cancelled and dropped.
* *Stream lifecycle.*
** Idle streams are half-closed after {{scm.container.client.idle.threshold}}
by a shared reaper thread, and immediately when the channel leaves {{READY}}
(e.g. the datanode sends GOAWAY on shutdown), so open streams do not hold up
the datanode's graceful shutdown.
** Half-closed streams that the datanode never completes are cancelled.
** A stream is dropped, never reused, when it fails, is completed by the
datanode, receives an unsolicited reply, or when its request is abandoned
(timeout, interrupt, cancel).
* *Unchanged paths.* Traced blocking calls still open one stream per call
(trace headers are per stream), but also complete on the waiting thread. The
async path ({{sendCommandAsync(ContainerCommandRequestProto)}}) is unchanged.
* *Smaller fixes along the way.* If issuing a fan-out {{sendCommand}} to one of
the datanodes fails, the requests already issued to the others are cancelled
instead of being left in flight. {{sortDatanodes}} returns early for
single-node pipelines.
* {{README.gRPC.md}} is updated to describe the new behavior.
h3. Compatibility
Client-only change. No protobuf, RPC or datanode changes; the datanode already
handles several sequential requests on one stream.
h3. Testing
New {{TestXceiverClientGrpcCallerRuns}} (33 tests against an in-process gRPC
server) covers:
* replies completing on the caller for {{get()}} and {{join()}}, and off the
submitter when nobody waits;
* stream reuse for sequential calls, distinct streams for concurrent calls, no
more streams than semaphore permits;
* deadline expiry, timed waits shorter than the request deadline, interrupted
waiters, cancel from another thread, late waiters;
* stream errors, datanode-completed streams, unsolicited replies, cancel racing
with the reply;
* idle-threshold reaping, half-closed streams never completed by the datanode,
server shutdown not being held up;
* traced calls using a stream per call, dead channels failing fast without
leaking permits, and fan-out cancellation.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
