[
https://issues.apache.org/jira/browse/HDDS-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-17767:
------------------------------------
Fix Version/s: 2.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Streaming reads hold XceiverClientGrpc request permits for the lifetime of
> the block stream
> -------------------------------------------------------------------------------------------
>
> Key: HDDS-17767
> URL: https://issues.apache.org/jira/browse/HDDS-17767
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Datanode
> Affects Versions: 2.2.2
> Reporter: Sergey Soldatov
> Assignee: Sergey Soldatov
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
>
> {{XceiverClientGrpc#initStreamRead}} takes a permit from the request
> semaphore that also limits unary requests
> ({{hdds.ratis.raft.client.async.outstanding-requests.max}}). The permit is
> only returned by {{completeStreamRead()}} when the block input stream closes,
> so each open streaming read holds one for as long as it stays open.
> This causes two problems:
> * *Unary requests are starved.* Long-lived block streams use up the permits
> that short requests to the same pipeline need. With enough streams open,
> those requests wait until a stream closes.
> * *Readers can hang.* {{semaphore.acquire()}} has no timeout. If every permit
> is held by open streams, the next stream open waits with no limit. A reader
> that already holds streams on the client waits for a permit only it can
> release, so the read never finishes. Two common cases are a positioned read
> while a sequential stream is open, and a key input stream that has seeked
> into several blocks.
> Streaming read concurrency also has no limit of its own. It shares the
> request limit, so it can't be tuned separately, even though each open stream
> can buffer up to {{ozone.client.stream.read.pre-read-size}} bytes.
> Fix: streaming reads get their own permit pool, sized by the new
> {{ozone.client.stream.read.max-concurrent-streams}}. When it is unset, it
> uses the request limit as before. Waiting for a stream permit can be bounded
> by {{ozone.client.stream.read.timeout}}, and a timed-out wait is retried
> under the read retry policy.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]