ss77892 opened a new pull request, #11080: URL: https://github.com/apache/ozone/pull/11080
## What changes were proposed in this pull request? HDDS-16241. gRPC deadline kills long-lived block streams after 30 seconds and the client never recovers XceiverClientGrpc.initStreamRead arms a gRPC deadline (withDeadlineAfter, ozone.client.read.timeout, default 30 seconds) on the long-lived streaming ReadBlock call introduced by HDDS-13974. A gRPC deadline bounds the entire call, not a single request, so every block stream is cancelled with DEADLINE_EXCEEDED 30 seconds after it opens, even when it is perfectly healthy. The failure then becomes permanent on the client side: 1. The cancellation lands in StreamingReader.onError and completes the reader future exceptionally. StreamBlockInputStream never re-initializes a reader whose future has failed, so checkError replays the stale exception on every subsequent read of that block for as long as the input stream stays open. 2. The failure is not classified as retryable: isConnectivityIssue only accepts UNAVAILABLE, and checkError wraps the original exception in a new IOException whose cause chain (IOException -> ExecutionException -> StatusRuntimeException) the retry classification cannot unwrap. So the failover/refresh path in handleExceptions never engages. Long-lived readers hit this hard. On an HBase-on-Ozone cluster, RegionServers keep store file input streams open indefinitely; after each stream's first 30 seconds, every pread through it fails instantly. A YCSB read workload showed a steady ~42 percent error rate (reads served from HBase block cache or memstore still succeeded, masking the problem for the first 30 seconds of each stream's life). Short-lived readers (CLI, file copies) close before the deadline fires. A related capacity problem: initStreamRead held a permit from the shared request semaphore for the stream's whole lifetime, so a client with many open files could exhaust the permits, starve short RPCs, and block new streams indefinitely (observed as 30-second RegionServer-wide stalls). What fix does: Remove the trigger, make recovery work, and separate stream capacity from request capacity: 1. Drop the call-lifetime deadline from the streaming call. Per-request timeliness is already enforced by ozone.client.stream.read.timeout in streamRead flow-control waits and StreamingReader.poll, which fails a stuck request with a retryable TimeoutIOException. 2. Never poison the stream. handleExceptions now marks the reader failed and, on the non-retryable path, tears down the failed stream and resets the request high-water mark before rethrowing, so the next read builds a fresh stream instead of replaying the stale failure. closeReader also resets requestedLength, since requested-but-unreceived data dies with the stream (this also fixes read-after-unbuffer resuming from a stale prefetch offset and stalling). 3. Make the failure classifiable. checkError rethrows the original IOException from the ExecutionException cause so retry classification sees the real failure, and isConnectivityIssue treats DEADLINE_EXCEEDED as a connectivity issue alongside UNAVAILABLE, enabling datanode failover. 4. Cancel failed streams instead of half-closing them. closeReader now calls cancel on a failed call; onCompleted only half-closes it, which would leave the RPC alive against a stuck datanode. 5. Account long-lived streams separately from short RPCs. Streams now take permits from a dedicated semaphore sized by a new config, ozone.client.stream.read.max-concurrent-streams. Any non-positive value (the default) inherits hdds.ratis.raft.client.async.outstanding-requests.max, preserving the capacity streams had when they shared the request semaphore. Acquisition waits up to ozone.client.stream.read.timeout and then fails with an actionable message instead of blocking indefinitely; a failed initStreamRead releases its permit. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-16241 ## How was this patch tested? UT has been added. Basic freon/cli workloads to confirm that basic functionality hasn't been broken HBase on Ozone cluster with YCSB workloads. The rate of failures dropped from ~60% to less than 1%. This 1% would be addressed as a separate jira because it has different root cause. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
