[
https://issues.apache.org/jira/browse/HADOOP-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18135695#comment-18135695
]
ASF GitHub Bot commented on HADOOP-20047:
-----------------------------------------
smengcl opened a new pull request, #8902:
URL: https://github.com/apache/hadoop/pull/8902
Generated-by: Claude Code (Opus 5.5)
JIRA: https://issues.apache.org/jira/browse/HADOOP-20047
### Description of PR
`RetryInvocationHandler` sets the thread local call id and retry count
before each attempt, and only `Client.Call` clears the call id when it is
created. When an attempt issues no RPC (calling `toString` or `hashCode` on the
retry proxy, or the protocol translator throwing before `Client.call`), the
call id stays set. The next attempt on the same thread then fails
`checkState(callId.get() == null)` in `Client.setCallIdAndRetryCount` with a
bare `IllegalStateException`, which aborts the retry and every later call made
on that thread. For asynchronous calls the retries run on the retry handler's
thread, so every later call on that handler is affected.
This patch clears the call id, retry count and external handler after each
attempt that set them, using the same reset `RequestHedgingProxyProvider`
already applies. It changes nothing when `SET_CALL_ID_FOR_TEST` is off.
This also covers HDFS-8409 and HDFS-11351. Apache Ozone hit the same fault
in its copy of this code and fixed it as HDDS-16745.
### How was this patch tested?
Added `TestRetryProxy#testCallIdClearedWhenNoRpcIsIssued`: through a retry
proxy over an RPC translator, calls `toString` and then twice a method that
throws before issuing the RPC. Without the fix the calls fail with
`IllegalStateException`; with it they fail with the expected `FatalException`.
Also ran `TestRetryProxy`, `TestFailoverProxy`, `TestDefaultRetryPolicy`,
`TestAsyncIPC`, `TestRetryCache`, `TestIPCServerResponder`, `TestIPC` and
`TestRPC` locally. All pass except five `TestIPC` tests and two `TestRPC` tests
that fail, or fail intermittently, the same way without this patch on the test
machine (connection resets and socket timeouts on its network).
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
### AI Tooling
- [x] The PR includes the phrase "Contains content generated by Claude Code
(Opus 5.5)".
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
Contains content generated by Claude Code (Opus 5.5).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
> RetryInvocationHandler leaves the call id set when an attempt issues no RPC
> ---------------------------------------------------------------------------
>
> Key: HADOOP-20047
> URL: https://issues.apache.org/jira/browse/HADOOP-20047
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, ipc
> Affects Versions: 3.5.0, 3.4.3
> Reporter: Siyao Meng
> Priority: Major
>
> RetryInvocationHandler sets the thread local call id and retry count in
> Client before every attempt on an RPC proxy, but only the creation of a
> Client.Call clears the call id. When an attempt issues no RPC, the call id
> stays set and the next attempt on that thread fails in
> Client.setCallIdAndRetryCount:
> {noformat}
> java.lang.IllegalStateException
> at
> org.apache.hadoop.util.Preconditions.checkState(Preconditions.java:254)
> at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:164)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:167)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:162)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:100)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:366)
> {noformat}
> Attempts that issue no RPC include calling toString or hashCode on the retry
> proxy (noted in HDFS-8409), and a protocol translator or
> ProtobufRpcEngine2.Invoker throwing before Client.call. After that, the retry
> is aborted and the next call through any retry proxy on the same thread
> fails. Asynchronous calls retry on the handler's own thread, so later calls
> on that handler fail as well.
> The code is in hadoop-common, so this affects every RetryProxy over a Hadoop
> RPC proxy, including HDFS clients (HDFS-8409, HDFS-11351) and YARN (RMProxy,
> ServerProxy). Apache Ozone fixed the same fault in its copy of this code in
> HDDS-16745.
> Proposed fix: clear the call id, retry count and external handler after every
> attempt that set them, with a regression test in TestRetryProxy.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
