[
https://issues.apache.org/jira/browse/HDFS-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15892062#comment-15892062
]
Yongtao Yang edited comment on HDFS-8409 at 3/2/17 11:05 AM:
-------------------------------------------------------------
I found a clue that may be useful to this problem. {{DFSClient#namenode}} is a
proxy whose InvocationHandler is RetryInvocationHandler. If namenode.toString()
or namenode.hashCode() or some other non-rpc methods are called, then the next
real rpc method will failed at setCallIdAndRetryCount(). The cause of this is
that toString() is also forwarded to RetryInvocationHandler.invoke() and
setCallIdAndRetryCount() is executed, then ipc.Client#callId.get() will not be
null any more. For real rpc methods, ipc.Client#callId will be set to null when
creating a Call instance(org.apache.hadoop.ipc.Client.Call.Call(RpcKind,
Writable)), but for toString() or hashCode(), it can't reach the Call
constructor, so ipc.Client#callId will never recover to be null and the next
method(whether it is a real rpc or not) will fail.
was (Author: ytyang):
I found a clue that may be useful to this problem. <tt>DFSClient#namenode</tt>
is a proxy whose InvocationHandler is RetryInvocationHandler. If
namenode.toString() or namenode.hashCode() or some other non-rpc methods are
called, then the next real rpc method will failed at setCallIdAndRetryCount().
The cause of this is that toString() is also forwarded to
RetryInvocationHandler.invoke() and setCallIdAndRetryCount() is executed, then
ipc.Client#callId.get() will not be null any more. For real rpc methods,
ipc.Client#callId will be set to null when creating a Call
instance(org.apache.hadoop.ipc.Client.Call.Call(RpcKind, Writable)), but for
toString() or hashCode(), it can't reach the Call constructor, so
ipc.Client#callId will never recover to be null and the next method(whether it
is a real rpc or not) will fail.
> HDFS client RPC call throws "java.lang.IllegalStateException"
> -------------------------------------------------------------
>
> Key: HDFS-8409
> URL: https://issues.apache.org/jira/browse/HDFS-8409
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Reporter: Juan Yu
> Assignee: Juan Yu
> Attachments: HDFS-8409.001.patch, HDFS-8409.002.patch,
> HDFS-8409.003.patch
>
>
> When the HDFS client RPC calls need to retry, it sometimes throws
> "java.lang.IllegalStateException" and retry is aborted and cause the client
> call will fail.
> {code}
> Caused by: java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:116)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:99)
> at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1912)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1089)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
> {code}
> Here is the check that throws exception
> {code}
> public static void setCallIdAndRetryCount(int cid, int rc) {
> ...
> Preconditions.checkState(callId.get() == null);
> }
> {code}
> The RetryInvocationHandler tries to call it with not null callId and causes
> exception.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
