[
https://issues.apache.org/jira/browse/HDFS-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15892062#comment-15892062
]
Yongtao Yang edited comment on HDFS-8409 at 3/2/17 11:07 AM:
-------------------------------------------------------------
I found a clue that may be useful to this problem. {{DFSClient#namenode}} is a
proxy whose {{InvocationHandler}} is {{RetryInvocationHandler}}. If
{{namenode.toString()}} or {{namenode.hashCode()}} or some other non-rpc
methods are called, then the next real rpc method will failed at
{{setCallIdAndRetryCount()}}. The cause of this is that {{toString()}} is also
forwarded to {{RetryInvocationHandler.invoke()}} and
{{setCallIdAndRetryCount()}} is executed there, then
{{ipc.Client#callId.get()}} will not be null any more. For real rpc methods,
{{ipc.Client#callId}} will be reset to null when creating a {{Call}}
instance({{org.apache.hadoop.ipc.Client.Call.Call(RpcKind, Writable)}}), but
for {{toString()}} or {{hashCode()}}, it can't reach the {{Call}} constructor,
so {{ipc.Client#callId}} will never recover to be null and the next
method(whether it is a real rpc or not) will fail.
was (Author: ytyang):
I found a clue that may be useful to this problem. {{DFSClient#namenode}} is a
proxy whose InvocationHandler is RetryInvocationHandler. If namenode.toString()
or namenode.hashCode() or some other non-rpc methods are called, then the next
real rpc method will failed at setCallIdAndRetryCount(). The cause of this is
that toString() is also forwarded to RetryInvocationHandler.invoke() and
setCallIdAndRetryCount() is executed, then ipc.Client#callId.get() will not be
null any more. For real rpc methods, ipc.Client#callId will be set to null when
creating a Call instance(org.apache.hadoop.ipc.Client.Call.Call(RpcKind,
Writable)), but for toString() or hashCode(), it can't reach the Call
constructor, so ipc.Client#callId will never recover to be null and the next
method(whether it is a real rpc or not) will fail.
> HDFS client RPC call throws "java.lang.IllegalStateException"
> -------------------------------------------------------------
>
> Key: HDFS-8409
> URL: https://issues.apache.org/jira/browse/HDFS-8409
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Reporter: Juan Yu
> Assignee: Juan Yu
> Attachments: HDFS-8409.001.patch, HDFS-8409.002.patch,
> HDFS-8409.003.patch
>
>
> When the HDFS client RPC calls need to retry, it sometimes throws
> "java.lang.IllegalStateException" and retry is aborted and cause the client
> call will fail.
> {code}
> Caused by: java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:116)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:99)
> at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1912)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1089)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
> {code}
> Here is the check that throws exception
> {code}
> public static void setCallIdAndRetryCount(int cid, int rc) {
> ...
> Preconditions.checkState(callId.get() == null);
> }
> {code}
> The RetryInvocationHandler tries to call it with not null callId and causes
> exception.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
