[
https://issues.apache.org/jira/browse/HDFS-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13709496#comment-13709496
]
Suresh Srinivas commented on HDFS-4942:
---------------------------------------
bq. I was thinking about it and I feel that adding the retry-related fields and
flags in the RPC layer is not the best way.
Most RPC protocols have a requirement to have unique request identifier/Message
ID/Transaction ID for execute-at-most-once semantics and for implementing retry
cache. Some protocols started out with simpler xid (ONC-RPC 32 bit number),
much like callId in hadoop RPC, and migrated to different request identifier
format since the previous scheme did not guarantee uniqueness. See -
http://ietf.10.n7.nabble.com/Sessions-RPC-XID-usage-td160272.html.
Initial proposal in HADOOP-9688 was to leave the callId field as is and add a
new RequestId field, that guarantees uniqueness. Later if you follow that jira,
we decided for the new schema. So clientId + callId is the unique Request
Identifier. This is simplified having to generated unique requestId every time.
We are also adding retry count, because it optimizes retry cache look up on the
server side. We can also provide more metrics around how many requests are
retried etc. This I think is useful.
bq. Mostly because the retry logic is intended for a few HDFS methods only,
while the new field and flag will be serialized and de-serialized by everybody
including DataNodes, Balancers, MapReduce and Yarn.
The current mechanism can be used by Yarn and Yarm related Application Masters.
In fact others having to rebuild the same mechanism at the application layer
seems unnecessary to me.
The new field is a 16 byte field. I think the impact of this should be in the
noise, compared the existing RPC payload, RTT related to RPC request and
response and time required for handling the RPC request itself. That is why I
clearly stated in the comment that optimizing a 16 byte field seems unnecessary
in -
https://issues.apache.org/jira/browse/HADOOP-9688?focusedCommentId=13698421&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13698421
bq. constrain changes to HDFS only
The current solution will be useful for all the other applications, instead of
every one building equivalent of client name. To do this, existing APIs need to
be changed for YARN and YARN applications. In fact for YARN, there is no
equivalent of DFSClient. Expecting AM developers to get generating unique
client name right could be error prone if client do not generate unique names.
I would also argue that DFSClient name belongs to the application layer and
API. It was an identity used for tracking the lease. I see changing that and
using it for retry cache as not a clean solution.
bq. avoid incompatible RPC changes that effect sub-projects
I am not sure what you mean by this. I plan to commit the bits required for
protocol compatibility to 2.1.0-beta. As you know 2.1.0-beta is incompatible
with older release already. Also for the sake argument, lets assume that we
want to use client name, instead of client Id. That means all the
non-idempotent method signature needs to change to include clientName as a
parameter much like create() and append() calls. Those changes are also not
compatible.
bq. limit serialization overhead to only the methods involved in the retry.
As I said earlier, the cost couple of bytes should be noise, compared to the
cost of end to end RPC and countless strings generated in logging on the client
and server side and numerous temporary short lived objects created in hadoop
daemons.
bq. This will require making clientName unique as many recently advocated for.
>From what I understood, this was brought up because the previous mechanism
>required unique client name. Not sure without the context of that change,
>people were advocating for it.
> Add retry cache support in Namenode
> -----------------------------------
>
> Key: HDFS-4942
> URL: https://issues.apache.org/jira/browse/HDFS-4942
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha, namenode
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: HDFSRetryCache.pdf
>
>
> In current HA mechanism with FailoverProxyProvider and non HA setups with
> RetryProxy retry a request from the RPC layer. If the retried request has
> already been processed at the namenode, the subsequent attempts fail for
> non-idempotent operations such as create, append, delete, rename etc. This
> will cause application failures during HA failover, network issues etc.
> This jira proposes adding retry cache at the namenode to handle these
> failures. More details in the comments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
