[
https://issues.apache.org/jira/browse/HDFS-16367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17452230#comment-17452230
]
Janus Chow commented on HDFS-16367:
-----------------------------------
[~ferhui] [~hexiaoqiao] [~tasanuma] Could you help to check the ticket and the
options?
> Out-sync of socket config between client and DataNode
> -----------------------------------------------------
>
> Key: HDFS-16367
> URL: https://issues.apache.org/jira/browse/HDFS-16367
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Janus Chow
> Priority: Major
>
> We met an issue of timeout while clients writing data to datanodes. The root
> cause is the different "dfs.client.socket-timeout" config used by Clients and
> DataNode.
> Clients updated the config to 3m, DataNodes are using the default config of
> 1m. During the stage of "DATA_STREAMING" when writing pipelines, the client
> will send heartbeat packets with an interval of half of the timeout, that is
> 90seconds, but the socket timeout between the client and the first DataNode
> is 60seconds, then the client fails to write the content.
>
> IMHO, the client config of "dfs.client.socket-timeout" is reasonable to be
> configured by clients themselves, thus in a cluster, we should allow the
> circumstances of different clients using different client configs.
> If so, the problem is the out-sync of some configs between the client and
> DataNodes. After checking the code, I think the following 2 options would be
> feasible:
> # while the client is invoking writeBlock of DataTransferProtocol, we can
> transfer the config from client to DataNode, and DataXceiver should process
> and update the config. The drawback of this option is we may need to add
> arguments to other methods like "readBlock".
> # Add a new interface like "syncConfig" to DataTransferProtocol, then after
> each time the connection is created, we can invoke the method and do the
> config sync up.
> Hope to get other options and ideas.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
