Repository: hadoop Updated Branches: refs/heads/branch-2 0bd7ba4ea -> 1513c0179
HDFS-9831. Document webhdfs retry configuration keys introduced by HDFS-5219/HDFS-5122. Contributed by Xiaobing Zhou. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/1513c017 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/1513c017 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/1513c017 Branch: refs/heads/branch-2 Commit: 1513c0179eea1992d7b9526dd6f13e4ceda5cfb2 Parents: 0bd7ba4 Author: Xiaoyu Yao <[email protected]> Authored: Fri Feb 26 14:14:12 2016 -0800 Committer: Xiaoyu Yao <[email protected]> Committed: Fri Feb 26 14:24:03 2016 -0800 ---------------------------------------------------------------------- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 3 + .../src/main/resources/hdfs-default.xml | 62 ++++++++++++++++++++ .../hadoop-hdfs/src/site/markdown/WebHDFS.md | 18 ++++++ 3 files changed, 83 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/1513c017/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt index 3705680..fa860e9 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt +++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt @@ -1025,6 +1025,9 @@ Release 2.8.0 - UNRELEASED HDFS-9843. Document distcp options required for copying between encrypted locations. (Xiaoyu Yao via cnauroth) + HDFS-9831.Document webhdfs retry configuration keys introduced by + HDFS-5219/HDFS-5122. (Xiaobing Zhou via xyao) + OPTIMIZATIONS HDFS-8026. Trace FSOutputSummer#writeChecksumChunks rather than http://git-wip-us.apache.org/repos/asf/hadoop/blob/1513c017/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml index a250eb1..c716607 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml @@ -2810,4 +2810,66 @@ prevention header. </description> </property> + +<property> + <name>dfs.http.client.retry.policy.enabled</name> + <value>false</value> + <description> + If "true", enable the retry policy of WebHDFS client. + If "false", retry policy is turned off. + Enabling the retry policy can be quite useful while using WebHDFS to + copy large files between clusters that could timeout, or + copy files between HA clusters that could failover during the copy. + </description> +</property> + +<property> + <name>dfs.http.client.retry.policy.spec</name> + <value>10000,6,60000,10</value> + <description> + Specify a policy of multiple linear random retry for WebHDFS client, + e.g. given pairs of number of retries and sleep time (n0, t0), (n1, t1), + ..., the first n0 retries sleep t0 milliseconds on average, + the following n1 retries sleep t1 milliseconds on average, and so on. + </description> +</property> + +<property> + <name>dfs.http.client.failover.max.attempts</name> + <value>15</value> + <description> + Specify the max number of failover attempts for WebHDFS client + in case of network exception. + </description> +</property> + +<property> + <name>dfs.http.client.retry.max.attempts</name> + <value>10</value> + <description> + Specify the max number of retry attempts for WebHDFS client, + if the difference between retried attempts and failovered attempts is + larger than the max number of retry attempts, there will be no more + retries. + </description> +</property> + +<property> + <name>dfs.http.client.failover.sleep.base.millis</name> + <value>500</value> + <description> + Specify the base amount of time in milliseconds upon which the + exponentially increased sleep time between retries or failovers + is calculated for WebHDFS client. + </description> +</property> + +<property> + <name>dfs.http.client.failover.sleep.max.millis</name> + <value>15000</value> + <description> + Specify the upper bound of sleep time in milliseconds between + retries or failovers for WebHDFS client. + </description> +</property> </configuration> http://git-wip-us.apache.org/repos/asf/hadoop/blob/1513c017/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md index 57909df..9df9179 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md @@ -24,6 +24,7 @@ WebHDFS REST API * [Authentication](#Authentication) * [Proxy Users](#Proxy_Users) * [Cross-Site Request Forgery Prevention](#Cross-Site_Request_Forgery_Prevention) + * [WebHDFS Retry Policy](#WebHDFS_Retry_Policy) * [File and Directory Operations](#File_and_Directory_Operations) * [Create and Write to a File](#Create_and_Write_to_a_File) * [Append to a File](#Append_to_a_File) @@ -300,6 +301,23 @@ custom header in the request. curl -i -L -X PUT -H 'X-XSRF-HEADER: ""' 'http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=CREATE' +WebHDFS Retry Policy +------------------------------------- + +WebHDFS supports an optional, configurable retry policy for resilient copy of +large files that could timeout, or copy file between HA clusters that could failover during the copy. + +The following properties control WebHDFS retry and failover policy. + +| Property | Description | Default Value | +|:---- |:---- |:---- +| `dfs.http.client.retry.policy.enabled` | If "true", enable the retry policy of WebHDFS client. If "false", retry policy is turned off. | `false` | +| `dfs.http.client.retry.policy.spec` | Specify a policy of multiple linear random retry for WebHDFS client, e.g. given pairs of number of retries and sleep time (n0, t0), (n1, t1), ..., the first n0 retries sleep t0 milliseconds on average, the following n1 retries sleep t1 milliseconds on average, and so on. | `10000,6,60000,10` | +| `dfs.http.client.failover.max.attempts` | Specify the max number of failover attempts for WebHDFS client in case of network exception. | `15` | +| `dfs.http.client.retry.max.attempts` | Specify the max number of retry attempts for WebHDFS client, if the difference between retried attempts and failovered attempts is larger than the max number of retry attempts, there will be no more retries. | `10` | +| `dfs.http.client.failover.sleep.base.millis` | Specify the base amount of time in milliseconds upon which the exponentially increased sleep time between retries or failovers is calculated for WebHDFS client. | `500` | +| `dfs.http.client.failover.sleep.max.millis` | Specify the upper bound of sleep time in milliseconds between retries or failovers for WebHDFS client. | `15000` | + File and Directory Operations -----------------------------
