bbeaudreault opened a new pull request #3271: URL: https://github.com/apache/hadoop/pull/3271
Per https://issues.apache.org/jira/browse/HDFS-16155, we would like the ability to customize the backoff strategy when BlockMissingException occurs. This can happen when the balancer moves blocks, and in low latency clusters the existing backoff is too conservative. Drastically reducing the existing window base config would help but expose the namenode to a potential DDOS if many blocks became missing, because the current backoff would grow slowly. Adding a configurable exponential component allows for aggressive early retries that back off quickly enough to mitigate stampeding herds. We make the backoff configurable by adding two new configs: - `dfs.client.retry.window.multiplier`: defaults to 1 to preserve existing behavior. Increasing this can result in a steeper backoff curve when desired - `dfs.client.retry.window.max`: defaults to Int.MAX to preserve existing behavior. Decreasing this can help put a ceiling on exponential backoffs that could quickly grow to effectively unlimited levels. As described, the default behavior is maintained and I've added a test case to verify that. Someone looking for a more aggressive initial retry that backs off quickly in case of continuous failure could try setting `window.base` to 10, `window.multiplier` to 5, and `window.max` to 10000. This would result in a quick initial retry of max 50ms, but quickly backoff to a few seconds within 3 retries. In order to improve the testability of this feature, I pulled out the existing refetchLocations retry configs into a FetchBlockLocationsRetryer class. I also improved the readability of the comment describing the backoff strategy, and fully tested the new retryer in TestFetchBlockLocationsRetryer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
