[
https://issues.apache.org/jira/browse/HADOOP-19093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17824028#comment-17824028
]
ASF GitHub Bot commented on HADOOP-19093:
-----------------------------------------
steveloughran commented on PR #6596:
URL: https://github.com/apache/hadoop/pull/6596#issuecomment-1980987839
When `mapreduce.manifest.committer.validate.output` I am going to add
pre-rename validation phase we scan the task attempt to directory tree to make
sure everything is there and with the expected etag. This is simply a
repetition of the tree walk.
I'm going to add rate limiting for all IO within the committer. Currently
there's a rate limiter for the committer rename -in the FS instance, so shared
across all threads- but nothing else.
my plan is in hadoop common to add a new interface `RateLimiterSource` where
you can then ask for the `getRateLimiter(String Path, String operation)`; with
well-known operation names. ABFS Will provide this API and just return the
existing rename limiter everywhere. In S3A I'll add separate limiters for read
and write operations which can be configured for the 5000 and 3500 limits under
a prefix. The WiP bulk delete API #6494 will use this for its rate limiting.
> add load tests for abfs rename resilience
> -----------------------------------------
>
> Key: HADOOP-19093
> URL: https://issues.apache.org/jira/browse/HADOOP-19093
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, test
> Affects Versions: 3.3.6
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> I need a load test to verify that the rename resilience of the manifest
> committer actually works as intended
> * test suite with name ILoadTest* prefix (as with s3)
> * parallel test running with many threads doing many renames
> * verify that rename recovery should be detected
> * and that all renames MUST NOT fail.
> maybe also: metrics for this in fs and doc update.
> Possibly; LogExactlyOnce to warn of load issues
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
