[
https://issues.apache.org/jira/browse/HDFS-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765885#action_12765885
]
Konstantin Shvachko commented on HDFS-708:
------------------------------------------
This is the result of a discussion held last week.
Participated: Dhruba, Nicholas, Rob, Suresh, Todd, shv.
Here is draft description of the tool as it was discussed.
Similar to DFSIO benchmark, the tool should run as a map-reduce job.
Each *map task* iterates choosing:
# a random hdfs operation (write, read, append)
possible extensions may include deletes, renames, write-with-hflush
random generator should be able to generate a configurable percentage of
reads, writes, appends, etc.
# random file. File names may be enumerated as in DFSIO, so choosing a file
means choosing its sequence number.
# random size of data to read, write or append;
the size should be in a configurable range [min, max]
# random block size for write
# random replication for write
# possible random offset for reads
After that the map task performs chosen operation with chosen parameters.
Reads must *verify read data* that it is exactly the same as it was written.
In order to do that write operation should store a seed for a standard
pseudo-random generator.
The seed can be either encoded in the file name or stored inside the file as a
header.
Appends may use the same seed for data generation.
An alternative would be to view a file as a set of records, where each record
starts with a header containing the seed. That way each append may choose an
independent seed.
*Job completion* can be defined by one of the following conditions:
- run T hours. In that case map tasks iterate until their elapsed time exceeds
T.
- run N operations. Each map performs N / numMaps operations and stops.
- terminate on first error flag. This may be useful for investigation purposes.
The cluster state is as close to the failure condition as it gets.
*Error reporting.* Errors are reported by maps as its output.
Map output also should include reasonable stats:
- number of operations performed in each category
- errors, % of errors;
The *reduce task* combines all error messages and stats, writes the results
into HDFS.
Potential extension with Fault Injection.
For example, one may choose to intentionally corrupt data while writing, and
then make sure it is corrupt on read.
> A stress-test tool for HDFS.
> ----------------------------
>
> Key: HDFS-708
> URL: https://issues.apache.org/jira/browse/HDFS-708
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: test, tools
> Affects Versions: 0.22.0
> Reporter: Konstantin Shvachko
> Fix For: 0.22.0
>
>
> It would be good to have a tool for automatic stress testing HDFS, which
> would provide IO-intensive load on HDFS cluster.
> The idea is to start the tool, let it run overnight, and then be able to
> analyze possible failures.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
