[
https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901998#comment-14901998
]
Kai Zheng commented on HDFS-8968:
---------------------------------
Hi Andrew,
It looks like to be a good idea to have a new module like *hadoop-benchmark*
for benchmark tools in *hadoop-tools*. Such tools should be helpful in a
production system for identifying and verifying some performance metrics, given
a certain cluster environment. This is particularly useful after HDFS-EC is
completed, since then in addition to existing storage policies for storage
types, we'll have various file forms (replication, striping, non-striping EC),
erasure coding policies using different codec algorithms, striping settings and
coder implementations, which will allow user to benchmark and make trade-offs
among these options. Currently the tool implemented in this issue isn't
perfect, and would be a good begining. Our on-going perf test effort found it
works fine. It would be great if you could give it mroe review and confirm how
we should proceed.
Thanks.
> New benchmark throughput tool for striping erasure coding
> ---------------------------------------------------------
>
> Key: HDFS-8968
> URL: https://issues.apache.org/jira/browse/HDFS-8968
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Kai Zheng
> Assignee: Rui Li
> Attachments: HDFS-8968-HDFS-7285.1.patch, HDFS-8968-HDFS-7285.2.patch
>
>
> We need a new benchmark tool to measure the throughput of client writing and
> reading considering cases or factors:
> * 3-replica or striping;
> * write or read, stateful read or positional read;
> * which erasure coder;
> * striping cell size;
> * concurrent readers/writers using processes or threads.
> The tool should be easy to use and better to avoid unnecessary local
> environment impact, like local disk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
