[
https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738375#comment-14738375
]
Kai Zheng commented on HDFS-8968:
---------------------------------
Thanks Rui for coming up the tool. Since HDFS-8833 was in, would you rebase it?
Additionally, some comments so far:
1. Maybe you could give your general overall idea about how to benchmark the
read/write throughput as comments in the class header?
2. Regarding the following usage, I thought *seq|pos* does only make sense for
*read*, so the usage would be better to be reorganized.
{code}
+ System.out.println("ErasureCodeBenchmarkThroughput <read|write|gen|clean> "
+ + "<size in MB> <ec|rep> [num clients] [seq|pos]");
{code}
3. The output as below is nice for human readable, meanwhile would be better to
also consider how scripts to post process:
{code}
+ System.out.println(type + " " + df.format(totalDataSizeMB) +
+ " MB data takes " + df.format(duration) + " s. Total throughput: " +
+ df.format(throughput) + " MB/s.");
{code}
4. In *ReadCallable* class:
1) sequenceRead => statefulRead for consistency;
2) please deduplicate the codes in sequenceRead and positionalRead.
> New benchmark throughput tool for striping erasure coding
> ---------------------------------------------------------
>
> Key: HDFS-8968
> URL: https://issues.apache.org/jira/browse/HDFS-8968
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Kai Zheng
> Assignee: Rui Li
> Attachments: HDFS-8968-HDFS-7285.1.patch
>
>
> We need a new benchmark tool to measure the throughput of client writing and
> reading considering cases or factors:
> * 3-replica or striping;
> * write or read, stateful read or positional read;
> * which erasure coder;
> * striping cell size;
> * concurrent readers/writers using processes or threads.
> The tool should be easy to use and better to avoid unnecessary local
> environment impact, like local disk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
