[
https://issues.apache.org/jira/browse/HADOOP-15020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhoutai.zt updated HADOOP-15020:
--------------------------------
Description:
When benchmark NameNode performance with NNBench, I start with pseudo
distributed deploy. Everything goes well with "-maps 1". BUT with -maps N (n>1)
and -operation create_write, many exceptions meet during the benchmark.
Hostname is part of the file path, which can differentiate hosts. With more
than two map tasks run on the same host, more than two map tasks may operate on
the same file, leading to exceptions.
17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: AL Total #1: 84
17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: AL Total #2: 43
17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: TPS Total (ms):
2570
17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: Longest Map Time (ms):
814.0
17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: Late maps: 0
17/11/07 15:22:32 INFO hdfs.NNBench: {color:red}RAW DATA: # of
exceptions: 3000{color}
2017-11-07 14:54:08,082 INFO org.apache.hadoop.hdfs.NNBench: Exception recorded
in op: Create/Write/Close
2017-11-07 14:54:08,082 INFO org.apache.hadoop.hdfs.NNBench: Exception recorded
in op: Create/Write/Close
2017-11-07 14:54:08,083 INFO org.apache.hadoop.hdfs.NNBench: Exception recorded
in op: Create/Write/Close
was:
When benchmark NameNode performance with NNBench. I start with pseudo
distributed deploy. Everything goes well with "-maps 1". BUT with -maps N (n>1)
and -operation create_write, many exceptions meet during the benchmark.
Hostname is part of the file path, which can differentiate hosts. With more
than two map tasks run on the same host, more than two map tasks may operate on
the same file, leading to exceptions.
17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: AL Total #1: 84
17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: AL Total #2: 43
17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: TPS Total (ms):
2570
17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: Longest Map Time (ms):
814.0
17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: Late maps: 0
17/11/07 15:22:32 INFO hdfs.NNBench: {color:red}RAW DATA: # of
exceptions: 3000{color}
2017-11-07 14:54:08,082 INFO org.apache.hadoop.hdfs.NNBench: Exception recorded
in op: Create/Write/Close
2017-11-07 14:54:08,082 INFO org.apache.hadoop.hdfs.NNBench: Exception recorded
in op: Create/Write/Close
2017-11-07 14:54:08,083 INFO org.apache.hadoop.hdfs.NNBench: Exception recorded
in op: Create/Write/Close
> NNBench not support run more than one map task on the same host
> ---------------------------------------------------------------
>
> Key: HADOOP-15020
> URL: https://issues.apache.org/jira/browse/HADOOP-15020
> Project: Hadoop Common
> Issue Type: Improvement
> Components: benchmarks
> Affects Versions: 2.7.2
> Environment: Hadoop 2.7.2
> Reporter: zhoutai.zt
> Priority: Minor
>
> When benchmark NameNode performance with NNBench, I start with pseudo
> distributed deploy. Everything goes well with "-maps 1". BUT with -maps N
> (n>1) and -operation create_write, many exceptions meet during the benchmark.
> Hostname is part of the file path, which can differentiate hosts. With more
> than two map tasks run on the same host, more than two map tasks may operate
> on the same file, leading to exceptions.
> 17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: AL Total #1:
> 84
> 17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: AL Total #2:
> 43
> 17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: TPS Total (ms):
> 2570
> 17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: Longest Map Time (ms):
> 814.0
> 17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: Late maps: 0
> 17/11/07 15:22:32 INFO hdfs.NNBench: {color:red}RAW DATA: # of
> exceptions: 3000{color}
> 2017-11-07 14:54:08,082 INFO org.apache.hadoop.hdfs.NNBench: Exception
> recorded in op: Create/Write/Close
> 2017-11-07 14:54:08,082 INFO org.apache.hadoop.hdfs.NNBench: Exception
> recorded in op: Create/Write/Close
> 2017-11-07 14:54:08,083 INFO org.apache.hadoop.hdfs.NNBench: Exception
> recorded in op: Create/Write/Close
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
