Hi, I did it on hadoop-2.0.3-alpha without HA as following:
[root@webdm test]# date +%Y-%m-%d_%H:%M:%S; hdfs dfs -put testspeed.tar.gz / ; date +%Y-%m-%d_%H:%M:%S 2013-02-18_15:20:01 13/02/18 15:20:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2013-02-18_15:20:30 so the performance is a little bit better than hadoop-1.0.4. On Mon, Feb 18, 2013 at 2:53 PM, Azuryy Yu <azury...@gmail.com> wrote: > Oh, yes, you are right, George. I'll probably do it in the next days. > > > On Mon, Feb 18, 2013 at 2:47 PM, George Datskos < > george.dats...@jp.fujitsu.com> wrote: > >> Hi Azuryy, >> >> So you have measurements for hadoop-1.0.4 and hadoop-2.0.3+QJM, but I >> think you should also measure hadoop-2.0.3 _wihout_ QJM so you can know for >> sure if the performance degrade is actually related to QJM or not. >> >> >> George >> >> >> Hi, >> >> HarshJ is a good guy, I've seen this JIRA: >> https://issues.apache.org/jira/browse/HDFS-4508 >> >> I have a test cluster hadoop-1.0.4, I've upgrade to hadoop-2.0.3-alpha. >> mu cluster is very small, four nodes totally. >> >> then I did some test on the original Hadoop and new Hadoop, the testing >> is very simple: I have a data file with 450MB, I just put it on the HDFS. >> >> block size: 128MB, replica: 2 >> >> the following is the result: >> >> [root@webdm test]# ll testspeed.tar.gz >> -rw-r--r-- 1 root root 452M Feb 18 13:54 testspeed.tar.gz >> [root@webdm test]# >> >> //On the hadoop-1.0.4 >> [root@webdm test]# date +%Y-%m-%d_%H:%M:%S; hadoop dfs -put >> testspeed.tar.gz / ; date +%Y-%m-%d_%H:%M:%S >> 2013-02-18_13:54:24 >> Warning: $HADOOP_HOME is deprecated. >> 2013-02-18_13:54:58 >> >> //On the hadoop-2.0.3-alpha with QJM >> [root@webdm test]# date +%Y-%m-%d_%H:%M:%S; hdfs dfs -put >> testspeed.tar.gz / ; date +%Y-%m-%d_%H:%M:%S >> 2013-02-18_14:13:29 >> 13/02/18 14:13:30 WARN util.NativeCodeLoader: Unable to load >> native-hadoop library for your platform... using builtin-java classes where >> applicable >> 2013-02-18_14:14:33 >> >> I do think QJM HA feature affect the performance, because each writer >> from QJM, it will do: fence old writer; sync in-progress log; start new log >> segment; then write. only if writer received a successful response from a >> quorum of JNs, writer finished for this time. >> >> But NFS HA just write log segment in the local and NFS, when it receive >> successful response from NFS, it finished this time. >> >> So, I just suggest we always keep these two HA features in future, even >> in the stable release. which one should be used, which depends on yourself >> based on your infrastructure. >> >> Thanks. >> >> >> >
