I don't think this is an issue. QJM talks with JN using RPC, the default handlers are enough for both DN and JN in my testing.
On Mon, Feb 18, 2013 at 4:02 PM, 谢良 <[email protected]> wrote: > IMHO, it's not a recommented deploy manner to deploy JN and DN into same > nodes. > > Regards, > Liang > ------------------------------ > *发件人:* Azuryy Yu [[email protected]] > *发送时间:* 2013年2月18日 15:56 > *收件人:* [email protected] > *主题:* Re: 答复: some ideas for QJM and NFS > > All JNs are deployed on the same node with DN. > > > On Mon, Feb 18, 2013 at 3:35 PM, 谢良 <[email protected]> wrote: > >> Hi Azuryy, just want to confirm one thing, your JN did not deploy on >> the same machines within DN, right ? >> >> Regards, >> Liang >> ------------------------------ >> *发件人:* Azuryy Yu [[email protected]] >> *发送时间:* 2013年2月18日 15:22 >> *收件人:* [email protected] >> *主题:* Re: some ideas for QJM and NFS >> >> Hi, >> >> I did it on hadoop-2.0.3-alpha without HA as following: >> >> [root@webdm test]# date +%Y-%m-%d_%H:%M:%S; hdfs dfs -put >> testspeed.tar.gz / ; date +%Y-%m-%d_%H:%M:%S >> 2013-02-18_15:20:01 >> 13/02/18 15:20:02 WARN util.NativeCodeLoader: Unable to load >> native-hadoop library for your platform... using builtin-java classes where >> applicable >> 2013-02-18_15:20:30 >> >> so the performance is a little bit better than hadoop-1.0.4. >> >> >> >> On Mon, Feb 18, 2013 at 2:53 PM, Azuryy Yu <[email protected]> wrote: >> >>> Oh, yes, you are right, George. I'll probably do it in the next days. >>> >>> >>> On Mon, Feb 18, 2013 at 2:47 PM, George Datskos < >>> [email protected]> wrote: >>> >>>> Hi Azuryy, >>>> >>>> So you have measurements for hadoop-1.0.4 and hadoop-2.0.3+QJM, but I >>>> think you should also measure hadoop-2.0.3 _wihout_ QJM so you can know for >>>> sure if the performance degrade is actually related to QJM or not. >>>> >>>> >>>> George >>>> >>>> >>>> Hi, >>>> >>>> HarshJ is a good guy, I've seen this JIRA: >>>> https://issues.apache.org/jira/browse/HDFS-4508 >>>> >>>> I have a test cluster hadoop-1.0.4, I've upgrade to >>>> hadoop-2.0.3-alpha. mu cluster is very small, four nodes totally. >>>> >>>> then I did some test on the original Hadoop and new Hadoop, the >>>> testing is very simple: I have a data file with 450MB, I just put it on the >>>> HDFS. >>>> >>>> block size: 128MB, replica: 2 >>>> >>>> the following is the result: >>>> >>>> [root@webdm test]# ll testspeed.tar.gz >>>> -rw-r--r-- 1 root root 452M Feb 18 13:54 testspeed.tar.gz >>>> [root@webdm test]# >>>> >>>> //On the hadoop-1.0.4 >>>> [root@webdm test]# date +%Y-%m-%d_%H:%M:%S; hadoop dfs -put >>>> testspeed.tar.gz / ; date +%Y-%m-%d_%H:%M:%S >>>> 2013-02-18_13:54:24 >>>> Warning: $HADOOP_HOME is deprecated. >>>> 2013-02-18_13:54:58 >>>> >>>> //On the hadoop-2.0.3-alpha with QJM >>>> [root@webdm test]# date +%Y-%m-%d_%H:%M:%S; hdfs dfs -put >>>> testspeed.tar.gz / ; date +%Y-%m-%d_%H:%M:%S >>>> 2013-02-18_14:13:29 >>>> 13/02/18 14:13:30 WARN util.NativeCodeLoader: Unable to load >>>> native-hadoop library for your platform... using builtin-java classes where >>>> applicable >>>> 2013-02-18_14:14:33 >>>> >>>> I do think QJM HA feature affect the performance, because each writer >>>> from QJM, it will do: fence old writer; sync in-progress log; start new log >>>> segment; then write. only if writer received a successful response from a >>>> quorum of JNs, writer finished for this time. >>>> >>>> But NFS HA just write log segment in the local and NFS, when it >>>> receive successful response from NFS, it finished this time. >>>> >>>> So, I just suggest we always keep these two HA features in future, >>>> even in the stable release. which one should be used, which depends on >>>> yourself based on your infrastructure. >>>> >>>> Thanks. >>>> >>>> >>>> >>> >> >
