Hi Everyone: I launched two experiments for sorting 1 Gb and 10 Gb data with hadoop, on (1) a single machine (2) 5-node clustrer in LAN
The cmd is: bin/hadoop jar hadoop-*-examples.jar sort [-m <#maps>] [-r <#reduces>] <in-dir> <out-dir> the result is shown here: [image: image.png] Mapping shows good scalability. The thing is, reduce takes much longer time than expected in cluster. As far as I know, hadoop sort uses identity function for reduce, which simply output the mapping result in a file. I tested LAN bandwidth, which is ~ 100Mbps, and the average LAN flow during reduce is about 10 Mbps (for sending and receiving). as a result, it appears a bit weird to me here... I am quite new in hadoop thus forgive me for any stupid questions here... Thanks. Best Regards Yours Sincerely Jingwei Lu
