Hi Experts, In Hadoop-2.0.4, the TeraSort leverage TeraSort#TotalOrderPartitioner as its Partitioner: 'job.setPartitionerClass(TotalOrderPartitioner.class);'. However, seems Yarn did not execute the methods of TeraSort#TotalOrderPartitioner at all. I did some tests to verify it as below:
Test 1: Add some code in the method readPartitions() and setConf() in TeraSort#TotalOrderPartitioner to print some words and write some word to a file. Expected Result: Some words should be printed and wrote into a file Actual Result: No word was printed and wrote into a file at all Test 2: Remove all existing methods in TeraSort#TotalOrderPartitioner, but only remaining some necessary but empty methods in it Expected Result: TeraSort job will ocurr some exception, as the specified Partitioner is not implemented at all Actual Result: TeraSort job completed successfully without any exception Above tests confused me a lot, because seems Yarn never use specified partitioner TeraSort#TotalOrderPartitioner at all during job execution. Any one can help provide the reasons? Thanks very much!
