Hi Fabio, There are test cases in the MapReduce project releases that test setting a custom partitioner and ensuring it works as intended.
But if you still wish to assert/assure self, you should be able to add a LOG statement to your custom Partitioner class's initialization methods, that may indicate its being initialized - so that you can see it on each map task's user logs. There are other ways as well but essentially, there is no "fallback" partitioner in case a user-specified partitioner is not initializable - tasks would fail if you've misconfigured the partitioner. For counters - there are no per-partition counters at the map end (they could end up being too many depending on the number of reducers you have for the job) but there are per-reduce-task input record counters in each reduce task you can use to get the count of number of keys that came into a specific partition. For generally testing your MR code end to end, I recommend using the Apache MRUnit library available at http://incubator.apache.org/mrunit/ On Thu, Feb 16, 2012 at 11:19 PM, <ext-fabio.alme...@nokia.com> wrote: > Hello All, > > I wrote my own partitioner and I would like to see if it’s working. > > By printing the return of method getPartition I could see that the > partitions were different, but were they really working? To answer that I > got the keys that every reducer task processed and that was what I expected. > It seems my partitioner is working properly. But not easy to discover > though. > > Does anyone know if there is an easier way to see if your customized > partitioner is working? For instance, a counter that shows how many > partitioners a map generated or a reducer received? > > Thanks in advance, > > Fabio Almeida -- Harsh J Customer Ops. Engineer Cloudera | http://tiny.cloudera.com/about
