Hi devs,
Does any one tried sorting with multiple reducers? I seem to hit this when
trying to implement the HFile bulk loader.
You can reproduce this as follow:
1. modify SortIT to run multiple reducers
2. run SortIT#testWritableSortDesc
I got exception:
java.lang.IllegalArgumentException: Can't read partitions file
at
org.apache.crunch.lib.sort.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:81)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:677)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:756)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:214)
Caused by: java.io.IOException: Wrong number of partitions in keyset
at
org.apache.crunch.lib.sort.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:77)
... 6 more
It seems that TotalOrderPartitioner does not receive the correct number of
reducers. Any ideas?
Thanks,
Chao