What kind of patterns do the url have ??This is my wild guess: you have a limited (surely less than 20) set of domains for the complete set of urls. HashPartitioner , which partitions the urls based on domains is the class to look at.And if this is true, you will have to write a custom Partitioner
Yes, this is indeed the case. Thanks for the tip.
