The de-duplication algorithm should be abstracted and separated into a utility method/class - currently both DeleteDuplicates and SegmentMergeTool perform de-duplication, but I'm afraid that each follows a slightly different, hardcoded routine...
Perhaps the IndexedDoc nested class from DeleteDuplicates.java could be used as a basis for this? The compareTo() method would need to be implemented, and the compare() method in each of the comparators, since I only implemented the optimized binary version.
Doug
