Makes sense to not write out unnecessary data. There's also a case where DefaultSorter writes out empty files when there's no output. Have to be careful about the configuration which enables empty partitions via events.
On Thu, Feb 2, 2017 at 3:57 PM, Kuhu Shukla <[email protected]> wrote: > Hello all, > > In DefaultSorter, for the OrderedPartitioned case, is the inclusion of > empty partition headers and segments in the file.out.index and file.out > needed? > > In the Unordered case, we prune out the empty partitions in the writer and > it might be worth doing the same in the Ordered case unless this behavior > was intentional. Would appreciate some comments on this. > > Thanks and Regards, > Kuhu >
