Hi All,
So I've built up over 100 segments + indexes and now when I run:
new IndexMerger(fs, indexesSubDirs.toArray(new File[] {}), index,
tmpDir).merge();
I'm getting the dreaded: "Too many open files" error.
I've checked my system settings for file-max:
$ cat /proc/sys/fs/file-nr
2677 1945 478412
$ cat /proc/sys/fs/file-max
478412
What would be the best way to work around (or fix) this. Merging 10 indexes
at a time and then merging the results down until I get just one index?
What about the dedup process. It seems to be able to manage the 100+ indexes
fine, but if I switch the process and merge the indexes first and then
remove dupes, I think it may speed up the process. Ideas?
Thanks!
Ben
