Hi,
We run into a problem with nutch using
MapFileOutputFormat#getReaders and getEntry.
In detail this happens until summary generation where we open for
each segment as much readers as much parts (part-0000 to part-n) we
have.
Having 80 tasktracker and 80 segments means:
80 x 80 x 4 (parseData, parseText, content, crawl). A search server
also needs to open as much files as required for the index searcher.
So the problem is a FileNotFoundException, (Too many open files).
Opening and closing Readers for each Detail makes no sense. We may
can limit the number of readers somehow and close the readers that
wasn't used since the longest time.
But I'm not that happy with this solution, so any thoughts how we can
solve this problem in general?
Thanks.
Stefan
P.S. We also note that the .crc also double the number of open files.
