Andrzej: Thank you for your response to my comments. The reason I said there may be bug in the fetcher is that in our case there was no JVM crash or OOM Exception during the fetch and the fetch process was successful by reading the log. file. So I cannot tell what caused the truncation (Unexpected EOF exception in the reader)
The problem with the MapFile is not that the performance drops, instead, it
simply hangs on a deadlock ( I looked at the thread dump). I do not
understand why a segread would need a write lock on the seg.
That's strange, indeed... Could you perhaps do a thread dump (^E or similar signal on Linux)? Are you sure you have killed the other process (fetcher) that was writing to this segment?
Your proposed manual step to fix the index may not work simply because UnExpected EOF is in the fetch output (seq file), not the index.
It will work, the "index" file I'm speaking here of, is not a Lucene index, it's one of the two files that constitute a MapFile ("data" and "index"). There is a pair of such files inside every segment's subdirectories, among others in fetcher_output.
-- Best regards, Andrzej Bialecki ___. ___ ___ ___ _ _ __________________________________ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com
