Fetching problems: Nutch 0.9 Hung Threads

[Luca Rondanini]

Hi all,

First of all....I've read all the posts regarding this problem in the 
mailing list!! :)

I'm try to index more than 200k documents. I'm reading those documents 
through an nfs mount partition. Everything seems fine till we arrive at 
40k-50k documents....then the fetcher fails with the error "Hung Threads"!!

These are the configurations that i've tried:

1)      topN=20.000
        fetcher.threads=10
        ulimit -n=1024
        MergeFactor=20
        file.limit=1M

----> Hung Threads

2)      topN=5000
        fetcher.threads=10
        ulimit -n=1024
        MergeFactor=20
        file.limit=1M

----> Hung Threads


3)      topN=5000
        fetcher.threads=5
        ulimit -n=1024
        MergeFactor=20
        file.limit=1M

----> Too many open file


4)      topN=5000
        fetcher.threads=5
        ulimit -n=4096
        MergeFactor=10
        file.limit=1M

----> Hung Threads



Can anyone please give me a clue as to what is going on?!?
Thanks,
Luca