Re: Fetching problems: Nutch 0.9 Hung Threads

[Luca Rondanini]

Hi,

Forgot to say the the problem doesn't occur if I crawl the same files on 
the local file system.

Thanks!

Luca

Luca Rondanini wrote:
Hi all,

First of all....I've read all the posts regarding this problem in the 
mailing list!! :)

I'm try to index more than 200k documents. I'm reading those documents 
through an nfs mount partition. Everything seems fine till we arrive at 
40k-50k documents....then the fetcher fails with the error "Hung Threads"!!

These are the configurations that i've tried:

1)    topN=20.000
    fetcher.threads=10
    ulimit -n=1024
    MergeFactor=20
    file.limit=1M

----> Hung Threads

2)    topN=5000
    fetcher.threads=10
    ulimit -n=1024
    MergeFactor=20
    file.limit=1M

----> Hung Threads


3)    topN=5000
    fetcher.threads=5
    ulimit -n=1024
    MergeFactor=20
    file.limit=1M

----> Too many open file


4)    topN=5000
    fetcher.threads=5
    ulimit -n=4096
    MergeFactor=10
    file.limit=1M

----> Hung Threads



Can anyone please give me a clue as to what is going on?!?
Thanks,
Luca