Thanks for your tips Xiao.
I am currently trying to use Nutch on a single machine so I didn't change
any Hadoop related configurations. Or should I? I assume Nutch sets the
default number of map/reduce task to 1. Is this true?
If I have to change the Hadoop mapreduce configurations in a single machine
environment, Could anyone help to tell me which is the file I should change?
I tried to specify the number of map and reduce task numbers but it didn't
work out.
Below is the configurations I added into mapred-site.xml:
<property>
<name>mapred.map.tasks</name>
<value>1</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>1</value>
</property>
Thanks,
-- Weiwei
On Sat, Nov 27, 2010 at 7:36 AM, xiao yang <[email protected]> wrote:
> Hi, Weiwei
>
> What about the configuration of Hadoop?
> Maybe there're 10 processes with 1 thread each.
>
> Thanks!
> Xiao
>
> On 11/27/10, Weiwei Xiong <[email protected]> wrote:
> > Hi All,
> >
> > I'am trying to use nutch to crawl some websites but got a full CPU usage
> > after it got to depth 2 or 3. I couldn't do anything with the machine but
> > have to stop the crawling. This happened even when I configured to use
> only
> > ONE fetcher thread.
> > One weird thing I noticed is that the number of threads keeps growing
> after
> > running sometime.
> >
> > Does anyone have any hint to solve this problem?
> >
> > Thanks.
> > -- ww
> >
>
