It works! Wow, I feel like I'm really starting to learn how nutch works.
Okay one more newbie question. I would like to crawl a list of domains,
but I would like crawling limited to just those domains. When I first
played around with nutch in a localsetup I just set the following
property in nutch-site.xml:
<property>
<name>urlfilter.prefix.file</name>
<value>urls.txt</value>
<description>Name of file on CLASSPATH containing url prefixes
used by urlfilter-prefix (PrefixURLFilter) plugin.</description>
</property>
Can I do this in a mapred system? Also how does the fetching work, does
each new round of generate crawldb fetch the next "level" of urls? I'm
wondering the best way to put the crawl/index system on autopilot so
pages are crawled and updated regularly.
Thanks again for you help Stefan.
Mike
On Fri, 2005-12-16 at 20:12 +0100, Stefan Groschupf wrote:
> looks good, now just install tomcat.
> uncompress your nutch-XXX.war file a folder called ROOT.war with
> unzip and change this in ROOT.war/WEB-INF/classes also.
> Than you can simply copy this folder into TOMCAT/webapps, that's it.
>
>
> Am 16.12.2005 um 20:09 schrieb Michael Taggart:
>
> > Sorry Stefan, I am so used to typing usr that I wrote my email
> > incorrectly. Here is exactly what is in my nutch-site.xml:
> >
> > <property>
> > <name>searcher.dir</name>
> > <value>/user/root</value>
> > <description>
> > Path to root of index directories. This directory is searched (in
> > order) for either the file search-servers.txt, containing a list of
> > distributed search servers, or the directory "index" containing
> > merged indexes, or the directory "segments" containing segment
> > indexes.
> > </description>
> > </property>
> >
> > On Fri, 2005-12-16 at 20:00 +0100, Stefan Groschupf wrote:
> >>> /user/root/urls <dir>
> >>>
> >>> configured with searcher.dir as /usr/root
> >> A typo? usr/root !=user/root !
> >>
> >>> However, I have seen people
> >>> talk of a web-inf folder but I can't find one on my system.
> >> may this helps:
> >> http://wiki.media-style.com/display/nutchDocu/install+user+interface
> >>
> >> part of the webapplication you had deployed to your tomcat:
> >> TOMCAT/webapps/ROOT/WEB-INF/classes/nutch-site.xml should b the
> >> location.
> >>
> >> Stefan
> >
>
