Hi Julien,
Many thanks for your help ! Your answer will save me a lot of hours in
research and test.I have found the plugin and testing it.
I have just one more question. Because I am really noob, I never installed a
plugin in Nutch. If I have understood correctly,
I need the following 3 files:
in the conf I have to add a domain-urlfilter.txt: this file will declare
which TLD will be crawled/fetched (I mean, URLs that don't match the TLD are
not fetched)
in the plugins I have to add a folder called urlfilter-domain with 2
files inside: DomainURLFilter.jar and plugin.xml. For this 2 files I
don't have to change anything in the configuration.
Am I right or should I change something?
Thanks again for your help !
Warm regards
Mauro
Hi Mauro,
Have a look at the domain filter plugin in the SVN version of the code. It
will allow you to filter based on the TLD.
HTH
Julien
--
DigitalPebble Ltd
http://www.digitalpebble.com
2009/3/19 Mauro Vignati vig...@gmail.com
Hi,
I'm testing Nutch and until now everything works fine (ok, some hours
spent
in reading, testing, testing and testing but it's normal.
I have a noob question: I have to crawl websites only within a ccTLD.
In the crawl-urlfilter.txt should I wright so:
# accept hosts in MY.DOMAIN.NAME
+^http://([a-z0-9]*\.)*.ch/
or so
# accept hosts in MY.DOMAIN.NAME
+^http://([a-z0-9]*\.)*ch/
The difference is the dot before the ch ccTLD. I mean, the dot before
the
bracket is already dividing the ccTLD and the name (or the root and a
subdomain) or sould I add one like in the first exemple? In the
installation
guide I can see:
+^http://([a-z0-9]*\.)*apache.org/
Is crawling every subdomain of apache.org (xxx.apache.org) or is
crawling apache.org?
Many thanks for any help
Mauro
