Any ideas? I've exhausted all the ends I know to take a look at.
matt
--- Begin Message ---
Here is my crawl-urlfilter.txt file.
Matt
# The url filter file used by the crawl command.
# Better for intranet crawling.
# Be sure to change MY.DOMAIN.NAME to your domain name.
# Each non-comment, non-blank line contains a regular expression
# prefixed by '+' or '-'. The first matching pattern in the file
# determines whether a URL is included or ignored. If no pattern
# matches, the URL is ignored.
# skip file:, ftp:, & mailto: urls
-^(file|ftp|mailto):
# skip image and other suffixes we can't yet parse
-\.(gif|GIF|jpg|JPG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe|png|PNG)$
# skip URLs containing certain characters as probable queries, etc.
[EMAIL PROTECTED]
# accept hosts in MY.DOMAIN.NAME
+^http://([a-z0-9]*\.)*corp.mydomain.com/
# skip everything else
-.
Stefan Neufeind wrote:
Matthew Holt wrote:
Just fyi,.. both of the sites I am trying to crawl are under the same
domain. The sub-domains just differ. Works for one, the other it o
nly appears to fetch 6 or so pages then doesn't fetch anymore. Do you
need any more information to solve the problem? I've tried everything
and havent' had any luck.. Thanks.
What does your crawl-urlfilter.txt look like?
Stefan
--- End Message ---
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
