DNS names use a bit richer character set than you have specified. In
this case I suspect that by specifying the inverse you will get
closer to what you intend:
+http://[^/]*\.(com|org|net|biz|mil|us|info|cc)/
-.*
A few days back there was a request for features that can be handled
by the RegexURLFilter. If you have any uncertainties about the regex
syntax supported, it is easy to put some test rules in a customized
conf directory and test them as follows:
$ #setup your shell environment as required. e.g. for bash:
$ export NUTCH_CONF_DIR=/Users/rcp/project/nutch/test/conf
$ export NUTCH_HOME=/Users/rcp/project/nutch/nutch-rcp
$ #next run
$ $NUTCH_HOME/bin/nutch net.nutch.net.RegexURLFilter
# you can now type in urls, one per line and see the results
http://www.somedomain.com/
+http://www.somedomain.com/
http://www.somedomain.co.uk/
-http://www.somedomain.co.uk/
of course they don't speak "American English" in the UK :-)
;rob
--
Robert C. Pettengill, Ph.D.
[EMAIL PROTECTED]
Questions about petroleum?
Goto: http://AskAboutOil.com/
Need help implementing search?
Goto: http://MesaVida.com/
On 2005, Aug 8, at 2:27 PM, Piotr Kosiorowski wrote:
Hello,
I am not sure which way is better but I would look for "dot":
orginal >http://([a-z0-9]*\.)*.(com|org|net|biz|edu|biz|mil|us|info|
cc)/
modified>http://([a-z0-9]*\.)*(com|org|net|biz|edu|biz|mil|us|info|
cc)/
In my opinion "dot" before com,org etc is already included in ([a-
z0-9]*\.)* and additional one (not escaped) means any character so
it would match eg:
http://www.abc.xcom/
but not
http://www.abc.com/.
Regards,
P.
Chirag Chaman wrote:
Here's a better way
http://([a-z0-9]*\.)*.(com|org|net|biz|edu|biz|mil|us|info|cc)/
FYI, this will not remove non-English sites -- but international
sites that
follow the two-letter convention.
CC-
-----Original Message-----
From: Jay Pound [mailto:[EMAIL PROTECTED] Sent:
Monday, August 08, 2005 2:37 PM
To: [email protected]; [email protected]
Subject: regex-url filter
I would like a confirmation from someone that this will work, I've
edited
the regex filter in hopes to weed out non-english sites from my
search
results, I'll be testing pruning on my current 40mil index to see
if it
works there, or maybe there is a way to set the search to return only
english results, but I'm trying it this way now, is this the right
way to
add just extensions without sites?
I'll try it soon but just wanted to not waste my time if its not
correct!!!
Thanks,
-Jay Pound
# The default url filter.
# Better for whole-internet crawling.
# Each non-comment, non-blank line contains a regular expression
# prefixed by '+' or '-'. The first matching pattern in the file
# determines whether a URL is included or ignored. If no pattern
# matches, the URL is ignored.
# skip file: ftp: and mailto: urls
-^(file|ftp|mailto):
# skip image and other suffixes we can't yet parse
-\.(gif|GIF|jpg|JPG|ico|ICO|css|sit|eps|wmf|rtf|zip|ppt|mpg|xls|gz|
rpm|tgz|m
ov|MOV|exe)$
# skip URLs containing certain characters as probable queries, etc.
[EMAIL PROTECTED]
# accept US only sites
+^http://([a-z0-9]*\.)*.com/
+^http://([a-z0-9]*\.)*.org/
+^http://([a-z0-9]*\.)*.edu/
+^http://([a-z0-9]*\.)*.net/
+^http://([a-z0-9]*\.)*.mil/
+^http://([a-z0-9]*\.)*.us/
+^http://([a-z0-9]*\.)*.info/
+^http://([a-z0-9]*\.)*.cc/
+^http://([a-z0-9]*\.)*.biz/
-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
