Cha,
Are you updating regex-urlfilter.txt or crawl-urlfilter.txt ?
If you use nutch crawl, you need to update crawl-urlfilter.txt
If you use whole web crawling you need to use regex-urlfilter.txt
Jason
On Mar 22, 2007, at 10:26 PM, cha wrote:
Hi,
I have try the filters u have provided ..but still its not working..
i have enable urlfilter-regex plugin in your configuration as well..
i cant find out whats the problem is.
Cheers,
cha
Jason Culverhouse wrote:
Cha,
You want something like this
-^http://([a-z0-9]*\.)*example.com/stores/[^/]+/(merch-cats-pg|merch-
cats|merch)/
Your regex fails to match because that last segment '/merch-cats-pg
\.*' requires a literal .
So it matches http://www.example.com/stores/abcd/merch-cats-pg.
not http://www.example.com/stores/abcd/merch-cats-pg/
You could also just change that to '/merch-cats-pg.*'
Get a copy of Mastering Regular Expressions By Jeffrey E. F. Friedl
http://www.oreilly.com/catalog/regex3/index.html
Jason
On Mar 21, 2007, at 8:37 AM, cha wrote:
Hi,
I want to ignore the following urls from crawling
for eg.
http://www.example.com/stores/abcd/merch-cats-pg/abcd.*
http://www.example.com/stores/abcd/merch-cats/abcd.*
http://www.example.com/stores/abcd/merch/abd.*
I have used regex-urlfilter.txt file and negate the following urls:
# skip URLs containing certain characters as probable queries, etc.
[EMAIL PROTECTED]
-http://([a-z0-9]*\.)*example.com/stores/.*/merch-cats-pg\.*
-http://([a-z0-9]*\.)*example.com/stores/.*/merch-cats\.*
-http://([a-z0-9]*\.)*example.com/stores/.*/merch\.*
The above filters still don't filters all the urls.
is there any way to solve this..any alternatives??
Awaiting,
Cha
--
View this message in context: http://www.nabble.com/help-needed-%3A-
filters-in-regex-urlfilter.txt-tf3441531.html#a9596460
Sent from the Nutch - User mailing list archive at Nabble.com.
--
View this message in context: http://www.nabble.com/help-needed-%3A-
filters-in-regex-urlfilter.txt-tf3441531.html#a9629082
Sent from the Nutch - User mailing list archive at Nabble.com.
