Pages with Specific URLS.

2009-07-23 Thread Zaihan
Hi All, I'm sure I've read somewhere before that URLs that is made like http://www.site.com/categories.asp?cid=25page=9 Can't be crawled. Is that true? Warmest Regards, Zaihan

Re: Pages with Specific URLS.

2009-07-23 Thread reinhard schwab
because? you mean urls which contain a query part? they can be crawled. the default nutch configuration excludes them by this filter rule in conf/crawl-urlfilter.txt # skip URLs containing certain characters as probable queries, etc. -[...@=] Zaihan schrieb: Hi All, I'm sure I've read