By default, nutch skips URLs containing certain characters. To change it, open regex-urlfilter.txt, comment out the following line.
# skip URLs containing certain characters as probable queries, etc. -[...@=] On Sun, 2009-09-27 at 03:55 +0800, Shawn Young wrote: > Hi all, > > I have a question, if a web page's url likes > http://www.test.com/test.php?gid=1111111 ,how can nutch crawl its content? > I've had a try, but it seems that nutch ignores the query string > 'gid=1111111' of the url. > > Can someone helps me? > Thanks. >
