It can work!First thanks, the urls's content includes news.yahoo.com <http://news.yahoo.com> game.yahoo.com <http://game.yahoo.com> blog.yahoo.com <http://blog.yahoo.com>
right? 2005/7/4, Ilia S. Yatsenko <[EMAIL PROTECTED]>: > > Try this > > +^http://([a-z0-9\.\-]*)\.yahoo\.com/ > > I hope it help you :) > > -----Original Message----- > From: zhangjin (JIRA) [mailto:[EMAIL PROTECTED] > Sent: Monday, July 04, 2005 6:42 AM > To: [email protected] > Subject: [jira] Created: (NUTCH-67) I want crawl the websites including > news.yahoo.com <http://news.yahoo.com>,game.yahoo.com<http://game.yahoo.com> > ,blog.yahoo.com <http://blog.yahoo.com>,etc! > > I want crawl the websites including > news.yahoo.com <http://news.yahoo.com>,game.yahoo.com<http://game.yahoo.com> > ,blog.yahoo.com <http://blog.yahoo.com>,etc! > > ---------------------------------------------------------------------------- > ----------- > > Key: NUTCH-67 > URL: http://issues.apache.org/jira/browse/NUTCH-67 > Project: Nutch > Type: Wish > Components: fetcher > Environment: Windows 2000,weblogic > Reporter: zhangjin > > how do I config them in the crawl-urlfilter.txt? I config them below,but > it is not successful. > # The url filter file used by the crawl command. > > # Better for intranet crawling. > # Be sure to change MY.DOMAIN.NAME <http://MY.DOMAIN.NAME> to your domain > name. > > # Each non-comment, non-blank line contains a regular expression > # prefixed by '+' or '-'. The first matching pattern in the file > # determines whether a URL is included or ignored. If no pattern > # matches, the URL is ignored. > > # skip file:, ftp:, & mailto: urls > -^(file|ftp|mailto): > > # skip image and other suffixes we can't yet parse > > -\.(gif|GIF|jpg|JPG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|M > OV|exe)$ > > # skip URLs containing certain characters as probable queries, etc. > [EMAIL PROTECTED] > > # accept hosts in MY.DOMAIN.NAME <http://MY.DOMAIN.NAME> > #+^http://([a-z0-9]*\.)*MY.DOMAIN.NAME/ <http://MY.DOMAIN.NAME/> > +^http://([a-z0-9]*\.)*yahoo.com/ <http://yahoo.com/> > # skip everything else > #-. > but It can not work, and can not crawl the domain name > (DOMAIN.NAME<http://DOMAIN.NAME> > ) > inluding news.yahoo.com > <http://news.yahoo.com>,game.yahoo.com<http://game.yahoo.com> > ,blog.yahoo.com <http://blog.yahoo.com> > why? > > -- > This message is automatically generated by JIRA. > - > If you think it was sent incorrectly contact one of the administrators: > http://issues.apache.org/jira/secure/Administrators.jspa > - > For more information on JIRA, see: > http://www.atlassian.com/software/jira > > -- TEL 0512-68251233-6966 MSN:[EMAIL PROTECTED] Mail:[EMAIL PROTECTED] QQ:58624951 BenQ.com <http://BenQ.com> 268 Shishan Road, New District, Suzhou, China
