Hi Sebastian,
I indeed changed the runtime/local/conf/regex-urlfilter.txt, so that's not
the problem.
After some unfruitful Googling, I turned myself to some on-line java regex
testers, finally I've been able to solve this problem.
It seems for regexes containing numbers to work, you need provide a 'full
match' regex
in the regex-urlfilter.txt file.
So by changing
-pn=[0-9]{4,}$
to
-^http://far\.boo\.com/f\?kw=SomeTopic&pn=[0-9]{4,}$
Nutch worked as expected.
2015-01-14 3:13 GMT+08:00 Sebastian Nagel <[email protected]>:
> Hi,
>
> the regular expression looks good.
> Which conf/regex-urlfilter.txt has been changed?
> runtime/local/conf/regex-urlfilter.txt ?
> If
> conf/regex-urlfilter.txt is changed
> you need to run "ant runtime" again
> to install the configuration changes
> into runtime/local/conf.
> For distributed mode you need to rebuild
> and deploy after any configuration change
> because configuration files are included
> in the job file.
>
> Sebastian
>
> On 01/13/2015 06:16 AM, fxmy wang wrote:
> > Hi Nutch users,
> >
> >
> > We are trying to crawl a forum site with the help of Nutch-2.2.1.
> >
> > The URLs are like far.boo.com/f?kw=SomeTopic&pn=150
> > where pn means PageNumber.
> >
> > The goal, is to filter out those old posts, say I want all those pn>1000
> > posts filtered.
> >
> > So in conf/regex-urlfilter.txt I added this above the '# accept anything
> > else' line.
> >
> > -[*!@] # skip certain queries
> > -pn=[0-9]{4,}$ # filter out pn>1000
> > +. # accept anything else
> >
> > And... no effect :(
> > After some generate-fetch-parse-updatedb circle the URL
> > far.boo.com/f?kw=SomeTopic&pn=649800 still got fetched.
> >
> > To verify furthermore I run the command below
> > bin/nutch plugin urlfilter-regex
> > org.apache.nutch.urlfilter.regex.RegexURLFilter [0]
> > and pasted 'far.boo.com/f?kw=SomeTopic&pn=649800' in, the output is
> > +far.boo.com/f?kw=SomeTopic&pn=649800
> > Seems nutch didn't filter it out.
> >
> > What is the proper way to deal with numbers in URLs?
> > Did I do something wrong?
> > Any advice will be very appreciated.
> >
> > ----------------------------------------------------------------------
> > [0]http://www.mail-archive.com/user%40nutch.apache.org/msg09536.html
> > ----------------------------------------------------------------------
> >
> > BR, fxmy
> >
>
>
