On Tue, May 22, 2012 at 2:00 PM, Markus Jelsma <[email protected]>wrote:
> > -----Original message----- > > From:Bai Shen <[email protected]> > > Sent: Tue 22-May-2012 19:40 > > To: [email protected] > > Subject: URL filtering and normalization > > > > Somehow my crawler started fetching youtube. I'm not really sure why as > I > > have db.ignore.external.links set to true. > > Weird! > > According to this link, it's due to redirects. http://lucene.472066.n3.nabble.com/db-ignore-external-links-true-and-redirects-td615411.html
