When someone uses the crawl method with, lets say 100 hundred sites, you establish your url filters to allow only those sites. In the first run, just those 100 sites are indexed, but in subsequent runs, the outlinks are indexed too, together with other hops of the seeds sites. This is fine, as someone gets some really good related sites, but if those sites do not comply with the url filter, how come are they indexed?
- URL filters and outlinks carmmello
- Re: URL filters and outlinks Byron Miller
- url filter prefix Gal Nitzan
