- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: Dan Waterloo Subject: Re: limit index to domain, text files with specific extensions, i.e. .lib, .spi, etc
Hi, Can you clarify what causes a website to be crawled, and what causes a url to be inserted into the database (indexed?)? I'd like to crawl the following domain: http://www.domain.com and only insert urls (from www.domain.com) into the database (index) for urls that end in .lib or .spi Sorry I'm a bit confused about this. Thank you for the help. Dan - - - - - - - - - - - - - - - - - - - - - - - - - - - - Read the full topic here: http://www.dataparksearch.org/cgi-bin/simpleforum.cgi?fid=02;topic_id=1226881847
