> I can only conclude that the way to succeed as a search startup is to > CRAWL DIFFERENTLY. Focus on websites in specific regions, specific > topics, specific data types. Crawl into the corners of websites that > contain interesting nuggets of data (listings, calendars, etc) that > won't ever have a high PageRank. Find a data-niche with an audience > you understand, and hammer away.
Yes, I totally agree that we need more control over the crawl process. -xiong
