I need help determining hardware specs for crawling 100 sites with 1000 pages each. Regular re-crawl is needed probably every day (maybe even more often). So will one server meet these crawling requirements (only crawling, searching will be handled by other machine)? If so, what hardware specification would be recommended (how much Ram, CPU's, hard disk space)?
Thanks,
Tomislav
