Hi! I am now crawling the internet in local mode in parallel with up to 10 instances on 3 computers. would it pay off for me to put a hadoop cluster on top of the 3 servers.
1.) a server would not be integrated directly into the crawl process as a master. 2.) can I run multiple crawl jobs on one server? Thanks
