>If you had to write a program to spider a website, what programming >language/OS/Web server software/DB/ would you use? > >I wrote one using cfhttp, but it is short of slow. Just cfhttp byitself is >slow. Looking for something with more kick to it.
Actually HTTP itself is kind of slow. It was designed for compatiblity and ease of development, not speed. >I was thinking along the lines of Linux/Apache/MySQL/Phthon or Perl... I can't see how changing web server or OS will really help you in any way. Your database does matter but the design of it matter most. You can destroy the performance of the best DBMS with a crappy schema. As for language everything you've mentioned are interperated languages. They're all going to be a little slow. If you want speed you need to go to compiled languages like Java or C++. If you want super fast then you still need to go to Assembler. ;^) If you want to stay interperated then you need to evaluate each based on their strengths and choose the one with the best balance. For example PERL is very good with Text Parsing, CF is very good with Database access and so forth. Personally I think you'd get a bigger bang for the buck by picking the language that you're best with and optimizing your code. Switching languages rarely make anything faster since you're producing amatuer code in the new language. Jim Davis ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Check out the new features and enhancements in the latest product release - download the "What's New PDF" now http://download.macromedia.com/pub/labs/coldfusion/cf8_beta_whatsnew_052907.pdf Archive: http://www.houseoffusion.com/groups/CF-Community/message.cfm/messageid:239516 Subscription: http://www.houseoffusion.com/groups/CF-Community/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.5
