"Nevertheless, perhaps you'll still post the answer here so that others who come along later can benefit from your experience in the same way that you benefited from reading whatever page you found (even if you didn't benefit from my suggestions...). "
Your funny :) Perhaps you should take your own advice. My guess is that the google search you described will return different results in a few months. Your first post won't benefit those who "come along later". "The spider hangs (waits for username and pass) when I hit a page that requires .htaccess authentication." I was using urllib. urllib2 doesn't have this problem. Simply switching urllib to urllib2 fixed the problem -- http://mail.python.org/mailman/listinfo/python-list
