Folks, I'm looking to use Scrapy to simplify some web scrapers that I have previously written.
But I'm curious if anyone else has had any luck in using Scrapy with forums based off of SMF forums? (e.g. http://www.simplemachines.org/community/). The main kicker, is that they require the password to be hashed when sent back to them. <form id="guest_form" action="http://www.simplemachines.org/community/index.php?action=login2"; method="post" accept-charset="UTF-8" *onsubmit="hashLoginPassword(this, '52c8d7f4468b42ca44664e302e2a5865');">* <div class="info">Please <a href="http://www.simplemachines.org/community/index.php?action=login";>login</a> or <a href="http://www.simplemachines.org/community/index.php?action=register";>register</a>.</div> <input type="text" name="user" size="10" class="input_text"> <input type="password" name="passwrd" size="10" class="input_password"> <select name="cookielength"> <option value="60">1 Hour</option> <option value="1440">1 Day</option> <option value="10080">1 Week</option> <option value="43200">1 Month</option> <option value="-1" selected="selected">Forever</option> </select> <input type="submit" value="Login" class="button_submit"><br> <div class="info">Login with username, password and session length</div> * <input type="hidden" value="52c8d7f4468b42ca44664e302e2a5865" name="b259f60">* * <input type="hidden" name="hash_passwrd" value="">* </form> Which uses the following Javascript, function hashLoginPassword(doForm, cur_session_id) { // Compatibility. if (cur_session_id == null) cur_session_id = smf_session_id; if (typeof(hex_sha1) == 'undefined') return; // Are they using an email address? if (doForm.user.value.indexOf('@') != -1) return; // Unless the browser is Opera, the password will not save properly. if (!('opera' in window)) doForm.passwrd.autocomplete = 'off'; doForm.hash_passwrd.value = hex_sha1(hex_sha1(doForm.user.value.php_to8bit().php_strtolower() + doForm.passwrd.value.php_to8bit()) + cur_session_id); // It looks nicer to fill it with asterisks, but Firefox will try to save that. if (is_ff != -1) doForm.passwrd.value = ''; else doForm.passwrd.value = doForm.passwrd.value.replace(/./g, '*'); } Has anyone had success with SMF Forums? And example or plugin code that I could use for this? I'm sure there is an python equivalent to the hex_sha1, and php_to8bit, but I had the opportunity to dig it up. - Benjamin -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
