jayvdb created this task. jayvdb added projects: Pywikibot-weblinkchecker.py, pywikibot-core. Herald added subscribers: pywikibot-bugs-list, StudiesWorld, Aklapper.
TASK DESCRIPTION With gso.gbv.de removed from the ignore list: ``` $ python pwb.py weblinkchecker -lang:de -family:wikipedia -page:Landeshauptarchiv_Schwerin Retrieving 1 pages from wikipedia:de. >>> Landeshauptarchiv Schwerin <<< checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+310140870 checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+358791324 checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+490440061 checking http://gso.gbv.de/CMD?ACT=SRCHA&IKT=1016&SRT=YOP&TRM=SER+Findbücher,+Inventare+und+kleine+Schriften+des+Landeshauptarchivs+Schwerin checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+241270014 checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+241270162 checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+241953677 checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+254003699 checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+310140870 checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+328953334 checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+342544136 checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+342544241 checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+328954845 checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+358791324 checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+372154581 checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+47082462X checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+490440061 checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+499794850 checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+537538801 checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+590233408 checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+611739003 checking http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+680277579 checking http://www.kulturwerte-mv.de/cms2/LAKD_prod/LAKD/content/de/Landesarchiv/Landeshauptarchiv_Schwerin/index.jsp checking http://www.kulturwerte-mv.de/cms2/LAKD_prod/LAKD/content_downloads/Landesarchiv/2010/Bautagung_Landeshauptarchiv/Der_Archivbau_Paul_Ehmigs_in_Schwerin,_Dieter_Zander.pdf checking http://mv.juris.de/mv/ArchivG_MV_rahmen.htm 1 pages read 0 pages written Execution time: 1 seconds Read operation time: 1 seconds Script terminated successfully. Waiting for remaining 25 threads to finish, please wait... ... Contacting server gso.gbv.de to find out its default encoding... Waiting for remaining 25 threads to finish, please wait... *[[Landeshauptarchiv Schwerin]] links to http://mv.juris.de/mv/ArchivG_MV_rahmen.htm - Socket Error: 'No address associated with hostname'. Error retrieving server's default charset. Using ISO 8859-1. Exception in thread Landeshauptarchiv Schwerin - http://gso.gbv.de/CMD?ACT=SRCHA&IKT=1016&SRT=YOP&TRM=SER+Findbücher,+Inventare+und+kleine+Schriften+des+Landeshauptarchivs+Schwerin: Traceback (most recent call last): File "/usr/lib64/python2.7/threading.py", line 813, in __bootstrap_inner self.run() File "./scripts/weblinkchecker.py", line 550, in run linkChecker = LinkChecker(self.url, HTTPignore=self.HTTPignore) File "./scripts/weblinkchecker.py", line 309, in __init__ self.changeUrl(url) File "./scripts/weblinkchecker.py", line 369, in changeUrl self.query = unicode(urllib.quote(self.query.encode(encoding), '=&')) File "/usr/lib64/python2.7/urllib.py", line 1301, in quote if not s.rstrip(safe): UnicodeDecodeError: 'ascii' codec can't decode byte 0xfc in position 41: ordinal not in range(128) Waiting for remaining 23 threads to finish, please wait... Waiting for remaining 23 threads to finish, please wait... http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+358791324 ok http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+490440061 ok Waiting for remaining 21 threads to finish, please wait... http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+342544136 ok http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+241953677 ok http://www.kulturwerte-mv.de/cms2/LAKD_prod/LAKD/content_downloads/Landesarchiv/2010/Bautagung_Landeshauptarchiv/Der_Archivbau_Paul_Ehmigs_in_Schwerin,_Dieter_Zander.pdf ok http://www.kulturwerte-mv.de/cms2/LAKD_prod/LAKD/content/de/Landesarchiv/Landeshauptarchiv_Schwerin/index.jsp ok http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+490440061 ok http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+499794850 ok http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+342544241 ok http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+328954845 ok http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+372154581 ok http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+537538801 ok http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+611739003 ok Waiting for remaining 10 threads to finish, please wait... http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+328953334 ok http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+241270162 ok http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+241270014 ok http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+47082462X ok http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+590233408 ok http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+310140870 ok http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+358791324 ok http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+680277579 ok Waiting for remaining 2 threads to finish, please wait... http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+310140870 ok http://gso.gbv.de/CMD?ACT=SRCHA&TRM=PPN+254003699 ok Saving history... ``` So there is a problem with http://gso.gbv.de/CMD?ACT=SRCHA&IKT=1016&SRT=YOP&TRM=SER+Findbücher,+Inventare+und+kleine+Schriften+des+Landeshauptarchivs+Schwerin , which does redirect to another page, and `requests` cant correctly decode it. ``` >>> r = requests.get('http://gso.gbv.de/CMD?ACT=SRCHA&IKT=1016&SRT=YOP&TRM=SER+Findbücher,+Inventare+und+kleine+Schriften+des+Landeshauptarchivs+Schwerin') >>> r.encoding 'ISO-8859-1' >>> r.content ... >>> r.url u'http://gso.gbv.de/CMD?ACT=SRCHA&IKT=1016&SRT=YOP&TRM=SER+Findb%C3%BCcher,+Inventare+und+kleine+Schriften+des+Landeshauptarchivs+Schwerin&COOKIE=U999,K999,D2.1,E2b9cf6b1-e,I0,B9994++++++,SY,A%5C9008+J,,1,,U,,4,,7,,M,H13-15,,17-23,,30,,50,,60-61,,73-78,,88-90,NGAST,R129.180.76.69,FN' ``` TASK DETAIL https://phabricator.wikimedia.org/T124140 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jayvdb Cc: Aklapper, StudiesWorld, pywikibot-bugs-list, jayvdb _______________________________________________ pywikibot-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
