jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1312258?usp=email )
Change subject: [cleanup] Fix incomplete URL substring sanitization (CodeQL) ...................................................................... [cleanup] Fix incomplete URL substring sanitization (CodeQL) Two places in the codebase were checking "is this URL trusted?" by looking for a known domain name anywhere inside the URL string, e.g. 'jstor.org' in link. That check can be fooled - a URL like https://jstor.org.attacker.com/phish contains the text "jstor.org" too, even though it isn't a JSTOR link at all. Replaced both checks with proper URL parsing, so we now compare against the actual hostname instead of just searching for a substring: * pywikibot/data/sparql.py: the code that detects a Wikimedia Commons login redirect now checks the parsed hostname instead of searching for "commons-query.wikimedia.org" anywhere in the URL. * scripts/reflinks.py: the JSTOR link skip-check now checks the parsed hostname (and its subdomains) instead of searching for "jstor.org" anywhere in the link. Bug: T432508 Change-Id: I0950512ec5bd00711ab17c8047ab36a0842fc4f2 --- M pywikibot/data/sparql.py M scripts/reflinks.py 2 files changed, 5 insertions(+), 3 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/data/sparql.py b/pywikibot/data/sparql.py index d259326..1d43f3a 100644 --- a/pywikibot/data/sparql.py +++ b/pywikibot/data/sparql.py @@ -8,7 +8,7 @@ from textwrap import fill from typing import Any, cast -from urllib.parse import quote +from urllib.parse import quote, urlparse from requests import JSONDecodeError, Response from requests.exceptions import Timeout @@ -170,7 +170,7 @@ # not in case the response otherwise might have it in between strcontent = self.last_response.content.decode() if (strcontent.startswith('<!DOCTYPE html>') - and 'https://commons-query.wikimedia.org' in url + and urlparse(url).hostname == 'commons-query.wikimedia.org' and ('Special:UserLogin' in strcontent or 'Special:OAuth' in strcontent)): raise NoUsernameError(fill( diff --git a/scripts/reflinks.py b/scripts/reflinks.py index 99f9344..2dac3ad 100755 --- a/scripts/reflinks.py +++ b/scripts/reflinks.py @@ -63,6 +63,7 @@ from http import HTTPStatus from pathlib import Path from textwrap import shorten +from urllib.parse import urlparse import pywikibot from pywikibot import comms, config, i18n, pagegenerators, textlib @@ -562,7 +563,8 @@ # for each link to change for match in linksInRef.finditer(raw_text): link = match['url'] - if 'jstor.org' in link: + hostname = urlparse(link).hostname or '' + if hostname == 'jstor.org' or hostname.endswith('.jstor.org'): # TODO: Clean URL blacklist continue -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1312258?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: I0950512ec5bd00711ab17c8047ab36a0842fc4f2 Gerrit-Change-Number: 1312258 Gerrit-PatchSet: 1 Gerrit-Owner: Nihar_Chakravarti <[email protected]> Gerrit-Reviewer: Xqt <[email protected]> Gerrit-Reviewer: jenkins-bot
_______________________________________________ Pywikibot-commits mailing list -- [email protected] To unsubscribe send an email to [email protected]
