[MediaWiki-commits] [Gerrit] pywikibot...FLOSSbot[master]: repository: make http verification more robust

[jenkins-bot (Code Review)]

jenkins-bot has submitted this change and it was merged.

Change subject: repository: make http verification more robust
..


repository: make http verification more robust

And add comments in the code to clarify why each improvement is actually
needed.

Change-Id: I34410d306c4caff5b7c05c30fd70d5c8db92be81
Signed-off-by: Loic Dachary 
---
M FLOSSbot/repository.py
1 file changed, 17 insertions(+), 2 deletions(-)

Approvals:
  Dachary: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/FLOSSbot/repository.py b/FLOSSbot/repository.py
index 3a3948e..5d4d9d0 100644
--- a/FLOSSbot/repository.py
+++ b/FLOSSbot/repository.py
@@ -397,9 +397,24 @@
 
 def verify_http(self, url):
 try:
-r = requests.head(url, allow_redirects=True)
+#
+# although head() would be more light weight, some
+# servers do not respond to it. For instance
+# https://src.openvz.org/projects/OVZ/ returned 405
+#
+# The user agent is required for some servers. For
+# instance http://marabunta.laotracara.com/descargas/
+# returns 406 if no User-Agent header is set.
+#
+r = requests.get(url,
+ headers={'User-Agent': 'FLOSSbot'},
+ verify=False)
+log.debug("GET " + url + " status " + str(r.status_code))
+if r.status_code != requests.codes.ok:
+log.debug("GET " + url + " " + r.text)
 return r.status_code == requests.codes.ok
-except:
+except Exception as e:
+log.debug("GET failed with " + str(e))
 return False
 
 def verify_protocol(self, url, protocol, credentials):

-- 
To view, visit https://gerrit.wikimedia.org/r/311843
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I34410d306c4caff5b7c05c30fd70d5c8db92be81
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/bots/FLOSSbot
Gerrit-Branch: master
Gerrit-Owner: Dachary 
Gerrit-Reviewer: Dachary 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] pywikibot...FLOSSbot[master]: repository: make http verification more robust

[Dachary (Code Review)]

Dachary has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/311843

Change subject: repository: make http verification more robust
..

repository: make http verification more robust

And add comments in the code to clarify why each improvement is actually
needed.

Change-Id: I34410d306c4caff5b7c05c30fd70d5c8db92be81
Signed-off-by: Loic Dachary 
---
M FLOSSbot/repository.py
1 file changed, 17 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/pywikibot/bots/FLOSSbot 
refs/changes/43/311843/1

diff --git a/FLOSSbot/repository.py b/FLOSSbot/repository.py
index 3a3948e..5d4d9d0 100644
--- a/FLOSSbot/repository.py
+++ b/FLOSSbot/repository.py
@@ -397,9 +397,24 @@
 
 def verify_http(self, url):
 try:
-r = requests.head(url, allow_redirects=True)
+#
+# although head() would be more light weight, some
+# servers do not respond to it. For instance
+# https://src.openvz.org/projects/OVZ/ returned 405
+#
+# The user agent is required for some servers. For
+# instance http://marabunta.laotracara.com/descargas/
+# returns 406 if no User-Agent header is set.
+#
+r = requests.get(url,
+ headers={'User-Agent': 'FLOSSbot'},
+ verify=False)
+log.debug("GET " + url + " status " + str(r.status_code))
+if r.status_code != requests.codes.ok:
+log.debug("GET " + url + " " + r.text)
 return r.status_code == requests.codes.ok
-except:
+except Exception as e:
+log.debug("GET failed with " + str(e))
 return False
 
 def verify_protocol(self, url, protocol, credentials):

-- 
To view, visit https://gerrit.wikimedia.org/r/311843
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I34410d306c4caff5b7c05c30fd70d5c8db92be81
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/bots/FLOSSbot
Gerrit-Branch: master
Gerrit-Owner: Dachary 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits