Hello community,
here is the log from the commit of package python-pytest-check-links for
openSUSE:Factory checked in at 2020-05-03 22:47:10
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Comparing /work/SRC/openSUSE:Factory/python-pytest-check-links (Old)
and /work/SRC/openSUSE:Factory/.python-pytest-check-links.new.2738 (New)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "python-pytest-check-links"
Sun May 3 22:47:10 2020 rev:5 rq:799795 version:0.4.0
Changes:
--------
---
/work/SRC/openSUSE:Factory/python-pytest-check-links/python-pytest-check-links.changes
2020-05-01 11:09:28.415341218 +0200
+++
/work/SRC/openSUSE:Factory/.python-pytest-check-links.new.2738/python-pytest-check-links.changes
2020-05-03 22:47:11.911150292 +0200
@@ -1,0 +2,8 @@
+Sun May 3 07:27:33 UTC 2020 - Tomáš Chvátal <[email protected]>
+
+- Update to 0.4.0:
+ * Drop py27 support and add badges #15
+ * Use requests(-cache) #12
+ * Add checking of self- and local html anchors #10
+
+-------------------------------------------------------------------
Old:
----
pytest_check_links-0.3.4.tar.gz
New:
----
pytest_check_links-0.4.0.tar.gz
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Other differences:
------------------
++++++ python-pytest-check-links.spec ++++++
--- /var/tmp/diff_new_pack.MxoRdA/_old 2020-05-03 22:47:12.547151587 +0200
+++ /var/tmp/diff_new_pack.MxoRdA/_new 2020-05-03 22:47:12.551151595 +0200
@@ -17,10 +17,11 @@
%{?!python_module:%define python_module() python-%{**} python3-%{**}}
+%define skip_python2 1
# Tests natually require internet
%bcond_with test
Name: python-pytest-check-links
-Version: 0.3.4
+Version: 0.4.0
Release: 0
Summary: Pytest plugin for checking links in files
License: BSD-3-Clause
@@ -34,7 +35,7 @@
Requires: python-docutils
Requires: python-html5lib
Requires: python-pytest >= 2.8
-Requires: python-six
+Requires: python-requests
Requires(post): update-alternatives
Requires(postun): update-alternatives
Recommends: python-jupyter_nbconvert
@@ -45,7 +46,7 @@
BuildRequires: %{python_module jupyter_nbconvert}
BuildRequires: %{python_module jupyter_nbformat}
BuildRequires: %{python_module pytest >= 2.8}
-BuildRequires: %{python_module six}
+BuildRequires: %{python_module requests}
%endif
%python_subpackages
++++++ pytest_check_links-0.3.4.tar.gz -> pytest_check_links-0.4.0.tar.gz ++++++
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pytest_check_links-0.3.4/.travis.yml
new/pytest_check_links-0.4.0/.travis.yml
--- old/pytest_check_links-0.3.4/.travis.yml 2020-03-30 11:40:43.000000000
+0200
+++ new/pytest_check_links-0.4.0/.travis.yml 2020-05-02 11:02:00.000000000
+0200
@@ -1,17 +1,17 @@
language: python
dist: xenial
python:
+ - 3.8
- 3.7
- 3.6
- 3.5
- - 3.4
- - 2.7
before_install:
- pip install --upgrade setuptools pip
install:
- - pip install --upgrade . pytest-cov
+ - pip install --upgrade .[cache] pytest-cov
script:
- - pytest --cov pytest_check_links
+ # disable automatic inclusion for coverage
+ - pytest -p no:check-links --cov pytest_check_links --cov-report term-missing
after_success:
- pip install codecov
- codecov
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pytest_check_links-0.3.4/CHANGELOG.md
new/pytest_check_links-0.4.0/CHANGELOG.md
--- old/pytest_check_links-0.3.4/CHANGELOG.md 1970-01-01 01:00:00.000000000
+0100
+++ new/pytest_check_links-0.4.0/CHANGELOG.md 2020-05-02 11:02:00.000000000
+0200
@@ -0,0 +1,13 @@
+# Change Log
+
+## 0.4.0
+* Drop py27 support and add badges
[#15](https://github.com/jupyterlab/pytest-check-links/pull/15)
+* Use requests(-cache)
[#12](https://github.com/jupyterlab/pytest-check-links/pull/12)
+* Add checking of self- and local html anchors
[#10](https://github.com/jupyterlab/pytest-check-links/pull/10)
+
+## 0.3.0
+* Honor Retry-After Header
[#9](https://github.com/jupyterlab/pytest-check-links/pull/9)
+* Update metadata for JupyterLab Org
[#7](https://github.com/jupyterlab/pytest-check-links/pull/7)
+
+## 0.2.0
+* Add rst handling
[#4](https://github.com/jupyterlab/pytest-check-links/pull/4)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pytest_check_links-0.3.4/ChangeLog
new/pytest_check_links-0.4.0/ChangeLog
--- old/pytest_check_links-0.3.4/ChangeLog 2020-04-21 17:09:34.000000000
+0200
+++ new/pytest_check_links-0.4.0/ChangeLog 2020-05-02 11:02:45.000000000
+0200
@@ -1,6 +1,31 @@
CHANGES
=======
+v0.4.0
+------
+
+* Add changelog
+* Add pypi badge
+* add badges
+* drop py27 support
+* remove allowable\_codes, attempt parsing backend-opts as json first
+* py27-compatible syntax
+* go ahead and handle/test remote #anchors
+* travis coverage tweaks
+* add handling (and test) of retries with cache
+* more py35/27 stringifying
+* more py27 syntax
+* whitespace
+* add test for custom cache location, stringify paths for <=3.5
+* reorganize cache tests, add memory backend test
+* hoist session to a singleton, ignore anchors for cache
+* some work on docs, etc
+* less-verbose protocol error handling
+* add requests-cache
+* use requests
+* Format release document
+* Add release notes
+
0.3.4
-----
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pytest_check_links-0.3.4/PKG-INFO
new/pytest_check_links-0.4.0/PKG-INFO
--- old/pytest_check_links-0.3.4/PKG-INFO 2020-04-21 17:09:34.000000000
+0200
+++ new/pytest_check_links-0.4.0/PKG-INFO 2020-05-02 11:02:46.000000000
+0200
@@ -1,6 +1,6 @@
Metadata-Version: 2.1
Name: pytest_check_links
-Version: 0.3.4
+Version: 0.4.0
Summary: Check links in files
Home-page: https://github.com/jupyterlab/pytest-check-links
Author: Jupyter Development Team
@@ -10,29 +10,97 @@
pytest plugin that checks URLs for HTML-containing files.
- Supported files:
+
[](https://codecov.io/gh/jupyterlab/pytest-check-links)
+ [](https://travis-ci.com/jupyterlab/pytest-check-links)
+ [](https://badge.fury.io/py/pytest-check-links)
+
+ ## Supported files
+
+ - `.html`
+ - `.rst`
+ - `.md` (TODO: select renderer)
+ - `.ipynb` (requires `nbconvert`)
- - .html
- - .rst
- - .md (TODO: select renderer)
- - .ipynb (requires nbconvert)
-
- Install:
+ ## Install
pip install pytest-check-links
- Use:
+ ## Use
pytest --check-links mynotebook.ipynb
+ ## Configure
+
+ #### --links-ext
+
+ > default: `md,rst,html,ipynb`
+
+ A comma-separated list of extensions to check
+
+ #### --check-anchors
+
+ Also check whether links with `#anchors` HTML files (either local, or
with
+ served with `html` in the `Content-Type`) actually exist, and point to
_exactly one_
+ named anchor.
+
+ ### Cache
+
+ Caching requires the installation of `requests-cache`.
+
+ pip install requests-cache
+
+ If enabled, each occurance of a link will be checked, no matter how
many times
+ it appears in a collection of files to check.
+
+ #### --check-links-cache
+
+ Cache requests when checking links. Caching is disabled by default,
and this option
+ must be provided, even if other cache configuration options are
provided.
+
+ #### --check-links-cache-name
+
+ > default: `.pytest-check-links-cache`
+
+ Name of link cache, either the base name of a file or similar,
depending on backend.
+
+ #### --check-links-cache-backend
+
+ > default: `sqlite3`
+
+ Cache persistence backend. The other known backends are:
+ - `memory`
+ - `redis`
+ - `mongodb`
+
+ See the [requests-cache
documentation](https://requests-cache.readthedocs.io)
+ for more information.
+
+ #### --check-links-cache-expire-after
+
+ > default: `None` (unlimited)
+
+ Time to cache link responses (seconds).
+
+ #### --check-links-cache-backend-opt
+
+ Backend-specific options for link cache, provided as `key:value`.
These are passed
+ directly to the `requests_cache.CachedSession` constructor, as they
vary depending
+ on the backend.
+
+ Values will be parsed as JSON first, so to overload the default of
caching all
+ HTTP response codes (which requires a list of `int`s):
+
+ ```bash
+ --check-links-backend-opt allowable_codes:[200]
+ ```
+
- TODO:
+ ## TODO
- pick a markdown renderer (probably commonmark) or make the markdown
renderer pluggable
- options for validating links (allow absolute links, only remote or
local, etc.)
- check internal links (`#anchors`)
- find URLs in Python docstrings
- - test myself, obvs!
Keywords: setup,distutils
@@ -43,3 +111,4 @@
Classifier: Programming Language :: Python
Classifier: Topic :: Software Development :: Testing
Description-Content-Type: text/markdown
+Provides-Extra: cache
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pytest_check_links-0.3.4/README.md
new/pytest_check_links-0.4.0/README.md
--- old/pytest_check_links-0.3.4/README.md 2020-04-11 23:53:54.000000000
+0200
+++ new/pytest_check_links-0.4.0/README.md 2020-05-02 11:02:00.000000000
+0200
@@ -2,26 +2,94 @@
pytest plugin that checks URLs for HTML-containing files.
-Supported files:
+[](https://codecov.io/gh/jupyterlab/pytest-check-links)
+[](https://travis-ci.com/jupyterlab/pytest-check-links)
+[](https://badge.fury.io/py/pytest-check-links)
+
+## Supported files
+
+- `.html`
+- `.rst`
+- `.md` (TODO: select renderer)
+- `.ipynb` (requires `nbconvert`)
-- .html
-- .rst
-- .md (TODO: select renderer)
-- .ipynb (requires nbconvert)
-
-Install:
+## Install
pip install pytest-check-links
-Use:
+## Use
pytest --check-links mynotebook.ipynb
+## Configure
+
+#### --links-ext
+
+> default: `md,rst,html,ipynb`
+
+A comma-separated list of extensions to check
+
+#### --check-anchors
+
+Also check whether links with `#anchors` HTML files (either local, or with
+served with `html` in the `Content-Type`) actually exist, and point to
_exactly one_
+named anchor.
+
+### Cache
+
+Caching requires the installation of `requests-cache`.
+
+ pip install requests-cache
+
+If enabled, each occurance of a link will be checked, no matter how many times
+it appears in a collection of files to check.
+
+#### --check-links-cache
+
+Cache requests when checking links. Caching is disabled by default, and this
option
+must be provided, even if other cache configuration options are provided.
+
+#### --check-links-cache-name
+
+> default: `.pytest-check-links-cache`
+
+Name of link cache, either the base name of a file or similar, depending on
backend.
+
+#### --check-links-cache-backend
+
+> default: `sqlite3`
+
+Cache persistence backend. The other known backends are:
+- `memory`
+- `redis`
+- `mongodb`
+
+See the [requests-cache documentation](https://requests-cache.readthedocs.io)
+for more information.
+
+#### --check-links-cache-expire-after
+
+> default: `None` (unlimited)
+
+Time to cache link responses (seconds).
+
+#### --check-links-cache-backend-opt
+
+Backend-specific options for link cache, provided as `key:value`. These are
passed
+directly to the `requests_cache.CachedSession` constructor, as they vary
depending
+on the backend.
+
+Values will be parsed as JSON first, so to overload the default of caching all
+HTTP response codes (which requires a list of `int`s):
+
+```bash
+--check-links-backend-opt allowable_codes:[200]
+```
+
-TODO:
+## TODO
- pick a markdown renderer (probably commonmark) or make the markdown renderer
pluggable
- options for validating links (allow absolute links, only remote or local,
etc.)
- check internal links (`#anchors`)
- find URLs in Python docstrings
-- test myself, obvs!
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pytest_check_links-0.3.4/RELEASE.md
new/pytest_check_links-0.4.0/RELEASE.md
--- old/pytest_check_links-0.3.4/RELEASE.md 1970-01-01 01:00:00.000000000
+0100
+++ new/pytest_check_links-0.4.0/RELEASE.md 2020-04-21 17:12:53.000000000
+0200
@@ -0,0 +1,10 @@
+
+
+```
+git pull origin master
+git tag ${NEW_VERSION}
+rm -rf dist build
+python setup.py sdist
+twine check dist/* && twine upload dist/*
+git push origin master --tags
+```
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/pytest_check_links-0.3.4/examples/anchors_remote.html
new/pytest_check_links-0.4.0/examples/anchors_remote.html
--- old/pytest_check_links-0.3.4/examples/anchors_remote.html 1970-01-01
01:00:00.000000000 +0100
+++ new/pytest_check_links-0.4.0/examples/anchors_remote.html 2020-05-02
11:02:00.000000000 +0200
@@ -0,0 +1,8 @@
+<!doctype html>
+<html>
+ <head></head>
+ <body>
+ <a
href="https://www.w3.org/TR/REC-html40/struct/links.html#anchors";>anchors
exists</a>
+ <a
href="https://www.w3.org/TR/REC-html40/struct/links.html#no-such-anchor";>doesn't
exist</a>
+ </body>
+</html>
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pytest_check_links-0.3.4/examples/httpbin.md
new/pytest_check_links-0.4.0/examples/httpbin.md
--- old/pytest_check_links-0.3.4/examples/httpbin.md 1970-01-01
01:00:00.000000000 +0100
+++ new/pytest_check_links-0.4.0/examples/httpbin.md 2020-05-02
11:02:00.000000000 +0200
@@ -0,0 +1,6 @@
+- [one](https://httpbin.org/delay/0.25#1)
+- [two](https://httpbin.org/delay/0.25#2)
+- [three](https://httpbin.org/delay/0.25#3)
+- [four](https://httpbin.org/delay/0.25#4)
+- [five](https://httpbin.org/delay/0.25#5)
+- [six](https://httpbin.org/delay/0.25#6)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pytest_check_links-0.3.4/pytest_check_links/args.py
new/pytest_check_links-0.4.0/pytest_check_links/args.py
--- old/pytest_check_links-0.3.4/pytest_check_links/args.py 2020-03-30
11:40:43.000000000 +0200
+++ new/pytest_check_links-0.4.0/pytest_check_links/args.py 2020-05-02
11:02:00.000000000 +0200
@@ -1,5 +1,5 @@
import argparse
-
+import json
class StoreExtensionsAction(argparse.Action):
@@ -14,3 +14,26 @@
def parse_extensions(self, csv):
return {'.%s' % ext.lstrip('.') for ext in csv.split(',')}
+
+
+class StoreCacheAction(argparse.Action):
+ """Build the cache session kwargs
+ """
+ def __call__(self, parser, namespace, values, option_string=None):
+ ns_name = "check_links_cache_kwargs"
+ if not hasattr(namespace, ns_name):
+ setattr(namespace, ns_name, {})
+ dest = self.dest.replace("check_links_cache_", "")
+ kwargs = namespace.check_links_cache_kwargs
+ if dest == "name":
+ kwargs["cache_name"] = values
+ elif dest == "expire_after":
+ kwargs["expire_after"] = float(values)
+ elif dest == "backend_opt":
+ key, value = str(values).split(":", 1)
+ try:
+ kwargs[key] = json.loads(value)
+ except:
+ kwargs[key] = value
+ else:
+ kwargs[dest] = values
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/pytest_check_links-0.3.4/pytest_check_links/plugin.py
new/pytest_check_links-0.4.0/pytest_check_links/plugin.py
--- old/pytest_check_links-0.3.4/pytest_check_links/plugin.py 2020-04-21
17:04:48.000000000 +0200
+++ new/pytest_check_links-0.4.0/pytest_check_links/plugin.py 2020-05-02
11:02:00.000000000 +0200
@@ -3,19 +3,26 @@
import os
import time
import warnings
-from six.moves.urllib.request import urlopen, Request
-from six.moves.urllib.parse import unquote
import html5lib
import pytest
+from requests import Session, Request
+from requests.compat import unquote
-from .args import StoreExtensionsAction
+from .args import StoreExtensionsAction, StoreCacheAction
_ENC = 'utf8'
default_extensions = {'.md', '.rst', '.html', '.ipynb'}
supported_extensions = {'.md', '.rst', '.html', '.ipynb'}
+default_cache = dict(
+ cache_name='.pytest-check-links-cache',
+ backend=None,
+ expire_after=None,
+ allowable_codes=list(range(200, 512)),
+)
+
def pytest_addoption(parser):
group = parser.getgroup("general")
@@ -30,6 +37,19 @@
"extensions are: %s." %
extensions_str(supported_extensions))
+ group.addoption('--check-links-cache', action='store_true',
+ help="Cache requests when checking links")
+ group.addoption('--check-links-cache-name', action=StoreCacheAction,
+ help="Name of link cache")
+ group.addoption('--check-links-cache-backend', action=StoreCacheAction,
+ help="Cache persistence backend")
+ group.addoption('--check-links-cache-expire-after',
action=StoreCacheAction,
+ help="Time to cache link responses (seconds)")
+ group.addoption('--check-links-cache-allowable-codes',
action=StoreCacheAction,
+ help="HTTP response codes to cache")
+ group.addoption('--check-links-cache-backend-opt', action=StoreCacheAction,
+ help="Backend-specific options for link cache, specfied as
`opt:value`")
+
def pytest_configure(config):
if config.option.links_ext:
@@ -39,15 +59,41 @@
def pytest_collect_file(path, parent):
config = parent.config
if config.option.check_links:
+ requests_session = ensure_requests_session(config)
if path.ext.lower() in config.option.links_ext:
- return CheckLinks(path, parent, config.option.check_anchors)
+ return CheckLinks(path, parent, requests_session,
config.option.check_anchors)
+
+
+def ensure_requests_session(config):
+ """Build the singleton requests.Session (or subclass)
+ """
+ session_attr = "check_links_requests_session"
+
+ if not hasattr(config.option, session_attr):
+ if config.option.check_links_cache:
+ from requests_cache import CachedSession
+ conf_kwargs = getattr(config.option, "check_links_cache_kwargs",
{})
+ kwargs = dict(default_cache)
+ kwargs.update(conf_kwargs)
+ requests_session = CachedSession(**kwargs)
+ if kwargs.get("expire_after"):
+ requests_session.remove_expired_responses()
+ else:
+ requests_session = Session()
+
+ requests_session.headers['User-Agent'] = 'pytest-check-links'
+
+ setattr(config.option, session_attr, requests_session)
+
+ return getattr(config.option, session_attr)
class CheckLinks(pytest.File):
"""Check the links in a file"""
- def __init__(self, path, parent, check_anchors):
+ def __init__(self, path, parent, requests_session, check_anchors=False):
super(CheckLinks, self).__init__(path, parent)
self.check_anchors = check_anchors
+ self.requests_session = requests_session
def _html_from_html(self):
"""Return HTML from an HTML file"""
@@ -159,7 +205,6 @@
def __init__(self, name, parent, target, parsed, description=''):
super(LinkItem, self).__init__(name, parent)
self.target = target
- self.retry_attempts = 0
self.parsed = parsed
self.description = description or '{}: {}'.format(self.fspath, target)
@@ -173,26 +218,31 @@
def reportinfo(self):
return self.fspath, 0, self.description
- def handle_retry(self, obj):
+ def sleep(self, response):
"""Handle responses with a Retry-After header.
https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.37
"""
- if self.retry_attempts < 3:
+ header = response.headers.get('Retry-After')
+
+ if header is None:
+ return False
+
+ if header == '1m0s':
+ sleep_time = 60
+ else:
try:
- sleep_time = int(obj.headers['Retry-After'])
+ sleep_time = int(header)
except ValueError:
sleep_time = 10
- # Github uses this non-conforming Retry-After
- if obj.headers['Retry-After'] == '1m0s':
- sleep_time = 60
- self.retry_attempts += 1
- time.sleep(sleep_time)
- return self.runtest()
- raise BrokenLinkError(self.target, "%s %s" % (obj.code, obj.reason))
+ time.sleep(sleep_time)
+
+ return True
def handle_anchor(self, parsed, anchor):
+ """Verify an anchor exists in the parsed HTML
+ """
anchors = parsed.findall('*//a[@name="{}"]'.format(anchor))
if not anchors:
@@ -206,29 +256,55 @@
)
)
+ def fetch_with_retries(self, url, retries=3):
+ """Fetch a URL, optionally retrying after a delay (by header)
+ """
+
+ url_no_anchor = url.split("#")[0]
+ session = self.parent.requests_session
+
+ try:
+ response = session.get(url_no_anchor)
+ except Exception as err:
+ raise BrokenLinkError(url, "%s" % err)
+
+ if response.status_code >= 400:
+ if retries and self.sleep(response):
+ self.uncache_url(url_no_anchor)
+ return self.fetch_with_retries(url, retries=retries - 1)
+
+ raise BrokenLinkError(url, "%d: %s" % (
+ response.status_code,
+ response.reason
+ ))
+
+ return response
+
+ def uncache_url(self, url):
+ uncached = False
+ session = self.parent.requests_session
+ if hasattr(session, "cache"):
+ request = Request('GET', url, headers=session.headers).prepare()
+ key = session.cache.create_key(request)
+ if session.cache.has_key(key):
+ session.cache.delete(key)
+ uncached = True
+ return uncached
+
def runtest(self):
- if ':' in self.target:
- # external reference, download
- req = Request(self.target)
- req.add_header('User-Agent', 'pytest-check-links')
- try:
- f = urlopen(req)
- except Exception as e:
- if hasattr(e, 'headers') and 'Retry-After' in e.headers:
- return self.handle_retry(e)
- raise BrokenLinkError(self.target, str(e))
- else:
- code = f.getcode()
- f.close()
- if code >= 400:
- if 'Retry-After' in f.headers:
- return self.handle_retry(e)
- raise BrokenLinkError(self.target, str(code))
+ url = self.target
+
+ if ':' in url:
+ response = self.fetch_with_retries(url)
+ if self.parent.check_anchors and '#' in url:
+ anchor = url.split('#')[1]
+ if anchor and "html" in response.headers.get("Content-Type"):
+ parsed = html5lib.parse(response.content,
namespaceHTMLElements=False)
+ return self.handle_anchor(parsed, anchor)
else:
- if self.target.startswith('/'):
- raise BrokenLinkError(self.target, "absolute path link")
+ if url.startswith('/'):
+ raise BrokenLinkError(url, "absolute path link")
# relative URL
- url = self.target
anchor = None
if '?' in url:
url = url.split('?')[0]
@@ -256,7 +332,7 @@
break
if not exists:
target_path = dirpath.join(url_path)
- raise BrokenLinkError(self.target, "No such file: %s" %
target_path)
+ raise BrokenLinkError(url, "No such file: %s" % target_path)
def extensions_str(extensions):
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/pytest_check_links-0.3.4/pytest_check_links.egg-info/PKG-INFO
new/pytest_check_links-0.4.0/pytest_check_links.egg-info/PKG-INFO
--- old/pytest_check_links-0.3.4/pytest_check_links.egg-info/PKG-INFO
2020-04-21 17:09:34.000000000 +0200
+++ new/pytest_check_links-0.4.0/pytest_check_links.egg-info/PKG-INFO
2020-05-02 11:02:46.000000000 +0200
@@ -1,6 +1,6 @@
Metadata-Version: 2.1
Name: pytest-check-links
-Version: 0.3.4
+Version: 0.4.0
Summary: Check links in files
Home-page: https://github.com/jupyterlab/pytest-check-links
Author: Jupyter Development Team
@@ -10,29 +10,97 @@
pytest plugin that checks URLs for HTML-containing files.
- Supported files:
+
[](https://codecov.io/gh/jupyterlab/pytest-check-links)
+ [](https://travis-ci.com/jupyterlab/pytest-check-links)
+ [](https://badge.fury.io/py/pytest-check-links)
+
+ ## Supported files
+
+ - `.html`
+ - `.rst`
+ - `.md` (TODO: select renderer)
+ - `.ipynb` (requires `nbconvert`)
- - .html
- - .rst
- - .md (TODO: select renderer)
- - .ipynb (requires nbconvert)
-
- Install:
+ ## Install
pip install pytest-check-links
- Use:
+ ## Use
pytest --check-links mynotebook.ipynb
+ ## Configure
+
+ #### --links-ext
+
+ > default: `md,rst,html,ipynb`
+
+ A comma-separated list of extensions to check
+
+ #### --check-anchors
+
+ Also check whether links with `#anchors` HTML files (either local, or
with
+ served with `html` in the `Content-Type`) actually exist, and point to
_exactly one_
+ named anchor.
+
+ ### Cache
+
+ Caching requires the installation of `requests-cache`.
+
+ pip install requests-cache
+
+ If enabled, each occurance of a link will be checked, no matter how
many times
+ it appears in a collection of files to check.
+
+ #### --check-links-cache
+
+ Cache requests when checking links. Caching is disabled by default,
and this option
+ must be provided, even if other cache configuration options are
provided.
+
+ #### --check-links-cache-name
+
+ > default: `.pytest-check-links-cache`
+
+ Name of link cache, either the base name of a file or similar,
depending on backend.
+
+ #### --check-links-cache-backend
+
+ > default: `sqlite3`
+
+ Cache persistence backend. The other known backends are:
+ - `memory`
+ - `redis`
+ - `mongodb`
+
+ See the [requests-cache
documentation](https://requests-cache.readthedocs.io)
+ for more information.
+
+ #### --check-links-cache-expire-after
+
+ > default: `None` (unlimited)
+
+ Time to cache link responses (seconds).
+
+ #### --check-links-cache-backend-opt
+
+ Backend-specific options for link cache, provided as `key:value`.
These are passed
+ directly to the `requests_cache.CachedSession` constructor, as they
vary depending
+ on the backend.
+
+ Values will be parsed as JSON first, so to overload the default of
caching all
+ HTTP response codes (which requires a list of `int`s):
+
+ ```bash
+ --check-links-backend-opt allowable_codes:[200]
+ ```
+
- TODO:
+ ## TODO
- pick a markdown renderer (probably commonmark) or make the markdown
renderer pluggable
- options for validating links (allow absolute links, only remote or
local, etc.)
- check internal links (`#anchors`)
- find URLs in Python docstrings
- - test myself, obvs!
Keywords: setup,distutils
@@ -43,3 +111,4 @@
Classifier: Programming Language :: Python
Classifier: Topic :: Software Development :: Testing
Description-Content-Type: text/markdown
+Provides-Extra: cache
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/pytest_check_links-0.3.4/pytest_check_links.egg-info/SOURCES.txt
new/pytest_check_links-0.4.0/pytest_check_links.egg-info/SOURCES.txt
--- old/pytest_check_links-0.3.4/pytest_check_links.egg-info/SOURCES.txt
2020-04-21 17:09:34.000000000 +0200
+++ new/pytest_check_links-0.4.0/pytest_check_links.egg-info/SOURCES.txt
2020-05-02 11:02:46.000000000 +0200
@@ -1,14 +1,18 @@
.travis.yml
AUTHORS
+CHANGELOG.md
ChangeLog
LICENSE
README.md
+RELEASE.md
pytest.ini
requirements.txt
setup.cfg
setup.py
examples/anchors_other.html
+examples/anchors_remote.html
examples/anchors_self.html
+examples/httpbin.md
examples/linkcheck.ipynb
examples/markdown.md
examples/rst.rst
@@ -24,5 +28,8 @@
pytest_check_links.egg-info/pbr.json
pytest_check_links.egg-info/requires.txt
pytest_check_links.egg-info/top_level.txt
+test/__init__.py
test/conftest.py
+test/test_anchors.py
+test/test_cache.py
test/test_check_links.py
\ No newline at end of file
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/pytest_check_links-0.3.4/pytest_check_links.egg-info/pbr.json
new/pytest_check_links-0.4.0/pytest_check_links.egg-info/pbr.json
--- old/pytest_check_links-0.3.4/pytest_check_links.egg-info/pbr.json
2020-04-21 17:09:34.000000000 +0200
+++ new/pytest_check_links-0.4.0/pytest_check_links.egg-info/pbr.json
2020-05-02 11:02:46.000000000 +0200
@@ -1 +1 @@
-{"git_version": "5d62197", "is_release": false}
\ No newline at end of file
+{"git_version": "3b1564b", "is_release": false}
\ No newline at end of file
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/pytest_check_links-0.3.4/pytest_check_links.egg-info/requires.txt
new/pytest_check_links-0.4.0/pytest_check_links.egg-info/requires.txt
--- old/pytest_check_links-0.3.4/pytest_check_links.egg-info/requires.txt
2020-04-21 17:09:34.000000000 +0200
+++ new/pytest_check_links-0.4.0/pytest_check_links.egg-info/requires.txt
2020-05-02 11:02:46.000000000 +0200
@@ -1,6 +1,9 @@
-pytest>=2.8
+docutils
html5lib
-six
-nbformat
nbconvert
-docutils
+nbformat
+pytest>=2.8
+requests
+
+[cache]
+requests-cache
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pytest_check_links-0.3.4/requirements.txt
new/pytest_check_links-0.4.0/requirements.txt
--- old/pytest_check_links-0.3.4/requirements.txt 2020-03-30
11:40:43.000000000 +0200
+++ new/pytest_check_links-0.4.0/requirements.txt 2020-05-02
11:02:00.000000000 +0200
@@ -1,6 +1,6 @@
-pytest >= 2.8
+docutils
html5lib
-six
-nbformat
nbconvert
-docutils
+nbformat
+pytest >= 2.8
+requests
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pytest_check_links-0.3.4/setup.cfg
new/pytest_check_links-0.4.0/setup.cfg
--- old/pytest_check_links-0.3.4/setup.cfg 2020-04-21 17:09:34.000000000
+0200
+++ new/pytest_check_links-0.4.0/setup.cfg 2020-05-02 11:02:46.000000000
+0200
@@ -30,6 +30,10 @@
console_scripts =
pytest-check-links = pytest_check_links.__main__:main
+[options.extras_require]
+cache =
+ requests-cache
+
[pep8]
ignore = E128
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pytest_check_links-0.3.4/test/__init__.py
new/pytest_check_links-0.4.0/test/__init__.py
--- old/pytest_check_links-0.3.4/test/__init__.py 1970-01-01
01:00:00.000000000 +0100
+++ new/pytest_check_links-0.4.0/test/__init__.py 2020-05-02
11:02:00.000000000 +0200
@@ -0,0 +1,4 @@
+import os
+
+here = os.path.dirname(os.path.abspath(__file__))
+examples = os.path.join(os.path.dirname(here), 'examples')
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pytest_check_links-0.3.4/test/test_anchors.py
new/pytest_check_links-0.4.0/test/test_anchors.py
--- old/pytest_check_links-0.3.4/test/test_anchors.py 1970-01-01
01:00:00.000000000 +0100
+++ new/pytest_check_links-0.4.0/test/test_anchors.py 2020-05-02
11:02:00.000000000 +0200
@@ -0,0 +1,28 @@
+from . import examples
+
+import pytest
+
+
[email protected]
+def anchor_args():
+ return ["-v", "--check-links", "--check-anchors"]
+
+
+def test_anchors_local_self(testdir, anchor_args):
+ testdir.copy_example('anchors_self.html')
+ result = testdir.runpytest(*anchor_args)
+ result.assert_outcomes(passed=1, failed=2)
+
+
+def test_anchors_local_other(testdir, anchor_args):
+ testdir.copy_example('anchors_self.html')
+ testdir.copy_example('anchors_other.html')
+ args = anchor_args + ["anchors_other.html"]
+ result = testdir.runpytest(*args)
+ result.assert_outcomes(passed=1, failed=2)
+
+
+def test_anchors_external(testdir, anchor_args):
+ testdir.copy_example('anchors_remote.html')
+ result = testdir.runpytest(*anchor_args)
+ result.assert_outcomes(passed=1, failed=1)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pytest_check_links-0.3.4/test/test_cache.py
new/pytest_check_links-0.4.0/test/test_cache.py
--- old/pytest_check_links-0.3.4/test/test_cache.py 1970-01-01
01:00:00.000000000 +0100
+++ new/pytest_check_links-0.4.0/test/test_cache.py 2020-05-02
11:02:00.000000000 +0200
@@ -0,0 +1,141 @@
+import os
+import time
+import shutil
+
+from glob import glob
+
+import pytest
+
+import requests_cache
+
+from . import examples
+
+
[email protected]
+def base_args():
+ return ["-v", "--check-links", "--check-links-cache"]
+
+
[email protected]
+def memory_args(base_args):
+ return base_args + ["--check-links-cache-backend", "memory"]
+
+
+def assert_sqlite(testdir, name=None, tmpdir=None, exists=True):
+ name = name or ".pytest-check-links-cache.sqlite"
+ tmpdir = str(tmpdir or testdir.tmpdir)
+ caches = list(glob(os.path.join(tmpdir, name)))
+ if exists:
+ assert caches
+ else:
+ assert not caches
+
+
[email protected]("cache_name", [
+ None,
+ "custom-cache"
+])
+def test_cache_expiry(testdir, base_args, cache_name, tmpdir):
+ """will the default sqlite3 backend persist and then expire?
+ """
+ testdir.copy_example('linkcheck.ipynb')
+
+ args = base_args + ["--check-links-cache-expire-after", "2"]
+ if cache_name:
+ args += ["--check-links-cache-name", os.path.join(str(tmpdir),
cache_name)]
+ expected = dict(passed=3, failed=3)
+ t0 = time.time()
+ result = testdir.runpytest(*args)
+ t1 = time.time()
+ result.assert_outcomes(**expected)
+
+ if cache_name:
+ assert_sqlite(testdir, name="{}.sqlite".format(cache_name),
tmpdir=tmpdir)
+ else:
+ assert_sqlite(testdir)
+
+ t2 = time.time()
+ result = testdir.runpytest(*args)
+ t3 = time.time()
+ result.assert_outcomes(**expected)
+
+ assert t1 - t0 > t3 - t2, "cache did not make second run faster"
+
+ time.sleep(2)
+
+ t4 = time.time()
+ result = testdir.runpytest(*args)
+ t5 = time.time()
+ result.assert_outcomes(**expected)
+
+ assert t5 - t4 > t3 - t2, "cache did not expire"
+
+
+def test_cache_memory(testdir, memory_args):
+ """will the memory backend cache links inside a run?
+ """
+ expected = dict(passed=3, failed=0)
+
+ testdir.copy_example('httpbin.md')
+
+ def run(passed):
+ t0 = time.time()
+ result = testdir.runpytest(*memory_args)
+ t1 = time.time()
+ result.assert_outcomes(passed=passed, failed=0)
+ assert_sqlite(testdir, exists=False)
+ return t1 - t0
+
+ d0 = run(6)
+
+ for i in range(5):
+ shutil.copy(
+ os.path.join(str(testdir.tmpdir), "httpbin.md"),
+ os.path.join(str(testdir.tmpdir), "httpbin{}.md".format(i))
+ )
+
+ d1 = run(36)
+ # allow a healthy savings margin for network flake
+ assert d1 < d0 * 4
+
+
+def test_cache_retry(testdir, memory_args):
+ """will a Retry-After header work with cache?
+ """
+
+ testdir.copy_example('httpbin.md')
+
+ attempts = []
+
+ _get = requests_cache.CachedSession.get
+
+ def mock_get(*args, **kwargs):
+ response = _get(*args, **kwargs)
+ if len(attempts) < 5:
+ response.status_code = 502
+ response.headers['Retry-After'] = '0'
+ attempts.append([args, kwargs])
+ return response
+
+ requests_cache.CachedSession.get = mock_get
+
+ result = testdir.runpytest(*memory_args)
+
+ try:
+ result.assert_outcomes(passed=5, failed=1)
+ assert len(attempts) == 10
+ finally:
+ requests_cache.CachedSession.get = _get
+
+
+def test_cache_backend_opts(testdir, base_args):
+ testdir.copy_example('httpbin.md')
+ args = base_args + [
+ "--check-links-cache-backend-opt", "fast_save:true",
+ "--check-links-cache-name", "foo",
+ "--check-links-cache-backend-opt", "extension:.db",
+ "--check-links-cache-backend-opt", "allowable_codes:[200]"
+ ]
+ result = testdir.runpytest(*args)
+ result.assert_outcomes(passed=6, failed=0)
+ assert_sqlite(testdir, name="foo.db")
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pytest_check_links-0.3.4/test/test_check_links.py
new/pytest_check_links-0.4.0/test/test_check_links.py
--- old/pytest_check_links-0.3.4/test/test_check_links.py 2020-04-21
17:04:48.000000000 +0200
+++ new/pytest_check_links-0.4.0/test/test_check_links.py 2020-05-02
11:02:00.000000000 +0200
@@ -1,7 +1,4 @@
-import os
-
-here = os.path.dirname(os.path.abspath(__file__))
-examples = os.path.join(os.path.dirname(here), 'examples')
+from . import examples
def test_ipynb(testdir):
@@ -25,14 +22,3 @@
testdir.copy_example('markdown.md')
result = testdir.runpytest("-v", "--check-links", "--links-ext=.md,rst")
result.assert_outcomes(passed=15, failed=6)
-
-def test_anchors_self(testdir):
- testdir.copy_example('anchors_self.html')
- result = testdir.runpytest("-v", "--check-links", "--check-anchors")
- result.assert_outcomes(passed=1, failed=2)
-
-def test_anchors_other(testdir):
- testdir.copy_example('anchors_self.html')
- testdir.copy_example('anchors_other.html')
- result = testdir.runpytest("-v", "--check-links", "--check-anchors",
"anchors_other.html")
- result.assert_outcomes(passed=1, failed=2)
