Karthikeyan Singaravelan <[email protected]> added the comment:
I did some more debugging. doctest patches linecache which does some regex
matching when filename is of the form <doctest <filename>[examplenumber]> to
return example source. Before the commit seems absolute path was present in
warning and hence this regex didn't match. With the commit returning the
filename of this format that matches the regex the example line is again
returned. This happens with warnings inside doctest because doctest patches
linecache which is used by warnings.py during formatting the warning. In
CPython for some reason presence of both single quote and double quote inside a
triple quoted string causes the single quote to be escaped. Any concatenation
with the escaped triple quoted string also escapes the resulting text. doctest
seems to store the examples as single quoted strings that are escaped and
escaping them during _formatwarnmsg_impl causes the other one also to be
escaped. It also happens with a normal string that has an escaped double quote.
>>> a = """Test '' b""" # Two single quotes
>>> a
"Test '' b"
>>> a = """Test " b'""" # One single and double quote
>>> a
'Test " b\''
>>> a + "'c'"
'Test " b\'\'c\''
>>> a = """Test ' b""" # Only single quote
>>> a
"Test ' b"
>>>> a + "'c'"
"Test ' b'c'"
>>>> a = "Test ' b\"" # Escaped double quote
>>>> a
'Test \' b"'
>>>> a + "'a'"
'Test \' b"\'a\''
Does anyone know why this happens with escaped quotes and single quote being
escaped? Is this expected and is it part of spec about how single and double
quote are swapped over representation?
Longer explanation :
Take the below sample doctest file
$ cat ../backups/bpo36695.rst
>>> import warnings # line 0
>>> warnings.warn("Test 'a'") # line 1
doctest patches linecache.getlines to a custom function
`__patched_linecache_getlines` [0]
linecache.getlines = __patched_linecache_getlines
__LINECACHE_FILENAME_RE = re.compile(r'<doctest '
r'(?P<name>.+)'
r'\[(?P<examplenum>\d+)\]>$')
def __patched_linecache_getlines(self, filename, module_globals=None):
m = self.__LINECACHE_FILENAME_RE.match(filename)
if m and m.group('name') == self.test.name:
example = self.test.examples[int(m.group('examplenum'))]
return example.source.splitlines(keepends=True)
else:
return self.save_linecache_getlines(filename, module_globals)
doctest forms a special filename as below that is passed to exec(compile()) and
hence as per the commit warning is now raised as the filename "<doctest
bpo36695.rst[1]>" in the warning. doctest also mocks sys.stdout internally to
have the output captured to a StringIO buffer. [1]
# Use a special filename for compile(), so we can retrieve
# the source code during interactive debugging (see
# __patched_linecache_getlines).
filename = '<doctest %s[%d]>' % (test.name, examplenum)
# Before commit
cpython git:(3b0b90c8c3) ./python.exe -m doctest ../backups/bpo36695.rst
/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/doctest.py:1:
UserWarning: Test 'a'
# Module doctest.
# After commit
$ cpython git:(11a896652e) ./python.exe -m doctest ../backups/bpo36695.rst
<doctest bpo36695.rst[1]>:1: UserWarning: Test 'a'
warnings.warn("Test 'a'")
formatting warning message [2] calls linecache.getline with filename as
"<doctest bpo36695.rst[1]>" after commit which in turn calls linecache.getlines
that is patched above by doctest and hence it matches the regex and returns the
example.source "warnings.warn("Test 'a'")". It seems to be a triple quoted
string that is already escaped and hence in the below line calling s += " %s\n"
% line causes the actual warning message and the example source line to be
escaped.
def _formatwarnmsg_impl(msg):
s = ("%s:%s: %s: %s\n"
% (msg.filename, msg.lineno, msg.category.__name__,
msg.message))
if msg.line is None:
try:
import linecache
line = linecache.getline(msg.filename, msg.lineno)
except Exception:
# When a warning is logged during Python shutdown, linecache
# and the import machinery don't work anymore
line = None
linecache = None
else:
line = msg.line
if line:
line = line.strip()
s += " %s\n" % line
[0]
https://github.com/python/cpython/blob/29d018aa63b72161cfc67602dc3dbd386272da64/Lib/doctest.py#L1468
[1]
https://github.com/python/cpython/blob/29d018aa63b72161cfc67602dc3dbd386272da64/Lib/doctest.py#L1452
[2]
https://github.com/python/cpython/blob/29d018aa63b72161cfc67602dc3dbd386272da64/Lib/warnings.py#L35
----------
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue36695>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
