[issue13848] io.open() doesn't check for embedded NUL characters
Hynek Schlawack h...@ox.cx added the comment: I have fixed the refleak, added a _PyUnicode_HasNULChars and integrated it into the Win32-unicode-if-branch. Couldn't test it due to lack of win32 – the function itself is tested though. -- Added file: http://bugs.python.org/file24355/open-nul.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13848 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13848] io.open() doesn't check for embedded NUL characters
Hynek Schlawack h...@ox.cx added the comment: With Georg's kind help I added some improvements: - I've been reluctant to waste heap for caching the nul string but he convinced me that I was being ridiculous ;) - For some reason there was a stray character inside, that should be fixed too. In related news, I'm also adding tests for fileio since the last patch. -- Added file: http://bugs.python.org/file24356/open-nul.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13848 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13848] io.open() doesn't check for embedded NUL characters
Antoine Pitrou pit...@free.fr added the comment: The patch works under Windows here (on branch default). -- stage: needs patch - patch review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13848 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13848] io.open() doesn't check for embedded NUL characters
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 572bb8c265c0 by Antoine Pitrou in branch '3.2': Issue #13848: open() and the FileIO constructor now check for NUL characters in the file name. http://hg.python.org/cpython/rev/572bb8c265c0 New changeset 6bb05ce1cd1f by Antoine Pitrou in branch 'default': Issue #13848: open() and the FileIO constructor now check for NUL characters in the file name. http://hg.python.org/cpython/rev/6bb05ce1cd1f -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13848 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13848] io.open() doesn't check for embedded NUL characters
Antoine Pitrou pit...@free.fr added the comment: I've made small changes and committed the patch in 3.2 and 3.3. 2.7 would need further changes and I don't think it's worth the bother. Thanks! -- resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13848 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13848] io.open() doesn't check for embedded NUL characters
Antoine Pitrou pit...@free.fr added the comment: Indeed, there seems to be no mechanism available to forbid NUL chars under Windows (for Python 3.x): open(LICENSE\x00foobar) _io.TextIOWrapper name='LICENSE\x00foobar' mode='r' encoding='cp1252' os.stat(LICENSE\x00foobar) nt.stat_result(st_mode=33206, st_ino=2251799813779714, st_dev=0, st_nlink=1, st_uid=0, st_gid=0, st_size=15132, st_atime=8589934592, st_mtime=8589934592, st_ctime=1301169903) I think PyUnicode_AsUnicode should grow a NUL char check in Python 3.3, since it doesn't return the size anyway. I don't think we can do that in previous versions, though, so we need an alternate strategy. Scanning the unicode string for NUL characters is enough. That should be easy by using PyUnicode_AsUnicodeAndSize. As for the patch: - the test should be in test_io; you may also add a separate in test_fileio - conv_name is never decref'ed, and so there will be a memory leak -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13848 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13848] io.open() doesn't check for embedded NUL characters
Antoine Pitrou pit...@free.fr added the comment: Since the NUL-scanning will be useful for Modules/posixmodule.c as well, perhaps it should be done as a private _PyUnicode_HasNULChars() function. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13848 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13848] io.open() doesn't check for embedded NUL characters
Terry J. Reedy tjre...@udel.edu added the comment: See also #13849 -- nosy: +terry.reedy ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13848 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13848] io.open() doesn't check for embedded NUL characters
Hynek Schlawack h...@ox.cx added the comment: I took a deep dive into parts of CPython that were unknown to me :) and dug up the following: Methods like os.stat or even os.open convert the file name using et in PyArg_ParseTuple[AndKeywords]. OTOH, open() and io.open() just hand through the object as O to the respective low-level io module. The result in 2.7 is that file() tries to convert it for it's own usage eventually – which fails as seen. While a more explicit error message wouldn't hurt, this seems safe to me insofar. In 3.3, file() aka Modules/_io/fileio.c , io_open does no such thing because it seems to handle fds as nameobj as well and does a wide range of checks on the argument. After io_open is certain that nameobj is a file name, it uses PyObject_AsCharBuffer()on bytes and PyUnicode_FromObject() + encoding magic on unicode to get an encoded string as a file name. Neither does a check for NUL bytes so the (w)open(er) that follows reacts as demonstrated by Antoine. I presume fixing/breaking PyObject_AsCharBuffer()/PyUnicode_FromObject() is out of question so the most obvious part to fix would be the conversion block inside io_open. Should I have a stab at it or do you disagree with my approach? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13848 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13848] io.open() doesn't check for embedded NUL characters
Antoine Pitrou pit...@free.fr added the comment: Yes, fixing the conversion block is probably the right approach. Apparently posixmodule.c uses PyUnicode_FSConverter, perhaps that would work? (also make sure that the case where a bytes string is given is fixed too: open(b\x00) Traceback (most recent call last): File stdin, line 1, in module FileNotFoundError: [Errno 2] No such file or directory: '' ) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13848 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13848] io.open() doesn't check for embedded NUL characters
Hynek Schlawack h...@ox.cx added the comment: JFTR, file()'s C equivalent is fileio_init and not io_open, I lost track of all the opens. ;) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13848 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13848] io.open() doesn't check for embedded NUL characters
Hynek Schlawack h...@ox.cx added the comment:
So I have good news and bad news. The good is: I fixed it for non-Win platforms
and the patch is truly beautiful:
Lib/test/test_builtin.py | 6 ++
Modules/_io/fileio.c | 25 -
2 files changed, 10 insertions(+), 21 deletions(-)
;)
Two problems:
1. I'm not sure if it's okay for me to put the test where I put it?
2. I'm not sure how to fix it for Win32 (and I also can't test it :().
It's just about the case when it's called with a Unicode path name. The current
code looks like as following:
if (PyUnicode_Check(nameobj)) {
widename = PyUnicode_AsUnicode(nameobj);
if (widename == NULL)
return -1;
}
We can't use the nifty PyUnicode_FSConverter because we want to keep Unicode.
So I assume the way to go would be the C equivalent of somthing like:
if '\0' in widename:
raise TypeError()
Right?
I hope someone would be so kind to implement it, otherwise the patch attached
passes all test on Linux and Mac (except for test_recursion_limit but that
fails currently for me on the Mac even without my patch).
--
keywords: +patch
Added file: http://bugs.python.org/file24316/open-nul.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13848
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13848] io.open() doesn't check for embedded NUL characters
New submission from Antoine Pitrou pit...@free.fr: open(\x00abc) Traceback (most recent call last): File stdin, line 1, in module FileNotFoundError: [Errno 2] No such file or directory: '' Contrast with 2.x open(): open(\x00abc) Traceback (most recent call last): File stdin, line 1, in module TypeError: file() argument 1 must be encoded string without NULL bytes, not str -- components: IO, Library (Lib) keywords: easy messages: 151876 nosy: hynek, pitrou priority: normal severity: normal stage: needs patch status: open title: io.open() doesn't check for embedded NUL characters type: behavior versions: Python 2.7, Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13848 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13848] io.open() doesn't check for embedded NUL characters
Changes by Alex Gaynor alex.gay...@gmail.com: -- nosy: +alex ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13848 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
