Re: [Python-Dev] cpython (merge 3.3 -> default): Issue #15539: Fix a number of bugs in Tools/scripts/pindent.py.
On 11.01.13 19:08, Brett Cannon wrote: This seems to have caused the Windows buildbots to fail. Yes, Ezio had already told me. I wrote too strong tests which caught yet one bug on Windows. Now it is fixed and all bots happy (except one). Is there a possibility to subscribe to email notifications about all buildbot's breaks in which I blamed? ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] Daily reference leaks (aef7db0d3893): sum=287
On Fri, Jan 11, 2013 at 8:08 AM, Christian Heimes wrote: > Am 11.01.2013 07:09, schrieb Nick Coghlan: > > On Fri, Jan 11, 2013 at 2:57 PM, wrote: > >> results for aef7db0d3893 on branch "default" > >> > >> > >> test_dbm leaked [2, 0, 0] references, sum=2 > >> test_dbm leaked [2, 2, 1] memory blocks, sum=5 > > > > Hmm, I'm starting to wonder if there's something to this one - it > > seems to be popping up a bit lately. > > > >> test_xml_etree_c leaked [56, 56, 56] references, sum=168 > >> test_xml_etree_c leaked [36, 38, 38] memory blocks, sum=112 > > > > I'm gonna take a wild guess and suggest there may be a problem with > > the recent pickling fix in the C extension :) > > It has more issues. Coverity has sent me some complains, see attachment. > The second report is indeed a false positive. Coverity doesn't know that PyList_GET_SIZE returns 0 for PyList_New(0). Maybe it can be taught some project/domain-specific information? The first report is legit, however. PyTuple_New(0) was called and its return value wasn't checked for NULL. I actually think Coverity is very useful for such cases because forgetting to check NULL returns from PyObject constructors is a common mistake and it's not something that would show up in tests. Anyway, this was fixed. Thanks for reporting Eli ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #15031: Refactor some code in importlib pertaining to validating
On Fri, Jan 11, 2013 at 9:55 PM, Nick Coghlan wrote:
> Nice improvement. Just a couple of minor cleanup suggestions.
>
> On Sat, Jan 12, 2013 at 9:09 AM, brett.cannon
> wrote:
>> +else:
>> +# To prevent having to make all messages have a conditional name.
>> +name = 'bytecode'
>
> For consistency with other default/implied names, I suggest wrapping
> this in angle brackets: "")
Good suggestion.
>
>> +if path is not None:
>> +exc_details['path'] = path
>> +magic = data[:4]
>> +raw_timestamp = data[4:8]
>> +raw_size = data[8:12]
>> +if magic != _MAGIC_BYTES:
>> +msg = 'bad magic number in {!r}: {!r}'.format(name, magic)
>> +raise ImportError(msg, **exc_details)
>> +elif len(raw_timestamp) != 4:
>> +message = 'bad timestamp in {!r}'.format(name)
>> +_verbose_message(message)
>> +raise EOFError(message)
>> +elif len(raw_size) != 4:
>> +message = 'bad size in {!r}'.format(name)
>> +_verbose_message(message)
>> +raise EOFError(message)
>
> For timestamp and size "incomplete" would probably be a better word
> than "bad" in the error messages (since we're only checking the length
> rather than the value).
True. Those were the original messages and in hindsight not accurate.
Since we don't consider exception messages a backwards-compatible
thing I'll update them.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (merge 3.3 -> default): Issue #15539: Fix a number of bugs in Tools/scripts/pindent.py.
On Sat, Jan 12, 2013 at 7:07 AM, Serhiy Storchaka wrote: > On 11.01.13 19:08, Brett Cannon wrote: >> >> This seems to have caused the Windows buildbots to fail. > > > Yes, Ezio had already told me. I wrote too strong tests which caught yet one > bug on Windows. Now it is fixed and all bots happy (except one). > > Is there a possibility to subscribe to email notifications about all > buildbot's breaks in which I blamed? Not that I know of. I usually leave the buildbot page up after a commit and just keep refreshing until a majority of the buildbots have completed building and are green. But hacking something together like that wouldn't be too hard. Could do that on App Engine on a free account as long as you didn't exhaust your free quota for fetching web pages, and with pubsubhubbub you probably wouldn't. Honestly the biggest pain I see is mapping commit real names to email addresses and maintaining the list of stable vs. unstable builders so people know how severe the breakage is (although I guess a break is a break and stable is more "always up" than "finicky failures"). And then from there you can do Chrome/Firefox extensions, build orbs, etc. It never ends! =) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
On Wed, Jan 9, 2013 at 12:14 AM, Guido van Rossum wrote: > But which half? A socket is two independent streams, one in each > direction. Twisted uses half_close() for this concept but unless you > already know what this is for you are left wondering which half. Which > is why I like using 'write' in the name. FWIW, "half-closed" is, IMHO, a well-known term. It's not just a Twisted thing. Either name is better than "shutdown"! Dustin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fork or exec?
2013/1/11 Ben Leslie : > Python is not UNIX, but I think if you are directly using the POSIX > interfaces they should > work (more or less) the same way the would if you were writing a C program. > (Some of us > still use Python to prototype things that will later be converted to C!). I completed the PEP to list advantages and drawbacks of the different alternatives. I moved the PEP to the PEP repository because I now consider it complete enough (for a first draft!): http://www.python.org/dev/peps/pep-0433/ I see 3 major alternatives: (a) add cloexec=False argument to some functions http://www.python.org/dev/peps/pep-0433/#proposition Drawbacks: http://www.python.org/dev/peps/pep-0433/#scope (b) always set close-on-exec flag and add cloexec=True argument to some functions (to be able to disable it) http://www.python.org/dev/peps/pep-0433/#always-set-close-on-exec-flag (c) add cloexec argument to some functions and a sys.setdefaultcloexec() function to change the default value of cloexec (will be False at startup) http://www.python.org/dev/peps/pep-0433/#add-a-function-to-set-close-on-exec-flag-by-default If you care about backward compatibility, (a) is the safest option. If you care about bugs and security, (b) is the best option. For example, (a) does not fix os.urandom(). (c) tries to be a compromise, but has its own drawbacks. Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 433: Add cloexec argument to functions creating file descriptors
HTML version: http://www.python.org/dev/peps/pep-0433/ *** PEP: 433 Title: Add cloexec argument to functions creating file descriptors Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 10-January-2013 Python-Version: 3.4 Abstract This PEP proposes to add a new optional argument ``cloexec`` on functions creating file descriptors in the Python standard library. If the argument is ``True``, the close-on-exec flag will be set on the new file descriptor. Rationale = On UNIX, subprocess closes file descriptors greater than 2 by default since Python 3.2 [#subprocess_close]_. All file descriptors created by the parent process are automatically closed. ``xmlrpc.server.SimpleXMLRPCServer`` sets the close-on-exec flag of the listening socket, the parent class ``socketserver.BaseServer`` does not set this flag. There are other cases creating a subprocess or executing a new program where file descriptors are not closed: functions of the os.spawn*() family and third party modules calling ``exec()`` or ``fork()`` + ``exec()``. In this case, file descriptors are shared between the parent and the child processes which is usually unexpected and causes various issues. This PEP proposes to continue the work started with the change in the subprocess, to fix the issue in any code, and not just code using subprocess. Inherited file descriptors issues - Closing the file descriptor in the parent process does not close the related resource (file, socket, ...) because it is still open in the child process. The listening socket of TCPServer is not closed on ``exec()``: the child process is able to get connection from new clients; if the parent closes the listening socket and create a new listening socket on the same address, it would get an "address already is used" error. Not closing file descriptors can lead to resource exhaustion: even if the parent closes all files, creating a new file descriptor may fail with "too many files" because files are still open in the child process. Security Leaking file descriptors is a major security vulnerability. An untrusted child process can read sensitive data like passwords and take control of the parent process though leaked file descriptors. It is for example a known vulnerability to escape from a chroot. Atomicity - Using ``fcntl()`` to set the close-on-exec flag is not safe in a multithreaded application. If a thread calls ``fork()`` and ``exec()`` between the creation of the file descriptor and the call to ``fcntl(fd, F_SETFD, new_flags)``: the file descriptor will be inherited by the child process. Modern operating systems offer functions to set the flag during the creation of the file descriptor, which avoids the race condition. Portability --- Python 3.2 added ``socket.SOCK_CLOEXEC`` flag, Python 3.3 added ``os.O_CLOEXEC`` flag and ``os.pipe2()`` function. It is already possible to set atomically close-on-exec flag in Python 3.3 when opening a file and creating a pipe or socket. The problem is that these flags and functions are not portable: only recent versions of operating systems support them. ``O_CLOEXEC`` and ``SOCK_CLOEXEC`` flags are ignored by old Linux versions and so ``FD_CLOEXEC`` flag must be checked using ``fcntl(fd, F_GETFD)``. If the kernel ignores ``O_CLOEXEC`` or ``SOCK_CLOEXEC`` flag, a call to ``fcntl(fd, F_SETFD, flags)`` is required to set close-on-exec flag. .. note:: OpenBSD older 5.2 does not close the file descriptor with close-on-exec flag set if ``fork()`` is used before ``exec()``, but it works correctly if ``exec()`` is called without ``fork()``. Scope - Applications still have to close explicitly file descriptors after a ``fork()``. The close-on-exec flag only closes file descriptors after ``exec()``, and so after ``fork()`` + ``exec()``. This PEP only change the close-on-exec flag of file descriptors created by the Python standard library, or by modules using the standard library. Third party modules not using the standard library should be modified to conform to this PEP. The new ``os.set_cloexec()`` function can be used for example. Impacted functions: * ``os.forkpty()`` * ``http.server.CGIHTTPRequestHandler.run_cgi()`` Impacted modules: * ``multiprocessing`` * ``socketserver`` * ``subprocess`` * ``tempfile`` * ``xmlrpc.server`` * Maybe: ``signal``, ``threading`` XXX Should ``subprocess.Popen`` set the close-on-exec flag on file XXX XXX descriptors of the constructor the ``pass_fds`` argument? XXX .. note:: See `Close file descriptors after fork`_ for a possible solution for ``fork()`` without ``exec()``. Proposal This PEP proposes to add a new optional argument ``cloexec`` on functions creating file descriptors in the Python standard library. If the argument is ``True``, the close-on-exec flag will be set on the ne
Re: [Python-Dev] PEP 433: Add cloexec argument to functions creating file descriptors
> PEP: 433 > Title: Add cloexec argument to functions creating file descriptors > Status: Draft The PEP is still a draft. I'm sending it to python-dev to get a first review. The main question is the choice between the 3 different options: * don't set close-on-exec flag by default * always set close-on-exec flag * add sys.setdefaultcloexec() to leave the choice to the application Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
