Re: [Python-Dev] should tests be thread-safe?
On 10 May 2014 06:53, akira 4kir4...@gmail.com wrote: Hi, May tests expect that unless they themselves start a thread then there are no threads to worry about? I see that some old tests are not thread-safe and I have not found it to be explicitly mentioned in the devguide. I've written a test for http://bugs.python.org/issue21332 that is known to be non-thread-safe. Is it acceptable for new tests? Thread safety is desirable, but not mandatory, since there is some process global state (e.g. the import system and the sys module in general) that the tests sometimes need to manipulate. Cheers, Nick. -- akira ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tix version needed to build 2.7 Windows installer?
On Fri, May 9, 2014 at 12:00 PM, Zachary Ware zachary.ware+py...@gmail.com wrote: On Thu, May 8, 2014 at 4:20 PM, Zachary Ware zachary.ware+py...@gmail.com wrote: I updated the 2.7 buildbot scripts to pull in Tcl/Tk 8.5.15 a couple of weeks ago (see http://bugs.python.org/issue21303), but hadn't gotten anything done with Tix yet. It should just need python.mak updated to point to Tcl/Tk 8.5.15; it's on my list to get fixed as soon as I can. Tix for 2.7 is now http://svn.python.org/projects/external/tix-8.4.3.5. You can build it with this monster of a command, run from tix-8.4.3.5\win: nmake -f python.mak DEBUG=0 MACHINE=IX86 COMPILERFLAGS=-DWINVER=0x0500 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk all Use install instead of all after building to install it to ..\..\tcltk. Set DEBUG and MACHINE as needed; DEBUG does not need to be set if you're building Release, but MACHINE always has to be set so that Tix uses the right build dir for Tk (IX86 for 32-bit, AMD64 for 64-bit). Awesome, thanks! So I now have a fully working setup, at least for 32-bit, and have backported the Path option from 3.3 (http://hg.python.org/cpython/rev/a9d34685ec47). I ran into an issue with win32com getting a 64-bit installer built but didn't have time to look into it yet. If anyone wants to try a 2.7 installer with that Path option, here's a copy: http://briancurtin.com/python-dev/python-2.7.msi (this is not signed, it'll warn you about that). ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tix version needed to build 2.7 Windows installer?
Brian Curtin wrote: On Fri, May 9, 2014 at 12:00 PM, Zachary Ware zachary.ware+py...@gmail.com wrote: On Thu, May 8, 2014 at 4:20 PM, Zachary Ware zachary.ware+py...@gmail.com wrote: I updated the 2.7 buildbot scripts to pull in Tcl/Tk 8.5.15 a couple of weeks ago (see http://bugs.python.org/issue21303), but hadn't gotten anything done with Tix yet. It should just need python.mak updated to point to Tcl/Tk 8.5.15; it's on my list to get fixed as soon as I can. Tix for 2.7 is now http://svn.python.org/projects/external/tix-8.4.3.5. You can build it with this monster of a command, run from tix-8.4.3.5\win: nmake -f python.mak DEBUG=0 MACHINE=IX86 COMPILERFLAGS=-DWINVER=0x0500 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk all Use install instead of all after building to install it to ..\..\tcltk. Set DEBUG and MACHINE as needed; DEBUG does not need to be set if you're building Release, but MACHINE always has to be set so that Tix uses the right build dir for Tk (IX86 for 32-bit, AMD64 for 64-bit). Awesome, thanks! So I now have a fully working setup, at least for 32-bit, and have backported the Path option from 3.3 (http://hg.python.org/cpython/rev/a9d34685ec47). I ran into an issue with win32com getting a 64-bit installer built but didn't have time to look into it yet. If the error is displayed immediately after the msm filename, it probably means you don't have that file. I had to pull the right ones out of our VS build directory, since I couldn't find a public version for SP1 (and apparently the QFE that updates the 32-bit MSM doesn't update the 64-bit one... guess I'll have to go look for that build now). If anyone wants to try a 2.7 installer with that Path option, here's a copy: http://briancurtin.com/python-dev/python-2.7.msi (this is not signed, it'll warn you about that). Looks good to me, and both of my builds worked fine. Cheers, Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] python process creation overhead
I was investigating speeding up Mercurial's test suite (it runs ~13,000 Python processes) and I believe I've identified CPython process/interpreter creation and destruction as sources of significant overhead and thus a concern for any CPython user. Full details are at [1]. tl;dr 10-18% of CPU time in test suite execution was spent doing the equivalent of `python -c 1` and 30-38% of CPU time was spent doing the equivalent of `hg version` (I suspect this is mostly dominated by module importing - and Mercurial has lazy module importing). My measurements show CPython is significantly slower than Perl and Ruby when it comes to process/interpreter creation/destruction. Furthermore, Python 3 appears to be 50% slower than Python 2. Any system spawning many Python processes will likely feel this overhead. I'm also a contributor to Firefox's build and testing infrastructure. Those systems collectively spawn thousands of Python processes. While I haven't yet measured, I fear that CPython process overhead is also contributing to significant overhead there. While more science and knowledge should be added before definite conclusions are reached, I'd like to ring the figurative bell about this problem. This apparent inefficiency has me concerned about the use of CPython in scenarios where process spawning is a common operation and decent latency is important. This includes CLI tools, build systems, and testing. I'm curious if others feel this is a problem and what steps can be taken to mitigate or correct it. [1] http://www.selenic.com/pipermail/mercurial-devel/2014-May/058854.html Gregory Szorc gregory.sz...@gmail.com P.S. I'm not subscribed, so please CC me on responses. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message.
Hi python-dev and Raymond, I think this change is a considerable usability regression for the documentation. Right now the warnings about CSPRNGs are hidden in the introductory paragraph, which users are likely to skip. I agree that there's no need to repeat the same advice twice, but I'd much rather we kept the .. warning:: version, so users are more likely to actually read it. Also, there's a few errors with your commit message. First, we can reasonably assert that urandom provides an acceptable CSPRNG, mostly because it does on every platform I'm aware of. Second, urandom is still a psuedo-random number generator, however they are cryptographically secure, it's not more random. Wikipedia does a good job laying out the necessary properties for a CSPRNG: https://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator#Requirements Cheers, Alex On Sat, May 10, 2014 at 2:05 PM, raymond.hettinger python-check...@python.org wrote: http://hg.python.org/cpython/rev/b466dc34b86e changeset: 90618:b466dc34b86e parent: 90616:ce070040e1a6 user:Raymond Hettinger pyt...@rcn.com date:Sat May 10 14:05:28 2014 -0700 summary: Remove the redundant and poorly worded warning message. The paragraph above already says, clearly and correctly, that However, being completely deterministic, it is not suitable for all purposes, and is completely unsuitable for cryptographic purposes. Also we should make any promises about SystemRandom or os.urandom() being cryptographically secure (they may be, but be can't validate that promise). Further, those are actual random number generators not psuedo-random number generators. files: Doc/library/random.rst | 6 -- 1 files changed, 0 insertions(+), 6 deletions(-) diff --git a/Doc/library/random.rst b/Doc/library/random.rst --- a/Doc/library/random.rst +++ b/Doc/library/random.rst @@ -43,12 +43,6 @@ uses the system function :func:`os.urandom` to generate random numbers from sources provided by the operating system. -.. warning:: - - The pseudo-random generators of this module should not be used for - security purposes. Use :func:`os.urandom` or :class:`SystemRandom` if - you require a cryptographically secure pseudo-random number generator. - Bookkeeping functions: -- Repository URL: http://hg.python.org/cpython ___ Python-checkins mailing list python-check...@python.org https://mail.python.org/mailman/listinfo/python-checkins -- I disapprove of what you say, but I will defend to the death your right to say it. -- Evelyn Beatrice Hall (summarizing Voltaire) The people's good is the highest law. -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message.
On May 10, 2014, at 2:18 PM, Alex Gaynor alex.gay...@gmail.com wrote: I think this change is a considerable usability regression for the documentation. Right now the warnings about CSPRNGs are hidden in the introductory paragraph, which users are likely to skip In the past couple of years, we've grown an unfortunate tendency to fill the docs with big warning boxes (the subprocess docs are an example of implicitly communicating that the module is dangerous and unusable). The preferred form of documentation is to be affirmatively worded, telling how to use a tool correctly and what its known limitations are. We save the warnings for cases of actual danger where otherwise well informed users get tripped-up. Tim Peters used to be around to articulate the principle that we don't write the docs with the assumption that the users are less bright than we are or that they can't read. People writing applications that need to be sure should probably have a howto guide. I don't think they are well served by filling the module docs with every possible way a person could make a security mistake). If you're writing a secure on-line poker game, you really need to know a great deal more about security than the warning message can supply. My understanding is that actual gaming systems use seeded CSPRNGs rather than urandom() because those systems need to be auditable. Raymond P.S. The docs for the random module had a successful 20 year history without the redundant big red warning box, so it is not really correct to say there has been a considerable usability regression. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] should tests be thread-safe?
If you need a well defined environement, run your test in a subprocess. Depending on the random function, your test may be run with more threads. On BSD, it changes for example which thread receives a signal. Importing the tkinter module creates a hidden C thread for the Tk loop. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python process creation overhead
Hello, On Sat, 10 May 2014 13:05:54 -0700 Gregory Szorc gregory.sz...@gmail.com wrote: I was investigating speeding up Mercurial's test suite (it runs ~13,000 Python processes) and I believe I've identified CPython process/interpreter creation and destruction as sources of significant overhead and thus a concern for any CPython user. This was recently discussed in https://mail.python.org/pipermail/python-dev/2014-April/134028.html To give numbers: $ time python -c pass real0m0.012s user0m0.008s sys 0m0.004s $ time python -c from mercurial import demandimport; demandimport.enable(); from mercurial.dispatch import request, dispatch; dispatch(request(['version'])) Mercurial Distributed SCM (version 3.0-rc+2-d924e387604f) (see http://mercurial.selenic.com for more information) Copyright (C) 2005-2014 Matt Mackall and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. real0m0.056s user0m0.044s sys 0m0.012s So, even for hg version, most of the time is spent initializing Mercurial and its modules, not initializing Python itself. In this precise measurement, an infinitely fast Python startup would make hg version 20% faster (and other hg commands probably even less, since they have to load more Mercurial code and data). I'm wondering why Mercurial couldn't have a transparent daemon mode, where each hg invocation on the CLI would only load whatever required to communicate with the persistent daemon process. It already has a commandserver. P.S. I'm not subscribed, so please CC me on responses. I would suggest following the list using the gmane NNTP gateway. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python process creation overhead
Le 10 mai 2014 22:51, Gregory Szorc gregory.sz...@gmail.com a écrit : Furthermore, Python 3 appears to be 50% slower than Python 2. Please mention the minor version. It looks like you compared 2.7 and 3.3. Please test 3.4, we made interesting progress on the startup time. There is still something to do, especially on OS X. Depending on the OS, different modules are loaded and some functions are implemented differently. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message.
On Sat, 10 May 2014 14:35:38 -0700 Raymond Hettinger raymond.hettin...@gmail.com wrote: In the past couple of years, we've grown an unfortunate tendency to fill the docs with big warning boxes (the subprocess docs are an example of implicitly communicating that the module is dangerous and unusable). The preferred form of documentation is to be affirmatively worded, telling how to use a tool correctly and what its known limitations are. We save the warnings for cases of actual danger where otherwise well informed users get tripped-up. Tim Peters used to be around to articulate the principle that we don't write the docs with the assumption that the users are less bright than we are or that they can't read. I agree with Alex. It's not about being bright or not, it's about being *willing* to eat walls of text. However pleasant it may be for some people to *write* documentation, for most readers (and especially non-native English readers, who read more slowly and more painfully than native ones), documentation is a piece of reference that they skim from, rather than read from start to end like a novel. So it makes sense to single out useful information. Knowing that the functions from the random module are not appropriate for cryptography is (IMO) more important than knowing that Python uses the Mersenne Twister as the core generator. It produces 53-bit precision floats and has a period of 2**19937-1. The underlying implementation in C is both fast and threadsafe Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message.
On 11 May 2014 07:37, Raymond Hettinger raymond.hettin...@gmail.com wrote: On May 10, 2014, at 2:18 PM, Alex Gaynor alex.gay...@gmail.com wrote: I think this change is a considerable usability regression for the documentation. Right now the warnings about CSPRNGs are hidden in the introductory paragraph, which users are likely to skip In the past couple of years, we've grown an unfortunate tendency to fill the docs with big warning boxes (the subprocess docs are an example of implicitly communicating that the module is dangerous and unusable). The preferred form of documentation is to be affirmatively worded, telling how to use a tool correctly and what its known limitations are. We save the warnings for cases of actual danger where otherwise well informed users get tripped-up. Tim Peters used to be around to articulate the principle that we don't write the docs with the assumption that the users are less bright than we are or that they can't read. That assumption has changed somewhat, as many users are now getting their education in programming (and how computers work in general) from introductory community workshops and the Python documentation. This means that writing our docs based on the assumption that the reader is already going to be a professional programmer is no longer adequate. This is especially essential in security related areas, as even professional programmers usually aren't sufficiently paranoid about all the ways their software can be attacked. As Alex notes, the short term way to eliminate the duplication here is to keep the security warnings and drop the material from the introductory paragraph, not go back to expecting readers to have already been alerted to randomness related cryptographic security issues in some other context. Readers that are already familiar with the security concerns will hopefully recognise that they're not a Python specific problem (and may even be appreciative of our attempts to convey the relevant knowledge to a broader audience), while readers that aren't yet aware of them may be more likely to account for them appropriately when writing their software. It's as much about cultivating a more paranoid more mindset in developers in general as it is about the contents of the specific security warning. A Security Considerations section in the module documentation can be a better way to tackle this than trying to jam everything into one warning box, but there should still be a Here be dragons warning early on noting that random *is* a potentially security sensitive module in a cryptographic context. Regards, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message.
On May 10, 2014, at 6:10 PM, Nick Coghlan ncogh...@gmail.com wrote: On 11 May 2014 07:37, Raymond Hettinger raymond.hettin...@gmail.com wrote: On May 10, 2014, at 2:18 PM, Alex Gaynor alex.gay...@gmail.com wrote: I think this change is a considerable usability regression for the documentation. Right now the warnings about CSPRNGs are hidden in the introductory paragraph, which users are likely to skip In the past couple of years, we've grown an unfortunate tendency to fill the docs with big warning boxes (the subprocess docs are an example of implicitly communicating that the module is dangerous and unusable). The preferred form of documentation is to be affirmatively worded, telling how to use a tool correctly and what its known limitations are. We save the warnings for cases of actual danger where otherwise well informed users get tripped-up. Tim Peters used to be around to articulate the principle that we don't write the docs with the assumption that the users are less bright than we are or that they can't read. That assumption has changed somewhat, as many users are now getting their education in programming (and how computers work in general) from introductory community workshops and the Python documentation. This means that writing our docs based on the assumption that the reader is already going to be a professional programmer is no longer adequate. This is especially essential in security related areas, as even professional programmers usually aren't sufficiently paranoid about all the ways their software can be attacked. As Alex notes, the short term way to eliminate the duplication here is to keep the security warnings and drop the material from the introductory paragraph, not go back to expecting readers to have already been alerted to randomness related cryptographic security issues in some other context. Readers that are already familiar with the security concerns will hopefully recognise that they're not a Python specific problem (and may even be appreciative of our attempts to convey the relevant knowledge to a broader audience), while readers that aren't yet aware of them may be more likely to account for them appropriately when writing their software. It's as much about cultivating a more paranoid more mindset in developers in general as it is about the contents of the specific security warning. A Security Considerations section in the module documentation can be a better way to tackle this than trying to jam everything into one warning box, but there should still be a Here be dragons warning early on noting that random *is* a potentially security sensitive module in a cryptographic context. I completely agree with Alex, Antoine, and Nick here. I’m both an experienced Python programmer and someone who is generally aware of the security implications of various parts of software. However I appreciate when I look at documentation that explicitly calls out the ways I might screw it up, especially in a security sensitive context, but I appreciate it any context really. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message.
On May 10, 2014, at 2:54 PM, Antoine Pitrou solip...@pitrou.net wrote: It's not about being bright or not, it's about being *willing* to eat walls of text. However pleasant it may be for some people to *write* documentation, for most readers (and especially non-native English readers, who read more slowly and more painfully than native ones), documentation is a piece of reference that they skim from, rather than read from start to end like a novel. Most users of the random module documentation are just normal people trying to create random numbers. People writing secure apps with cryptographically secure random numbers are not the primary audience. But we have this big red security warning that essentially says, if you read only one thing, read this. Before proceeding further with stamping distracting security warnings all over the module documentation, we should look to other languages to see what others have found necessary. This warning does not appear anywhere else I've looked (MS Excel docs, Java docs, Go lang docs, etc.) http://docs.oracle.com/javase/6/docs/api/java/util/Random.html http://golang.org/pkg/math/rand/ Those docs are clear, concise, not preachy, and not littered with distractions. Raymond ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message.
Give it up, Raymond. On Saturday, May 10, 2014, Raymond Hettinger raymond.hettin...@gmail.com wrote: On May 10, 2014, at 2:54 PM, Antoine Pitrou solip...@pitrou.netjavascript:_e(%7B%7D,'cvml','solip...@pitrou.net'); wrote: It's not about being bright or not, it's about being *willing* to eat walls of text. However pleasant it may be for some people to *write* documentation, for most readers (and especially non-native English readers, who read more slowly and more painfully than native ones), documentation is a piece of reference that they skim from, rather than read from start to end like a novel. Most users of the random module documentation are just normal people trying to create random numbers. People writing secure apps with cryptographically secure random numbers are not the primary audience. But we have this big red security warning that essentially says, if you read only one thing, read this. Before proceeding further with stamping distracting security warnings all over the module documentation, we should look to other languages to see what others have found necessary. This warning does not appear anywhere else I've looked (MS Excel docs, Java docs, Go lang docs, etc.) http://docs.oracle.com/javase/6/docs/api/java/util/Random.html http://golang.org/pkg/math/rand/ Those docs are clear, concise, not preachy, and not littered with distractions. Raymond -- --Guido van Rossum (on iPad) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python process creation overhead
On May 10, 2014, at 5:46 PM, Victor Stinner victor.stin...@gmail.com wrote: Le 10 mai 2014 22:51, Gregory Szorc gregory.sz...@gmail.com a écrit : Furthermore, Python 3 appears to be 50% slower than Python 2. Please mention the minor version. It looks like you compared 2.7 and 3.3. Please test 3.4, we made interesting progress on the startup time. There is still something to do, especially on OS X. Depending on the OS, different modules are loaded and some functions are implemented differently. For what it's worth pip is the same way, about half of our test suite involves invoking (multiple) python processes. This has historically be really slow (~30 minutes to run ~200 tests of that type). We've been able to get the wall clock run time down by parallelizing these but the sequential time is still really slow. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message.
Hi, On Sun, May 11, 2014 at 12:35 AM, Raymond Hettinger raymond.hettin...@gmail.com wrote: On May 10, 2014, at 2:18 PM, Alex Gaynor alex.gay...@gmail.com wrote: I think this change is a considerable usability regression for the documentation. Right now the warnings about CSPRNGs are hidden in the introductory paragraph, which users are likely to skip In the past couple of years, we've grown an unfortunate tendency to fill the docs with big warning boxes If the problem is the scary red boxes, http://bugs.python.org/issue13515 still has a patch to make them less intimidating (and some interesting discussion about it). Best Regards, Ezio Melotti (the subprocess docs are an example of implicitly communicating that the module is dangerous and unusable). The preferred form of documentation is to be affirmatively worded, telling how to use a tool correctly and what its known limitations are. We save the warnings for cases of actual danger where otherwise well informed users get tripped-up. Tim Peters used to be around to articulate the principle that we don't write the docs with the assumption that the users are less bright than we are or that they can't read. People writing applications that need to be sure should probably have a howto guide. I don't think they are well served by filling the module docs with every possible way a person could make a security mistake). If you're writing a secure on-line poker game, you really need to know a great deal more about security than the warning message can supply. My understanding is that actual gaming systems use seeded CSPRNGs rather than urandom() because those systems need to be auditable. Raymond P.S. The docs for the random module had a successful 20 year history without the redundant big red warning box, so it is not really correct to say there has been a considerable usability regression. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] should tests be thread-safe?
You might have forgotten to include Python-dev in the reply. Indeed, adding it back! Thank you for the reply. I might have expressed the question poorely. I meant: I have a script that I know is not thread-safe but it doesn't matter because the test itself doesn't run any threads and the current tests are never(?) run in multiple threads (-j uses processes). Should this *new* test be fixed if e.g., there is a desire to be able to run (at least some) tests in multiple threads concurrently in the future? The short answer is: no, you don't have to make you thread thread safe, as long as it can reliably run even in the presence of background threads (like the tkinter threads Victor mentions). ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python process creation overhead
Yeah, but 200 test in 30 minutes is 9 *seconds* per test -- the Python startup time is only a tiny fraction of that (20-40 *milliseconds*). On Sat, May 10, 2014 at 3:33 PM, Donald Stufft don...@stufft.io wrote: On May 10, 2014, at 5:46 PM, Victor Stinner victor.stin...@gmail.com wrote: Le 10 mai 2014 22:51, Gregory Szorc gregory.sz...@gmail.com a écrit : Furthermore, Python 3 appears to be 50% slower than Python 2. Please mention the minor version. It looks like you compared 2.7 and 3.3. Please test 3.4, we made interesting progress on the startup time. There is still something to do, especially on OS X. Depending on the OS, different modules are loaded and some functions are implemented differently. For what it's worth pip is the same way, about half of our test suite involves invoking (multiple) python processes. This has historically be really slow (~30 minutes to run ~200 tests of that type). We've been able to get the wall clock run time down by parallelizing these but the sequential time is still really slow. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message.
On 11 May 2014 08:24, Raymond Hettinger raymond.hettin...@gmail.com wrote: Before proceeding further with stamping distracting security warnings all over the module documentation, we should look to other languages to see what others have found necessary. This warning does not appear anywhere else I've looked (MS Excel docs, Java docs, Go lang docs, etc.) http://docs.oracle.com/javase/6/docs/api/java/util/Random.html http://golang.org/pkg/math/rand/ Those docs are clear, concise, not preachy, and not littered with distractions. The fact that many (most?) programmers treat security considerations as a distraction is a core part of the problem we're trying to address. As you point out, most language development teams do very little to try to educate their users about security issues. The consequences of that are clearly visible in the world around us: when security is treated as an optional afterthought, you get widespread deployment of insecure software. At this point, we have two options: * continue with the same model as everyone else, and treat security as an optional extra users should feel free to ignore (or treat as an advanced topic only specialists need to worry about) * change our documentation practices to try to encourage the growth of a security aware development community around Python, trusting that our users will recognise that the security issues we're discussing are inherent in the way computers work, rather than being specific to Python. I'm obviously a strong advocate for the second path. Users aren't stupid, they'll figure out that almost all the security concerns we're warning about are inherent in the problem being solved, rather than being a Python-specific issue. Cheers, Nick. Raymond ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message.
Nick Coghlan, 11.05.2014 01:01: As you point out, most language development teams do very little to try to educate their users about security issues. The consequences of that are clearly visible in the world around us: when security is treated as an optional afterthought, you get widespread deployment of insecure software. At this point, we have two options: * continue with the same model as everyone else, and treat security as an optional extra users should feel free to ignore (or treat as an advanced topic only specialists need to worry about) * change our documentation practices to try to encourage the growth of a security aware development community around Python, trusting that our users will recognise that the security issues we're discussing are inherent in the way computers work, rather than being specific to Python. I'm obviously a strong advocate for the second path. Users aren't stupid, they'll figure out that almost all the security concerns we're warning about are inherent in the problem being solved, rather than being a Python-specific issue. Even if I know the problematic parts of a certain corner of software development or just of a specific tool, I prefer reading in the documentation that the authors of that tool are also aware of the (potential) problems. Makes me feel more comfortable with trusting the software. Total +1 on keeping these little bits around. Stefan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python process creation overhead
Yes right, sorry I didn’t mean to imply that all that time was spent in the Python start up time. I’ve personally never actually spent time to figure out which part of that was slow because getting visibility inside of a subprocess.Popen is a pain and I’m slowly trying to rewrite our tests to not require subprocesses at all. I was only trying to chime in with a me too here about slow subprocess tests since our test suite ends up starting a couple thousand subprocesses as well and those tests are definitely our slowest. On May 10, 2014, at 6:50 PM, Guido van Rossum gu...@python.org wrote: Yeah, but 200 test in 30 minutes is 9 *seconds* per test -- the Python startup time is only a tiny fraction of that (20-40 *milliseconds*). On Sat, May 10, 2014 at 3:33 PM, Donald Stufft don...@stufft.io wrote: On May 10, 2014, at 5:46 PM, Victor Stinner victor.stin...@gmail.com wrote: Le 10 mai 2014 22:51, Gregory Szorc gregory.sz...@gmail.com a écrit : Furthermore, Python 3 appears to be 50% slower than Python 2. Please mention the minor version. It looks like you compared 2.7 and 3.3. Please test 3.4, we made interesting progress on the startup time. There is still something to do, especially on OS X. Depending on the OS, different modules are loaded and some functions are implemented differently. For what it's worth pip is the same way, about half of our test suite involves invoking (multiple) python processes. This has historically be really slow (~30 minutes to run ~200 tests of that type). We've been able to get the wall clock run time down by parallelizing these but the sequential time is still really slow. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message.
On May 10, 2014, at 4:15 PM, Stefan Behnel stefan...@behnel.de wrote: Total +1 on keeping these little bits around. Since all of you want a warning, I'll add one back but with improved wording. I'm not all at comfortable with the wording of the second sentence. I was the author of the SystemRandom() class and I only want to guarantee that it provides access to the operating system's source of random numbers. It is a bold claim to guarantee that it is cryptographically secure (many such claims in the past have turned-out to be false). We don't really know what it is going to do on a VM for example. Also, I don't want to call SystemRandom() a pseudo-random number generator. It purports to be an actual random number generator (or at least it purports to have used some real source of entropy at some stage). To me (the module maintainer), that is an important distinction. Raymond___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Remove the redundant and poorly worded warning message.
[Raymond Hettinger] ... I'm not all at comfortable with the wording of the second sentence. I was the author of the SystemRandom() class and I only want to guarantee that it provides access to the operating system's source of random numbers. It is a bold claim to guarantee that it is cryptographically secure (many such claims in the past have turned-out to be false). We don't really know what it is going to do on a VM for example. Also, I don't want to call SystemRandom() a pseudo-random number generator. It purports to be an actual random number generator (or at least it purports to have used some real source of entropy at some stage). To me (the module maintainer), that is an important distinction. It should be sufficient to say that SystemRandom() inherits all the properties of the operating system's os.urandom() implementation, yes? Since Python has nothing to do with that, it's most accurate and helpful to tell the user to refer to their OS urandom docs. On all platforms I'm aware of (two ;-)), urandom() is in fact a CSPRNG, not an actual random number generator. That's why urandom() can get away with never blocking, potentially producing bits far faster than the system random() can accumulate fresh entropy. But all the hideous details don't belong in the Python docs - they belong in the OS's urandom docs. A phrasing I've found helpful is to tell users that urandom() is as secure as the people who wrote your operating system knew how to make it. Linux users smile then, and Windows users groan ;-) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Values and objects
[bringing back on-list] On 05/10/2014 07:30 PM, Devin Jeanpierre wrote: On Sat, May 10, 2014 at 2:38 PM, Ethan Furman et...@stoneleaf.us wrote: On 05/10/2014 02:03 PM, Devin Jeanpierre wrote: spam is referring to a local variable that has not been bound. This is not an implementation detail. The implementation detail is that in cpython there is a spot already reserved for what will be the 'spam' variable, as opposed to the module level where no such spots are reserved. I don't know what a spot reserved would even mean in a language spec, so sure. But Chris did not bring up the idea of a reserved spot in the post you were responding to, and I didn't either. Chris claimed that Python had unbound variables, you disagreed, and I pointed at the docs, which agree with Chris systematically. Chris was claiming that the exception UnboundLocalError, with complete text of: UnboundLocalError: local variable 'not_here' referenced before assignment is saying that the local variable does exist before it is assigned to; it is my contention that it does not. UnboundLocalError came into existence to aid in debugging a specific subclass of errors [1], and the text was chosen for that specific purpose. Because module level variables work differently from local variables. Not by language definition. There are pythons where modifying function locals works, but the working or not-working is not a language guarantee. I did make the mistake of saying locals, when I meant function-level locals. That was wrong. Um, what other kind of locals are there? See https://docs.python.org/3.4/library/exceptions.html#UnboundLocalError Raised when a reference is made to a local variable in a function or method, but no value has been bound to that variable. Hmm, looks like a doc-patch is needed. ;) Seriously though, error messages are chosen to provide a simple and clear description that will help the user track down what went wrong, not for enshrining in exact detail the language semantics. Would you really rather have: Traceback (most recent call last): File stdin, line 1, in module File stdin, line 2, in func UnboundLocalError: the name 'not_here' does not yet exist as you have not yet assigned anything to it so there is currently no variable by that name although at some point (in the future of this function, or perhaps in a branch that has been passed and did not execute) you will or did assign something to it so it will exist in the future of this function or may exist at this point in a future run of this function. or: Traceback (most recent call last): File stdin, line 1, in module File stdin, line 2, in func UnboundLocalError: local variable 'not_here' referenced before assignment -- ~Ethan~ [1] The conditions being that a variable was referenced before is was created, but the text of NameError offered no help in figuring that out (what do you mean it doesn't exist?!? It's right *there*!) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
