Re: [Python-Dev] PEP 416: Add a frozendict builtin type
Is your implementation (adapted to a standalone type) something you could put up on the cheeseshop? Short answer: no. My implementation (attached to the issue #14162) reuses most of private PyDict functins which are not exported and these functions have to be modified to accept a frozendict as input. One of the advantage of reusing PyDict functions is also to have a frozendict type compatible with the PyDict (public) API: PyDict_GetItem(), PyDict_SetItem(), etc. This property allows to do further changes like accepting a frozendict for __builtins__ or use freezing a type dict (use frozendict for type.__dict__). If you only want to a frozendict type, you can copy/paste PyDict code or implement it complelty differently. Or you can write a read-only proxy. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 updated
Nick Coghlan ncoghlan at gmail.com writes: My rewritten version of PEP 414 is now up (http://www.python.org/dev/peps/pep-0414/). It describes in detail a lot more of the historical background that was taken as read when Guido accepted the PEP. Nice work - thanks! I've implemented a first attempt at an import hook as mentioned in the PEP: https://bitbucket.org/vinay.sajip/uprefix/ It's used as follows: assume you have a simple package hierarchy of code containing u-prefixed literals: frob +-- __init__.py +-- subwob | +-- __init__.py | +-- subsubwob.py +-- wob.py with the following contents: # frob/subwob/__init__.py z = u'def' #- # frob/subwob/subsubwob.py w = u'tuv' #- # frob/__init__.py y = u'xyz' #- # frob/wob.py x = u'abc' #- You can now import these in Python 3.2 using the hook: Python 3.2.2 (default, Sep 5 2011, 21:17:14) [GCC 4.6.1] on linux2 Type help, copyright, credits or license for more information. import uprefix; uprefix.register_hook() import frob.subwob.subsubwob frob.subwob.subsubwob.w 'tuv' frob.subwob module 'frob.subwob' from 'frob/subwob/__init__.py' frob.subwob.z 'def' import frob.wob frob.wob.x 'abc' frob module 'frob' from 'frob/__init__.py' frob.y 'xyz' The import uprefix; uprefix.register_hook() is all that's needed to enable the hook. You can also unregister the hook by calling uprefix.unregister_hook(). The project is set up with a setup.py and (basic) test suite, though it's too early to put it on PyPI. I have done basic testing with it, and it should also work as expected in virtual environments. The implementation uses lib2to3, though that could be changed if needed. You can also, of course, use it in Python 3.3 right now (before the PEP gets implemented). Please take a look at it, try it out, and give some feedback. Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Assertion in _PyManagedBuffer_FromObject()
Thomas Wouters tho...@python.org wrote: Do you test against pydebug builds of Python, or otherwise a build that actually enables asserts? Yes, I do (and much more than that): http://hg.python.org/features/cdecimal/file/40917e4b51aa/Modules/_decimal/python/runall-memorydebugger.sh http://hg.python.org/features/cdecimal/file/40917e4b51aa/Modules/_decimal/python/runall.bat It's automated, so it's not a big deal. You get 100% coverage, with and without threads, all machine configurations, pydebug, refleaks, release build and release build with Valgrind. The version on PyPI has had the same tests for a long time (i.e. also before I became involved with core development). Because I suspect most people don't, so they don't trigger the assert. Python is normally (that is, a release build on Windows or a regular, non-pydebug build on the rest) built without asserts. Asserts are disabled by the NDEBUG symbol, which Python passes for regular builds. If many C-extension authors don't know the benefits of --with-pydebug and the consensus here is to protect these authors and their users, then of course I agree with the exception approach for a (now hypothetical) API change. I would have some comments about valid uses of explicit aborts in a library that essentially perform the same function as compiling said library with -D_FORTIFY_SOURCE=2 and -ftrapv (i.e. crash when an external program violates a function contract), but I suspect that would be OT now. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Defending against stack overflow (was Sandboxing Python)
Having a look at the crashers in Lib/test/crashers it seems to me that they fall into four groups. 1. Unsafe gc functions like getreferrers() 2. Stack overflows. 3. Normal bugs that can be fixed on a case-by-case basis (like borrowed_ref_1.py and borrowed_ref_2.py) 4. Things that don't crash CPython anymore and should be moved. 1. can be dealt with by removing the offending function(s), 3. by fixing the problem directly. 4. no need to fix, just move :) So, how to handle stack overflows (of the C stack)? To prevent a stack overflow an exception must be raised before the VM runs out C stack. To do this we need 2 pieces of info: a) How much stack we've used b) How much stack is available. (a) can be easily, if not strictly portably, determined by taking the address of a local variable. (b) is tougher and is almost certainly OS dependent, but a conservative estimate is easy to do. A different approach is to separate the Python stack from the C stack, like stackless. This is a much more elegant approach, but is also a *lot* more work. I think it is a reasonable aim for 3.3 that Lib/test/crashers should be empty. Cheers, Mark. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Matrix testing against CPython releases (was: Re: Assertion in _PyManagedBuffer_FromObject())
Stefan Krah, 04.03.2012 12:33: Thomas Wouters wrote: Do you test against pydebug builds of Python, or otherwise a build that actually enables asserts? Yes, I do (and much more than that): http://hg.python.org/features/cdecimal/file/40917e4b51aa/Modules/_decimal/python/runall-memorydebugger.sh http://hg.python.org/features/cdecimal/file/40917e4b51aa/Modules/_decimal/python/runall.bat It's automated, so it's not a big deal. You get 100% coverage, with and without threads, all machine configurations, pydebug, refleaks, release build and release build with Valgrind. Same for Cython. We continuously test against the debug builds of all CPython branches since 2.4 (the oldest we support), as well as the latest developer branch, using both our own test suite and Python's regression test suite. https://sage.math.washington.edu:8091/hudson/ https://sage.math.washington.edu:8091/hudson/view/python/ BTW, I can warmly recommend Jenkins' matrix builds for this kind of compatibility testing. Here's an example: https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/ Basically, you write a build script and Jenkins configures it using environment variables that define the specific setup, e.g. Python 2.7 with C backend. It'll then run all combinations in parallel (optionally filtering out nonsense combinations or preferring combinations that should fail the build early) and present the results both as an aggregated view and in separate per-setup views. It also uses file hashes to remember where the dependencies came from, e.g. which build created the CPython installation that was used for testing, so that you can jump right to the build log of the dependency to check for relevant changes that may have triggered a test failure. Oh, and you can just copy such a job config to set up a separate set of test jobs for a developer's branch, for example. A huge help in distributed developer settings, or when you want to get a GSoC student up and running. Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 updated
On 03/04/2012 10:34 AM, Vinay Sajip wrote:
https://bitbucket.org/vinay.sajip/uprefix/
import uprefix; uprefix.register_hook()
import frob.subwob.subsubwob
frob.subwob.subsubwob.w
Hi,
it's pretty cool that 150 lines is enough to have this functionality.
This guard:
if sys.version_info[0] 3:
raise NotImplementedError('This hook is implemented for Python 3 only')
Wouldn't it be better if the hook did nothing when on python 2?
I think it'll make it necessary to use something like
import sys
if sys.version_info[0] 3:
import uprefix
uprefix.register_hook()
in the calling code to enable the code to run unchanged on both branches.
Also: have you though about providing a context manager which does
register_hook() in __enter__() and unregister_hook() in __exit__()?
I think that some code will want to enable the hook only for specific
modules. The number of lines could be minimized with something like this:
import uprefix
with uprefix.hook:
import abcde_with_u
import bcdef_with_u
import other_module_without_u
Regards,
Zbyszek
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 updated
Zbigniew Jędrzejewski-Szmek zbyszek at in.waw.pl writes:
if sys.version_info[0] 3:
raise NotImplementedError('This hook is implemented for Python 3 only')
Wouldn't it be better if the hook did nothing when on python 2?
I think it'll make it necessary to use something like
Actually I've realised the guard won't be invoked on Python 2, anyway: I later
added a raise ImportError() from e in an exception handler, which leads to a
syntax error in Python 2 before the guard even gets executed.
So, I'll remove the guard (as it does nothing useful anyway) and think a bit
more about not failing on Python 2. Perhaps - not use the from syntax in the
exception handler, and do a no-op in register_hook if on Python 2.
Also: have you though about providing a context manager which does
register_hook() in __enter__() and unregister_hook() in __exit__()?
Of course, things like this can be added without too much trouble.
Thanks for the feedback.
Regards,
Vinay Sajip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 updated
Hi, It should also be added that the Python 3.3 alpha will release with support: Python 3.3.0a0 (default:042e7481c7b4, Mar 4 2012, 12:37:26) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type help, copyright, credits or license for more information. uHello + ' World!' 'Hello World!' Regards, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 updated
On Sun, Mar 4, 2012 at 11:46 PM, Armin Ronacher armin.ronac...@active-4.com wrote: Hi, It should also be added that the Python 3.3 alpha will release with support: Python 3.3.0a0 (default:042e7481c7b4, Mar 4 2012, 12:37:26) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type help, copyright, credits or license for more information. uHello + ' World!' 'Hello World!' Nice :) Do you have any more updates left to do? I saw the change, the tests, the docs and the tokenizer updates go by on python-checkins, so if you're done we can mark the PEP as Final (at which point the inclusion in the first alpha is implied). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] slice subscripts for sequences and mappings
On Sat, 3 Mar 2012 18:20:22 -0800 Thomas Wouters tho...@python.org wrote: I'm not sure how the ABCs, which are abstract declarations of semantics, tie into this specific implementation detail. ABCs work just as well for Python types as for C types, and Python types don't have this distinction. The distinction in C types has been *practically* useless for years, so why should it stay? What is the actual benefit here? For one, it's certainly easier to implement an extension type if your getitem function receives a Py_ssize_t directly, rather than a PyObject. (it can be more efficient too) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 updated
Hi, On 3/4/12 2:01 PM, Nick Coghlan wrote: Nice :) Do you have any more updates left to do? I saw the change, the tests, the docs and the tokenizer updates go by on python-checkins, so if you're done we can mark the PEP as Final (at which point the inclusion in the first alpha is implied). Docs just have a minor notice regarding the reintroduced support for 'u' prefixes, someone might want to add more to it. Especially regarding the intended use for them. Regards, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Install Hook [Was: Re: PEP 414 updated]
Hi, Jut to reiterate what I wrote on IRC: Please do not write or advocate for import hooks, especially not for porting purposes. It would either mean that people start adding that hook on their own to the code (and that awfully reminds me of the days of 'require rubygems' in the Ruby world) or that the __init__.py has to do that and that's a non trivial thing. The hook on install time works perfectly fine and the only situation where it might not work is when you're trying to use Python 3.2 for development and also support down to 2.x by using the newly introduced u-prefixes. In this case I would recommend using Python 3.3 for development and running the testsuite periodically from Python 3.2 after installating the library (into a virtualenv for instance). The current work in progress install time hook can be found here: https://github.com/mitsuhiko/unicode-literals-pep/tree/master/install-hook Regards, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxing Python
$ python execfile.py badhash.py Hang up. class badhash: __hash__ = int(42).__hash__ set([badhash() for _ in range(10)]) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxing Python
There is even easier way to exceed the time-limit timeout and to eat CPU: sum(xrange(10)). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Install / import hooks
Armin Ronacher armin.ronacher at active-4.com writes: Please do not write or advocate for import hooks, especially not for porting purposes. It would either mean that people start adding that hook on their own to the code (and that awfully reminds me of the days of 'require rubygems' in the Ruby world) or that the __init__.py has to do that and that's a non trivial thing. Well, we have to treat people as grown ups, don't we? The hook on install time works perfectly fine and the only situation where it might not work is when you're trying to use Python 3.2 for development and also support down to 2.x by using the newly introduced u-prefixes. This is exactly my use case on some projects; I can't always afford the time delay of a processing step between every edit and test (the same complaint as was made about 2to3 in the PEP). People should be free to use what's best for their use cases and development work processes. Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 updated
On Sat, Mar 3, 2012 at 11:34 PM, Nick Coghlan ncogh...@gmail.com wrote: My rewritten version of PEP 414 is now up (http://www.python.org/dev/peps/pep-0414/). It describes in detail a lot more of the historical background that was taken as read when Guido accepted the PEP. Thanks very much! It looks great to me. Can we let the interminable discussion die now? Please? Regards, Nick. P.S. If you find an actual factual *error* in the PEP, let me know by private email. If you just disagree with Guido's acceptance of the PEP, or want to quibble about my personal choice of wording on a particular point, please just let it rest. +1 -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Install Hook [Was: Re: PEP 414 updated]
On Sun, Mar 4, 2012 at 6:43 AM, Armin Ronacher armin.ronac...@active-4.com wrote: Please do not write or advocate for import hooks, especially not for porting purposes. It would either mean that people start adding that hook on their own to the code (and that awfully reminds me of the days of 'require rubygems' in the Ruby world) or that the __init__.py has to do that and that's a non trivial thing. I'd love a pointer to the rubygems debacle... The hook on install time works perfectly fine and the only situation where it might not work is when you're trying to use Python 3.2 for development and also support down to 2.x by using the newly introduced u-prefixes. In this case I would recommend using Python 3.3 for development and running the testsuite periodically from Python 3.2 after installating the library (into a virtualenv for instance). +1 The current work in progress install time hook can be found here: https://github.com/mitsuhiko/unicode-literals-pep/tree/master/install-hook Yee! -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxing Python
Hi all, On Sun, Mar 4, 2012 at 03:51, Guido van Rossum gu...@python.org wrote: Could we put asserts in the places where segfaults may happen? No. I checked Lib/test/crashers/*.py and none of them would be safe with just a failing assert. If they were, we'd have written the assert long ago :-( mutation_inside_cyclegc.py is not tied to a particular place in the source; loosing_mro_ref.py requires an extra INCREF/DECREF in a performance-critical path; etc. Changing CPython to make it truly secure is definitely either a lost cause or a real major effort, and pysandbox just gives another such example. My advise is to give up and move security at some other level. (Or else, if you want to play this game, there is PyPy's sandboxing, which is just an unpolished proof a concept so far. I can challenge anyone to attack it, and this time it includes attempts to consume too much time or memory, to crash the process in any other way than a clean fatal error! message, and more generally to exploit issues that are dismissed by pysandbox as irrelevant.) A bientôt, Armin. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxing Python
Armin Rigo wrote: Hi all, On Sun, Mar 4, 2012 at 03:51, Guido van Rossum gu...@python.org wrote: Could we put asserts in the places where segfaults may happen? No. I checked Lib/test/crashers/*.py and none of them would be safe with just a failing assert. If they were, we'd have written the assert long ago :-( mutation_inside_cyclegc.py is not tied to a particular place in the source; loosing_mro_ref.py requires an extra INCREF/DECREF in a performance-critical path; etc. Changing CPython to make it truly secure is definitely either a lost cause or a real major effort, and pysandbox just gives another such example. My advise is to give up and move security at some other level. I don't think it is as hard as all that. All the crashers can be fixed, and with minimal effect on performance. (although the gc module might need couple of function removed) (Or else, if you want to play this game, there is PyPy's sandboxing, which is just an unpolished proof a concept so far. I can challenge anyone to attack it, and this time it includes attempts to consume too much time or memory, to crash the process in any other way than a clean fatal error! message, and more generally to exploit issues that are dismissed by pysandbox as irrelevant.) Using too much memory can be dealt with at one place (in the allocator). You can't solve the too much time, without solving the halting problem, but you can make sure all code is interruptable (i.e. Cntrl-C works). Cheers, Mark. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 updated
On Mar 04, 2012, at 05:34 PM, Nick Coghlan wrote: My rewritten version of PEP 414 is now up (http://www.python.org/dev/peps/pep-0414/). It describes in detail a lot more of the historical background that was taken as read when Guido accepted the PEP. Nick, really great job with your rewrite of PEP 414. I think you nailed it from the technical side while bringing some much needed balance to the social side. Not to diminish Armin's contribution to the PEP - after all this, I'm really glad he was able to bring it up and despite the heat of the discussion, get this resolved to his satisfaction. One factual omission: In the section on WSGI native strings, you say * binary data: handled as str in Python 2 and bytes in Python 3 While true, this omits that binary data can *also* be handled as bytes in Python 2.6 and 2.7, where using `bytes` can be a more descriptive alias for `str`. If you can do it in a readable way within the context of that section I think it's worth mentioning this. Cheers, -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 updated
On Sun, 2012-03-04 at 17:34 +1000, Nick Coghlan wrote: My rewritten version of PEP 414 is now up (http://www.python.org/dev/peps/pep-0414/). It describes in detail a lot more of the historical background that was taken as read when Guido accepted the PEP. Just as support for string exceptions was eliminated from Python 2 using the normal deprecation process, support for redundant string prefix characters (specifically, B, R, u, U) may be eventually eliminated from Python 3, regardless of the current acceptance of this PEP. We might need to clarify the feature's longevity. I take the above to mean that use of u'' and/or U'' won't emit a deprecation warning in 3.3. But that doesn't necessarily mean its usage won't emit a deprecation warning in 3.4 or 3.5 or 3.6, or whenever it feels right? Does that sound about right? - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414
On Sat, Mar 3, 2012 at 5:02 AM, Lennart Regebro rege...@gmail.com wrote: I'm not sure that's true at all. In most cases where you support both Python 2 and Python 3, most strings will be native, ie, without prefix in either Python 2 or Python 3. The native case is the most common case. Exactly. The reason native strings even exist as a concept in WSGI was to make it so that the idiomatic manipulation of header data in both Python 2 and 3 would use plain old string constants with no special wrappers or markings. What's thrown the monkey wrench in here for the WSGI case is the use of unicode_literals. If you simply skip using unicode_literals for WSGI code, you should be fine with a single 2/3 codebase. But then you need some way to mark some things as unicode... which is how we end up back at this PEP. I suppose WSGI could have gone the route of using byte strings for headers instead, but I'm not sure it would have helped. The design goals for PEP were to sanely support both 2to3 and 2+3 single codebases, and WSGI does actually do that... for the code that's actually doing WSGI stuff. Ironically enough, the effect of the WSGI API is that it's all the *non* WSGI-specific code in the same module that ends up needing to mark its strings as unicode... or else it has to use unicode_literals and mark all the WSGI code with str(). There's really no good way to deal with a *mixed* WSGI/non-WSGI module, except to use explicit markers on one side or the other. Perhaps the simplest solution of all might be to just isolate direct WSGI code in modules that don't import unicode_literals. Web frameworks usually hide WSGI stuff away from the user anyway, and many are already natively unicode in their app-facing APIs. So, if a framework or library encapsulates WSGI in a str-safe/unicode-friendly API, this really shouldn't be an issue for the library's users. But I suppose somebody's got to port the libraries first. ;-) If anyone's updating porting strategy stuff, a mention of this in the tips regarding unicode_literals would be a good idea. i.e., something like: If you have 2.x modules which work with WSGI and also contain explicit u'' strings, you should not use unicode_literals unless you are willing to explicitly mark all WSGI environment and header strings as native strings using 'str()'. This is necessary because WSGI headers and environment keys/values are defined as byte strings in Python 2.x, and unicode strings in 3.x. Alternatively, you may continue to use u'' strings if you are targeting Python 3.3+ only, or can use the import or install hooks provided for Python 3.2, or if you are using 2to3... but in this case you should not use unicode_literals. That could probably be written a lot more clearly. ;-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defending against stack overflow (was Sandboxing Python)
So, how to handle stack overflows (of the C stack)? To prevent a stack overflow an exception must be raised before the VM runs out C stack. To do this we need 2 pieces of info: a) How much stack we've used b) How much stack is available. Python has already dedicated counters for stack depth, which just need proper updating and conservative values. I also think that we need to avoid allocating large arrays on the stack in recursive functions, and always heap-allocate such memory, to be stack-conservative. I think it is a reasonable aim for 3.3 that Lib/test/crashers should be empty. I agree. If you have patches to review, just put me on the nosy list. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Install Hook [Was: Re: PEP 414 updated]
Armin Ronacher armin.ronacher at active-4.com writes: The current work in progress install time hook can be found here: https://github.com/mitsuhiko/unicode-literals-pep/tree/master/install-hook I realise that the implementation is different, using tokenize rather than lib2to3, but in terms of its effect on the transformed code, what are the differences between this hook and running 2to3 with just the fix_unicode fixer? Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] slice subscripts for sequences and mappings
Thomas Wouters wrote: Why even have separate tp_as_sequence and tp_as_mapping anymore? That particular distinction never existed for Python types, so why should it exist for C types at all? I forget if there was ever a real point to it, I imagine the original motivation was to provide a fast path for types that take ints as indexes. Also, it dates from the very beginnings of Python, before it had user defined classes. At that time the archetypal sequence (list) and the archetypal mapping (dict) were very distinct -- I don't think dicts supported 'in' then, so there was no overlap. It looks like a case of it seemed like a good idea at the time. The distinction broke down fairly soon after, but it's so embedded in the extension module API that it's been very hard to get rid of. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxing Python
Maciej Fijalkowski wrote: Segfaults (most of them) can generally be made into arbitrary code execution, Can you give an example of how this can be done? -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxing Python
Mark Shannon wrote: You can't solve the too much time, without solving the halting problem, but you can make sure all code is interruptable (i.e. Cntrl-C works). If you can arrange for Ctrl-C to interrupt the process cleanly, then (at least on Unix) you can arrange to receive a signal after a timeout and recover cleanly from that as well.. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxing Python
Hi Mark, On Sun, Mar 4, 2012 at 18:34, Mark Shannon m...@hotpy.org wrote: I don't think it is as hard as all that. All the crashers can be fixed, and with minimal effect on performance. I will assume that you don't mean just to fix the files in Lib/test/crashers, but to fix the general issues that each is a particular case for. I suppose there is no point in convincing you about my point of view, so I can just say feel free and have fun. Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxing Python
Hi Greg, On Sun, Mar 4, 2012 at 22:44, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Segfaults (most of them) can generally be made into arbitrary code execution, Can you give an example of how this can be done? You should find tons of documented examples of various attacks. It's not easy, but it's possible. For example, let's assume we can decref a object to 0 before its last usage, at address x. All you need is the skills and luck to arrange that the memory at x becomes occupied by a new bigger string object allocated at x - small_number. This is enough to control exactly all the bytes that are put at address x and following, just by choosing the characters of the string. For example the bytes can be built to make address x look like a built-in function object, which you can call --- which will call an arbitrary chosen address in memory. This is enough to run arbitrary machine code and do anything. A bientôt, Armin. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxing Python
Hi Mark, On Sun, Mar 4, 2012 at 18:34, Mark Shannon m...@hotpy.org wrote: You can't solve the too much time, without solving the halting problem, Not sure what you mean by that. It seems to me that it's particularly easy to do in a roughly portable way, with alarm() for example on all UNIXes. A bientôt, Armin. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Install Hook [Was: Re: PEP 414 updated]
Hi, On 3/4/12 4:44 PM, Guido van Rossum wrote: I'd love a pointer to the rubygems debacle... Setuptools worked because Python had .pth files for a long, long time. When the Ruby world started moving packages into nonstandard locations (GameName/the files in that gem) something needed to activate that import machinery hack. For a while all Ruby projects had the line require 'rubygems' somewhere in the project. Some libraries even shipped that line to bootstrap rubygems. I think an article about that should be found here: http://tomayko.com/writings/require-rubygems-antipattern But since the page errors out currently I don't know if that is the one I'm referring to. Considering such an import hook has to run over all imports because it would not know which to rewrite and which not I think it would be equally problematic, especially if libraries would magically activate that hook. Regards, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Install Hook [Was: Re: PEP 414 updated]
Hi, On 3/4/12 9:00 PM, Vinay Sajip wrote: I realise that the implementation is different, using tokenize rather than lib2to3, but in terms of its effect on the transformed code, what are the differences between this hook and running 2to3 with just the fix_unicode fixer? I would hope they both have the same effect. Namely stripping the 'u' prefix in all variations. Why did I go with the tokenize approach? Because I never even considered a 2to3 solution. Part of the reason why I wrote this PEP was that 2to3 is so awfully slow and I was assuming that this would be largely based on the initial parsing step and not the fixers themselves. Why did I not time it with just the unicode fixer? Because if you look at how simple the tokenize version is you can see that this one did not take me more than a good minute and maybe 10 more for the distutils hooking. Regards, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxing Python
Armin Rigo wrote: Hi Mark, On Sun, Mar 4, 2012 at 18:34, Mark Shannon m...@hotpy.org wrote: You can't solve the too much time, without solving the halting problem, Not sure what you mean by that. It seems to me that it's particularly easy to do in a roughly portable way, with alarm() for example on all UNIXes. What time should you set the alarm for? How much time is enough before you decide that a piece of code is taking too long? The halting problem is not that you can't breaking out of an infinite loop, but that you can't *in general* decide when you are in an infinite loop. I think that Mark's point is that you can't, in general, tell when you are in a too much time attack (or bug) that needs to be broken out of rather than just a legitimately long calculation which will terminate if you wait just a little longer. -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Install Hook [Was: Re: PEP 414 updated]
Armin Ronacher armin.ronacher at active-4.com writes: Considering such an import hook has to run over all imports because it would not know which to rewrite and which not I think it would be equally problematic, especially if libraries would magically activate that hook. You could be right, but it sounds a little alarmist to me - problematic - magical. For example, in the current implementation of uprefix, the hook does nothing for files in the stdlib, could be refined to be more intelligent about what to run on, etc. Plus, as Zbigniew pointed out in his post, ways could be found (e.g. via a context manager) to give users control of when the hook runs. I'm not sure your rubygems example is analogous - I would have thought the equivalent for Python would be to stick import setuptools everywhere, which is not an anti-pattern we lose sleep over, AFAIK. It's early days, but it seems reasonable to document in the usage of the hook that it is intended to be used in certain ways and not in others. IIRC Ryan's post was doing just that - telling people how the requiring of rubygems should work. AFAIK the approach hasn't been tried before, and was suggested by Nick (so I assume is not completely off the wall). My particular implementation might have holes in it (feedback welcome on any such, and I'll try to fix them) but I would think the approach could be given a chance in some realistic scenarios to see what problems emerge in practice, rather than trying to shoot it down before it's even got going. Regards, Vinay Sajip It is not enough merely to win; others must lose. - Gore Vidal ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxing Python
Am 04.03.2012 23:53, schrieb Steven D'Aprano: Armin Rigo wrote: Hi Mark, On Sun, Mar 4, 2012 at 18:34, Mark Shannon m...@hotpy.org wrote: You can't solve the too much time, without solving the halting problem, Not sure what you mean by that. It seems to me that it's particularly easy to do in a roughly portable way, with alarm() for example on all UNIXes. What time should you set the alarm for? How much time is enough before you decide that a piece of code is taking too long? The halting problem is not that you can't breaking out of an infinite loop, but that you can't *in general* decide when you are in an infinite loop. I think that Mark's point is that you can't, in general, tell when you are in a too much time attack (or bug) that needs to be broken out of rather than just a legitimately long calculation which will terminate if you wait just a little longer. This is getting off-topic, but you can *certainly* solve the too much time problem without solving the halting problem. The too much time problem typically has a subjective, local, application-specific specification. Therefore, the too much time problem is *easily* solved with timeouts. Too much is just too much, even if it would eventually complete with a useful result. I'd say that a single request should not take more than 20 seconds, else it's too much. It must be less than 2 seconds for interactive use, and less than 1s if you get more than 100 requests per second. If these numbers sound arbitrary to you: they are. They are still useful to me. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Install Hook [Was: Re: PEP 414 updated]
Armin Ronacher armin.ronacher at active-4.com writes: I would hope they both have the same effect. Namely stripping the 'u' prefix in all variations. Okay, that's all I was curious about. Why did I go with the tokenize approach? Because I never even considered a 2to3 solution. Part of the reason why I wrote this PEP was that 2to3 is so awfully slow and I was assuming that this would be largely based on the initial parsing step and not the fixers themselves. Why did I not time it with just the unicode fixer? Because if you look at how simple the tokenize version is you can see that this one did not take me more than a good minute and maybe 10 more for the distutils hooking. You don't need to justify your approach - to me, anyway ;-) I suppose tokenize needed changing because of the grammar change, so it seems reasonable to put the changed version to work. I agree that 2to3 seems slow sometimes, but I can't say I've pinned it down as to exactly where the time is spent. I assumed that it was just because it seems to run over a lot of files each time, regardless of whether they've been changed since the last run or not. (I believe there might be ways of optimising that, but my understanding is that in the default/simple cases it runs over everything.) I factored out the transformation step in my hook into a method, so I should be able to swap out the lib2to3 approach with a tokenize approach without too much work, should that prove necessary or desirable. Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defending against stack overflow (was Sandboxing Python)
On Sun, Mar 4, 2012 at 12:35 PM, Martin v. Löwis mar...@v.loewis.de wrote: So, how to handle stack overflows (of the C stack)? To prevent a stack overflow an exception must be raised before the VM runs out C stack. To do this we need 2 pieces of info: a) How much stack we've used b) How much stack is available. Python has already dedicated counters for stack depth, which just need proper updating and conservative values. I also think that we need to avoid allocating large arrays on the stack in recursive functions, and always heap-allocate such memory, to be stack-conservative. I think it is a reasonable aim for 3.3 that Lib/test/crashers should be empty. I agree. If you have patches to review, just put me on the nosy list. Regards, Martin Maybe as a point of reference. PyPy, with the interpreter being largely modelled after CPython, automatically inserts about 750 checks for stack exhaustion. CPython has about 15 checks so far, I suggest looking at all the places that pypy inserts such checks is a useful start. Cheers, fijal ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] [RELEASED] Python 3.3.0 alpha 1
On behalf of the Python development team, I'm happy to announce the first alpha release of Python 3.3.0. This is a preview release, and its use is not recommended in production settings. Python 3.3 includes a range of improvements of the 3.x series, as well as easier porting between 2.x and 3.x. Major new features in the 3.3 release series are: * PEP 380, Syntax for Delegating to a Subgenerator (yield from) * PEP 393, Flexible String Representation (doing away with the distinction between wide and narrow Unicode builds) * PEP 409, Suppressing Exception Context * PEP 3151, Reworking the OS and IO exception hierarchy * The new packaging module, building upon the distribute and distutils2 projects and deprecating distutils * The new lzma module with LZMA/XZ support * PEP 3155, Qualified name for classes and functions * PEP 414, explicit Unicode literals to help with porting * The new faulthandler module that helps diagnosing crashes * Wrappers for many more POSIX functions in the os and signal modules, as well as other useful functions such as sendfile() For a more extensive list of changes in 3.3.0, see http://docs.python.org/3.3/whatsnew/3.3.html To download Python 3.3.0 visit: http://www.python.org/download/releases/3.3.0/ Please consider trying Python 3.3.0a1 with your code and reporting any bugs you may notice to: http://bugs.python.org/ Enjoy! -- Georg Brandl, Release Manager georg at python.org (on behalf of the entire python-dev team and 3.3's contributors) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
