Re: [Python-Dev] Non-string keys in type dict
Nick Coghlan wrote: On Thu, Mar 8, 2012 at 11:42 AM, Benjamin Peterson benja...@python.org wrote: 2012/3/7 Victor Stinner victor.stin...@gmail.com: Can't we simply raise an error if the dict contains non-string keys? Sounds okay to me. For 3.3, the most we can do is trigger a deprecation warning, since removing this feature *will* break currently running code. I don't have any objection to us starting down that path, though. I think it would be sad to lose that functionality. If we are going to, though, we may as well check the string to make sure it's a valid identifier: -- class A: -- pass -- setattr(A, '42', 'hrm') -- A.42 File stdin, line 1 A.42 ^ SyntaxError: invalid syntax Doesn't seem very useful. ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Non-string keys in type dict
On Thu, Mar 8, 2012 at 08:46, Ethan Furman et...@stoneleaf.us wrote: I think it would be sad to lose that functionality. If we are going to, though, we may as well check the string to make sure it's a valid identifier: That would break even more code. I have encountered many cases of attributes that aren't valid identifiers, in particular using dots or dashes. Admittedly this is often in cases where the object has both attribute access and key access, so you can make foo['bar-frotz'] instead. But when should we then require that it is a valid identifier and when not? -- class A: -- pass -- setattr(A, '42', 'hrm') -- A.42 File stdin, line 1 A.42 ^ SyntaxError: invalid syntax Doesn't seem very useful. You have to set it with setattr, so you have to get it with getattr. I don't see the problem. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
Stefan Behnel wrote: Hi, I found a problem in the current yield from implementation that I think is worth discussing: http://bugs.python.org/issue14220 [snip] I've been experimenting with the implementation of PEP 380, and I found a couple of interesting things. First of all, the semantics described in the PEP do not match the tests. If you substitute the supposedly semantically equivalent code based on normal yields for each yield from in the test code (Lib/test/test_pep380.py) and run it, then it fails. My second experiment involved stripping away all the code relating to yield from outside of the interpreter and changing the YIELD_FROM bytecode to repeat itself, by setting the last instruction to the instruction immediately before itself. To do this I added 4 lines of code and removed over 120 lines :) This fails many of the tests*, but works for the most straightforward use cases. Many of these failures seem to me to be more 'natural' than the current behaviour It might be possible to fix most, or all?, of the other failures by compiling yield from into a two opcode sequence: YIELD_FROM_START and YIELD_FROM_REPEAT. Both opcodes should be reasonably simple and __next__() and send() would not have to worry about the subiterator, although close() and throw() might (the subiterator would be top-of-stack in the generator's frame) Overall the semantics of PEP 380 seem far too complex, trying to do several things at once. An example: Plain yield makes no distinction between a receiving a None and any other value, so send(None) and __next__() are the same. Yield from makes this distinction so has to test for None, meaning the semantics of send now changes with its argument. I would recommend changing one of two things in the PEP: Either, close and throw should not close/throw in subiterators (this would simplify the semantics and implementation immensely) Or, only allow subgenerators, not subiterators (this would fix the next/send problem). I would also suggest a change in implementation to perform all yields within the interpreter. A simpler implementation is likely to be a more reliable one. Finally, the PEP itself makes no mention of coroutines, stackless or greenlet in the alternatives or criticisms section. Perhaps it should. Cheers, Mark. *Tests in PEP 380. It passes all other tests. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
On Thu, Mar 8, 2012 at 9:52 PM, Mark Shannon m...@hotpy.org wrote: I would recommend changing one of two things in the PEP: Either, close and throw should not close/throw in subiterators (this would simplify the semantics and implementation immensely) Or, only allow subgenerators, not subiterators (this would fix the next/send problem). Either of those changes would completely defeat the point of the PEP. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding a builtins parameter to eval(), exec() and __import__().
Jim J. Jewett wrote: http://mail.python.org/pipermail/python-dev/2012-March/117395.html Brett Cannon posted: [in reply to Mark Shannon's suggestion of adding a builtins parameter to match locals and globals] It's a mess right now to try to grab the __import__() implementation and this would actually help clarify import semantics by saying that __import__() for any chained imports comes from __import__()s locals, globals, or builtins arguments (in that order) or from the builtins module itself (i.e. tstate-builtins). How does that differ from today? The idea is that you can change, presumable restrict, the builtins separately from the globals for an import. If you're saying that the locals and (module-level) globals aren't always checked in order, then that is a semantic change. Probably a good change, but still a change -- and it can be made indepenently of Mark's suggestion. Also note that I would assume this was for sandboxing, Actually, I just think it's a cleaner implementation, but sandboxing is a good excuse :) and that missing names should *not* fall back to the real globals, although I would understand if bootstrapping required the import statement to get special treatment. (Note that I like Mark's proposed change; I just don't see how it cleans up import.) I don't think it cleans up import, but I'll defer to Brett on that. I've included __import__() along with exec and eval as it is a place where new namespaces can be introduced into an execution. There may be others I haven't though of. Cheers, Mark. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
On Thu, Mar 8, 2012 at 9:52 PM, Mark Shannon m...@hotpy.org wrote: First of all, the semantics described in the PEP do not match the tests. If you substitute the supposedly semantically equivalent code based on normal yields for each yield from in the test code (Lib/test/test_pep380.py) and run it, then it fails. What's more important is whether or not it matches the semantics of inlining the subgenerator bodies. The expansion in the PEP was an attempt to define a way to achieve that in current Python without interpreter support. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding a builtins parameter to eval(), exec() and __import__().
On Thu, Mar 8, 2012 at 10:06 PM, Mark Shannon m...@hotpy.org wrote: I don't think it cleans up import, but I'll defer to Brett on that. I've included __import__() along with exec and eval as it is a place where new namespaces can be introduced into an execution. There may be others I haven't though of. runpy is another one. However, the problem I see with builtins as a separate argument is that it would be a lie. The element that's most interesting about locals vs globals vs builtins is the scope of visibility of their contents. When I call out to another function in the same module, locals are not shared, but globals and builtins are. When I call out to code in a *different* module, neither locals nor globals are shared, but builtins are still common. So there are two ways this purported extra builtins parameter could work: 1. Sandboxing - you try to genuinely give the execution context a different set of builtins that's shared by all code executed, even imports from other modules. However, I assume this isn't what you meant, since it is the domain of sandboxing utilities like Victor's pysandbox and is known to be incredibly difficult to get right (hence the demise of both rexec and Bastion and recent comments about known segfault vulnerabilities that are tolerable in the normal case of merely processing untrusted data with trusted code but anathema to a robust CPython native sandboxing scheme that can still cope even when the code itself is untrusted). 2. chained globals - just an extra namespace that's chained behind the globals dictionary for name lookup, not actually shared with code invoked from other modules. The second approach is potentially useful, but: 1. builtins is *not* the right name for it (because any other code invoked will still be using the original builtins) 2. it's already trivial to achieve such chained lookups in 3.3 by passing a collections.ChainMap instance as the globals parameter: http://docs.python.org/dev/library/collections#collections.ChainMap collections.ChainMap also has the virtue of working with any current API that accepts a globals argument and can be extended to an arbitrary level of chaining, whereas this suggestion requires that all such APIs be expanded to accept a third parameter, and could still only chain lookups one additional step in doing so. So a big -1 from me. Cheers, Nick. P.S. I've referenced this talk before, but Tim Dawborn's effort from PyCon AU last year about the sandboxing setup for http://www.ncss.edu.au/ should be required viewing for anyone wanting to understand the kind of effort it takes to fairly comprehensively protect host servers from attacks when executing arbitrary untrusted Python code on CPython. Implementing such protection is certainly *possible* (since Tim's talk is all about one way to do it), but it's not easy, and Tim's approach uses Linux OS level sandboxing rather than rather than relying on a Python language level sandbox. This was largely due to a university requirement that the sandbox solution be language agnostic, but it also serves to protect the sandbox from the documented attacks against the CPython interpreter. Tim reviews a few interesting attempts to break the sandbox around the 5 minute mark in https://www.youtube.com/watch?v=y-WPPdhTKBU. (I did suggest he grab our test_crashers directory to see what happened when they were run in the sandbox, but I doubt it would be much more interesting than merely calling sys.exit()) -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] steps to solve bugs
Hi, I am a novice python programmer and am learning to be able to solve some issues. Well following the steps given in the PSF website, I have * installed vc++ 2008 and even finished till building the cpython code and I have got the console for python 3.0x Having done this, I am not able to quite follow the further steps to solve the bugs . Currently I am wondering in the issues tracker though not still working on it. So I request you my kind folks to kindly make me understand the further things I must have to take up to get it done. Thanks Regards, Ejaj ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] steps to solve bugs
On Thu, Mar 8, 2012 at 11:09 PM, Ejaj Hassan ejjy...@gmail.com wrote: Hi, I am a novice python programmer and am learning to be able to solve some issues. Well following the steps given in the PSF website, I have * installed vc++ 2008 and even finished till building the cpython code and I have got the console for python 3.0x Having done this, I am not able to quite follow the further steps to solve the bugs . Currently I am wondering in the issues tracker though not still working on it. So I request you my kind folks to kindly make me understand the further things I must have to take up to get it done. Hi Ejaj, If you're interested in getting started working on items on the issue tracker, I suggest signing up for the core-mentorship list (core-mentors...@python.org) and reposting your question there. It's specifically for this kind of question, whereas python-dev is intended for design and development discussions. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] steps to solve bugs
On Thu, Mar 08, 2012 at 06:39:42PM +0530, Ejaj Hassan wrote: I am a novice python programmer and am learning to be able to solve some issues. Well following the steps given in the PSF website, I have installed vc++ 2008 and even finished till building the cpython code and I have got the console for python 3.0x Having done this, I am not able to quite follow the further steps to solve the bugs . Currently I am wondering in the issues tracker though not still working on it. Are you debugging the Python core, the standard Python library or your own code written in Python? Except for the core you don't need to recompile the Python interpreter - just download ready one. If you are debugging your own code - this mailing list is a wrong place to look for help. This mailing list is to work on developing Python (adding new features to Python itself and fixing bugs); if you're having problems learning, understanding or using Python, please find another forum. Probably python-list/comp.lang.python mailing list/news group is the best place; there are Python developers who participate in it; you may get a faster, and probably more complete, answer there. See http://www.python.org/community/ for other lists/news groups/fora. Oleg. -- Oleg Broytmanhttp://phdru.name/p...@phdru.name Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding a builtins parameter to eval(), exec() and __import__().
Nick Coghlan wrote: On Thu, Mar 8, 2012 at 10:06 PM, Mark Shannon m...@hotpy.org wrote: I don't think it cleans up import, but I'll defer to Brett on that. I've included __import__() along with exec and eval as it is a place where new namespaces can be introduced into an execution. There may be others I haven't though of. runpy is another one. Add that to the list. However, the problem I see with builtins as a separate argument is that it would be a lie. The element that's most interesting about locals vs globals vs builtins is the scope of visibility of their contents. When I call out to another function in the same module, locals are not shared, but globals and builtins are. When I call out to code in a *different* module, neither locals nor globals are shared, but builtins are still common. Not necessarily. All functions in a module will inherit their globals *and* builtins from the module, which gets them from __import__(). So there are two ways this purported extra builtins parameter could work: 1. Sandboxing - you try to genuinely give the execution context a different set of builtins that's shared by all code executed, even imports from other modules. Victor's pysandbox seems pretty good to me, I had a go at breaking it and failed, but it is too restrictive. Rather than make pysandbox more secure, I think my proposal could make it more usable, as clearer guarantees about access and visibility can be provided to the sandbox developer. You shouldn't need to cripple introspection in order to limit access to the builtins. However, I assume this isn't what you meant, since it is the domain of sandboxing utilities like Victor's pysandbox and is known to be incredibly difficult to get right (hence the demise of both rexec and Bastion and recent comments about known segfault vulnerabilities that are tolerable in the normal case of merely processing untrusted data with trusted code but anathema to a robust CPython native sandboxing scheme that can still cope even when the code itself is untrusted). By changing the implementation to be based around immutable execution contexts means that the compiler will enforce things for us. Static typing has its advantages, occasionally :) As I stated elsewhere, the crashers can be fixed. I think Victor has already fixed a couple. 2. chained globals - just an extra namespace that's chained behind the globals dictionary for name lookup, not actually shared with code invoked from other modules. That's exactly what builtins already are. They are a fall back for LOAD_GLOBAL and similar when something isn't found in the globals. The second approach is potentially useful, but: 1. builtins is *not* the right name for it (because any other code invoked will still be using the original builtins) Other code will use whatever builtins they were given at __import__. The key point is that every piece of code already inherits locals, globals and builtins from somewhere else. We can already control locals (by which parameters are passed in) and globals via exec, eval, __import__, and runpy (any others?) but we can't control builtins. One last point is that this is a low-impact change. All code using eval, etc. will continue to work as before. It also may speed things up a little. Cheers, Mark. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
Nick Coghlan wrote: On Thu, Mar 8, 2012 at 9:52 PM, Mark Shannon m...@hotpy.org wrote: First of all, the semantics described in the PEP do not match the tests. If you substitute the supposedly semantically equivalent code based on normal yields for each yield from in the test code (Lib/test/test_pep380.py) and run it, then it fails. What's more important is whether or not it matches the semantics of inlining the subgenerator bodies. The expansion in the PEP was an attempt to define a way to achieve that in current Python without interpreter support. So yield from X means inline X here, if X is a generator. That is much, much easier to understand than the big block of code in the PEP. It really ought say that yield from is equivalent to inlining in the PEP. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 412 has been accepted
On Wed, 7 Mar 2012 18:52:43 -0500 Brett Cannon br...@yvrsfo.ca wrote: Since PEP 412 has code that doesn't break tests anymore (thanks to hash randomization), it was just accepted. Mark, can you make sure there is an up-to-date patch in the tracker so people can potentially look at the code at the sprints here at PyCon? And also please apply for core dev privileges (http://docs.python.org/devguide/coredev.html) so that we can make you fix bugs. =) For the record (I had to look it up), PEP 412 is Mark Shannon's Key-Sharing Dictionary, an optimization that decreases memory consumption of instances. http://www.python.org/dev/peps/pep-0412/ Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding a builtins parameter to eval(), exec() and __import__().
On Thu, Mar 8, 2012 at 11:40 PM, Mark Shannon m...@hotpy.org wrote: Other code will use whatever builtins they were given at __import__. Then they're not builtins - they're module-specific chained globals. The thing that makes the builtins special is *who else* can see them (i.e. all the other code in the process). If you replace builtins.open, you replace if for everyone (that hasn't either shadowed it or cached a reference to the original). The key point is that every piece of code already inherits locals, globals and builtins from somewhere else. We can already control locals (by which parameters are passed in) and globals via exec, eval, __import__, and runpy (any others?) but we can't control builtins. Correct - because controlling builtins is the domain of sandboxes. One last point is that this is a low-impact change. All code using eval, etc. will continue to work as before. It also may speed things up a little. Passing in a ChainMap instance as the globals when you want to include an additional namespace in the lookup chain is even lower impact. A reference implementation and concrete use cases might change my mind, but for now, I'm just seeing a horrendously complicated approach with huge implications for the runtime data model semantics for something that 3.3 already supports in a much simpler fashion. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Request for clarification of PEP 380
Hi, The scenario is this: A generator, G, has a non-generator sub-iterator, S, (ie G includes a yield from S experssion and S is not a generator) and either G.close() or G.throw(GeneratorExit) is called. In the current implementation, S.close() is called and, if that call raises an exception, then that exception is suppressed. Should close() be called at all? I know that it helps non-generators to support the protocol, but there is the problem of iterables that happen to have a close method. This may cause unwanted side effects. Why is the exception suppressed? The text of the PEP seems to implicitly assume that all sub-iterators will be generators, so it is not clear on the above points. Cheers, Mark. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
On Thu, Mar 8, 2012 at 11:45 PM, Mark Shannon m...@hotpy.org wrote: It really ought say that yield from is equivalent to inlining in the PEP. That's what the motivation section is about. There's also an entire subsection called The Refactoring Principle that describes this intent. However, we needed something more concrete to flesh out the original implementation details, which is what the code equivalent was designed to provide (it was also designed to point out that the days of you can just use a simple for loop actually went away when PEP 342 was implemented). Now, it may be that we fixed things during implementation that should be reflected back into the formal semantic spec in the PEP - so if you can point out specific cases where what we implemented doesn't match the nominal behaviour, I'm open to updating the PEP accordingly. (Of course, if there are any tests that fail solely due to the two known differences in semantics that are already noted in the PEP, they don't count). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Request for clarification of PEP 380
On Fri, Mar 9, 2012 at 12:06 AM, Mark Shannon m...@hotpy.org wrote: The text of the PEP seems to implicitly assume that all sub-iterators will be generators, so it is not clear on the above points. On the contrary, this question is explicitly addressed in the PEP: http://www.python.org/dev/peps/pep-0380/#finalization If you want to block an inadvertent close() call, you need to either wrap the object so it doesn't implement unwanted parts of the generator API (which is the iterator protocol plus send(), throw() and close()), or else you should continue to use simple iteration rather than delegation. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why does Mac OS X python share site-packages with apple python?
Just to follow up. With Robin's help over in wxPython land I have given Robin a patch to wxPython to fix the site-packages issue. Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Request for clarification of PEP 380
Nick Coghlan wrote: On Fri, Mar 9, 2012 at 12:06 AM, Mark Shannon m...@hotpy.org wrote: The text of the PEP seems to implicitly assume that all sub-iterators will be generators, so it is not clear on the above points. On the contrary, this question is explicitly addressed in the PEP: http://www.python.org/dev/peps/pep-0380/#finalization I should have read it more carefully. If you want to block an inadvertent close() call, you need to either wrap the object so it doesn't implement unwanted parts of the generator API (which is the iterator protocol plus send(), throw() and close()), or else you should continue to use simple iteration rather than delegation. What about the exception being suppressed? That doesn't seem to be mentioned. Cheers, Mark. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Non-string keys in type dict
On Thu, Mar 8, 2012 at 1:10 AM, Lennart Regebro rege...@gmail.com wrote: On Thu, Mar 8, 2012 at 08:46, Ethan Furman et...@stoneleaf.us wrote: I think it would be sad to lose that functionality. If we are going to, though, we may as well check the string to make sure it's a valid identifier: That would break even more code. I have encountered many cases of attributes that aren't valid identifiers, in particular using dots or dashes. Admittedly this is often in cases where the object has both attribute access and key access, so you can make foo['bar-frotz'] instead. But when should we then require that it is a valid identifier and when not? -- class A: -- pass -- setattr(A, '42', 'hrm') -- A.42 File stdin, line 1 A.42 ^ SyntaxError: invalid syntax Doesn't seem very useful. You have to set it with setattr, so you have to get it with getattr. I don't see the problem. I'm with Lennart. I've spoken out on this particular question several times before; it is a *feature* that you can use any arbitrary string with getattr() and setattr(). However these functions should (and do!) reject non-strings. I'm lukewarm on forbidding non-strings in namespace dicts if you can get them in there by other means. I presume that some people might use this to hide state they don't want accessible using regular attribute notation (x.foo) and maybe somebody is even using some namespace's keys as a dict to store many things with non-string keys, but that feels like abuse of the namespace to me, and there are plenty of other ways to manage such state, so if it gives a significant speed-up to use string-only dicts, so be it. (Although Python's dicts already have some string-only optimizations -- they just dynamically adapt to a more generic and slightly slower approach once the first non-key string shows up.) As Nick says we should introduce a deprecation period if we do this. On Wed, Mar 7, 2012 at 11:43 PM, Ethan Furman et...@stoneleaf.us wrote: Are you able to modify classes after class creation in Python 3? Without using a metaclass? Yes, by assignment to attributes. The __dict__ is a read-only proxy, but attribute assignment is allowed. (This is because the new type system introduced in Python 2.2 needs to *track* changes to the dict; it does this by tracking setattr/delattr calls, because dict doesn't have a way to trigger a hook on changes.) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Non-string keys in type dict
Guido van Rossum wrote:
On Wed, Mar 7, 2012 at 11:43 PM, Ethan Furman wrote:
Are you able to modify classes after class creation in Python 3? Without
using a metaclass?
Yes, by assignment to attributes. The __dict__ is a read-only proxy,
but attribute assignment is allowed. (This is because the new type
system introduced in Python 2.2 needs to *track* changes to the dict;
it does this by tracking setattr/delattr calls, because dict doesn't
have a way to trigger a hook on changes.)
Poorly phrased question -- I meant is it possible to add non-string-name
attributes to classes after class creation. During class creation we
can do this:
-- class Test:
... ns = vars()
... ns[42] = 'green eggs'
... del ns
...
-- Test
class '__main__.Test'
-- Test.__dict__
dict_proxy({
'__module__': '__main__',
42: 'green eggs',
'__doc__': None,
'__dict__': attribute '__dict__' of 'Test' objects,
'__weakref__': attribute '__weakref__' of 'Test' objects,
'__locals__': {
42: 'green eggs',
'__module__': '__main__',
'__locals__': {...}}
})
-- Test.__dict__[42]
'green eggs'
A little more experimentation shows that not all is well, however:
-- dir(Test)
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: unorderable types: int() str()
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Non-string keys in type dict
On Thu, Mar 8, 2012 at 8:22 AM, Ethan Furman et...@stoneleaf.us wrote:
Guido van Rossum wrote:
On Wed, Mar 7, 2012 at 11:43 PM, Ethan Furman wrote:
Are you able to modify classes after class creation in Python 3? Without
using a metaclass?
Yes, by assignment to attributes. The __dict__ is a read-only proxy,
but attribute assignment is allowed. (This is because the new type
system introduced in Python 2.2 needs to *track* changes to the dict;
it does this by tracking setattr/delattr calls, because dict doesn't
have a way to trigger a hook on changes.)
Poorly phrased question -- I meant is it possible to add non-string-name
attributes to classes after class creation. During class creation we can do
this:
-- class Test:
... ns = vars()
... ns[42] = 'green eggs'
... del ns
...
-- Test
class '__main__.Test'
-- Test.__dict__
dict_proxy({
'__module__': '__main__',
42: 'green eggs',
'__doc__': None,
'__dict__': attribute '__dict__' of 'Test' objects,
'__weakref__': attribute '__weakref__' of 'Test' objects,
'__locals__': {
42: 'green eggs',
'__module__': '__main__',
'__locals__': {...}}
})
-- Test.__dict__[42]
'green eggs'
A little more experimentation shows that not all is well, however:
-- dir(Test)
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: unorderable types: int() str()
So what conclusion do you draw?
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Non-string keys in type dict
On Thu, Mar 8, 2012 at 2:43 AM, Ethan Furman et...@stoneleaf.us wrote: PJ Eby wrote: Short version: AddOns are things you can use to dynamically extend instances -- a bit like the decorator in decorator pattern (not to be confused with Python decorators). Rather than synthesize a unique string as a dictionary key, I just used the AddOn classes themselves as keys. This works fine for object instances, but gets hairy once classes come into play. Are you able to modify classes after class creation in Python 3? Without using a metaclass? For ClassAddOns, it really doesn't matter; you can't remove them from the class they attach to. Addons created after the class is finalized use a weakref dictionary to attach to their classes. Now that I've gone back and looked at the code, the only reason that ClassAddOns even use the class __dict__ in the first place is because it's a convenient place to put them while the class is being built. With only slightly hairier code, I could use an __addons__ dict in the class namespace while it's being built, but there'll then be a performance hit at look up time to do cls.__dict__['__addons__'][key] instead of cls.__dict__[key]. Actually, now that I'm thinking about it, the non-modifiability of class dictionaries is actually a feature for this use case: if I make an __addons__ dict, that dict is mutable. That means I'll have to move to string keys or have some sort of immutable dict type available... ;-) (Either that, or do some other, more complex refactoring.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Non-string keys in type dict
On Wed, Mar 7, 2012 at 5:39 PM, Victor Stinner victor.stin...@gmail.com wrote: Hi, During the Language Summit 2011 (*), it was discussed that PyPy and Jython don't support non-string key in type dict. An issue was open to emit a warning on such dict, but the patch has not been commited yet. It should be noted that Jython started supporting non-string dict keys in version 2.5. IIRC this was done to get Django running, but in general we decided to go with compatibility. -Frank ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding a builtins parameter to eval(), exec() and __import__().
On Thu, Mar 8, 2012 at 5:57 AM, Nick Coghlan ncogh...@gmail.com wrote: On Thu, Mar 8, 2012 at 11:40 PM, Mark Shannon m...@hotpy.org wrote: Other code will use whatever builtins they were given at __import__. Then they're not builtins - they're module-specific chained globals. The thing that makes the builtins special is *who else* can see them (i.e. all the other code in the process). If you replace builtins.open, you replace if for everyone (that hasn't either shadowed it or cached a reference to the original). Looks like you two are talking about different things. There is only one 'builtins' *module*. But the __builtins__ that are actually used by any particular piece of code is *not* taken by importing builtins. It is taken from what the globals store under the key __builtins__. This is a feature that was added specifically for sandboxing purposes, but I believe it has found other uses too. The key point is that every piece of code already inherits locals, globals and builtins from somewhere else. We can already control locals (by which parameters are passed in) and globals via exec, eval, __import__, and runpy (any others?) but we can't control builtins. Correct - because controlling builtins is the domain of sandboxes. Incorrect (unless I misunderstand the context) -- when you control the globals you control the __builtins__ set there. One last point is that this is a low-impact change. All code using eval, etc. will continue to work as before. It also may speed things up a little. Passing in a ChainMap instance as the globals when you want to include an additional namespace in the lookup chain is even lower impact. A reference implementation and concrete use cases might change my mind, but for now, I'm just seeing a horrendously complicated approach with huge implications for the runtime data model semantics for something that 3.3 already supports in a much simpler fashion. I can't say I'm completely following the discussion. It's not clear whether what I just explained was already implicit in the coversation or is new information. In any case, the locals / globals / builtins chain is a simplification; there are also any number of intermediate scopes (between locals and globals) from which nonlocal variables may be used. Like optimized function globals, these don't use a dict lookup at all, they are determined by compile-time analysis. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 412 has been accepted
On Thu, Mar 8, 2012 at 5:41 AM, Antoine Pitrou solip...@pitrou.net wrote: For the record (I had to look it up), PEP 412 is Mark Shannon's Key-Sharing Dictionary, an optimization that decreases memory consumption of instances. http://www.python.org/dev/peps/pep-0412/ Thanks for reminding us. I've gotten into the habit of listing the PEP number *and* title whenever first referencing a given PEP in a discussion -- if the PEP is mentioned in the message subject, its title should be too. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Non-string keys in type dict
Guido van Rossum wrote:
On Thu, Mar 8, 2012 at 8:22 AM, Ethan Furman et...@stoneleaf.us wrote:
Guido van Rossum wrote:
On Wed, Mar 7, 2012 at 11:43 PM, Ethan Furman wrote:
Are you able to modify classes after class creation in Python 3? Without
using a metaclass?
Yes, by assignment to attributes. The __dict__ is a read-only proxy,
but attribute assignment is allowed. (This is because the new type
system introduced in Python 2.2 needs to *track* changes to the dict;
it does this by tracking setattr/delattr calls, because dict doesn't
have a way to trigger a hook on changes.)
Poorly phrased question -- I meant is it possible to add non-string-name
attributes to classes after class creation. During class creation we can do
this:
-- class Test:
... ns = vars()
... ns[42] = 'green eggs'
... del ns
...
-- Test
class '__main__.Test'
-- Test.__dict__
dict_proxy({
'__module__': '__main__',
42: 'green eggs',
'__doc__': None,
'__dict__': attribute '__dict__' of 'Test' objects,
'__weakref__': attribute '__weakref__' of 'Test' objects,
'__locals__': {
42: 'green eggs',
'__module__': '__main__',
'__locals__': {...}}
})
-- Test.__dict__[42]
'green eggs'
A little more experimentation shows that not all is well, however:
-- dir(Test)
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: unorderable types: int() str()
So what conclusion do you draw?
That other changes (that have definitely been for the better) are making
the 'feature' of non-string keys in namespace dicts less and less
friendly. Rather than letting it slowly fall into complete shambles we
should go ahead and deprecate, then remove, that functionality.
Because namespace dicts already have tacit approval to not support
non-string keys, it doesn't make much sense to spend developer resources
on fixing dir and whatever other functions exist that deal with
namespace dicts and assume string-only keys.
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Non-string keys in type dict
Hi, 2012/3/8 Ethan Furman et...@stoneleaf.us: A little more experimentation shows that not all is well, however: -- dir(Test) Traceback (most recent call last): File stdin, line 1, in module TypeError: unorderable types: int() str() So what conclusion do you draw? That other changes (that have definitely been for the better) are making the 'feature' of non-string keys in namespace dicts less and less friendly. Rather than letting it slowly fall into complete shambles we should go ahead and deprecate, then remove, that functionality. Not that I disagree with the conclusion, but the obvious thing to do here is to fix dir() and return only string attributes, i.e. those you can access with getattr. Cheers, -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxing Python
Hi Stefan, On Wed, Mar 7, 2012 at 23:16, Stefan Behnel stefan...@behnel.de wrote: Well, there's a bug tracker that lists some of them, which is not *that* hard to find. Does your claim about a significantly harder endeavour refer to finding a crash or to finding a fix for it? Are you talking about the various crashes about the JIT? That's one of the reasons why the sandboxed PyPy does not include the JIT. A bientôt, Armin. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
Stefan Behnel, 07.03.2012 21:40:
I found a problem in the current yield from implementation
... and here's another one, also in genobject.c:
int
PyGen_FetchStopIterationValue(PyObject **pvalue) {
PyObject *et, *ev, *tb;
PyObject *value = NULL;
if (PyErr_ExceptionMatches(PyExc_StopIteration)) {
PyErr_Fetch(et, ev, tb);
Py_XDECREF(et);
Py_XDECREF(tb);
if (ev) {
value = ((PyStopIterationObject *)ev)-value;
Py_INCREF(value);
Py_DECREF(ev);
}
} else if (PyErr_Occurred()) {
return -1;
}
if (value == NULL) {
value = Py_None;
Py_INCREF(value);
}
*pvalue = value;
return 0;
}
When the StopIteration was set using PyErr_SetObject(), ev points to the
value, not the exception instance, so this code lacks exception
normalisation. I use slightly different code in Cython (which needs to be
compatible with Py2.x), but CPython 3.3 could do something along these lines:
if (ev) {
if (PyObject_IsInstance(ev, PyExc_StopIteration)) {
value = ((PyStopIterationObject *)ev)-value;
Py_INCREF(value); // or maybe XINCREF()?
Py_DECREF(ev);
} else {
/* PyErr_SetObject() puts the value directly into ev */
value = ev;
}
} else ...
Would that be acceptable for CPython as well or would you prefer full
fledged normalisation?
Stefan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding a builtins parameter to eval(), exec() and __import__().
On 8 March 2012 12:52, Nick Coghlan ncogh...@gmail.com wrote: 2. it's already trivial to achieve such chained lookups in 3.3 by passing a collections.ChainMap instance as the globals parameter: http://docs.python.org/dev/library/collections#collections.ChainMap Somewhat OT, but collections.ChainMap is really cool. I hadn't noticed it get added into 3.3, and as far as I can see, it's not in the What's New in 3.3 document. But it's little things like this that *really* make the difference for me in new versions. So thanks to whoever added it, and could we have a whatsnew entry, please? Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
2012/3/8 Stefan Behnel stefan...@behnel.de: Would that be acceptable for CPython as well or would you prefer full fledged normalisation? I think we have to normalize for correctness. Consider that it may be some StopIteration subclass which set value on construction. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
On Thu, 8 Mar 2012 14:36:06 -0600 Benjamin Peterson benja...@python.org wrote: 2012/3/8 Stefan Behnel stefan...@behnel.de: Would that be acceptable for CPython as well or would you prefer full fledged normalisation? I think we have to normalize for correctness. Consider that it may be some StopIteration subclass which set value on construction. Perhaps it would be time to drop the whole delayed normalization thing, provided the benchmarks don't exhibit a slowdown? It complicates a lot of code paths. cheers Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
2012/3/8 Antoine Pitrou solip...@pitrou.net: On Thu, 8 Mar 2012 14:36:06 -0600 Benjamin Peterson benja...@python.org wrote: 2012/3/8 Stefan Behnel stefan...@behnel.de: Would that be acceptable for CPython as well or would you prefer full fledged normalisation? I think we have to normalize for correctness. Consider that it may be some StopIteration subclass which set value on construction. Perhaps it would be time to drop the whole delayed normalization thing, provided the benchmarks don't exhibit a slowdown? It complicates a lot of code paths. +1 Also, it delays errors from exception initialization to arbitrary points. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP
On Thu, Feb 9, 2012 at 5:18 PM, Guido van Rossum gu...@python.org wrote:
A dictionary would (then) be a SET of these. (Voila! things have already
gotten simplified.)
Really? So {a:1, a:2} would be a dict of length 2?
Eventually, I also think this will seque and integrate nicely into Mark
Shannon's shared-key dict proposal (PEP 412).
I just noticed something in Guido's example. Something gives me a strange
feeling that using a variable as a key doesn't smell right. Presumably
Python just hashes the variable's id, or uses the id itself as the key, but
I wonder if anyone's noticed any problems with this, and whether the hash
collision problems could be solved by removing this?? Does anyone even
use this functionality -- of a *variable* (not a string) as a dict key?
mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP
On 2012-03-08, at 22:08 , Mark Janssen wrote: I just noticed something in Guido's example. Something gives me a strange feeling that using a variable as a key doesn't smell right. Presumably Python just hashes the variable's id, or uses the id itself as the key Python calls ``hash`` on the object and uses the result. , but I wonder if anyone's noticed any problems with this, and whether the hash collision problems could be solved by removing this?? No. Not that it makes sense, people could ask for object hashes on their own and end up with the same result. Does anyone even use this functionality -- of a *variable* (not a string) as a dict key? What you're asking does not make sense, the dict key is not the name but whatever object is bound to the name. And yes I've used non-string objects as names before: tuples, frozensets, integers, my own objects, … ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding a builtins parameter to eval(), exec() and __import__().
On 07/03/2012 16:33, Mark Shannon wrote: It should also help with sandboxing, as it would make it easier to analyse and thus control access to builtins, since the execution context of all code would be easier to determine. pysandbox patchs __builtins__ in: - the caller frame - the interpreter state - all modules It uses a read-only dict with only a subset of __builtins__. It is important for: - deny replacing a builtin function - deny adding a new superglobal variable - deny accessing a blocked function If a module or something else leaks the real builtins dict, it would be a vulnerability. pysandbox is able to replace temporary __builtins__ everywhere and then restore the previous state. Can you please explain why/how pysandbox is too restrictive and how your proposition would make it more usable? Currently, it is impossible to allow one function access to sensitive functions like open(), while denying it to others, as any code can then get the builtins of another function via f.__globals__['builtins__']. Separating builtins from globals could solve this. For a sandbox, it's a feature, or maybe a requirement :-) It is a problem if a function accessing to the trusted builtins dict is also accessible in the sandbox. I don't remember why it is a problem: pysandbox blocks access to the __globals__ attribute of functions. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 - some numbers from the Django port
On Wed, Mar 7, 2012 at 2:36 PM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Armin Ronacher armin.ronacher at active-4.com writes: What are you trying to argue? That the overall Django testsuite does not do a lot of string processing, less processing with native strings? I'm surprised you see a difference at all over the whole Django testsuite and I wonder why you get a slowdown at all for the ported Django on 2.7. The point of the figures is to show there is *no* difference (statistically speaking) between the three sets of samples. Of course, any individual run or set of runs could be higher or lower due to other things happening on the machine (not that I was running any background tasks), so the idea of the simple statistical analysis is to determine whether these samples could all have come from the same populations. According to ministat, they could have (with a 95% confidence level). But the stuff you run is not really benchmarking anything. As far as I know django benchmarks benchmark something like mostly DB creation and deletion, although that might differ between CPython and PyPy. How about running *actual* django benchmarks, instead of the test suite? Not that proving anything is necessary, but if you try to prove something, make it right. Cheers, fijal ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxing Python
On 05/03/2012 23:11, Victor Stinner wrote: 3 tests are crashing pysandbox: - modify a dict during a dict lookup: I proposed two different fixes in issue #14205 - type MRO changed during a type lookup (modify __bases__ during the lookup): I proposed a fix in issue #14199 (keep a reference to the MRO during the lookup) - stack overflow because of a compiler recursion: we should limit the depth in the compiler (i didn't write a patch yet) pysandbox should probably hide __bases__ special attribute, or at least make it read-only. I opened the following issues to fix these crashers: #14205: Raise an error if a dict is modified during a lookup. Fixed in Python 3.3. #14199: Keep a refence to mro in _PyType_Lookup() and super_getattro(). Fixed in Python 3.3. #14211: Don't rely on borrowed _PyType_Lookup() reference in PyObject_GenericSetAttr(). Fixed in Python 3.3. #14231: Fix or drop Lib/test/crashers/borrowed_ref_1.py, it looks like it was already fixed 3 years ago. The compiler recursion is not fixed yet. Fixes may be backported to Python 2.7 and 3.2. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding a builtins parameter to eval(), exec() and __import__().
On Fri, Mar 9, 2012 at 3:31 AM, Guido van Rossum gu...@python.org wrote: But the __builtins__ that are actually used by any particular piece of code is *not* taken by importing builtins. It is taken from what the globals store under the key __builtins__. This is a feature that was added specifically for sandboxing purposes, but I believe it has found other uses too. Agreed, but swapping out builtins for a different namespace is still the exception rather than the rule. My Impression of Mark's proposal was that this approach would become the *preferred* way of doing things, and that's the part I don't like at a conceptual level. The key point is that every piece of code already inherits locals, globals and builtins from somewhere else. We can already control locals (by which parameters are passed in) and globals via exec, eval, __import__, and runpy (any others?) but we can't control builtins. Correct - because controlling builtins is the domain of sandboxes. Incorrect (unless I misunderstand the context) -- when you control the globals you control the __builtins__ set there. And this is where I don't like the idea at a practical level. We already have a way to swap in a different set of builtins for a certain execution context (i.e. set __builtins__ in the global namespace) for a small chunk of code, as well as allowing collections.ChainMap to insert additional namespaces into the name lookup path. This proposal suggests adding an additional mapping argument to every API that currently accepts a locals and/or globals mapping, thus achieving... well, nothing substantial, as far as I can tell (aside from a lot of pointless churn in a bunch of APIs, not all of which are under our direct control). In any case, the locals / globals / builtins chain is a simplification; there are also any number of intermediate scopes (between locals and globals) from which nonlocal variables may be used. Like optimized function globals, these don't use a dict lookup at all, they are determined by compile-time analysis. Acknowledged, but code executed via the exec API with both locals and globals passed in is actually one of the few places where that lookup chain survives in its original form (module level class definitions being the other). Now, rereading Mark's original message, a simpler proposal of having *function objects* do an early lookup of self.__globals__['__builtins__'] at creation time and caching that somewhere such that the frame objects can get hold of it (rather than having to do the lookup every time the function gets called or a builtin gets referenced) might be a nice micro-optimisation. It's the gratuitous API changes that I'm objecting to, not the underlying idea of binding the reference to the builtins namespace earlier in the function definition process. I'd even be OK with leaving the default builtins reference *out* of the globals namespace in favour of storing a hidden reference on the frame objects. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding a builtins parameter to eval(), exec() and __import__().
On Thu, Mar 8, 2012 at 4:33 PM, Nick Coghlan ncogh...@gmail.com wrote: On Fri, Mar 9, 2012 at 3:31 AM, Guido van Rossum gu...@python.org wrote: But the __builtins__ that are actually used by any particular piece of code is *not* taken by importing builtins. It is taken from what the globals store under the key __builtins__. This is a feature that was added specifically for sandboxing purposes, but I believe it has found other uses too. Agreed, but swapping out builtins for a different namespace is still the exception rather than the rule. My Impression of Mark's proposal was that this approach would become the *preferred* way of doing things, and that's the part I don't like at a conceptual level. The key point is that every piece of code already inherits locals, globals and builtins from somewhere else. We can already control locals (by which parameters are passed in) and globals via exec, eval, __import__, and runpy (any others?) but we can't control builtins. Correct - because controlling builtins is the domain of sandboxes. Incorrect (unless I misunderstand the context) -- when you control the globals you control the __builtins__ set there. And this is where I don't like the idea at a practical level. We already have a way to swap in a different set of builtins for a certain execution context (i.e. set __builtins__ in the global namespace) for a small chunk of code, as well as allowing collections.ChainMap to insert additional namespaces into the name lookup path. This proposal suggests adding an additional mapping argument to every API that currently accepts a locals and/or globals mapping, thus achieving... well, nothing substantial, as far as I can tell (aside from a lot of pointless churn in a bunch of APIs, not all of which are under our direct control). In any case, the locals / globals / builtins chain is a simplification; there are also any number of intermediate scopes (between locals and globals) from which nonlocal variables may be used. Like optimized function globals, these don't use a dict lookup at all, they are determined by compile-time analysis. Acknowledged, but code executed via the exec API with both locals and globals passed in is actually one of the few places where that lookup chain survives in its original form (module level class definitions being the other). Now, rereading Mark's original message, a simpler proposal of having *function objects* do an early lookup of self.__globals__['__builtins__'] at creation time and caching that somewhere such that the frame objects can get hold of it (rather than having to do the lookup every time the function gets called or a builtin gets referenced) might be a nice micro-optimisation. It's the gratuitous API changes that I'm objecting to, not the underlying idea of binding the reference to the builtins namespace earlier in the function definition process. I'd even be OK with leaving the default builtins reference *out* of the globals namespace in favour of storing a hidden reference on the frame objects. Agreed on the gratuitous API changes. I'd like to hear Mark's response. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sandboxing Python
On 01/03/2012 22:59, Victor Stinner wrote: I challenge anymore to break pysandbox! I would be happy if anyone breaks it because it would make it more stronger. Results, one week later. Nobody found a vulnerability giving access to the filesystem or to the sandbox. Armin Rigo complained that CPython has known crasher bugs. Except of the compiler recursion, I fixed those bugs in CPython 3.3. Serhiy Storchaka found a bug in the pysandbox timeout: long operations implemented in C hangs the sandbox, the timeout contrain is not applied. Guido proposed to abort the process (use the default SIGALRM action). I proposed to add an option to use a subprocess. Both solutions are not exclusive. Armin Rigo also noticed that PyPy sandbox design is more robust than pysandbox design, I agree with him even if I think a CPython sandbox is useful and users ask for such protection. I have no idea how many developers tried to break the pysandbox security. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
