Re: [Python-Dev] Import semantics
Frank Wierzbicki wrote: > On 7/5/06, Guido van Rossum <[EMAIL PROTECTED]> wrote: > >>Hi Frank, >> >>Have you and/or the Jython community made up your mind about this? The >>thread seems to have disappeared after you posted (or perhaps it >>continued only on jython-dev, which I don't read?). > > The thread pretty much stopped there. I think a modification of the > import semantics won't really come up again until 2.3 is out, since > releases and reviving Jython take priority over backwards incompatible > features. For my part I would like to think that a goal for Jython > should be: > > Pure Python code developed first on Jython should run without change on > CPython. > > Which would mean we will eventually need to change the import > semantics for importing Python code (though I doubt we will change the > semantics for importing Java packages any time soon). Whether that > can be done in 2.x, or if this change is so incompatible that we need > to think about it in a "Jython 3000" way, I don't really know. Changing the behavior of getattr for python modules and leave it as it is for java packages seems a reasonable compromise. > > >>Also, I just realized that you're the new Jython maintainer. Is *that* >>official? > > It is official, at least in the unofficial way that Jython appears to > work: Brian handed the baton to me after (I presume) Samuele Pedroni > had handed the baton to Brian. > > Brian's email is here: > http://sourceforge.net/mailarchive/message.php?msg_id=13859029 > > That said, I still regard Samuele Pedroni as the ultimate authority on > Jython and give him pretty much full veto power. He fortunately > continues to watch the checkins and prods me when I go in the wrong > direction. > as things stand this a reasonable way to go about things, I'm the only person with a bit historical prospective, a bit of time to help (less than I would like still), knowing both python semantics well and Java and knowing the Jython code base, also I designed how the new-style classes implemantion work in Jython atm, so I should know how things are supposed to hang together in that respect, also for things that involve the Java glueing aspects of Jython one needs to grow some language design experience and sense, and I'm around since enough time for Jython and on python-dev to have some of that :) regards. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.5 and beyond
Neal Norwitz schrieb: > On 7/4/06, Thomas Heller <[EMAIL PROTECTED]> wrote: >> >> I would like to ask about the possibility to add some improvements to ctypes >> in Python 2.5, although the feature freeze is now in effect. Hopefully >> former >> third-party libraries can have the freeze relaxed somewhat;-). > > Ok, former third-party libraries get a 1e-308 reprieve. :-) > > I think the general answer is to plan some way that you could make an > external release to support the other communities like numpy. I don't > know the details, but I believe Barry does this with email. This > would allow Python to be more stable, but allow additional features > for the communities that are interested in installing an additional > ctypes release. > >> I intend to do these changes, the first is a small and trivial one, but >> allows >> a lot of flexibility: >> >> - Remove the restriction that the argtypes attribute of foreign functions >> must >> be ctypes types. Instead they are only required to implement a .from_param >> class method. > > This patch is ok. I will add comments to the patch on SF. Patch 1517790 is now committed, thanks. >> The second one is more involved, and not yet complete. I can post the patch >> or a link to it for review when it is implemented completely: >> >> - Implement the __array_struct__ attribute as describes by the numpy pep at >> http://numpy.scipy.org/array_interface.html. > > This is too much to add to a beta. Perhaps you could work on a branch > to add these features and integrate the branch back in after 2.5 has > been released. You could make an external release from the branch. We will see how this works out. Not what I expected to hear, but I'm sure it is a wise decision. Thanks, Thomas ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] zlib module build failure on Mac OSX 10.4.7
On Jul 5, 2006, at 7:35 AM, Ronald Oussoren wrote: On Jul 4, 2006, at 11:21 PM, Neal Norwitz wrote: Ronald, Bob, I know Skip found and fixed his problem, however, is this problem likely to affect other users? Is there anything we can do to help alleviate/diagnose this problem? I'll either enhance configure or roll back my change to setup.py. I'd prefer to do the former, but if beta2 gets too close I'll just change setup.py. I have checked in a patch that fixes this issue (that is, configure will correctly set HAVE_ZLIB_COPY even if you have an old static library of zlib in /usr/local/lib) in revision 47267. I don't really like that patch, it doesn't smell right. I wonder if we should just add '-Wl,-search_path_first' to the default LDFLAGS on OSX, that way the linker would behave more like the one on other unix- y platforms and special-casing for several libraries could be dropped. To recap the problem I'm trying to solve: python uses several 3th- party libraries like libz and libbz2 to build extensions. Some of these libraries are shipped with OSX as shared libraries. I want to make it possible to link with local copies of those libraries, mostly because that makes it possible to run the resulting binary on older versions of the OS but also because some of the libraries on OSX are pretty old. I also want to link using static libraries instead of dynamic libraries. To do that you must use -Wl,-search_paths_first because otherwise the OSX linker will look for a dylib anywhere on the path before looking for a static library. Ronald Ronald ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ ronaldoussoren%40mac.com smime.p7s Description: S/MIME cryptographic signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] what can we do to hide the 'file' type?
Hi Brett, On Wed, Jul 05, 2006 at 05:01:48PM -0700, Brett Cannon wrote: > And if Armin and/or Samuele sign off that what we find is most likely (with > "most likely" equalling 99% chance) all there is, then bonus points and I > will *really* be convinced. =) I don't think I can "sign off" that. Really hiding Python objects is quite hard IMHO. Armin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposed beta 2 changes (Q for Anthony/Neal)
Anthony Baxter wrote: > On Tuesday 04 July 2006 22:32, Nick Coghlan wrote: >> 1. Finishing the __module_name__ workaround to allow relative >> imports from the main module when using -m. > > I have some nervousness about this. Are there cases in the stdlib > where this is an issue, today? Are there any cases which can't be > addressed by using absolute imports? For 2.5, wouldn't it be better > to simply say "use absolute imports in this case"? > My nervousness is that I don't want a late change to introduce a > language misfeature that we'll all regret later. I actually come to agree with this POV - particularly after spending some time thinking about how I'd like to it to look in Py3k (with a simple boolean check replacing the __name__ == "__main__" idiom). I'll revert the change that added __module_name__, too. I'd love to fix this properly for 2.5 but the timing of the discovery of the limitation is unfortunately lousy :( I'll add something to PEP 338, and possibly elsewhere, noting that the -m switch adds the current directory to sys.path, so absolute imports will work fine so long as you invoke Python from the directory containing the relevant top level package directory, even if it isn't anywhere else on sys.path. Some time between now and late 2007 I'll write something up about the interaction between __name__, sys.path[0] and implicit relative imports that actually makes implicit relative imports appear to work when a module inside a package is executed directly instead of via -m (probably when I get around to writing a Python 2.6 PEP on the topic). Heh. I was going to be clever with the PEP number relating to this for 2.6, and I think I've discovered that trying to mix PEP 328 with 338 is going to be a doomed effort - add the PEP numbers together to find out why ;) >> 2. Adding an 'ignore' filter for ImportWarning at the end of >> warnings.py > > Already covered this one in another email... Yes, this seems the best > approach for 2.5. OK, I'll make the change to warnings.py, and update the warnings module documentation. AMK will still need to update What's New. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Phillip J. Eby wrote: > At 07:27 PM 7/5/2006 +0200, Guido van Rossum wrote: >> However I still don't believe "global" has the stretchiness in its >> meaning that you claim it has. Have you ever heard a Python programmer >> talking about closures use the word "global variable"? >> >> Are there any other native speakers who side with Michael? > > I don't, and am -1 on using "global" to mean "outer". Another -1 on stretching the meaning of global here. I quite like 'nonlocal', though. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] zipfile.ZipFile('foo.zip', 'a'): file not found -> create?
Hi all, In reference to: http://sourceforge.net/tracker/index.php?func=detail&aid=1514451&group_id=5470&atid=105470 I wrote a patch for this "bug", but a valid point was raised by Ronald Oussorren: this borders on being more of a "feature" than a bug fix, although - IMHO - this fix improves consistency with the rest of the Python standard library. Can I get some opinions on this? My patch for this issue currently lives here: http://sourceforge.net/tracker/index.php?func=detail&aid=1517891&group_id=5470&atid=305470 Cheers, Tom ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] doc for new restricted execution design for Python
I wrote: > I would still rather not spread FUD. Brett Cannon responds: > I don't consider it FUD. Armin in an email said that he thought it > was a losing battle to try to hide 'file' from an interpreter. That > is what I am worried about, period. Everythign else can be > protected through resource hiding. I never intended to say that "we can't hide 'file' from the user" was FUD. Only objects crossing between interpreters. And it *might* not be FUD, but so far I haven't heard anyone say that they thought objects *could* cross between interpreters. I think your proposal would read better with a categorical statement that objects cannot cross between interpreters (right along with your categorical statement that Python code cannot directly access dangerous resources without help from the C level), so I present the following challenge: can anyone think of a way that an object could "cross" interpreters? For instance, are certain basic objects shared (str and int for instance) and objects could be passed between interpreters by attaching them as attributes of these core objects? I just don't know enough about multiple interpreters to be sure -- but somehow I thought they had separate object pools. -- Michael Chermside ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Restricted execution: what's the threat model?
Ka-Ping Yee writes: > i'm starting to think > that it would be good to clarify what kinds of threats we are > trying to defend against, and specify what invariants we are > intending to preserve. Yes! > So here are a couple of questions for clarification (some with my > guesses as to their answers): Okay, I'll throw in my thoughts also. > 1. When we say "restricted/untrusted/ interpreter" we > don't really mean that the *interpreter* is untrusted, right? > We mean that the Python code that runs in that interpreter is > untrusted (i.e. to be prevented from doing harm), right? Agreed. My interpretation of the proposal was that interpreters were either "sandboxed" or "trusted". "Sandboxed" means that there are security restrictions imposed at some level (perhaps even NO restrictions). "Trusted" means that the interpreter implements no security restrictions (beyond what CPython already implements, which isn't much) and thus runs faster. > 2. I'm assuming that the implementation of the Python interpreter > is always trusted Sure... it's got to be. > What do > we take the Trusted Computing Base to include? The Python VM > implementation -- plus all the builtin objects and C modules? > Plus the whole standard library? My interpretation of Brett's proposal is that the CPython developers would try to ensure that Python VM had no "security holes" when running in sandboxed mode. Of course, we also "try" to ensure no crashes are possible also, and while we're quite good, we're not perfect. Beyond that, all pure-python modules with source available (whether in the stdlib or not) can be "trusted" because they run in a sandboxed VM. All C modules are *up to the user*. Brett proposes to provide a default list of useful-but-believed-to-be-safe modules in the stdlib, but the user can configure the C-module whitelist to whatever she desires. > 3. Is it part of the plan that we want to protect Python code from > other Python code? For example, should a Python program/function > X be able to say "i want to launch/call program/function Y with > *these* parameters and have it run under *these* limitations?" > This has a big impact on the model. Now *that* is a good question. I would say the answer is a partial "no", because there are pieces of Brett's security model that are tied to the interpreter instance. Python code cannot launch another interpreter (but perhaps it *should* be able to?), so it cannot modify those restrictions for new Python code it launches. However, I would rather like to allow Python code to execute other code with greater restrictions, although I would accept all kinds of limitations and performance penalties to do so. I would be satisfied if the caller could restrict certain things (like web and file access) but not others (like memory limits or use of stdout). I would satisfied if the caller paid huge overhead costs of launching a separate interpreter -- heck, even a separate process. And if it is willing to launch a separate process, then Brett's model works just fine: allow the calling code to start a new (restricted) Python VM. > We want to be able to guarantee that... > > A. The interpreter will not crash no matter what Python code > it is given to execute. Agreed. We already want to guarantee that, with the caveat that the guarantee doesn't apply to a few special modules (like ctypes). > B. Python programs running in different interpreters embedded > in the same process cannot communicate with each other. I don't want to guarantee this, does someone else? It's astonishingly hard... there are all kinds of clever "knock on the walls" tricks. For instance, communicate by varying your CPU utilization up and down in regular patterns. I'd be satisfied if they could pass information (perhaps even someday provide a library making it *easy* to do so), but could not pass unforgable items like Python object references, open file descriptors, and so forth. > C. Python programs running in different interpreters embedded > in the same process cannot access each other's Python objects. I strengthen that slightly to all "unforgable" items, not just object references. > D. A given piece of Python code cannot access or communicate > with certain Python objects in the same interpreter. > > E. A given piece of Python code can access only a limited set > of Python objects in the same interpreter. Hmmm. I'm not sure. -- Michael Chermside ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Michael Chermside <[EMAIL PROTECTED]> writes: > Phillip Eby writes: >> I don't see a problem with requiring '.x' to be used for both >> reading and writing of outer-scope names; it just shouldn't be >> required for an outer-scope name that you don't rebind in the >> current scope. >> >> def counter(num): >> def inc(): >> .num += 1 >> return .num >> return inc > > I am reminded of Tim Peter's declaration in response to a similar > proposal some time ago: > > Syntax should not look like grit on my monitor. > > (Sorry, no reference... but I swear it's word-for-word accurate because > the quote burned itself into my memory.) I think it was Anthony: http://mail.python.org/pipermail/python-dev/2005-July/054581.html Cheers, mwh -- SCSI is not magic. There are fundamental technical reasons why it is necessary to sacrifice a young goat to your SCSI chain now and then. -- John Woods ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Ka-Ping Yee wrote: > On Wed, 5 Jul 2006, Guido van Rossum wrote: >> On 7/5/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote: >>> Using the classic nonsense example: >>> >>> def counter(num): >>> def inc(): >>> .num += 1 >>> return .num >>> return inc >>> >> Would this also use ..num to refer to num in an outer scope two >> levels removed? > > I don't think there's any need for that. I see '.num' as just another > way of saying "num, but don't make a new binding". > > I agree with Guido that the best proposals so far are converging on > the idea that it's more Pythonic to say "don't make a new binding" > when a variable is used, than to declare "this is the scope for this > binding" ahead of time. > > Of those there are two kinds: > > (a) State once (anywhere in a scope, but preferably at the > beginning) that a variable is non-local. This is like > the "global" keyword works now, and this category includes: > > - Change the meaning of 'global'. > - Add a new keyword 'outer' or 'nonlocal', etc. > > (b) Indicate, when mentioning a variable, that the variable > is non-local. This category includes: > > - Say 'global.x' or 'outer.x' instead of 'x'. > - Say '.x' instead of 'x'. > > My favourite so far is to use a new keyword -- i think changing the > meaning of 'global' would be misleading. '.x' is probably my next > favourite, though i share Guido's concern about allowing both 'x' > and '.x' to refer to the same thing. > > I see that 'outer' is used as an identifier in hmac.py in the > standard library and also as a variable in test_set*.py. On the > other hand 'nonlocal' does not appear anywhere in the standard > library. Thus, 'nonlocal' is the best option that i've seen so far; > it's less likely to break anything and it says exactly what it means. > I can't think of a more accurate keyword. 'extant' (= already existing) is probably pretty rare in existing code, and has pretty close to exactly the correct meaning, but may be too obscure. -tim > > > -- ?!ng > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
So.. are we only thinking about implementing this outer scope assignment because there's lots of talk about it on the list, or are there actually use cases that would become clearer if assigning to an outer scope variable was allowed? I tend to think that almost _any_ piece of code that could be written using outer scope assignment would become less clear by doing so as opposed to some other way. So, I'm not keen on the whole idea. If we must, however, I think a declaration like 'nonlocal' would be best (well, least-bad, that is). ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
jan-python> So.. are we only thinking about implementing this outer jan-python> scope assignment because there's lots of talk about it on jan-python> the list, ... :-) jan-python> ... or are there actually use cases that would become jan-python> clearer if assigning to an outer scope variable was allowed? I think full lexical scoping will only be of use to people who use nested scopes heavily. The more typical user will be happy to just refer to values in outser scopes without modifying them and rely on classes to save changed state across calls. I think it's almost a YAGNI, but I'm sure others will disagree. Skip Repeat after me: Python is not Lisp... ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] what can we do to hide the 'file' type?
Armin Rigo writes: > I don't think I can "sign off" [on hiding the file type]. Really hiding > Python objects is quite hard IMHO. I agree. But we don't have to give up yet. How about instead of hiding file, we cripple it. Completely. Modify the file type so that when executing on a sandboxed interpreter, all of the dangerous methods and attributes of file throw exceptions. Then we create a separate thing (in C) called a "SecureFileWrapper". It has methods that are passed a reference to a file object and can invoke the methods without error. We provide a means for obtaining a SecureFileWrapper bound to a given file (perhaps open()). Essentially, we give up on hiding file, which is a frequently-used type, and very hard to hide, and instead we rely on our ability to write a reliably secure "SecureFileWrapper" class (in C). -- Michael Chermside ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
> However I still don't believe "global" has the stretchiness in its > meaning that you claim it has. Have you ever heard a Python programmer > talking about closures use the word "global variable"? I guess the term I've heard most often is "free variable," but I wouldn't be surprised if I saw the term "global" used to describe a free variable. However, I should point out that Dijkstra used "global" in a similar way in his 1971 book "A Discipline of Programming." The program examples in that book are in a language he devised for the purpose; one of the language's features is that every variable used in every block must be explicitly declared, even if it is taken from a surrounding block. If I remember correctly, the terms he used for variables taken from a surrounding block and variables defined and used in the same block were "global" and "private," respectively. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
On 7/6/06, Guido van Rossum <[EMAIL PROTECTED]> wrote: +1 on nonlocal.I think that the := operator is also in case (b), but as I don't likeit I'm find with not mentioning it. :-)Could someone write a PEP for this? Doesn't have to be very long butI'd like it to summarize the main options proposed and discuss them, like I did for the switch PEP. It's a p3yk PEP. (We really need tomove this to the py3k list...)Drat, too bad this wasn't back in February when I was all for writing the PEP--sadly, I don't have time to do this, maybe later if no one steps up to the plate... For reference, here is a link to the other, rather large, thread on this back then:http://thread.gmane.org/gmane.comp.python.devel/76532/focus=76532 My option one, is essentially "nonlocal", though I spelled it as "use". Really, I am personally agreeable to almost any spelling of such a keyword. I am +1 on "nonlocal", though I know the biggest dissent against it is adding another "global"-like keyword that is not "pythonic". Also in regard to the "prefix-dot" notation (i.e. ".x"), I am -1 for the reason that it introduces a subtle alternate way of spelling local variables. The rules for the usage would have to be strict enough to prevent subtle code obscurity (like what is the semantics for using ".x" and "x" in a scope--are both spellings allowed, and if so, what is the meaning if assignment is involved or is not involved). Of course, if the semantics were well defined such that it would be difficult for users to trap themselves with different spellings of local variables, then I would be +0 for it. Best Regards,Almann-- Almann T. Goo[EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
At 10:05 AM 7/6/2006 -0500, [EMAIL PROTECTED] wrote: > jan-python> So.. are we only thinking about implementing this outer > jan-python> scope assignment because there's lots of talk about it on > jan-python> the list, ... > >:-) > > jan-python> ... or are there actually use cases that would become > jan-python> clearer if assigning to an outer scope variable was allowed? > >I think full lexical scoping will only be of use to people who use nested >scopes heavily. The more typical user will be happy to just refer to values >in outser scopes without modifying them and rely on classes to save changed >state across calls. I think it's almost a YAGNI, but I'm sure others will >disagree. Here's the reason I think this keeps coming up, and why Guido's "just use a class" argument doesn't really address the actual problem that's taking place. When you are writing some function, and you find yourself using a nested function because it's the closest match for something you're doing, there occasionally comes a point at which you realize that, gosh, you need to mutate something in an outer scope. At that point, your choices are to either kludge it with a mutable, or to reorganize the whole thing. Both of these *feel* like warts because they're making you do less-than-optimal things. Your mental "flow" is disrupted because the language is forcing you to work around it, rather than having a way of working with it. This is a user experience issue, not a technical one. The fact that you can say, "you should've done it differently in the first place" doesn't do anything for your flow. In theory, you could design cars without any brakes, because people could just coast to a stop if they planned well enough in advance. ;-) In practice, you need the brakes because people often don't discover their need to stop until much later in the process. This is a flow issue that's specific to *incremental* development. What happens is that first you refactor the code in a function to include nested functions. The variable references don't change, you're just indenting some code. *Then*, it later comes up that you need to rebind a variable, and now you have to globally change the variable to make it work as a mutable instead, or else you have to refactor all the variables to be 'self.' references and put a class in somewhere. This destroys the flow of incrementally developing whatever it is you're developing, and makes you stop to do excise work. That's why being able to rebind a variable without redefining it is important. In short: in *theory*, a rebinding operator or "nonlocal" declaration is unnecessary. In *practice*, having one seems quite useful every time you wander down the path that leads to having to rewrite your code just because the language won't let you do that one tiny thing -- or so it feels like to the person who's experiencing it. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python]
On 7/5/06, Talin <[EMAIL PROTECTED]> wrote: Brett Cannon wrote:> On 7/5/06, Michael Chermside <[EMAIL PROTECTED]> wrote:>> If you were using capabilities, you would need to ensure that>> restricted interpreters could only get the file object that they >> were given. But then _all_ of these fancy versions of the>> restrictions would be immediately supported: it would be up to the>> users to create secure wrappers implementing the specific >> restrictions desired.>> I agree. I would prefer this way of doing it. But as I have said, making> sure that 'file' does not get out into the wild is tough.I seem to recall someone mentioned earlier in this discussion the notion of somehow throwing an exception when sandboxed code attempts to push afile reference onto the interpreter stack.That was AMK. I'm not an expert in these matters, so perhaps what I am going to saywill make no sense, but here goes:What if there were two copies of the evaluator function. One copy wouldbe a slightly slower 'checked' function, that would test all objects for a 'check' bit. Any attempt to evaluate a reference to an object with acheck bit set would throw an exception. The other eval function would be the 'unchecked' version that would runat full speed, just like it does today.Transitioning from the checked to the unchecked state could only be donevia C code. So the 'file' wrapper, for example, would switch over to the unchecked interpreter before calling the actual methods of 'file'. ThatC wrapper might also check the current permission state to see whatoperations were legal.So add the proper checks in Python/ceval.c:call_function() to check for this flag on every object passed in that is called? Note that whenever a C function sets the interpreter state to'unchecked', that fact is saved on the stack, so that when the function returns, the previous state is restored. The function for setting theinterpreter state is something like PyCall_Unchecked( ... ), whichrestores the interpreter state back to where it was.Transitioning from unchecked to checked is trickier. Specifically, you don't want to ever run sandboxed code in the unchecked state - this is aproblem for generators, callbacks, and so on. I can think of twoapproaches to handling this:First approach is to mark all sandboxed code with a bit indicating the code is untrusted. Any attempt to call or otherwise invoke a functionthat has this bit set would throw the interpreter into the 'checked'state. (Note that transitioning the other way is *not* automatic - i.e. calling trusted code does not automatically transition to unchecked state.)The approach is good because it means that if you have intermediary codebetween the wrapper and the sandboxed code, the interpreter still does the right thing - it sets the interpreter into checked state.One problem is how to restore the 'unchecked' state when a function callreturns. Probably you would have to build this into the code that does the state transition.If marking the sandboxed code isn't feasible, then you'd have to havethe wrapper objects wrap all of the callbacks with code that goes tochecked state before calling the callbacks. This means finding all the possible holes - however, I suspect that there are far fewer such holesthan trying to hide all possible 'file' methods. However, one advantageof doing this is that restoring the 'unchecked' state after the call returns is fairly straightforward.The advantage of the this whole approach is that once you set the'check' bit on 'file', it doesn't matter whether 'file' gets loose ornot - you can't do anything with it without throwing an exception. Moreover, it also doesn't matter what code path you go through to accessit. Only C code that flips the interpreter into unchecked state can callmethods on 'file' without blowing up.So essentially, what I propose is to define a simple security primitive - which essentially comes down to checking a single bit - and use thatas a basis to create more complex and subtle security mechanisms.Right, but it does require that the proper verification function be turned on so that the permission bit on 'file' is checked. It kind of seems like 'rexec' and its f_restricted flag it set on execution frames, except you are adding an object-level flag as well. Either way, the trick is not fouling up switching between the two checking functions.-Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] doc for new restricted execution design for Python
On 7/6/06, Michael Chermside <[EMAIL PROTECTED]> wrote: I wrote:> I would still rather not spread FUD.Brett Cannon responds:> I don't consider it FUD. Armin in an email said that he thought it> was a losing battle to try to hide 'file' from an interpreter. That > is what I am worried about, period. Everythign else can be> protected through resource hiding.I never intended to say that "we can't hide 'file' from the user"was FUD. Only objects crossing between interpreters. And it *might* not be FUD, but so far I haven't heard anyone say that they thoughtobjects *could* cross between interpreters. I think your proposalwould read better with a categorical statement that objects cannotcross between interpreters (right along with your categorical statement that Python code cannot directly access dangerous resourceswithout help from the C level), so I present the following challenge:can anyone think of a way that an object could "cross" interpreters? For instance, are certain basic objects shared (str and int forinstance) and objects could be passed between interpreters byattaching them as attributes of these core objects? I just don'tknow enough about multiple interpreters to be sure -- but somehow I thought they had separate object pools.C extension module attributes will share those objects across interpreters. Because the initializing function is not called again on extension modules when imported into another interpreter anything that is exposed by the module is the same across all interpreters. -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] doc for new restricted execution design for Python
On 7/5/06, Ka-Ping Yee <[EMAIL PROTECTED]> wrote: On Wed, 5 Jul 2006, Brett Cannon wrote:> On 7/4/06, Ka-Ping Yee <[EMAIL PROTECTED]> wrote:> > In response to Guido's comment about confusing the words "trusted" and > > "untrusted", how about "empowered" and "restricted"?>> Maybe. I am really starting to lean towards trusted and sandboxed.It can be risky to use words of the form '*trust*' when talking about security because 'trust' can have different meanings indifferent contexts. (Examples: 'X is trusted' might mean 'X issomething i feel safe running without restrictions' or 'X *is*in fact allowed to run without restrictions' or 'i need to run X without restrictions in order to accomplish my task' or 'X issomething i am relying on for the security of my system'.)The most common context for 'trusted' that i seem to come acrossis in the phrase 'trusted computing base' (TCB), which refers to 'the thing that is supposed to be enforcing security restrictions'as opposed to 'something i'm willing to let run unrestricted'.So when i read 'trusted code' what comes to mind is the C implementationof the Python interpreter, and it may be a good idea to reserve that phrase for that purpose, if it's to be used at all.In any case, let's try to nail down clear names for the differentpieces we're talking about here, and i gently advise avoiding'*trust*' words or using them with very limited meaning. For the next draft I am going with "trusted" and "sandboxed" just because I have already revised a decent amount and it is what my brain is defaulting to right now, but I can change the wording later. -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
I quoted this unwritten bit of Python Zen, attributing it to Tim: > Syntax should not look like grit on my monitor. mwh writes: > I think it was Anthony: > > http://mail.python.org/pipermail/python-dev/2005-July/054581.html But that's not the original. Turns out, it WAS Anthony, and I had misquoted. The correct source appears to be: http://mail.python.org/pipermail/python-dev/2004-August/047179.html And the correct quote (for the record) is: Syntax Shall Not Look Like Grit On Tim's Monitor. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] doc for new restricted execution design for Python
On 7/5/06, Greg Ewing <[EMAIL PROTECTED]> wrote: Brett Cannon wrote:> Armin in an email said that he thought it was> a losing battle to try to hide 'file' from an interpreter.And I would change file() so that it didn't openfiles. Then it would be harmless for code to have access to the file class.Right, that is essentially what I proposed initially with the whole crippling idea.What the capabilities supporters are saying is that if we go that route we will be constantly finding objects that require similar crippling. It will degenerate into this constant chasing of our tail to plug security holes where an object that we did not account for leaked out and wreaked havoc. What they are saying is that if we harden Python so that references don't get out without us knowing about it we won't have this run-around. But then my question has been what makes us trying to cripple objects any less of a run-around then finding new ways to get at references of 'file' or any other object? I have been suggesting the former requires less running around than the latter. That is why I have asked that people see how many ways they can come up with to get to 'file' from a standard interpreter prompt so we can gauge how bad hiding references might be. -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] introducing __dir__?
i'd guess this has been brought up before, but i'm not aware of it. anyway -- by what i've gathered, today's dir() "collects" the attributes of the object from __dict__, __methods__, __members__, and ultimately __class__. also, if i understood correctly, __methods__ and __members__ are deprecated. moreover, they are sequence objects, rather than callbale functions. that means my code can't my suggestion is simple -- replace this mechanism with a __dir__ - a special method that returns the list of attributes of the object. rationale: * remove deprecated __methods__, etc. * symmetry -- just like hex() calls __hex__, etc. * __methods__ and __members__ are lists rather than callable objects, which means they cannot be updated on-demand and the reason for bringing this up in the first place -- allowing proxy objects to show the attributes of the internal object. for example, in rpyc, i had to override __builtin__.dir to achieve this functionallity, which is of course not the optimal sollution. implementation: very straight forward and similar to the rest of the special methods: * dir() - returns the current scope * dir(obj) - retuns obj.__dir__(); if the object does not provide __dir__, uses the current mechanism as a fallback, or something similar the ultimate object type would grow a default __dir__ method that uses the same modus operandi as today's dir, which deriving classes could override to provide custom listings. therefore, i don't expect any backward-compatibility issues. aye or nay? -tomer ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Restricted execution: what's the threat model?
[replying to both Ping and Michael in the same email]On 7/6/06, Michael Chermside <[EMAIL PROTECTED]> wrote: Ka-Ping Yee writes:> i'm starting to think> that it would be good to clarify what kinds of threats we are > trying to defend against, and specify what invariants we are> intending to preserve.Yes!> So here are a couple of questions for clarification (some with my> guesses as to their answers): Okay, I'll throw in my thoughts also.> 1. When we say "restricted/untrusted/ interpreter" we> don't really mean that the *interpreter* is untrusted, right?> We mean that the Python code that runs in that interpreter is > untrusted (i.e. to be prevented from doing harm), right?Agreed. My interpretation of the proposal was that interpreterswere either "sandboxed" or "trusted". "Sandboxed" means that there are security restrictions imposed at some level (perhaps even NOrestrictions). "Trusted" means that the interpreter implementsno security restrictions (beyond what CPython already implements,which isn't much) and thus runs faster. Yep. > 2. I'm assuming that the implementation of the Python interpreter > is always trustedSure... it's got to be.Yep. > What do> we take the Trusted Computing Base to include? The Python VM> implementation -- plus all the builtin objects and C modules?> Plus the whole standard library?My interpretation of Brett's proposal is that the CPython developers would try to ensure that Python VM had no "security holes" whenrunning in sandboxed mode. Of course, we also "try" to ensure nocrashes are possible also, and while we're quite good, we're not perfect.Beyond that, all pure-python modules with source available (whetherin the stdlib or not) can be "trusted" because they run in asandboxed VM. All C modules are *up to the user*. Brett proposes to provide a default list of useful-but-believed-to-be-safe modulesin the stdlib, but the user can configure the C-module whitelistto whatever she desires.Michael has it on the money. > 3. Is it part of the plan that we want to protect Python code from> other Python code? For example, should a Python program/function > X be able to say "i want to launch/call program/function Y with> *these* parameters and have it run under *these* limitations?"> This has a big impact on the model.Now *that* is a good question. I would say the answer is a partial "no", because there are pieces of Brett's security model that aretied to the interpreter instance. Python code cannot launch anotherinterpreter (but perhaps it *should* be able to?), so it cannot modify those restrictions for new Python code it launches.However, I would rather like to allow Python code to execute othercode with greater restrictions, although I would accept all kindsof limitations and performance penalties to do so. I would be satisfied if the caller could restrict certain things (like weband file access) but not others (like memory limits or use ofstdout). I would satisfied if the caller paid huge overhead costsof launching a separate interpreter -- heck, even a separate process. And if it is willing to launch a separate process, thenBrett's model works just fine: allow the calling code to starta new (restricted) Python VM.The plan is that there is no sandboxed eval() that runs unsafe code from a trusted interpreter within its namespace. I hope to provide Python code access to running a sandboxed interpreter where you can pass in a string to be executed, but the namespace for that sandboxed interpreter will be fresh and will not carry over in any way from the trusted interpreter. > We want to be able to guarantee that...>> A. The interpreter will not crash no matter what Python code > it is given to execute.Agreed. We already want to guarantee that, with the caveat that theguarantee doesn't apply to a few special modules (like ctypes).Right, which is why I have been trying to plug the various known crashers that do not rely upon a specific extension module from being imported. > B. Python programs running in different interpreters embedded> in the same process cannot communicate with each other. I don't want to guarantee this, does someone else? It'sastonishingly hard... there are all kinds of clever "knock on thewalls" tricks. For instance, communicate by varying your CPUutilization up and down in regular patterns. I'd be satisfied if they could pass information (perhaps evensomeday provide a library making it *easy* to do so), but couldnot pass unforgable items like Python object references, open filedescriptors, and so forth. Or at least cannot communicate without explicit allowances to do so.As for knocking on the walls, if you protect access to that kind of information well, it shouldn't be a problem. > C. Python programs running in different interpreters embedded> in the same process cannot access each other's Python objects. I strengthen that slightly to all "unforgable" items, not justobject references.I would change tha
Re: [Python-Dev] introducing __dir__?
On Thursday 06 July 2006 13:22, tomer filiba wrote: > my suggestion is simple -- replace this mechanism with a __dir__ - > a special method that returns the list of attributes of the object. > > rationale: > * remove deprecated __methods__, etc. > * symmetry -- just like hex() calls __hex__, etc. > * __methods__ and __members__ are lists rather than callable > objects, which means they cannot be updated on-demand +1 -Fred -- Fred L. Drake, Jr. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] what can we do to hide the 'file' type?
On 7/6/06, Michael Chermside <[EMAIL PROTECTED]> wrote: Armin Rigo writes:> I don't think I can "sign off" [on hiding the file type]. Really hiding> Python objects is quite hard IMHO. I agree. But we don't have to give up yet. How about instead of hidingfile, we cripple it. Completely. Modify the file type so that whenexecuting on a sandboxed interpreter, all of the dangerous methodsand attributes of file throw exceptions. This is basically what I proposed in the first place! Then we create a separate thing (in C) called a "SecureFileWrapper".It has methods that are passed a reference to a file object andcan invoke the methods without error. We provide a means for obtaining a SecureFileWrapper bound to a given file (perhaps open()).Yeah, it would be through open() if we returned wrappers instead of performing the checks directly in file itself. Essentially, we give up on hiding file, which is a frequently-usedtype, and very hard to hide, and instead we rely on our ability towrite a reliably secure "SecureFileWrapper" class (in C). That is another possibility. Should simplify the code as well by having less checks and just have pure PySandbox_IsTrusted() checks in 'file' itself in unsafe places instead of a ton checks that the file being accessed is allowed. -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] what can we do to hide the 'file' type?
On 7/6/06, Armin Rigo <[EMAIL PROTECTED]> wrote: Hi Brett,On Wed, Jul 05, 2006 at 05:01:48PM -0700, Brett Cannon wrote:> And if Armin and/or Samuele sign off that what we find is most likely (with> "most likely" equalling 99% chance) all there is, then bonus points and I > will *really* be convinced. =)I don't think I can "sign off" that. Really hiding Python objects isquite hard IMHO.=) That's fine. I didn't expect you to, especially without people either finding more instances of ways to get to 'file' or stating that they really tried and couldn't find any way to get to it. -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] what can we do to hide the 'file' type?
Me: > I agree. But we don't have to give up yet. How about instead of hiding > file, we cripple it. Completely. Modify the file type so that when > executing on a sandboxed interpreter, all of the dangerous methods > and attributes of file throw exceptions. Brett: > This is basically what I proposed in the first place! in circles, pulling at his hair like a crazy man> Not quite. Your original proposal had the file type throwing exceptions when the user did something they weren't allowed to (access a file not on the list, etc). This version proposes that it *always* fails, and creates a separate beast (the SecureFileWrapper) for applying the restrictions. Why? Because if the C code in file enforces the rules, then all possible rules need to be written in advance, and you have to hold long arguments about whether to allow subdirectories, restrict file sizes, etc. Whereas SecureFileWrapper could delegate its restrictions to Python functions provided by the USER and then USERS could design whatever level of restriction they wanted. Imagine we want some code to be able to open files only if they contain the letter 'w': # get my own wrapper from __builtins__ myFileWrapper = file # define a function to constrain my callers def filenameContainsLetterW(path, mode): filename = os.path.basename(path) if 'w' not in filename and 'W' not in filename: raise PyXXX_SecurityException # create more restrictive wrapper class MustHaveW_FileWrapper: __metaclass__ = SecureFileWrapperMeta wrapped_file = init_condition = filenameContainsLetterW # register the wrapper so it applies to any code # in this stack frame or lower. The restriction is # automatically lifted when the current stack # frame exits. PyXXX_RegisterFileWrapper( MustHaveW_FileWrapper ) # Invoke less-trusted code with restrictions in place less_trusted_code() If the code fragment shown above ALREADY received a wrapped form of file which was restricted to read-only access, then less_trusted_code() would be restricted to read-only access to files containing 'w'. Okay, the syntax needs work, but the idea is that I can defin restrictions *in python* and apply them to other code. Using the stack to enforce wrappers is something Python code cannot get around (although it does prevent passing more-powerful callbacks to later stackframes). It all depends on whether we think that a simple set of restrictions implementable in C (such as 4 lists: read-only files, read-write files, read-only dirs, write-only dirs) will be sufficient, or if it is valuable to allow end users to fine-tune the restrictions. -- Michael Chermside ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Subversion outage Friday 15:00 GMT
I plan to do some subversion administration tomorrow; in order to be able to roll back changes, I have to disable write access during these changes. The outage shouldn't last longer than one hour; most likely, it will be much faster. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] what can we do to hide the 'file' type?
On 7/6/06, Michael Chermside <[EMAIL PROTECTED]> wrote: Me:> I agree. But we don't have to give up yet. How about instead of hiding> file, we cripple it. Completely. Modify the file type so that when> executing on a sandboxed interpreter, all of the dangerous methods > and attributes of file throw exceptions.Brett:> This is basically what I proposed in the first place! > in circles, pulling at his hair like a crazy man>Not quite. Your original proposal had the file type throwing exceptions when the user did something they weren't allowed to(access a file not on the list, etc). This version proposes thatit *always* fails, and creates a separate beast (theSecureFileWrapper) for applying the restrictions. Why? Because if the C code in file enforces the rules, then all possiblerules need to be written in advance, and you have to hold longarguments about whether to allow subdirectories, restrict filesizes, etc. Whereas SecureFileWrapper could delegate its restrictions to Python functions provided by the USER and thenUSERS could design whatever level of restriction they wanted.Ah, OK, that makes more sense. I was not thinking about allowing for specifying a factory function to return the specific object to use when using open(). That could be rather handy and cool. I will definitely see if I can work it into the API in a reasonable way. -BrettImagine we want some code to be able to open files only if they contain the letter 'w': # get my own wrapper from __builtins__ myFileWrapper = file # define a function to constrain my callers def filenameContainsLetterW(path, mode): filename = os.path.basename(path) if 'w' not in filename and 'W' not in filename: raise PyXXX_SecurityException # create more restrictive wrapper class MustHaveW_FileWrapper: __metaclass__ = SecureFileWrapperMeta wrapped_file = init_condition = filenameContainsLetterW # register the wrapper so it applies to any code # in this stack frame or lower. The restriction is # automatically lifted when the current stack # frame exits. PyXXX_RegisterFileWrapper( MustHaveW_FileWrapper ) # Invoke less-trusted code with restrictions in place less_trusted_code()If the code fragment shown above ALREADY received a wrapped form of file which was restricted to read-onlyaccess, then less_trusted_code() would be restrictedto read-only access to files containing 'w'.Okay, the syntax needs work, but the idea is that I can defin restrictions *in python* and apply them to othercode. Using the stack to enforce wrappers is somethingPython code cannot get around (although it does preventpassing more-powerful callbacks to later stackframes). It all depends on whether we think that a simple setof restrictions implementable in C (such as 4 lists:read-only files, read-write files, read-only dirs,write-only dirs) will be sufficient, or if it is valuable to allow end users to fine-tune therestrictions.-- Michael Chermside ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Musings on concurrency and scoping ("replacing" Javascript)
On Wed, 21 Jun 2006, Brett Cannon wrote:
> I have been working on a design doc for restricted execution of Python
> as part of my dissertation for getting Python into Firefox to replace
> JavaScript on the web.
I've been doing a bunch of Firefox extension programming in Javascript
and suddenly a few of the recent topics here came together in my head
in a silent kapow of thoughts. This is kind of a side note to the
security discussion, but they're all interconnected: network
programming, concurrency, lexical scoping, security.
Client-side web scripting tends to have a callback/continuation-ish
concurrency style because it has to deal with network transactions
(which can stall for long periods of time) in a user interface that
is expected to stay always responsive. The Firefox API is full of
listeners/observers, events, and continuation-like things. So one
thing to consider is that, when Python is used for these purposes,
it may be written in a specialized style.
As i write JavaScript in this style i find i use nested functions
a lot. When i want to set up a callback that uses variables in the
current context, the natural thing to do is to define a new function
in the local namespace. And if that function has to also provide a
callback, then it has another function nested within it and so on.
function spam() {
var local_A = do_work();
do_network_transaction(
new function(result_1) {
var local_B = do_work(result_1);
do_network_transaction(
new function(result_2) {
do_work(local_A, local_B, result_1, result_2);
...
}
);
}
);
}
So it is a possible consequence of embedding Python in Firefox that
people will be using nested functions and lexical scoping in Python
more often, which makes the recent discussion about access to
enclosing scopes more significant.
This is even related to security as well. Namespaces and lexical
scoping are a natural and visually apparent way to limit access.
If, for example, result_1 and result_2 in the above example are
security-sensitive objects like secret keys or networking functions,
you can see just by inspection that they cannot leak outside of
spam() except by directly being passed in an outgoing function call.
The standard Pythonic response to nested functions is to translate
them into classes. But the nested function style has two advantages:
1. Variables are more appropriately scoped; they exist
only where they are meaningful. (In a class, all the
variables would be mixed into one namespace, where
some of them would be invalid some of the time.)
2. Local variables are private. (Class instances don't
get their own private namespaces to play in.)
The first becomes more significant when in a more continuation-y
style because it helps keep the various continuations from
interfering with each other. The second becomes more significant
if you care about restricting untrusted Python code.
-- ?!ng
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] About a month until PSF call for test trackers closes!
Back at the beginning of June, the Python Software Foundation's Infrastructure committee sent out an email requesting people to help us find a replacement tracker for SourceForge (the original announcement can be found at http://wiki.python.org/moin/CallForTrackers ). We asked that people put test trackers loaded with a data dump of Python's SourceForge tracker data online for us to play with. We said that we would close acceptance of test trackers as of August 7. That close date is a little over a month away! If you want to help get your favorite tracker up and going for the Infrastructure committee to evaluate, please visit http://wiki.python.org/moin/CallForTrackers and see if you can help out! We already have a tracker for JIRA up and loaded with the SF data to poke around with. There are also Trac and Roundup trackers in the planning stages that could use some help in getting the SF data imported and getting the servers up. In order for a tracker to be considered it *must* have the SF data loaded up. This will be a necessary step if the tracker is accepted, plus it lets us see how well the tracker lets us manage a large number of bugs. If you have questions or concerns, please email infrastructure at python.org (it is subscription-protected, but your email will be accepted; the subscription page is at http://mail.python.org/mailman/listinfo/infrastructure ). Thank you to those that have helped so far and those that do in the future.-Brett CannonChairman, PSF Infrastructure Committee ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Musings on concurrency and scoping ("replacing" Javascript)
At 05:17 PM 7/6/2006 -0500, Ka-Ping Yee wrote:
>On Wed, 21 Jun 2006, Brett Cannon wrote:
> > I have been working on a design doc for restricted execution of Python
> > as part of my dissertation for getting Python into Firefox to replace
> > JavaScript on the web.
>
>I've been doing a bunch of Firefox extension programming in Javascript
>and suddenly a few of the recent topics here came together in my head
>in a silent kapow of thoughts. This is kind of a side note to the
>security discussion, but they're all interconnected: network
>programming, concurrency, lexical scoping, security.
>
>Client-side web scripting tends to have a callback/continuation-ish
>concurrency style because it has to deal with network transactions
>(which can stall for long periods of time) in a user interface that
>is expected to stay always responsive. The Firefox API is full of
>listeners/observers, events, and continuation-like things. So one
>thing to consider is that, when Python is used for these purposes,
>it may be written in a specialized style.
>
>As i write JavaScript in this style i find i use nested functions
>a lot. When i want to set up a callback that uses variables in the
>current context, the natural thing to do is to define a new function
>in the local namespace. And if that function has to also provide a
>callback, then it has another function nested within it and so on.
>
> function spam() {
> var local_A = do_work();
> do_network_transaction(
> new function(result_1) {
> var local_B = do_work(result_1);
> do_network_transaction(
> new function(result_2) {
> do_work(local_A, local_B, result_1, result_2);
> ...
> }
> );
> }
> );
> }
>
>So it is a possible consequence of embedding Python in Firefox that
>people will be using nested functions and lexical scoping in Python
>more often, which makes the recent discussion about access to
>enclosing scopes more significant.
As much as I'd love to have the nested scope feature, I think it's only
right to point out that the above can be rewritten as something like this
in Python 2.5:
def spam():
local_A = do_work()
result_1 = yield do_network_transaction()
local_B = do_work(result_1)
result_2 = yield do_network_transaction()
do_work(local_A, local_B, result_1, result_2)
...
All you need is an appropriate trampoline (possibly just a decorator) that
takes the objects yielded by the function, and uses them up to set up
callbacks that resume the generator with the returned result.
>This is even related to security as well. Namespaces and lexical
>scoping are a natural and visually apparent way to limit access.
>If, for example, result_1 and result_2 in the above example are
>security-sensitive objects like secret keys or networking functions,
>you can see just by inspection that they cannot leak outside of
>spam() except by directly being passed in an outgoing function call.
>
>The standard Pythonic response to nested functions is to translate
>them into classes. But the nested function style has two advantages:
>
> 1. Variables are more appropriately scoped; they exist
> only where they are meaningful. (In a class, all the
> variables would be mixed into one namespace, where
> some of them would be invalid some of the time.)
>
> 2. Local variables are private. (Class instances don't
> get their own private namespaces to play in.)
>
>The first becomes more significant when in a more continuation-y
>style because it helps keep the various continuations from
>interfering with each other. The second becomes more significant
>if you care about restricting untrusted Python code.
These are all legitimate benefits of a functional style; it's just not
necessarily the case that they also argue in favor of write access to outer
function scopes, since this isn't necessary for writing callbacks, as in
your original example. Co-routines are clearly superior to nested-function
callbacks when the intended control flow is relatively linear. This is, if
I recall correctly, why we added them. :)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Talin wrote: > I propose to create a new type of scoping rule, which I will call > "explicit" lexical scoping, that will co-exist with the current > "implicit" scoping rule that exists in Python today. I'd like to toss one more variant into the mix. If we really need to address variables in an intermediate scope, the most explicit way that I can think of doing so is to write (using Philip's example): def counter(num): scope as outer # "outer" is an arbitrary identifier def inc(): outer.num += 1 return outer.num return inc This is somewhat similar to the unworkable "use the function name" method that's been suggested. The "scope as X" statement associates the name "X" with the namespace of local variables in the scope in which it is executed. Such names are "lexically scoped", and only allow access to or rebinding of existing names from the originating scope (i.e. no "del outer.num" allowed). Cheers, Evan @ 4-am ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
On Thu, 06 Jul 2006 18:28:12 +0200, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > Here's the reason I think this keeps coming up, and why Guido's "just > use a class" argument doesn't really address the actual problem that's > taking place. I agree this argument is not generally applicable in every case, but why not in this specific situation? > In short: in *theory*, a rebinding operator or "nonlocal" declaration is > unnecessary. In *practice*, having one seems quite useful every time > you wander down the path that leads to having to rewrite your code just > because the language won't let you do that one tiny thing - I think this argument is a too general one. To me it is too close to "let's add every possible feature we can find, because it might be usefull to someone" :) One of the things I like about python is that it doesn't do this, and therefore the manual stays relatively small and I don't have to remember all kinds of rarely used features to make best use of the language. (I assume this is not a point of debate. repeating: Python is not Lisp ;-) ) Most of the arguments I've seen on the list are about 'how can we implement this', I'd like to see more arguments on whether this should be implemented at all. I was hoping someone would come up with a good example, so does anyone have one?? > - or so it feels like to the person who's experiencing it. Have you ever been that person, or come across such a situation? O, and I don't think the inc() example is a good one. In this incrementer the function call is all about the side effects, it's even in the name 'increment'. Incrementing is useless unless you increment /something/, so this should be better implemented as a class. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
At 01:00 AM 7/7/2006 +0200, Jan Kanis wrote: >On Thu, 06 Jul 2006 18:28:12 +0200, Phillip J. Eby <[EMAIL PROTECTED]> >wrote: > >>Here's the reason I think this keeps coming up, and why Guido's "just >>use a class" argument doesn't really address the actual problem that's >>taking place. > >I agree this argument is not generally applicable in every case, but why >not in this specific situation? I'm saying that "just use a class" doesn't help in the incremental case. If you happen to know *ahead of time* that you will need a callback to modify state, you can perhaps do this, just like you can always coast your car to a stop if you have enough advance notice. However, sometimes you really want to have brakes. :) >>In short: in *theory*, a rebinding operator or "nonlocal" declaration is >>unnecessary. In *practice*, having one seems quite useful every time >>you wander down the path that leads to having to rewrite your code just >>because the language won't let you do that one tiny thing - > >I think this argument is a too general one. To me it is too close to >"let's add every possible feature we can find, because it might be usefull >to someone" :) Not at all. It's an argument regarding the incremental evolution of code. Python tries to keep the growth of complexity in a code base relatively flat -- incremental effort should be rewarded with incremental benefit. Discontinuities of complexity are to be avoided. The classic example is "Hello world", where in Java you must learn about six or eight different things, but in Python it is just 'print "Hello world"'. If you want to then make it a function, you indent it and add a 'def'. It is all very minimal and incremental, and for the most part, the things that people want to add to Python tend to be wherever discontinuities can occur. It is also true that, because Python is already so well-designed, that these remaining feature areas tend to lean toward the minor and/or obscure: coroutines, conditional expressions, etc. etc. >One of the things I like about python is that it doesn't do this, and >therefore the manual stays relatively small and I don't have to remember >all kinds of rarely used features to make best use of the language. (I >assume this is not a point of debate. repeating: Python is not Lisp ;-) ) And one of the things that makes it not Lisp is that not everyone is free to go around and actually add the new syntax to support their use cases. Guido must first be convinced. Discussions like this one are how we convince him. ;) >>- or so it feels like to the person who's experiencing it. > >Have you ever been that person, or come across such a situation? Many times. The hard thing about trying to provide use cases for this is that of course you can always find another way to write it. It's just that sometimes the nested function is a perfect solution at point in time A, and then at point in time B, a change in the program requires that the nested function mutate a bit of state, resulting in either a rewrite to do it the "right" way, or hacking mutable objects. Both are user-experience discontinuities: a sudden sharp rise in effort compared to reward. This then produces the perception of a "wart". Yes, it's purely a superficial problem. That's why it's called a "wart". :) >O, and I don't think the inc() example is a good one. In this incrementer >the function call is all about the side effects, it's even in the name >'increment'. Incrementing is useless unless you increment /something/, so >this should be better implemented as a class. Any example that's intended to motivate lexical scope modification is going to suffer from either being bogus due to oversimplification, or confusing due to application-specificity. That doesn't make it something that doesn't happen, it's just that the circumstances of it happening are somewhat more organic. By the way, I'm leaving out the people who have backgrounds in Lisp or Scheme and are trying to write it in Python. I don't really care for Lisp, myself, although my Python has become increasingly functional over time. Not because of Lisp (which I've never used apart from a few Emacs hacks 12 years ago, when I didn't even know what functional programming *was*), but because it's often the easiest way to do something. At least, until you bump into the rebinding problem. Personally, the idea to use a simple namespace object to store mutable variables as attributes seems rather appealing as a workaround solution in the absence of a rebinding operator. The other solution I thought of was providing a function called 'rebind()' that could be used like this: rebind(x=23) to change 'x' in the nearest enclosing scope. This solves the problem without adding any syntax, and it's incremental at the point of use. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listin
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
On Fri, 07 Jul 2006 01:25:19 +0200, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > >>> - or so it feels like to the person who's experiencing it. >> >> Have you ever been that person, or come across such a situation? > > Many times. The hard thing about trying to provide use cases for this > is that of course you can always find another way to write it. It's > just that sometimes the nested function is a perfect solution at point > in time A, and then at point in time B, a change in the program requires > that the nested function mutate a bit of state, resulting in either a > rewrite to do it the "right" way, or hacking mutable objects. Well, not an actual example, but close enough. Leaving the rest to Guido then. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Musings on concurrency and scoping ("replacing" Javascript)
On Thu, 6 Jul 2006, Phillip J. Eby wrote: > As much as I'd love to have the nested scope feature, I think it's only > right to point out that the above can be rewritten as something like this > in Python 2.5: > > def spam(): > local_A = do_work() > result_1 = yield do_network_transaction() > local_B = do_work(result_1) > result_2 = yield do_network_transaction() > do_work(local_A, local_B, result_1, result_2) > ... > > All you need is an appropriate trampoline (possibly just a decorator) that > takes the objects yielded by the function, and uses them up to set up > callbacks that resume the generator with the returned result. Clever! Could you help me understand what goes on in do_network_transaction() when you write it this way? In the Firefox/JavaScript world, the network transaction is fired off in another thread, and when it's done it posts an event back to the JavaScript thread, which triggers the callback. And what happens if you want to supply more than one continuation? In my JavaScript code i'm setting up two continuations per step -- one for success and one for failure, since with a network you never know what might happen. -- ?!ng ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
On 7/6/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > Here's the reason I think this keeps coming up, and why Guido's "just use a > class" argument doesn't really address the actual problem that's taking place. (And note that I've recently gone on record as doubting that argument myself.) > When you are writing some function, and you find yourself using a nested > function because it's the closest match for something you're doing, there > occasionally comes a point at which you realize that, gosh, you need to > mutate something in an outer scope. At that point, your choices are to > either kludge it with a mutable, or to reorganize the whole thing. Both of > these *feel* like warts because they're making you do less-than-optimal > things. Your mental "flow" is disrupted because the language is forcing > you to work around it, rather than having a way of working with it. Right. I'm actually not a big fan of the word "flow" -- it often doesn't apply to the way I work myself, and it tends to emphasize the needs of the writer at the expense of the reader. (Perl programmers have excellent flow due to the many ways of doing things in that language. 'nuff said.) But I agree that the need for a refactoring like this to be a simple, local thing is important. (Compare my argument for making print a function -- which by the way received an applause at EuroPython once I explained it, after getting boohs when the slide first went up.) > This is a user experience issue, not a technical one. The fact that you > can say, "you should've done it differently in the first place" doesn't do > anything for your flow. In theory, you could design cars without any > brakes, because people could just coast to a stop if they planned well > enough in advance. ;-) In practice, you need the brakes because people > often don't discover their need to stop until much later in the process. I don't like the brakeless car example either. It is too extreme and opens your argument up to easy objections (the similarity between the cases isn't that great anyway). > This is a flow issue that's specific to *incremental* development. What > happens is that first you refactor the code in a function to include nested > functions. The variable references don't change, you're just indenting > some code. *Then*, it later comes up that you need to rebind a variable, > and now you have to globally change the variable to make it work as a > mutable instead, or else you have to refactor all the variables to be > 'self.' references and put a class in somewhere. > > This destroys the flow of incrementally developing whatever it is you're > developing, and makes you stop to do excise work. That's why being able to > rebind a variable without redefining it is important. Quite apart from the coder's flow, if the code is being developed incrementally, there are probably readers reviewing it before and after the refactoring. It's a lot easier to verify that a refactoring is correct when it doesn't choose a completely different approach. > In short: in *theory*, a rebinding operator or "nonlocal" declaration is > unnecessary. In *practice*, having one seems quite useful every time you > wander down the path that leads to having to rewrite your code just because > the language won't let you do that one tiny thing -- or so it feels like to > the person who's experiencing it. There really is hardly any need to argue the point any more; I've said that I will accept the feature if we can find a mutually acceptable syntax. I still only see these two alternatives as viable: a new keyword (nonlocal being my current favorite) or stretching the meaning of global. After Andrew Koenig's Dijkstra reference I'm beginning to think that the latter may actually be fine. (Oh, and somebody can add "get rid of global" to PEP 3099 -- that ain't gonna happen. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Musings on concurrency and scoping ("replacing" Javascript)
On Jul 6, 2006, at 5:04 PM, Ka-Ping Yee wrote: > On Thu, 6 Jul 2006, Phillip J. Eby wrote: >> As much as I'd love to have the nested scope feature, I think it's >> only >> right to point out that the above can be rewritten as something >> like this >> in Python 2.5: >> >> def spam(): >> local_A = do_work() >> result_1 = yield do_network_transaction() >> local_B = do_work(result_1) >> result_2 = yield do_network_transaction() >> do_work(local_A, local_B, result_1, result_2) >> ... >> >> All you need is an appropriate trampoline (possibly just a >> decorator) that >> takes the objects yielded by the function, and uses them up to set up >> callbacks that resume the generator with the returned result. > > Clever! Could you help me understand what goes on in > do_network_transaction() when you write it this way? In the > Firefox/JavaScript world, the network transaction is fired off > in another thread, and when it's done it posts an event back > to the JavaScript thread, which triggers the callback. > > And what happens if you want to supply more than one continuation? > In my JavaScript code i'm setting up two continuations per step -- > one for success and one for failure, since with a network you never > know what might happen. When you have a failure the yield expression raises an exception instead of returning a result. -bob ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
On 7/6/06, Evan Simpson <[EMAIL PROTECTED]> wrote:
Talin wrote:> I propose to create a new type of scoping rule, which I will call> "explicit" lexical scoping, that will co-exist with the current> "implicit" scoping rule that exists in Python today.
I'd like to toss one more variant into the mix. If we really need toaddress variables in an intermediate scope, the most explicit way that Ican think of doing so is to write (using Philip's example):
def counter(num):scope as outer # "outer" is an arbitrary identifierdef inc():outer.num += 1return outer.numreturn incWhy not extend the interface to the locals builtin and add a __getitem__ that returns a proxy to access locals defined in other lexical scopes via __{get/set/del}attr_:
def counter(num): num = 1def inc(): locals[1].num += 1return outer.numreturn incWhere, for CPython, locals[n] gives access to NamespaceProxy(sys._getframe(n).f_locals). In addition to having a relatively pleasing and explicit syntax, this may be a feasible method for allowing portable introspection into outer scopes without having to export the whole frame object a la sys._getframe(n). I strongly suspect that Jython, IronPython, and PyPy would have little difficulty supporting (and optimizing) this construct.
Lacking core language support, it is easy to roll an object that does just what I suggest. Actual implementation is left to a more motivated reader, of course.Just another crazy idea to throw into the pot.
-Kevin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Musings on concurrency and scoping ("replacing" Javascript)
At 07:04 PM 7/6/2006 -0500, Ka-Ping Yee wrote: >On Thu, 6 Jul 2006, Phillip J. Eby wrote: > > As much as I'd love to have the nested scope feature, I think it's only > > right to point out that the above can be rewritten as something like this > > in Python 2.5: > > > > def spam(): > > local_A = do_work() > > result_1 = yield do_network_transaction() > > local_B = do_work(result_1) > > result_2 = yield do_network_transaction() > > do_work(local_A, local_B, result_1, result_2) > > ... > > > > All you need is an appropriate trampoline (possibly just a decorator) that > > takes the objects yielded by the function, and uses them up to set up > > callbacks that resume the generator with the returned result. > >Clever! Could you help me understand what goes on in >do_network_transaction() when you write it this way? In the >Firefox/JavaScript world, the network transaction is fired off >in another thread, and when it's done it posts an event back >to the JavaScript thread, which triggers the callback. The only difference here is that the callback is to the generator instance's send() method. How the actual callback machinery works, and whether threads are involved are orthogonal to the code itself, making it an ideal implementation-independent way to express asynchronous algorithms. >And what happens if you want to supply more than one continuation? >In my JavaScript code i'm setting up two continuations per step -- >one for success and one for failure, since with a network you never >know what might happen. Errors can be passed back via the generator instance's throw() method. See also peak.events (which could be considered a prototype implementation of PEP 342 and associated trampoline features) and Twisted's Deferred machinery (which allows sending either a result or a "failure" (where failures can be converted back into thrown exceptions). ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python]
Brett Cannon wrote: > On 7/5/06, Talin <[EMAIL PROTECTED]> wrote: >> Transitioning from the checked to the unchecked state could only be done >> via C code. So the 'file' wrapper, for example, would switch over to the >> unchecked interpreter before calling the actual methods of 'file'. That >> C wrapper might also check the current permission state to see what >> operations were legal. > > So add the proper checks in Python/ceval.c:call_function() to check for > this > flag on every object passed in that is called? Right. I also realized that you would need to add code that propagates the checked bit from the class to any instances of the class. So whenever you call a class to create an object, if the class has the checked bit, the instance will have it set as well. >> So essentially, what I propose is to define a simple security primitive >> - which essentially comes down to checking a single bit - and use that >> as a basis to create more complex and subtle security mechanisms. > > Right, but it does require that the proper verification function be turned > on so that the permission bit on 'file' is checked. It kind of seems like > 'rexec' and its f_restricted flag it set on execution frames, except you > are > adding an object-level flag as well. > > Either way, the trick is not fouling up switching between the two checking > functions. > > -Brett I wasn't aware of how rexec worked, but that seems correct to me. Given a 'restricted' flag on a stack frame, and one on the object as well, then the code for checking for permission violations is nothing more than: if (object.restricted && exec_frame.restricted) raise SecurityException In particular, there's no need to call a function to check a "permission level" or "access rights" or anything of the sort - all that stuff is implemented at a higher level. By making the check very simple, it can also be made very fast. And by making it fast, we can afford to call it a lot - for every operation in fact. And if we can call it for every operation, then we don't have to spend time hunting down all of the possible loopholes and ways in which 'file' or other restricted objects might be accessed. Originally I had thought to simply add a check like the above into the interpreter. However, that would mean that *all* code, whether restricted or not, would have to pay the (slight) performance penalty of checking that flag. So instead, I thought it might be more efficient to have two different code paths, one with the check and one without. But all this is based on profound ignorance of the interpreter - there might be a hundred other, better ways to do this without having to create two versions of ceval. Another interesting think about the check bit idea is that you can set it on any kind of object. For example, you could set it on individual methods of a class rather than the class as a whole. However, that's probably needlessly elaborate, since fine-grained access control will be much more elegantly achieved via trusted wrappers. -- Talin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
