Re: [Python-Dev] Import semantics
On 6/25/06, Frank Wierzbicki <[EMAIL PROTECTED]> wrote: > Sorry for the untrimmed conversation, but I've cc'ed jython-dev, my > comments are at the bottom. > > On 6/12/06, Guido van Rossum <[EMAIL PROTECTED]> wrote: > > On 6/12/06, Samuele Pedroni <[EMAIL PROTECTED]> wrote: > > > Fabio Zadrozny wrote: > > > > Python and Jython import semantics differ on how sub-packages should be > > > > accessed after importing some module: > > > > > > > > Jython 2.1 on java1.5.0 (JIT: null) > > > > Type "copyright", "credits" or "license" for more information. > > > > >>> import xml > > > > >>> xml.dom > > > > > > > > > > > > Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] on > > > > win32 > > > > Type "help", "copyright", "credits" or "license" for more information. > > > > >>> import xml > > > > >>> xml.dom > > > > Traceback (most recent call last): > > > > File "", line 1, in ? > > > > AttributeError: 'module' object has no attribute 'dom' > > > > >>> from xml.dom import pulldom > > > > >>> xml.dom > > > > > > > > > > > > Note that in Jython importing a module makes all subpackages beneath it > > > > available, whereas in python, only the tokens available in __init__.py > > > > are accessible, but if you do load the module later even if not getting > > > > it directly into the namespace, it gets accessible too -- this seems > > > > more like something unexpected to me -- I would expect it to be > > > > available only if I did some "import xml.dom" at some point. > > > > > > > > My problem is that in Pydev, in static analysis, I would only get the > > > > tokens available for actually imported modules, but that's not true for > > > > Jython, and I'm not sure if the current behaviour in Python was > > > > expected. > > > > > > > > So... which would be the right semantics for this? > > > > > > the difference in Jython is deliberate. I think the reason was to mimic > > > more the Java style for this, in java fully qualified names always work. > > > In jython importing the top level packages is enough to get a similar > > > effect. > > > > > > This is unlikely to change for backward compatibility reasons, at least > > > from my POV. > > > > IMO it should do this only if the imported module is really a Java > > package. If it's a Python package it should stick to python semantics > > if possible. > > > > -- > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > This is a tough one since the BDFL and Samuele disagree here. Perhaps > we should document the Java import behavior as permanent, but document > the Python imports in Jython as being deprecated but available until > some future release? I believe we would keep it at least through > Jython 2.3. Hi Frank, Have you and/or the Jython community made up your mind about this? The thread seems to have disappeared after you posted (or perhaps it continued only on jython-dev, which I don't read?). Also, I just realized that you're the new Jython maintainer. Is *that* official? I'd like to offer you my congratulations, and, more importantly, any support you might need. I find Jython an important part for Python's long-term stategy. I'm asked occasionally what the status of Jython is; people point out that the last release was 2.1 many years ago and the website has no news since early 2005; thy're afraid that Jython is dying and that it's not a viable choice for new projects. I'm very happy to be able to tell them that soon there will be a 2.3 release and yes there *is* continued support... So if you need anything from me or from the PSF, please let me know! -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Import semantics
On Wednesday 05 July 2006 18:12, Guido van Rossum wrote: > I'm asked > occasionally what the status of Jython is; people point out that > the last release was 2.1 many years ago and the website has no news > since early 2005; thy're afraid that Jython is dying and that it's > not a viable choice for new projects. I'm very happy to be able to > tell them that soon there will be a 2.3 release and yes there *is* > continued support... So if you need anything from me or from the > PSF, please let me know! In that case, why not post a news item saying this? The website is probably the first place people look... Anthony -- Anthony Baxter <[EMAIL PROTECTED]> It's never too late to have a happy childhood. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
On 7/5/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > At 05:49 AM 7/5/2006 +0200, Guido van Rossum wrote: > >>* Alternate spelling of outer names when binding (e.g. ".x = whatever" to > >>bind an outer x) > > > >We looked at and rejected "globals.x = whatever". I think the same > >reasoning applies here. > > I thought the 'globals.x' proposal required that 'x' always be accessed > using 'globals', even if it wasn't being rebound. I don't see a problem > with requiring '.x' to be used for both reading and writing of outer-scope > names; it just shouldn't be required for an outer-scope name that you don't > rebind in the current scope. That symmetry requirement can't be > implemented with the 'globals.x' approach unless 'globals' is treated > specially by the compiler. > > Using the classic nonsense example: > > def counter(num): > def inc(): > .num += 1 > return .num > return inc > > If inc() only needed to *read* num, it could just use 'num' without the > '.', and be nicely backward compatible with today's Python. > > (Note: It should be illegal to use both '.num' and 'num' in the same scope, > whether writing or reading the value, to prevent readers from becoming > confused about what variable you mean. It should also be required that the > compiler can see a definition of 'num' in an outer scope if you use the > '.num' syntax, so that misspelling a name doesn't create a global variable.) > > I personally think this approach could be the overall least-intrusive > solution as far as syntax goes. It also allows for dropping the 'global' > keyword in 3.x, and it has a nice EIBTI feel to it, as it allows you to > highlight closure variables in an inner function by using the '.'. It's > often not obvious when an inner function (such as a decorator returned by a > decorator factory) is using variables that were defined in the outer scope; > the leading '.' would make them stand out, and so could be considered the > recommended code style when referring to outer variables. > > In addition, there's a nice symmetry between nested functions and top-level > functions, e.g. in this global version of the counter example: > > num = 0 > def inc(): > .num += 1 > return .num > > The principle downside taht I see is that it uses semi-random punctuation > in place of keywords. OTOH, we are already using more-or-less this syntax > for relative imports, so reusing it to mean "relative variables" seems to > at least avoid creating any entirely new principles. :) > > Anyway, I won't argue this one further; I just wanted to make sure it had > been considered, as I'm not sure that you were reading the thread where it > was first brought up (possibly as long as a few months ago). Thanks for bringing this up. I'm not sure what I think of it yet. One problem I see is that there might end up being two ways to reference variables in outer scopes: .num if you plan to assign to it, or just num if you only reference it. I find that the most disurbing issue so far; modified global declarations or outer declarations don't have this problem. Would this also use ..num to refer to num in an outer scope two levels removed? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Import semantics
On 7/5/06, Anthony Baxter <[EMAIL PROTECTED]> wrote: > On Wednesday 05 July 2006 18:12, Guido van Rossum wrote: > > I'm asked > > occasionally what the status of Jython is; people point out that > > the last release was 2.1 many years ago and the website has no news > > since early 2005; thy're afraid that Jython is dying and that it's > > not a viable choice for new projects. I'm very happy to be able to > > tell them that soon there will be a 2.3 release and yes there *is* > > continued support... So if you need anything from me or from the > > PSF, please let me know! > > In that case, why not post a news item saying this? The website is > probably the first place people look... I'm all for that; but I don't have webmaster privileges (nor do I want them :-). Frank? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
On Wednesday 05 July 2006 18:21, Guido van Rossum wrote: > Would this also use ..num to refer to num in an outer scope two > levels removed? Ew! I don't want to even think about debugging ...x vs x Anthony -- Anthony Baxter <[EMAIL PROTECTED]> It's never too late to have a happy childhood. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Sorry, I should have added a ... :-) On 7/5/06, Anthony Baxter <[EMAIL PROTECTED]> wrote: > On Wednesday 05 July 2006 18:21, Guido van Rossum wrote: > > Would this also use ..num to refer to num in an outer scope two > > levels removed? > > Ew! > > I don't want to even think about debugging > > ...x > vs > x > > Anthony > -- > Anthony Baxter <[EMAIL PROTECTED]> > It's never too late to have a happy childhood. > -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Guido van Rossum wrote: > On 7/5/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote: >> At 12:18 AM 7/5/2006 +0200, Guido van Rossum wrote: >> >I don't see anything else that's attractive. The realistic options are: >> > >> >1. do nothing >> >2. extend global's meaning >> >3. add outer keyword >> >> Did you also consider and reject: >> >> * Alternate binding operators (e.g. ":=", ".=", etc.) > > Brr. > >> * Alternate spelling of outer names when binding (e.g. ".x = whatever" to >> bind an outer x) > > We looked at and rejected "globals.x = whatever". I think the same > reasoning applies here. > >> If so, then these should probably be added to the "rejected alternatives" >> for Py3K so they don't get rehashed. > > Georgbot? I added the alternative binding operators. The discussion about ".x" seems to be still in progress. Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Guido van Rossum wrote: > Would this also use ..num to refer to num in an outer scope two levels > removed? I realize this was a wink, but it is a valid problem with the "dot"-proposal. def foo(n): def bar(n): def baz(): return .n So, which 'n' outer 'n' is being referenced? Seems like you need to either be able to do multiple dots (ugly, hard to read) or only do a single-step outwards reference. But then that has it's own problems, if I meant the 'n' passed into 'foo', then I have to resort to such nonsense as: def foo(n): def bar(n): foon = .n def baz(): return .foon It would almost be cute if you could do something like ".foo.n" to get to the correct variable. If python maintains it's current scoping rules, then it seems like it works out, but I haven't thought this one all the way through. def foo(n): def bar(n): def baz(): return .foo.n + .bar.n -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Guido van Rossum <[EMAIL PROTECTED]> wrote: > 1. do nothing > 2. extend global's meaning > 3. add outer keyword 2.5. extend global syntax to cover both [really global] and [innermost matching scope]. eg. global x, y outer # trailing non-keyword global in x, y # re-use keyword not global x# ceci n'est pas un global ... # something less ugly? > Personally it's not a burning need Agreed. Inability to write as well as read nested scopes is more of an aesthetic wart than a practical one IMO. -- And Clover mailto:[EMAIL PROTECTED] http://www.doxdesk.com/ -- And Clover mailto:[EMAIL PROTECTED] http://www.doxdesk.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Guido van Rossum wrote:
> On 7/5/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote:
> > Did you also consider and reject:
> >
> > * Alternate binding operators (e.g. ":=", ".=", etc.)
>
> Brr.
That's too bad :(
I still find a rebinding operator (":=" being my favorite) much, *much*
more appealing than any of the alternative proposals. It's beautifully
symmetrical with "assignment means local". It also pretty much makes the
global statement redundant.
The only downside I see is that it may cause a fairly big shift in
style: I for one would use := for rebinding local names. While I think
that would be an improvement (eg. by catching typo's earlier), it's
*different*.
Just
--
Change is bad. We fear change. -- Garth Algar
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
On 7/5/06, Just van Rossum <[EMAIL PROTECTED]> wrote:
> Guido van Rossum wrote:
>
> > On 7/5/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote:
> > > Did you also consider and reject:
> > >
> > > * Alternate binding operators (e.g. ":=", ".=", etc.)
> >
> > Brr.
>
> That's too bad :(
>
> I still find a rebinding operator (":=" being my favorite) much, *much*
> more appealing than any of the alternative proposals. It's beautifully
> symmetrical with "assignment means local". It also pretty much makes the
> global statement redundant.
>
> The only downside I see is that it may cause a fairly big shift in
> style: I for one would use := for rebinding local names. While I think
> that would be an improvement (eg. by catching typo's earlier), it's
> *different*.
Hallo broer! :-)
I wonder what this should mean then:
def outer():
def inner():
x := 1
What is x's scope?
Also, a := operator allows all sorts of left-hand sides that don't
necessarily make sense, e.g.
x.foo := 1
x[0] := 1
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
2006/7/5, Just van Rossum <[EMAIL PROTECTED]>:
> Guido van Rossum wrote:
>
> > On 7/5/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote:
> > > Did you also consider and reject:
> > >
> > > * Alternate binding operators (e.g. ":=", ".=", etc.)
> >
> > Brr.
>
> That's too bad :(
>
> I still find a rebinding operator (":=" being my favorite) much, *much*
> more appealing than any of the alternative proposals. It's beautifully
> symmetrical with "assignment means local". It also pretty much makes the
> global statement redundant.
>
> The only downside I see is that it may cause a fairly big shift in
> style: I for one would use := for rebinding local names. While I think
> that would be an improvement (eg. by catching typo's earlier), it's
> *different*.
I suggest <- as an assignment operator instead of := - it's used in
OCaml and it looks *very* different, yet still makes sense.
x = 0
print x
def f():
x = 1 # bind locally
print x
def g():
x <- 42 # assign "lexically"
print x
f()
print x
g()
print x
prints
0
1
0
42
42
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 PCTotD (pre-coffee thought of the day): On Jul 5, 2006, at 6:17 AM, Marek "Baczek" Baczyński wrote: > 2006/7/5, Just van Rossum <[EMAIL PROTECTED]>: >> Guido van Rossum wrote: >> >>> On 7/5/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote: Did you also consider and reject: * Alternate binding operators (e.g. ":=", ".=", etc.) >>> >>> Brr. Clearly we need the "as if in" operator: x = 0 def foo(): x = 1 def bar(): x = 2 def baz(): x as if in foo = 3 x as if in global += 1 (Personally, I've never really needed this much, but if you have to have it, be explicit! :) - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.3 (Darwin) iQCVAwUBRKuY+XEjvBPtnXfVAQI7WAQAp9fp8SZU8lJdyllrrbFhR9lWxF+pbZ9d +OBv0VEFiFD3aLuKc7y7rAyCWfRVMsoO3FsapeC0/CFJDpRoDDRB7qLzX7He9Fyf JXh3g489RdokxwZClIl38RVgM7WNVK8lzFr8Kf8eTjEUItJoJjk5m2vhymQpfWYG /OeU5xCABxw= =OBRd -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Guido van Rossum wrote: > Hallo broer! :-) Yo :) > I wonder what this should mean then: > > def outer(): > def inner(): > x := 1 > > What is x's scope? UnboundVariableError: variable 'x' referenced before assignment Or a SyntaxError if the compiler can detect it. > Also, a := operator allows all sorts of left-hand sides that don't > necessarily make sense, e.g. > > x.foo := 1 > x[0] := 1 True, although maybe they could be made to make sense by defining special methods: __rebindattr__ __rebinditem__ <0.5 wink> Just ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
I know this is very similar to the "global.x =" syntax, which was already shot down?, but wouldn't allowing access to a functions locals from within, by prefixing the name, be a good way to disambiguate what happens (instead of any operator to indicate outer scope, like .x = 3 or the like)? I guess this necessitates "global.x =" as well, though. def foo(): def bar(): foo.x = 3 print x # prints 3 I seem to recall that this syntax has been proposed before, though not in this discussion. But my memory is murky. //Simon ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Marek "Baczek" Baczyñski wrote: > I suggest <- as an assignment operator instead of := - it's used in > OCaml and it looks *very* different, yet still makes sense. Except it's currently valid Python syntax: >>> x = 0 >>> x <- 42 False >>> Just ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Barry> Clearly we need the "as if in" operator: Why not be more direct? x = 0 def foo(): x = 1 def bar(): x = 2 def baz(): x in foo = 3 x in global += 1 By naming the function in which the binding is to occur you avoid problems of someone coming along and adding or deleting functions between the assignment (in baz) and the target of the assignment (x in foo) but then forgetting to increment or decrement the counters that refer to a fixed number of levels above the current function. Barry> (Personally, I've never really needed this much, but if you have Barry> to have it, be explicit! :) Nor I. I can't think of any situations in my programming where I've used nested functions, but I was never a LISPer... Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
On 5 jul 2006, at 11.40, Scott Dial wrote: > Guido van Rossum wrote: >> Would this also use ..num to refer to num in an outer scope two >> levels removed? > > I realize this was a wink, but it is a valid problem with the > "dot"-proposal. > > def foo(n): >def bar(n): > def baz(): >return .n > > So, which 'n' outer 'n' is being referenced? Seems like you need to > either be able to do multiple dots (ugly, hard to read) or only do a > single-step outwards reference. But then that has it's own > problems, if > I meant the 'n' passed into 'foo', then I have to resort to such > nonsense as: No, it's actually not a problem. foo()'s "n" should just be hidden. If you don't want it to be hidden, don't write your function that way. If you find you need deeply nested functions where local names shadow names in outer scopes that you need to access you might want to think of another way to solve your problem. //Simon ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Simon Percivall wrote: > On 5 jul 2006, at 11.40, Scott Dial wrote: > > Guido van Rossum wrote: > >> Would this also use ..num to refer to num in an outer scope two > >> levels removed? > > > > I realize this was a wink, but it is a valid problem with the > > "dot"-proposal. > > > > def foo(n): > >def bar(n): > > def baz(): > >return .n > > > > So, which 'n' outer 'n' is being referenced? Seems like you need to > > either be able to do multiple dots (ugly, hard to read) or only do a > > single-step outwards reference. But then that has it's own > > problems, if > > I meant the 'n' passed into 'foo', then I have to resort to such > > nonsense as: > > > No, it's actually not a problem. foo()'s "n" should just be > hidden. If you don't want it to be hidden, don't write your > function that way. If you find you need deeply nested functions > where local names shadow names in outer scopes that you need to > access you might want to think of another way to solve your > problem. +1, YAGNI. Just ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Jul 5, 2006, at 9:39 AM, [EMAIL PROTECTED] wrote: > > Barry> Clearly we need the "as if in" operator: > > Why not be more direct? Sure, why not? :) Then we can reserve the "as if" operator for those things that Guido has rejected, but that we sneak in while he's not looking. like-omg-gag-me-with-a-spoon-ly y'rs, - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.3 (Darwin) iQCVAwUBRKvK8nEjvBPtnXfVAQLbswQArfvIoCdCHmryk3qkOvG6BE0Q1iW7dk0O eI178nG1tY+02JLyrPb1RcjdJG0W0wPwugvVNwVlz29cNkt048uEme6ZBfv3wCt/ bQSWTnDym/OWtQhUtsaw7V5K1o/bP5noqS2MQAcafk4lARv7TAWbBNkPqpk/yFmp 2yNhIfngjts= =thYd -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Patch for commands.py to provide callback
Greetings all,I have attached a patch for commands.py to provide a callback ability. Example use:-import commandscmd = 'top -b -n2'def fancy(out):
print 'GOT(%s)' % out.strip()commands.cb = fancy(s,o) = commands.getstatusoutput(cmd)print 'OUTPUT (%s)' % o-I am not sure if this is the proper forum or means to submit something like this, so please forgive me and advise accordingly if I am in error. The basic idea is obvious, to allow long-running commands to call back whenever there is output. This is against python
2.4. Please let me know if you have any questions or suggestions.thanks!!-brad
--- commands.py.ORIG 2006-06-30 12:19:40.0 -0600
+++ commands.py 2006-07-05 09:15:29.0 -0600
@@ -47,16 +47,28 @@
# Ditto but preserving the exit status.
# Returns a pair (sts, output)
#
+
def getstatusoutput(cmd):
-"""Return (status, output) of executing cmd in a shell."""
+
import os
-pipe = os.popen('{ ' + cmd + '; } 2>&1', 'r')
-text = pipe.read()
-sts = pipe.close()
+
+try:
+myCB = cb
+except:
+myCB = None
+
+text = ''
+
+(stdin,stdout) = os.popen2( '{ ' + cmd + '; } 2>&1', 'r')
+for line in stdout:
+if myCB != None:
+myCB(line)
+text += line
+sts = stdout.close()
+
if sts is None: sts = 0
if text[-1:] == '\n': text = text[:-1]
-return sts, text
-
+return sts,text
# Make command argument from directory and pathname (prefix space, add quotes).
#
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Patch for commands.py to provide callback
Brad Doctor wrote: > I am not sure if this is the proper forum or means to submit something > like this, so please forgive me and advise accordingly if I am in error. to make sure that they don't just disappear under a zillion other mails, patches should be submitted to the patch tracker: http://sourceforge.net/patch/?group_id=5470 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Patch for commands.py to provide callback
Cool, thank you Fredrik -- going there now.-bradOn 7/5/06, Fredrik Lundh <[EMAIL PROTECTED] > wrote:Brad Doctor wrote:> I am not sure if this is the proper forum or means to submit something > like this, so please forgive me and advise accordingly if I am in error.to make sure that they don't just disappear under a zillion other mails,patches should be submitted to the patch tracker: http://sourceforge.net/patch/?group_id=5470___Python-Dev mailing list Python-Dev@python.orghttp://mail.python.org/mailman/listinfo/python-devUnsubscribe: http://mail.python.org/mailman/options/python-dev/brad.doctor%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Patch for commands.py to provide callback
Since commands.getstatusoutput() is such a trivial wrapper around os.popen(), why bother patching commands.py? On 7/5/06, Brad Doctor <[EMAIL PROTECTED]> wrote: > Greetings all, > > I have attached a patch for commands.py to provide a callback ability. > Example use: > > - > import commands > > cmd = 'top -b -n2' > > def fancy(out): > print 'GOT(%s)' % out.strip() > > commands.cb = fancy > > (s,o) = commands.getstatusoutput(cmd) > print 'OUTPUT (%s)' % o > - > > I am not sure if this is the proper forum or means to submit something like > this, so please forgive me and advise accordingly if I am in error. The > basic idea is obvious, to allow long-running commands to call back whenever > there is output. This is against python 2.4. Please let me know if you > have any questions or suggestions. > > thanks!! > -brad > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Patch for commands.py to provide callback
Because it is a great way to consistently use popen(). Rather than write something specific each time, our site/company prefers to use commands to keep it all consistent.-brad On 7/5/06, Guido van Rossum <[EMAIL PROTECTED]> wrote: Since commands.getstatusoutput() is such a trivial wrapper aroundos.popen(), why bother patching commands.py?On 7/5/06, Brad Doctor <[EMAIL PROTECTED]> wrote: > Greetings all,>> I have attached a patch for commands.py to provide a callback ability.> Example use:>> -> import commands> > cmd = 'top -b -n2'>> def fancy(out):> print 'GOT(%s)' % out.strip()>> commands.cb = fancy>> (s,o) = commands.getstatusoutput(cmd)> print 'OUTPUT (%s)' % o > ->> I am not sure if this is the proper forum or means to submit something like> this, so please forgive me and advise accordingly if I am in error. The> basic idea is obvious, to allow long-running commands to call back whenever > there is output. This is against python 2.4. Please let me know if you> have any questions or suggestions.>> thanks!!> -brad>> ___ > Python-Dev mailing list> Python-Dev@python.org> http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe:> http://mail.python.org/mailman/options/python-dev/guido%40python.org Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Import semantics
> In that case, why not post a news item saying this? The website is > probably the first place people look... > I think any news other than "here is the beta -- follow this link to download it" would be kind of a waste at this point. And that will come when I finish __slots__, subclassable "type", and figure out how to put a release together, which will be "real soon now" -- but really. Thanks, -Frank ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Patch for commands.py to provide callback
Hm. It sounds like your company would be better off developing a library of handy tools that it uses over and over, instead of on the one hand using only standard library, and then if the standard library doesn't provide the feature you need, proposing a patch. I wouldn't be so critical if I thought others would benefit from your patch. But it seems a rather arbitrary place to insert a debugging hook (if that's what you need it for), and I don't think I've seen others with the same need. Or perhaps you could start by explaining the problem you are trying to solve before proposing a specific patch? --Guido On 7/5/06, Brad Doctor <[EMAIL PROTECTED]> wrote: > Because it is a great way to consistently use popen(). Rather than write > something specific each time, our site/company prefers to use commands to > keep it all consistent. > > -brad > > > On 7/5/06, Guido van Rossum <[EMAIL PROTECTED]> wrote: > > Since commands.getstatusoutput() is such a trivial wrapper around > > os.popen(), why bother patching commands.py? > > > > On 7/5/06, Brad Doctor <[EMAIL PROTECTED]> wrote: > > > Greetings all, > > > > > > I have attached a patch for commands.py to provide a callback ability. > > > Example use: > > > > > > - > > > import commands > > > > > > cmd = 'top -b -n2' > > > > > > def fancy(out): > > > print 'GOT(%s)' % out.strip() > > > > > > commands.cb = fancy > > > > > > (s,o) = commands.getstatusoutput(cmd) > > > print 'OUTPUT (%s)' % o > > > - > > > > > > I am not sure if this is the proper forum or means to submit something > like > > > this, so please forgive me and advise accordingly if I am in error. The > > > basic idea is obvious, to allow long-running commands to call back > whenever > > > there is output. This is against python 2.4. Please let me know if you > > > have any questions or suggestions. > > > > > > thanks!! > > > -brad > > > > > > ___ > > > Python-Dev mailing list > > > Python-Dev@python.org > > > http://mail.python.org/mailman/listinfo/python-dev > > > Unsubscribe: > > > > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > > > > > > > > > > > > > > -- > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Time-out in URL Open
2006/7/4, Guido van Rossum <[EMAIL PROTECTED]>: > > This affect all the sockets. > > So, assuming your app is single-threaded, set the timeout, call > urlopen(), and reset the timeout to None. No, it's multithreaded, :D > > And I hit the problem when servicing > > information with a web service (TCPServer), and I need to timeout the > > connection of the URLOpen *only*. > > That's not so easy even if you were to have a timeout parameter to > urlopen(). You'd have to implement that peculiarity in all the layers > (in this case, urllib and httplib; and possibly ftplib, gopherlib etc. > :-) Yes, it's not SO easy, because, as you said, you have to dig into the layers until you hit the actual socket creation and modify the timeout for that socket only. That's why I think that this should be handled in the standard library and not left to implement to whoever will need it, :) -- .Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
At 10:21 AM 7/5/2006 +0200, Guido van Rossum wrote: >Thanks for bringing this up. I'm not sure what I think of it yet. One >problem I see is that there might end up being two ways to reference >variables in outer scopes: .num if you plan to assign to it, or just >num if you only reference it. I find that the most disurbing issue so >far; modified global declarations or outer declarations don't have >this problem. Well, you could make it mandatory in Py3K I suppose, though I'm not sure I like it being mandatory, due to the frequent need to reference top-level names that would cause an awful lot of dots to start popping up. But for existing Python, the optional nature of the '.' allows existing code to run unchanged. And for versions that support the new syntax, using a leading '.' for all non-global, non-locals should be considered "good style" since it highlights the dependency and practically shouts "tricky stuff here, pay attention." Ironically, having *only* one way to refer to outer variables makes it impossible to communicate this distinction in the common read-only case. >Would this also use ..num to refer to num in an outer scope two levels >removed? I think that's unnecessary; it would be much better to use variables with distinct names. By the way, an interesting thought for Py3K is that you could maybe use this syntax to do away with explicit 'self', if you consider the class' namespace to be part of a function's closure. E.g.: class Foo: whee = 42 def bar(baz): print .whee Consider this: if Foo were a function rather than a class, each invocation of Foo would yield a new namespace in which 'whee' is defined. However, each invocation of a *class* also yields a new namespace. So there's a definite symmetry in using .whee to refer to an instance attribute of Foo. The big problem that comes to mind with that idea is that it makes it impossible to have argument names that are the same as attribute names, unless the 'whee'/'.whee' prohibition were relaxed. :( But it's an intriguing thought, nonetheless. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Time-out in URL Open
OK, you've convinced me. Now where's that SF patch you were promising? :-) --Guido On 7/5/06, Facundo Batista <[EMAIL PROTECTED]> wrote: > 2006/7/4, Guido van Rossum <[EMAIL PROTECTED]>: > > > > This affect all the sockets. > > > > So, assuming your app is single-threaded, set the timeout, call > > urlopen(), and reset the timeout to None. > > No, it's multithreaded, :D > > > > > And I hit the problem when servicing > > > information with a web service (TCPServer), and I need to timeout the > > > connection of the URLOpen *only*. > > > > That's not so easy even if you were to have a timeout parameter to > > urlopen(). You'd have to implement that peculiarity in all the layers > > (in this case, urllib and httplib; and possibly ftplib, gopherlib etc. > > :-) > > Yes, it's not SO easy, because, as you said, you have to dig into the > layers until you hit the actual socket creation and modify the timeout > for that socket only. > > That's why I think that this should be handled in the standard library > and not left to implement to whoever will need it, :) > > -- > .Facundo > > Blog: http://www.taniquetil.com.ar/plog/ > PyAr: http://www.python.org/ar/ > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
At 05:40 AM 7/5/2006 -0400, Scott Dial wrote: >Guido van Rossum wrote: > > Would this also use ..num to refer to num in an outer scope two levels > removed? > >I realize this was a wink, but it is a valid problem with the >"dot"-proposal. Actually, it isn't. :) See below. >def foo(n): >def bar(n): > def baz(): >return .n > >So, which 'n' outer 'n' is being referenced? Notice that this is a made-up example. Existing Python scoping rules don't allow this! Thus your example is not a bug, it's a feature request. And I say we say "no" to adding this feature. ;-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
On 7/5/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > By the way, an interesting thought for Py3K is that you could maybe use > this syntax to do away with explicit 'self', if you consider the class' > namespace to be part of a function's closure. Sorry, but now I am *definitely* -1. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Import semantics
On 7/5/06, Guido van Rossum <[EMAIL PROTECTED]> wrote: > Hi Frank, > > Have you and/or the Jython community made up your mind about this? The > thread seems to have disappeared after you posted (or perhaps it > continued only on jython-dev, which I don't read?). The thread pretty much stopped there. I think a modification of the import semantics won't really come up again until 2.3 is out, since releases and reviving Jython take priority over backwards incompatible features. For my part I would like to think that a goal for Jython should be: Pure Python code developed first on Jython should run without change on CPython. Which would mean we will eventually need to change the import semantics for importing Python code (though I doubt we will change the semantics for importing Java packages any time soon). Whether that can be done in 2.x, or if this change is so incompatible that we need to think about it in a "Jython 3000" way, I don't really know. > > Also, I just realized that you're the new Jython maintainer. Is *that* > official? It is official, at least in the unofficial way that Jython appears to work: Brian handed the baton to me after (I presume) Samuele Pedroni had handed the baton to Brian. Brian's email is here: http://sourceforge.net/mailarchive/message.php?msg_id=13859029 That said, I still regard Samuele Pedroni as the ultimate authority on Jython and give him pretty much full veto power. He fortunately continues to watch the checkins and prods me when I go in the wrong direction. > I'd like to offer you my congratulations, and, more > importantly, any support you might need. Thanks! > I find Jython an important > part for Python's long-term stategy. That's good to know. > I'm asked occasionally what the > status of Jython is; people point out that the last release was 2.1 > many years ago and the website has no news since early 2005; they're > afraid that Jython is dying and that it's not a viable choice for new > projects. > I'm very happy to be able to tell them that soon there will > be a 2.3 release and yes there *is* continued support... Perhaps for large values of "soon" -- but seriously, I am working hard to polish the coming release, make it easier for new developers to read the code, and when I have a chance, update the website. Jython is my first serious plunge into open source contributions (it is really too bad that I couldn't have been a journeyman for another year or so first, but circumstances did not allow that). I have suffered from some over-optimism when asked for release dates, so I'm really afraid to give too sharp of a definition for "soon". That said, I believe a 2.2 version will be out sometime this summer, and a 2.3 should follow relatively quickly (maybe 6 months or so) The 2.2->2.3 will (hopefully) be relatively quick because 2.2 is an unfortunate but at this point unavoidable mix of 2.2 and 2.3 features, with heavy favoring of 2.3 features. Jython has had a history of having a very small number of developers, and a history of too much off-list discussions so that new developers have a very hard time catching up. I'm trying hard to change that culture. There are 4 or 5 developers contributing patches lately -- so I'm actually spending more time trying to help them along instead of concentrating 100% on a release -- I really think they are the most important predictor of the future of Jython. I have high hopes for the project's future health. Anyhow, this post is getting long, so to sum up: Jython is alive, and we aren't going to let it die. Regards, -Frank ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Time-out in URL Open
What about doing it with a per-thread-timeout in TLS (overriding the global one if a thread does have it set in its TLS)? Not as clean, but perhaps far easier to implement than patching dozens of modules/functions/classes to provide timeout= options everywhere... Alex On 7/5/06, Guido van Rossum <[EMAIL PROTECTED]> wrote: > OK, you've convinced me. Now where's that SF patch you were promising? :-) > > --Guido > > On 7/5/06, Facundo Batista <[EMAIL PROTECTED]> wrote: > > 2006/7/4, Guido van Rossum <[EMAIL PROTECTED]>: > > > > > > This affect all the sockets. > > > > > > So, assuming your app is single-threaded, set the timeout, call > > > urlopen(), and reset the timeout to None. > > > > No, it's multithreaded, :D > > > > > > > > And I hit the problem when servicing > > > > information with a web service (TCPServer), and I need to timeout the > > > > connection of the URLOpen *only*. > > > > > > That's not so easy even if you were to have a timeout parameter to > > > urlopen(). You'd have to implement that peculiarity in all the layers > > > (in this case, urllib and httplib; and possibly ftplib, gopherlib etc. > > > :-) > > > > Yes, it's not SO easy, because, as you said, you have to dig into the > > layers until you hit the actual socket creation and modify the timeout > > for that socket only. > > > > That's why I think that this should be handled in the standard library > > and not left to implement to whoever will need it, :) > > > > -- > > .Facundo > > > > Blog: http://www.taniquetil.com.ar/plog/ > > PyAr: http://www.python.org/ar/ > > ___ > > Python-Dev mailing list > > Python-Dev@python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/aleaxit%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Time-out in URL Open
Guido> OK, you've convinced me. Now where's that SF patch you were Guido> promising? :-) A starting point is probably the patch Georg referred to a couple days ago: Georg> There was one patch that did this: http://python.org/sf/723312. Alas, it's assigned to me and I let it get so stale that Martin asked the author to update it for 2.5 a couple months ago. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Import semantics
Frank> That said, I still regard Samuele Pedroni as the ultimate Frank> authority on Jython and give him pretty much full veto power. He Frank> fortunately continues to watch the checkins and prods me when I Frank> go in the wrong direction. Does that make Samele the DBPV (Dictator benevolo per vita)? ;-) Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Import semantics
In Italian that would be DBAV (Dittatore benevolo a vita)...;-) Alex On 7/5/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > > Frank> That said, I still regard Samuele Pedroni as the ultimate > Frank> authority on Jython and give him pretty much full veto power. He > Frank> fortunately continues to watch the checkins and prods me when I > Frank> go in the wrong direction. > > Does that make Samele the DBPV (Dictator benevolo per vita)? ;-) > > Skip > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/aleaxit%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Import semantics
On 7/5/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > > Frank> That said, I still regard Samuele Pedroni as the ultimate > Frank> authority on Jython and give him pretty much full veto power. He > Frank> fortunately continues to watch the checkins and prods me when I > Frank> go in the wrong direction. > > Does that make Samele the DBPV (Dictator benevolo per vita)? ;-) > > Skip > I wonder if Samuele even wants that role? Anyway, I believe Samuele has more experience with Jython as it is currently implemented than anyone else, so I must take anything he says about Jython very seriously. However, when it comes to pure *Python* matters (no C and no J) which includes half of Jython's import semantics, I think it is still Guido's opinion that matters most. Without that, there is too much chaos. -Frank ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Guido writes: [discussion of how to fix the can't-bind-outer-scope-vars wart] > I think we have to continue to search for a solution that extends the > idea of global declarations. > > I've proposed extending its meaning to refer to the nearest outer > scope where the variable is set; if there is no such scope it's an > error. This will break a small number of program but probably not very > many; still, it'll require a future statement or waiting until Python > 3.0. The downside is that "global" is not a very intuitive word for > this new meaning. I disagree with your last statement -- I think "global" _is_ a very intuitive word for this. As I understand it, in programming "global" has two meanings, closely intertwined. One is "universal, same throughout the system". For instance, "The singleton pattern is used to create a single, global instance of a type." The second meaning is the term "global variable". This term developed (I believe) in languages that had only two scopes: local-to-current-function and global-to-entire-program. But the term "global variable" refers to any variable whose assignment is a "side effect", regardless of whether that variable is global-to-entire-program, global-to-module, or even global-to-enclosing-function. I have even heard the term "global variable" (mis)used to refer to any kind of side effect. Anyhow, in Python only builtins is _really_ global -- even today's global keyword only refers to module scope. So I believe that it would be a very reasonable interpretation of "global" to mean "not local", and implement as "search enclosing scopes in order to find the binding". -- Michael Chermside ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Import semantics
Skip> Does that make Samele the DBPV (Dictator benevolo per vita)? ;-) Alex> In Italian that would be DBAV (Dittatore benevolo a vita)...;-) Damn Google Translator. File a bug report for me please Alex (or Guido or Jeremy or Neal or ...). ;-) Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
On 7/5/06, Michael Chermside <[EMAIL PROTECTED]> wrote: > Guido writes: >[discussion of how to fix the can't-bind-outer-scope-vars wart] > > I think we have to continue to search for a solution that extends the > > idea of global declarations. > > > > I've proposed extending its meaning to refer to the nearest outer > > scope where the variable is set; if there is no such scope it's an > > error. This will break a small number of program but probably not very > > many; still, it'll require a future statement or waiting until Python > > 3.0. The downside is that "global" is not a very intuitive word for > > this new meaning. > > I disagree with your last statement -- I think "global" _is_ a very > intuitive word for this. As I understand it, in programming "global" > has two meanings, closely intertwined. One is "universal, same > throughout the system". For instance, "The singleton pattern is used > to create a single, global instance of a type." The second meaning is > the term "global variable". This term developed (I believe) in > languages that had only two scopes: local-to-current-function and > global-to-entire-program. But the term "global variable" refers to > any variable whose assignment is a "side effect", regardless of > whether that variable is global-to-entire-program, global-to-module, > or even global-to-enclosing-function. I have even heard the term > "global variable" (mis)used to refer to any kind of side effect. > > Anyhow, in Python only builtins is _really_ global -- even today's > global keyword only refers to module scope. So I believe that it > would be a very reasonable interpretation of "global" to mean > "not local", and implement as "search enclosing scopes in order > to find the binding". I really wish I could agree with you, because that would make the choice so much easier. However I still don't believe "global" has the stretchiness in its meaning that you claim it has. Have you ever heard a Python programmer talking about closures use the word "global variable"? Are there any other native speakers who side with Michael? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Import semantics
On 7/5/06, Frank Wierzbicki <[EMAIL PROTECTED]> wrote: > On 7/5/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > > > > Frank> That said, I still regard Samuele Pedroni as the ultimate > > Frank> authority on Jython and give him pretty much full veto power. He > > Frank> fortunately continues to watch the checkins and prods me when I > > Frank> go in the wrong direction. > > > > Does that make Samele the DBPV (Dictator benevolo per vita)? ;-) > > > > Skip > > > I wonder if Samuele even wants that role? > > Anyway, I believe Samuele has more experience with Jython as it is > currently implemented than anyone else, so I must take anything he > says about Jython very seriously. However, when it comes to pure > *Python* matters (no C and no J) which includes half of Jython's > import semantics, I think it is still Guido's opinion that matters > most. Without that, there is too much chaos. Actually, I started this discussion because Samuele told me that there was a new Jython maintainer, and I jumped out of my chair of joy. Samuele is happy to give advice and otherwise help out when asked, but he also seems happy to delegate everything to you, which sounds like a great endorsement. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Phillip Eby writes: > I don't see a problem with requiring '.x' to be used for both > reading and writing of outer-scope names; it just shouldn't be > required for an outer-scope name that you don't rebind in the > current scope. > > def counter(num): > def inc(): > .num += 1 > return .num > return inc I am reminded of Tim Peter's declaration in response to a similar proposal some time ago: Syntax should not look like grit on my monitor. (Sorry, no reference... but I swear it's word-for-word accurate because the quote burned itself into my memory.) -- Michael Chermside ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Phillip Eby writes: > The big problem that comes to mind with that idea is that it makes > it impossible to have argument names that are the same as attribute > names, unless the 'whee'/'.whee' prohibition were relaxed. :( But > it's an intriguing thought, nonetheless. My three-year-old has been working on that 'whee'/'.whee' prohibition, but he hasn't mastered it yet. Gotta-go-wash-another-load-of-underpants -lly yours, Michael Chermside ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
At 07:27 PM 7/5/2006 +0200, Guido van Rossum wrote: >However I still don't believe "global" has the stretchiness in its >meaning that you claim it has. Have you ever heard a Python programmer >talking about closures use the word "global variable"? > >Are there any other native speakers who side with Michael? I don't, and am -1 on using "global" to mean "outer". Of course, I also don't really care for using "global" to refer to a module level variable, or it being the only non-executable statement in Python, but these are very minor things and relatively easily explained to the programmers I've worked with. Using "global" to mean "outer", on the other hand, would surely lead to much wailing and grinding of teeth in response to any attempt at rational explanation. :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Time-out in URL Open
Alex Martelli <[EMAIL PROTECTED]> wrote: > What about doing it with a per-thread-timeout in TLS (overriding the > global one if a thread does have it set in its TLS)? Not as clean, > but perhaps far easier to implement than patching dozens of > modules/functions/classes to provide timeout= options everywhere... Yes please! I wrote a sketch of a module which did this on c.l.py recently http://groups.google.com/group/comp.lang.python/browse_thread/thread/d897c00b67cadca5/fd2ceb4e014de7ce?lnk=st&q=TimeoutError&rnum=2&hl=en#fd2ceb4e014de7ce It would be much better if it had help from the core though. -- Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] doc for new restricted execution design for Python
On 7/4/06, Ka-Ping Yee <[EMAIL PROTECTED]> wrote:
Hi Brett,Here are some comments on the description of the restricted executionmodel that you posted.> When referring to the state of an interpreter, it is either "trusted" or> "untrusted". A trusted interpreter has no restrictions imposed upon any
> resource. An untrusted interpreter has at least one, possibly more, resource> with a restriction placed upon it.In response to Guido's comment about confusing the words "trusted" and
"untrusted", how about "empowered" and "restricted"?Maybe. I am really starting to lean towards trusted and sandboxed.
> When the Interpreter Is Embedded> >> Single Untrusted Interpreter> >> This use case is when an application embeds the interpreter and never has more
> than one interpreter running.>> The main security issue to watch out for is not having default abilities be> provided to the interpreter by accident.I'd rather rephrase this in the opposite direction. The onus shouldn't
be on the application to hunt down each possible dangerous authority anddeactivate them all one by one. The main security issue is to let theapplication choose which abilities it wants the restricted interpreter
to have, and then ensure that the restricted interpreter gets only thoseabilities.Right. I am thinking more of an implementation screw up that somehow provides access to an object that has escalated rights.
> Multiple Untrusted Interpreters> ---
>> When multiple interpreters, all untrusted at varying levels, need to be> running within a single application. This is the key use case that this> proposed design is targetted for.>> On top of the security issues from a single untrusted interpreter,
> there is one additional worry. Resources cannot end up being leaked> into other interpreters where they are given escalated rights.What is your model here for communication between interpreters? If two
interpreters can communicate, any attempt to "prevent leakage" ofresources is meaningless. When you say "leaked into other interpreters"are you talking about a Python object leaking or something else at a
lower level?I am talking about Python objects.As for communication, I was planning on something included directly in globals or some custom object to handle that. I have not been focusing on that aspect so far.
Suppose for example that the application wants to embed two interpreters,P and Q, and that the application wants P to be able to write files but
Q to be restricted against writing files. When you say "leaked" above,that suggests to me that you want to prevent something like# code running in Pimport spamf = open('/home/doofus/.ssh/authorized_keys', 'a')
spam.f = f# code running in Qimport spamspam.f.write('blargh')The above example supposes that P and Q can communicate through ashared module, spam, where they can pass Python objects.
Right. But Python modules are separate per interpreter and only C extension modules are in any way shared between interpreters. But sharing an open file like that is bad and why C extension modules must be whitelisted to be used.
But notice that even if you prevent them from passing Python objectslike open files, any form of communication is sufficient to leak
resources:# code running in Pdef add_key(key):f = open('/home/doofus/.ssh/authorized_keys', 'a')f.write(key + '\n')f.close()import sockets =
socket.socket()s.bind(('', ))s.listen(1)ns, addr = s.accept()add_key(ns.recv(100))# code running in Qimport webbrowserwebbrowser.open('
http://localhost:/zebra')As long as P can listen for instructions from Q, it can give Qthe power to write to the filesystem.Right, which is why sockets and files are restricted and turned off by default. You have to give explicit permission to use either resource.
> Filesystem> ===>> The most obvious facet of a filesystem to protect is reading from it.
> One does not want what is stored in ``/etc/passwd`` to get out. And> one also does not want writing to the disk unless explicitly allowed> for basically the same reason; if someone can write ``/etc/passwd``
> then they can set the password for the root account.There's a big difference between modifying (or erasing) an existing fileand writing a new file (e.g. for temporary storage). If i give you alittle filesystem of your own to play in, and it starts out empty, you
can put whatever you want in it without violating my secrecy or theintegrity of my files.I think you should be talking about this in terms of specificallywhat abilities you want to be able to allow, based on examples of
real-life applications.Fair enough. But since you have the ability to only list files specifically, you can give temporary file access by giving access to such a non-existent file for writing. If you don't like an existing file then you
[Python-Dev] branch for sandox work created: bcannon-sandboxing
I have created a branch in svn off of HEAD for holding the sandboxing work. It's called bcannon-sandboxing and I have checked in the design doc in the root as sandboxing_design_doc.txt . You can keep an eye on the checkout message for incremental changes, but I will email the list once I have gone through at least one thorough revision. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] doc for new restricted execution design for Python
Ka-Ping Yee writes: > If you mean getting from a trusted interpreter to an untrusted > interpreter -- then how is a resource going to travel between > interpreters? Brett Cannon responds: > Beats me, but I am always scared of Armin and Samuele. =) Okay, those two scare me also, but I would still rather not spread FUD. Your proposal contains lots of details about how to address the danger that Python objects can cross from one interpreter to another. Could we instead attack that straight-on and try to find a convincing proof that objects cannot possibly cross the interpreter barrier? If so, it would simplify a bit of your proposal, and make me feel a little less worried. -- Michael Chermside ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python]
In response to Ka-Ping's comments on the subject of "Resource Hiding" vs "Resource Crippling", Brett says: > It seems that your criticisms are aimed at resource crippling > being a "plug holes as needed but if you foul up you are screwed" > with resource hiding being more "fix the fundamental issues and > just don't present access to resources you don't want to give > access to (or wrap accordingly)". And in general I agree with > this assessment. But I also realize that Python was not designed > for security in mind and there seems to be new ways to get access > to 'file'. If I felt confident that I could find and hide 'file' > as needed, I would go that route immediately. But I don't think I > can (and Armin has said this as well). I agree completely. "Resource Hiding" (specifically, Capabilities) has the cleanest concept, yet there are valid reasons to worry about implementing it in Python. However, I would like to point out one other advantage that capabilities has over "Resource Crippling". Resource Crippling implements the restrictions as changes in the underlying C-level objects capable of performing dangerous operations. That means that the restrictions are implemented by the interpreter. The advantage is obvious: we can trust the interpreter. But the disadvantages are (1) it's slow to fix (requires a bugfix release followed by everyone in the world upgrading), and (2) it cannot be extended by the user. With resource crippling, you will need to decide just what kind of restrictions the file type will implement. I believe you are planning to restrict to a list of known filenames and known directories for reading and for writing. (Actually, you said mode string, but I presume that you won't maintain separate lists for 'r' and 'rb' modes.) Then there was discussion of whether the directories ought to be recursive, whether the total number of files opened ought to be restricted, whether the total size written should be restricted, and even whether the size should be measured in bytes or blocks. Such conversations could go on for a long time, and in the end you must make some compromises. If you were using capabilities, you would need to ensure that restricted interpreters could only get the file object that they were given. But then _all_ of these fancy versions of the restrictions would be immediately supported: it would be up to the users to create secure wrappers implementing the specific restrictions desired. I really like this feature of capabilities: that they can be extended (well, restricted) by the user, not just by the language implementer. That's a powerful feature, and I don't want to give it up. But on the other hand, I don't quite see how to maintain it -- here are my best ideas, perhaps they will help. Python already has one essential ingredient for capabilities: unforgable references. But it fails in two other ways: having secure data, and having no auxiliary means of accessing objects. Python's powerful introspection and visible implementation (eg: __dict__) make it impossible to encapsulate data in an object in a way that prevents malicious users from accessing it. But that is actually surprisingly easy to fix. Just create a new type (maybe a new metaclass), implemented in C, which contains private data and a means to control access to it. You would provide a dict which would be stored privately without access from Python, and then provide methods and attributes along with a Python function for evaluating access to each. The type would ensure that the access test was evaluated with access to the private dict before any method or attribute was accessed. Such a construct is simple enough that I believe we could implement it and be reasonably confident that it was reliably secure. (I have played with this in the past and been reasonably pleased with the results.) Adding restrictions would then incur some performance penalties, but that seems unproblematic. That leaves the other problem: auxiliary means of accessing objects. There are things like gc.get_objects(). In the special case of file, which is a type that's also dangerous, there are tricks like "object().__class__.__subclasses__()". I would love to believe that we could plug all of these holes, but experience (rexec) proves otherwise. For something like sockets, I am fairly sure that there's a clear bottleneck (the low-level socket module), but still numerous existing libraries that use this low-level module without being handed a capability. So this is where my alternative plan starts to fall apart. Your (Brett's) plan to use resource crippling for these kinds of restrictions involves putting C code around all direct access to files, sockets, or whatever resource is being accessed. Perhaps instead of your C code doing the security checks directly, it could make sure that the objects returned were contained within the correct secure wrappers. That's OK so far as it goes, but the C checks are per interpreter instance,
Re: [Python-Dev] doc for new restricted execution design for Python
On 7/5/06, Michael Chermside <[EMAIL PROTECTED]> wrote: Ka-Ping Yee writes:> If you mean getting from a trusted interpreter to an untrusted> interpreter -- then how is a resource going to travel between> interpreters?Brett Cannon responds:> Beats me, but I am always scared of Armin and Samuele. =) Okay, those two scare me also, but I would still rather notspread FUD.I don't consider it FUD. Armin in an email said that he thought it was a losing battle to try to hide 'file' from an interpreter. That is what I am worried about, period. Everythign else can be protected through resource hiding. Your proposal contains lots of details about how toaddress the danger that Python objects can cross from one interpreter to another. Could we instead attack that straight-onand try to find a convincing proof that objects cannot possiblycross the interpreter barrier? If so, it would simplify a bitof your proposal, and make me feel a little less worried. As I said to Ping, if people *really* think this is doable and are willing to help out with this, then fine, I am willing to give this a shot. But I know I don't personally know enough about every random corner of the code base like Armin and Samuele know in order to feel comfortable in claiming I can pull this off by myself. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python]
On 7/5/06, Michael Chermside <[EMAIL PROTECTED]> wrote: In response to Ka-Ping's comments on the subject of "Resource Hiding"vs "Resource Crippling", Brett says:> It seems that your criticisms are aimed at resource crippling> being a "plug holes as needed but if you foul up you are screwed" > with resource hiding being more "fix the fundamental issues and> just don't present access to resources you don't want to give> access to (or wrap accordingly)". And in general I agree with > this assessment. But I also realize that Python was not designed> for security in mind and there seems to be new ways to get access> to 'file'. If I felt confident that I could find and hide 'file' > as needed, I would go that route immediately. But I don't think I> can (and Armin has said this as well).I agree completely. "Resource Hiding" (specifically, Capabilities)has the cleanest concept, yet there are valid reasons to worry about implementing it in Python. However, I would like to point outone other advantage that capabilities has over "Resource Crippling".Resource Crippling implements the restrictions as changes in the underlying C-level objects capable of performing dangerous operations.That means that the restrictions are implemented by the interpreter.The advantage is obvious: we can trust the interpreter. But thedisadvantages are (1) it's slow to fix (requires a bugfix release followed by everyone in the world upgrading), and (2) it cannot beextended by the user.With resource crippling, you will need to decide just what kind ofrestrictions the file type will implement. I believe you are planning to restrict to a list of known filenames and knowndirectories for reading and for writing. (Actually, you said modestring, but I presume that you won't maintain separate lists for'r' and 'rb' modes.) Then there was discussion of whether the directories ought to be recursive, whether the total number offiles opened ought to be restricted, whether the total size writtenshould be restricted, and even whether the size should be measuredin bytes or blocks. Such conversations could go on for a long time, and in the end you must make some compromises.If you were using capabilities, you would need to ensure thatrestricted interpreters could only get the file object that theywere given. But then _all_ of these fancy versions of the restrictions would be immediately supported: it would be up to theusers to create secure wrappers implementing the specificrestrictions desired.I agree. I would prefer this way of doing it. But as I have said, making sure that 'file' does not get out into the wild is tough. I really like this feature of capabilities: that they can beextended (well, restricted) by the user, not just by the language implementer. That's a powerful feature, and I don't want to giveit up. But on the other hand, I don't quite see how to maintainit -- here are my best ideas, perhaps they will help.Python already has one essential ingredient for capabilities: unforgable references. But it fails in two other ways: havingsecure data, and having no auxiliary means of accessingobjects.Right. Private attributes only exist at the C level. Python's powerful introspection and visible implementation(eg: __dict__) make it impossible to encapsulate data in an object in a way that prevents malicious users from accessingit. But that is actually surprisingly easy to fix. Just createa new type (maybe a new metaclass), implemented in C, whichcontains private data and a means to control access to it. You would provide a dict which would be stored privately withoutaccess from Python, and then provide methods and attributesalong with a Python function for evaluating access to each. Thetype would ensure that the access test was evaluated with access to the private dict before any method or attribute wasaccessed. Such a construct is simple enough that I believe wecould implement it and be reasonably confident that it wasreliably secure. (I have played with this in the past and been reasonably pleased with the results.) Adding restrictions wouldthen incur some performance penalties, but that seemsunproblematic.Right, but as you mention below this still does not protect the C level objects such as 'file'. If you can prevent references to 'file' from getting out, you can change open() to return wrapped instances of 'file' with the desired security measures in place. That leaves the other problem: auxiliary means of accessingobjects. There are things like gc.get_objects(). In the specialcase of file, which is a type that's also dangerous, there aretricks like "object().__class__.__subclasses__()". I would loveto believe that we could plug all of these holes, but experience (rexec) proves otherwise. For something like sockets, I amfairly sure that there's a clear bottleneck (the low-levelsocket module), but still numerous existing libraries that usethis low-level module without being handed a capability. Right, but for modules that cheat, you just d
[Python-Dev] what can we do to hide the 'file' type?
To make sure I don't unfairly block out capabilities as a complete security model instead of just crippling 'file's constructor (I do like capabilities and think it is a good model, really!), let's discuss how one can get to the 'file' type without importing any extension modules (that can be protected at the import level so I am ignoring the 'gc' module trick and such). First, it's in __builtin__. That reference can just be left out of the dict at the PyInterpreterState stuct's dict for built-ins. But we all know it isn't that simple.Second, there is __subclasses__(). That method could just not be allowed to be in the 'type' class at the Python level (hiding it, crippling it, whatever), but would that break much code? I don't know, but I doubt it. Third, for any wrappers returned by open(), it cannot be a subclass because chaining __class__ attribute, mro() (or any of the other methods provided on 'object' or 'type'), or type() will get you to the original 'file' type. The actual 'file' reference will need to be stored at the C struct level for the wrapper and not accessed except by the wrapper directly which would be implemented in C. Can anyone think of any other way to gain access to 'file' without importing a module? At that point one would need to be *very* careful about what an extension module exported to the world, but I can live with that (as that is already part of the plan). Please seriously try to think of ways to get to 'file' everybody. If we really cannot come up with anything beyond these three ways, then I am totally willing to go with a much more complete capabilities system for security in Python and really minimize any crippling. I just need to be convinced that we won't be plugging holes in how to hide 'file' rather than plugging holes from crippling 'file' (which, at this point, I am not convinced of). And if Armin and/or Samuele sign off that what we find is most likely (with "most likely" equalling 99% chance) all there is, then bonus points and I will *really* be convinced. =) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Marek "Baczek" Baczyński wrote: > I suggest <- as an assignment operator instead of := - it's used in > OCaml and it looks *very* different, yet still makes sense. But assigning to an outer scope isn't *very* different, it's only slightly different. -- "And now for something slightly different..." Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Simon Percivall wrote: > def foo(): > def bar(): > foo.x = 3 That already had a different meaning - it assigns to an attribute of the function object created by executing def foo(). -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
[EMAIL PROTECTED] wrote: > By naming the function in which the binding is to occur you avoid problems > of someone coming along and adding or deleting functions between the > assignment (in baz) and the target of the assignment (x in foo) but then > forgetting to increment or decrement the counters that refer to a fixed > number of levels above the current function. But it doesn't do anything for the (I expect much more common) case of factoring out something in a function body and making it a nested function -- you'd still have to change the form of all the references to the name in that case. Better not to have a scheme that uses counters or scope names at all, I think. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python]
Michael Chermside wrote: > That leaves the other problem: auxiliary means of accessing > objects. There are things like gc.get_objects(). In the special > case of file, which is a type that's also dangerous, there are > tricks like "object().__class__.__subclasses__()". My approach to that would be to not provide access to these kinds of things via attributes, but via builtin functions. E.g there wouldn't be a __subclasses__ attribute, but a subclasses() function. Then that capability can be denied by not providing that function. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] doc for new restricted execution design for Python
Brett Cannon wrote: > Armin in an email said that he thought it was > a losing battle to try to hide 'file' from an interpreter. And I would change file() so that it didn't open files. Then it would be harmless for code to have access to the file class. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python]
On 7/5/06, Greg Ewing <[EMAIL PROTECTED]> wrote: Michael Chermside wrote:> That leaves the other problem: auxiliary means of accessing> objects. There are things like gc.get_objects(). In the special> case of file, which is a type that's also dangerous, there are > tricks like "object().__class__.__subclasses__()".My approach to that would be to not provide access tothese kinds of things via attributes, but via builtinfunctions. E.g there wouldn't be a __subclasses__ attribute, but a subclasses() function. Then thatcapability can be denied by not providing thatfunction.__subclasses__ is a function. And yes, if we go this route, that is what would happen most likely. The trick is figuring out any and all ways one can get to 'file' from a standard interpreter prompt. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] ImportWarning decision
So there's 3 choices here: a) revert the importwarning entirely b) make it suppressed by default c) more complicated code in import.c to only emit the warning if the import fails. After a bit of a chat with Neal, I think the best combination of prudence and functionality is (b). (a) also works for me. (c) is a bit too scary, post-beta. import.c is a bad place where programmers go to die - I'd rather not mess about with it at this stage of the release cycle. Unless I hear screams in the near future, (b) is what's going to be done for beta2, and therefore 2.5 final. We can re-evaluate the situation for 2.6 - maybe a more complex solution like (c) can be done for that. This means Google can just turn it on in sitecustomize.py and Guido can avoid the hordes of peasants with pitchforks and burning torches. Anthony -- Anthony Baxter <[EMAIL PROTECTED]> It's never too late to have a happy childhood. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposed beta 2 changes (Q for Anthony/Neal)
On Tuesday 04 July 2006 22:32, Nick Coghlan wrote: > 1. Finishing the __module_name__ workaround to allow relative > imports from the main module when using -m. > >I'd really like to finish this, because having PEP 328 and 338 > not playing well together is a wart that's quite visible to end > users. I'd rather not have people's first encounter with the > features provided by either PEP involve discovering that they're > "broken". > >The patch to fix this also takes care of adding a couple of > paragraphs to the tutorial about explicit relative imports (which > aren't currently written up in the main documentation). The patch > is attached directly to the beta 1 bug report about the problem > [1]. > >(Guido gave a +1 to the concept, but explicitly deferred to > Anthony and Neal as to whether or not the fix should go in for beta > 2) I have some nervousness about this. Are there cases in the stdlib where this is an issue, today? Are there any cases which can't be addressed by using absolute imports? For 2.5, wouldn't it be better to simply say "use absolute imports in this case"? My nervousness is that I don't want a late change to introduce a language misfeature that we'll all regret later. > 2. Adding an 'ignore' filter for ImportWarning at the end of > warnings.py Already covered this one in another email... Yes, this seems the best approach for 2.5. Anthony -- Anthony Baxter <[EMAIL PROTECTED]> It's never too late to have a happy childhood. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.5 and beyond
On 7/4/06, Thomas Heller <[EMAIL PROTECTED]> wrote: > > I would like to ask about the possibility to add some improvements to ctypes > in Python 2.5, although the feature freeze is now in effect. Hopefully former > third-party libraries can have the freeze relaxed somewhat;-). Ok, former third-party libraries get a 1e-308 reprieve. :-) I think the general answer is to plan some way that you could make an external release to support the other communities like numpy. I don't know the details, but I believe Barry does this with email. This would allow Python to be more stable, but allow additional features for the communities that are interested in installing an additional ctypes release. > I intend to do these changes, the first is a small and trivial one, but allows > a lot of flexibility: > > - Remove the restriction that the argtypes attribute of foreign functions must > be ctypes types. Instead they are only required to implement a .from_param > class method. This patch is ok. I will add comments to the patch on SF. > The second one is more involved, and not yet complete. I can post the patch > or a link to it for review when it is implemented completely: > > - Implement the __array_struct__ attribute as describes by the numpy pep at > http://numpy.scipy.org/array_interface.html. This is too much to add to a beta. Perhaps you could work on a branch to add these features and integrate the branch back in after 2.5 has been released. You could make an external release from the branch. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ImportWarning decision
At 01:24 PM 7/6/2006 +1000, Anthony Baxter wrote: >This means Google can just turn it on in sitecustomize.py and Guido >can avoid the hordes of peasants with pitchforks and burning torches. Is that really true? It seems to me that Guido indicated a sitecustomize-solution wasn't possible, in which case suppression is no better than elimination. By the way, while I haven't formally reviewed any of the patches to delay the warning, the one I saw that went by on Python-Dev looked quite straightforward and minimal. Until I saw it, I also believed that trying to implement something like that would be difficult and delicate. But the patch I saw looked very simple and direct. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] import screwiness
In import.c starting around line 1210 (I removed a bunch of code that
doesn't matter for the problem):
if (PyUnicode_Check(v)) {
copy = PyUnicode_Encode(PyUnicode_AS_UNICODE(v),
PyUnicode_GET_SIZE(v),
Py_FileSystemDefaultEncoding, NULL);
v = copy;
}
len = PyString_GET_SIZE(v);
if (len + 2 + namelen + MAXSUFFIXSIZE >= buflen) {
Py_XDECREF(copy);
continue; /* Too long */
}
strcpy(buf, PyString_AS_STRING(v));
***
So if v is originally unicode, then copy is unicode from the second
line, right? Then we assign v to copy, so v is still unicode. Then
later on we do PyString_GET_SIZE and PyString_AS_STRING. That doesn't
work, does it? What am I missing?
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] import screwiness
[Neal Norwitz]
> In import.c starting around line 1210 (I removed a bunch of code that
> doesn't matter for the problem):
>
> if (PyUnicode_Check(v)) {
> copy = PyUnicode_Encode(PyUnicode_AS_UNICODE(v),
> PyUnicode_GET_SIZE(v),
> Py_FileSystemDefaultEncoding, NULL);
> v = copy;
> }
> len = PyString_GET_SIZE(v);
> if (len + 2 + namelen + MAXSUFFIXSIZE >= buflen) {
> Py_XDECREF(copy);
> continue; /* Too long */
> }
> strcpy(buf, PyString_AS_STRING(v));
>
> ***
> So if v is originally unicode, then copy is unicode from the second
> line, right?
No. An encoded unicode string is of type str, and PyUnicode_Encode()
returns an encoded string. Like so:
>>> u"\u1122".encode('utf-8')
'\xe1\x84\xa2'
>>> type(_)
> Then we assign v to copy, so v is still unicode.
Almost ;-)
> Then later on we do PyString_GET_SIZE and PyString_AS_STRING. That doesn't
> work, does it? What am I missing?
The conceptual type of the object returned by PyUnicode_Encode().
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] import screwiness
On 7/5/06, Tim Peters <[EMAIL PROTECTED]> wrote: > > > Then later on we do PyString_GET_SIZE and PyString_AS_STRING. That doesn't > > work, does it? What am I missing? > > The conceptual type of the object returned by PyUnicode_Encode(). Phew, I sure am glad I was missing that. :-) I saw as the first line in PyUnicode_Encode that it was creating a unicode string. I then went to look at PyString_Encode and saw the same. There I realized that the string created initially wasn't the string returned (it's the object from _AsEncodedString). Which naturally tells me what you just did. Thanks. Now can you fix test_subprocess hanging? :-) Please, I'll even buy you lunch and a coke at the sprints at Google. You are coming, right? n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
On Wed, 5 Jul 2006, Guido van Rossum wrote: > On 7/5/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > > Using the classic nonsense example: > > > > def counter(num): > > def inc(): > > .num += 1 > > return .num > > return inc > > > Would this also use ..num to refer to num in an outer scope two > levels removed? I don't think there's any need for that. I see '.num' as just another way of saying "num, but don't make a new binding". I agree with Guido that the best proposals so far are converging on the idea that it's more Pythonic to say "don't make a new binding" when a variable is used, than to declare "this is the scope for this binding" ahead of time. Of those there are two kinds: (a) State once (anywhere in a scope, but preferably at the beginning) that a variable is non-local. This is like the "global" keyword works now, and this category includes: - Change the meaning of 'global'. - Add a new keyword 'outer' or 'nonlocal', etc. (b) Indicate, when mentioning a variable, that the variable is non-local. This category includes: - Say 'global.x' or 'outer.x' instead of 'x'. - Say '.x' instead of 'x'. My favourite so far is to use a new keyword -- i think changing the meaning of 'global' would be misleading. '.x' is probably my next favourite, though i share Guido's concern about allowing both 'x' and '.x' to refer to the same thing. I see that 'outer' is used as an identifier in hmac.py in the standard library and also as a variable in test_set*.py. On the other hand 'nonlocal' does not appear anywhere in the standard library. Thus, 'nonlocal' is the best option that i've seen so far; it's less likely to break anything and it says exactly what it means. I can't think of a more accurate keyword. -- ?!ng ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python]
Brett Cannon wrote: > On 7/5/06, Michael Chermside <[EMAIL PROTECTED]> wrote: >> If you were using capabilities, you would need to ensure that >> restricted interpreters could only get the file object that they >> were given. But then _all_ of these fancy versions of the >> restrictions would be immediately supported: it would be up to the >> users to create secure wrappers implementing the specific >> restrictions desired. > > I agree. I would prefer this way of doing it. But as I have said, making > sure that 'file' does not get out into the wild is tough. I seem to recall someone mentioned earlier in this discussion the notion of somehow throwing an exception when sandboxed code attempts to push a file reference onto the interpreter stack. I'm not an expert in these matters, so perhaps what I am going to say will make no sense, but here goes: What if there were two copies of the evaluator function. One copy would be a slightly slower 'checked' function, that would test all objects for a 'check' bit. Any attempt to evaluate a reference to an object with a check bit set would throw an exception. The other eval function would be the 'unchecked' version that would run at full speed, just like it does today. Transitioning from the checked to the unchecked state could only be done via C code. So the 'file' wrapper, for example, would switch over to the unchecked interpreter before calling the actual methods of 'file'. That C wrapper might also check the current permission state to see what operations were legal. Note that whenever a C function sets the interpreter state to 'unchecked', that fact is saved on the stack, so that when the function returns, the previous state is restored. The function for setting the interpreter state is something like PyCall_Unchecked( ... ), which restores the interpreter state back to where it was. Transitioning from unchecked to checked is trickier. Specifically, you don't want to ever run sandboxed code in the unchecked state - this is a problem for generators, callbacks, and so on. I can think of two approaches to handling this: First approach is to mark all sandboxed code with a bit indicating the code is untrusted. Any attempt to call or otherwise invoke a function that has this bit set would throw the interpreter into the 'checked' state. (Note that transitioning the other way is *not* automatic - i.e. calling trusted code does not automatically transition to unchecked state.) The approach is good because it means that if you have intermediary code between the wrapper and the sandboxed code, the interpreter still does the right thing - it sets the interpreter into checked state. One problem is how to restore the 'unchecked' state when a function call returns. Probably you would have to build this into the code that does the state transition. If marking the sandboxed code isn't feasible, then you'd have to have the wrapper objects wrap all of the callbacks with code that goes to checked state before calling the callbacks. This means finding all the possible holes - however, I suspect that there are far fewer such holes than trying to hide all possible 'file' methods. However, one advantage of doing this is that restoring the 'unchecked' state after the call returns is fairly straightforward. The advantage of the this whole approach is that once you set the 'check' bit on 'file', it doesn't matter whether 'file' gets loose or not - you can't do anything with it without throwing an exception. Moreover, it also doesn't matter what code path you go through to access it. Only C code that flips the interpreter into unchecked state can call methods on 'file' without blowing up. So essentially, what I propose is to define a simple security primitive - which essentially comes down to checking a single bit - and use that as a basis to create more complex and subtle security mechanisms. -- Talin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
+1 on nonlocal. I think that the := operator is also in case (b), but as I don't like it I'm find with not mentioning it. :-) Could someone write a PEP for this? Doesn't have to be very long but I'd like it to summarize the main options proposed and discuss them, like I did for the switch PEP. It's a p3yk PEP. (We really need to move this to the py3k list...) --Guido On 7/6/06, Ka-Ping Yee <[EMAIL PROTECTED]> wrote: > On Wed, 5 Jul 2006, Guido van Rossum wrote: > > On 7/5/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > > > Using the classic nonsense example: > > > > > > def counter(num): > > > def inc(): > > > .num += 1 > > > return .num > > > return inc > > > > > Would this also use ..num to refer to num in an outer scope two > > levels removed? > > I don't think there's any need for that. I see '.num' as just another > way of saying "num, but don't make a new binding". > > I agree with Guido that the best proposals so far are converging on > the idea that it's more Pythonic to say "don't make a new binding" > when a variable is used, than to declare "this is the scope for this > binding" ahead of time. > > Of those there are two kinds: > > (a) State once (anywhere in a scope, but preferably at the > beginning) that a variable is non-local. This is like > the "global" keyword works now, and this category includes: > > - Change the meaning of 'global'. > - Add a new keyword 'outer' or 'nonlocal', etc. > > (b) Indicate, when mentioning a variable, that the variable > is non-local. This category includes: > > - Say 'global.x' or 'outer.x' instead of 'x'. > - Say '.x' instead of 'x'. > > My favourite so far is to use a new keyword -- i think changing the > meaning of 'global' would be misleading. '.x' is probably my next > favourite, though i share Guido's concern about allowing both 'x' > and '.x' to refer to the same thing. > > I see that 'outer' is used as an identifier in hmac.py in the > standard library and also as a variable in test_set*.py. On the > other hand 'nonlocal' does not appear anywhere in the standard > library. Thus, 'nonlocal' is the best option that i've seen so far; > it's less likely to break anything and it says exactly what it means. > I can't think of a more accurate keyword. > > > -- ?!ng > -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] what can we do to hide the 'file' type?
Brett Cannon wrote: > Can anyone think of any other way to gain access to 'file' without > importing a module? In principle, it might be possible to find file in the func_defaults or func_globals of some function, which might be defined as orig_file = file def file(...): ... I couldn't find any such function in the standard library, though. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ImportWarning decision
On 7/6/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > At 01:24 PM 7/6/2006 +1000, Anthony Baxter wrote: > >This means Google can just turn it on in sitecustomize.py and Guido > >can avoid the hordes of peasants with pitchforks and burning torches. > > Is that really true? It seems to me that Guido indicated a > sitecustomize-solution wasn't possible, in which case suppression is no > better than elimination. Don't worry about it. It's irrelevant anyway since we're not upgrading to 2.5 any time soon. BTW The pitchforks pitch was widely misunderstood -- it was a dramatization to emphasize what I still consider a general problem; if I believed the problem was limited to Google I would have proposed an internal solution. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] doc for new restricted execution design for Python
On Wed, 5 Jul 2006, Brett Cannon wrote: > On 7/4/06, Ka-Ping Yee <[EMAIL PROTECTED]> wrote: > > In response to Guido's comment about confusing the words "trusted" and > > "untrusted", how about "empowered" and "restricted"? > > Maybe. I am really starting to lean towards trusted and sandboxed. It can be risky to use words of the form '*trust*' when talking about security because 'trust' can have different meanings in different contexts. (Examples: 'X is trusted' might mean 'X is something i feel safe running without restrictions' or 'X *is* in fact allowed to run without restrictions' or 'i need to run X without restrictions in order to accomplish my task' or 'X is something i am relying on for the security of my system'.) The most common context for 'trusted' that i seem to come across is in the phrase 'trusted computing base' (TCB), which refers to 'the thing that is supposed to be enforcing security restrictions' as opposed to 'something i'm willing to let run unrestricted'. So when i read 'trusted code' what comes to mind is the C implementation of the Python interpreter, and it may be a good idea to reserve that phrase for that purpose, if it's to be used at all. In any case, let's try to nail down clear names for the different pieces we're talking about here, and i gently advise avoiding '*trust*' words or using them with very limited meaning. -- ?!ng ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] import screwiness
[Neal] >>> Then later on we do PyString_GET_SIZE and PyString_AS_STRING. That doesn't >>> work, does it? What am I missing? [Tim] >> The conceptual type of the object returned by PyUnicode_Encode(). [Neal] > Phew, I sure am glad I was missing that. :-) > > I saw as the first line in PyUnicode_Encode that it was creating a > unicode string. I then went to look at PyString_Encode and saw the > same. There I realized that the string created initially wasn't the > string returned (it's the object from _AsEncodedString). Which > naturally tells me what you just did. Thanks. Good! You're welcome. > Now can you fix test_subprocess hanging? :-) Sure: break it on Windows and I'll fix it tonight :-) > Please, I'll even buy you lunch and a coke at the sprints at Google. You > are coming, right? Sorry, doesn't look like it. I'm living on investment income this year, and the markets haven't been kind so far -- I don't want to go more in the hole than is truly necessary. I even switched from Campbell's soup to the store generic brand last week ;-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Restricted execution: what's the threat model?
After reading the messages on this thread i'm starting to think
that it would be good to clarify what kinds of threats we are
trying to defend against, and specify what invariants we are
intending to preserve.
For example, here are a few things Brett mentioned:
> Right. I am thinking more of an implementation screw up that somehow
> provides access to an object that has escalated rights.
> But you are correct, I am only concerned with preventing a crash of a
> sandboxed interperter.
[on what is meant by "it getting out"]
> Out of a trusted interpreter and ending up in a sandboxed interpreter
> some how.
So here are a couple of questions for clarification (some with my
guesses as to their answers):
1. When we say "restricted/untrusted/ interpreter" we
don't really mean that the *interpreter* is untrusted, right?
We mean that the Python code that runs in that interpreter is
untrusted (i.e. to be prevented from doing harm), right?
2. I'm assuming that the implementation of the Python interpreter
is always trusted. As a starting point it seems to me we have
to draw the line somewhere -- around at least the C code that
implements the Python interpreter, and possibly more. What do
we take the Trusted Computing Base to include? The Python VM
implementation -- plus all the builtin objects and C modules?
Plus the whole standard library?
("trusted" = "behaves safely because it's our job to write it
correctly, not due to something else imposing
restrictions upon it";
"untrusted" = "we wish to be able to impose restrictions on it")
3. Is it part of the plan that we want to protect Python code from
other Python code? For example, should a Python program/function
X be able to say "i want to launch/call program/function Y with
*these* parameters and have it run under *these* limitations?"
This has a big impact on the model.
And here are some possible goals or invariants to consider. It
will be helpful to decide on some of these so that, when someone
points to what they think is a flaw in the security implementation,
we can say "yes, that is our responsibility" or "no, it isn't".
We want to be able to guarantee that...
A. The interpreter will not crash no matter what Python code
it is given to execute.
B. Python programs running in different interpreters embedded
in the same process cannot communicate with each other.
C. Python programs running in different interpreters embedded
in the same process cannot access each other's Python objects.
D. A given piece of Python code cannot access or communicate
with certain Python objects in the same interpreter.
E. A given piece of Python code can access only a limited set
of Python objects in the same interpreter.
I think in order to get truly useful restricted interpreters we
will end up wanting to make guarantees of all of these kinds.
There may be others i haven't thought of -- feel free to edit or
add others.
-- ?!ng
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
