Re: [Python-Dev] Possible optimization for LOAD_FAST ?
On 04/01/2011 01:02, Guido van Rossum wrote: On Mon, Jan 3, 2011 at 9:52 AM, David Malcolmdmalc...@redhat.com wrote: On Sun, 2011-01-02 at 19:18 -0800, Guido van Rossum wrote: On Sun, Jan 2, 2011 at 5:50 PM, Alex Gaynoralex.gay...@gmail.com wrote: No, it's singularly impossible to prove that any global load will be any given value at compile time. Any optimization based on this premise is wrong. True. My proposed way out of this conundrum has been to change the language semantics slightly so that global names which (a) coincide with a builtin, and (b) have no explicit assignment to them in the current module, would be fair game for such optimizations, with the understanding that the presence of e.g. len = len anywhere in the module (even in dead code!) would be sufficient to disable the optimization. But barring someone interested in implementing something based on this rule, the proposal has languished for many years. Is there a PEP for this? Not that I know of, otherwise I'd have mentioned it. :-) It would be useful if someone wrote it up, since the idea comes back in one form or another regularly. FWIW, this is reminiscent of Fortran's rules for intrinsics (its name for builtins), which have a similar optimization behavior (except there the potential overrides that the compiler doesn't need to take into account are load-time definitions). I've been attempting another way in: http://bugs.python.org/issue10399 using a new JUMP_IF_SPECIALIZABLE opcode. This compares what a value is against a compile-time prediction, branching to an optimized implementation if the guess was correct. I use this to implement function-call inlining within the generated bytecode. Yeah, that's what everybody proposes to keep the language semantics unchanged. But I claim that an easier solution is to say to hell with those semantics, let's change them to make the implementation simpler. That's from the Zen of Python: If the implementation is easy to explain, it may be a good idea. I guess few people can seriously propose to change Python's semantics, that's why *I* am proposing it. :-) Note that the semantics of locals (e.g. UnboundLocalError) were also changed specifically to allow a significant optimization -- again by me. I think someone else pointed this out, but replacing builtins externally to a module is actually common for testing. In particular replacing the open function, but also other builtins, is often done temporarily to replace it with a mock. It seems like this optimisation would break those tests. Michael Caveat-of-doom: That code's very much a work-in-progress at this stage, though: sometimes it doesn't segfault :) and the way that I track the predicted values is taking some other liberties with semantics (see that URL and the dmalcolm-ast-optimization-branch in SVN). (There's probably at least 2 PEPs in the above idea, though have yet to write my first PEP) If you want to write up a PEP for the semantics change I am proposing, everything you need is in this thread. -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3333: wsgi_string() function
On Tue, 04 Jan 2011 03:44:53 +0100
Victor Stinner victor.stin...@haypocalc.com wrote:
def wsgi_string(u):
# Convert an environment variable to a WSGI bytes-as-unicode
string
return u.encode(enc, esc).decode('iso-8859-1')
def run_with_cgi(application):
environ = {k: wsgi_string(v) for k,v in os.environ.items()}
environ['wsgi.input']= sys.stdin
environ['wsgi.errors'] = sys.stderr
environ['wsgi.version'] = (1, 0)
...
--
What is this horrible encoding bytes-as-unicode? os.environ is
supposed to be correctly decoded and contain valid unicode characters.
If WSGI uses another encoding than the locale encoding (which is a bad
idea), it should use os.environb and decodes keys and values using its
own encoding.
If you really want to store bytes in unicode, str is not the right type:
use the bytes type and use os.environb instead.
+1. We should minimize such reencoding dances, and avoid promoting them.
Regards
Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3333: wsgi_string() function
Le mardi 04 janvier 2011 à 13:20 +0100, Antoine Pitrou a écrit :
On Tue, 04 Jan 2011 03:44:53 +0100
Victor Stinner victor.stin...@haypocalc.com wrote:
def wsgi_string(u):
# Convert an environment variable to a WSGI bytes-as-unicode
string
return u.encode(enc, esc).decode('iso-8859-1')
def run_with_cgi(application):
environ = {k: wsgi_string(v) for k,v in os.environ.items()}
environ['wsgi.input']= sys.stdin
environ['wsgi.errors'] = sys.stderr
environ['wsgi.version'] = (1, 0)
...
--
What is this horrible encoding bytes-as-unicode? os.environ is
supposed to be correctly decoded and contain valid unicode characters.
If WSGI uses another encoding than the locale encoding (which is a bad
idea), it should use os.environb and decodes keys and values using its
own encoding.
If you really want to store bytes in unicode, str is not the right type:
use the bytes type and use os.environb instead.
+1. We should minimize such reencoding dances, and avoid promoting them.
The example from the PEP is specific to CGI and is a little bit special.
The reference implementation (wsgiref in py3k) only redecodes
(transcode) some variables:
---
_is_request = {
'SCRIPT_NAME', 'PATH_INFO', 'QUERY_STRING', 'REQUEST_METHOD',
'AUTH_TYPE',
'CONTENT_TYPE', 'CONTENT_LENGTH', 'HTTPS', 'REMOTE_USER',
'REMOTE_IDENT',
}.__contains__
def _needs_transcode(k):
return _is_request(k) or k.startswith('HTTP_') or
k.startswith('SSL_') \
or (k.startswith('REDIRECT_') and _needs_transcode(k[9:]))
---
My problem is that I don't understand how I can know if a variable was
converted to bytes-as-unicode or not. GrahamDumpleton told me on IRC,
that the framework is supposed to redecodes one more time some variables
(eg. PATH_INFO). But this is not explicit in the PEP and
_needs_transcode() is a private function.
Since the environ already contain different types (eg. wsgi.version is a
tuple, wsgi.multithread is a boolean, ...), why not keeping these
variables as raw bytes?
Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3333: wsgi_string() function
On Tue, 04 Jan 2011 14:33:37 +0100
Victor Stinner victor.stin...@haypocalc.com wrote:
Le mardi 04 janvier 2011 à 13:20 +0100, Antoine Pitrou a écrit :
On Tue, 04 Jan 2011 03:44:53 +0100
Victor Stinner victor.stin...@haypocalc.com wrote:
def wsgi_string(u):
# Convert an environment variable to a WSGI bytes-as-unicode
string
return u.encode(enc, esc).decode('iso-8859-1')
def run_with_cgi(application):
environ = {k: wsgi_string(v) for k,v in os.environ.items()}
environ['wsgi.input']= sys.stdin
environ['wsgi.errors'] = sys.stderr
environ['wsgi.version'] = (1, 0)
...
--
What is this horrible encoding bytes-as-unicode? os.environ is
supposed to be correctly decoded and contain valid unicode characters.
If WSGI uses another encoding than the locale encoding (which is a bad
idea), it should use os.environb and decodes keys and values using its
own encoding.
If you really want to store bytes in unicode, str is not the right type:
use the bytes type and use os.environb instead.
+1. We should minimize such reencoding dances, and avoid promoting them.
The example from the PEP is specific to CGI and is a little bit special.
Well, it would be better if it used os.environb anyway ;)
Regards
Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
On Dec 30, 2010, at 02:50 AM, R. David Murray wrote: You are welcome; thanks for the feedback. (I sometimes feel like I'm working in a bit of a vacuum, though Barry does chime in occasionally...but I do realize that people are busy; that's why I inherited this job in the first place, after all :) It's you're own fault for doing such a damn good job. :) -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
On Tue, Jan 4, 2011 at 8:49 PM, Michael Foord fuzzy...@voidspace.org.uk wrote: I think someone else pointed this out, but replacing builtins externally to a module is actually common for testing. In particular replacing the open function, but also other builtins, is often done temporarily to replace it with a mock. It seems like this optimisation would break those tests. print() and input() come to mind. However, so long as appropriate tools are provided to recompile a module with this optimisation disabled for certain builtins (with some builtins such as open(), print() and input() blacklisted by default) then that issue should be manageable. I've extracted the full list of 68 builtin functions from the table in the docs below. I've placed asterisks next to the ones I think we would *want* to be eligible for optimisation. Aside from the 3 mentioned above, we could fairly easily omit ones which are used primarily at the interactive prompt (i.e. dir(), help()), as well as __import__() (since that lookup is handled specially anyway). Cheers, Nick. __import__() abs() * all() * any() * ascii() * bin() * bool() * bytearray() * bytes() * callable() * chr() * classmethod() * compile() * complex() * delattr() * dict() * dir() divmod() * enumerate() * eval() * exec() * filter() * float() * format() * frozenset() * getattr() * globals() * hasattr() * hash() * help() hex() * id() * input() int() * isinstance() * issubclass() * iter() * len() * list() * locals() * map() * max() * memoryview() * min() * next() * object() * oct() * open() ord() * pow() * print() property() * range() * repr() * reversed() * round() * set() * setattr() * slice() * sorted() * staticmethod() * str() * sum() * super() * tuple() * type() * vars() * zip() * -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
On Tue, Jan 4, 2011 at 2:49 AM, Michael Foord fuzzy...@voidspace.org.uk wrote: I think someone else pointed this out, but replacing builtins externally to a module is actually common for testing. In particular replacing the open function, but also other builtins, is often done temporarily to replace it with a mock. It seems like this optimisation would break those tests. Hm, I already suggested to make an exception for open, (and one should be added for __import__) but if this is done for other builtins that is indeed a problem. Can you point to example code doing this? -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Checking input range in time.asctime and time.ctime
On Mon, Jan 3, 2011 at 7:47 PM, Guido van Rossum gu...@python.org wrote: Given the rule garbage in - garbage out, I'd do the most useful thing, which would be to produce a longer output string (and update the docs). I did not know that GIGO was a design rule, but after thinking about it some more, I agree. It is very unlikely that a Python program would care about precise length of the string produced by time.asctime() and these strings are not well suited for passing timestamps to other programs that may care. (Use of asctime() timestamps in internet protocols has been long deprecated and surely won't be in use in 10-th millennium :-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
On Wed, Jan 5, 2011 at 1:58 AM, Guido van Rossum gu...@python.org wrote: On Tue, Jan 4, 2011 at 2:49 AM, Michael Foord fuzzy...@voidspace.org.uk wrote: I think someone else pointed this out, but replacing builtins externally to a module is actually common for testing. In particular replacing the open function, but also other builtins, is often done temporarily to replace it with a mock. It seems like this optimisation would break those tests. Hm, I already suggested to make an exception for open, (and one should be added for __import__) but if this is done for other builtins that is indeed a problem. Can you point to example code doing this? I've seen it done to write tests for simple CLI behaviour by mocking input() and print() (replacing sys.stdin and sys.stdout instead is far more common, but replacing the functions works too). If compile() accepted a blacklist of builtins that it wasn't allowed to optimise, then that should deal with the core of the problem as far as testing goes. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
On Tue, Jan 4, 2011 at 10:20 AM, Nick Coghlan ncogh...@gmail.com wrote: On Wed, Jan 5, 2011 at 1:58 AM, Guido van Rossum gu...@python.org wrote: On Tue, Jan 4, 2011 at 2:49 AM, Michael Foord fuzzy...@voidspace.org.uk wrote: I think someone else pointed this out, but replacing builtins externally to a module is actually common for testing. In particular replacing the open function, but also other builtins, is often done temporarily to replace it with a mock. It seems like this optimisation would break those tests. Hm, I already suggested to make an exception for open, (and one should be added for __import__) but if this is done for other builtins that is indeed a problem. Can you point to example code doing this? I've seen it done to write tests for simple CLI behaviour by mocking input() and print() (replacing sys.stdin and sys.stdout instead is far more common, but replacing the functions works too). If compile() accepted a blacklist of builtins that it wasn't allowed to optimise, then that should deal with the core of the problem as far as testing goes. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia Ugh, I can't be the only one who finds these special cases to be a little nasty? Special cases aren't special enough to break the rules. Alex -- I disapprove of what you say, but I will defend to the death your right to say it. -- Evelyn Beatrice Hall (summarizing Voltaire) The people's good is the highest law. -- Cicero Code can always be simpler than you think, but never as simple as you want -- Me ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3333: wsgi_string() function
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 01/03/2011 09:44 PM, Victor Stinner wrote:
Hi,
In the PEP , I read:
--
import os, sys
enc, esc = sys.getfilesystemencoding(), 'surrogateescape'
def wsgi_string(u):
# Convert an environment variable to a WSGI bytes-as-unicode
string
return u.encode(enc, esc).decode('iso-8859-1')
def run_with_cgi(application):
environ = {k: wsgi_string(v) for k,v in os.environ.items()}
environ['wsgi.input']= sys.stdin
environ['wsgi.errors'] = sys.stderr
environ['wsgi.version'] = (1, 0)
...
--
What is this horrible encoding bytes-as-unicode? os.environ is
supposed to be correctly decoded and contain valid unicode characters.
If WSGI uses another encoding than the locale encoding (which is a bad
idea), it should use os.environb and decodes keys and values using its
own encoding.
If you really want to store bytes in unicode, str is not the right type:
use the bytes type and use os.environb instead.
I'm not clear on the semantics here, but I'm pretty sure you'll find
that the web-SIG does know them well. I have CC'ed that list (via gmane).
Note that Guido just recently wrote on that list that he considers that
PEP to be de facto accepted.
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk0jSTUACgkQ+gerLs4ltQ4cCQCgyc9QsRfzC2lrtnDO0v8TvK6W
rVwAnjvvwD47J1chgupqM3unt5c2jd6p
=8LEf
-END PGP SIGNATURE-
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3333: wsgi_string() function
At 03:44 AM 1/4/2011 +0100, Victor Stinner wrote:
Hi,
In the PEP , I read:
--
import os, sys
enc, esc = sys.getfilesystemencoding(), 'surrogateescape'
def wsgi_string(u):
# Convert an environment variable to a WSGI bytes-as-unicode
string
return u.encode(enc, esc).decode('iso-8859-1')
def run_with_cgi(application):
environ = {k: wsgi_string(v) for k,v in os.environ.items()}
environ['wsgi.input']= sys.stdin
environ['wsgi.errors'] = sys.stderr
environ['wsgi.version'] = (1, 0)
...
--
What is this horrible encoding bytes-as-unicode? os.environ is
supposed to be correctly decoded and contain valid unicode characters.
If WSGI uses another encoding than the locale encoding (which is a bad
idea), it should use os.environb and decodes keys and values using its
own encoding.
If you really want to store bytes in unicode, str is not the right type:
use the bytes type and use os.environb instead.
If you want to discuss this, the Web-SIG is the appropriate
place. Also, it was the appropriate place months ago, when the final
decision on the environ encoding was made. ;-)
IOW, the above change to the PEP is merely fixing the code example to
be correct for Python 3, where it previously was correct only for
Python 2. The PEP itself has already required this since the
previous revisions, and wsgiref in the stdlib is already compliant
with the above (although it uses a more sophisticated approach for
dealing with win32 compatibility).
The rationale for this choice is described in the PEP, and was also
discussed in the mailing list emails back when the work was being done.
IOW, this particular ship already sailed a long time ago. In fact,
for Jython this bytes-as-unicode approach has been the PEP
333-defined encoding for at least *six years*... so it's REALLY late
to complain about it now! ;-)
PEP is merely a mapping of PEP 333 to allow WSGI apps to be
ported from Python 2 to Python 3. There is work in progress on the
Web-SIG now on PEP 444, which will support only Python 2.6+, where
'b' literals and the 'bytes' alias are available. It is as yet
uncertain what environ encoding will be used, but at the moment I'm
not convinced that either pure bytes or pure unicode are acceptable
replacements for the PEP 333-compatible approach.
In any event, that is a discussion for the Web-SIG, not Python-Dev.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
On Tue, Jan 4, 2011 at 8:21 AM, Alex Gaynor alex.gay...@gmail.com wrote: Ugh, I can't be the only one who finds these special cases to be a little nasty? Special cases aren't special enough to break the rules. Alex +1, I don't think nailing down a few builtins is that helpful for optimizing Python. Anyone attempting to seriously optimize Python is going to need to use more general techniques that apply to non-builtins as well. In unladen swallow (I'm sure something similar is done in PyPy) we have some infrastructure for watching dictionaries for changes, and in particular we tend to watch the builtins and module dictionaries. Reid ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
On Jan 04, 2011, at 10:21 AM, Alex Gaynor wrote: Ugh, I can't be the only one who finds these special cases to be a little nasty? Special cases aren't special enough to break the rules. Yeah, I agree. Still it would be interesting to see what kind of performance improvement this would result in. That seems to be the only way to decide whether the cost is worth the benefit. Outside of testing, I do agree that most of the builtins could be pretty safely optimized (even open()). There needs to be a way to stop all optimizations for testing purposes. Perhaps a sys variable, plus command line option and/or environment variable? -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
On 04/01/2011 16:54, Barry Warsaw wrote: On Jan 04, 2011, at 10:21 AM, Alex Gaynor wrote: Ugh, I can't be the only one who finds these special cases to be a little nasty? Special cases aren't special enough to break the rules. Yeah, I agree. Still it would be interesting to see what kind of performance improvement this would result in. That seems to be the only way to decide whether the cost is worth the benefit. Outside of testing, I do agree that most of the builtins could be pretty safely optimized (even open()). There needs to be a way to stop all optimizations for testing purposes. Perhaps a sys variable, plus command line option and/or environment variable? Although testing in an environment deliberately different from production is a recipe for hard to diagnose bugs. Michael -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
Doesnt this all boil down to being able to monitor PyDict for changes to it's key-space? The keys are immutable anyway so the instances of PyDict could manage a opaque value (in fact, a counter) that changes every time a new value is written to any key. Once we get a reference out of the dict, we can can do very fast lookups by passing the key, the reference we know from the last lookup and our last state. The lookup returns a new reference and the new state. If the dict has not changed, the state doesnt change and the reference is simply taken from the passed value passed to the lookup. That way the code remains the same no matter if the dict has changed or not. 2011/1/4 Michael Foord fuzzy...@voidspace.org.uk: On 04/01/2011 16:54, Barry Warsaw wrote: On Jan 04, 2011, at 10:21 AM, Alex Gaynor wrote: Ugh, I can't be the only one who finds these special cases to be a little nasty? Special cases aren't special enough to break the rules. Yeah, I agree. Still it would be interesting to see what kind of performance improvement this would result in. That seems to be the only way to decide whether the cost is worth the benefit. Outside of testing, I do agree that most of the builtins could be pretty safely optimized (even open()). There needs to be a way to stop all optimizations for testing purposes. Perhaps a sys variable, plus command line option and/or environment variable? Although testing in an environment deliberately different from production is a recipe for hard to diagnose bugs. Michael -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lukas.lueg%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
Barry Warsaw ba...@python.org wrote: On Jan 04, 2011, at 10:21 AM, Alex Gaynor wrote: Ugh, I can't be the only one who finds these special cases to be a little nasty? Special cases aren't special enough to break the rules. Yeah, I agree. Still it would be interesting to see what kind of performance improvement this would result in. That seems to be the only way to decide whether the cost is worth the benefit. Yuck from me as well. I would guess that attribute lookups would be just as significant as global variable lookups (depending on coding style, of course). In contrast, the local variable semantic change provided a big speed increase for a minor language complexity cost. Neil ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3333: wsgi_string() function
On Tue, Jan 4, 2011 at 8:22 AM, Tres Seaver tsea...@palladion.com wrote: Note that Guido just recently wrote on that list that he considers that PEP to be de facto accepted. That was conditional on there not being any objections in the next 24 hours. There have been plenty, so I'm retracting that. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
Guido van Rossum wrote: On Tue, Jan 4, 2011 at 2:49 AM, Michael Foord fuzzy...@voidspace.org.uk wrote: I think someone else pointed this out, but replacing builtins externally to a module is actually common for testing. In particular replacing the open function, but also other builtins, is often done temporarily to replace it with a mock. It seems like this optimisation would break those tests. Hm, I already suggested to make an exception for open, (and one should be added for __import__) but if this is done for other builtins that is indeed a problem. Can you point to example code doing this? I've been known to monkey-patch builtins in the interactive interpreter and in test code. One example that comes to mind is that I had some over-complicated recursive while loop (!), and I wanted to work out the Big Oh behaviour so I knew exactly how horrible it was. Working it out from first principles was too hard, so I cheated: I knew each iteration called len() exactly once, so I monkey-patched len() to count how many times it was called. Problem solved. I also have a statistics package that has its own version of sum, and I rely on calls to sum() from within the package picking up my version rather than the builtin one. -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
On Tue, Jan 4, 2011 at 1:50 PM, Steven D'Aprano st...@pearwood.info wrote: I've been known to monkey-patch builtins in the interactive interpreter and in test code. One example that comes to mind is that I had some over-complicated recursive while loop (!), and I wanted to work out the Big Oh behaviour so I knew exactly how horrible it was. Working it out from first principles was too hard, so I cheated: I knew each iteration called len() exactly once, so I monkey-patched len() to count how many times it was called. Problem solved. But why couldn't you edit the source code? I also have a statistics package that has its own version of sum, and I rely on calls to sum() from within the package picking up my version rather than the builtin one. As long as you have a definition or import of sum at the top of (or really anywhere in) the module, that will still work. It's only if you were to do things like import builtins builtins.len = ... (whether inside your package or elsewhere) that things would stop working with the proposed optimization. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
On Tue, Jan 4, 2011 at 9:33 AM, Lukas Lueg lukas.l...@googlemail.comwrote: The keys are immutable anyway so the instances of PyDict could manage a opaque value (in fact, a counter) that changes every time a new value is written to any key. Once we get a reference out of the dict, we can can do very fast lookups by passing the key, the reference we know from the last lookup and our last state. The lookup returns a new reference and the new state. If the dict has not changed, the state doesnt change and the reference is simply taken from the passed value passed to the lookup. That way the code remains the same no matter if the dict has changed or not. I have had similar ideas in the past but have never found time to explore them. The same mechanism could also be used to speed up attribute access on objects. -- Daniel Stutzbach ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
On Tue, Jan 4, 2011 at 2:15 PM, Daniel Stutzbach stutzb...@google.com wrote: On Tue, Jan 4, 2011 at 9:33 AM, Lukas Lueg lukas.l...@googlemail.com wrote: The keys are immutable anyway so the instances of PyDict could manage a opaque value (in fact, a counter) that changes every time a new value is written to any key. Once we get a reference out of the dict, we can can do very fast lookups by passing the key, the reference we know from the last lookup and our last state. The lookup returns a new reference and the new state. If the dict has not changed, the state doesnt change and the reference is simply taken from the passed value passed to the lookup. That way the code remains the same no matter if the dict has changed or not. I have had similar ideas in the past but have never found time to explore them. The same mechanism could also be used to speed up attribute access on objects. Check out the various approaches in PEP 266, PEP 267, and PEP 280. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
Guido van Rossum wrote: On Tue, Jan 4, 2011 at 1:50 PM, Steven D'Aprano st...@pearwood.info wrote: I've been known to monkey-patch builtins in the interactive interpreter and in test code. One example that comes to mind is that I had some over-complicated recursive while loop (!), and I wanted to work out the Big Oh behaviour so I knew exactly how horrible it was. Working it out from first principles was too hard, so I cheated: I knew each iteration called len() exactly once, so I monkey-patched len() to count how many times it was called. Problem solved. But why couldn't you edit the source code? Because there was no source code -- I was experimenting in the interactive interpreter. I could have just re-created the function by using the readline history, but it was just easier to redefine len. Oh... it's just occurred to me that you were asking for use-cases for assigning to builtins.len directly, rather than just to len. No, I've never done that -- sorry for the noise. I also have a statistics package that has its own version of sum, and I rely on calls to sum() from within the package picking up my version rather than the builtin one. As long as you have a definition or import of sum at the top of (or really anywhere in) the module, that will still work. It's only if you were to do things like import builtins builtins.len = ... (whether inside your package or elsewhere) that things would stop working with the proposed optimization. Ha, well, that's the sort of thing that gives monkey-patching a bad name, surely? Is there a use-case for globally replacing builtins for all modules, everywhere? I suppose that's what you're asking. The only example I can think of might be the use of mocks for testing purposes, but even there I'd prefer to inject the mock into the module I was testing: mymodule.len = mylen But I haven't done much work with mocks, so I'm just guessing. -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
I very much like the fact that python has *very* little black magic revealed to the user. Strong -1 on optimizing picked builtins in a picked way. 2011/1/4 Steven D'Aprano st...@pearwood.info: Guido van Rossum wrote: On Tue, Jan 4, 2011 at 2:49 AM, Michael Foord fuzzy...@voidspace.org.uk wrote: I think someone else pointed this out, but replacing builtins externally to a module is actually common for testing. In particular replacing the open function, but also other builtins, is often done temporarily to replace it with a mock. It seems like this optimisation would break those tests. Hm, I already suggested to make an exception for open, (and one should be added for __import__) but if this is done for other builtins that is indeed a problem. Can you point to example code doing this? I've been known to monkey-patch builtins in the interactive interpreter and in test code. One example that comes to mind is that I had some over-complicated recursive while loop (!), and I wanted to work out the Big Oh behaviour so I knew exactly how horrible it was. Working it out from first principles was too hard, so I cheated: I knew each iteration called len() exactly once, so I monkey-patched len() to count how many times it was called. Problem solved. I also have a statistics package that has its own version of sum, and I rely on calls to sum() from within the package picking up my version rather than the builtin one. -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lukas.lueg%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
On Tue, Jan 4, 2011 at 3:13 PM, Steven D'Aprano st...@pearwood.info wrote: Guido van Rossum wrote: But why couldn't you edit the source code? Because there was no source code -- I was experimenting in the interactive interpreter. I could have just re-created the function by using the readline history, but it was just easier to redefine len. The interactive interpreter will always be excluded from this kind of optimization (well, in my proposal anyway). Oh... it's just occurred to me that you were asking for use-cases for assigning to builtins.len directly, rather than just to len. No, I've never done that -- sorry for the noise. There are two versions of the assign to global named 'len' idiom. One is benign: if the assignment occurs in the source code of the module (i.e., where the compiler can see it when it is compiling the module) the optimization will be disabled in that module. The second is not: if a module-global named 'len' is set in a module from outside that module, the compiler cannot see that assignment when it considers the optimization, and it may generate optimized code that will not take the global by that name into account (it would use an opcode that computes the length of an object directly). The third way to mess with the optimization is messing with builtins.len. This one is also outside what the compiler can see. [...] Ha, well, that's the sort of thing that gives monkey-patching a bad name, surely? Monkey-patching intentionally has a bad name -- there's always a code smell. (And it looks like one, too. :-) Is there a use-case for globally replacing builtins for all modules, everywhere? I suppose that's what you're asking. I think the folks referring to monkey-patching builtins in unittests were referring to this. But they might also be referring to the second option above. The only example I can think of might be the use of mocks for testing purposes, but even there I'd prefer to inject the mock into the module I was testing: mymodule.len = mylen But I haven't done much work with mocks, so I'm just guessing. Same here. But it would fail (i.e. not be picked up by the optimization) either way. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
On Tue, Jan 4, 2011 at 2:13 PM, Lukas Lueg lukas.l...@googlemail.com wrote: I very much like the fact that python has *very* little black magic revealed to the user. Strong -1 on optimizing picked builtins in a picked way. That's easy for you to say. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
On 04/01/2011 23:29, Guido van Rossum wrote: On Tue, Jan 4, 2011 at 3:13 PM, Steven D'Apranost...@pearwood.info wrote: Guido van Rossum wrote: But why couldn't you edit the source code? Because there was no source code -- I was experimenting in the interactive interpreter. I could have just re-created the function by using the readline history, but it was just easier to redefine len. The interactive interpreter will always be excluded from this kind of optimization (well, in my proposal anyway). Oh... it's just occurred to me that you were asking for use-cases for assigning to builtins.len directly, rather than just to len. No, I've never done that -- sorry for the noise. There are two versions of the assign to global named 'len' idiom. One is benign: if the assignment occurs in the source code of the module (i.e., where the compiler can see it when it is compiling the module) the optimization will be disabled in that module. The second is not: if a module-global named 'len' is set in a module from outside that module, the compiler cannot see that assignment when it considers the optimization, and it may generate optimized code that will not take the global by that name into account (it would use an opcode that computes the length of an object directly). The third way to mess with the optimization is messing with builtins.len. This one is also outside what the compiler can see. [...] Ha, well, that's the sort of thing that gives monkey-patching a bad name, surely? Monkey-patching intentionally has a bad name -- there's always a code smell. (And it looks like one, too. :-) Is there a use-case for globally replacing builtins for all modules, everywhere? I suppose that's what you're asking. I think the folks referring to monkey-patching builtins in unittests were referring to this. But they might also be referring to the second option above. I prefer monkey patching builtins (where I do such a thing) in the namespace where they are used. I know it is *common* to monkeypatch __builtins__.open (python 2) however. I don't recall monkey patching anything other than open and raw_input myself but I *bet* there are people doing it for reasons they see as legitimate in tests. :-) Michael The only example I can think of might be the use of mocks for testing purposes, but even there I'd prefer to inject the mock into the module I was testing: mymodule.len = mylen But I haven't done much work with mocks, so I'm just guessing. Same here. But it would fail (i.e. not be picked up by the optimization) either way. -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
On 04/01/2011 23:29, Guido van Rossum wrote: On Tue, Jan 4, 2011 at 3:13 PM, Steven D'Apranost...@pearwood.info wrote: Guido van Rossum wrote: But why couldn't you edit the source code? Because there was no source code -- I was experimenting in the interactive interpreter. I could have just re-created the function by using the readline history, but it was just easier to redefine len. The interactive interpreter will always be excluded from this kind of optimization (well, in my proposal anyway). Oh... it's just occurred to me that you were asking for use-cases for assigning to builtins.len directly, rather than just to len. No, I've never done that -- sorry for the noise. There are two versions of the assign to global named 'len' idiom. One is benign: if the assignment occurs in the source code of the module (i.e., where the compiler can see it when it is compiling the module) the optimization will be disabled in that module. The second is not: if a module-global named 'len' is set in a module from outside that module, the compiler cannot see that assignment when it considers the optimization, and it may generate optimized code that will not take the global by that name into account (it would use an opcode that computes the length of an object directly). The third way to mess with the optimization is messing with builtins.len. This one is also outside what the compiler can see. [...] Ha, well, that's the sort of thing that gives monkey-patching a bad name, surely? Monkey-patching intentionally has a bad name -- there's always a code smell. (And it looks like one, too. :-) Is there a use-case for globally replacing builtins for all modules, everywhere? I suppose that's what you're asking. I think the folks referring to monkey-patching builtins in unittests were referring to this. But they might also be referring to the second option above. The only examples I could find from a quick search (using the patch decorator from my mock module which is reasonably well used) patch __builtins__.open and raw_input. https://www.google.com/codesearch?hl=enlr=q=%22patch%28%27__builtin__.%22+lang%3Apython+case%3Ayes :-) Michael Foord The only example I can think of might be the use of mocks for testing purposes, but even there I'd prefer to inject the mock into the module I was testing: mymodule.len = mylen But I haven't done much work with mocks, so I'm just guessing. Same here. But it would fail (i.e. not be picked up by the optimization) either way. -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
On Tue, Jan 4, 2011 at 3:36 PM, Michael Foord fuzzy...@voidspace.org.uk wrote: The only examples I could find from a quick search (using the patch decorator from my mock module which is reasonably well used) patch __builtins__.open and raw_input. https://www.google.com/codesearch?hl=enlr=q=%22patch%28%27__builtin__.%22+lang%3Apython+case%3Ayes So, that significantly weakens the argument that this optimization will break unit tests, since I am happy to promise never to optimize these builtins, and any other builtins intended for I/O. Surely it will break *somebody's* code. That hasn't stopped us with other changes. The crux is whether it breaks significant amounts of code or code that would be really hard to write in another way. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Steroidal builtins (was: Possible optimization for LOAD_FAST ?)
Steven I've been known to monkey-patch builtins in the interactive Steven interpreter and in test code. Me too. I use a slightly beefed up dir() funcion which identifies modules within a package which haven't been imported yet. Handy for quick-n-dirty introspection. import email dir(email) ['Charset', 'Encoders', 'Errors', 'FeedParser', 'Generator', 'Header', 'Iterators', 'LazyImporter', 'MIMEAudio', 'MIMEBase', 'MIMEImage', 'MIMEMessage', 'MIMEMultipart', 'MIMENonMultipart', 'MIMEText', 'Message', 'Parser', 'Utils', '[_parseaddr]', '[base64mime]', '[charset]', '[encoders]', '[errors]', '[feedparser]', '[generator]', '[header]', '[iterators]', '[message]', '[parser]', '[quoprimime]', '[test/]', '[utils]', '_LOWERNAMES', '_MIMENAMES', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', '_name', 'base64MIME', 'email', 'importer', 'message_from_file', 'message_from_string', 'mime', 'quopriMIME', 'sys'] Those names with [...] bracketing the names are submodules or subpackages of the email package which haven't been imported yet. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Started my PSF core grant today
For those of you who don't know, the PSF has given me a two month grant to work on the core. It's mostly focused on the long overdue overhaul of the dev docs (now being called the devguide) and writing a HOWTO on porting Python 2 code to Python 3. If I have time left over it will be spent on the test suite. I have a blog post with links to my original grant proposal at http://sayspy.blogspot.com/2011/01/psf-core-grant-day-1.html . ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Started my PSF core grant today
On Jan 4, 2011, at 4:36 PM, Brett Cannon wrote: For those of you who don't know, the PSF has given me a two month grant to work on the core. It's mostly focused on the long overdue overhaul of the dev docs (now being called the devguide) and writing a HOWTO on porting Python 2 code to Python 3. If I have time left over it will be spent on the test suite. Woohoo. Nice to have you working on these tasks. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] devguide: Strip out all generic svn instructions from the FAQ. It's not only
On Wed, Jan 5, 2011 at 5:55 AM, brett.cannon python-check...@python.org wrote: brett.cannon pushed 72a286c3452d to devguide: http://hg.python.org/devguide/rev/72a286c3452d changeset: 13:72a286c3452d user: Brett Cannon br...@python.org date: Tue Jan 04 11:48:38 2011 -0800 summary: Strip out all generic svn instructions from the FAQ. It's not only silly to duplicate instructions that can be found all over the internet that are maintained by the creators of the tools under discussion, but it's a maintenance burden that is unneeded. Your call as the author, but please reconsider this one. I've found it *hugely* convenient over the years to have these task oriented answers in the FAQ. The problem with the answers all over the internet is that I (or someone new to our source control tool) may not know enough to ask the right question, and hence those answers may as well not exist. Even if these FAQ answers don't always provide everything needed, they usually provide enough information to let me search for the full answers. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] devguide: Strip out all generic svn instructions from the FAQ. It's not only
On Wed, Jan 5, 2011 at 04:13, Nick Coghlan ncogh...@gmail.com wrote: On Wed, Jan 5, 2011 at 5:55 AM, brett.cannon python-check...@python.org wrote: brett.cannon pushed 72a286c3452d to devguide: http://hg.python.org/devguide/rev/72a286c3452d changeset: 13:72a286c3452d user:Brett Cannon br...@python.org date:Tue Jan 04 11:48:38 2011 -0800 summary: Strip out all generic svn instructions from the FAQ. It's not only silly to duplicate instructions that can be found all over the internet that are maintained by the creators of the tools under discussion, but it's a maintenance burden that is unneeded. Your call as the author, but please reconsider this one. I've found it *hugely* convenient over the years to have these task oriented answers in the FAQ. The problem with the answers all over the internet is that I (or someone new to our source control tool) may not know enough to ask the right question, and hence those answers may as well not exist. Even if these FAQ answers don't always provide everything needed, they usually provide enough information to let me search for the full answers. I agree with Nick here. I also found these instructions useful in the past, although I'm quite familiar with SVN. New devs interested in contributing to Python but not too familiar with the source-control tool it's using at the time will benefit even more from this. As for maintenance nightmare, I'm sure it's simple enough to attract contributors. For example, I can volunteer to maintain it. Eli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] [issue8033] sqlite: broken long integer handling for arguments to user-defined functions
Hello, I am quite new to development in python, and as a first contribution to the community, I have provided a patch to the issue 8033 (quite trivial). I then ran the test suite an everything was ok. However, the status has not changed, and nobody has answered so far (patch provided one month ago). So my question : has I missed something in the procedure that I read carefully, to deliver a patch, or something else? Thank you for your help, and for taking care of python. Philippe ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
