[Python-Dev] issues found by Coverity
Hello, maybe you have noticed a bunch of commits I made the last couple of days. They were all related to resource leaks and other issues that were detected by Coverity. Maybe you have seen the CID in some checkin messages. Most memory and reference leaks were found in the error branch of functions and predates even Python 2.7. The leaks are usually in highly unlikely branches and cause no serious problem. Coverity is really great in following up all possible code branches. A few issues are 3.3 regressions, for example #15900, #15895 and 85cb90f79cbf are IMHO serious issues that should be fixed in the next rc. AFAIK Georg wants to create another rc. I didn't create NEWS entries for my commits yet and just a few tickets for modification I wasn't sure about. Is it sufficient to write just one news entry that lists all fixed functions for 2.7 and 3.2? Georg: Shall I create a list of commits with just regressions so you can cherry pick them? Or do you want to pull all memory leak fixes? Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Release of astoptimizer 0.3
Hi,
Here are some progress on my astoptimizer project. If you are interested by
the optimizer, run it on your own project or help me to implement more
optimizations.
http://pypi.python.org/pypi/astoptimizer
https://bitbucket.org/haypo/astoptimizer
---
The last version (0.3) works on Python 2.6-3.3 and implements the
following optimizations:
* Call builtin functions if arguments are constants. Examples:
- len("abc") => 3
- ord("A") => 65
* Call methods of builtin types if the object and arguments are constants.
Examples:
- u"h\\xe9ho".encode("utf-8") => b"h\\xc3\\xa9ho"
- "python2.7".startswith("python") => True
- (32).bit_length() => 6
- float.fromhex("0x1.8p+0") => 1.5
* Call functions of math and string modules for functions without
border effect. Examples:
- math.log(32) / math.log(2) => 5.0
- string.atoi("5") => 5
* Format strings for str%args and print(arg1, arg2, ...) if arguments
are constants and the format string is valid.
Examples:
- "x=%s" % 5 => "x=5"
- print(1.5) => print("1.5")
* Simplify expressions. Examples:
- not(x in y) => x not in y
- 4 and 5 and x and 6 => x and 6
* Loop: replace range() with xrange() on Python 2, and list with
tuple. Examples:
- for x in range(n): ... => for x in xrange(n): ...
- for x in [1, 2, 3]: ... => for x in (1, 2, 3): ...
* Evaluate unary and binary operators, subscript and comparaison if all
arguments are constants. Examples:
- 1 + 2 * 3 => 7
- not True => False
- "abc" * 3 => "abcabcabc"
- abcdef[:3] => abc
- (2, 7, 3)[1] => 7
- frozenset("ab") | frozenset("bc") => frozenset("abc")
- None is None => True
- "2" in "python2.7" => True
- "def f(): return 2 if 4 < 5 else 3" => "def f(): return 2"
* Remove dead code. Examples:
- def f(): return 1; return 2 => def f(): return 1
- if DEBUG: print("debug") => pass with DEBUG declared as False
- while 0: ... => pass
---
Unsafe optimizations are disabled by default. Optimizations can be enabled
using a Config class with "features" like "builtin_funcs" (builtin functions
like len()) or "pythonbin" (optimized code will be execute by the same
Python binary executable).
astoptimizer.patch_compile() can be used to hook the optimizer in the
compile() builtin function. On Python 3.3, it is enough to use the optimizer
on imports (thanks to the importlib). On older versions, the compileall
module can be used to compile a whole project using the optimizer.
I didn't start to benchmark anything yet, I focused on fixing bugs (not
generating invalid code). I will start benchmarks when the "variables"
feature (ex: "x=1; print(x)" => "x=1; print(1)") will work. There is an
experimental support of variables, but it is still too agressive and
generate invalid code in some cases (see the TODO file).
I plan to implement other optimizations like unrolling loop or convert
a loop to a list comprehension, see the TODO file.
Don't hesitate to propose more optimizations if you have some ideas ;-)
Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Release of astoptimizer 0.3
Am 11.09.2012 12:41, schrieb Victor Stinner: > Hi, > > Here are some progress on my astoptimizer project. If you are interested by > the optimizer, run it on your own project or help me to implement more > optimizations. Wow, that's an amazing list of optimizations. Keep up the good work! Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Release of astoptimizer 0.3
On Tue, Sep 11, 2012 at 12:41 PM, Victor Stinner
wrote:
> Hi,
>
> Here are some progress on my astoptimizer project. If you are interested by
> the optimizer, run it on your own project or help me to implement more
> optimizations.
>
> http://pypi.python.org/pypi/astoptimizer
> https://bitbucket.org/haypo/astoptimizer
>
> ---
>
> The last version (0.3) works on Python 2.6-3.3 and implements the
> following optimizations:
>
> * Call builtin functions if arguments are constants. Examples:
>
> - len("abc") => 3
> - ord("A") => 65
Does it preserve python semantics? What if you change the len builtin?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Release of astoptimizer 0.3
>> * Call builtin functions if arguments are constants. Examples:
>>
>> - len("abc") => 3
>> - ord("A") => 65
>
> Does it preserve python semantics? What if you change the len builtin?
This optimization is disabled by default (in the version 0.3), because
builtin functions may be shadowed. Examples: "len=ord;
print(len('A'))" or exec(compile("print(len('A'))", "test", "exec"),
{'len': ord}).
If you know that one specific builtin function is shadowed (ex:
print), you can modify the configuration to enable optimizations on
builtin functions except the specified function.
Do you projects where builtin functions are shadowed?
The idea is that the author of the application knows its application
(... and all modules used by applications) and is able to explicitly
specify what is that "constant". You can for example declare some
variables as constant using Config.add_constant("module.name", value).
In the same manner, you can declare "pure" functions (without border
effect).
Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Snakebite v0.1: ready for beta testing.
Quick start: % cd ~ && svn co http://svn.snakebite.net/.snakebite && cd .snakebite && sh snakebite.subr If all goes well, you should see something like this: A.snakebite/snakebite.subr A.snakebite/ssh_config_ext A.snakebite/ssh_known_hosts U .snakebite Checked out revision 58. Created link for 'sb'. Created link for 'sbx'. Created link for 'sby'. Created link for 'sbctl'. Fixed permissions for /Users/Trent/.snakebite/snakebite.subr. The following commands can now be executed directly: sb sbx sby sbctl Testing connection.done. Getting a list of your projects...done. Looking up your username for project 'cpython'...done. Getting project details for 'trent.nelson@cpython'...done. Setting current project to 'cpython'...done. Trent, you're now configured for cpython. Enjoy! +---+ |Available Hosts| | (Last Update: 2012-09-11 11:08:01Z) | +---+ | Alias | OS | Arch | +---||--+ | a7|AIX 7.1 | Power4 | | d3|DragonFlyBSD 3.0.2 | x64| |d3x|DragonFlyBSD 3.0.2 | x86| | h2|HP-UX 11iv2 | PA-RISC | | h3|HP-UX 11iv3 | Itanium2 | | i6|IRIX 6.5.30 | MIPS | |n51|NetBSD 5.1.2| x64| | n51x|NetBSD 5.1.2| x86| | o51x|OpenBSD 5.1 | x86| |o51|OpenBSD 5.1 | x64| |s10|Solaris 10 | SPARC | | s9|Solaris 9 | SPARC | +---+ Enter alias: Simply enter any of the aliases in the table and it'll ssh you into that box as cpython@, i.e.: Enter alias: a7 AIX arsenic 1 7 000BF95F4C00 :::. :::...::. .,:: :::.:::. :::.,-: ;;`;;`` ;;;`` `, `;;; ;;; ,;;;'' ,[[ '[[, [[[,/[[[' '[==/, [[ [. '[[ [[[ [[[ c$$$cc$$$c $$c '''$ $$ $$$ "Y$c$$ $$$ $$$ 888 888, 888b "88bo, 88bdP 888oo,__ 888Y88 888 `88bo,__,o, YMM ""` "W" "YMmMY" YUMMM MMM YM MMM"YUMP" AIX 7.1 IBM IntelliStation 9114-275 2 x 1.4GHz Power4 CPUs 2 x 2Gbps LP9802 FC HBAs 8GB RAM, 4 x 36GB [cpython@arsenic]~% General notes: - Almost all of the hosts have a corresponding cpython build slave, which always lives in ~/buildslave. - You're more than welcome to set up local builds on each box. Keep everything in ~/hg. Some hosts already have a ~/hg dir, others don't. The layout should be: ~/hg/3.x ~/hg/3.2 ~/hg/2.7 If they don't exist, feel free to create them. It's going to be easiest to just clone the corresponding build directory from ~/buildslave, i.e. if you want a local 3.x area but no ~/hg/3.x exists: % cd ~/hg % hg clone ~/buildslave/3.x-*/build 3.x Once a base repo has been created, you can clone a local copy: hg clone 3.x 3.x.trent.issue2811 Try follow that naming convention as it'll make it easier for other developers to figure out what each directory is for. Also, try and keep tabs on local builds and remove things you don't need once you're done. I haven't finished hooking up the SAN yet so everything is on local disks at the moment; disk space is a bit light in some places. - If you're not used to vi shell key bindings, you're going to have a bad time :-) - Almost all of the hosts (except for the *BSDs) have been set up to use a common ~/.zsh and ~/.vim: http:/svn.snakebite.net/home/trunk/ http:/svn.snakebite.net/skel/trunk/ They're both based on random dotfile hacking I've done over the years and are far from elegant -- so, suggestions welcome. If I'm awake and working, I'll be on #python-dev, so that'll be the best place to get me if you need immediate assistance. So, log in and have a play around! Oh, X11 forwarding works, too, just invoke `sbx` (or `sby`) instead of `sb` and it'll invoke ssh with -X or -Y respectively. All the proprietary UNIX hosts have X11 installed, complete with glorious circa-late-nineties Motif eye candy. For those looking for tangible things to do... take a look at the current buildslaves with [SB] in the name -- almost all of them are failing in some way/shape/form, so there's plenty of stuff to get your teeth stuck into :-) Trent. ___ Python-Dev mailing list Python-Dev@pytho
Re: [Python-Dev] Snakebite v0.1: ready for beta testing.
Very cool, Trent! I also love the retro use of svn as a tie-in to how long you have been fighting to bring this project to fruition. =) On Tue, Sep 11, 2012 at 8:20 AM, Trent Nelson wrote: > Quick start: > > % cd ~ && svn co http://svn.snakebite.net/.snakebite && cd .snakebite && > sh snakebite.subr > > If all goes well, you should see something like this: > > A.snakebite/snakebite.subr > A.snakebite/ssh_config_ext > A.snakebite/ssh_known_hosts > U .snakebite > Checked out revision 58. > Created link for 'sb'. > Created link for 'sbx'. > Created link for 'sby'. > Created link for 'sbctl'. > Fixed permissions for /Users/Trent/.snakebite/snakebite.subr. > The following commands can now be executed directly: > sb > sbx > sby > sbctl > Testing connection.done. > Getting a list of your projects...done. > Looking up your username for project 'cpython'...done. > Getting project details for 'trent.nelson@cpython'...done. > Setting current project to 'cpython'...done. > Trent, you're now configured for cpython. Enjoy! > +---+ > |Available Hosts| > | (Last Update: 2012-09-11 11:08:01Z) | > +---+ > | Alias | OS | Arch | > +---||--+ > | a7|AIX 7.1 | Power4 | > | d3|DragonFlyBSD 3.0.2 | x64| > |d3x|DragonFlyBSD 3.0.2 | x86| > | h2|HP-UX 11iv2 | PA-RISC | > | h3|HP-UX 11iv3 | Itanium2 | > | i6|IRIX 6.5.30 | MIPS | > |n51|NetBSD 5.1.2| x64| > | n51x|NetBSD 5.1.2| x86| > | o51x|OpenBSD 5.1 | x86| > |o51|OpenBSD 5.1 | x64| > |s10|Solaris 10 | SPARC | > | s9|Solaris 9 | SPARC | > +---+ > Enter alias: > > Simply enter any of the aliases in the table and it'll ssh you into > that box as cpython@, i.e.: > > Enter alias: a7 > AIX arsenic 1 7 000BF95F4C00 > > :::. :::...::. .,:: :::.:::. :::.,-: > ;;`;;`` ;;;`` `, `;;; ;;; ,;;;'' >,[[ '[[, [[[,/[[[' '[==/, [[ [. '[[ [[[ [[[ > c$$$cc$$$c $$c '''$ $$ $$$ "Y$c$$ $$$ $$$ >888 888, 888b "88bo, 88bdP 888oo,__ 888Y88 888 `88bo,__,o, >YMM ""` "W" "YMmMY" YUMMM MMM YM MMM > "YUMP" > > AIX 7.1 >IBM IntelliStation 9114-275 > 2 x 1.4GHz Power4 CPUs > 2 x 2Gbps LP9802 FC HBAs > 8GB RAM, 4 x 36GB > > [cpython@arsenic]~% > > General notes: > > - Almost all of the hosts have a corresponding cpython build slave, > which always lives in ~/buildslave. > > - You're more than welcome to set up local builds on each box. > Keep everything in ~/hg. Some hosts already have a ~/hg dir, > others don't. The layout should be: > > ~/hg/3.x > ~/hg/3.2 > ~/hg/2.7 > > If they don't exist, feel free to create them. It's going to > be easiest to just clone the corresponding build directory > from ~/buildslave, i.e. if you want a local 3.x area but no > ~/hg/3.x exists: > > % cd ~/hg > % hg clone ~/buildslave/3.x-*/build 3.x > > Once a base repo has been created, you can clone a local copy: > hg clone 3.x 3.x.trent.issue2811 > > Try follow that naming convention as it'll make it easier for > other developers to figure out what each directory is for. > > Also, try and keep tabs on local builds and remove things you > don't need once you're done. I haven't finished hooking up > the SAN yet so everything is on local disks at the moment; > disk space is a bit light in some places. > > - If you're not used to vi shell key bindings, you're going to > have a bad time :-) > > - Almost all of the hosts (except for the *BSDs) have been set > up to use a common ~/.zsh and ~/.vim: > http:/svn.snakebite.net/home/trunk/ > http:/svn.snakebite.net/skel/trunk/ > They're both based on random dotfile hacking I've done over > the years and are far from elegant -- so, suggestions welcome. > > If I'm awake and working, I'll be on #python-dev, so that'll be the > best place to get me if you need immediate assistance. > > So, log in and have a play around! Oh, X11 forwarding works, too, > just invoke `sbx` (or `sby`) instead of `sb` and it'll invoke ssh > with -X or -Y respectively. All the proprietary UNIX hosts have > X11 installed, complete with glorious circa-
Re: [Python-Dev] Snakebite v0.1: ready for beta testing.
On Tue, Sep 11, 2012 at 05:20:01AM -0700, Trent Nelson wrote: > Quick start: > > % cd ~ && svn co http://svn.snakebite.net/.snakebite && cd .snakebite && sh > snakebite.subr For those that already have ~/.snakebite, one of these will work: - sbctl hard-reset, or - svn update && sh snakebite.subr In general, from now on (I tweaked it quite a bit this morning), 'sb' automatically updates itself as necessary. (Pro-tip though: `sh snakebite.subr` will always do its best to re-initialize everything, like fixing permissions when svn update strips them, etc.) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Snakebite v0.1: ready for beta testing.
On Tue, Sep 11, 2012 at 05:23:34AM -0700, Brett Cannon wrote: >Very cool, Trent! I also love the retro use of svn as a tie-in to how long >you have been fighting to bring this project to fruition. =) Haha. I probably shouldn't mention that I started writing all the wrapper .snakebite/svn stuff from scratch last Wednesday :P Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Release of astoptimizer 0.3
On Tue, Sep 11, 2012 at 8:41 PM, Victor Stinner
wrote:
> * Call builtin functions if arguments are constants. Examples:
>
> - len("abc") => 3
> - ord("A") => 65
This is fine in an external project, but should never be added to the
standard library. The barrier to semantic changes that break
monkeypatching should be high.
Yes, this is frustrating as it eliminates a great many interesting
static optimisations that are *probably* OK. That's one of the reasons
why PyPy uses tracing - it can perform these optimisations *and* still
include the appropriate dynamic checks.
However, the double barrier of third party module + off by default is
a suitable activation barrier for ensuring people know that what
they're doing is producing bytecode that doesn't behave like standard
Python any more (e.g. tests won't be able to shadow builtins or
optimised module references). Optimisations that break the language
semantics are heading towards the same territory as the byteplay and
withhacks modules (albeit not as evil internally).
> * Call methods of builtin types if the object and arguments are constants.
> Examples:
>
> - u"h\\xe9ho".encode("utf-8") => b"h\\xc3\\xa9ho"
> - "python2.7".startswith("python") => True
> - (32).bit_length() => 6
> - float.fromhex("0x1.8p+0") => 1.5
That last one isn't constant, it's a name lookup. Very cool
optimisations for literals, though.
> * Call functions of math and string modules for functions without
> border effect. Examples:
>
> - math.log(32) / math.log(2) => 5.0
> - string.atoi("5") => 5
Same comment applies here as for the builtin optimisation: fine in an
external project, not in the standard library (even if it's off by
default - merely having it there is still an official endorsement of
deliberately breaking the dynamic lookup semantics of our own
language).
> * Format strings for str%args and print(arg1, arg2, ...) if arguments
> are constants and the format string is valid.
> Examples:
>
> - "x=%s" % 5 => "x=5"
> - print(1.5) => print("1.5")
The print example runs afoul of the general rule above: not in the
standard library, because you're changing the values seen by a mocked
version of print()
> * Simplify expressions. Examples:
>
> - not(x in y) => x not in y
This (and the "is") equivalent should be OK
> - 4 and 5 and x and 6 => x and 6
So long as this is just constant folding, that should be fine, too.
>
> * Loop: replace range() with xrange() on Python 2, and list with
> tuple. Examples:
>
> - for x in range(n): ... => for x in xrange(n): ...
> - for x in [1, 2, 3]: ... => for x in (1, 2, 3): ...
Name lookup optimisations again: not in the standard library.
> * Evaluate unary and binary operators, subscript and comparaison if all
> arguments are constants. Examples:
>
> - 1 + 2 * 3 => 7
> - not True => False
> - "abc" * 3 => "abcabcabc"
> - abcdef[:3] => abc
> - (2, 7, 3)[1] => 7
> - frozenset("ab") | frozenset("bc") => frozenset("abc")
> - None is None => True
> - "2" in "python2.7" => True
> - "def f(): return 2 if 4 < 5 else 3" => "def f(): return 2"
Yep, literals are good.
> * Remove dead code. Examples:
>
> - def f(): return 1; return 2 => def f(): return 1
> - if DEBUG: print("debug") => pass with DEBUG declared as False
> - while 0: ... => pass
Dangerous.
def f(): return 1; yield
if DEBUG: yield
while 0: yield
>>> def f():
... if 0:
... global x
... return x
...
>>> f()
Traceback (most recent call last):
File "", line 1, in
File "", line 4, in f
NameError: global name 'x' is not defined
> Unsafe optimizations are disabled by default. Optimizations can be enabled
> using a Config class with "features" like "builtin_funcs" (builtin functions
> like len()) or "pythonbin" (optimized code will be execute by the same
> Python binary executable).
>
> astoptimizer.patch_compile() can be used to hook the optimizer in the
> compile() builtin function. On Python 3.3, it is enough to use the optimizer
> on imports (thanks to the importlib). On older versions, the compileall
> module can be used to compile a whole project using the optimizer.
>
> I didn't start to benchmark anything yet, I focused on fixing bugs (not
> generating invalid code). I will start benchmarks when the "variables"
> feature (ex: "x=1; print(x)" => "x=1; print(1)") will work. There is an
> experimental support of variables, but it is still too agressive and
> generate invalid code in some cases (see the TODO file).
>
> I plan to implement other optimizations like unrolling loop or convert
> a loop to a list comprehension, see the TODO file.
>
> Don't hesitate to propose more optimizations if you have some ideas ;-)
Mainly just a request to be *very*, *very* clear that the unsafe
optimisations will produce bytecode that *does not behave like Python*
with respect to name lookup semantics, thus mock based testing that
relies on name shadowing will not work correctly, and neither will
direct monkeypa
Re: [Python-Dev] Snakebite v0.1: ready for beta testing.
On Tue, Sep 11, 2012 at 10:28 PM, Trent Nelson wrote: > (Pro-tip though: `sh snakebite.subr` will always do its best to > re-initialize everything, like fixing permissions when svn update > strips them, etc.) That's actually causing some problems - if SELinux security context info or ACLs are stored for the directory, then the extra character (either '.' or '+') in the ls -l output causes the permissions check to fail. Easy enough to fix though, the __get_perms command just needs to be changed to: ls -l $1 | cut -b 1-10 Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Snakebite v0.1: ready for beta testing.
On Tue, Sep 11, 2012 at 06:16:52AM -0700, Nick Coghlan wrote: > On Tue, Sep 11, 2012 at 10:28 PM, Trent Nelson wrote: > > (Pro-tip though: `sh snakebite.subr` will always do its best to > > re-initialize everything, like fixing permissions when svn update > > strips them, etc.) > > That's actually causing some problems - if SELinux security context > info or ACLs are stored for the directory, then the extra character > (either '.' or '+') in the ls -l output causes the permissions check > to fail. > > Easy enough to fix though, the __get_perms command just needs to be changed > to: > > ls -l $1 | cut -b 1-10 Ta, fixed. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Issue 15906; regression in argparse in Python 3.3, 3.2, and 2.7
Issue 15906 describes a problem with argparse that is breaking lots of code in
Ubuntu. This is a recent regression caused by the fix for issue 12776, and it
affects Python 2.7, 3.2, and 3.3.
I posted a diff that should fix the problem, but at the heart of it is a
semantic ambiguity in argparse that needs clarification. This needs to be
cleared up before a proper patch can be applied. I have submitted a patch for
what *I* think reasonable semantics should be, but let's see what you think.
Issue 12776 tried to fix a problem illustrated by this example:
-snip snip-
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--file', type=open, default='/etc/passwd')
args = parser.parse_args()
print(args.file.read())
args = parser.parse_args(['--file', '/etc/group'])
print(args.file.read())
-snip snip-
What this code is (IMO, sensibly) trying to do is say that args.file will
always be an open file, regardless of whether the path comes from the default
value or is given on the command line.
The problem is that this breaks some other, also sensible code:
-snip snip-
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--test", dest="test", type=str,
default=[], action='append')
args = parser.parse_args(['--test', 'bar'])
args.test.append('baz')
args = parser.parse_args()
args.test.append('baz')
-snip snip-
This code is saying, I want to accumulate string arguments into a list,
regardless of whether any arguments are given on the command line.
The fix for issue 12776 broke the last two lines of the second example,
because in the no-command-line-arguments-given case, arg.test is the *string*
"[]" and not the actual empty list object.
It seems to me that the semantics could reasonably be implied to mean that the
type converter should only be applied to the default value when
action='store', as is the default. Then in the second example, because
action='append', the type conversion would not be applied (it makes no sense
to do so).
I have attached a diff to issue 15906 that implements these semantics. If you
agree, then I will apply this to all of 3.3, 3.2, and 2.7, which are all
affected by this bug (because the original fix for 12776 was applied to all
three branches).
Georg, I would like to apply this to the 3.3 branch.
Cheers,
-Barry
signature.asc
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 3.3.0 release candidate 2
Just a small note, MD5 for RC2 file python-3.3.0rc2.msi is not correct on
http://python.org/download/releases/3.3.0/
it would be nice if someone can update it
cheers,
Perica
On Sunday, September 9, 2012 4:25:39 AM UTC-5, Georg Brandl wrote:
> -BEGIN PGP SIGNED MESSAGE-
>
> Hash: SHA1
>
>
>
> On behalf of the Python development team, I'm delighted to announce the
>
> second release candidate of Python 3.3.0.
>
>
>
> This is a preview release, and its use is not recommended in
>
> production settings.
>
>
>
> Python 3.3 includes a range of improvements of the 3.x series, as well
>
> as easier porting between 2.x and 3.x. Major new features and changes
>
> in the 3.3 release series are:
>
>
>
> * PEP 380, syntax for delegating to a subgenerator ("yield from")
>
> * PEP 393, flexible string representation (doing away with the
>
> distinction between "wide" and "narrow" Unicode builds)
>
> * A C implementation of the "decimal" module, with up to 80x speedup
>
> for decimal-heavy applications
>
> * The import system (__import__) now based on importlib by default
>
> * The new "lzma" module with LZMA/XZ support
>
> * PEP 397, a Python launcher for Windows
>
> * PEP 405, virtual environment support in core
>
> * PEP 420, namespace package support
>
> * PEP 3151, reworking the OS and IO exception hierarchy
>
> * PEP 3155, qualified name for classes and functions
>
> * PEP 409, suppressing exception context
>
> * PEP 414, explicit Unicode literals to help with porting
>
> * PEP 418, extended platform-independent clocks in the "time" module
>
> * PEP 412, a new key-sharing dictionary implementation that
>
> significantly saves memory for object-oriented code
>
> * PEP 362, the function-signature object
>
> * The new "faulthandler" module that helps diagnosing crashes
>
> * The new "unittest.mock" module
>
> * The new "ipaddress" module
>
> * The "sys.implementation" attribute
>
> * A policy framework for the email package, with a provisional (see
>
> PEP 411) policy that adds much improved unicode support for email
>
> header parsing
>
> * A "collections.ChainMap" class for linking mappings to a single unit
>
> * Wrappers for many more POSIX functions in the "os" and "signal"
>
> modules, as well as other useful functions such as "sendfile()"
>
> * Hash randomization, introduced in earlier bugfix releases, is now
>
> switched on by default
>
>
>
> In total, almost 500 API items are new or improved in Python 3.3.
>
> For a more extensive list of changes in 3.3.0, see
>
>
>
> http://docs.python.org/3.3/whatsnew/3.3.html
>
>
>
> To download Python 3.3.0 visit:
>
>
>
> http://www.python.org/download/releases/3.3.0/
>
>
>
> Please consider trying Python 3.3.0 with your code and reporting any bugs
>
> you may notice to:
>
>
>
> http://bugs.python.org/
>
>
>
>
>
> Enjoy!
>
>
>
> - --
>
> Georg Brandl, Release Manager
>
> georg at python.org
>
> (on behalf of the entire python-dev team and 3.3's contributors)
>
> -BEGIN PGP SIGNATURE-
>
> Version: GnuPG v2.0.19 (GNU/Linux)
>
>
>
> iEYEARECAAYFAlBMYJMACgkQN9GcIYhpnLCc5ACfcufn57tkNBPFU7qCpZ74GzjW
>
> msMAn3sIwWHLdqixypnnyMBOw1ijILjo
>
> =+e0e
>
> -END PGP SIGNATURE-
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 15906; regression in argparse in Python 3.3, 3.2, and 2.7
On Tue, 11 Sep 2012 11:34:30 -0400, Barry Warsaw wrote:
> Issue 15906 describes a problem with argparse that is breaking lots of code in
> Ubuntu. This is a recent regression caused by the fix for issue 12776, and it
> affects Python 2.7, 3.2, and 3.3.
>
> I posted a diff that should fix the problem, but at the heart of it is a
> semantic ambiguity in argparse that needs clarification. This needs to be
> cleared up before a proper patch can be applied. I have submitted a patch for
> what *I* think reasonable semantics should be, but let's see what you think.
>
> Issue 12776 tried to fix a problem illustrated by this example:
>
> -snip snip-
> import argparse
>
> parser = argparse.ArgumentParser()
> parser.add_argument('--file', type=open, default='/etc/passwd')
> args = parser.parse_args()
>
> print(args.file.read())
>
> args = parser.parse_args(['--file', '/etc/group'])
> print(args.file.read())
> -snip snip-
>
> What this code is (IMO, sensibly) trying to do is say that args.file will
> always be an open file, regardless of whether the path comes from the default
> value or is given on the command line.
Actually, what 12776 was trying to fix was that the conversion function
was *always* called, and thus the file was *always* opened, even when
a file argument was passed in the arguments to be parsed.
> The problem is that this breaks some other, also sensible code:
>
> -snip snip-
> import argparse
>
> parser = argparse.ArgumentParser()
> parser.add_argument("--test", dest="test", type=str,
> default=[], action='append')
>
> args = parser.parse_args(['--test', 'bar'])
> args.test.append('baz')
>
> args = parser.parse_args()
> args.test.append('baz')
> -snip snip-
>
> This code is saying, I want to accumulate string arguments into a list,
> regardless of whether any arguments are given on the command line.
>
> The fix for issue 12776 broke the last two lines of the second example,
> because in the no-command-line-arguments-given case, arg.test is the *string*
> "[]" and not the actual empty list object.
But apparently in doing so we broke something else.
There is no question that the current state of the tip of each branch is
a regression. So at a minimum the old behavior needs to be restored,
which is that the above code should do as you say: args.test should end
up initialized to the empty list.
As interesting aside: if the above is changed to
args = parser.parse_args(['--test], 'x'])
the append will work. The fact that that is true is doubtless a clue
as to why the fix for 12776 broke things unexpectedly.
> It seems to me that the semantics could reasonably be implied to mean that the
> type converter should only be applied to the default value when
> action='store', as is the default. Then in the second example, because
> action='append', the type conversion would not be applied (it makes no sense
> to do so).
There is another possible semantic, which is that when the store type is
append, the converter should be applied to each of the individual items
in the default list.
Which brings us to another issue: as things stand now, if we have, say,
'default=['abc']', then passing in '--test x' on the command line would
result in args.test being equal to
['abc', 'x']
which is consistent with optparse but not necessarily the desired
semantics.
> I have attached a diff to issue 15906 that implements these semantics. If you
> agree, then I will apply this to all of 3.3, 3.2, and 2.7, which are all
> affected by this bug (because the original fix for 12776 was applied to all
> three branches).
>
> Georg, I would like to apply this to the 3.3 branch.
The 12776 fix isn't going to be in 3.3, so I don't think this is a
pressing issue. We can take our time to make sure we have the correct
fix. It is, however, a release blocker for 2.7.4, 3.2.4, and 3.3.1.
--David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Release of astoptimizer 0.3
On 11/09/2012 13:06, Victor Stinner wrote:
* Call builtin functions if arguments are constants. Examples:
- len("abc") => 3
- ord("A") => 65
Does it preserve python semantics? What if you change the len builtin?
This optimization is disabled by default (in the version 0.3), because
builtin functions may be shadowed. Examples: "len=ord;
print(len('A'))" or exec(compile("print(len('A'))", "test", "exec"),
{'len': ord}).
If you know that one specific builtin function is shadowed (ex:
print), you can modify the configuration to enable optimizations on
builtin functions except the specified function.
Do you projects where builtin functions are shadowed?
The idea is that the author of the application knows its application
(... and all modules used by applications) and is able to explicitly
specify what is that "constant". You can for example declare some
variables as constant using Config.add_constant("module.name", value).
In the same manner, you can declare "pure" functions (without border
effect).
Do you know what the cost would be of, say, replacing:
len("abc")
with:
3 if len is __builtins__.len else len("abc")
if possible where the lookup __builtins__.len is been done early, such
as at compile time?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 15906; regression in argparse in Python 3.3, 3.2, and 2.7
On 9/11/2012 11:34 AM, Barry Warsaw wrote:
Issue 15906 describes a problem with argparse that is breaking lots of code in
Ubuntu. This is a recent regression caused by the fix for issue 12776, and it
affects Python 2.7, 3.2, and 3.3.
I posted a diff that should fix the problem, but at the heart of it is a
semantic ambiguity in argparse that needs clarification. This needs to be
cleared up before a proper patch can be applied. I have submitted a patch for
what *I* think reasonable semantics should be, but let's see what you think.
Issue 12776 tried to fix a problem illustrated by this example:
-snip snip-
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--file', type=open, default='/etc/passwd')
args = parser.parse_args()
print(args.file.read())
args = parser.parse_args(['--file', '/etc/group'])
print(args.file.read())
-snip snip-
What this code is (IMO, sensibly) trying to do is say that args.file will
always be an open file, regardless of whether the path comes from the default
value or is given on the command line.
The problem is that this breaks some other, also sensible code:
-snip snip-
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--test", dest="test", type=str,
default=[], action='append')
args = parser.parse_args(['--test', 'bar'])
args.test.append('baz')
args = parser.parse_args()
args.test.append('baz')
-snip snip-
This code is saying, I want to accumulate string arguments into a list,
regardless of whether any arguments are given on the command line.
This second example strikes me (naively, as English speaker but not
argparse user) as 'wrong' in that 'default' is being misused to mean
'start value that is always used to generate the final value' [as in
sum(iterable, start=0)], rather than 'final value that is only used if
nothing else is given' (as in nearly all uses of 'default' in Python).
Perhaps this is what you meant by "semantic ambiguity".
As I see it, storing is done *with* a default or explicit value,
appending is done *to* a start value *with* whatever. Perhaps reusing
'default' instead of using a new name such as 'start' was a bit too
clever for our own good ;-).
The fix for issue 12776 broke the last two lines of the second example,
because in the no-command-line-arguments-given case, arg.test is the *string*
"[]" and not the actual empty list object.
This seems even more wrong (as in slightly crazy) as it switches the
meaning of 'default' within one parser example rather than between
parser examples.
It seems to me that the semantics could reasonably be implied to mean that the
type converter should only be applied to the default value when
action='store', as is the default. Then in the second example, because
action='append', the type conversion would not be applied (it makes no sense
to do so).
I arrive at the same conclusion, I believe, by saying that for a given
parser, the type converter should always or never be applied to
'default', which should mean converting or not when the parser is
created. Append to 'default as base or start' should mean not converting.
--
Terry Jan Reedy
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 3.3.0 release candidate 2
Indeed, thanks for catching that.
The GPG signatures are good, so the downloads are the original ones
built by Martin.
Georg
On 11.09.2012 18:11, Perica Zivkovic wrote:
> Just a small note, MD5 for RC2 file python-3.3.0rc2.msi is not correct on
> http://python.org/download/releases/3.3.0/
>
> it would be nice if someone can update it
>
> cheers,
>
> Perica
>
> On Sunday, September 9, 2012 4:25:39 AM UTC-5, Georg Brandl wrote:
> Hash: SHA1
>
>
>
> On behalf of the Python development team, I'm delighted to announce the
>
> second release candidate of Python 3.3.0.
>
>
>
> This is a preview release, and its use is not recommended in
>
> production settings.
>
>
>
> Python 3.3 includes a range of improvements of the 3.x series, as well
>
> as easier porting between 2.x and 3.x. Major new features and changes
>
> in the 3.3 release series are:
>
>
>
> * PEP 380, syntax for delegating to a subgenerator ("yield from")
>
> * PEP 393, flexible string representation (doing away with the
>
> distinction between "wide" and "narrow" Unicode builds)
>
> * A C implementation of the "decimal" module, with up to 80x speedup
>
> for decimal-heavy applications
>
> * The import system (__import__) now based on importlib by default
>
> * The new "lzma" module with LZMA/XZ support
>
> * PEP 397, a Python launcher for Windows
>
> * PEP 405, virtual environment support in core
>
> * PEP 420, namespace package support
>
> * PEP 3151, reworking the OS and IO exception hierarchy
>
> * PEP 3155, qualified name for classes and functions
>
> * PEP 409, suppressing exception context
>
> * PEP 414, explicit Unicode literals to help with porting
>
> * PEP 418, extended platform-independent clocks in the "time" module
>
> * PEP 412, a new key-sharing dictionary implementation that
>
> significantly saves memory for object-oriented code
>
> * PEP 362, the function-signature object
>
> * The new "faulthandler" module that helps diagnosing crashes
>
> * The new "unittest.mock" module
>
> * The new "ipaddress" module
>
> * The "sys.implementation" attribute
>
> * A policy framework for the email package, with a provisional (see
>
> PEP 411) policy that adds much improved unicode support for email
>
> header parsing
>
> * A "collections.ChainMap" class for linking mappings to a single unit
>
> * Wrappers for many more POSIX functions in the "os" and "signal"
>
> modules, as well as other useful functions such as "sendfile()"
>
> * Hash randomization, introduced in earlier bugfix releases, is now
>
> switched on by default
>
>
>
> In total, almost 500 API items are new or improved in Python 3.3.
>
> For a more extensive list of changes in 3.3.0, see
>
>
>
> http://docs.python.org/3.3/whatsnew/3.3.html
>
>
>
> To download Python 3.3.0 visit:
>
>
>
> http://www.python.org/download/releases/3.3.0/
>
>
>
> Please consider trying Python 3.3.0 with your code and reporting any bugs
>
> you may notice to:
>
>
>
> http://bugs.python.org/
>
>
>
>
>
> Enjoy!
>
>
>
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Release of astoptimizer 0.3
On Tue, Sep 11, 2012 at 10:06 AM, MRAB wrote:
> On 11/09/2012 13:06, Victor Stinner wrote:
* Call builtin functions if arguments are constants. Examples:
- len("abc") => 3
- ord("A") => 65
>>>
>>>
>>> Does it preserve python semantics? What if you change the len builtin?
>>
>>
>> This optimization is disabled by default (in the version 0.3), because
>> builtin functions may be shadowed. Examples: "len=ord;
>> print(len('A'))" or exec(compile("print(len('A'))", "test", "exec"),
>> {'len': ord}).
>>
>> If you know that one specific builtin function is shadowed (ex:
>> print), you can modify the configuration to enable optimizations on
>> builtin functions except the specified function.
>>
>> Do you projects where builtin functions are shadowed?
>>
>> The idea is that the author of the application knows its application
>> (... and all modules used by applications) and is able to explicitly
>> specify what is that "constant". You can for example declare some
>> variables as constant using Config.add_constant("module.name", value).
>> In the same manner, you can declare "pure" functions (without border
>> effect).
>>
> Do you know what the cost would be of, say, replacing:
>
> len("abc")
>
> with:
>
> 3 if len is __builtins__.len else len("abc")
>
> if possible where the lookup __builtins__.len is been done early, such
> as at compile time?
Doing the lookup at compile time would defeat the whole purpose (the
dynamic shadowing of len could happen at any time). Also there are two
forms of shadowing: patching the globals of the module where the call
occurs, or patching the builtins.
Doing the lookup at run time would also defeat the purpose, because
most of the cost of calling len() *is* the lookup. Actually two
lookups -- the LOAD_GLOBAL opcode is used, which first looks in the
globals (this will normally always fail) and then in the builtins.
FWIW, I expect that there are few places where len()
is actually used. However I expect it to be quite common for ord(),
where the same reasoning applies.
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 15906; regression in argparse in Python 3.3, 3.2, and 2.7
On Tue, Sep 11, 2012 at 10:17 AM, Terry Reedy wrote: > On 9/11/2012 11:34 AM, Barry Warsaw wrote: >> It seems to me that the semantics could reasonably be implied to mean that >> the >> type converter should only be applied to the default value when >> action='store', as is the default. Then in the second example, because >> action='append', the type conversion would not be applied (it makes no >> sense >> to do so). > > I arrive at the same conclusion, I believe, by saying that for a given > parser, the type converter should always or never be applied to 'default', > which should mean converting or not when the parser is created. Append to > 'default as base or start' should mean not converting. I noted this in the comment to the issue, but the argparse documentation says this about the type argument: "type= can take any callable that takes a single string argument and returns the converted value." (from http://docs.python.org/dev/library/argparse.html#type ) So type is meant for converting (command-line) strings to other types. What is causing one of the problems here is calling type on a non-string (an empty list in one of the cases above), which doesn't look like it's meant to be supported.. So another way out could simply be not to call type on non-strings. Indeed, this was done before. One of the changes that was made in the patch for issue 12776 was to remove the str type check prior to calling type. --Chris ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Release of astoptimizer 0.3
On 11.09.12 13:41, Victor Stinner wrote:
Here are some progress on my astoptimizer project. If you are interested by
the optimizer, run it on your own project or help me to implement more
optimizations.
http://pypi.python.org/pypi/astoptimizer
https://bitbucket.org/haypo/astoptimizer
It's a very interesting work.
* Call builtin functions if arguments are constants. Examples:
- len("abc") => 3
- ord("A") => 65
Does this mean transformation print("A") => None and output at compile time?
* Loop: replace range() with xrange() on Python 2,
range(0, 10**10, 10**9)
* Evaluate unary and binary operators, subscript and comparaison if all
arguments are constants. Examples:
- 1 + 2 * 3 => 7
Does this mean 1/0 raises exception at compile time?
* Remove dead code. Examples:
- def f(): return 1; return 2 => def f(): return 1
- if DEBUG: print("debug") => pass with DEBUG declared as False
- while 0: ... => pass
As Nick said, it could change the semantics, if there were local
variable assignment or yield in removed code.
Unsafe optimizations are disabled by default. Optimizations can be enabled
using a Config class with "features" like "builtin_funcs" (builtin functions
like len()) or "pythonbin" (optimized code will be execute by the same
Python binary executable).
It would be good if the optimizer checked if the built-in names
overridden in this module or function and disabled this dangerous
optimization in such case.
I plan to implement other optimizations like unrolling loop or convert
a loop to a list comprehension, see the TODO file.
Don't hesitate to propose more optimizations if you have some ideas ;-)
set([1, 2, 3]) => {1, 2, 3}
set([x for ...]) => {x for ...}
dict([(k, v) for ...]) => {k: v for ...}
dict((k, v) for ...) => {k: v for ...}
''.join([s for ...]) => ''.join(s for ...)
a.extend([s for ...]) => a.extend(s for ...)
(f(x) for x in a) => map(f, a)
(x.y for x in a) => map(operator.attrgetter('y'), a)
(x[0] for x in a) => map(operator.itemgetter(0), a)
(2 * x for x in a) => map((2).__mul__, a)
(x in b for x in a) => map(b.__contains__, a)
map(lambda x: x.strip(), a) => (x.strip() for x in a)
x in ['i', 'em', 'cite'] => x in {'i', 'em', 'cite'}
x == 'i' or x == 'em' or x == 'cite'] => x in {'i', 'em', 'cite'}
a = []; for ...: a.append(x) => a = [x for ...]
a = (); for ...: a.add(x) => a = {x for ...}
a = {}; for ...: a[k] = v => a = {k: v for ...}
for ...: f.write(...) => __fwrite = f.write; for ...: __fwrite(...)
x = x + 1 => x += 1
x = x + ' ' => x += ' '
x = x + [y] => x.append(y)
x = x + [y, z] => x.extend([y, z])
'x=%s' % repr(x) => 'x=%a' % (x,)
'x=%s' % x + s => 'x=%s%s' % (x, s)
x = x + ', [%s]' % y => x = '%s, [%s]' % (x, y)
range(0, x) => range(x)
for i in range(len(a)): x = a[i] ... => for i, x in enumerate(a): ...
i = 0; for x in a: i += 1 ... => for i, x in enumerate(a, 1): ...
i = 1; for x in a: ... i += 1 => for i, x in enumerate(a, 1): ...
s = 0; for ...: if ...: s += 1 => s = sum(1 for ... if ...)
while True: s = f.readline(); if not s: break; ... => for s in f: ...
def f(x): ... len() ... => def f(x, __len = len): ... __len() ...
Not all such transformations are always safe (and I know code in stdlib
where they are not).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Release of astoptimizer 0.3
Nick Coghlan, 11.09.2012 14:57:
> On Tue, Sep 11, 2012 at 8:41 PM, Victor Stinner wrote:
>> * Loop: replace range() with xrange() on Python 2, and list with
>> tuple. Examples:
>>
>> - for x in range(n): ... => for x in xrange(n): ...
>> - for x in [1, 2, 3]: ... => for x in (1, 2, 3): ...
>
> Name lookup optimisations again: not in the standard library.
I assume you meant the "range" part, not the second example (which will end
up caching a constant tuple).
>> * Evaluate unary and binary operators, subscript and comparaison if all
>> arguments are constants. Examples:
>>
>> - 1 + 2 * 3 => 7
>> - not True => False
>> - "abc" * 3 => "abcabcabc"
>> - abcdef[:3] => abc
>> - (2, 7, 3)[1] => 7
>> - frozenset("ab") | frozenset("bc") => frozenset("abc")
That's a name lookup, too.
>> - None is None => True
>> - "2" in "python2.7" => True
>> - "def f(): return 2 if 4 < 5 else 3" => "def f(): return 2"
>
> Yep, literals are good.
Except that evaluating something like '"abc" * constant' can eat up all
memory, imagine this code:
KILL_MEMORY = sys.argv[1] == 'NEVER PASS THIS VALUE'
def test():
if KILL_MEMORY:
return 'abc' * 1
else:
return 'abc'
Stefan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Release of astoptimizer 0.3
Serhiy Storchaka, 11.09.2012 20:48:
> set([1, 2, 3]) => {1, 2, 3}
> set([x for ...]) => {x for ...}
> dict([(k, v) for ...]) => {k: v for ...}
> dict((k, v) for ...) => {k: v for ...}
> ''.join([s for ...]) => ''.join(s for ...)
> a.extend([s for ...]) => a.extend(s for ...)
> (f(x) for x in a) => map(f, a)
> (x.y for x in a) => map(operator.attrgetter('y'), a)
> (x[0] for x in a) => map(operator.itemgetter(0), a)
> (2 * x for x in a) => map((2).__mul__, a)
> (x in b for x in a) => map(b.__contains__, a)
> map(lambda x: x.strip(), a) => (x.strip() for x in a)
> x in ['i', 'em', 'cite'] => x in {'i', 'em', 'cite'}
> x == 'i' or x == 'em' or x == 'cite'] => x in {'i', 'em', 'cite'}
> a = []; for ...: a.append(x) => a = [x for ...]
> a = (); for ...: a.add(x) => a = {x for ...}
> a = {}; for ...: a[k] = v => a = {k: v for ...}
> for ...: f.write(...) => __fwrite = f.write; for ...: __fwrite(...)
> x = x + 1 => x += 1
> x = x + ' ' => x += ' '
> x = x + [y] => x.append(y)
> x = x + [y, z] => x.extend([y, z])
> 'x=%s' % repr(x) => 'x=%a' % (x,)
> 'x=%s' % x + s => 'x=%s%s' % (x, s)
> x = x + ', [%s]' % y => x = '%s, [%s]' % (x, y)
> range(0, x) => range(x)
> for i in range(len(a)): x = a[i] ... => for i, x in enumerate(a): ...
> i = 0; for x in a: i += 1 ... => for i, x in enumerate(a, 1): ...
> i = 1; for x in a: ... i += 1 => for i, x in enumerate(a, 1): ...
> s = 0; for ...: if ...: s += 1 => s = sum(1 for ... if ...)
> while True: s = f.readline(); if not s: break; ... => for s in f: ...
> def f(x): ... len() ... => def f(x, __len = len): ... __len() ...
>
> Not all such transformations are always safe (and I know code in stdlib
> where they are not).
Actually, many of them are not, and some of them are even plain wrong or
may at least turn out not to be optimisations.
Stefan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 15906; regression in argparse in Python 3.3, 3.2, and 2.7
On Sep 11, 2012, at 12:19 PM, R. David Murray wrote:
>The 12776 fix isn't going to be in 3.3, so I don't think this is a
>pressing issue. We can take our time to make sure we have the correct
>fix. It is, however, a release blocker for 2.7.4, 3.2.4, and 3.3.1.
Are you sure about that?
% ./python
Python 3.3.0rc2+ (default:6fea947edead, Sep 11 2012, 15:03:16)
[GCC 4.7.1 20120908 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import argparse
>>> p = argparse.ArgumentParser()
>>> p.add_argument('--test', dest='test', type=str, default=[], action='append')
_AppendAction(option_strings=['--test'], dest='test', nargs=None, const=None,
default=[], type=, choices=None, help=None, metavar=None)
>>> args = p.parse_args()
>>> args.test
'[]'
Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 15906; regression in argparse in Python 3.3, 3.2, and 2.7
On Tue, 11 Sep 2012 15:08:26 -0400, Barry Warsaw wrote: > On Sep 11, 2012, at 12:19 PM, R. David Murray wrote: > > >The 12776 fix isn't going to be in 3.3, so I don't think this is a > >pressing issue. We can take our time to make sure we have the correct > >fix. It is, however, a release blocker for 2.7.4, 3.2.4, and 3.3.1. > > Are you sure about that? > > % ./python > Python 3.3.0rc2+ (default:6fea947edead, Sep 11 2012, 15:03:16) The actual 3.3.0 release will not be made from default, but from a dedicated clone/branch that does not have the 12776-fix changeset in it. --David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 15906; regression in argparse in Python 3.3, 3.2, and 2.7
On Sep 11, 2012, at 03:08 PM, Barry Warsaw wrote:
>Are you sure about that?
>
>% ./python
>Python 3.3.0rc2+ (default:6fea947edead, Sep 11 2012, 15:03:16)
Never mind. Georg didn't pull that patch into his release clone.
(Aside: Georg, maybe you could fiddle with the default branch's version
numbers so that it reflects being 3.3.1a1?)
(Aside 2: releasing/3.3.0 is broken:
Traceback (most recent call last):
File "/home/barry/projects/python/3.3.0/Parser/asdl.py", line 309, in visit
meth(object, *args)
File "./Parser/asdl_c.py", line 1043, in visitSum
self.simpleSum(sum, name)
File "./Parser/asdl_c.py", line 1067, in simpleSum
self.emit("default:" % name, 2)
TypeError: not all arguments converted during string formatting
make: *** [Python/Python-ast.c] Error 1
)
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 15906; regression in argparse in Python 3.3, 3.2, and 2.7
On Sep 11, 2012, at 01:17 PM, Terry Reedy wrote: >This second example strikes me (naively, as English speaker but not argparse >user) as 'wrong' in that 'default' is being misused to mean 'start value that >is always used to generate the final value' [as in sum(iterable, start=0)], >rather than 'final value that is only used if nothing else is given' (as in >nearly all uses of 'default' in Python). Perhaps this is what you meant by >"semantic ambiguity". Well, in a sense yes, that is an ambiguity so I won't quibble about whether it's the same one or not. :) >As I see it, storing is done *with* a default or explicit value, appending is >done *to* a start value *with* whatever. Perhaps reusing 'default' instead of >using a new name such as 'start' was a bit too clever for our own good ;-). I suspect that it's too late to change this, by say adding a 'start' parameter or some such, at least until Python 3.4. >> The fix for issue 12776 broke the last two lines of the second example, >> because in the no-command-line-arguments-given case, arg.test is the >> *string* "[]" and not the actual empty list object. > >This seems even more wrong (as in slightly crazy) as it switches the meaning >of 'default' within one parser example rather than between parser examples. And yet, that's how it works in 2.7, 3.2, and 3.3.1. >> It seems to me that the semantics could reasonably be implied to mean that >> the type converter should only be applied to the default value when >> action='store', as is the default. Then in the second example, because >> action='append', the type conversion would not be applied (it makes no >> sense to do so). > >I arrive at the same conclusion, I believe, by saying that for a given >parser, the type converter should always or never be applied to 'default', >which should mean converting or not when the parser is created. Append to >'default as base or start' should mean not converting. Does that mean that for fixing the regression, you favor applying the type conversion only for action='store' or by only applying it when the default is a string? It seems better to only apply the type conversion for action='store' but more backward compatible for the original behavior to only apply it when default is a string. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 15906; regression in argparse in Python 3.3, 3.2, and 2.7
On Sep 11, 2012, at 10:39 AM, Chris Jerdonek wrote: >So another way out could simply be not to call type on non-strings. >Indeed, this was done before. One of the changes that was made in the >patch for issue 12776 was to remove the str type check prior to >calling type. Yep, so probably that's the right fix for the regression. I'll try to whip up a patch. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 15906; regression in argparse in Python 3.3, 3.2, and 2.7
On Sep 11, 2012, at 12:19 PM, R. David Murray wrote: >There is another possible semantic, which is that when the store type is >append, the converter should be applied to each of the individual items in >the default list. Yep. Maybe for 3.4 . >Which brings us to another issue: as things stand now, if we have, say, >'default=['abc']', then passing in '--test x' on the command line would >result in args.test being equal to > >['abc', 'x'] > >which is consistent with optparse but not necessarily the desired >semantics. Right. This gets to Chris's observation that we're blurring the meaning of 'default'. For action='append' should that be the default value when no arguments are given, or the starting value for the append operations? We might need to add a 'start' argument when action='append'. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 15906; regression in argparse in Python 3.3, 3.2, and 2.7
On 11.09.2012 21:25, Barry Warsaw wrote:
> On Sep 11, 2012, at 03:08 PM, Barry Warsaw wrote:
>
>>Are you sure about that?
>>
>>% ./python
>>Python 3.3.0rc2+ (default:6fea947edead, Sep 11 2012, 15:03:16)
>
> Never mind. Georg didn't pull that patch into his release clone.
>
> (Aside: Georg, maybe you could fiddle with the default branch's version
> numbers so that it reflects being 3.3.1a1?)
Well, it's not yet.
> (Aside 2: releasing/3.3.0 is broken:
>
> Traceback (most recent call last):
> File "/home/barry/projects/python/3.3.0/Parser/asdl.py", line 309, in visit
> meth(object, *args)
> File "./Parser/asdl_c.py", line 1043, in visitSum
> self.simpleSum(sum, name)
> File "./Parser/asdl_c.py", line 1067, in simpleSum
> self.emit("default:" % name, 2)
> TypeError: not all arguments converted during string formatting
> make: *** [Python/Python-ast.c] Error 1
> )
This bug is not specific to that repository. Not sure how you hit the bug; a
fresh checkout doesn't do it for me here.
Georg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 15906; regression in argparse in Python 3.3, 3.2, and 2.7
On Sep 11, 2012, at 10:13 PM, Georg Brandl wrote:
>> Traceback (most recent call last):
>> File "/home/barry/projects/python/3.3.0/Parser/asdl.py", line 309, in visit
>> meth(object, *args)
>> File "./Parser/asdl_c.py", line 1043, in visitSum
>> self.simpleSum(sum, name)
>> File "./Parser/asdl_c.py", line 1067, in simpleSum
>> self.emit("default:" % name, 2)
>> TypeError: not all arguments converted during string formatting
>> make: *** [Python/Python-ast.c] Error 1
>> )
>
>This bug is not specific to that repository. Not sure how you hit the bug; a
>fresh checkout doesn't do it for me here.
http://bugs.python.org/issue15923
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 15906; regression in argparse in Python 3.3, 3.2, and 2.7
On 9/11/2012 3:31 PM, Barry Warsaw wrote: On Sep 11, 2012, at 01:17 PM, Terry Reedy wrote: As I see it, storing is done *with* a default or explicit value, appending is done *to* a start value *with* whatever. Perhaps reusing 'default' instead of using a new name such as 'start' was a bit too clever for our own good ;-). Such cleverness is something that I might have done, having been 'raised' on IBM mainframes with statements with 8-10 position-only parameters (JCL DD? don't quite remember), and having had to carefully count commas writing things like XX arg1, arg2,,,arg5,,,arg8,, The cost of positions rewards multiplexing the meaning of positions. The availability of keyword-only parameters changes the tradeoffs. The fix for issue 12776 broke the last two lines of the second example, because in the no-command-line-arguments-given case, arg.test is the *string* "[]" and not the actual empty list object. This seems even more wrong (as in slightly crazy) as it switches the meaning of 'default' within one parser example rather than between parser examples. And yet, that's how it works in 2.7, 3.2, and 3.3.1. In all 2.7 or 3.2? It seems to me that the semantics could reasonably be implied to mean that the type converter should only be applied to the default value when action='store', as is the default. Then in the second example, because action='append', the type conversion would not be applied (it makes no sense to do so). I arrive at the same conclusion, I believe, by saying that for a given parser, the type converter should always or never be applied to 'default', which should mean converting or not when the parser is created. Append to 'default as base or start' should mean not converting. Does that mean that for fixing the regression, you favor applying the type conversion only for action='store' or by only applying it when the default is a string? How about a third rule which amounts to the second *plus* (maybe) the first: apply the conversion when it fulfills the purpose of having type conversions, which is to make sure that the caller get objects of the desired class. Also, only add necessary restrictions to argparse users. In particular: parser.parse_args gets a sequence of strings. But callers of .parse_args often want non-string objects. argparse could return args as named strings and make the user convert to the types wanted, but there are good reasons to do the conversion at the point of parsing. So the conversion function *must* accept strings, but it seems an unnecessary restriction to require it to accept anything else, including its output type or a non-string 'default'. When the 'default' is a substitute for user input, as with Store, *and* it is a string, it should be converted just like user input. But the parser designer should not have to convert a non-string default object to a string just so it can be converted back to the real default non-string object (another unnecessary restriction). (If type=int, there is no sense in requiring default='100' instead of default=100.) Indeed, the default object might not even be a possible output of the conversion function (for instance, type=int, default=None). If the 'default' is not a backup/replacement for a string input to parse_args, as with append, then it seems it should not be converted either. It seems better to only apply the type conversion for action='store' but more backward compatible for the original behavior to only apply it when default is a string. Are there any other actions where the default might be a string that should be converted the same way as user input? If not, both rules apply. As I said, I am not familiar with the details of argparse (or the previous bugfix), so I let you sort out how these thoughts apply to previous, current, and desired future behavior and doc claims. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Release of astoptimizer 0.3
2012/9/11 Nick Coghlan :
> This is fine in an external project, but should never be added to the
> standard library. The barrier to semantic changes that break
> monkeypatching should be high.
The version 0.3 has a known bug: "len=chr; print(len('A'))" is
optimized, whereas it should not. It is now fixed in the new version
0.3.1.
>> * Call methods of builtin types if the object and arguments are constants.
>> Examples:
>>
>> - u"h\\xe9ho".encode("utf-8") => b"h\\xc3\\xa9ho"
>> - "python2.7".startswith("python") => True
>> - (32).bit_length() => 6
>> - float.fromhex("0x1.8p+0") => 1.5
>
> That last one isn't constant, it's a name lookup.
Well, yes, but in CPython, it is not possible to modify float.fromhex
attribute, nor unicode.encode.
>> - print(1.5) => print("1.5")
>
> The print example runs afoul of the general rule above: not in the
> standard library, because you're changing the values seen by a mocked
> version of print()
Ah yes, you found a bug. I forgot to disable this optimization by
default (when builtin_funcs feature is disabled). Fixed in 0.3.1.
>> - not(x in y) => x not in y
>
> This (and the "is") equivalent should be OK
Note: not(x is y) is also implemented.
>> * Loop: replace range() with xrange() on Python 2, and list with
>> tuple. Examples:
>>
>> - for x in range(n): ... => for x in xrange(n): ...
>> - for x in [1, 2, 3]: ... => for x in (1, 2, 3): ...
>
> Name lookup optimisations again: not in the standard library.
Correct, same issue than print(): forgot in version 0.3, and also
fixed in 0.3.1.
> def f(): return 1; yield
> if DEBUG: yield
> while 0: yield
I'm using the test suite of Python 2.7 and 3.3 using my optimizer. "if
0: yield" is a known captcha (there is a test) and it is handled
correctly by astoptimizer.
By the way, "return 3" is not removed in a generator because it must
raise a SyntaxError.
def f():
> ... if 0:
> ... global x
> ... return x
> ...
f()
> Traceback (most recent call last):
> File "", line 1, in
> File "", line 4, in f
> NameError: global name 'x' is not defined
Oh, nice catch. But it is possible to detect the usage of global and
nonlocal, and don't remove an expression if they are present. Fixed in
0.3.1.
Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Release of astoptimizer 0.3
>> * Call builtin functions if arguments are constants. Examples:
>>
>>- len("abc") => 3
>>- ord("A") => 65
>
> Does this mean transformation print("A") => None and output at compile time?
Only a subset of builtin functions are called at compile time, and not
with any constant value. See astoptimizer/config_builtin_funcs.py.
print() has other optimizations, but it is not called at compile time.
>> * Loop: replace range() with xrange() on Python 2,
>
> range(0, 10**10, 10**9)
Oh. I didn't know this difference between range and xrange. I should
add more checks on arguments.
>> * Evaluate unary and binary operators, subscript and comparaison if all
>>arguments are constants. Examples:
>>
>>- 1 + 2 * 3 => 7
>
> Does this mean 1/0 raises exception at compile time?
Such trap are handled carefully. There are many checks to decide if an
operator can be optimized or not.
Example: "\U0010"[0] is not optimized by default.
>> * Remove dead code. Examples:
>>
>>- def f(): return 1; return 2 => def f(): return 1
>>- if DEBUG: print("debug") => pass with DEBUG declared as False
>>- while 0: ... => pass
>
> As Nick said, it could change the semantics, if there were local variable
> assignment or yield in removed code.
yield is handled correctly, but local variable is a new problem.
I should find something to detect local variables. I already have code
to handle variable, but it not fully enabled because there are known
bugs.
> It would be good if the optimizer checked if the built-in names overridden
> in this module or function and disabled this dangerous optimization in such
> case.
Simple cases like "len=chr" or "from math import floor as int" are now
handled correctly in the new version 0.3.1.
> set([1, 2, 3]) => {1, 2, 3}
I don't see how to create literal set or frozen in AST.
set([...]) may be converted to set((...)) at least.
It would be nice to have a new ast.Lit() in Python 3.4: the idea was
already proposed (with a patch, 0_ast.patch) in the issue #11549.
> set([x for ...]) => {x for ...}
> dict([(k, v) for ...]) => {k: v for ...}
> dict((k, v) for ...) => {k: v for ...}
> ''.join([s for ...]) => ''.join(s for ...)
> a.extend([s for ...]) => a.extend(s for ...)
These optimizations look correct.
> (f(x) for x in a) => map(f, a)
> (x.y for x in a) => map(operator.attrgetter('y'), a)
> (x[0] for x in a) => map(operator.itemgetter(0), a)
> (2 * x for x in a) => map((2).__mul__, a)
> (x in b for x in a) => map(b.__contains__, a)
> map(lambda x: x.strip(), a) => (x.strip() for x in a)
Is it faster? :-)
> x in ['i', 'em', 'cite'] => x in {'i', 'em', 'cite'}
A list can contain non-hashable objects, whereas a set can not.
> x == 'i' or x == 'em' or x == 'cite'] => x in {'i', 'em', 'cite'}
You need to know the type of x. Depending on the type, x.__eq__ and
x.__contains__ may be completly different.
> a = []; for ...: a.append(x) => a = [x for ...]
> a = (); for ...: a.add(x) => a = {x for ...}
> a = {}; for ...: a[k] = v => a = {k: v for ...}
Yeah, the first example is already in the TODO list :-) But it's not
trivial. I prefer to start with unrolling loops :-)
> for ...: f.write(...) => __fwrite = f.write; for ...: __fwrite(...)
f.write lookup cannot be optimized.
> x = x + 1 => x += 1
> x = x + ' ' => x += ' '
I don't know if these optimizations are safe.
> x = x + [y] => x.append(y)
> x = x + [y, z] => x.extend([y, z])
It behaves differently if x is not a list, but str for example.
> 'x=%s' % repr(x) => 'x=%a' % (x,)
I don't understand this one.
> 'x=%s' % x + s => 'x=%s%s' % (x, s)
> x = x + ', [%s]' % y => x = '%s, [%s]' % (x, y)
Doesn't work if s type is not str.
> range(0, x) => range(x)
Is it faster?
> for i in range(len(a)): x = a[i] ... => for i, x in enumerate(a): ...
> i = 0; for x in a: i += 1 ... => for i, x in enumerate(a, 1): ...
> i = 1; for x in a: ... i += 1 => for i, x in enumerate(a, 1): ...
> s = 0; for ...: if ...: s += 1 => s = sum(1 for ... if ...)
These optimizations look interesting.
> while True: s = f.readline(); if not s: break; ... => for s in f: ...
Too much assumptions on f type.
> def f(x): ... len() ... => def f(x, __len = len): ... __len() ...
Ah yes, I read somewhere than locals are faster than builtins. But
this specific example is wrong, f(1, 2) must fail. It is possible to
write something like:
def create_f():
len = len
def f(x):
... len ...
return f
f = create_f(); del f
Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Release of astoptimizer 0.3
>>> - frozenset("ab") | frozenset("bc") => frozenset("abc")
>
> That's a name lookup, too.
Yes, and it is only optimized if the "builtin_funcs" feature is
enabled explictly.
> Except that evaluating something like '"abc" * constant' can eat up all
> memory, imagine this code:
>
> KILL_MEMORY = sys.argv[1] == 'NEVER PASS THIS VALUE'
>
> def test():
> if KILL_MEMORY:
> return 'abc' * 1
> else:
> return 'abc'
The optimizer always check the result of an optimization for
constraints. For example, an optimization is cancelled if the result
is not immutable. Integers must be in the range [-2^128; 2^128-1] by
default. Strings are limited to 4096 bytes/characters. Tuples are
limited to 20 items.
The compiler may waste some seconds to evaluate an expression and then
drop the result, but I don't really care right now. I prefer to spend
minutes to compile if the application is faster at runtime.
astoptimizer tries to avoid expensive operations when it knows that
the result will be too big. For your specific example, str * int is
*not* evulated if the result will be longer than the limit. Similar
sanity checks may be added later for pow(int, int) and
math.factorial(int) for example.
Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Release of astoptimizer 0.3
On 11/09/2012 22:46, Victor Stinner wrote:
2012/9/11 Nick Coghlan :
This is fine in an external project, but should never be added to the
standard library. The barrier to semantic changes that break
monkeypatching should be high.
The version 0.3 has a known bug: "len=chr; print(len('A'))" is
optimized, whereas it should not. It is now fixed in the new version
0.3.1.
* Call methods of builtin types if the object and arguments are constants.
Examples:
- u"h\\xe9ho".encode("utf-8") => b"h\\xc3\\xa9ho"
- "python2.7".startswith("python") => True
- (32).bit_length() => 6
- float.fromhex("0x1.8p+0") => 1.5
That last one isn't constant, it's a name lookup.
Well, yes, but in CPython, it is not possible to modify float.fromhex
attribute, nor unicode.encode.
It's possible to shadow 'float':
>>> float = "foo"
>>> float
'foo'
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Release of astoptimizer 0.3
Hello,
i am a longtime Reader of this list and this is the first time i a dare to
speak up.
Apology in advance for any noise, silly comments and not posting to
python-ideas ;).
Am 11.09.2012 12:41, schrieb Victor Stinner:
I plan to implement other optimizations like unrolling loop or convert
a loop to a list comprehension, see the TODO file.
Don't hesitate to propose more optimizations if you have some ideas ;-)
Well how about implementing guards like in pypy?
They are not optimizations by themselves but should enable a whole lot of
them.
I am not good at explaining so i've written this short example (python
2.7.3).
I hope it makes clear what i mean.
Thank you for reading
larudwer
#
# Example for implementing an guard
# by freezing global and local dictionary
# it is not an optimization by itself
# but should open the possibility for them.
#
class WorldChangedException(Exception): pass
class WorldShadowedException(Exception): pass
class frozendict(dict):
def __setitem__(self, key, value):
if key in globals():
if self == locals():
raise WorldShadowedException(key)
else:
raise WorldChangedException(key)
def put(string):
print string
g = frozendict({'put': put, 'foo': 2})
l = frozendict({'bar': 'work fast'})
#
# This is just an poor example.
# astoptimizer should be able to generate
# much better code
#
f = compile("""
put( "trying fast path")
put(bar)
put = 27
""", "", "exec")
def test():
global put # Workaround for UnboundLocalError
put("trying slow path")
put = 27
#
# Guard: since the world is frozen, nothing can change.
# if the fast path succeeds we take the result, else
# we use the slow path.
#
try:
exec(f, g, l)
except WorldChangedException, e:
print "WorldChangedException Caught", e
test()
except WorldShadowedException, e:
print "WorldShadowedException Caught", e
test()
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 15906; regression in argparse in Python 3.3, 3.2, and 2.7
On Sep 11, 2012, at 05:30 PM, Terry Reedy wrote: >> And yet, that's how it works in 2.7, 3.2, and 3.3.1. > >In all 2.7 or 3.2? It will be broken in 2.7.4 and 3.2.4, but the Ubuntu packages in 12.10 are currently affected because they pulled in some Mercurial updates which included these unreleased changes. That's where it was first noticed. >How about a third rule which amounts to the second *plus* (maybe) the first: >apply the conversion when it fulfills the purpose of having type conversions, >which is to make sure that the caller get objects of the desired class. Also, >only add necessary restrictions to argparse users. I've tried the various suggestions out, and I think from a practical point of view, a fix for the regression in the 2.7, 3.2 and 3.3 branches should be to apply the one line check for action being a _StoreAction instance. This seems to be the easiest way to preserve the current behavior (the fix for issues #12776 and #11839) and fix issue #15906. I'll update the issue and apply this change to the three branches. Georg can ignore the change for 3.3.0. -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Release of astoptimizer 0.3
> i am a longtime Reader of this list and this is the first time i a dare to > speak up. > Apology in advance for any noise, silly comments and not posting to > python-ideas ;). Welcome! > Well how about implementing guards like in pypy? Guards would allow to generate specialized functions without a JIT. Dummy example: def func(arg): arg *= 1 arg += 0 return arg * 16 Can be optimized to: def func(arg): if type(arg) is int: # fast path return arg << 4 else: # slow path arg *= 1 arg += 0 return arg * 16 I prefer a check before executing the code, rather than an exception and reexecute the same code twice. Specializing all functions would waste a lot of memory, so it should only be applied on some very specific cases... It can also be slower if the guard check is slower than the function body! But I bet that guards would help to enable more aggressive optimizations, or at least make some optimizations safe. Dave Malcolm wrote a patch modifying eval.c to support specialized functions. See the http://bugs.python.org/issue10399 I don't know yet what is the best approach for CPython. -- For the specific case of builtin functions and types, I made two changes in Python 3.3: * the type of the builtins mapping can now be any mapping (Python <= 3.2 requires the dict type, dict subtypes are disallowed) * "new" types.MappingProxyType type to create a read-only proxy for a mapping (see also the rejected PEP 416) We may combine these two changes to use a read-only mapping for builtins. It would at least help to ensure that an application does not monkeypatch builtin functions/types. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Release of astoptimizer 0.3
2012/9/11 Guido van Rossum :
> FWIW, I expect that there are few places where len()
> is actually used.
I found one revelant example in the stdlib.
if path.endswith('.dylib'):
yield path[:-len('.dylib')] + suffix + '.dylib'
else:
yield path + suffix
Cool, I'm not the only developer too lazy to copy/paste len('suffix')
in a Python interpreter and then copy/paste the result in my code :-)
"if text.endswith('suffix'): ... text[:-len('suffix')] ..." is a
common pattern in my code.
> However I expect it to be quite common for ord(),
> where the same reasoning applies.
Projects using the same code base for Python 2 and Python 3 contain a
lot of inefficient code. For example, using the six library, a simple
Unicode literal strings becomes a function code: u('unicode').
I expect that astoptimizer will be able to remove (or at least
reduce!) the overhead of the six library and all checks on the Python
version ("if PYTHON3: ... else: ...").
Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Release of astoptimizer 0.3
On Tue, Sep 11, 2012 at 5:40 PM, Victor Stinner
wrote:
> 2012/9/11 Guido van Rossum :
>> FWIW, I expect that there are few places where len()
>> is actually used.
>
> I found one revelant example in the stdlib.
>
> if path.endswith('.dylib'):
> yield path[:-len('.dylib')] + suffix + '.dylib'
> else:
> yield path + suffix
>
> Cool, I'm not the only developer too lazy to copy/paste len('suffix')
> in a Python interpreter and then copy/paste the result in my code :-)
>
> "if text.endswith('suffix'): ... text[:-len('suffix')] ..." is a
> common pattern in my code.
Ok, that's a good use case.
>> However I expect it to be quite common for ord(),
>> where the same reasoning applies.
>
> Projects using the same code base for Python 2 and Python 3 contain a
> lot of inefficient code. For example, using the six library, a simple
> Unicode literal strings becomes a function code: u('unicode').
But are you able to do enough static analysis to feel comfortable that
this is the right u() function? IIRC you said earlier that you're not
even capable of recognizing "len = ord; print(len('a'))" -- if that is
really true, I'm very worried about your optimizer's capacity for
breaking code. I'm not talking about "behind-their-back" changes to
__builtins__ or patching of the module globals. I'm talking about
detecting straightforward definitions that override the identifiers
you are replacing.
> I expect that astoptimizer will be able to remove (or at least
> reduce!) the overhead of the six library and all checks on the Python
> version ("if PYTHON3: ... else: ...").
Hm. Wouldn't it be just as easy to run a source-to-source translator
to remove six artefacts instead of an ast optimizer? Surely some
convention could be adopted that is easy to use, and the tool to do
the translation could be a lot simpler than an ast optimizer.
Sorry for being skeptical, but I'm not excited about advertising this
as a general optimization tool unless you can make it a lot safer.
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (merge 3.2 -> default): - Issue #15906: Fix a regression in argparse caused by the preceding change,
On Tue, Sep 11, 2012 at 9:13 PM, barry.warsaw
wrote:
> http://hg.python.org/cpython/rev/25e41fdc4e60
> changeset: 79001:25e41fdc4e60
> parent: 78998:6fea947edead
> parent: 79000:bc342cd7ed96
> user:Barry Warsaw
> date:Wed Sep 12 00:12:29 2012 -0400
> summary:
> - Issue #15906: Fix a regression in argparse caused by the preceding change,
> when action='append', type='str' and default=[].
I didn't have time to respond Barry's e-mail from four hours ago
before this was committed. I think this change may be problematic.
At the least, I think people should have an opportunity to air their
specific concerns and talk through the implications.
Also, from the discussion it seemed like the sentiment was leaning
towards a different approach for the fix.
I made a comment on the issue with some more extended remarks:
http://bugs.python.org/msg170351
--Chris
>
> files:
> Lib/argparse.py | 1 +
> Lib/test/test_argparse.py | 10 ++
> Misc/NEWS | 3 +++
> 3 files changed, 14 insertions(+), 0 deletions(-)
>
>
> diff --git a/Lib/argparse.py b/Lib/argparse.py
> --- a/Lib/argparse.py
> +++ b/Lib/argparse.py
> @@ -1961,6 +1961,7 @@
> # twice (which may fail) if the argument was given, but
> # only if it was defined already in the namespace
> if (action.default is not None and
> +isinstance(action, _StoreAction) and
> hasattr(namespace, action.dest) and
> action.default is getattr(namespace, action.dest)):
> setattr(namespace, action.dest,
> diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py
> --- a/Lib/test/test_argparse.py
> +++ b/Lib/test/test_argparse.py
> @@ -4607,6 +4607,16 @@
> args = parser.parse_args([])
> self.assertEqual(NS(foo='foo_converted'), args)
>
> +def test_issue_15906(self):
> +# Issue #15906: When action='append', type=str, default=[] are
> +# providing, the dest value was the string representation "[]" when
> it
> +# should have been an empty list.
> +parser = argparse.ArgumentParser()
> +parser.add_argument('--test', dest='test', type=str,
> +default=[], action='append')
> +args = parser.parse_args([])
> +self.assertEqual(args.test, [])
> +
> # ==
> # parse_known_args tests
> # ==
> diff --git a/Misc/NEWS b/Misc/NEWS
> --- a/Misc/NEWS
> +++ b/Misc/NEWS
> @@ -56,6 +56,9 @@
>especially problematic for the FileType type, as a default file would
> always
>be opened, even if a file argument was specified on the command line.
>
> +- Issue #15906: Fix a regression in argparse caused by the preceding change,
> + when action='append', type='str' and default=[].
> +
> Extension Modules
> -
>
>
> --
> Repository URL: http://hg.python.org/cpython
>
> ___
> Python-checkins mailing list
> python-check...@python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
