Re: [Python-Dev] AST optimizer implemented in Python

On 08/14/2012 03:32 PM, Victor Stinner wrote: I had the idea (perhaps not an original one) that peephole optimization would be much better done in python than in C. The C code is clunky and unwieldly, wheras python would be much better suited, being able to use nifty regexes and the like.

Re: [Python-Dev] AST optimizer implemented in Python

> -Original Message- > From: Victor Stinner [mailto:victor.stin...@gmail.com] > Sent: 14. ágúst 2012 13:32 > To: Kristján Valur Jónsson > Cc: Python Dev > Subject: Re: [Python-Dev] AST optimizer implemented in Python > > The problem is, there exists only b

Re: [Python-Dev] AST optimizer implemented in Python

2012/8/14 Kristján Valur Jónsson : >> I moved the script to a new dedicated project on Bitbucket: >> https://bitbucket.org/haypo/astoptimizer >> >> Join the project if you want to help me to build a better optimizer! >> >> It now works on Python 2.5-3.3. > > I had the idea (perhaps not an original

Re: [Python-Dev] AST optimizer implemented in Python

> -Original Message- > I moved the script to a new dedicated project on Bitbucket: > https://bitbucket.org/haypo/astoptimizer > > Join the project if you want to help me to build a better optimizer! > > It now works on Python 2.5-3.3. I had the idea (perhaps not an original one) that p

Re: [Python-Dev] AST optimizer implemented in Python

Implementations are currently required to *have* an AST (or else declare non compliance with that particular flag to compile). They're definitely not required to have the *same* AST, thus all AST manipulation, like bytecode manipulation, is necessarily implementation dependent. We don't even guara

Re: [Python-Dev] AST optimizer implemented in Python

On 8/13/2012 10:45 AM, Guido van Rossum wrote: Not so fast. If you make this a language feature you force all Python implementations to support an identical AST API. That's a big step. I have been wondering about this. One could think from the manuals that we are there already. From the beginn

Re: [Python-Dev] AST optimizer implemented in Python

Not so fast. If you make this a language feature you force all Python implementations to support an identical AST API. That's a big step. Not that AST manipulation isn't cool -- but I'd like to warn against over-enthusiasm that might backfire on the language (or its community) as a whole. --Guido

Re: [Python-Dev] AST optimizer implemented in Python

Brett Cannon wrote: On Sat, Aug 11, 2012 at 8:16 PM, Eric Snow > wrote: On Sat, Aug 11, 2012 at 6:03 PM, Brett Cannon mailto:br...@python.org>> wrote: > It would also be very easy to expand importlib.abc.SourceLoader to add a > method whic

Re: [Python-Dev] AST optimizer implemented in Python

On Mon, Aug 13, 2012 at 7:05 AM, Meador Inge wrote: > This is an interesting project and I would happily volunteer to help flesh > out the details of a prototype and working on a PEP. Also, if there are possible AST improvements that would help in 3.4+, that option *is* on the table (this was an

Re: [Python-Dev] AST optimizer implemented in Python

On Sun, Aug 12, 2012 at 6:07 PM, Eric Snow wrote: > On Aug 12, 2012 12:56 PM, "PJ Eby" wrote: > > I'm not sure if this is directly related or not, but making this > > mechanism support custom compilation for new filename suffixes would > > be nice, especially for various e.g. HTML/XML templating

Re: [Python-Dev] AST optimizer implemented in Python

On Aug 12, 2012 12:56 PM, "PJ Eby" wrote: > I'm not sure if this is directly related or not, but making this > mechanism support custom compilation for new filename suffixes would > be nice, especially for various e.g. HTML/XML templating systems that > compile to Python or bytecode. > > Specifica

Re: [Python-Dev] AST optimizer implemented in Python

On Sat, Aug 11, 2012 at 1:30 PM, Victor Stinner wrote: > Hi, > > I started to implement an AST optimizer in Python. It's easy to create > a new AST tree, so I'm surprised that I didn't find any existing > project. > > https://bitbucket.org/haypo/misc/src/tip/python/ast_optimizer.py Very cool. >

Re: [Python-Dev] AST optimizer implemented in Python

> I started to implement an AST optimizer in Python. It's easy to create > a new AST tree, so I'm surprised that I didn't find any existing > project. I done more research. I found the AST optimizer of PyPy, which implements basic optimizations: https://bitbucket.org/pypy/pypy/src/default/pypy/int

Re: [Python-Dev] AST optimizer implemented in Python

I'm not sure if this is directly related or not, but making this mechanism support custom compilation for new filename suffixes would be nice, especially for various e.g. HTML/XML templating systems that compile to Python or bytecode. Specifically, having a way to add a new source suffix (e.g. ".

Re: [Python-Dev] AST optimizer implemented in Python

On Sat, Aug 11, 2012 at 8:03 PM, Brett Cannon wrote: > > It would also be very easy to expand importlib.abc.SourceLoader to add a > method which is called with source and returns the bytecode to be written > out which people could override with AST optimizations before sending the > bytecode back.

Re: [Python-Dev] AST optimizer implemented in Python

Stefan Behnel, 12.08.2012 08:00: > Victor Stinner, 11.08.2012 20:30: >> I started to implement an AST optimizer in Python. It's easy to create >> a new AST tree, so I'm surprised that I didn't find any existing >> project. >> >> https://bitbucket.org/haypo/misc/src/tip/python/ast_optimizer.py > >

Re: [Python-Dev] AST optimizer implemented in Python

"Martin v. Löwis", 11.08.2012 23:27: >>> +1 We should add some form of setastoptimizer API in 3.4. Please start a >>> PEP for this. It would be nice to include the ability to properly cache >>> the >>> ast optimizer output so that it does not have to run every time (in pyc >>> files or similar,

Re: [Python-Dev] AST optimizer implemented in Python

Victor Stinner, 11.08.2012 20:30: > I started to implement an AST optimizer in Python. It's easy to create > a new AST tree, so I'm surprised that I didn't find any existing > project. > > https://bitbucket.org/haypo/misc/src/tip/python/ast_optimizer.py Since you're about to do pretty much the sa

Re: [Python-Dev] AST optimizer implemented in Python

Stefan Behnel, 12.08.2012 06:42: > Chris Angelico, 12.08.2012 01:22: >>> Other idea to improve this optimizer: >>> - move invariant out of loops. Example: "x=[]; for i in range(10): >>> x.append(i)" => "x=[]; x_append=x.append; for i in range(10): >>> x_append(i)". Require to infer the type of var

Re: [Python-Dev] AST optimizer implemented in Python

Chris Angelico, 12.08.2012 01:22: >> Other idea to improve this optimizer: >> - move invariant out of loops. Example: "x=[]; for i in range(10): >> x.append(i)" => "x=[]; x_append=x.append; for i in range(10): >> x_append(i)". Require to infer the type of variables. > > But this is risky. It's th

Re: [Python-Dev] AST optimizer implemented in Python

On Sun, Aug 12, 2012 at 11:17 AM, Victor Stinner wrote: > The idea would be to allow the developer to specify explicitly what he > wants to optimize. I'm using a configuration class with a list of what > can be optimized (ex: len(int)), but it can be changed to something > different later. > > It

Re: [Python-Dev] AST optimizer implemented in Python

>> Other idea to improve this optimizer: >> - move invariant out of loops. Example: "x=[]; for i in range(10): >> x.append(i)" => "x=[]; x_append=x.append; for i in range(10): >> x_append(i)". Require to infer the type of variables. > > But this is risky. It's theoretically possible for x.append t

Re: [Python-Dev] AST optimizer implemented in Python

On Sat, Aug 11, 2012 at 8:16 PM, Eric Snow wrote: > On Sat, Aug 11, 2012 at 6:03 PM, Brett Cannon wrote: > > It would also be very easy to expand importlib.abc.SourceLoader to add a > > method which is called with source and returns the bytecode to be written > > out > > Yes, please. Not having

Re: [Python-Dev] AST optimizer implemented in Python

On Sat, Aug 11, 2012 at 6:03 PM, Brett Cannon wrote: > It would also be very easy to expand importlib.abc.SourceLoader to add a > method which is called with source and returns the bytecode to be written > out Yes, please. Not having to hack around this would be nice. > which people could overr

Re: [Python-Dev] AST optimizer implemented in Python

On Sat, Aug 11, 2012 at 2:30 PM, Victor Stinner wrote: > Hi, > > I started to implement an AST optimizer in Python. It's easy to create > a new AST tree, so I'm surprised that I didn't find any existing > project. > > https://bitbucket.org/haypo/misc/src/tip/python/ast_optimizer.py > > To test its

Re: [Python-Dev] AST optimizer implemented in Python

On Sun, Aug 12, 2012 at 4:30 AM, Victor Stinner wrote: > I started to implement an AST optimizer in Python. It's easy to create > a new AST tree, so I'm surprised that I didn't find any existing > project. Very nice idea! > Other idea to improve this optimizer: > - move invariant out of loops.

Re: [Python-Dev] AST optimizer implemented in Python

On Sat, Aug 11, 2012 at 3:27 PM, "Martin v. Löwis" wrote: > I think you misunderstood. What gps is concerned about (IIUC) that some > people add ast optimizers in some run of Python, but other AST optimizers in > a different run. Then, if you use a Python byte code > file, you should be able to fi

Re: [Python-Dev] AST optimizer implemented in Python

On 8/11/2012 2:30 PM, Victor Stinner wrote: Hi, I started to implement an AST optimizer in Python. It's easy to create a new AST tree, so I'm surprised that I didn't find any existing project. https://bitbucket.org/haypo/misc/src/tip/python/ast_optimizer.py To test its peephole optimizations (

Re: [Python-Dev] AST optimizer implemented in Python

+1 We should add some form of setastoptimizer API in 3.4. Please start a PEP for this. It would be nice to include the ability to properly cache the ast optimizer output so that it does not have to run every time (in pyc files or similar, etc) but can verify that it is the specific ast optimi

Re: [Python-Dev] AST optimizer implemented in Python

On Sat, Aug 11, 2012 at 11:47 AM, Gregory P. Smith wrote: > > On Sat, Aug 11, 2012 at 11:30 AM, Victor Stinner > wrote: >> >> Hi, >> >> I started to implement an AST optimizer in Python. It's easy to create >> a new AST tree, so I'm surprised that I didn't find any existing >> project. >> >> http

Re: [Python-Dev] AST optimizer implemented in Python

On Sat, Aug 11, 2012 at 11:30 AM, Victor Stinner wrote: > Hi, > > I started to implement an AST optimizer in Python. It's easy to create > a new AST tree, so I'm surprised that I didn't find any existing > project. > > https://bitbucket.org/haypo/misc/src/tip/python/ast_optimizer.py Neat! > >

[Python-Dev] AST optimizer implemented in Python

Hi, I started to implement an AST optimizer in Python. It's easy to create a new AST tree, so I'm surprised that I didn't find any existing project. https://bitbucket.org/haypo/misc/src/tip/python/ast_optimizer.py To test its peephole optimizations (by checking manually its final bytecode), I wr