Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Nick Coghlan
Brett Cannon wrote: I think the question the PEP is posing is will python-dev accept the Unladen Swallow work to add an LLVM JIT into the py3k trunk? My personal answer is yes as it's a nice speed improvement for code that runs more than once, all while being an optional enhancement for those

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Nick Coghlan
Barry Warsaw wrote: On Jan 21, 2010, at 04:07 PM, Jeffrey Yasskin wrote: I could imagine a problem if Python+LLVM link in one libstdc++, and an extension module links in a different one, even if no C++ objects are passed across the boundary. Does that cause problems in practice? We'd have

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread sstein...@gmail.com
On Jan 21, 2010, at 11:32 PM, Reid Kleckner wrote: On Thu, Jan 21, 2010 at 5:07 PM, David Malcolm dmalc...@redhat.com wrote: To what extent would it be possible to use (conditionally) use full ahead-of-time compilation as well as JIT? It would be possible to do this, but it doesn't have

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread sstein...@gmail.com
On Jan 21, 2010, at 11:32 PM, Chris Bergstresser wrote: On Thu, Jan 21, 2010 at 9:49 PM, Tres Seaver tsea...@palladion.com wrote: Generally, that's not going to be the case. But the broader point--that you've no longer got an especially good idea of what's taking time to run in your

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread skip
Reid You could do a static compilation of all code objects in a .pyc to Reid LLVM IR and compile that to a .so that you load at runtime, but it Reid just eliminates the interpreter overhead. OTOH, it would solve the problem some people have distributing their proprietary apps as .py

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Michael Foord
On 22/01/2010 11:45, sstein...@gmail.com wrote: On Jan 21, 2010, at 11:32 PM, Reid Kleckner wrote: On Thu, Jan 21, 2010 at 5:07 PM, David Malcolmdmalc...@redhat.com wrote: To what extent would it be possible to use (conditionally) use full ahead-of-time compilation as well as JIT?

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread John Arbash Meinel
sstein...@gmail.com wrote: On Jan 21, 2010, at 11:32 PM, Chris Bergstresser wrote: On Thu, Jan 21, 2010 at 9:49 PM, Tres Seaver tsea...@palladion.com wrote: Generally, that's not going to be the case. But the broader point--that you've no longer got an especially good idea of what's

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Stefan Behnel
John Arbash Meinel, 22.01.2010 15:56: For example, I've never seen a Pyrex __init__ function show up in timing, the time spent always gets assigned to the calling function. So if I want to see it, I set up a 'create_foo(*args, **kwargs)' function that just does return Foo(*args, **kwargs).

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Stefan Behnel
Collin Winter, 21.01.2010 21:14: Cython is a super-set of Python, and files annotated for maximum Cython performance are no longer valid Python code, and will not run on any other Python implementation. Except when you use plain Python annotations, in which case you get the speed of Cython for

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Tony Nelson
On 10-01-22 02:53:21, Collin Winter wrote: On Thu, Jan 21, 2010 at 11:37 PM, Glyph Lefkowitz gl...@twistedmatrix.com wrote: On Jan 21, 2010, at 6:48 PM, Collin Winter wrote: ... There's been a recent thread on our mailing list about a patch that dramatically reduces the memory footprint

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Collin Winter
Hey Tony, On Fri, Jan 22, 2010 at 10:11 AM, Tony Nelson tonynel...@georgeanelson.com wrote: On 10-01-22 02:53:21, Collin Winter wrote: On Thu, Jan 21, 2010 at 11:37 PM, Glyph Lefkowitz gl...@twistedmatrix.com wrote: On Jan 21, 2010, at 6:48 PM, Collin Winter wrote:  ... There's been a

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Collin Winter
Hey Jake, On Thu, Jan 21, 2010 at 10:48 AM, Jake McGuire mcgu...@google.com wrote: On Thu, Jan 21, 2010 at 10:19 AM, Reid Kleckner r...@mit.edu wrote: On Thu, Jan 21, 2010 at 12:27 PM, Jake McGuire mcgu...@google.com wrote: On Wed, Jan 20, 2010 at 2:27 PM, Collin Winter collinwin...@google.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Alex Gaynor
On Fri, Jan 22, 2010 at 1:07 PM, Collin Winter collinwin...@google.com wrote: Hey Jake, On Thu, Jan 21, 2010 at 10:48 AM, Jake McGuire mcgu...@google.com wrote: On Thu, Jan 21, 2010 at 10:19 AM, Reid Kleckner r...@mit.edu wrote: On Thu, Jan 21, 2010 at 12:27 PM, Jake McGuire

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread John Arbash Meinel
Collin Winter wrote: Hey Jake, ... Hmm. So cProfile doesn't break, but it causes code to run under a completely different execution model so the numbers it produces are not connected to reality? We've found the call graph and associated execution time information from cProfile to be

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Martin v. Löwis
I'm not an expert here, but this is why we've tried to avoid exposing C++ functions into the public API. As long as the public API stays C, we only require that LLVM and Python are compiled with the same compiler, right? Possibly not. You may end up loading two different C++ runtimes into

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread James Abbatiello
On Thu, Jan 21, 2010 at 5:24 PM, Collin Winter collinwin...@google.com wrote: On Thu, Jan 21, 2010 at 12:56 PM, Paul Moore p.f.mo...@gmail.com wrote: Windows compatibility is a big deal to me. And IMHO, it's a great strength of Python at the moment that it has solid Windows support. I would be

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Martin v. Löwis
But I believe they would be the same set of problems that PEP 384 addresses for cross-version compatibility. Because the linker only sees the C symbols, it shouldn't be possible for the C++ specific parts of the runtimes to interfere with each other. Unfortunately, on systems with a global

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Martin v. Löwis
It is provided as a separate tool (and often invoked by application installers) rather than allowing the native code to be distributed because the results can be system specific. Actually, they have now changed the implementation: it is now a service which will do the ngen compilation in

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Michael Foord
On 22/01/2010 22:43, Martin v. Löwis wrote: It is provided as a separate tool (and often invoked by application installers) rather than allowing the native code to be distributed because the results can be system specific. Actually, they have now changed the implementation: it is now a

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Michael Foord
On 21/01/2010 06:54, Gregory P. Smith wrote: +1 My biggest concern is memory usage but it sounds like addressing that is already in your mind. I don't so much mind an additional up front constant and per-line-of-code hit for instrumentation but leaks are unacceptable. Any instrumentation

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Barry Warsaw
On Jan 20, 2010, at 11:05 PM, Jack Diederich wrote: Does disabling the LLVM change binary compatibility between modules targeted at the same version? At tonight's Boston PIG we had some binary package maintainers but most people (including myself) only cared about source compatibility.I

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Barry Warsaw
On Jan 20, 2010, at 10:09 PM, Benjamin Peterson wrote: Does disabling the LLVM change binary compatibility between modules targeted at the same version?  At tonight's Boston PIG we had some binary package maintainers but most people (including myself) only cared about source compatibility.    

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Antoine Pitrou
Hello, If we have done any original work, it is by accident. :-) The increased memory usage comes from a) LLVM code generation, analysis and optimization libraries; b) native code; c) memory usage issues or leaks in LLVM; d) data structures needed to optimize and generate machine code; e)

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Floris Bruynooghe
On Wed, Jan 20, 2010 at 02:27:05PM -0800, Collin Winter wrote: Platform Support [...] In order to support hardware and software platforms where LLVM's JIT does not work, Unladen Swallow provides a ``./configure --without-llvm`` option. This flag carves out any part of Unladen

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Glyph Lefkowitz
On Jan 21, 2010, at 7:25 AM, Antoine Pitrou wrote: We seek guidance from the community on an acceptable level of increased memory usage. I think a 10-20% increase would be acceptable. It would be hard for me to put an exact number on what I would find acceptable, but I was really hoping

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Collin Winter
Hi Dirkjan, On Wed, Jan 20, 2010 at 10:55 PM, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Thu, Jan 21, 2010 at 02:56, Collin Winter collinwin...@google.com wrote: Agreed. We are actively working to improve the startup time penalty. We're interested in getting guidance from the CPython

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread A.M. Kuchling
On Wed, Jan 20, 2010 at 10:54:11PM -0800, Gregory P. Smith wrote: I think having a run time flag (or environment variable for those who like that) to disable the use of JIT at python3 execution time would be a good idea. Another approach could be to compile two binaries, 'python' which is

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread John Arbash Meinel
Collin Winter wrote: Hi Dirkjan, On Wed, Jan 20, 2010 at 10:55 PM, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Thu, Jan 21, 2010 at 02:56, Collin Winter collinwin...@google.com wrote: Agreed. We are actively working to improve the startup time penalty. We're interested in getting guidance

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Reid Kleckner
On Thu, Jan 21, 2010 at 7:25 AM, Antoine Pitrou solip...@pitrou.net wrote: 32-bit; gcc 4.0.3 +-+---+---+--+ | Binary size | CPython 2.6.4 | CPython 3.1.1 | Unladen Swallow r988 |

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Lennart Regebro
My 2 cents: For environments like Plone that has a lot of code and uses a lot of memory the current speed increases is probaly not worth it if the memory increases as much as it does now. It would for almost all cases mean you need to skimp on data caching instead, which would probably slow down

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Hanno Schlichting
Hi, I'm a relative outsider to core development (I'm just a Plone release manager), but'll allow myself a couple of questions. Feel free to ignore them, if you think they are not relevant at this point :-) I'd note that I'm generally enthusiastic and supportive of the proposal :) As a data point,

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Antoine Pitrou
Reid Kleckner rnk at mit.edu writes: This is positively humongous. Is there any way to shrink these numbers dramatically (I'm talking about the release builds)? Large executables or libraries may make people anxious about the interpreter's memory efficiency; and they will be a nuisance

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Reid Kleckner
On Thu, Jan 21, 2010 at 9:35 AM, Floris Bruynooghe floris.bruynoo...@gmail.com wrote: I just compiled with the --without-llvm option and see that the binary, while only an acceptable 4.1M, still links with libstdc++.  Is it possible to completely get rid of the C++ dependency if this option is

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Reid Kleckner
On Thu, Jan 21, 2010 at 12:27 PM, Jake McGuire mcgu...@google.com wrote: On Wed, Jan 20, 2010 at 2:27 PM, Collin Winter collinwin...@google.com wrote: Profiling - Unladen Swallow integrates with oProfile 0.9.4 and newer [#oprofile]_ to support assembly-level profiling on Linux

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Jake McGuire
On Wed, Jan 20, 2010 at 2:27 PM, Collin Winter collinwin...@google.com wrote: Profiling - Unladen Swallow integrates with oProfile 0.9.4 and newer [#oprofile]_ to support assembly-level profiling on Linux systems. This means that oProfile will correctly symbolize JIT-compiled

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Jake McGuire
On Thu, Jan 21, 2010 at 10:19 AM, Reid Kleckner r...@mit.edu wrote: On Thu, Jan 21, 2010 at 12:27 PM, Jake McGuire mcgu...@google.com wrote: On Wed, Jan 20, 2010 at 2:27 PM, Collin Winter collinwin...@google.com wrote: Profiling - Unladen Swallow integrates with oProfile 0.9.4 and

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Dirkjan Ochtman
On Thu, Jan 21, 2010 at 18:32, Collin Winter collinwin...@google.com wrote: I added startup benchmarks for Mercurial and Bazaar yesterday (http://code.google.com/p/unladen-swallow/source/detail?r=1019) so we can use them as more macro-ish benchmarks, rather than merely starting the CPython

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Toshio Kuratomi
On Thu, Jan 21, 2010 at 12:25:59PM +, Antoine Pitrou wrote: We seek guidance from the community on an acceptable level of increased memory usage. I think a 10-20% increase would be acceptable. I'm just a user of the core interpreter but the bottleneck in using python in my environment

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Antoine Pitrou
Dirkjan Ochtman dirkjan at ochtman.nl writes: What I personally would consider interesting for the PEP is a (not too big) section evaluating where other Python-performance efforts are at. E.g. does it make sense to propose a u-s merge now when, by the time 3.3 (or whatever) is released,

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Toshio Kuratomi
On Thu, Jan 21, 2010 at 09:32:23AM -0800, Collin Winter wrote: Hi Dirkjan, On Wed, Jan 20, 2010 at 10:55 PM, Dirkjan Ochtman dirk...@ochtman.nl wrote: For some apps (like Mercurial, which I happen to sometimes hack on), increased startup time really sucks. We already have our demandimport

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Collin Winter
Hey Dirkjan, On Thu, Jan 21, 2010 at 11:16 AM, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Thu, Jan 21, 2010 at 18:32, Collin Winter collinwin...@google.com wrote: I added startup benchmarks for Mercurial and Bazaar yesterday (http://code.google.com/p/unladen-swallow/source/detail?r=1019) so

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Collin Winter
Hey Greg, On Wed, Jan 20, 2010 at 10:54 PM, Gregory P. Smith g...@krypto.org wrote: +1 My biggest concern is memory usage but it sounds like addressing that is already in your mind.  I don't so much mind an additional up front constant and per-line-of-code hit for instrumentation but leaks

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Reid Kleckner
On Thu, Jan 21, 2010 at 3:14 PM, Collin Winter collinwin...@google.com wrote: P.S. Is there any chance of LLVM doing something like tracing JITs? Those seem somewhat more promising to me (even though I understand they're quite hard in the face of Python features like stack frames). Yes, you

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Collin Winter
Hey Barry, On Thu, Jan 21, 2010 at 3:34 AM, Barry Warsaw ba...@python.org wrote: On Jan 20, 2010, at 11:05 PM, Jack Diederich wrote: Does disabling the LLVM change binary compatibility between modules targeted at the same version?  At tonight's Boston PIG we had some binary package maintainers

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Paul Moore
2010/1/20 Collin Winter collinwin...@google.com: Hello python-dev, I've just committed the initial draft of PEP 3146, proposing to merge Unladen Swallow into CPython's source tree and roadmap. The initial draft is included below. I've also uploaded the PEP to Rietveld at

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Steve Steiner (listsin)
On Jan 21, 2010, at 3:20 PM, Collin Winter wrote: Hey Greg, On Wed, Jan 20, 2010 at 10:54 PM, Gregory P. Smith g...@krypto.org wrote: +1 My biggest concern is memory usage but it sounds like addressing that is already in your mind. I don't so much mind an additional up front constant

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Paul Moore
2010/1/21 Paul Moore p.f.mo...@gmail.com: 2010/1/20 Collin Winter collinwin...@google.com: Hello python-dev, [...] We're looking forward to discussing this with everyone. I'll comment on a number of points here - I've read the thread but it'd get too complex trying to quote specific items.

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Alex Gaynor
On Thu, Jan 21, 2010 at 3:00 PM, Steve Steiner (listsin) list...@integrateddevcorp.com wrote: On Jan 21, 2010, at 3:20 PM, Collin Winter wrote: Hey Greg, On Wed, Jan 20, 2010 at 10:54 PM, Gregory P. Smith g...@krypto.org wrote: +1 My biggest concern is memory usage but it sounds like

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Martin v. Löwis
1. What are the implications for PEP 384 (Stable ABI) if U-S is added? I haven't studied U-S yet, but I'd hope that there might be no implications. No basic API should change, and everything the JIT compiler does should be well shielded from the object API (which PEP 384 deals with). 2. What

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Martin v. Löwis
Reid Kleckner wrote: On Thu, Jan 21, 2010 at 7:25 AM, Antoine Pitrou solip...@pitrou.net wrote: 32-bit; gcc 4.0.3 +-+---+---+--+ | Binary size | CPython 2.6.4 | CPython 3.1.1 | Unladen Swallow r988 |

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Martin v. Löwis
How large is the LLVM shared library? One surprising data point is that the binary is much larger than some of the memory footprint measurements given in the PEP. Could it be that you need to strip the binary, or otherwise remove unneeded debug information? Regards, Martin

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Martin v. Löwis
There is freeze: http://wiki.python.org/moin/Freeze Which IIRC Robert Collins tried in the past, but didn't see a huge gain. It at least tries to compile all of your python files to C files and then build an executable out of that. to C files is a bit of an exaggeration, though. It embeds

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Dirkjan Ochtman
Hey Collin, Thanks for the good answers so far! On Thu, Jan 21, 2010 at 21:14, Collin Winter collinwin...@google.com wrote: We used to run the Mercurial correctness tests at every revision, but they were incredibly slow and a bit flaky under CPython 2.6. Bazaar's tests were faster, but were

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Martin v. Löwis
On Wed, Jan 20, 2010 at 10:54:11PM -0800, Gregory P. Smith wrote: I think having a run time flag (or environment variable for those who like that) to disable the use of JIT at python3 execution time would be a good idea. Another approach could be to compile two binaries, 'python' which is

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread John Arbash Meinel
Martin v. Löwis wrote: There is freeze: http://wiki.python.org/moin/Freeze Which IIRC Robert Collins tried in the past, but didn't see a huge gain. It at least tries to compile all of your python files to C files and then build an executable out of that. to C files is a bit of an

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Martin v. Löwis
Sure, though it sounds quite similar to what they were mentioning with: the creation of hermetic Python binaries, with all necessary modules preloaded I wondered whethe this hermetic binary would also include the result of JIT compilation - if so, it would go beyond freeze, and contain actual

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Collin Winter
On Wed, Jan 20, 2010 at 2:27 PM, Collin Winter collinwin...@google.com wrote: [snip] Incremental builds, however, are significantly slower. The table below shows incremental rebuild times after touching ``Objects/listobject.c``.

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Reid Kleckner
On Thu, Jan 21, 2010 at 4:34 PM, Martin v. Löwis mar...@v.loewis.de wrote: How large is the LLVM shared library? One surprising data point is that the binary is much larger than some of the memory footprint measurements given in the PEP. Could it be that you need to strip the binary, or

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread David Malcolm
On Wed, 2010-01-20 at 14:27 -0800, Collin Winter wrote: [snip] At a high level, the Unladen Swallow JIT compiler works by translating a function's CPython bytecode to platform-specific machine code, using data collected at runtime, as well as classical compiler optimizations, to improve the

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Collin Winter
Hi Paul, On Thu, Jan 21, 2010 at 12:56 PM, Paul Moore p.f.mo...@gmail.com wrote: I'm concerned about the memory and startup time penalties. It's nice that you're working on them - I'd like to see them remain a priority. Ultimately a *lot* of people use Python for short-running transient

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Martin v. Löwis
Reid Kleckner wrote: On Thu, Jan 21, 2010 at 4:34 PM, Martin v. Löwis mar...@v.loewis.de wrote: How large is the LLVM shared library? One surprising data point is that the binary is much larger than some of the memory footprint measurements given in the PEP. Could it be that you need to

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Collin Winter
On Thu, Jan 21, 2010 at 2:24 PM, Collin Winter collinwin...@google.com wrote: I'll also update the PEP with these benchmark results, since they're important to a lot of people. Done; see http://codereview.appspot.com/186247/diff2/4:8/9 for the wording change and new startup data. Collin Winter

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Martin v. Löwis
With my downstream distributor of Python hat on, I'm wondering if it would be feasible to replace the current precompiled .pyc/.pyo files in marshal format with .so/.dll files in platform-specific shared-library format, so that the pre-compiled versions of the stdlib could be memory-mapped

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Jeffrey Yasskin
On Thu, Jan 21, 2010 at 10:09 AM, Hanno Schlichting ha...@hannosch.eu wrote: I'm a relative outsider to core development (I'm just a Plone release manager), but'll allow myself a couple of questions. Feel free to ignore them, if you think they are not relevant at this point :-) I'd note that

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread David Malcolm
On Thu, 2010-01-21 at 23:42 +0100, Martin v. Löwis wrote: With my downstream distributor of Python hat on, I'm wondering if it would be feasible to replace the current precompiled .pyc/.pyo files in marshal format with .so/.dll files in platform-specific shared-library format, so that the

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Scott Dial
On 1/21/2010 1:09 PM, Hanno Schlichting wrote: - Who holds the copyright, is there a foundation or is there a risk of the project getting into trouble because of copyright issues? - What licence is the code and the tools under and what affect does that have on the code generated by the JIT

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Collin Winter
Hey Antoine, On Thu, Jan 21, 2010 at 4:25 AM, Antoine Pitrou solip...@pitrou.net wrote: The increased memory usage comes from a) LLVM code generation, analysis and optimization libraries; b) native code; c) memory usage issues or leaks in LLVM; d) data structures needed to optimize and

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Collin Winter
On Thu, Jan 21, 2010 at 10:14 AM, Reid Kleckner r...@mit.edu wrote: On Thu, Jan 21, 2010 at 9:35 AM, Floris Bruynooghe floris.bruynoo...@gmail.com wrote: I just compiled with the --without-llvm option and see that the binary, while only an acceptable 4.1M, still links with libstdc++.  Is it

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Collin Winter
Hey Glyph, On Thu, Jan 21, 2010 at 9:11 AM, Glyph Lefkowitz gl...@twistedmatrix.com wrote: It would be hard for me to put an exact number on what I would find acceptable, but I was really hoping that we could get a *reduced* memory footprint in the long term. My real concern here is not

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Barry Warsaw
On Jan 21, 2010, at 12:42 PM, Collin Winter wrote: Hi Collin, I don't believe that introducing the Unladen Swallow JIT will make maintaining a stable ABI per PEP 384 more difficult. We've been careful about not exporting any C++ symbols via PyAPI_FUNC(), so I don't believe that will be an issue

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Barry Warsaw
On Jan 21, 2010, at 02:46 PM, Jeffrey Yasskin wrote: LLVM's under the University of Illinois/NCSA license, which is BSD-like: http://www.opensource.org/licenses/UoI-NCSA.php or http://llvm.org/releases/2.6/LICENSE.TXT. Cool. This is a GPL compatible license, so it would presumably not change

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Jeffrey Yasskin
On Thu, Jan 21, 2010 at 3:49 PM, Barry Warsaw ba...@python.org wrote: Martin's follow up reminds me what the issues with C++ here are.  They center around which C++ compilers you use on which platforms.  Solaris, and to some extent Windows IIRC, were the most problematic for the work I was

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Barry Warsaw
On Jan 21, 2010, at 04:07 PM, Jeffrey Yasskin wrote: I could imagine a problem if Python+LLVM link in one libstdc++, and an extension module links in a different one, even if no C++ objects are passed across the boundary. Does that cause problems in practice? We'd have the same problems as from

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Collin Winter
On Thu, Jan 21, 2010 at 12:20 PM, Collin Winter collinwin...@google.com wrote: Hey Greg, On Wed, Jan 20, 2010 at 10:54 PM, Gregory P. Smith g...@krypto.org wrote: I think having a run time flag (or environment variable for those who like that) to disable the use of JIT at python3 execution

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jake McGuire wrote: Hmm. So cProfile doesn't break, but it causes code to run under a completely different execution model so the numbers it produces are not connected to reality? We've found the call graph and associated execution time

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Reid Kleckner
On Thu, Jan 21, 2010 at 5:07 PM, David Malcolm dmalc...@redhat.com wrote: To what extent would it be possible to use (conditionally) use full ahead-of-time compilation as well as JIT? It would be possible to do this, but it doesn't have nearly the same benefits as JIT compilation, as Alex

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Chris Bergstresser
On Thu, Jan 21, 2010 at 9:49 PM, Tres Seaver tsea...@palladion.com wrote: IIUC, optimizing your application using standard (non-JITed) profiling tools would still be a win for the app when run under the JIT, because your are going to be trimming code / using better algorithms, which will tend

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Glyph Lefkowitz
On Jan 21, 2010, at 6:48 PM, Collin Winter wrote: Hey Glyph, There's been a recent thread on our mailing list about a patch that dramatically reduces the memory footprint of multiprocess concurrency by separating reference counts from objects. We're looking at possibly incorporating this

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-21 Thread Collin Winter
On Thu, Jan 21, 2010 at 11:37 PM, Glyph Lefkowitz gl...@twistedmatrix.com wrote: On Jan 21, 2010, at 6:48 PM, Collin Winter wrote: Hey Glyph, There's been a recent thread on our mailing list about a patch that dramatically reduces the memory footprint of multiprocess concurrency by

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-20 Thread Martin v. Löwis
We're looking forward to discussing this with everyone. I think the PEP is asking too much (although I can understand how marketing may have influenced that), and also asks for permission where none is needed. It is too broad: it asks (in its title) for the integration of Unladen Swallow, when

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-20 Thread Terry Reedy
Some comments from a non-developer: The proposal to add this to 3.x seems a bit premature until you have a version that runs with 3.x. Not that I expect that to be a problem though. If CPython development moves to distributed hg, the notion of 'blessed' branches (other than the PSF release

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-20 Thread Collin Winter
Hi Terry, On Wed, Jan 20, 2010 at 5:14 PM, Terry Reedy tjre...@udel.edu wrote: Some comments from a non-developer: The proposal to add this to 3.x seems a bit premature until you have a version that runs with 3.x. Not that I expect that to be a problem though. If CPython development moves to

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-20 Thread Jeffrey Yasskin
On Wed, Jan 20, 2010 at 5:56 PM, Collin Winter collinwin...@google.com wrote: On Wed, Jan 20, 2010 at 5:14 PM, Terry Reedy tjre...@udel.edu wrote: Given the slight benefits compared to the costs, I think this, in its current state, should be optional, such as is psyco. Psyco has a similar

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-20 Thread Reid Kleckner
On Wed, Jan 20, 2010 at 8:14 PM, Terry Reedy tjre...@udel.edu wrote: If CPython development moves to distributed hg, the notion of 'blessed' branches (other than the PSF release branch) will, as I understand it, become somewhat obsolete. If you make a branch publicly available, anyone can grab

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-20 Thread Bill Janssen
Reid Kleckner r...@mit.edu wrote: On Wed, Jan 20, 2010 at 8:14 PM, Terry Reedy tjre...@udel.edu wrote: If CPython development moves to distributed hg, the notion of 'blessed' branches (other than the PSF release branch) will, as I understand it, become somewhat obsolete. If you make a

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-20 Thread Jack Diederich
On Wed, Jan 20, 2010 at 5:27 PM, Collin Winter collinwin...@google.com wrote: [big snip] In order to support hardware and software platforms where LLVM's JIT does not work, Unladen Swallow provides a ``./configure --without-llvm`` option. This flag carves out any part of Unladen Swallow that

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-20 Thread Benjamin Peterson
2010/1/20 Jack Diederich jackd...@gmail.com: On Wed, Jan 20, 2010 at 5:27 PM, Collin Winter collinwin...@google.com wrote: [big snip] In order to support hardware and software platforms where LLVM's JIT does not work, Unladen Swallow provides a ``./configure --without-llvm`` option. This

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-20 Thread Brett Cannon
On Wed, Jan 20, 2010 at 19:57, Bill Janssen jans...@parc.com wrote: Reid Kleckner r...@mit.edu wrote: On Wed, Jan 20, 2010 at 8:14 PM, Terry Reedy tjre...@udel.edu wrote: If CPython development moves to distributed hg, the notion of 'blessed' branches (other than the PSF release branch)

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-20 Thread Andrew McNabb
On Wed, Jan 20, 2010 at 10:13:56PM -0500, Reid Kleckner wrote: 2) As a command line option, you can pass -j never. If you have a short-lived script, you can just stick this in your #! line and forget about it. This has more overhead, since all of the JIT machinery is loaded into memory but

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-20 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jack Diederich wrote: Does disabling the LLVM change binary compatibility between modules targeted at the same version? At tonight's Boston PIG we had some binary package maintainers but most people (including myself) only cared about source

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-20 Thread Martin v. Löwis
It's true that as Martin said, we can rebase our code to Py3K in a branch on python.org any time we like, the question is more if we do the work, will the Python community accept it. I've stated my personal preference already. Let me add an observation on top of that: even if the core

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-20 Thread Gregory P. Smith
+1 My biggest concern is memory usage but it sounds like addressing that is already in your mind. I don't so much mind an additional up front constant and per-line-of-code hit for instrumentation but leaks are unacceptable. Any instrumentation data or jit caches should be managed (and tunable

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-20 Thread Dirkjan Ochtman
On Thu, Jan 21, 2010 at 02:56, Collin Winter collinwin...@google.com wrote: Agreed. We are actively working to improve the startup time penalty. We're interested in getting guidance from the CPython community as to what kind of a startup slow down would be sufficient in exchange for greater

<    1   2