Re: [Python-Dev] A new JIT compiler for a faster CPython?
On 18/07/2012 06:55, [email protected] wrote: Zitat von Mark Lawrence : On 17/07/2012 23:20, Victor Stinner wrote: http://psyco.sourceforge.net/ says: "News, 12 March 2012 Psyco is unmaintained and dead. Please look at PyPy for the state-of-the-art in JIT compilers for Python." Victor A search on pypi for JIT compilers gives no matches. I think you misread: PyPy, not pypi. Regards, Martin No think about it I did misread, my apologies for time wasting :-( -- Cheers. Mark Lawrence. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A new JIT compiler for a faster CPython?
In article <[email protected]>, [email protected] wrote: > I don't think it is. It is still slow and memory hungry. The fact that > the version that Apple ships with Xcode still miscompiles Python 3.3 > tells me that it is still buggy. Whether LLVM is suitable for a JIT is an interesting question but it's not LLVM per se that is the problem with compiling 3.3. Apple ships two C compiler chains with Xcode 4 for OS X 10.7, both of them are based on LLVM. It's the Apple transitional gcc-4.2 frontend with an old LLVM backend that is problematic (and not to be confused with the "pure" gcc-4.2 shipped with Xcode 3). That compiler was the default compiler for early releases of Xcode 4 and for building OS X 10.7. It has been frozen for a long time because Apple's efforts have been going into transitioning the OS X world to the new compiler: a clang frontend with a more current LLVM backend. The latest releases of Xcode 4 now use clang-llvm as the default and that's what we now use as a default for building Python 3.3 with Xcode 4. That transition will be complete with the imminent release of OS X 10.8 Mountain Lion when the whole OS is built with clang-llvm. The iOS world is already there. -- Ned Deily, [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A new JIT compiler for a faster CPython?
On Wed, Jul 18, 2012 at 8:27 AM, Stefan Behnel wrote: > [email protected], 18.07.2012 07:53: > > [Victor Stinner] > >> I don't want to write my own library to generate machine code. > > > > I plan to use nanojit. > > As I said, generating machine code is the uninteresting part of it and > won't give you much of a win. The changes you make to the way the code > works while walking the path from seeing the Python code to generating > machine code is what makes the difference. > > You could even skip the machine code generation all together and just go > with triggering pre-implemented high-level patterns from the interpreter. > If you code up the right patterns, your win would be bigger than with a > bare 1:1 mapping from Python code to machine code. Both Cython and WPython > are clear examples for that. > > Stefan > > It's uninteresting but it's completely necessary and it's still quite a bit of work. For the PyPy needs llvm failed to provide some features (besides being buggy), like dynamic patching of compiled assembler (you kind of need that for the tracing JIT when you discover new paths) or speed of execution. Cheers, fijal ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A new JIT compiler for a faster CPython?
> > > However, it requires that the JIT compiler knows about a lot of > optimisations. PyPy's JIT is full of those. It's not the fact that it has a > JIT compiler at all that makes it fast and not the fact that they compile > Python to machine code, it's the fact that they came up with a huge bunch > of specialisations that makes lots of code patterns fast once it detected > them. LLVM (or any other low-level JIT compiler) won't help at all with > that. > > Stefan > Very good point Stefan I would just like to add that a lot of those also require changes in the object model which might render changes in CPython C API (like the introduction of maps). Certainly you can't keep the current C structures, which will already break some code. Cheers, fijal ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A new JIT compiler for a faster CPython?
Some of my (reasonably well informed) opinions on this subject... The theory -- Don't think in terms of speeding up your program. Think in terms of reducing the time spent executing your program. Performance is improved by removing aspects of the execution overhead. In a talk I gave at EuroPython 2010(1), I divided the overhead into 5 parts: Interpretive Imprecise type information Parameter Handling & Call overhead Lookups (globals/builtins/attributes) Memory management (garbage collection) For optimising CPython, we cannot change the GC from ref-counting, but the other 4 apply, plus boxing and unboxing of floats and ints. Compilation (by which I assume people mean converting bytecodes to machine code) addresses only the first point (by definition). I worry that Victor is proposing to make the same mistake made by Unladen Swallow, which is to attack the interpretive overhead first, then attack the other overheads. This is the wrong way around. If you want good performance, JIT compilation should come last not first. Results from my PhD thesis(2) show that the original HotPy without any JIT compilation outperformed Unladen Swallow using JIT compilation. In other words, an optimising interpreter for Python will be faster than a naive JIT compiler. The optimised bytecode traces in an optimising interpreter are much better input for a JIT compiler than the original bytecodes. The practice If you want modest speedup for modest effort, then look at Cesare's WPython. Also take a look at Stefan Brunthaler's work on inline caching in an interpreter. If you want a larger speedup then you need to tackle most or all of the causes of execution overhead listed above. HotPy (version 2, a fork of CPython) aims to tackle all of these causes except the GC overhead. As far as I am aware, it is the only project that does so. Please take a look at www.hotpy.org for more information on HotPy. You can see my talk from EuroPython 2011(3) on the ideas behind it and from EuroPython 2012(4) on the current implementation. Finally, a defence of LLVM. LLVM is a quality piece of software. It may have some bugs, so does all software. The code-generation components are designed with static compilation in mind, so they do use a lot of memory and run slowly for a JIT compiler, but they produce excellent quality code. And don't forget the old saying about blaming your tools ;) If HotPy (version 2) were to have an (optional) JIT I would expect it to be LLVM based. The JIT can run in a separate thread, while the optimised code continues to run in the interpreter, patching in the machine code when it is complete. Cheers, Mark. 1) Talk at EuroPython 2010 Slides: www.dcs.gla.ac.uk/~marks/comparison.pdf Video: http://blip.tv/europythonvideos/mark_shannon-_hotpy_a_comparison-3999872 The information in the talk is a bit out of date; PyPy now includes out-of-line guards. 2) theses.gla.ac.uk/2975/01/2011shannonphd.pdf 3) Talk at EuroPython 2011 https://ep2012.europython.eu/conference/talks/making-cpython-fast-using-trace-based-optimisations 4) Talk at EuroPythnon 2012 https://ep2012.europython.eu/conference/talks/hotpy-2-a-high-performance-binary-compatible-virtual-machine-for-python ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A new JIT compiler for a faster CPython?
On Wed, Jul 18, 2012 at 7:45 PM, Mark Shannon wrote: > The practice > > > If you want modest speedup for modest effort, then look at Cesare's WPython. > Also take a look at Stefan Brunthaler's work on inline caching in an > interpreter. > > If you want a larger speedup then you need to tackle most or all of the > causes of > execution overhead listed above. > HotPy (version 2, a fork of CPython) aims to tackle all of these causes > except > the GC overhead. As far as I am aware, it is the only project that does so. Indeed, there's a lot that could be done in the CPython compiler and eval loop, and the bytecode design used to communicate between them. Thanks for summarising that so clearly. There are a couple of other compiler related patches that anyone interested in optimisation of CPython should at least glance at: - Dave Malcolm's patch that adds a Python level AST optimisation step to the compiler (effectively trading slower compile times for faster execution of the compiled bytecode) (http://bugs.python.org/issue10399) - Eugene Toder's patch to add an AST optimisation step to the compiler chain (http://bugs.python.org/issue11549) (I've asked Eugene about this patch more recently and his current thought is that subsequent improvements to the peephole optimisation have rendered it less valuable. However, the patch is still a potentially useful resource for anyone investigating bytecode optimisation ideas) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A new JIT compiler for a faster CPython?
On 18 Jul, 2012, at 7:53, [email protected] wrote: >> What is the status of LLVM nowadays? Is it not a good solution to >> write a portable JIT? > > I don't think it is. It is still slow and memory hungry. The fact that > the version that Apple ships with Xcode still miscompiles Python 3.3 > tells me that it is still buggy. Does it miscompile? I regularly run the 3.3 testsuite using the latest Xcode from the Appstore on a OSX Lion machine and that works properly. The only unexpected test failures are in ctypes, which is probably caused by the way the clang developers interpret the ABI w.r.t. passing small values. Ronald smime.p7s Description: S/MIME cryptographic signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyBytes_Join
On Wed, Jul 18, 2012 at 3:35 PM, Eli Bendersky wrote: > Hi, > > PyUnicode_Join is a public C API function. However, its cousin for the > bytes object is tucked privately in Objects/bytesobject.c as _PyBytes_Join. > Is there any harm in exposing it publicly? > > A more correctly formulated question would be: why is _PyBytes_Join part of the limited API, while PyUnicode_Join is in the full API. Is there a reason for the former to be less stable than the latter? Thanks in advance, Eli ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PyBytes_Join
Hi, PyUnicode_Join is a public C API function. However, its cousin for the bytes object is tucked privately in Objects/bytesobject.c as _PyBytes_Join. Is there any harm in exposing it publicly? Eli ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A new JIT compiler for a faster CPython?
Zitat von Ronald Oussoren : On 18 Jul, 2012, at 7:53, [email protected] wrote: What is the status of LLVM nowadays? Is it not a good solution to write a portable JIT? I don't think it is. It is still slow and memory hungry. The fact that the version that Apple ships with Xcode still miscompiles Python 3.3 tells me that it is still buggy. Does it miscompile? I'm talking about the bug in http://mail.python.org/pipermail/python-dev/2011-September/113731.html I regularly run the 3.3 testsuite using the latest Xcode from the Appstore on a OSX Lion machine and that works properly. I'm not actually using the latest Xcode. So if you could test my test program, that would be much appreciated. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A new JIT compiler for a faster CPython?
On 18 Jul, 2012, at 16:59, [email protected] wrote: > > Zitat von Ronald Oussoren : > >> On 18 Jul, 2012, at 7:53, [email protected] wrote: >> What is the status of LLVM nowadays? Is it not a good solution to write a portable JIT? >>> >>> I don't think it is. It is still slow and memory hungry. The fact that >>> the version that Apple ships with Xcode still miscompiles Python 3.3 >>> tells me that it is still buggy. >> >> Does it miscompile? > > I'm talking about the bug in > > http://mail.python.org/pipermail/python-dev/2011-September/113731.html > >> I regularly run the 3.3 testsuite using the latest Xcode from the Appstore >> on a OSX Lion machine and that works properly. > > I'm not actually using the latest Xcode. So if you could test my test > program, that would be much appreciated. That bug in llvm-gcc still exists, and is unlikely to get fixed. That's a bug in the integretion of the GCC frontend and LLVM backend, clang (LLVM project frontend + LLVM backend) does work. Ronald smime.p7s Description: S/MIME cryptographic signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Close #15387: inspect.getmodulename() now uses a new
Why is inspect.getmoduleinfo() deprecated? Is it just to remove circular dependencies? FWIW, I much prefer an API like: tell_me_about(object) to one like: for test_data in (X, Y, Z): usable = tester(object, test_data) if valid(usable): return possible_results[test_data] and to me, inspect.getmoduleinfo(path) looks like the first, while checking the various import.machinery.*SUFFIXES looks like the second. -jJ On 7/18/12, nick.coghlan wrote: > http://hg.python.org/cpython/rev/af7961e1c362 > changeset: 78161:af7961e1c362 > user:Nick Coghlan > date:Wed Jul 18 23:14:57 2012 +1000 > summary: > Close #15387: inspect.getmodulename() now uses a new > importlib.machinery.all_suffixes() API rather than the deprecated > inspect.getmoduleinfo() > > files: > Doc/library/importlib.rst | 13 - > Doc/library/inspect.rst| 15 --- > Lib/importlib/machinery.py | 4 > Lib/inspect.py | 11 +-- > Misc/NEWS | 3 +++ > 5 files changed, 40 insertions(+), 6 deletions(-) > > > diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst > --- a/Doc/library/importlib.rst > +++ b/Doc/library/importlib.rst > @@ -533,12 +533,23 @@ > > .. attribute:: EXTENSION_SUFFIXES > > - A list of strings representing the the recognized file suffixes for > + A list of strings representing the recognized file suffixes for > extension modules. > > .. versionadded:: 3.3 > > > +.. func:: all_suffixes() > + > + Returns a combined list of strings representing all file suffixes for > + Python modules recognized by the standard import machinery. This is a > + helper for code which simply needs to know if a filesystem path > + potentially represents a Python module (for example, > + :func:`inspect.getmodulename`) > + > + .. versionadded:: 3.3 > + > + > .. class:: BuiltinImporter > > An :term:`importer` for built-in modules. All known built-in modules > are > diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst > --- a/Doc/library/inspect.rst > +++ b/Doc/library/inspect.rst > @@ -198,9 +198,18 @@ > .. function:: getmodulename(path) > > Return the name of the module named by the file *path*, without > including the > - names of enclosing packages. This uses the same algorithm as the > interpreter > - uses when searching for modules. If the name cannot be matched > according to the > - interpreter's rules, ``None`` is returned. > + names of enclosing packages. The file extension is checked against all > of > + the entries in :func:`importlib.machinery.all_suffixes`. If it matches, > + the final path component is returned with the extension removed. > + Otherwise, ``None`` is returned. > + > + Note that this function *only* returns a meaningful name for actual > + Python modules - paths that potentially refer to Python packages will > + still return ``None``. > + > + .. versionchanged:: 3.3 > + This function is now based directly on :mod:`importlib` rather than > the > + deprecated :func:`getmoduleinfo`. > > > .. function:: ismodule(object) > diff --git a/Lib/importlib/machinery.py b/Lib/importlib/machinery.py > --- a/Lib/importlib/machinery.py > +++ b/Lib/importlib/machinery.py > @@ -13,3 +13,7 @@ > from ._bootstrap import ExtensionFileLoader > > EXTENSION_SUFFIXES = _imp.extension_suffixes() > + > +def all_suffixes(): > +"""Returns a list of all recognized module suffixes for this process""" > +return SOURCE_SUFFIXES + BYTECODE_SUFFIXES + EXTENSION_SUFFIXES > diff --git a/Lib/inspect.py b/Lib/inspect.py > --- a/Lib/inspect.py > +++ b/Lib/inspect.py > @@ -450,8 +450,15 @@ > > def getmodulename(path): > """Return the module name for a given file, or None.""" > -info = getmoduleinfo(path) > -if info: return info[0] > +fname = os.path.basename(path) > +# Check for paths that look like an actual module file > +suffixes = [(-len(suffix), suffix) > +for suffix in importlib.machinery.all_suffixes()] > +suffixes.sort() # try longest suffixes first, in case they overlap > +for neglen, suffix in suffixes: > +if fname.endswith(suffix): > +return fname[:neglen] > +return None > > def getsourcefile(object): > """Return the filename that can be used to locate an object's source. > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -41,6 +41,9 @@ > Library > --- > > +- Issue #15397: inspect.getmodulename() is now based directly on importlib > + via a new importlib.machinery.all_suffixes() API. > + > - Issue #14635: telnetlib will use poll() rather than select() when > possible >to avoid failing due to the select() file descriptor limit. > > > -- > Repository URL: http://hg.python.org/cpython > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailma
Re: [Python-Dev] [Python-checkins] cpython: Close #15387: inspect.getmodulename() now uses a new
Because the concepts it is based on are no longer used internally - determining the kind of module is now the province of importlib's finders and loaders. -- Sent from my phone, thus the relative brevity :) On Jul 19, 2012 2:50 AM, "Jim Jewett" wrote: > Why is inspect.getmoduleinfo() deprecated? Is it just to remove > circular dependencies? > > FWIW, I much prefer an API like: > > tell_me_about(object) > > to one like: > > for test_data in (X, Y, Z): > usable = tester(object, test_data) > if valid(usable): > return possible_results[test_data] > > and to me, inspect.getmoduleinfo(path) looks like the first, while > checking the various import.machinery.*SUFFIXES looks like the second. > > -jJ > > On 7/18/12, nick.coghlan wrote: > > http://hg.python.org/cpython/rev/af7961e1c362 > > changeset: 78161:af7961e1c362 > > user:Nick Coghlan > > date:Wed Jul 18 23:14:57 2012 +1000 > > summary: > > Close #15387: inspect.getmodulename() now uses a new > > importlib.machinery.all_suffixes() API rather than the deprecated > > inspect.getmoduleinfo() > > > > files: > > Doc/library/importlib.rst | 13 - > > Doc/library/inspect.rst| 15 --- > > Lib/importlib/machinery.py | 4 > > Lib/inspect.py | 11 +-- > > Misc/NEWS | 3 +++ > > 5 files changed, 40 insertions(+), 6 deletions(-) > > > > > > diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst > > --- a/Doc/library/importlib.rst > > +++ b/Doc/library/importlib.rst > > @@ -533,12 +533,23 @@ > > > > .. attribute:: EXTENSION_SUFFIXES > > > > - A list of strings representing the the recognized file suffixes for > > + A list of strings representing the recognized file suffixes for > > extension modules. > > > > .. versionadded:: 3.3 > > > > > > +.. func:: all_suffixes() > > + > > + Returns a combined list of strings representing all file suffixes for > > + Python modules recognized by the standard import machinery. This is a > > + helper for code which simply needs to know if a filesystem path > > + potentially represents a Python module (for example, > > + :func:`inspect.getmodulename`) > > + > > + .. versionadded:: 3.3 > > + > > + > > .. class:: BuiltinImporter > > > > An :term:`importer` for built-in modules. All known built-in modules > > are > > diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst > > --- a/Doc/library/inspect.rst > > +++ b/Doc/library/inspect.rst > > @@ -198,9 +198,18 @@ > > .. function:: getmodulename(path) > > > > Return the name of the module named by the file *path*, without > > including the > > - names of enclosing packages. This uses the same algorithm as the > > interpreter > > - uses when searching for modules. If the name cannot be matched > > according to the > > - interpreter's rules, ``None`` is returned. > > + names of enclosing packages. The file extension is checked against > all > > of > > + the entries in :func:`importlib.machinery.all_suffixes`. If it > matches, > > + the final path component is returned with the extension removed. > > + Otherwise, ``None`` is returned. > > + > > + Note that this function *only* returns a meaningful name for actual > > + Python modules - paths that potentially refer to Python packages will > > + still return ``None``. > > + > > + .. versionchanged:: 3.3 > > + This function is now based directly on :mod:`importlib` rather > than > > the > > + deprecated :func:`getmoduleinfo`. > > > > > > .. function:: ismodule(object) > > diff --git a/Lib/importlib/machinery.py b/Lib/importlib/machinery.py > > --- a/Lib/importlib/machinery.py > > +++ b/Lib/importlib/machinery.py > > @@ -13,3 +13,7 @@ > > from ._bootstrap import ExtensionFileLoader > > > > EXTENSION_SUFFIXES = _imp.extension_suffixes() > > + > > +def all_suffixes(): > > +"""Returns a list of all recognized module suffixes for this > process""" > > +return SOURCE_SUFFIXES + BYTECODE_SUFFIXES + EXTENSION_SUFFIXES > > diff --git a/Lib/inspect.py b/Lib/inspect.py > > --- a/Lib/inspect.py > > +++ b/Lib/inspect.py > > @@ -450,8 +450,15 @@ > > > > def getmodulename(path): > > """Return the module name for a given file, or None.""" > > -info = getmoduleinfo(path) > > -if info: return info[0] > > +fname = os.path.basename(path) > > +# Check for paths that look like an actual module file > > +suffixes = [(-len(suffix), suffix) > > +for suffix in importlib.machinery.all_suffixes()] > > +suffixes.sort() # try longest suffixes first, in case they overlap > > +for neglen, suffix in suffixes: > > +if fname.endswith(suffix): > > +return fname[:neglen] > > +return None > > > > def getsourcefile(object): > > """Return the filename that can be used to locate an object's > source. > > diff --git a/Misc/NEWS b/Misc/NEWS > > --- a/Misc/NEWS > > +++
