Re: On-topic: alternate Python implementations
Am 04.08.2012 15:53, schrieb Stefan Behnel: So, if a C++ compiler takes a .c file and compiles it with C language semantics, it doesn't qualify as a C compiler? That implies a rather weird definition of a C compiler, I'd say. I'd say that even a brainfuck compiler compiling a .py file with C language semantics can shamelessly call itself a C compiler. :P If a C++ compiler is given C code, it may or may not produce equivalent executables. In most non-trivial cases it will just barf on the valid C / invalid C++ code and refuse to compile it. In few rare cases, it will compile the code and produce different behaviour at runtime (e.g. for sizeof 'a'). Uli -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
On 8/4/2012 7:19 PM, Steven D'Aprano wrote: On Sat, 04 Aug 2012 18:38:33 -0700, Paul Rubin wrote: Steven D'Aprano steve+comp.lang.pyt...@pearwood.info writes: Runtime optimizations that target the common case, but fall back to unoptimized code in the rare cases that the optimization doesn't apply, offer the opportunity of big speedups for most code at the cost of trivial slowdowns when you do something unusual. The problem is you can't always tell if the unusual case is being exercised without an expensive dynamic check, which in some cases must be repeated in every iteration of a critical inner loop, even though it turns out that the program never actually uses the unusual case. There are other approaches. PyPy uses two interpreters and a JIT compiler to handle the hard cases. When code does something unexpected to other code, the backup interpreter is used to get control out of the trouble spot so that the JIT compiler can then recompile the code. (I think; I've read the paper but haven't looked at the internals.) This is hard to implement and hard to get right. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
alex23, 06.08.2012 05:40: On Aug 4, 4:15 pm, Steven D'Aprano steve +comp.lang.pyt...@pearwood.info wrote: But the Python ecosystem is a lot bigger than just those four. Here are just a few other implementations that you might be interested in: There's also HotPy: http://code.google.com/p/hotpy/ http://www.hotpy.org/ And just in case anyone was wondering where the others are: http://wiki.python.org/moin/PythonImplementations Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Jürgen A. Erhard, 05.08.2012 14:28: On Sun, Aug 05, 2012 at 07:46:59AM +0200, Stefan Behnel wrote: Jürgen A. Erhard, 05.08.2012 01:25: None of the other implementations require Python for actually compiling or running Python source. Nuitka was on the list as well. True, which I realized only after my missive. But doesn't change much, only that the list is wrong. Agreed. My definition, to also answer your following post, is does not rely on any executable part of the CPython source (which includes .c files and executable code in header files if any, but of course can exclude the stdlib). Not sure that's precise enough, but... if it can't run/work on a system that has no shred of CPython installed, it's not an alternative *implementation*. I can live with that definition. Cython is (by design) not an independent reimplementation of Python. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Stefan Behnel, 05.08.2012 07:46: Jürgen A. Erhard, 05.08.2012 01:25: None of the other implementations require Python for actually compiling or running Python source. Nuitka was on the list as well. Oh, and Stackless was also on Steven's list, as well as WPython. That means that 50% of the other implementations that Steven presented are not implementations according to your apparent definition. BTW, what is you definition? Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
On Sun, Aug 05, 2012 at 07:46:59AM +0200, Stefan Behnel wrote: Jürgen A. Erhard, 05.08.2012 01:25: On Sat, Aug 04, 2012 at 08:40:16AM +0200, Stefan Behnel wrote: Steven D'Aprano, 04.08.2012 08:15: Most people are aware, if only vaguely, of the big Four Python implementations: And not to forget Cython, which is the only static Python compiler that is widely used. Compiles and optimises Python to C code that uses the CPython runtime and allows for easy manual optimisations to get C-like performance out of it. Cython is certainly *not* a Python *implementation*, since it always uses the CPython runtime (and compiling Cython C files requires Python.h). Yes, it avoids an unnecessary duplication of effort as well as a substantial loss of compatibility that all non-CPython based implementations suffer from. But it's not an Python *implementation*, just an extension. Mind you, this is not intended as a slight of Cython as such. I really like it, though I haven't had need for it yet, but I sure prefer it to writing extensions in pure C. *b* None of the other implementations require Python for actually compiling or running Python source. Nuitka was on the list as well. True, which I realized only after my missive. But doesn't change much, only that the list is wrong. Oh, yes, you can create a stand-alone... wait, a stand-alone app. By embedding the Python runtime (dynamic linking with libpythonX.Y... maybe static too? Sure, that works. My definition, to also answer your following post, is does not rely on any executable part of the CPython source (which includes .c files and executable code in header files if any, but of course can exclude the stdlib). Not sure that's precise enough, but... if it can't run/work on a system that has no shred of CPython installed, it's not an alternative *implementation*. The big three don't need CPython (except PyPy for building, and even it can use a precompile PyPy I think). Grits, J -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Mark Lawrence wrote: With arrogance like that German by any chance? Comments like that are not appropriate on this list. Please don't make them. ~Ethan~ -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Mark Lawrence breamore...@yahoo.co.uk writes: With arrogance like that German by any chance? Please keep derogatory national stereotypes off this forum and out of our community. They are counter to our goals of diversity URL:http://www.python.org/community/diversity/; you don't have to subscribe to that, but if not then you thereby exclude yourself. -- \ “You can stand tall without standing on someone. You can be a | `\ victor without having victims.” —Harriet Woods, 1927–2007 | _o__) | Ben Finney -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
On Aug 4, 4:15 pm, Steven D'Aprano steve +comp.lang.pyt...@pearwood.info wrote: But the Python ecosystem is a lot bigger than just those four. Here are just a few other implementations that you might be interested in: There's also HotPy: http://code.google.com/p/hotpy/ http://www.hotpy.org/ -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
On Sat, Aug 4, 2012 at 4:15 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: CLPython, an implementation of Python written in Common Lisp. Berp - a compiler which works by translating Python to Haskell and compiling that. Okay. WHY? CLPython gives some reason, but how often do you need to bridge that particular pair of languages? And why compile Python via Haskell, when C is available as a high level assembly language? The mind boggles... ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Steven D'Aprano, 04.08.2012 08:15: Most people are aware, if only vaguely, of the big Four Python implementations: CPython, or just Python, the reference implementation written in C. IronPython, written in .NET. Jython, written in Java. PyPy, the optimizing implementation written in Python (actually, it's written in a subset of Python, RPython). But the Python ecosystem is a lot bigger than just those four. Here are just a few other implementations that you might be interested in: Stackless - the forgetten Python, Stackless is, I believe, the oldest implementation behind only CPython itself. It's a fork of CPython with the calling stack removed and fast and lightweight microthreads, and is used extensively in EVE Online. http://www.stackless.com/ Nuitka - optimising Python compiler written in C++, supports Python 2.6 and 2.7, claims to be up to twice as fast as CPython. http://nuitka.net/pages/overview.html WPython - another optimizing version of Python with wordcodes instead of bytecodes. http://code.google.com/p/wpython/ CLPython, an implementation of Python written in Common Lisp. http://common-lisp.net/project/clpython/ CapPython is an experimental restricted version of Python with capabilities. http://plash.beasts.org/wiki/CapPython http://en.wikipedia.org/wiki/Object-capability_model Berp - a compiler which works by translating Python to Haskell and compiling that. https://github.com/bjpop/berp/wiki And not to forget Cython, which is the only static Python compiler that is widely used. Compiles and optimises Python to C code that uses the CPython runtime and allows for easy manual optimisations to get C-like performance out of it. http://cython.org/ Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
On Sat, 04 Aug 2012 08:40:16 +0200, Stefan Behnel wrote: And not to forget Cython, which is the only static Python compiler that is widely used. Compiles and optimises Python to C code that uses the CPython runtime and allows for easy manual optimisations to get C-like performance out of it. http://cython.org/ Cython is great, but I question that it is a *Python* implementation. That's not a criticism of Cython, but it is different from Python. Take this example code from the tutorial: from libc.math cimport sin cdef double f(double x): return sin(x*x) If that's Python code, then I'm Ethel the Aardvark. Cython is very Python-like, but there is no doubt in my mind that it is a superset of Python and therefore a different language. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Steven D'Aprano, 04.08.2012 09:49: On Sat, 04 Aug 2012 08:40:16 +0200, Stefan Behnel wrote: And not to forget Cython, which is the only static Python compiler that is widely used. Compiles and optimises Python to C code that uses the CPython runtime and allows for easy manual optimisations to get C-like performance out of it. http://cython.org/ Cython is great, but I question that it is a *Python* implementation. That's not a criticism of Cython, but it is different from Python. Take this example code from the tutorial: from libc.math cimport sin cdef double f(double x): return sin(x*x) If that's Python code, then I'm Ethel the Aardvark. We never met in person, so I can't comment on the last part. However, the above is Cython code and, yes, that's a different language. Note that it uses a different file extension: .pyx. Try putting the above code into a .py file and compiling that. Cython will reject it and tell you that cimport is not valid Python syntax. Cython is very Python-like, but there is no doubt in my mind that it is a superset of Python and therefore a different language. As long as you don't use any features of the Cython language, it's plain Python. That makes it a Python compiler in my eyes. The fact that you can easily use C features to optimise your code (also in Python syntax, BTW) doesn't impact that. You mentioned a couple of other implementations and I'm sure they have features (and bugs) that CPython doesn't have. Interaction with Lisp code in CLPython, for example. I don't think additional features or language implementation bugs make a Python implementation non-Python per se. Also note that most of the less widely known alternative Python implementations do not even publish their results of running the CPython test suite, so how do you actually know they can run Python code? Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
On 04/08/2012 08:49, Steven D'Aprano wrote: On Sat, 04 Aug 2012 08:40:16 +0200, Stefan Behnel wrote: And not to forget Cython, which is the only static Python compiler that is widely used. Compiles and optimises Python to C code that uses the CPython runtime and allows for easy manual optimisations to get C-like performance out of it. http://cython.org/ Cython is great, but I question that it is a *Python* implementation. That's not a criticism of Cython, but it is different from Python. Take this example code from the tutorial: from libc.math cimport sin cdef double f(double x): return sin(x*x) If that's Python code, then I'm Ethel the Aardvark. Cython is very Python-like, but there is no doubt in my mind that it is a superset of Python and therefore a different language. I agree so it's off topic and can't be discussed here. Isn't that right, Stefan? -- Cheers. Mark Lawrence. -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
On Sat, 04 Aug 2012 16:34:17 +1000, Chris Angelico wrote: On Sat, Aug 4, 2012 at 4:15 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: CLPython, an implementation of Python written in Common Lisp. Berp - a compiler which works by translating Python to Haskell and compiling that. Okay. WHY? CLPython gives some reason, but how often do you need to bridge that particular pair of languages? And why compile Python via Haskell, when C is available as a high level assembly language? For much the same reason that PyPy uses RPython when C is available. Because Haskell is available as a high level non-assembly language. Berp is based on the Glasgow Haskell Compiler, which is a modern, efficient, optimizing compiler capable of producing excellent quality machine code on Windows, Mac, Linux and many Unixes. It gives you all the advantages of a high-level language with high-level data structures, type inference, and a compiler capable of generating optimized, fast, machine code. Who would want to deal with C's idiosyncrasies, low-powered explicit type system, difficult syntax, and core-dumps, when you could use something better? Apart from C programmers, of course. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Mark Lawrence, 04.08.2012 12:05: I agree so it's off topic and can't be discussed here. Isn't that right, Stefan? Hmm, in case you are referring to a recent friendly and diplomatic request of mine regarding a couple of people who were burdening a public high volume mailing list with a purely private back-and-forth chat about having beer and getting drunk - then, no, I don't think the discussion in this thread qualifies as yet another example for that so far. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: Who would want to deal with C's idiosyncrasies, low-powered explicit type system, difficult syntax, and core-dumps, when you could use something better? In the free software world, apparently many people like C. C is also quite popular in the zero-fault software world: Several verification tools do exist and Leroy et al. are writing a certified compiler for C to plug the hole between the verified source code and the generated assembly. Stefan Krah -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Steven D'Aprano, 04.08.2012 12:54: Berp is based on the Glasgow Haskell Compiler, which is a modern, efficient, optimizing compiler capable of producing excellent quality machine code on Windows, Mac, Linux and many Unixes. It gives you all the advantages of a high-level language with high-level data structures, type inference, and a compiler capable of generating optimized, fast, machine code. Although all those optimisations don't mean that Python code would run fast on top of it. Just because you translate Python to another language and platform doesn't mean that there's any benefit from the underlying platform optimisations. Both PyPy and Cython run Python code faster than CPython, but not because they eventually translate it into machine code but because they optimise and specialise it along the way, based on its high-level code constructs. One big success of the Unladen Swallow project was to show that bare JIT compilation is mostly worthless for high level languages. Who would want to deal with C's idiosyncrasies, low-powered explicit type system, difficult syntax, and core-dumps, when you could use something better? The core developers of both CPython and Cython aim for exactly that. They write C so you don't have to. But keep in mind that C is still *the* lingua franca of software development. A major reason why Python is (slowly) catching up these days is that the main implementation is written in C and makes it easy to interface with C code. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Am 04.08.2012 11:10 schrieb Stefan Behnel: As long as you don't use any features of the Cython language, it's plain Python. That makes it a Python compiler in my eyes. Tell that the C++ guys. C++ is mainly a superset of C. But nevertheless, C and C++ are distinct languages and so are Python and Cython. Thomas -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
The first time I did reply not 'reply all', so I'm posting again. ;-) I think Cython is a Python implementation because you can only use the Python features, not the extra features. C++ is different because of the different rules (C was in a time of assembly and costly computers, C++ was made in the time of (relatively) cheaper computers and computers doing more things) On 4 August 2012 18:21, Thomas Rachel nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa...@spamschutz.glglgl.de wrote: Am 04.08.2012 11:10 schrieb Stefan Behnel: As long as you don't use any features of the Cython language, it's plain Python. That makes it a Python compiler in my eyes. Tell that the C++ guys. C++ is mainly a superset of C. But nevertheless, C and C++ are distinct languages and so are Python and Cython. Thomas -- http://mail.python.org/**mailman/listinfo/python-listhttp://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Thomas Rachel, 04.08.2012 14:51: Am 04.08.2012 11:10 schrieb Stefan Behnel: As long as you don't use any features of the Cython language, it's plain Python. That makes it a Python compiler in my eyes. Tell that the C++ guys. C++ is mainly a superset of C. But nevertheless, C and C++ are distinct languages and so are Python and Cython. So, if a C++ compiler takes a .c file and compiles it with C language semantics, it doesn't qualify as a C compiler? That implies a rather weird definition of a C compiler, I'd say. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Stefan Behnel, 04.08.2012 15:53: Thomas Rachel, 04.08.2012 14:51: Am 04.08.2012 11:10 schrieb Stefan Behnel: As long as you don't use any features of the Cython language, it's plain Python. That makes it a Python compiler in my eyes. Tell that the C++ guys. C++ is mainly a superset of C. But nevertheless, C and C++ are distinct languages and so are Python and Cython. So, if a C++ compiler takes a .c file and compiles it with C language semantics, it doesn't qualify as a C compiler? That implies a rather weird definition of a C compiler, I'd say. Ah, sorry. Got it. You were again talking about Cython the language. Sure, Cython the language is different from Python the language. Cython the compiler can compile both. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Stefan Krah stefan-use...@bytereef.org writes: In the free software world, apparently many people like C. C is also quite popular in the zero-fault software world: Several verification tools do exist and Leroy et al. are writing a certified compiler for C to plug the hole between the verified source code and the generated assembly. C is pretty poor as a compiler target: how would you translate Python generators into C, for example? How would you handle garbage collection? C isn't so great for high-assurance stuff either, compared to (say) Ada. People do use it in critical apps, but that's just because it is (or anyway used to be) so ubiquitous. I'm wondering what you mean about verification tools, other than analyzers like Coverity that mainly check for bugs that in a safer language would be caught by the compiler. Compcert is not all that C-specific it has been adapted to compile a Haskell-derived language (Habit). Haskell doesn't sound all that great as a translation target for Python either, unfortunately, because its execution semantics are so different. GHC is a very powerful compiler but it was made to compile Haskell code that people actually write, and may do less good of a job with compiler output from an imperative language like Python. Compiling Python to Scheme and then using a Scheme compiler might be a more natural fit. But, compiling to Haskell was probably pretty convenient for that particular project. Finally, Python itself isn't all that well suited for compilation, given its high dynamicity. It will be interesting to see if the language evolves due to PyPy. -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Paul Rubin, 04.08.2012 17:59: Stefan Krah writes: In the free software world, apparently many people like C. C is also quite popular in the zero-fault software world: Several verification tools do exist and Leroy et al. are writing a certified compiler for C to plug the hole between the verified source code and the generated assembly. C is pretty poor as a compiler target: how would you translate Python generators into C, for example? Depends. If you have CPython available, that'd be a straight forward extension type. Otherwise, I guess you'd either have a class for them in C++ or a struct in C. Not exactly complex. For the yielding, you can use labels and goto. Given that you generate the code, that's pretty straight forward as well. How would you handle garbage collection? CPython does it automatically for us at least. Lacking that, you'd use one of the available garbage collection implementations, or provide none at all. Haskell doesn't sound all that great as a translation target for Python either, unfortunately, because its execution semantics are so different. GHC is a very powerful compiler but it was made to compile Haskell code that people actually write, and may do less good of a job with compiler output from an imperative language like Python. Compiling Python to Scheme and then using a Scheme compiler might be a more natural fit. But, compiling to Haskell was probably pretty convenient for that particular project. You'd have some kind of emulation layer that does most of the translation at runtime. That's why I said that you shouldn't expect too much of a performance gain from what the platform gives you for the underlying implementation. It can optimise the emulator, but it won't see enough of the Python code to make anything efficient out of it. Jython is an example for that. Finally, Python itself isn't all that well suited for compilation, given its high dynamicity. You can get pretty far with static code analysis, optimistic optimisations and code specialisation. We've decided against whole program analysis in Cython not only for compiler complexity reasons but also because it would let the normal compilation time explode for gains that you can much easier get by manual optimisation. Obviously, optimising JIT compilers can do much more here (because they actually have to do less), although they won't always be able to figure out the right thing to do either. That's where manual optimisation wins again. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Stefan Behnel stefan...@behnel.de writes: C is pretty poor as a compiler target: how would you translate Python generators into C, for example? Depends. If you have CPython available, that'd be a straight forward extension type. Calling CPython hardly counts as compiling Python into C. For the yielding, you can use labels and goto. Given that you generate the code, that's pretty straight forward as well. You're going to compile the whole Python program into a single C function so that you can do gotos inside of it? What happens if the program imports a generator? How would you handle garbage collection? CPython does it automatically for us at least. You mean you're going to have all the same INCREF/DECREF stuff on every operation in compiled data? Ugh. Lacking that, you'd use one of the available garbage collection implementations, What implementations would those be? There's the Boehm GC which is useful for some purposes but not really suitable at large scale, from what I can tell. Is there something else? or provide none at all. You're going to let the program just leak memory until it crashes?? you shouldn't expect too much of a performance gain from what the platform gives you for the underlying implementation. It can optimise the emulator, but it won't see enough of the Python code to make anything efficient out of it. Jython is an example for that. Compare that to the performance gain of LuaJIT and it starts to look like something is wrong with that approach, or maybe some issue inherent in Python itself. You can get pretty far with static code analysis, optimistic optimisations and code specialisation. It seems very hard to do reasonable optimizations in the presence of standard Python techniques like dynamically poking class instance attributes. I guess some optimizations are still possible, like storing attributes named as literals in the program in fixed slots, saving some dictionary lookups even though the slot contents would have to still be mutable. -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
On 04/08/2012 11:59, Stefan Behnel wrote: Mark Lawrence, 04.08.2012 12:05: I agree so it's off topic and can't be discussed here. Isn't that right, Stefan? Hmm, in case you are referring to a recent friendly and diplomatic request of mine regarding a couple of people who were burdening a public high volume mailing list with a purely private back-and-forth chat about having beer and getting drunk - then, no, I don't think the discussion in this thread qualifies as yet another example for that so far. Stefan With arrogance like that German by any chance? -- Cheers. Mark Lawrence. -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
On Sat, 04 Aug 2012 19:24:12 +0100, Mark Lawrence breamore...@yahoo.co.uk wrote: On 04/08/2012 11:59, Stefan Behnel wrote: Mark Lawrence, 04.08.2012 12:05: I agree so it's off topic and can't be discussed here. Isn't that right, Stefan? Hmm, in case you are referring to a recent friendly and diplomatic request of mine regarding a couple of people who were burdening a public high volume mailing list with a purely private back-and-forth chat about having beer and getting drunk - then, no, I don't think the discussion in this thread qualifies as yet another example for that so far. Stefan With arrogance like that German by any chance? Hey now, cool it with the passive-aggression. We're here to discuss code, right? If you want to fight it out, you can gladly do it by e-mail. ~Temia -- Invective! Verb your expletive nouns! -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
: On 4 August 2012 14:24, Mark Lawrence breamore...@yahoo.co.uk wrote: With arrogance like that German by any chance? I didn't give a monkeys about the beer conversation personally, but can we leave the national stereotypes out of it? -[]z. -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
From: Zero Piraeus sche...@gmail.com To: Mark Lawrence breamore...@yahoo.co.uk Cc: python-list@python.org Sent: Saturday, 4 August 2012, 19:42 Subject: Re: On-topic: alternate Python implementations : On 4 August 2012 14:24, Mark Lawrence breamore...@yahoo.co.uk wrote: With arrogance like that German by any chance? I didn't give a monkeys about the beer conversation personally, but can we leave the national stereotypes out of it? -[]z. No. Next question?-- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
: On 4 August 2012 14:50, Mark Lawrence breamore...@yahoo.co.uk wrote: No. Next question? *plonk* -[]z. -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Paul Rubin, 04.08.2012 20:18: Stefan Behnel writes: C is pretty poor as a compiler target: how would you translate Python generators into C, for example? Depends. If you have CPython available, that'd be a straight forward extension type. Calling CPython hardly counts as compiling Python into C. CPython is written in C, though. So anything that CPython does can be done in C. It's not like the CPython project used a completely unusual way of writing C code. Besides, I find your above statement questionable. You will always need some kind of runtime infrastructure when you compile Python into C, so you can just as well use CPython for that instead of reimplementing it completely from scratch. Both Cython and Nuitka do exactly that, and one of the major advantages of that approach is that they can freely interact with arbitrary code (Python or not) that was written for CPython, regardless of its native dependencies. What good would it be to throw all of that away, just for the sake of having pure C code generation? For the yielding, you can use labels and goto. Given that you generate the code, that's pretty straight forward as well. You're going to compile the whole Python program into a single C function so that you can do gotos inside of it? What happens if the program imports a generator? No, you are going to compile only the generator function into a function that uses gotos, maybe with an additional in-out struct parameter that holds its state. Then, on entry, you read the label (or its ID) from the previous state, reset local variables and jump to the label. On exit, you store the state back end return. Cython does it that way. Totally straight forward, as I said. How would you handle garbage collection? CPython does it automatically for us at least. You mean you're going to have all the same INCREF/DECREF stuff on every operation in compiled data? Ugh. If you don't like that, you can experiment with anything from a dedicated GC to transactional memory. Lacking that, you'd use one of the available garbage collection implementations, What implementations would those be? There's the Boehm GC which is useful for some purposes but not really suitable at large scale, from what I can tell. Is there something else? No idea - I'll look it up when I need one. Last I heard, PyPy had a couple of GCs to choose from, but I don't know how closely the are tied into its infrastructure. or provide none at all. You're going to let the program just leak memory until it crashes?? Well, it's not like CPython leaks memory until it crashes, now does it? And it's written in C. So there must be ways to handle this also in C. Remember that CPython didn't even have a GC before something around 2.0, IIRC. That worked quite ok in most cases and simply left the tricky cases to the programmers. It really depends on what your requirements are. Small embedded systems, time critical code and real-time systems are often much better off without garbage collection. It's pure convenience, after all. you shouldn't expect too much of a performance gain from what the platform gives you for the underlying implementation. It can optimise the emulator, but it won't see enough of the Python code to make anything efficient out of it. Jython is an example for that. Compare that to the performance gain of LuaJIT and it starts to look like something is wrong with that approach, or maybe some issue inherent in Python itself. Huh? LuaJIT is a reimplementation of Lua that uses an optimising JIT compiler specifically for Lua code. How is that similar to the Jython runtime that runs *on top of* the JVM with its generic byte code based JIT compiler? Basically, LuaJIT's JIT compiler works at the same level as the one in PyPy, which is why both can theoretically provide the same level of performance gains. You can get pretty far with static code analysis, optimistic optimisations and code specialisation. It seems very hard to do reasonable optimizations in the presence of standard Python techniques like dynamically poking class instance attributes. I guess some optimizations are still possible, like storing attributes named as literals in the program in fixed slots, saving some dictionary lookups even though the slot contents would have to still be mutable. Sure. Even when targeting the CPython runtime with the generated C code (like Cython or Nuitka), you can still do a lot. And sure, static code analysis will never be able to infer everything that a JIT compiler can see. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
On 04/08/2012 20:06, Stefan Behnel wrote: Paul Rubin, 04.08.2012 20:18: Stefan Behnel writes: C is pretty poor as a compiler target: how would you translate Python generators into C, for example? Depends. If you have CPython available, that'd be a straight forward extension type. Calling CPython hardly counts as compiling Python into C. CPython is written in C, though. So anything that CPython does can be done in C. It's not like the CPython project used a completely unusual way of writing C code. Besides, I find your above statement questionable. You will always need some kind of runtime infrastructure when you compile Python into C, so you can just as well use CPython for that instead of reimplementing it completely from scratch. Both Cython and Nuitka do exactly that, and one of the major advantages of that approach is that they can freely interact with arbitrary code (Python or not) that was written for CPython, regardless of its native dependencies. What good would it be to throw all of that away, just for the sake of having pure C code generation? For the yielding, you can use labels and goto. Given that you generate the code, that's pretty straight forward as well. You're going to compile the whole Python program into a single C function so that you can do gotos inside of it? What happens if the program imports a generator? No, you are going to compile only the generator function into a function that uses gotos, maybe with an additional in-out struct parameter that holds its state. Then, on entry, you read the label (or its ID) from the previous state, reset local variables and jump to the label. On exit, you store the state back end return. Cython does it that way. Totally straight forward, as I said. How would you handle garbage collection? CPython does it automatically for us at least. You mean you're going to have all the same INCREF/DECREF stuff on every operation in compiled data? Ugh. If you don't like that, you can experiment with anything from a dedicated GC to transactional memory. Lacking that, you'd use one of the available garbage collection implementations, What implementations would those be? There's the Boehm GC which is useful for some purposes but not really suitable at large scale, from what I can tell. Is there something else? No idea - I'll look it up when I need one. Last I heard, PyPy had a couple of GCs to choose from, but I don't know how closely the are tied into its infrastructure. or provide none at all. You're going to let the program just leak memory until it crashes?? Well, it's not like CPython leaks memory until it crashes, now does it? And it's written in C. So there must be ways to handle this also in C. Remember that CPython didn't even have a GC before something around 2.0, IIRC. That worked quite ok in most cases and simply left the tricky cases to the programmers. It really depends on what your requirements are. Small embedded systems, time critical code and real-time systems are often much better off without garbage collection. It's pure convenience, after all. [snip] CPython relied entirely on reference counting, so memory could leak you if inadvertently created a cycle of memory references. That problem was fixed when a mark-and-sweep mechanism was added (it's called occasionally to collect any unreachable cycles). -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: Most people are aware, if only vaguely, of the big Four Python implementations: CPython, or just Python, the reference implementation written in C. IronPython, written in .NET. Technicality: .NET is not a language, it is a run-time framework. IronPython is written in C#. It generates code that runs in the .NET Framework. -- Tim Roberts, t...@probo.com Providenza Boekelheide, Inc. -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Stefan Behnel stefan...@behnel.de writes: Calling CPython hardly counts as compiling Python into C. CPython is written in C, though. So anything that CPython does can be done in C. It's not like the CPython project used a completely unusual way of writing C code. CPython is a relatively simple interpreter, and executing code by invoking such an interpreter IMHO doesn't count as compiling it in any meaningful way. You will always need some kind of runtime infrastructure when you compile Python into C, so you can just as well use CPython for that instead of reimplementing it completely from scratch. Maybe there's parts of Cpython you can re-use, but having the CPython interpreter be the execution engine for compiled Python generators again fails the seriousness test of what it means to compile code. If you mean something other than that, you might explain more clearly. Both Cython and Nuitka do exactly that, I didn't know about Nuitka; it looks interesting but (at least after a few minutes looking) I don't have much sense of how it works. No, you are going to compile only the generator function into a function that uses gotos, maybe with an additional in-out struct parameter that holds its state. Yeah, ok, I guess that can work, given python generators are limited to returning through just one stack level. You might want to avoid copying locals by just putting everything into a struct, that has to be retained across entries/exits. If you don't like that, you can experiment with anything from a dedicated GC to transactional memory. OK, but then CPython is no longer managing the memory. Last I heard, PyPy had a couple of GCs to choose from, PyPy doesn't compile to C, but I guess compiling to C doesn't preclude precise GC, as long as the generated C code carefully tracks what C objects can contain GC-able pointers, and follows some constraints about when the GC can run. Some other compilers do this so it's not as big a deal as it sounded like at first. OK. or provide none at all. You're going to let the program just leak memory until it crashes?? Well, it's not like CPython leaks memory until it crashes... I was counting CPython's reference counting as a rudimentary form of GC, though I guess that's terminology that not everyone agrees on. Huh? LuaJIT is a reimplementation of Lua that uses an optimising JIT compiler specifically for Lua code. How is that similar to the Jython runtime that runs *on top of* the JVM with its generic byte code based JIT compiler? I thought LuaJIT compiles the existing Lua VM code, but I haven't looked at it closely or used it. It seems very hard to do reasonable optimizations in the presence of standard Python techniques Sure. Even when targeting the CPython runtime with the generated C code (like Cython or Nuitka), you can still do a lot. And sure, static code analysis will never be able to infer everything that a JIT compiler can see. I think even a JIT can't avoid a lot of pain and slowdown, without complex whole-program analysis and requiring the application to follow some special conventions, like never importing at runtime. -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Paul Rubin, 04.08.2012 22:43: Stefan Behnel writes: Calling CPython hardly counts as compiling Python into C. CPython is written in C, though. So anything that CPython does can be done in C. It's not like the CPython project used a completely unusual way of writing C code. CPython is a relatively simple interpreter, and executing code by invoking such an interpreter IMHO doesn't count as compiling it in any meaningful way. Oh, CPython is substantially more than an interpreter. The eval loop is only *one* way to use the runtime environment. Remember that it has many builtin types and functions as well as a huge standard library. Much of the runtime environment is already written in C or can be compiled down to C. If you compile Python code into C code that avoids the eval loop and only uses the CPython runtime environment (which is what Cython does), I think that qualifies as compiling Python code to C. It's definitely the most practical and user friendly way to do it. You will always need some kind of runtime infrastructure when you compile Python into C, so you can just as well use CPython for that instead of reimplementing it completely from scratch. Maybe there's parts of Cpython you can re-use, but having the CPython interpreter be the execution engine for compiled Python generators again fails the seriousness test of what it means to compile code. If you mean something other than that, you might explain more clearly. See above. Both Cython and Nuitka do exactly that, I didn't know about Nuitka; it looks interesting but (at least after a few minutes looking) I don't have much sense of how it works. It's mostly like Cython but without the type system, i.e. without all the stuff that makes it useful in real life. Just a bare Python-to-C++-in-CPython compiler, without much of a way to make it do what you want. Last I heard, PyPy had a couple of GCs to choose from, PyPy doesn't compile to C RPython (usually) does, though, and my guess is that the memory management part of the runtime is written in RPython. but I guess compiling to C doesn't preclude precise GC, as long as the generated C code carefully tracks what C objects can contain GC-able pointers, and follows some constraints about when the GC can run. Some other compilers do this so it's not as big a deal as it sounded like at first. OK. Yep, C really becomes a lot nicer when you generate it. Huh? LuaJIT is a reimplementation of Lua that uses an optimising JIT compiler specifically for Lua code. How is that similar to the Jython runtime that runs *on top of* the JVM with its generic byte code based JIT compiler? I thought LuaJIT compiles the existing Lua VM code, but I haven't looked at it closely or used it. Ok. It obviously reuses code, but the VM part of it is really different from standard Lua. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
On Friday, August 3, 2012 11:15:20 PM UTC-7, Steven D'Aprano wrote: WPython - another optimizing version of Python with wordcodes instead of bytecodes. http://code.google.com/p/wpython/ I remember reading about this a while ago. I thought this was eventually going to be committed to CPython... =\ -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
On Sat, Aug 04, 2012 at 08:40:16AM +0200, Stefan Behnel wrote: Steven D'Aprano, 04.08.2012 08:15: Most people are aware, if only vaguely, of the big Four Python implementations: And not to forget Cython, which is the only static Python compiler that is widely used. Compiles and optimises Python to C code that uses the CPython runtime and allows for easy manual optimisations to get C-like performance out of it. Cython is certainly *not* a Python *implementation*, since it always uses the CPython runtime (and compiling Cython C files requires Python.h). None of the other implementations require Python for actually compiling or running Python source. Oh, yes, you can create a stand-alone... wait, a stand-alone app. By embedding the Python runtime (dynamic linking with libpythonX.Y... maybe static too? Didn't test, because it's irrelevant for making the point). Grits, J -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
On Sat, 04 Aug 2012 08:59:18 -0700, Paul Rubin wrote: C isn't so great for high-assurance stuff either, compared to (say) Ada. People do use it in critical apps, but that's just because it is (or anyway used to be) so ubiquitous. And then they are shocked, SHOCKED I say!, when their app has enough buffer overflow security vulnerabilities to sink a battleship. [half a wink] Haskell doesn't sound all that great as a translation target for Python either, unfortunately, because its execution semantics are so different. I have no opinion on that either way, except to say that if some developer wants to experiment with Python-in-Haskell, good on him or her. Trying something new is how progress is made. [...] Finally, Python itself isn't all that well suited for compilation, given its high dynamicity. It will be interesting to see if the language evolves due to PyPy. Python is a dynamic language, but most Python code is relatively static. Runtime optimizations that target the common case, but fall back to unoptimized code in the rare cases that the optimization doesn't apply, offer the opportunity of big speedups for most code at the cost of trivial slowdowns when you do something unusual. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Steven D'Aprano steve+comp.lang.pyt...@pearwood.info writes: Runtime optimizations that target the common case, but fall back to unoptimized code in the rare cases that the optimization doesn't apply, offer the opportunity of big speedups for most code at the cost of trivial slowdowns when you do something unusual. The problem is you can't always tell if the unusual case is being exercised without an expensive dynamic check, which in some cases must be repeated in every iteration of a critical inner loop, even though it turns out that the program never actually uses the unusual case. -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
On Sat, 04 Aug 2012 18:38:33 -0700, Paul Rubin wrote: Steven D'Aprano steve+comp.lang.pyt...@pearwood.info writes: Runtime optimizations that target the common case, but fall back to unoptimized code in the rare cases that the optimization doesn't apply, offer the opportunity of big speedups for most code at the cost of trivial slowdowns when you do something unusual. The problem is you can't always tell if the unusual case is being exercised without an expensive dynamic check, which in some cases must be repeated in every iteration of a critical inner loop, even though it turns out that the program never actually uses the unusual case. I never said optimizing Python was easy :) Obviously if the check is expensive enough, the optimization isn't going to be worth doing. But often the check is not so expensive, or is just a matter of tedious and careful book-keeping. I don't wish to dispute that optimizing Python is hard, but it's not a Hard Problem like factorizing huge integers, or solving the Palestine/ Israeli conflict. It's hard like cleaning your house after a gang of drunken frat boys have partied all weekend. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Paul Rubin, 05.08.2012 03:38: Steven D'Aprano writes: Runtime optimizations that target the common case, but fall back to unoptimized code in the rare cases that the optimization doesn't apply, offer the opportunity of big speedups for most code at the cost of trivial slowdowns when you do something unusual. The problem is you can't always tell if the unusual case is being exercised without an expensive dynamic check, which in some cases must be repeated in every iteration of a critical inner loop, even though it turns out that the program never actually uses the unusual case. Cython does a lot of optimistic optimisations. That's where a large part of that huge C file comes from that Cython generates from even simple Python code. For example, in CPython, C function calls are so ridiculously faster than Python function calls that it's worth some effort if it saves you from packing an argument tuple to call into a Python function. In fact, we've been thinking about ways to export C signatures from Python function objects, so that code implemented in C (or a C compatible language) can be called directly from other code implemented in C. That's very common in the CPython ecosystem. There are a lot of simple things that quickly add up into a much better performance on average. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: On-topic: alternate Python implementations
Jürgen A. Erhard, 05.08.2012 01:25: On Sat, Aug 04, 2012 at 08:40:16AM +0200, Stefan Behnel wrote: Steven D'Aprano, 04.08.2012 08:15: Most people are aware, if only vaguely, of the big Four Python implementations: And not to forget Cython, which is the only static Python compiler that is widely used. Compiles and optimises Python to C code that uses the CPython runtime and allows for easy manual optimisations to get C-like performance out of it. Cython is certainly *not* a Python *implementation*, since it always uses the CPython runtime (and compiling Cython C files requires Python.h). Yes, it avoids an unnecessary duplication of effort as well as a substantial loss of compatibility that all non-CPython based implementations suffer from. You'd be surprised to see how much of Python we implement, though, including some of the builtins. You might want to revise your opinion once you start digging into it. It's always easy to disagree at the surface. None of the other implementations require Python for actually compiling or running Python source. Nuitka was on the list as well. Oh, yes, you can create a stand-alone... wait, a stand-alone app. By embedding the Python runtime (dynamic linking with libpythonX.Y... maybe static too? Sure, that works. Stefan -- http://mail.python.org/mailman/listinfo/python-list
