Author: Manuel Jacob Branch: improve-docs Changeset: r63965:5b37afd9acfd Date: 2013-05-09 16:54 +0200 http://bitbucket.org/pypy/pypy/changeset/5b37afd9acfd/
Log: Split architecture document and replace internal URLs by ref roles. diff --git a/pypy/doc/architecture.rst b/pypy/doc/architecture.rst --- a/pypy/doc/architecture.rst +++ b/pypy/doc/architecture.rst @@ -1,93 +1,27 @@ -=============================== Goals and Architecture Overview =============================== .. contents:: - This document gives an overview of the goals and architecture of PyPy. -See `getting started`_ for a practical introduction and starting points. +See :doc:`getting started <getting-started>` for a practical introduction and starting points. Mission statement -================= +----------------- -We aim to provide: +We aim to provide a compliant, flexible and fast implementation of the Python_ +Language which uses the RPython toolchain to enable new advanced high-level +features without having to encode the low-level details. We call this PyPy. - * a common translation and support framework for producing - implementations of dynamic languages, emphasizing a clean - separation between language specification and implementation - aspects. We call this the `RPython toolchain`_. +.. _Python: http://docs.python.org/reference/ - * a compliant, flexible and fast implementation of the Python_ Language - which uses the above toolchain to enable new advanced high-level features - without having to encode the low-level details. We call this PyPy. - -By separating concerns in this way, our implementation -of Python - and other dynamic languages - is able to automatically -generate a Just-in-Time compiler for any dynamic language. It also -allows a mix-and-match approach to implementation decisions, including -many that have historically been outside of a user's control, such as -target platform, memory and -threading models, garbage collection strategies, and optimizations applied, -including whether or not to have a JIT in the first place. High Level Goals -================ - -RPython - the Translation Toolchain ------------------------------------ - -Traditionally, language interpreters are written in a target platform language -such as C/Posix, Java or C#. Each implementation provides -a fundamental mapping between application source code and the target -environment. One of -the goals of the "all-encompassing" environments, such as the .NET framework -and to some extent the Java virtual machine, is to provide standardized -and higher level functionalities in order to support language implementers -for writing language implementations. - -PyPy is experimenting with a more ambitious approach. We are using a -subset of the high-level language Python, called RPython_, in which we -write languages as simple interpreters with few references to and -dependencies on lower level details. The `RPython toolchain`_ -produces a concrete virtual machine for the platform of our choice by -inserting appropriate lower level aspects. The result can be customized -by selecting other feature and platform configurations. - -Our goal is to provide a possible solution to the problem of language -implementers: having to write ``l * o * p`` interpreters for ``l`` -dynamic languages and ``p`` platforms with ``o`` crucial design -decisions. PyPy aims at making it possible to change each of these -variables independently such that: - -* ``l``: the language that we analyze can be evolved or entirely replaced; - -* ``o``: we can tweak and optimize the translation process to produce - platform specific code based on different models and trade-offs; - -* ``p``: we can write new translator back-ends to target different - physical and virtual platforms. - -By contrast, a standardized target environment - say .NET - -enforces ``p=1`` as far as it's concerned. This helps making ``o`` a -bit smaller by providing a higher-level base to build upon. Still, -we believe that enforcing the use of one common environment -is not necessary. PyPy's goal is to give weight to this claim - at least -as far as language implementation is concerned - showing an approach -to the ``l * o * p`` problem that does not rely on standardization. - -The most ambitious part of this goal is to `generate Just-In-Time -Compilers`_ in a language-independent way, instead of only translating -the source interpreter into an interpreter for the target platform. -This is an area of language implementation that is commonly considered -very challenging because of the involved complexity. - - -PyPy - the Python Interpreter ------------------------------ +---------------- Our main motivation for developing the translation framework is to -provide a full featured, customizable, fast_ and `very compliant`_ Python +provide a full featured, customizable, :ref:`fast <how-fast-is-pypy>` and +:doc:`very compliant <cpython_differences>` Python implementation, working on and interacting with a large variety of platforms and allowing the quick introduction of new advanced language features. @@ -106,72 +40,6 @@ and fully orthogonal to the interpreter source code. -PyPy Architecture -================= - -As you would expect from a project implemented using ideas from the world -of `Extreme Programming`_, the architecture of PyPy has evolved over time -and continues to evolve. Nevertheless, the high level architecture is -stable. As described above, there are two rather independent basic -subsystems: the `PyPy Python Interpreter`_ and the `RPython Translation Toolchain`_. - -.. _`translation framework`: - -RPython Translation Toolchain ------------------------------ - -The job of the RPython toolchain is to translate RPython_ programs -into an efficient version of that program for one of the various target -platforms, generally one that is considerably lower-level than Python. - -The approach we have taken is to reduce the level of abstraction of the -source RPython program in several steps, from the high level down to the -level of the target platform, whatever that may be. Currently we -support two broad flavours of target platforms: the ones that assume a -C-like memory model with structures and pointers, and the ones that -assume an object-oriented model with classes, instances and methods (as, -for example, the Java and .NET virtual machines do). - -The RPython toolchain never sees the RPython source code or syntax -trees, but rather starts with the *code objects* that define the -behaviour of the function objects one gives it as input. It can be -considered as "freezing" a pre-imported RPython program into an -executable form suitable for the target platform. - -The steps of the translation process can be summarized as follows: - -* The code object of each source functions is converted to a `control - flow graph` by the `Flow Object Space`_. - -* The control flow graphs are processed by the Annotator_, which - performs whole-program type inference to annotate each variable of - the control flow graph with the types it may take at run-time. - -* The information provided by the annotator is used by the RTyper_ to - convert the high level operations of the control flow graphs into - operations closer to the abstraction level of the target platform. - -* Optionally, `various transformations`_ can then be applied which, for - example, perform optimizations such as inlining, add capabilities - such as stackless-style concurrency, or insert code for the - `garbage collector`_. - -* Then, the graphs are converted to source code for the target platform - and compiled into an executable. - -This process is described in much more detail in the `document about -the RPython toolchain`_ and in the paper `Compiling dynamic language -implementations`_. - -.. _`control flow graph`: translation.html#the-flow-model -.. _`Flow Object Space`: objspace.html#the-flow-object-space -.. _Annotator: translation.html#the-annotation-pass -.. _RTyper: rtyper.html#overview -.. _`various transformations`: translation.html#the-optional-transformations -.. _`document about the RPython toolchain`: translation.html -.. _`garbage collector`: garbage_collection.html -.. _`RPython toolchain`: translation.html - .. _python-interpreter: PyPy Python Interpreter @@ -184,10 +52,10 @@ - a bytecode compiler responsible for producing Python code objects from the source code of a user application; -- a `bytecode evaluator`_ responsible for interpreting +- a :doc:`bytecode evaluator <interpreter>` responsible for interpreting Python code objects; -- a `standard object space`_, responsible for creating and manipulating +- a :ref:`standard object space <standard-object-space>`, responsible for creating and manipulating the Python objects seen by the application. The *bytecode compiler* is the preprocessing phase that produces a @@ -200,61 +68,11 @@ lists, as well as the operations between them, like addition or truth-value-testing. +.. TODO edit the sentence about the flow object space + This division between bytecode evaluator and object space is very important, as it gives a lot of flexibility. One can plug in -different `object spaces`_ to get different or enriched behaviours +different :doc:`object spaces <objspace>` to get different or enriched behaviours of the Python objects. Additionally, a special more abstract object -space, the `flow object space`_, allows us to reuse the bytecode +space, the :ref:`flow object space <flow-object-space>`, allows us to reuse the bytecode evaluator for our translation framework. - -.. _`bytecode evaluator`: interpreter.html -.. _`standard object space`: objspace.html#the-standard-object-space -.. _`object spaces`: objspace.html -.. _`flow object space`: objspace.html#the-flow-object-space - -.. _`the translation framework`: - - -Further reading -=============== - -All of PyPy's documentation can be reached from the `documentation -index`_. Of particular interest after reading this document might be: - - * `getting-started`_: a hands-on guide to getting involved with the - PyPy source code. - - * `PyPy's approach to virtual machine construction`_: a paper - presented to the Dynamic Languages Symposium attached to OOPSLA - 2006. - - * `The translation document`_: a detailed description of our - translation process. - - * `JIT Generation in PyPy`_, describing how we produce a Just-in-time - Compiler from an interpreter. - - * A tutorial of how to use the `RPython toolchain`_ to `implement your own - interpreter`_. - -.. _`documentation index`: index.html#project-documentation -.. _`getting-started`: getting-started-dev.html -.. _`PyPy's approach to virtual machine construction`: https://bitbucket.org/pypy/extradoc/raw/tip/talk/dls2006/pypy-vm-construction.pdf -.. _`the translation document`: translation.html -.. _`RPython toolchain`: translation.html -.. _`Compiling dynamic language implementations`: https://bitbucket.org/pypy/extradoc/raw/tip/eu-report/D05.1_Publish_on_translating_a_very-high-level_description.pdf -.. _`Technical reports`: index-report.html - -.. _`getting started`: getting-started.html -.. _`Extreme Programming`: http://www.extremeprogramming.org/ - -.. _fast: faq.html#how-fast-is-pypy -.. _`very compliant`: cpython_differences.html - -.. _`RPython`: coding-guide.html#rpython - -.. _Python: http://docs.python.org/reference/ -.. _Psyco: http://psyco.sourceforge.net -.. _`generate Just-In-Time Compilers`: jit/index.html -.. _`JIT Generation in PyPy`: jit/index.html -.. _`implement your own interpreter`: http://morepypy.blogspot.com/2011/04/tutorial-writing-interpreter-with-pypy.html diff --git a/pypy/doc/faq.rst b/pypy/doc/faq.rst --- a/pypy/doc/faq.rst +++ b/pypy/doc/faq.rst @@ -99,6 +99,8 @@ .. __: extending.html +.. _how-fast-is-pypy: + ----------------- How fast is PyPy? ----------------- diff --git a/pypy/doc/objspace.rst b/pypy/doc/objspace.rst --- a/pypy/doc/objspace.rst +++ b/pypy/doc/objspace.rst @@ -283,7 +283,7 @@ non-wrapped objects). -.. _`standard object space`: +.. _`standard-object-space`: The Standard Object Space ========================= @@ -483,7 +483,7 @@ ``add__Long_Long`` and there is no ``add__Int_Long``), which leads to ``6L.__radd__(5)`` being called, as in CPython. -.. _`Flow Object Space`: +.. _flow-object-space: The Flow Object Space ===================== diff --git a/rpython/doc/architecture.rst b/rpython/doc/architecture.rst new file mode 100644 --- /dev/null +++ b/rpython/doc/architecture.rst @@ -0,0 +1,141 @@ +Goals and Architecture Overview +=============================== + +.. contents:: + +Mission statement +----------------- + +We aim to provide a common translation and support framework for producing +implementations of dynamic languages, emphasizing a clean separation between +language specification and implementation aspects. We call this the +:doc:`RPython toolchain <translation>`. + +By separating concerns in this way, our implementation +of Python - and other dynamic languages - is able to automatically +generate a Just-in-Time compiler for any dynamic language. It also +allows a mix-and-match approach to implementation decisions, including +many that have historically been outside of a user's control, such as +target platform, memory and +threading models, garbage collection strategies, and optimizations applied, +including whether or not to have a JIT in the first place. + + +High Level Goals +---------------- + +Traditionally, language interpreters are written in a target platform language +such as C/Posix, Java or C#. Each implementation provides +a fundamental mapping between application source code and the target +environment. One of +the goals of the "all-encompassing" environments, such as the .NET framework +and to some extent the Java virtual machine, is to provide standardized +and higher level functionalities in order to support language implementers +for writing language implementations. + +PyPy is experimenting with a more ambitious approach. We are using a +subset of the high-level language Python, called :doc:`rpython`, in which we +write languages as simple interpreters with few references to and +dependencies on lower level details. The RPython toolchain +produces a concrete virtual machine for the platform of our choice by +inserting appropriate lower level aspects. The result can be customized +by selecting other feature and platform configurations. + +Our goal is to provide a possible solution to the problem of language +implementers: having to write ``l * o * p`` interpreters for ``l`` +dynamic languages and ``p`` platforms with ``o`` crucial design +decisions. PyPy aims at making it possible to change each of these +variables independently such that: + +* ``l``: the language that we analyze can be evolved or entirely replaced; + +* ``o``: we can tweak and optimize the translation process to produce + platform specific code based on different models and trade-offs; + +* ``p``: we can write new translator back-ends to target different + physical and virtual platforms. + +By contrast, a standardized target environment - say .NET - +enforces ``p=1`` as far as it's concerned. This helps making ``o`` a +bit smaller by providing a higher-level base to build upon. Still, +we believe that enforcing the use of one common environment +is not necessary. PyPy's goal is to give weight to this claim - at least +as far as language implementation is concerned - showing an approach +to the ``l * o * p`` problem that does not rely on standardization. + +The most ambitious part of this goal is to :doc:`generate Just-In-Time +Compilers <jit/index>` in a language-independent way, instead of only translating +the source interpreter into an interpreter for the target platform. +This is an area of language implementation that is commonly considered +very challenging because of the involved complexity. + + +Architecture +------------ + +The job of the RPython toolchain is to translate :doc:`rpython` programs +into an efficient version of that program for one of the various target +platforms, generally one that is considerably lower-level than Python. + +The approach we have taken is to reduce the level of abstraction of the +source RPython program in several steps, from the high level down to the +level of the target platform, whatever that may be. Currently we +support two broad flavours of target platforms: the ones that assume a +C-like memory model with structures and pointers, and the ones that +assume an object-oriented model with classes, instances and methods (as, +for example, the Java and .NET virtual machines do). + +The RPython toolchain never sees the RPython source code or syntax +trees, but rather starts with the *code objects* that define the +behaviour of the function objects one gives it as input. It can be +considered as "freezing" a pre-imported RPython program into an +executable form suitable for the target platform. + +The steps of the translation process can be summarized as follows: + +* The code object of each source functions is converted to a :ref:`control + flow graph <flow-model>` by the :ref:`Flow Object Space <flow-object-space>`. + +* The control flow graphs are processed by the :ref:`Annotator <annotator>`, which + performs whole-program type inference to annotate each variable of + the control flow graph with the types it may take at run-time. + +* The information provided by the annotator is used by the :doc:`RTyper <rtyper>` to + convert the high level operations of the control flow graphs into + operations closer to the abstraction level of the target platform. + +* Optionally, `various transformations <optional-transformations>` can then be applied which, for + example, perform optimizations such as inlining, add capabilities + such as stackless-style concurrency, or insert code for the + :doc:`garbage collector <garbage_collection>`. + +* Then, the graphs are converted to source code for the target platform + and compiled into an executable. + +This process is described in much more detail in the :doc:`document about +the RPython toolchain <translation>` and in the paper `Compiling dynamic language +implementations`_. + +.. _`Compiling dynamic language implementations`: https://bitbucket.org/pypy/extradoc/raw/tip/eu-report/D05.1_Publish_on_translating_a_very-high-level_description.pdf + +Further reading +--------------- + + * :doc:`getting-started`: a hands-on guide to getting involved with the + PyPy source code. + + * `PyPy's approach to virtual machine construction`_: a paper + presented to the Dynamic Languages Symposium attached to OOPSLA + 2006. + + * :doc:`The translation document <translation>`: a detailed description of our + translation process. + + * :doc:`JIT Generation in PyPy <jit/index>`, describing how we produce a Just-in-time + Compiler from an interpreter. + + * A tutorial of how to use the :doc:`RPython toolchain <translation>` to `implement your own + interpreter`_. + +.. _`PyPy's approach to virtual machine construction`: https://bitbucket.org/pypy/extradoc/raw/tip/talk/dls2006/pypy-vm-construction.pdf +.. _`implement your own interpreter`: http://morepypy.blogspot.com/2011/04/tutorial-writing-interpreter-with-pypy.html diff --git a/rpython/doc/translation.rst b/rpython/doc/translation.rst --- a/rpython/doc/translation.rst +++ b/rpython/doc/translation.rst @@ -100,7 +100,7 @@ .. _`Flow Object Space`: objspace.html#the-flow-object-space .. _`interactive interface`: getting-started-dev.html#try-out-the-translator -.. _`flow model`: +.. _`flow-model`: .. _`control flow graphs`: The Flow Model @@ -436,7 +436,7 @@ ``int_add`` is that code generators no longer have to worry about what kind of addition (or concatenation maybe?) it means. -.. _`optional transformations`: +.. _`optional-transformations`: The Optional Transformations ============================ _______________________________________________ pypy-commit mailing list [email protected] http://mail.python.org/mailman/listinfo/pypy-commit
