Author: Armin Rigo <[email protected]> Branch: Changeset: r71364:a421f5063e35 Date: 2014-05-07 12:22 +0200 http://bitbucket.org/pypy/pypy/changeset/a421f5063e35/
Log: Move this document from the stm branch diff --git a/pypy/doc/stm.rst b/pypy/doc/stm.rst --- a/pypy/doc/stm.rst +++ b/pypy/doc/stm.rst @@ -1,70 +1,77 @@ -====================== -Transactional Memory -====================== + +============================= +Software Transactional Memory +============================= .. contents:: This page is about ``pypy-stm``, a special in-development version of PyPy which can run multiple independent CPU-hungry threads in the same -process in parallel. It is side-stepping what is known in the Python -world as the "global interpreter lock (GIL)" problem. +process in parallel. It is a solution to what is known in the Python +world as the "global interpreter lock (GIL)" problem --- it is an +implementation of Python without the GIL. -"STM" stands for Software Transactional Memory, the technique used +"STM" stands for Software `Transactional Memory`_, the technique used internally. This page describes ``pypy-stm`` from the perspective of a user, describes work in progress, and finally gives references to more implementation details. -This work was done mostly by Remi Meier and Armin Rigo. Thanks to all -donors for crowd-funding the work so far! Please have a look at the -`2nd call for donation`_. +This work was done by Remi Meier and Armin Rigo. Thanks to all donors +for crowd-funding the work so far! Please have a look at the `2nd call +for donation`_. +.. _`Transactional Memory`: http://en.wikipedia.org/wiki/Transactional_memory .. _`2nd call for donation`: http://pypy.org/tmdonate2.html Introduction ============ -``pypy-stm`` is a variant of the regular PyPy interpreter. With caveats -listed below, it should be in theory within 25%-50% slower than a +``pypy-stm`` is a variant of the regular PyPy interpreter. With caveats_ +listed below, it should be in theory within 20%-50% slower than a regular PyPy, comparing the JIT version in both cases. It is called STM for Software Transactional Memory, which is the internal technique used (see `Reference to implementation details`_). -What you get in exchange for this slow-down is that ``pypy-stm`` runs -any multithreaded Python program on multiple CPUs at once. Programs -running two threads or more in parallel should ideally run faster than -in a regular PyPy, either now or soon as issues are fixed. In one way, -that's all there is to it: this is a GIL-less Python, feel free to -`download and try it`__. However, the deeper idea behind the -``pypy-stm`` project is to improve what is so far the state-of-the-art -for using multiple CPUs, which for cases where separate processes don't -work is done by writing explicitly multi-threaded programs. Instead, -``pypy-stm`` is pushing forward an approach to *hide* the threads, as -described below in `atomic sections`_. +The benefit is that the resulting ``pypy-stm`` can execute multiple +threads of Python code in parallel. Programs running two threads or +more in parallel should ideally run faster than in a regular PyPy +(either now, or soon as bugs are fixed). +* ``pypy-stm`` is fully compatible with a GIL-based PyPy; you can use + it as a drop-in replacement and multithreaded programs will run on + multiple cores. -.. __: +* ``pypy-stm`` does not impose any special API to the user, but it + provides a new pure Python module called `transactional_memory`_ with + features to inspect the state or debug conflicts_ that prevent + parallelization. This module can also be imported on top of a non-STM + PyPy or CPython. -Current status -============== +* Building on top of the way the GIL is removed, we will talk + about `Atomic sections, Transactions, etc.: a better way to write + parallel programs`_. + + +Getting Started +=============== **pypy-stm requires 64-bit Linux for now.** Development is done in the branch `stmgc-c7`_. If you are only -interested in trying it out, you can download a Ubuntu 12.04 binary -here__ (``pypy-2.2.x-stm*.tar.bz2``; this version is a release mode, -but not stripped of debug symbols). The current version supports four -"segments", which means that it will run up to four threads in parallel, -in other words it is running a thread pool up to 4 threads emulating normal -threads. +interested in trying it out, you can download a Ubuntu binary here__ +(``pypy-2.3.x-stm*.tar.bz2``, Ubuntu 12.04-14.04; these versions are +release mode, but not stripped of debug symbols). The current version +supports four "segments", which means that it will run up to four +threads in parallel. To build a version from sources, you first need to compile a custom -version of clang; we recommend downloading `llvm and clang like -described here`__, but at revision 201645 (use ``svn co -r 201645 ...`` +version of clang(!); we recommend downloading `llvm and clang like +described here`__, but at revision 201645 (use ``svn co -r 201645 <path>`` for all checkouts). Then apply all the patches in `this directory`__: -they are fixes for the very extensive usage that pypy-stm does of a -clang-only feature (without them, you get crashes of clang). Then get +they are fixes for a clang-only feature that hasn't been used so heavily +in the past (without the patches, you get crashes of clang). Then get the branch `stmgc-c7`_ of PyPy and run:: rpython/bin/rpython -Ojit --stm pypy/goal/targetpypystandalone.py @@ -75,23 +82,26 @@ .. __: https://bitbucket.org/pypy/stmgc/src/default/c7/llvmfix/ -Caveats: +.. _caveats: -* So far, small examples work fine, but there are still a number of - bugs. We're busy fixing them. +Current status +-------------- + +* So far, small examples work fine, but there are still a few bugs. + We're busy fixing them as we find them; feel free to `report bugs`_. * Currently limited to 1.5 GB of RAM (this is just a parameter in - `core.h`__). Memory overflows are not detected correctly, so may - cause segmentation faults. + `core.h`__). Memory overflows are not correctly handled; they cause + segfaults. -* The JIT warm-up time is abysmal (as opposed to the regular PyPy's, - which is "only" bad). Moreover, you should run it with a command like - ``pypy-stm --jit trace_limit=60000 args...``; the default value of - 6000 for ``trace_limit`` is currently too low (6000 should become - reasonable again as we improve). Also, in order to produce machine - code, the JIT needs to enter a special single-threaded mode for now. - This all means that you *will* get very bad performance results if - your program doesn't run for *many* seconds for now. +* The JIT warm-up time improved recently but is still bad. In order to + produce machine code, the JIT needs to enter a special single-threaded + mode for now. This means that you will get bad performance results if + your program doesn't run for several seconds, where *several* can mean + *many.* When trying benchmarks, be sure to check that you have + reached the warmed state, i.e. the performance is not improving any + more. This should be clear from the fact that as long as it's + producing more machine code, ``pypy-stm`` will run on a single core. * The GC is new; although clearly inspired by PyPy's regular GC, it misses a number of optimizations for now. Programs allocating large @@ -108,111 +118,197 @@ * The STM system is based on very efficient read/write barriers, which are mostly done (their placement could be improved a bit in JIT-generated machine code). But the overall bookkeeping logic could - see more improvements (see Statistics_ below). - -* You can use `atomic sections`_, but the most visible missing thing is - that you don't get reports about the "conflicts" you get. This would - be the first thing that you need in order to start using atomic - sections more extensively. Also, for now: for better results, try to - explicitly force a transaction break just before (and possibly after) - each large atomic section, with ``time.sleep(0)``. + see more improvements (see `Low-level statistics`_ below). * Forking the process is slow because the complete memory needs to be - copied manually right now. + copied manually. A warning is printed to this effect. -* Very long-running processes should eventually crash on an assertion - error because of a non-implemented overflow of an internal 29-bit - number, but this requires at the very least ten hours --- more - probably, several days or more. +* Very long-running processes (on the order of days) will eventually + crash on an assertion error because of a non-implemented overflow of + an internal 29-bit number. .. _`report bugs`: https://bugs.pypy.org/ .. __: https://bitbucket.org/pypy/pypy/raw/stmgc-c7/rpython/translator/stm/src_stm/stm/core.h -Statistics +User Guide ========== + -When a non-main thread finishes, you get statistics printed to stderr, -looking like that:: +Drop-in replacement +------------------- - thread 0x7f73377fe600: - outside transaction 42182 0.506 s - run current 85466 0.000 s - run committed 34262 3.178 s - run aborted write write 6982 0.083 s - run aborted write read 550 0.005 s - run aborted inevitable 388 0.010 s - run aborted other 0 0.000 s - wait free segment 0 0.000 s - wait write read 78 0.027 s - wait inevitable 887 0.490 s - wait other 0 0.000 s - bookkeeping 51418 0.606 s - minor gc 162970 1.135 s - major gc 1 0.019 s - sync pause 59173 1.738 s - spin loop 129512 0.094 s +Multithreaded, CPU-intensive Python programs should work unchanged on +``pypy-stm``. They will run using multiple CPU cores in parallel. -The first number is a counter; the second number gives the associated -time (the amount of real time that the thread was in this state; the sum -of all the times should be equal to the total time between the thread's -start and the thread's end). The most important points are "run -committed", which gives the amount of useful work, and "outside -transaction", which should give the time spent e.g. in library calls -(right now it seems to be a bit larger than that; to investigate). -Everything else is overhead of various forms. (Short-, medium- and -long-term future work involves reducing this overhead :-) +The existing semantics of the GIL (Global Interpreter Lock) are +unchanged: although running on multiple cores in parallel, ``pypy-stm`` +gives the illusion that threads are run serially, with switches only +occurring between bytecodes, not in the middle of them. Programs can +rely on this: using ``shared_list.append()/pop()`` or +``shared_dict.setdefault()`` as synchronization mecanisms continues to +work as expected. -These statistics are not printed out for the main thread, for now. +This works by internally considering the points where a standard PyPy or +CPython would release the GIL, and replacing them with the boundaries of +"transaction". Like their database equivalent, multiple transactions +can execute in parallel, but will commit in some serial order. They +appear to behave as if they were completely run in this serialization +order. Atomic sections -=============== +--------------- -While one of the goal of pypy-stm is to give a GIL-free but otherwise -unmodified Python, the other goal is to push for a better way to use -multithreading. For this, you (as the Python programmer) get an API -in the ``__pypy__.thread`` submodule: +PyPy supports *atomic sections,* which are blocks of code which you want +to execute without "releasing the GIL". *This is experimental and may +be removed in the future.* In STM terms, this means blocks of code that +are executed while guaranteeing that the transaction is not interrupted +in the middle. -* ``__pypy__.thread.atomic``: a context manager (i.e. you use it in - a ``with __pypy__.thread.atomic:`` statement). It runs the whole - block of code without breaking the current transaction --- from - the point of view of a regular CPython/PyPy, this is equivalent to - saying that the GIL will not be released at all between the start and - the end of this block of code. +Here is a usage example:: -The obvious usage is to use atomic blocks in the same way as one would -use locks: to protect changes to some shared data, you do them in a -``with atomic`` block, just like you would otherwise do them in a ``with -mylock`` block after ``mylock = thread.allocate_lock()``. This allows -you not to care about acquiring the correct locks in the correct order; -it is equivalent to having only one global lock. This is how -transactional memory is `generally described`__: as a way to efficiently -execute such atomic blocks, running them in parallel while giving the -illusion that they run in some serial order. + with __pypy__.thread.atomic: + assert len(lst1) == 10 + x = lst1.pop(0) + lst1.append(x) -.. __: http://en.wikipedia.org/wiki/Transactional_memory +In this (bad) example, we are sure that the item popped off one end of +the list is appened again at the other end atomically. It means that +another thread can run ``len(lst1)`` or ``x in lst1`` without any +particular synchronization, and always see the same results, +respectively ``10`` and ``True``. It will never see the intermediate +state where ``lst1`` only contains 9 elements. Atomic sections are +similar to re-entrant locks (they can be nested), but additionally they +protect against the concurrent execution of *any* code instead of just +code that happens to be protected by the same lock in other threads. -However, the less obvious intended usage of atomic sections is as a -wide-ranging replacement of explicit threads. You can turn a program -that is not multi-threaded at all into a program that uses threads -internally, together with large atomic sections to keep the behavior -unchanged. This capability can be hidden in a library or in the -framework you use; the end user's code does not need to be explicitly -aware of using threads. For a simple example of this, see -`transaction.py`_ in ``lib_pypy``. The idea is that if you have a -program where the function ``f(key, value)`` runs on every item of some -big dictionary, you can replace the loop with:: +Note that the notion of atomic sections is very strong. If you write +code like this:: + + with __pypy__.thread.atomic: + time.sleep(10) + +then, if you think about it as if we had a GIL, you are executing a +10-seconds-long atomic transaction without releasing the GIL at all. +This prevents all other threads from progressing at all. While it is +not strictly true in ``pypy-stm``, the exact rules for when other +threads can progress or not are rather complicated; you have to consider +it likely that such a piece of code will eventually block all other +threads anyway. + +Note that if you want to experiment with ``atomic``, you may have to add +manually a transaction break just before the atomic block. This is +because the boundaries of the block are not guaranteed to be the +boundaries of the transaction: the latter is at least as big as the +block, but maybe bigger. Therefore, if you run a big atomic block, it +is a good idea to break the transaction just before. This can be done +e.g. by the hack of calling ``time.sleep(0)``. (This may be fixed at +some point.) + +There are also issues with the interaction of locks and atomic blocks. +This can be seen if you write to files (which have locks), including +with a ``print`` to standard output. If one thread tries to acquire a +lock while running in an atomic block, and another thread has got the +same lock, then the former may fail with a ``thread.error``. The reason +is that "waiting" for some condition to become true --while running in +an atomic block-- does not really make sense. For now you can work +around it by making sure that, say, all your prints are either in an +``atomic`` block or none of them are. (This kind of issue is +theoretically hard to solve.) + + +Locks +----- + +**Not Implemented Yet** + +The thread module's locks have their basic semantic unchanged. However, +using them (e.g. in ``with my_lock:`` blocks) starts an alternative +running mode, called `Software lock elision`_. This means that PyPy +will try to make sure that the transaction extends until the point where +the lock is released, and if it succeeds, then the acquiring and +releasing of the lock will be "elided". This means that in this case, +the whole transaction will technically not cause any write into the lock +object --- it was unacquired before, and is still unacquired after the +transaction. + +This is specially useful if two threads run ``with my_lock:`` blocks +with the same lock. If they each run a transaction that is long enough +to contain the whole block, then all writes into the lock will be elided +and the two transactions will not conflict with each other. As usual, +they will be serialized in some order: one of the two will appear to run +before the other. Simply, each of them executes an "acquire" followed +by a "release" in the same transaction. As explained above, the lock +state goes from "unacquired" to "unacquired" and can thus be left +unchanged. + +This approach can gracefully fail: unlike atomic sections, there is no +guarantee that the transaction runs until the end of the block. If you +perform any input/output while you hold the lock, the transaction will +end as usual just before the input/output operation. If this occurs, +then the lock elision mode is cancelled and the lock's "acquired" state +is really written. + +Even if the lock is really acquired already, a transaction doesn't have +to wait for it to become free again. It can enter the elision-mode anyway +and tentatively execute the content of the block. It is only at the end, +when trying to commit, that the thread will pause. As soon as the real +value stored in the lock is switched back to "unacquired", it can then +proceed and attempt to commit its already-executed transaction (which +can fail and abort and restart from the scratch, as usual). + +Note that this is all *not implemented yet,* but we expect it to work +even if you acquire and release several locks. The elision-mode +transaction will extend until the first lock you acquired is released, +or until the code performs an input/output or a wait operation (for +example, waiting for another lock that is currently not free). In the +common case of acquiring several locks in nested order, they will all be +elided by the same transaction. + +.. _`software lock elision`: https://www.repository.cam.ac.uk/handle/1810/239410 + + +Atomic sections, Transactions, etc.: a better way to write parallel programs +---------------------------------------------------------------------------- + +(This section is based on locks as we plan to implement them, but also +works with the existing atomic sections.) + +In the cases where elision works, the block of code can run in parallel +with other blocks of code *even if they are protected by the same lock.* +You still get the illusion that the blocks are run sequentially. This +works even for multiple threads that run each a series of such blocks +and nothing else, protected by one single global lock. This is +basically the Python application-level equivalent of what was done with +the interpreter in ``pypy-stm``: while you think you are writing +thread-unfriendly code because of this global lock, actually the +underlying system is able to make it run on multiple cores anyway. + +This capability can be hidden in a library or in the framework you use; +the end user's code does not need to be explicitly aware of using +threads. For a simple example of this, there is `transaction.py`_ in +``lib_pypy``. The idea is that you write, or already have, some program +where the function ``f(key, value)`` runs on every item of some big +dictionary, say:: + + for key, value in bigdict.items(): + f(key, value) + +Then you simply replace the loop with:: for key, value in bigdict.items(): transaction.add(f, key, value) transaction.run() This code runs the various calls to ``f(key, value)`` using a thread -pool, but every single call is done in an atomic section. The end -result is that the behavior should be exactly equivalent: you don't get -any extra multithreading issue. +pool, but every single call is executed under the protection of a unique +lock. The end result is that the behavior is exactly equivalent --- in +fact it makes little sense to do it in this way on a non-STM PyPy or on +CPython. But on ``pypy-stm``, the various locked calls to ``f(key, +value)`` can tentatively be executed in parallel, even if the observable +result is as if they were executed in some serial order. This approach hides the notion of threads from the end programmer, including all the hard multithreading-related issues. This is not the @@ -223,41 +319,176 @@ only requires that the end programmer identifies where this parallelism is likely to be found, and communicates it to the system, using for example the ``transaction.add()`` scheme. - + .. _`transaction.py`: https://bitbucket.org/pypy/pypy/raw/stmgc-c7/lib_pypy/transaction.py .. _OpenMP: http://en.wikipedia.org/wiki/OpenMP -================== -Other APIs in pypy-stm: +.. _`transactional_memory`: -* ``__pypy__.thread.getsegmentlimit()``: return the number of "segments" - in this pypy-stm. This is the limit above which more threads will not - be able to execute on more cores. (Right now it is limited to 4 due - to inter-segment overhead, but should be increased in the future. It +API of transactional_memory +--------------------------- + +The new pure Python module ``transactional_memory`` runs on both CPython +and PyPy, both with and without STM. It contains: + +* ``getsegmentlimit()``: return the number of "segments" in + this pypy-stm. This is the limit above which more threads will not be + able to execute on more cores. (Right now it is limited to 4 due to + inter-segment overhead, but should be increased in the future. It should also be settable, and the default value should depend on the - number of actual CPUs.) + number of actual CPUs.) If STM is not available, this returns 1. -* ``__pypy__.thread.exclusive_atomic``: same as ``atomic``, but - raises an exception if you attempt to nest it inside another - ``atomic``. +* ``print_abort_info(minimum_time=0.0)``: debugging help. Each thread + remembers the longest abort or pause it did because of cross-thread + contention_. This function prints it to ``stderr`` if the time lost + is greater than ``minimum_time`` seconds. The record is then + cleared, to make it ready for new events. This function returns + ``True`` if it printed a report, and ``False`` otherwise. -* ``__pypy__.thread.signals_enabled``: a context manager that runs - its block with signals enabled. By default, signals are only - enabled in the main thread; a non-main thread will not receive - signals (this is like CPython). Enabling signals in non-main threads - is useful for libraries where threads are hidden and the end user is - not expecting his code to run elsewhere than in the main thread. -Note that all of this API is (or will be) implemented in a regular PyPy -too: for example, ``with atomic`` will simply mean "don't release the -GIL" and ``getsegmentlimit()`` will return 1. +API of __pypy__.thread +---------------------- -================== +The ``__pypy__.thread`` submodule is a built-in module of PyPy that +contains a few internal built-in functions used by the +``transactional_memory`` module, plus the following: + +* ``__pypy__.thread.atomic``: a context manager to run a block in + fully atomic mode, without "releasing the GIL". (May be eventually + removed?) + +* ``__pypy__.thread.signals_enabled``: a context manager that runs its + block with signals enabled. By default, signals are only enabled in + the main thread; a non-main thread will not receive signals (this is + like CPython). Enabling signals in non-main threads is useful for + libraries where threads are hidden and the end user is not expecting + his code to run elsewhere than in the main thread. + + +.. _contention: + +Conflicts +--------- + +Based on Software Transactional Memory, the ``pypy-stm`` solution is +prone to "conflicts". To repeat the basic idea, threads execute their code +speculatively, and at known points (e.g. between bytecodes) they +coordinate with each other to agree on which order their respective +actions should be "committed", i.e. become globally visible. Each +duration of time between two commit-points is called a transaction. + +A conflict occurs when there is no consistent ordering. The classical +example is if two threads both tried to change the value of the same +global variable. In that case, only one of them can be allowed to +proceed, and the other one must be either paused or aborted (restarting +the transaction). If this occurs too often, parallelization fails. + +How much actual parallelization a multithreaded program can see is a bit +subtle. Basically, a program not using ``__pypy__.thread.atomic`` or +eliding locks, or doing so for very short amounts of time, will +parallelize almost freely (as long as it's not some artificial example +where, say, all threads try to increase the same global counter and do +nothing else). + +However, using if the program requires longer transactions, it comes +with less obvious rules. The exact details may vary from version to +version, too, until they are a bit more stabilized. Here is an +overview. + +Parallelization works as long as two principles are respected. The +first one is that the transactions must not *conflict* with each other. +The most obvious sources of conflicts are threads that all increment a +global shared counter, or that all store the result of their +computations into the same list --- or, more subtly, that all ``pop()`` +the work to do from the same list, because that is also a mutation of +the list. (It is expected that some STM-aware library will eventually +be designed to help with conflict problems, like a STM-aware queue.) + +A conflict occurs as follows: when a transaction commits (i.e. finishes +successfully) it may cause other transactions that are still in progress +to abort and retry. This is a waste of CPU time, but even in the worst +case senario it is not worse than a GIL, because at least one +transaction succeeds (so we get at worst N-1 CPUs doing useless jobs and +1 CPU doing a job that commits successfully). + +Conflicts do occur, of course, and it is pointless to try to avoid them +all. For example they can be abundant during some warm-up phase. What +is important is to keep them rare enough in total. + +Another issue is that of avoiding long-running so-called "inevitable" +transactions ("inevitable" is taken in the sense of "which cannot be +avoided", i.e. transactions which cannot abort any more). Transactions +like that should only occur if you use ``__pypy__.thread.atomic``, +generally become of I/O in atomic blocks. They work, but the +transaction is turned inevitable before the I/O is performed. For all +the remaining execution time of the atomic block, they will impede +parallel work. The best is to organize the code so that such operations +are done completely outside ``__pypy__.thread.atomic``. + +(This is related to the fact that blocking I/O operations are +discouraged with Twisted, and if you really need them, you should do +them on their own separate thread.) + +In case of lock elision, we don't get long-running inevitable +transactions, but a different problem can occur: doing I/O cancels lock +elision, and the lock turns into a real lock, preventing other threads +from committing if they also need this lock. (More about it when lock +elision is implemented and tested.) + + + +Implementation +============== + +XXX this section mostly empty for now + + +Low-level statistics +-------------------- + +When a non-main thread finishes, you get low-level statistics printed to +stderr, looking like that:: + + thread 0x7f73377fe600: + outside transaction 42182 0.506 s + run current 85466 0.000 s + run committed 34262 3.178 s + run aborted write write 6982 0.083 s + run aborted write read 550 0.005 s + run aborted inevitable 388 0.010 s + run aborted other 0 0.000 s + wait free segment 0 0.000 s + wait write read 78 0.027 s + wait inevitable 887 0.490 s + wait other 0 0.000 s + sync commit soon 1 0.000 s + bookkeeping 51418 0.606 s + minor gc 162970 1.135 s + major gc 1 0.019 s + sync pause 59173 1.738 s + longest recordered marker 0.000826 s + "File "x.py", line 5, in f" + +On each line, the first number is a counter, and the second number gives +the associated time --- the amount of real time that the thread was in +this state. The sum of all the times should be equal to the total time +between the thread's start and the thread's end. The most important +points are "run committed", which gives the amount of useful work, and +"outside transaction", which should give the time spent e.g. in library +calls (right now it seems to be larger than that; to investigate). The +various "run aborted" and "wait" entries are time lost due to +conflicts_. Everything else is overhead of various forms. (Short-, +medium- and long-term future work involves reducing this overhead :-) + +The last two lines are special; they are an internal marker read by +``transactional_memory.print_abort_info()``. + +These statistics are not printed out for the main thread, for now. Reference to implementation details -=================================== +----------------------------------- The core of the implementation is in a separate C library called stmgc_, in the c7_ subdirectory. Please see the `README.txt`_ for more @@ -282,3 +513,15 @@ .. __: https://bitbucket.org/pypy/pypy/raw/stmgc-c7/rpython/translator/stm/src_stm/stmgcintf.c .. __: https://bitbucket.org/pypy/pypy/raw/stmgc-c7/rpython/jit/backend/llsupport/stmrewrite.py .. __: https://bitbucket.org/pypy/pypy/raw/stmgc-c7/rpython/jit/backend/x86/assembler.py + + + +See also +======== + +See also +https://bitbucket.org/pypy/pypy/raw/default/pypy/doc/project-ideas.rst +(section about STM). + + +.. include:: _ref.txt _______________________________________________ pypy-commit mailing list [email protected] https://mail.python.org/mailman/listinfo/pypy-commit
