Author: Maciej Fijalkowski <[email protected]> Branch: extradoc Changeset: r4957:48522e7e372b Date: 2013-03-15 09:42 -0700 http://bitbucket.org/pypy/extradoc/changeset/48522e7e372b/
Log: (fijal, arigo) in the typical pypy style copy-paste beamerdefs diff --git a/talk/pycon2013/pypy_without_gil/Makefile b/talk/pycon2013/pypy_without_gil/Makefile new file mode 100644 --- /dev/null +++ b/talk/pycon2013/pypy_without_gil/Makefile @@ -0,0 +1,16 @@ +# you can find rst2beamer.py and inkscapeslide.py here: +# http://bitbucket.org/antocuni/env/src/619f486c4fad/bin/rst2beamer.py +# http://bitbucket.org/antocuni/env/src/619f486c4fad/bin/inkscapeslide.py + + +talk.pdf: talk.rst author.latex title.latex stylesheet.latex + rst2beamer --stylesheet=stylesheet.latex --documentoptions=14pt talk.rst talk.latex || exit + sed 's/\\date{}/\\input{author.latex}/' -i talk.latex || exit + #sed 's/\\maketitle/\\input{title.latex}/' -i talk.latex || exit + pdflatex talk.latex || exit + +view: talk.pdf + evince talk.pdf & + +xpdf: talk.pdf + xpdf talk.pdf & diff --git a/talk/pycon2013/pypy_without_gil/author.latex b/talk/pycon2013/pypy_without_gil/author.latex new file mode 100644 --- /dev/null +++ b/talk/pycon2013/pypy_without_gil/author.latex @@ -0,0 +1,8 @@ +\definecolor{rrblitbackground}{rgb}{0.0, 0.0, 0.0} + +\title[Python without the GIL]{Python without the GIL} +\author[arigo, fijal] +{Armin Rigo \\ Maciej Fijałkowski} + +\institute{Pycon US 2013} +\date{March 15 2013} diff --git a/talk/pycon2013/pypy_without_gil/beamerdefs.txt b/talk/pycon2013/pypy_without_gil/beamerdefs.txt new file mode 100644 --- /dev/null +++ b/talk/pycon2013/pypy_without_gil/beamerdefs.txt @@ -0,0 +1,108 @@ +.. colors +.. =========================== + +.. role:: green +.. role:: red + + +.. general useful commands +.. =========================== + +.. |pause| raw:: latex + + \pause + +.. |small| raw:: latex + + {\small + +.. |end_small| raw:: latex + + } + +.. |scriptsize| raw:: latex + + {\scriptsize + +.. |end_scriptsize| raw:: latex + + } + +.. |strike<| raw:: latex + + \sout{ + +.. closed bracket +.. =========================== + +.. |>| raw:: latex + + } + + +.. example block +.. =========================== + +.. |example<| raw:: latex + + \begin{exampleblock}{ + + +.. |end_example| raw:: latex + + \end{exampleblock} + + + +.. alert block +.. =========================== + +.. |alert<| raw:: latex + + \begin{alertblock}{ + + +.. |end_alert| raw:: latex + + \end{alertblock} + + + +.. columns +.. =========================== + +.. |column1| raw:: latex + + \begin{columns} + \begin{column}{0.45\textwidth} + +.. |column2| raw:: latex + + \end{column} + \begin{column}{0.45\textwidth} + + +.. |end_columns| raw:: latex + + \end{column} + \end{columns} + + + +.. |snake| image:: ../../img/py-web-new.png + :scale: 15% + + + +.. nested blocks +.. =========================== + +.. |nested| raw:: latex + + \begin{columns} + \begin{column}{0.85\textwidth} + +.. |end_nested| raw:: latex + + \end{column} + \end{columns} diff --git a/talk/pycon2013/pypy_without_gil/pypy_without_gil.txt b/talk/pycon2013/pypy_without_gil/pypy_without_gil.txt --- a/talk/pycon2013/pypy_without_gil/pypy_without_gil.txt +++ b/talk/pycon2013/pypy_without_gil/pypy_without_gil.txt @@ -46,7 +46,7 @@ * needs careful usage of locks * mostly hindered by the GIL in CPython - (but not in Jython) + (but not in Jython or IronPython) Classic solutions @@ -56,9 +56,12 @@ * no shared mutable state at all -* copying, keeping in sync are your problem +* copying, keeping in sync is your problem -* memory usage is often multiplied (unless you're lucky with ``fork``) +* serializing and deserializing is expensive and hard + +* memory usage is often multiplied (unless you're lucky with ``fork``, + but not on Python) Classic solutions @@ -68,7 +71,7 @@ * MPI: message passing, with limited shared state -* etc.: tons of experiments that never caught on +* etc.: tons of experiments that never caught on in the mainstream Classic solutions @@ -84,7 +87,7 @@ Demo ---------------------------- +---- * ``pypy-stm`` @@ -92,7 +95,7 @@ Demo ---------------------------- +---- - demo: @@ -104,13 +107,18 @@ * transactions2.py +Status of the implementation +---------------------------- +* mostly works -- missing in pypy-stm: +* major GC collections missing (leaks slowly) - * GC major collection - * JIT - * some more optimizations possible (= tons of) +* JIT integration is not done + +* tons of optimizations possible + + - __pypy__.thread.atomic diff --git a/talk/pycon2013/pypy_without_gil/stylesheet.latex b/talk/pycon2013/pypy_without_gil/stylesheet.latex new file mode 100644 --- /dev/null +++ b/talk/pycon2013/pypy_without_gil/stylesheet.latex @@ -0,0 +1,11 @@ +\usetheme{Boadilla} +\usecolortheme{whale} +\setbeamercovered{transparent} +\setbeamertemplate{navigation symbols}{} + +\definecolor{darkgreen}{rgb}{0, 0.5, 0.0} +\newcommand{\docutilsrolegreen}[1]{\color{darkgreen}#1\normalcolor} +\newcommand{\docutilsrolered}[1]{\color{red}#1\normalcolor} + +\newcommand{\green}[1]{\color{darkgreen}#1\normalcolor} +\newcommand{\red}[1]{\color{red}#1\normalcolor} diff --git a/talk/pycon2013/pypy_without_gil/talk.rst b/talk/pycon2013/pypy_without_gil/talk.rst new file mode 100644 --- /dev/null +++ b/talk/pycon2013/pypy_without_gil/talk.rst @@ -0,0 +1,233 @@ +.. include:: beamerdefs.txt + +==================== +PyPy without the GIL +==================== + +Intro +--------------------------- + +- PyPy intro + +- thanks to contributors for STM + + +Problem +------- + +* An existing complex, large program that does stuff + +* "stuff" consists of bits that are mostly independent from each other + +|pause| + +* ... but not quite + + +Problem +------- + +* We want to parallelize the program to use all these cores + +* We have some shared mutable state + +|pause| + +* Not too much of it --- otherwise, no chance for parallelism + +* But still some + + +Classic solutions +----------------- + +Bare-metal multi-threading: + +* large shared state + +* needs careful usage of locks + +* mostly hindered by the GIL in CPython + (but not in Jython or IronPython) + + +Classic solutions +----------------- + +Multi-processing: + +* no shared mutable state at all + +* copying, keeping in sync is your problem + +* serializing and deserializing is expensive and hard + +* memory usage is often multiplied (unless you're lucky with ``fork``, + but not on Python) + + +Classic solutions +----------------- + +A range of intermediate solutions: + +* MPI: message passing, with limited shared state + +* etc.: tons of experiments that never caught on in the mainstream + + +Classic solutions +----------------- + +The typical solution for web servers: + +* run independent processes + +* share data only via the database + +* the database itself handles concurrency with transactions + + +Demo +---- + +* ``pypy-stm`` + +* internally based on "transactions" (STM, HTM) + + +Demo +---- + +- demo: + + * multithreading.py + + * multithreading2.py + + * message_passing.py + + * transactions2.py + +Status of the implementation +---------------------------- + +* mostly works + +* major GC collections missing (leaks slowly) + +* JIT integration is not done + +* tons of optimizations possible + + + + +- __pypy__.thread.atomic + + * with atomic: + print "hello", username + + * the illusion of serialization + + +Lower level: STM +--------------------------- + +- pypy and not cpython? + + +- STM = Software Transactional Memory + + +- Basic idea: each thread runs in parallel, but everything it does is + in a series of "transactions" + + +- A transaction keeps all changes to pre-existing memory "local" + + +- The changes are made visible only when the transaction "commits" + + +- The transaction will "abort" if a conflict is detected, and + it will be transparently retried + + +- Non-reversible operations like I/O turn the transaction "inevitable" + and stop progress in the other threads + + +- __pypy__.thread.last_abort_info() -> traceback-like information + + +- (GC-supported structure: when we want to modify a pre-existing object, + we first copy it into "local" memory, and when we commit, it becomes + the newer version of the old pre-existing object; we end up with a + chained list of versions, and we have to make sure we always use the + latest one. We rely on GC major collections to free them eventually.) + + +- alternative: HTM...? + + +Higher level: not threads +--------------------------- + + +- xxx memory usage good + + +- based on (and fully compatible with) threads + + * existing multithreaded programs work + + +- but opens up unexpected alternatives + + * we can run multiple threads but at the same time use ``atomic`` + + * with the GIL-based idea of ``atomic`` it doesn't make sense: + we have multiple threads but they're all using ``atomic``, i.e. + only one at a time will ever run... except no :-) + + +- transaction.py + + * demo + + * illusion of serial execution: can "sanely" reason about algorithms + + * "Conflict tracebacks" + + * Might need to debug them --- but they are performance bugs, not + correctness bugs + + * The idea is that we fix only XX% of the bugs and we are done + + * Maybe some new "performance debugger" tools might help too + + +- TigerQuoll, 1st March: same idea with JavaScript (for servers) + + * various models possible: + + . events dispatchers + + . futures + + . map/reduce, scatter/gather + + +- Event dispatchers + + * twisted, tulip, etc. + + * run the event dispatcher in one thread (e.g. the main thread), + and schedule the actual events to run on a different thread from + a pool + + * the events are run with ``atomic``, so that they appear to run + serially + + * does not rely on any change to the user program _______________________________________________ pypy-commit mailing list [email protected] http://mail.python.org/mailman/listinfo/pypy-commit
