Author: Philip Jenvey <pjen...@underboss.org> Branch: extradoc Changeset: r5151:1694aba0aca8 Date: 2014-02-07 11:35 -0800 http://bitbucket.org/pypy/extradoc/changeset/1694aba0aca8/
Log: merge upstream diff --git a/planning/jit.txt b/planning/jit.txt --- a/planning/jit.txt +++ b/planning/jit.txt @@ -48,6 +48,18 @@ - ovfcheck(a << b) will do ``result >> b`` and check that the result is equal to ``a``, instead of looking at the x86 flags. +- Track whether floats in the JIT could possibly have overflowed into + ``inf``/``nan`` + + f81 = cast_int_to_float(i79) + f82 = float_add(f81, 11235582092889474423308157442431404585112356118389416079589380072358292237843810195794279832650471001320007117491962084853674360550901038905802964414967132773610493339054092829768888725077880882465817684505312860552384417646403930092119569408801702322709406917786643639996702871154982269052209770601514008576.000000) + i83 = float_eq(f82, f81) + guard_false(i83, descr=<Guard0x104d2c6b0>) + + For example, here this is the test for ``isinf(i81)``, but it's impossible + for ``i81`` to be ``inf`` because ``float(sys.maxint)`` is a finite value. + + OPTIMIZATIONS ------------- diff --git a/sprintinfo/leysin-winter-2014/announcement.txt b/sprintinfo/leysin-winter-2014/announcement.txt new file mode 100644 --- /dev/null +++ b/sprintinfo/leysin-winter-2014/announcement.txt @@ -0,0 +1,62 @@ +===================================================================== + PyPy Leysin Winter Sprint (11-19st January 2014) +===================================================================== + +The next PyPy sprint will be in Leysin, Switzerland, for the ninth time. +This is a fully public sprint: newcomers and topics other than those +proposed below are welcome. + +------------------------------ +Goals and topics of the sprint +------------------------------ + +* Py3k: work towards supporting Python 3 in PyPy + +* NumPyPy: work towards supporting the numpy module in PyPy + +* STM: work towards supporting Software Transactional Memory + +* And as usual, the main side goal is to have fun in winter sports :-) + We can take a day off for ski. + +----------- +Exact times +----------- + +For a change, and as an attempt to simplify things, I specified the +dates as 11-19 January 2014, where 11 and 19 are travel days. We will +work full days between the 12 and the 18. You are of course allowed to +show up for a part of that time only, too. + +----------------------- +Location & Accomodation +----------------------- + +Leysin, Switzerland, "same place as before". Let me refresh your +memory: both the sprint venue and the lodging will be in a very spacious +pair of chalets built specifically for bed & breakfast: +http://www.ermina.ch/. The place has a good ADSL Internet connexion +with wireless installed. You can of course arrange your own lodging +anywhere (as long as you are in Leysin, you cannot be more than a 15 +minutes walk away from the sprint venue), but I definitely recommend +lodging there too -- you won't find a better view anywhere else (though +you probably won't get much worse ones easily, either :-) + +Please *confirm* that you are coming so that we can adjust the +reservations as appropriate. The rate so far has been around 60 CHF a +night all included in 2-person rooms, with breakfast. There are larger +rooms too (less expensive per person) and maybe the possibility to get a +single room if you really want to. + +Please register by Mercurial:: + + https://bitbucket.org/pypy/extradoc/ + https://bitbucket.org/pypy/extradoc/raw/extradoc/sprintinfo/leysin-winter-2014 + +or on the pypy-dev mailing list if you do not yet have check-in rights: + + http://mail.python.org/mailman/listinfo/pypy-dev + +You need a Swiss-to-(insert country here) power adapter. There will be +some Swiss-to-EU adapters around -- bring a EU-format power strip if you +have one. diff --git a/sprintinfo/leysin-winter-2014/people.txt b/sprintinfo/leysin-winter-2014/people.txt new file mode 100644 --- /dev/null +++ b/sprintinfo/leysin-winter-2014/people.txt @@ -0,0 +1,63 @@ + +People coming to the Leysin sprint Winter 2014 +================================================== + +People who have a ``?`` in their arrive/depart or accomodation +column are known to be coming but there are no details +available yet from them. + + +==================== ============== ======================= + Name Arrive/Depart Accomodation +==================== ============== ======================= +Armin Rigo private +Romain Guillebert 11-19 Ermina +Christian Clauss 11-12 & 18-19 I live nearby +Maciej Fijalkowski 11-18 Ermina +Remi Meier 11-19 Ermina +Johan Råde 11-18 Ermina +Antonio Cuni 14-18 Ermina +Manuel Jacob 12-19 private +==================== ============== ======================= + + +People on the following list were present at previous sprints: + +==================== ============== ===================== + Name Arrive/Depart Accomodation +==================== ============== ===================== +Romain Guillebert ? ? +Michael Foord ? ? +David Schneider ? ? +Jacob Hallen ? ? +Laura Creighton ? ? +Hakan Ardo ? ? +Carl Friedrich Bolz ? ? +Samuele Pedroni ? ? +Anders Hammarquist ? ? +Christian Tismer ? ? +Niko Matsakis ? ? +Toby Watson ? ? +Paul deGrandis ? ? +Michael Hudson ? ? +Anders Lehmann ? ? +Niklaus Haldimann ? ? +Lene Wagner ? ? +Amaury Forgeot d'Arc ? ? +Valentino Volonghi ? ? +Boris Feigin ? ? +Andrew Thompson ? ? +Bert Freudenberg ? ? +Beatrice Duering ? ? +Richard Emslie ? ? +Johan Hahn ? ? +Stephan Diehl ? ? +Alexander Schremmer ? ? +Anders Chrigstroem ? ? +Eric van Riet Paap ? ? +Holger Krekel ? ? +Guido Wesdorp ? ? +Leonardo Santagada ? ? +Alexandre Fayolle ? ? +Sylvain Thénault ? ? +==================== ============== ===================== diff --git a/sprintinfo/leysin-winter-2014/planning.txt b/sprintinfo/leysin-winter-2014/planning.txt new file mode 100644 --- /dev/null +++ b/sprintinfo/leysin-winter-2014/planning.txt @@ -0,0 +1,46 @@ + +People +------ + +Johan Rade +Remi Meier +Maciej Fijalkowski +Romain Guillebert +Armin Rigo +Manuel Jacob +Antonio Cuni + +Topics +------ + +* numpy stuff, fix bugs from bug tracker (rguillebert, antocuni around) + +* look at codespeed2 + +* resume-refactor branch (fijal, rguillebert) MORE PROGRESS + +* GC pinning + +* asmgcc bug with greenlets and --shared (FIXED) + +* think about --shared by default + +* CFFI 1.0 + +* STM (remi, armin) DONE in transaction breaks, started c7 + +* discuss about C++ / cppyy, look into importing pyshiboken (johan pessimistic, ?) + +* try cppyy to run on windows (johan) IN PROGRESS + +* ctypes: https://bugs.pypy.org/issue1671 DONE + +* longs multiplication: patch at https://bugs.pypy.org/issue892 + +* look into merging refactor-str-types (mjacob, antocuni) FIX TRANSLATION + +* tweaking ast classes: https://bugs.pypy.org/issue1673 (mjacob) + +* skiing (fijal, DONE) + +* add jit_merge_point to tuple_contains (anybody) diff --git a/talk/ep2014/status/abstract.rst b/talk/ep2014/status/abstract.rst new file mode 100644 --- /dev/null +++ b/talk/ep2014/status/abstract.rst @@ -0,0 +1,36 @@ +PyPy status talk (a.k.a.: no no, PyPy is not dead) +=================================================== + +Abstract +-------- + +The current status of PyPy, with a particular focus on what happened in +the last two years, since the last EuroPython PyPy talk. We will give a +brief overview of the current speed and the on-going development efforts +on the JIT, the GC, NumPy, Python 3 compatibility, CFFI, STM... + + +Description +----------- + +In this talk we will present the current status of PyPy, with a +particular focus on what happened in the last two years, since the last +EuroPython PyPy talk. We will give an overview of the current speed and +the on-going development efforts, including but not limited to: + +- the status of the Just-in-Time Compiler (JIT) and PyPy performance in + general; + +- the improvements on the Garbage Collector (GC); + +- the status of the NumPy and Python 3 compatibility subprojects; + +- CFFI, which aims to be a general C interface mechanism for both + CPython and PyPy; + +- a quick overview of the STM (Software Transactional Memory) research + project, which aims to solve the GIL problem. + +This is the "general PyPy status talk" that we give every year at +EuroPython (except last year; hence the "no no, PyPy is not dead" part +of the title of this talk). diff --git a/talk/ep2014/stm/abstract.rst b/talk/ep2014/stm/abstract.rst new file mode 100644 --- /dev/null +++ b/talk/ep2014/stm/abstract.rst @@ -0,0 +1,41 @@ +Using All These Cores: Transactional Memory in PyPy +=================================================== + +Abstract +-------- + +PyPy, the Python implementation written in Python, experimentally +supports Transactional Memory (TM). The strength of TM is to enable a +novel use of multithreading, inheritently safe, and not limited to +special use cases like other approaches. This talk will focus on how it +works under the hood. + + +Description +----------- + +PyPy is a fast alternative Python implementation. Software +Transactional Memory (STM) is a current academic research topic. Put +the two together --brew for a couple of years-- and we get a version of +PyPy that runs on multiple cores, without the infamous Global +Interpreter Lock (GIL). + +The current research is based on a recent new insight that promises to +give really good performance. The speed of STM is generally measured by +two factors: the ability to scale with the number of CPUs, and the +amount of overhead when compared with other approaches in a single CPU +(in this case, with the regular PyPy with the GIL). Scaling is not +really a problem here, but single-CPU performance is --or used to be. +This new approach gives a single-threaded overhead that should be very +low, maybe 20%, which would definitely be news for STM systems. Right +now (February 2014) we are still implementing it, so we cannot give +final numbers yet, but early results on a small interpreter for a custom +language are around 15%. This looks like a deal-changer for STM. + +In the talk, I will describe our progress, hopefully along with real +numbers and demos. I will then dive under the hood of PyPy to give an +idea about how it works. I will conclude with a picture of how the +future of multi-threaded programming might looks like, for high-level +languages like Python. I will also mention CPython: how hard (or not) +it would be to change the CPython source code to use the same approach. + diff --git a/talk/fosdem2014/Makefile b/talk/fosdem2014/Makefile new file mode 100644 --- /dev/null +++ b/talk/fosdem2014/Makefile @@ -0,0 +1,10 @@ +# Note to myself (arigo): run in the 64-bit environment + +pypy-stm.pdf: pypy-stm.tex + pdflatex pypy-stm.tex + +pypy-stm.tex: pypy-stm.rst + rst2beamer.py --stylesheet=stylesheet.latex --documentoptions=14pt --input-encoding=utf8 --output-encoding=utf8 $< | python expand-itemize.py > pypy-stm.tex + +clean: + rm -f pypy-stm.tex pypy-stm.pdf diff --git a/talk/fosdem2014/expand-itemize.py b/talk/fosdem2014/expand-itemize.py new file mode 100644 --- /dev/null +++ b/talk/fosdem2014/expand-itemize.py @@ -0,0 +1,10 @@ +import sys + +def expand(in_file, out_file): + for line in in_file: + line = line.replace(r'\begin{itemize}', + r'\begin{itemize}\setlength{\itemsep}{10pt}') + out_file.write(line) + +if __name__ == '__main__': + expand(sys.stdin, sys.stdout) diff --git a/talk/fosdem2014/pypy-jit/Makefile b/talk/fosdem2014/pypy-jit/Makefile new file mode 100644 --- /dev/null +++ b/talk/fosdem2014/pypy-jit/Makefile @@ -0,0 +1,16 @@ +# you can find rst2beamer.py and inkscapeslide.py here: +# http://bitbucket.org/antocuni/env/src/619f486c4fad/bin/rst2beamer.py +# http://bitbucket.org/antocuni/env/src/619f486c4fad/bin/inkscapeslide.py + + +talk.pdf: talk.rst author.latex stylesheet.latex + rst2beamer.py --input-encoding=utf8 --output-encoding=utf8 --stylesheet=stylesheet.latex --documentoptions=14pt talk.rst talk.latex || exit + sed 's/\\date{}/\\input{author.latex}/' -i talk.latex || exit + #sed 's/\\maketitle/\\input{title.latex}/' -i talk.latex || exit + pdflatex talk.latex || exit + +view: talk.pdf + evince talk.pdf > /dev/null 2>&1 & + +xpdf: talk.pdf + xpdf talk.pdf & diff --git a/talk/fosdem2014/pypy-jit/Speed.png b/talk/fosdem2014/pypy-jit/Speed.png new file mode 100644 index 0000000000000000000000000000000000000000..796a1ed2ef8f48d701a54242e78694ac16a70762 GIT binary patch [cut] diff --git a/talk/fosdem2014/pypy-jit/author.latex b/talk/fosdem2014/pypy-jit/author.latex new file mode 100644 --- /dev/null +++ b/talk/fosdem2014/pypy-jit/author.latex @@ -0,0 +1,8 @@ +\definecolor{rrblitbackground}{rgb}{0.0, 0.0, 0.0} + +\title[How PyPy makes your code run fast]{How PyPy makes your code run fast} +\author[rguillebert] +{Romain Guillebert} + +\institute{FOSDEM} +\date{February 2nd, 2014} diff --git a/talk/fosdem2014/pypy-jit/beamerdefs.txt b/talk/fosdem2014/pypy-jit/beamerdefs.txt new file mode 100644 --- /dev/null +++ b/talk/fosdem2014/pypy-jit/beamerdefs.txt @@ -0,0 +1,108 @@ +.. colors +.. =========================== + +.. role:: green +.. role:: red + + +.. general useful commands +.. =========================== + +.. |pause| raw:: latex + + \pause + +.. |small| raw:: latex + + {\small + +.. |end_small| raw:: latex + + } + +.. |scriptsize| raw:: latex + + {\scriptsize + +.. |end_scriptsize| raw:: latex + + } + +.. |strike<| raw:: latex + + \sout{ + +.. closed bracket +.. =========================== + +.. |>| raw:: latex + + } + + +.. example block +.. =========================== + +.. |example<| raw:: latex + + \begin{exampleblock}{ + + +.. |end_example| raw:: latex + + \end{exampleblock} + + + +.. alert block +.. =========================== + +.. |alert<| raw:: latex + + \begin{alertblock}{ + + +.. |end_alert| raw:: latex + + \end{alertblock} + + + +.. columns +.. =========================== + +.. |column1| raw:: latex + + \begin{columns} + \begin{column}{0.45\textwidth} + +.. |column2| raw:: latex + + \end{column} + \begin{column}{0.45\textwidth} + + +.. |end_columns| raw:: latex + + \end{column} + \end{columns} + + + +.. |snake| image:: ../../img/py-web-new.png + :scale: 15% + + + +.. nested blocks +.. =========================== + +.. |nested| raw:: latex + + \begin{columns} + \begin{column}{0.85\textwidth} + +.. |end_nested| raw:: latex + + \end{column} + \end{columns} diff --git a/talk/fosdem2014/pypy-jit/rst2beamer.py b/talk/fosdem2014/pypy-jit/rst2beamer.py new file mode 100755 --- /dev/null +++ b/talk/fosdem2014/pypy-jit/rst2beamer.py @@ -0,0 +1,267 @@ +#!/usr/bin/env python +# encoding: utf-8 +""" +A docutils script converting restructured text into Beamer-flavoured LaTeX. + +Beamer is a LaTeX document class for presentations. Via this script, ReST can +be used to prepare slides. It can be called:: + + rst2beamer.py infile.txt > outfile.tex + +where ``infile.tex`` contains the produced Beamer LaTeX. + +See <http:www.agapow.net/programming/python/rst2beamer> for more details. + +""" +# TODO: modifications for handout sections? +# TOOD: sections and subsections? +# TODO: enable beamer themes? +# TODO: convert document metadata to front page fields? +# TODO: toc-conversion? +# TODO: fix descriptions + +# Unless otherwise stated, created by P-M Agapow on 2007-08-21 +# and open for academic & non-commercial use and modification . + +__docformat__ = 'restructuredtext en' +__author__ = "Paul-Michael Agapow <aga...@bbsrc.ac.uk>" +__version__ = "0.2" + + +### IMPORTS ### + +import locale +from docutils.core import publish_cmdline, default_description +from docutils.writers.latex2e import Writer as Latex2eWriter +from docutils.writers.latex2e import LaTeXTranslator, DocumentClass +from docutils import nodes + +## Syntax highlighting: + +""" + .. sourcecode:: python + + My code goes here. + + + :copyright: 2007 by Georg Brandl. + :license: BSD, see LICENSE for more details. +""" + +from pygments.formatters import HtmlFormatter, LatexFormatter + +# The default formatter +DEFAULT = LatexFormatter() + + +from docutils.parsers.rst import directives + +from pygments import highlight +from pygments.lexers import get_lexer_by_name, TextLexer + +VARIANTS = { + 'linenos': LatexFormatter(linenos=True), +} + +def pygments_directive(name, arguments, options, content, lineno, + content_offset, block_text, state, state_machine): + try: + lexer = get_lexer_by_name(arguments[0]) + except ValueError: + # no lexer found - use the text one instead of an exception + lexer = TextLexer() + formatter = DEFAULT + parsed = highlight(u'\n'.join(content), lexer, formatter) + return [nodes.raw('', parsed, format='latex')] + +pygments_directive.arguments = (1, 0, 1) +pygments_directive.content = 1 +pygments_directive.options = dict([(key, directives.flag) for key in VARIANTS]) + +directives.register_directive('sourcecode', pygments_directive) + + +## multiple images as a single animation + +""" + .. animage:: foo-p*.pdf + :align: center + :scale: 50% +""" + +from glob import glob +import copy +from docutils.parsers.rst import directives +from docutils.parsers.rst.directives.images import Image +import docutils + +class Animage(Image): # Animated Image :-) + + def run(self): + def raw(text): + return docutils.nodes.raw('', text, format='latex') + + nodes = Image.run(self) + img = nodes[0] + if not isinstance(img, docutils.nodes.image): + return nodes # not an image, WTF? + newnodes = [] + pattern = img.attributes['uri'] + filenames = sorted(glob(pattern)) + for i, filename in enumerate(filenames): + newimg = copy.deepcopy(img) + newimg.attributes['uri'] = filename + newnodes += [raw(r'\only<%d>{' % (i+1)), + newimg, + raw('}')] + return newnodes + +directives.register_directive('animage', Animage) + + + + +## CONSTANTS & DEFINES: ### + +BEAMER_SPEC = ( + 'Beamer options', + 'These are derived almost entirely from the LaTeX2e options', + tuple ( + [ + ( + 'Specify theme.', + ['--theme'], + {'default': '', } + ), + ( + 'Specify document options. Multiple options can be given, ' + 'separated by commas. Default is "10pt,a4paper".', + ['--documentoptions'], + {'default': '', } + ), + ] + list (Latex2eWriter.settings_spec[2][2:]) + ), +) + +BEAMER_DEFAULTS = { + 'output_encoding': 'latin-1', + 'documentclass': 'beamer', +} + + +### IMPLEMENTATION ### + +try: + locale.setlocale (locale.LC_ALL, '') +except: + pass + +class BeamerTranslator (LaTeXTranslator): + """ + A converter for docutils elements to beamer-flavoured latex. + """ + + def __init__ (self, document): + LaTeXTranslator.__init__ (self, document) + self.head_prefix = [x for x in self.head_prefix if ('{typearea}' not in x)] + hyperref_posn = [i for i in range (len (self.head_prefix)) if ('{hyperref}' in self.head_prefix[i])] + if not hyperref_posn: + self.head_prefix.append(None) + hyperref_posn = [-1] # XXX hack + self.head_prefix[hyperref_posn[0]] = ('\\usepackage{hyperref}\n' + + '\\usepackage{fancyvrb}\n' + + LatexFormatter(style="manni").get_style_defs() + + "\n") + + self.head_prefix.extend ([ + '\\definecolor{rrblitbackground}{rgb}{0.55, 0.3, 0.1}\n', + '\\newenvironment{rtbliteral}{\n', + '\\begin{ttfamily}\n', + '\\color{rrblitbackground}\n', + '}{\n', + '\\end{ttfamily}\n', + '}\n', + ]) + # this fixes the hardcoded section titles in docutils 0.4 + self.d_class = DocumentClass ('article') + + def begin_frametag (self, node): + if "verbatim" in str(node).lower(): + return '\\begin{frame}[containsverbatim,fragile]\n' + else: + return '\\begin{frame}\n' + + def end_frametag (self): + return '\\end{frame}\n' + + def visit_section (self, node): + if (self.section_level == 0): + self.body.append (self.begin_frametag(node)) + LaTeXTranslator.visit_section (self, node) + + def depart_section (self, node): + # Remove counter for potential subsections: + LaTeXTranslator.depart_section (self, node) + if (self.section_level == 0): + self.body.append (self.end_frametag()) + + def visit_title (self, node): + if (self.section_level == 1): + self.body.append ('\\frametitle{%s}\n\n' % self.encode(node.astext())) + raise nodes.SkipNode + else: + LaTeXTranslator.visit_title (self, node) + + def depart_title (self, node): + if (self.section_level != 1): + LaTeXTranslator.depart_title (self, node) + + def visit_literal_block(self, node): + if not self.active_table.is_open(): + self.body.append('\n\n\\smallskip\n\\begin{rtbliteral}\n') + self.context.append('\\end{rtbliteral}\n\\smallskip\n\n') + else: + self.body.append('\n') + self.context.append('\n') + if (self.settings.use_verbatim_when_possible and (len(node) == 1) + # in case of a parsed-literal containing just a "**bold**" word: + and isinstance(node[0], nodes.Text)): + self.verbatim = 1 + self.body.append('\\begin{verbatim}\n') + else: + self.literal_block = 1 + self.insert_none_breaking_blanks = 1 + + def depart_literal_block(self, node): + if self.verbatim: + self.body.append('\n\\end{verbatim}\n') + self.verbatim = 0 + else: + self.body.append('\n') + self.insert_none_breaking_blanks = 0 + self.literal_block = 0 + self.body.append(self.context.pop()) + + +class BeamerWriter (Latex2eWriter): + """ + A docutils writer that modifies the translator and settings for beamer. + """ + settings_spec = BEAMER_SPEC + settings_defaults = BEAMER_DEFAULTS + + def __init__(self): + Latex2eWriter.__init__(self) + self.translator_class = BeamerTranslator + + + + +if __name__ == '__main__': + description = ( + "Generates Beamer-flavoured LaTeX for PDF-based presentations." + default_description) + publish_cmdline (writer=BeamerWriter(), description=description) + + +### END ###################################################################### + diff --git a/talk/fosdem2014/pypy-jit/stylesheet.latex b/talk/fosdem2014/pypy-jit/stylesheet.latex new file mode 100644 --- /dev/null +++ b/talk/fosdem2014/pypy-jit/stylesheet.latex @@ -0,0 +1,11 @@ +\usetheme{Boadilla} +\usecolortheme{whale} +\setbeamercovered{transparent} +\setbeamertemplate{navigation symbols}{} + +\definecolor{darkgreen}{rgb}{0, 0.5, 0.0} +\newcommand{\docutilsrolegreen}[1]{\color{darkgreen}#1\normalcolor} +\newcommand{\docutilsrolered}[1]{\color{red}#1\normalcolor} + +\newcommand{\green}[1]{\color{darkgreen}#1\normalcolor} +\newcommand{\red}[1]{\color{red}#1\normalcolor} diff --git a/talk/fosdem2014/pypy-jit/talk.pdf b/talk/fosdem2014/pypy-jit/talk.pdf new file mode 100644 index 0000000000000000000000000000000000000000..78643be7cc20a8371fc7ae6eebf74543936b90f7 GIT binary patch [cut] diff --git a/talk/fosdem2014/pypy-jit/talk.rst b/talk/fosdem2014/pypy-jit/talk.rst new file mode 100644 --- /dev/null +++ b/talk/fosdem2014/pypy-jit/talk.rst @@ -0,0 +1,116 @@ +================================= +How PyPy makes your code run fast +================================= + +Introduction +============ + +* Romain Guillebert, @rguillebert + +* PyPy contributor for ~3 years + +* NumPyPy contributor + +* Please interrupt me + +* How the PyPy JIT works (kind of) + +* Warning : May contain traces of machine code + +speed.pypy.org +============== + +.. image:: Speed.png + :scale: 40% + :align: center + +AOT +=== + +* Ahead of time compilation + +* GCC + +* Can optimize only on what it knows before running the program + +Interpreter +=========== + +* CPython, PyPy + +* Executes an abstract representation of the program + +* Not very smart + +JIT +=== + +* PyPy + +* Gathers information at runtime + +* Produces optimized machine code + +RPython +======= + +* Statically typed subset of Python + +* The RPython compiler automatically generates the JIT from the annotated RPython code + +* The JIT can be added with just one line of code + +* More hints are needed to have an efficient JIT + +Tracing JIT +=========== + +* Optimizes loops + +* Traces one iteration of a loop + +* Produces a linear trace of execution + +* Inlines almost everything + +* The trace is then optimized and compiled + +Guard +===== + +* The JIT produces a linear trace, but the code isn't + +* The JIT can make assumptions that are not always true + +* Guard : If this is true, continue, otherwise return to the interpreter + +* guard_true, guard_class, guard_no_exception, ... + +Bridge +====== + +* After a guard has failed X times, the other path is traced, compiled and attached to the trace + +Optimizations +============= + +* Virtuals + +* Virtualizables + +* Promotion + +Jitviewer +========= + +* Jitviewer demo + +Demo +==== + +* Edge detection algorithm + +Questions +========= + +* Questions ? diff --git a/talk/fosdem2014/pypy-stm.pdf b/talk/fosdem2014/pypy-stm.pdf new file mode 100644 index 0000000000000000000000000000000000000000..78ca0e1e4a6f24dd1d5ebb89e45fe4c8f6bf95e8 GIT binary patch [cut] diff --git a/talk/fosdem2014/pypy-stm.rst b/talk/fosdem2014/pypy-stm.rst new file mode 100644 --- /dev/null +++ b/talk/fosdem2014/pypy-stm.rst @@ -0,0 +1,246 @@ +========================================================== +Using All These Cores: Transactional Memory under the hood +========================================================== + + +.. summary: + - Intro + - Using multiple threads: C++, Java; Jython, IronPython + - the GIL in CPython + - "bytecode" is uninteresting for the Python programmer + - but larger blocks are + - if we can make these larger blocks atomic, we win + - "with atomic:" + - theoretical only so far! + - best example: event-driven *non-multithreaded* systems + - under the hood: transactional memory + + +Introduction +============ + +* Armin Rigo, PyPy dev, CPython dev + +* Co-author: Remi Meier, ETHZ + +* This talk applies to Python or any similar language + + +Problem +======= + +* Most computer's CPUs today have multiple cores + +* How to use them? + + +Multithread programming +======================= + +* C, C++, Java, .NET, ... + +* Jython, IronPython + + +CPython, PyPy +============= + +* No story so far + +* Alternatives for various cases + +* Some fine and some horrible + + +The GIL +======= + +* Global Interpreter Lock + +* "Each bytecode is executed atomically" + + +Transactional Memory +==================== + +* Recent research (past ~10 years) + +* Optimistically runs multiple threads even if they + are supposed to be waiting on the same lock + +* Usually, high overheads + + +Expected results +================ + +* Runs multiple threads despite a single GIL + +* Does not remove the GIL, but solves the original problem anyway + + +Transactional Memory +==================== + +* STM: Software Transactional Memory + +* HTM: Hardware Transactional Memory + +* Hybrids + + +Status +====== + +* STM is still at least 2x slower (on one core) + +* HTM: tested in Ruby with Intel Haswell CPUs, not bad but + still disappointing (imo) + + +STM C7 +====== + +* c7 is our group's research (there were a lot of previous + research that failed to give good results) + +* Hope: much less than 2x slower for "PyPy-like" usages + +* (insert description here) + + +Atomic sections +=============== + +* GIL = "each bytecode is atomic" + +* One bytecode? Obscure for the regular Python programmer + +* Larger atomic sections: + +:: + + with atomic: + ... + + +Larger atomic sections +====================== + +* New way to synchronize multiple threads + +* All ``atomic`` blocks appear to run serialized + +* With STM/HTM, they actually run in parallel as far as possible + + +No threads? +=========== + +* Works even if you don't use threads! + +* If the Twisted reactor (say) was modified to start a pool of threads, + and to run all events in "``with atomic:``" + +* ...Then the end result is the same, for any Twisted application + + +Behind-the-scene threads +======================== + +* The thread pool added behind the scene lets a STM/HTM-enabled + Python run on several cores + +* The "``with atomic:``" means that the semantics of the Twisted + application didn't change + + +Summary (optimistic) +==================== + +* If you are using Twisted... + +* ...Your program will run on multiple cores ``:-)`` + + +Conflicts +========= + +* Actually, your program will likely fail to use multiple cores + out of the box + +* ...Because of "conflicts": each event should be "often" independent, + but may not be + +* Example: incrementing a global counter, or otherwise changing some + global object systematically + + +Some work left for you to do +============================ + +* You need to figure out where the conficts are + +* Maybe using some debugger-like tools that report conflicts + +* Then you need (hopefully small) rewrites to avoid them + + +Some work left for us to do, first +================================== + +* Additional conflicts come from Twisted itself + +* Example: the logging system, which may need to use queues + +* This means that some of the core Python data structures (dicts, + queues...) may need refactorings too + + +What is the point? +================== + +* The point is that with STM/HTM your program is always *correct* + (as much as the single-core version is) + +* You need to work in order to fix the most obvious conflicts + +* If you don't, it won't be faster than the single-core original + + +What did we win? +================ + +* Regular approach to multithreading: your program is always *fast* + +* You need to work in order to fix the bugs (races, deadlocks...) + +* You need to find and fix *all* bugs -- as opposed to the STM/HTM + version where you only fix *some* issues until it is fast enough + + +Scope +===== + +* Twisted / Tornado / Eventlet / Stackless / etc.: event-driven programming + +* Any program computing something complicated, e.g. over all items in + a dictionary, occasionally updating a shared state, etc. + +* In general, any CPU-bound program with identifiable sections that + have a good chance to be parallelizable: "a good chance" is enough + + +Conclusion +========== + +* Mostly theoretical for now: there is a risk it won't work in + practice [1] + +* Expect progress in the following months: http://morepypy.blogspot.com/ + +:: + + - + +[1] I bet it will, eventually ``:-)`` diff --git a/talk/fosdem2014/stylesheet.latex b/talk/fosdem2014/stylesheet.latex new file mode 100644 --- /dev/null +++ b/talk/fosdem2014/stylesheet.latex @@ -0,0 +1,10 @@ +\usetheme{Warsaw} +\usecolortheme{whale} +\setbeamercovered{transparent} +\definecolor{darkgreen}{rgb}{0, 0.5, 0.0} +\newcommand{\docutilsrolegreen}[1]{\color{darkgreen}#1\normalcolor} +\newcommand{\docutilsrolered}[1]{\color{red}#1\normalcolor} +\addtobeamertemplate{block begin}{}{\setlength{\parskip}{35pt plus 1pt minus 1pt}} + +\newcommand{\green}[1]{\color{darkgreen}#1\normalcolor} +\newcommand{\red}[1]{\color{red}#1\normalcolor} diff --git a/talk/pycon2014/language-summit.rst b/talk/pycon2014/language-summit.rst new file mode 100644 --- /dev/null +++ b/talk/pycon2014/language-summit.rst @@ -0,0 +1,7 @@ +---------------------------- +Language summit presentation +---------------------------- + +We should give a ~10 minute presentation about the status of PyPy. + +(Asked by Michael Foord) _______________________________________________ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
