[pypy-commit] extradoc extradoc: more slides
Author: Antonio Cuni
Branch: extradoc
Changeset: r5620:e0d0635116b3
Date: 2016-04-02 15:32 +0200
http://bitbucket.org/pypy/extradoc/changeset/e0d0635116b3/
Log:more slides
diff --git a/talk/bucharest2016/jit-frontend/diagrams/architecture.svg
b/talk/bucharest2016/jit-frontend/diagrams/architecture.svg
new file mode 100644
--- /dev/null
+++ b/talk/bucharest2016/jit-frontend/diagrams/architecture.svg
@@ -0,0 +1,700 @@
+
+
+
+http://purl.org/dc/elements/1.1/";
+ xmlns:cc="http://creativecommons.org/ns#";
+ xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#";
+ xmlns:svg="http://www.w3.org/2000/svg";
+ xmlns="http://www.w3.org/2000/svg";
+ xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd";
+ xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape";
+ width="1685.75"
+ height="949.56055"
+ id="svg3076"
+ version="1.1"
+ inkscape:version="0.48.2 r9819"
+ sodipodi:docname="architecture.svg">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+image/svg+xml
+http://purl.org/dc/dcmitype/StillImage"; />
+
+
+
+
+
+rpython+codewriter+jitcode+timeline+metatracer+optimizer+backend+jitted
+
+
+
+
+
+ def LOAD_GLOBAL(self):...
+
+
+
+ def STORE_FAST(self):...
+
+
+
+ def BINARY_ADD(self):...
+
+
+
+
+RPYTHON
+
+
+
+ CODEWRITER
+
+
+
+
+
+
+
+
+ ...p0 = getfield_gc(p0, 'func_globals')p2 = getfield_gc(p1, 'strval')call(dict_lookup, p0, p2)
+
+
+
+
+
+ ...p0 = getfield_gc(p0, 'locals_w')setarrayitem_gc(p0, i0, p1)
+
+
+
+
+...promote_class(p0)i0 = getfield_gc(p0, 'intval')promote_class(p1)i1 = getfield_gc(p1, 'intval')i2 = int_add(i0, i1)if (overflowed) goto ...p2 = new_with_vtable('W_IntObject')setfield_gc(p2, i2, 'intval')
+
+
+
+
+
+
+
+
+JITCODE
+
+
+
+compile-time
+runtime
+
+
+META-TRACER
+
+
+
+
+OPTIMIZER
+
+
+
+
+BACKEND
+
+
+
+
+ASSEMBLER
+
+
+
+
diff --git a/talk/bucharest2016/jit-frontend/diagrams/pypytrace.svg
b/talk/bucharest2016/jit-frontend/diagrams/pypytrace.svg
new file mode 100644
--- /dev/null
+++ b/talk/bucharest2016/jit-frontend/diagrams/pypytrace.svg
@@ -0,0 +1,346 @@
+
+
+
+http://purl.org/dc/elements/1.1/";
+ xmlns:cc="http://creativecommons.org/ns#";
+ xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#";
+ xmlns:svg="http://www.w3.org/2000/svg";
+ xmlns="http://www.w3.org/2000/svg";
+ xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd";
+ xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape";
+ width="847.5791"
+ height="534.64679"
+ id="svg5724"
+ version="1.1"
+ inkscape:version="0.48.2 r9819"
+ sodipodi:docname="pypytrace.svg">
+
+
+
+
+
+image/svg+xml
+http://purl.org/dc/dcmitype/StillImage"; />
+
+
+
+
+
+python+dis+trace0+trace1+trace2+trace3
+
+
+def fn():c = a+b...
+
+
+LOAD_GLOBAL ALOAD_GLOBAL BBINARY_ADDSTORE_FAST C
+
+
+
+ ...p0 = getfield_gc(p0, 'func_globals')p2 = getfield_gc(p1, 'strval')call(dict_lookup, p0, p2)...
+
+
+
+...p0 = getfield_gc(p0, 'func_globals')p2 = getfield_gc(p1, 'strval')call(dict_lookup, p0, p2)...
+
+
+...guard_class(p0, W_IntObject)i0 = getfield_gc(p0, 'intval')guard_class(p1, W_IntObject)i1 = getfield_gc(p1, 'intval')i2 = int_add(00, i1)guard_not_overflow()p2 = new_with_vtable('W_IntObject')setfield_gc(p2, i2, 'intval')...
+
+
+...p0 = getfield_gc(p0, 'locals_w')setarrayitem_gc(p0, i0, p1)
+
+
diff --git a/talk/bucharest2016/jit-frontend/talk.rst
b/talk/bucharest2016/jit-frontend/talk.rst
--- a/talk/bucharest2016/jit-frontend/talk.rst
+++ b/talk/bucharest2016/jit-frontend/talk.rst
@@ -93,7 +93,7 @@
-Part 1
+Part 2
--
**Overview of tracing JITs**
@@ -147,3 +147,320 @@
.. animage:: diagrams/tracing-phases-p*.pdf
:align: center
:scale: 100%
+
+
+Trace trees
+---
+
+WRITE ME
+
+Part 3
+--
+
+**The PyPy JIT**
+
+General architecture
+-
+
+.. animage:: diagrams/architecture-p*.pdf
+ :align: center
+ :scale: 24%
+
+
+PyPy trace example
+---
+
+.. animage:: diagrams/pypytrace-p*.pdf
+ :align: center
+ :scale: 40%
+
+
+PyPy optimizer
+---
+
+- intbounds
+
+- constant folding / pure operations
+
+- virtuals
+
+- string optimizations
+
+- heap (multiple get/setfield, etc)
+
+- unr
[pypy-commit] extradoc extradoc: more slides
Author: Antonio Cuni
Branch: extradoc
Changeset: r5621:5bbdec29c586
Date: 2016-04-02 16:03 +0200
http://bitbucket.org/pypy/extradoc/changeset/5bbdec29c586/
Log:more slides
diff --git a/talk/bucharest2016/jit-frontend/talk.rst
b/talk/bucharest2016/jit-frontend/talk.rst
--- a/talk/bucharest2016/jit-frontend/talk.rst
+++ b/talk/bucharest2016/jit-frontend/talk.rst
@@ -54,6 +54,9 @@
- "like GCC" (it statically produces a binary)
+ - you can run RPython programs on top of CPython (veeery slow, for
+development only)
+
|pause|
* ``pypy``: a Python interpreter
@@ -159,6 +162,77 @@
**The PyPy JIT**
+
+Terminology (1)
+
+
+* **translation time**: when you run "rpython targetpypy.py" to get the
+ ``pypy` binary
+
+* **runtime**: everything which happens after you start ``pypy``
+
+* **interpretation**, **tracing**, **compiling**
+
+* **assembler/machine code**: the output of the JIT compiler
+
+* **execution time**: when your Python program is being executed
+
+ - by the interpreter
+
+ - by the machine code
+
+
+Terminology (2)
+
+
+* **interp-level**: things written in RPython
+
+* **[PyPy] interpreter**: the RPython program which executes the final Python
+ programs
+
+* **bytecode**: "the output of dis.dis". It is executed by the PyPy
+ interpreter.
+
+* **app-level**: things written in Python, and executed by the PyPy
+ Interpreter
+
+
+Terminology (3)
+---
+
+* (the following is not 100% accurate but it's enough to understand the
+ general principle)
+
+* **low level op or ResOperation**
+
+ - low-level instructions like "add two integers", "read a field out of a
+struct", "call this function"
+
+ - (more or less) the same level of C ("portable assembler")
+
+ - knows about GC objects (e.g. you have ``getfield_gc`` vs ``getfield_raw``)
+
+
+* **jitcodes**: low-level representation of RPython functions
+
+ - sequence of low level ops
+
+ - generated at **translation time**
+
+ - 1 RPython function --> 1 C function --> 1 jitcode
+
+
+* **JIT traces or loops**
+
+ - a very specific sequence of llops as actually executed by your Python
program
+
+ - generated at **runtime** (more specifically, during **tracing**)
+
+* **JIT optimizer**: takes JIT traces and emits JIT traces
+
+* **JIT backend**: takes JIT traces and emits machine code
+
+
General architecture
-
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] extradoc extradoc: add some slides
Author: Antonio Cuni
Branch: extradoc
Changeset: r5619:e3308dd709fd
Date: 2016-04-02 15:26 +0200
http://bitbucket.org/pypy/extradoc/changeset/e3308dd709fd/
Log:add some slides
diff --git a/talk/bucharest2016/jit-frontend/Makefile
b/talk/bucharest2016/jit-frontend/Makefile
new file mode 100644
--- /dev/null
+++ b/talk/bucharest2016/jit-frontend/Makefile
@@ -0,0 +1,32 @@
+# you can find rst2beamer.py and inkscapeslide.py here:
+# http://bitbucket.org/antocuni/env/src/619f486c4fad/bin/rst2beamer.py
+# http://bitbucket.org/antocuni/env/src/619f486c4fad/bin/inkscapeslide.py
+
+
+talk.pdf: talk.rst author.latex stylesheet.latex
diagrams/tracing-phases-p0.pdf diagrams/trace-p0.pdf diagrams/tracetree-p0.pdf
diagrams/architecture-p0.pdf diagrams/pypytrace-p0.pdf
+ rst2beamer.py --stylesheet=stylesheet.latex --documentoptions=14pt
talk.rst talk.latex || exit
+ sed 's/\\date{}/\\input{author.latex}/' -i talk.latex || exit
+ #sed 's/\\maketitle/\\input{title.latex}/' -i talk.latex || exit
+ pdflatex talk.latex || exit
+
+view: talk.pdf
+ evince talk.pdf &
+
+xpdf: talk.pdf
+ xpdf talk.pdf &
+
+
+diagrams/tracing-phases-p0.pdf: diagrams/tracing-phases.svg
+ cd diagrams && inkscapeslide.py tracing-phases.svg
+
+diagrams/trace-p0.pdf: diagrams/trace.svg
+ cd diagrams && inkscapeslide.py trace.svg
+
+diagrams/tracetree-p0.pdf: diagrams/tracetree.svg
+ cd diagrams && inkscapeslide.py tracetree.svg
+
+diagrams/architecture-p0.pdf: diagrams/architecture.svg
+ cd diagrams && inkscapeslide.py architecture.svg
+
+diagrams/pypytrace-p0.pdf: diagrams/pypytrace.svg
+ cd diagrams && inkscapeslide.py pypytrace.svg
diff --git a/talk/bucharest2016/jit-frontend/diagrams/tracing-phases.svg
b/talk/bucharest2016/jit-frontend/diagrams/tracing-phases.svg
new file mode 100644
--- /dev/null
+++ b/talk/bucharest2016/jit-frontend/diagrams/tracing-phases.svg
@@ -0,0 +1,473 @@
+
+
+
+http://purl.org/dc/elements/1.1/";
+ xmlns:cc="http://creativecommons.org/ns#";
+ xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#";
+ xmlns:svg="http://www.w3.org/2000/svg";
+ xmlns="http://www.w3.org/2000/svg";
+ xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd";
+ xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape";
+ id="svg2"
+ version="1.1"
+ inkscape:version="0.48.2 r9819"
+ width="397.1475"
+ height="228.3725"
+ xml:space="preserve"
+ sodipodi:docname="tracing-phases.svg">image/svg+xmlhttp://purl.org/dc/dcmitype/StillImage"; />interp+tracing+compilation+running+cold_guard+compiled_loop+bridge+hot_guard
+
+Interpretation
+
+
+
+Tracing
+
+
+hotloopdetected
+
+
+Compilation
+
+
+Running
+
+
+coldguardfailed
+
+
+enteringcompiledloop
+
+
+guardfailure
+
+
+→
+
+
+hot
+
+
+hotguardfailed
+
+
+
\ No newline at end of file
diff --git a/talk/bucharest2016/jit-frontend/talk.rst
b/talk/bucharest2016/jit-frontend/talk.rst
--- a/talk/bucharest2016/jit-frontend/talk.rst
+++ b/talk/bucharest2016/jit-frontend/talk.rst
@@ -15,3 +15,135 @@
- "how we manage to make things fast"
+
+Part 1
+---
+
+**PyPy introduction**
+
+What is PyPy?
+--
+
+* For most people, the final product:
+
+|scriptsize|
+
+.. sourcecode:: python
+
+$ pypy
+Python 2.7.10 (173add34cdd2, Mar 15 2016, 23:00:19)
+[PyPy 5.1.0-alpha0 with GCC 4.8.4] on linux2
+>>>> import test.pystone
+>>>> test.pystone.main()
+Pystone(1.1) time for 5 passes = 0.0473992
+This machine benchmarks at 1.05487e+06 pystones/second
+
+|end_scriptsize|
+
+* More in general: a broader project, ecosystem and community
+
+
+PyPy as a project
+--
+
+* ``rpython``: a fancy compiler
+
+ - source code: "statically typed Python with type inference and
metaprogramming"
+
+ - fancy features: C-like performance, GC, meta-JIT
+
+ - "like GCC" (it statically produces a binary)
+
+|pause|
+
+* ``pypy``: a Python interpreter
+
+ - "like CPython", but written in RPython
+
+ - CPython : GCC = PyPy : RPython
+
+
+
+Important fact
+---
+
+* We **did not** write a JIT compiler for Python
+
+* The "meta JIT" works with all RPython programs
+
+* The "Python JIT" is automatically generated from the interpreter
+
+* Writing an interpreter is vastly easier than a compiler
+
+* Other interpreters: smalltalk, prolog, ruby, php, ...
+
+
+The final product
+--
+
+* ``rpython`` + ``pypy``: the final binary you download and execute
+
+ - a Python interpreter
+
+ - with a GC
+
+ - with a JIT
+
+ - fast
+
+
+
+Part 1
+--
+
+**Overview of tracing JITs**
+
+
+Assumptions
+---
+
+* Pareto Principle (80-20 rul
[pypy-commit] extradoc extradoc: fix makefile
Author: Antonio Cuni
Branch: extradoc
Changeset: r5622:8188a609f5bf
Date: 2016-04-02 16:04 +0200
http://bitbucket.org/pypy/extradoc/changeset/8188a609f5bf/
Log:fix makefile
diff --git a/talk/bucharest2016/jit-frontend/Makefile
b/talk/bucharest2016/jit-frontend/Makefile
--- a/talk/bucharest2016/jit-frontend/Makefile
+++ b/talk/bucharest2016/jit-frontend/Makefile
@@ -3,7 +3,7 @@
# http://bitbucket.org/antocuni/env/src/619f486c4fad/bin/inkscapeslide.py
-talk.pdf: talk.rst author.latex stylesheet.latex
diagrams/tracing-phases-p0.pdf diagrams/trace-p0.pdf diagrams/tracetree-p0.pdf
diagrams/architecture-p0.pdf diagrams/pypytrace-p0.pdf
+talk.pdf: talk.rst author.latex stylesheet.latex
diagrams/tracing-phases-p0.pdf diagrams/architecture-p0.pdf
diagrams/pypytrace-p0.pdf
rst2beamer.py --stylesheet=stylesheet.latex --documentoptions=14pt
talk.rst talk.latex || exit
sed 's/\\date{}/\\input{author.latex}/' -i talk.latex || exit
#sed 's/\\maketitle/\\input{title.latex}/' -i talk.latex || exit
@@ -19,11 +19,11 @@
diagrams/tracing-phases-p0.pdf: diagrams/tracing-phases.svg
cd diagrams && inkscapeslide.py tracing-phases.svg
-diagrams/trace-p0.pdf: diagrams/trace.svg
- cd diagrams && inkscapeslide.py trace.svg
+# diagrams/trace-p0.pdf: diagrams/trace.svg
+# cd diagrams && inkscapeslide.py trace.svg
-diagrams/tracetree-p0.pdf: diagrams/tracetree.svg
- cd diagrams && inkscapeslide.py tracetree.svg
+# diagrams/tracetree-p0.pdf: diagrams/tracetree.svg
+# cd diagrams && inkscapeslide.py tracetree.svg
diagrams/architecture-p0.pdf: diagrams/architecture.svg
cd diagrams && inkscapeslide.py architecture.svg
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] extradoc extradoc: start my talk
Author: Antonio Cuni
Branch: extradoc
Changeset: r5618:a1afb047aee1
Date: 2016-04-02 15:00 +0200
http://bitbucket.org/pypy/extradoc/changeset/a1afb047aee1/
Log:start my talk
diff --git a/talk/bucharest2016/jit-frontend/author.latex
b/talk/bucharest2016/jit-frontend/author.latex
new file mode 100644
--- /dev/null
+++ b/talk/bucharest2016/jit-frontend/author.latex
@@ -0,0 +1,8 @@
+\definecolor{rrblitbackground}{rgb}{0.0, 0.0, 0.0}
+
+\title[PyPy Intro]{PyPy Intro and JIT Frontend}
+\author[antocuni]
+{Antonio Cuni}
+
+\institute{Intel@Bucharest}
+\date{April 4 2016}
diff --git a/talk/bucharest2016/jit-frontend/beamerdefs.txt
b/talk/bucharest2016/jit-frontend/beamerdefs.txt
new file mode 100644
--- /dev/null
+++ b/talk/bucharest2016/jit-frontend/beamerdefs.txt
@@ -0,0 +1,108 @@
+.. colors
+.. ===
+
+.. role:: green
+.. role:: red
+
+
+.. general useful commands
+.. ===
+
+.. |pause| raw:: latex
+
+ \pause
+
+.. |small| raw:: latex
+
+ {\small
+
+.. |end_small| raw:: latex
+
+ }
+
+.. |scriptsize| raw:: latex
+
+ {\scriptsize
+
+.. |end_scriptsize| raw:: latex
+
+ }
+
+.. |strike<| raw:: latex
+
+ \sout{
+
+.. closed bracket
+.. ===
+
+.. |>| raw:: latex
+
+ }
+
+
+.. example block
+.. ===
+
+.. |example<| raw:: latex
+
+ \begin{exampleblock}{
+
+
+.. |end_example| raw:: latex
+
+ \end{exampleblock}
+
+
+
+.. alert block
+.. ===
+
+.. |alert<| raw:: latex
+
+ \begin{alertblock}{
+
+
+.. |end_alert| raw:: latex
+
+ \end{alertblock}
+
+
+
+.. columns
+.. ===
+
+.. |column1| raw:: latex
+
+ \begin{columns}
+ \begin{column}{0.45\textwidth}
+
+.. |column2| raw:: latex
+
+ \end{column}
+ \begin{column}{0.45\textwidth}
+
+
+.. |end_columns| raw:: latex
+
+ \end{column}
+ \end{columns}
+
+
+
+.. |snake| image:: ../../img/py-web-new.png
+ :scale: 15%
+
+
+
+.. nested blocks
+.. ===
+
+.. |nested| raw:: latex
+
+ \begin{columns}
+ \begin{column}{0.85\textwidth}
+
+.. |end_nested| raw:: latex
+
+ \end{column}
+ \end{columns}
diff --git a/talk/bucharest2016/jit-frontend/stylesheet.latex
b/talk/bucharest2016/jit-frontend/stylesheet.latex
new file mode 100644
--- /dev/null
+++ b/talk/bucharest2016/jit-frontend/stylesheet.latex
@@ -0,0 +1,11 @@
+\usetheme{Boadilla}
+\usecolortheme{whale}
+\setbeamercovered{transparent}
+\setbeamertemplate{navigation symbols}{}
+
+\definecolor{darkgreen}{rgb}{0, 0.5, 0.0}
+\newcommand{\docutilsrolegreen}[1]{\color{darkgreen}#1\normalcolor}
+\newcommand{\docutilsrolered}[1]{\color{red}#1\normalcolor}
+
+\newcommand{\green}[1]{\color{darkgreen}#1\normalcolor}
+\newcommand{\red}[1]{\color{red}#1\normalcolor}
diff --git a/talk/bucharest2016/jit-frontend/talk.rst
b/talk/bucharest2016/jit-frontend/talk.rst
new file mode 100644
--- /dev/null
+++ b/talk/bucharest2016/jit-frontend/talk.rst
@@ -0,0 +1,17 @@
+.. include:: beamerdefs.txt
+
+
+PyPy Intro and JIT Frontend
+
+
+About this talk
+
+
+* What is PyPy? What is RPython?
+
+* Tracing JIT 101
+
+* PyPy JIT frontend and optimizer
+
+ - "how we manage to make things fast"
+
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] extradoc extradoc: split into two slides
Author: Antonio Cuni Branch: extradoc Changeset: r5623:3bd035f4d6ec Date: 2016-04-02 16:07 +0200 http://bitbucket.org/pypy/extradoc/changeset/3bd035f4d6ec/ Log:split into two slides diff --git a/talk/bucharest2016/jit-frontend/talk.rst b/talk/bucharest2016/jit-frontend/talk.rst --- a/talk/bucharest2016/jit-frontend/talk.rst +++ b/talk/bucharest2016/jit-frontend/talk.rst @@ -167,7 +167,7 @@ * **translation time**: when you run "rpython targetpypy.py" to get the - ``pypy` binary + ``pypy`` binary * **runtime**: everything which happens after you start ``pypy`` @@ -200,6 +200,7 @@ Terminology (3) --- + * (the following is not 100% accurate but it's enough to understand the general principle) @@ -222,6 +223,9 @@ - 1 RPython function --> 1 C function --> 1 jitcode +Terminology (4) +--- + * **JIT traces or loops** - a very specific sequence of llops as actually executed by your Python program ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] extradoc extradoc: generate talk.pdf
Author: Antonio Cuni Branch: extradoc Changeset: r5624:bd451b255eab Date: 2016-04-02 16:08 +0200 http://bitbucket.org/pypy/extradoc/changeset/bd451b255eab/ Log:generate talk.pdf diff --git a/talk/bucharest2016/jit-frontend/talk.pdf b/talk/bucharest2016/jit-frontend/talk.pdf new file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..4ad8022637420b08f8a93bc24ddfd00d4b13adb2 GIT binary patch [cut] ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] extradoc extradoc: write the slides about trace trees
Author: Antonio Cuni
Branch: extradoc
Changeset: r5628:b7ac115da338
Date: 2016-04-03 16:54 +0200
http://bitbucket.org/pypy/extradoc/changeset/b7ac115da338/
Log:write the slides about trace trees
diff --git a/talk/bucharest2016/jit-frontend/Makefile
b/talk/bucharest2016/jit-frontend/Makefile
--- a/talk/bucharest2016/jit-frontend/Makefile
+++ b/talk/bucharest2016/jit-frontend/Makefile
@@ -3,7 +3,7 @@
# http://bitbucket.org/antocuni/env/src/619f486c4fad/bin/inkscapeslide.py
-talk.pdf: talk.rst author.latex stylesheet.latex
diagrams/tracing-phases-p0.pdf diagrams/architecture-p0.pdf
diagrams/pypytrace-p0.pdf
+talk.pdf: talk.rst author.latex stylesheet.latex
diagrams/tracing-phases-p0.pdf diagrams/architecture-p0.pdf
diagrams/pypytrace-p0.pdf diagrams/tracetree-p0.pdf
rst2beamer.py --stylesheet=stylesheet.latex --documentoptions=14pt
talk.rst talk.latex || exit
sed 's/\\date{}/\\input{author.latex}/' -i talk.latex || exit
#sed 's/\\maketitle/\\input{title.latex}/' -i talk.latex || exit
@@ -22,8 +22,8 @@
# diagrams/trace-p0.pdf: diagrams/trace.svg
# cd diagrams && inkscapeslide.py trace.svg
-# diagrams/tracetree-p0.pdf: diagrams/tracetree.svg
-# cd diagrams && inkscapeslide.py tracetree.svg
+diagrams/tracetree-p0.pdf: diagrams/tracetree.svg
+ cd diagrams && inkscapeslide.py tracetree.svg
diagrams/architecture-p0.pdf: diagrams/architecture.svg
cd diagrams && inkscapeslide.py architecture.svg
diff --git a/talk/bucharest2016/jit-frontend/diagrams/tracetree.svg
b/talk/bucharest2016/jit-frontend/diagrams/tracetree.svg
new file mode 100644
--- /dev/null
+++ b/talk/bucharest2016/jit-frontend/diagrams/tracetree.svg
@@ -0,0 +1,488 @@
+
+
+
+http://purl.org/dc/elements/1.1/";
+ xmlns:cc="http://creativecommons.org/ns#";
+ xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#";
+ xmlns:svg="http://www.w3.org/2000/svg";
+ xmlns="http://www.w3.org/2000/svg";
+ xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd";
+ xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape";
+ width="1017.0221"
+ height="787.14459"
+ id="svg8609"
+ version="1.1"
+ inkscape:version="0.48.4 r9939"
+ sodipodi:docname="tracetree.svg">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+image/svg+xml
+http://purl.org/dc/dcmitype/StillImage"; />
+
+
+
+
+
+trace+looptrace, guard_sign+blackhole+interp+call_jittedtrace+guard_sign+bridge+loop2+loop
+
+
+label(start, i0, a0)v0 = int_lt(i0, 2000)guard_true(v0)v1 = int_mod(i0, 2)v2 = int_eq(v1, 0)guard_true(v1)a1 = int_add(a0, 10)i1 = int_add(i0, 1)jump(start, i1, a1)
+
+
+
+
+
+
+
+
+
+
+
+ BLACKHOLE
+
+COLD FAIL
+
+
+
+
+ INTERPRETER
+
+
+
+
+
+
+
+a1 = int_mul(a0, 2)i1 = int_add(i0, 1)jump(start, i1, a1)
+
+HOT FAIL
+
+
+
+
+
diff --git a/talk/bucharest2016/jit-frontend/talk.rst
b/talk/bucharest2016/jit-frontend/talk.rst
--- a/talk/bucharest2016/jit-frontend/talk.rst
+++ b/talk/bucharest2016/jit-frontend/talk.rst
@@ -152,10 +152,38 @@
:scale: 100%
-Trace trees
+Trace trees (1)
+---
-WRITE ME
+|scriptsize|
+|example<| |small| tracetree.py |end_small| |>|
+
+.. sourcecode:: python
+
+def foo():
+ a = 0
+ i = 0
+ N = 100
+ while i < N:
+if i%2 == 0:
+a += 1
+else:
+a *= 2;
+i += 1
+ return a
+
+|end_example|
+|end_scriptsize|
+
+
+Trace trees (2)
+---
+
+.. animage:: diagrams/tracetree-p*.pdf
+ :align: center
+ :scale: 30%
+
+
Part 3
--
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] extradoc extradoc: regenerate pdf
Author: Antonio Cuni Branch: extradoc Changeset: r5629:bb172b3eaf26 Date: 2016-04-04 00:20 +0200 http://bitbucket.org/pypy/extradoc/changeset/bb172b3eaf26/ Log:regenerate pdf diff --git a/talk/bucharest2016/jit-frontend/talk.pdf b/talk/bucharest2016/jit-frontend/talk.pdf index 4ad8022637420b08f8a93bc24ddfd00d4b13adb2..c74ff0a9eb5d7e988a9424ae7c94708182ad5a1b GIT binary patch [cut] ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] extradoc extradoc: add an XXX
Author: Antonio Cuni Branch: extradoc Changeset: r5630:8de7beaf20d3 Date: 2016-04-04 00:25 +0200 http://bitbucket.org/pypy/extradoc/changeset/8de7beaf20d3/ Log:add an XXX diff --git a/blog/draft/jit-leaner-frontend.rst b/blog/draft/jit-leaner-frontend.rst --- a/blog/draft/jit-leaner-frontend.rst +++ b/blog/draft/jit-leaner-frontend.rst @@ -46,6 +46,7 @@ pointers to store the "operations" object after tracing, we use a compact list of 16-bit integers (with 16bit pointers in between). On 64bit machine the wins are tremendous - it's 4x more efficient to use 16bit pointers than full 64bit pointers. +.. XXX: I assume you are talking about "memory efficiency": we should be clearer Additionally those pointers have a much better defined lifespan, so we don't need to bother tracking them by the GC, which also saves quite a bit of time. ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy resource_warning: move the logic from __del__ to destructor, because we are not allowed to call arbitrary code from the interp-level __del__
Author: Antonio Cuni
Branch: resource_warning
Changeset: r83551:110eae852734
Date: 2016-04-06 19:39 +
http://bitbucket.org/pypy/pypy/changeset/110eae852734/
Log:move the logic from __del__ to destructor, because we are not
allowed to call arbitrary code from the interp-level __del__
diff --git a/pypy/module/_file/interp_file.py b/pypy/module/_file/interp_file.py
--- a/pypy/module/_file/interp_file.py
+++ b/pypy/module/_file/interp_file.py
@@ -49,15 +49,16 @@
# thread that runs __del__, so no race condition should be possible
self.clear_all_weakrefs()
if self.stream is not None:
-if self.space.sys.resource_warning_enabled:
-w_repr = self.space.repr(self)
-str_repr = self.space.str_w(w_repr)
-self.space.resource_warning("WARNING: unclosed file: " +
str_repr)
self.enqueue_for_destruction(self.space, W_File.destructor,
'close() method of ')
def destructor(self):
assert isinstance(self, W_File)
+if self.space.sys.resource_warning_enabled:
+w_repr = self.space.repr(self)
+str_repr = self.space.str_w(w_repr)
+self.space.resource_warning("WARNING: unclosed file: " + str_repr)
+#
try:
self.direct_close()
except StreamErrors, e:
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy resource_warning: (antocuni, florinpapa): fix the test, it was because we returned None from space.format_traceback and the following appexec in space.resource_warning got confused
Author: Antonio Cuni
Branch: resource_warning
Changeset: r83559:67909a688ba0
Date: 2016-04-07 10:37 +0200
http://bitbucket.org/pypy/pypy/changeset/67909a688ba0/
Log:(antocuni, florinpapa): fix the test, it was because we returned
None from space.format_traceback and the following appexec in
space.resource_warning got confused
diff --git a/pypy/interpreter/baseobjspace.py b/pypy/interpreter/baseobjspace.py
--- a/pypy/interpreter/baseobjspace.py
+++ b/pypy/interpreter/baseobjspace.py
@@ -1751,7 +1751,7 @@
""")
def format_traceback(self):
-self.appexec([],
+return self.appexec([],
"""():
import traceback
return "".join(traceback.format_stack())
diff --git a/pypy/module/_file/interp_file.py b/pypy/module/_file/interp_file.py
--- a/pypy/module/_file/interp_file.py
+++ b/pypy/module/_file/interp_file.py
@@ -60,7 +60,7 @@
if self.space.sys.resource_warning_enabled:
w_repr = self.space.repr(self)
str_repr = self.space.str_w(w_repr)
-w_msg = self.space.wrap("WARNING: unclosed file:" + str_repr)
+w_msg = self.space.wrap("WARNING: unclosed file: " + str_repr)
self.space.resource_warning(w_msg, self.w_tb)
#
try:
diff --git a/pypy/module/_file/test/test_file.py
b/pypy/module/_file/test/test_file.py
--- a/pypy/module/_file/test/test_file.py
+++ b/pypy/module/_file/test/test_file.py
@@ -275,12 +275,6 @@
assert fn() == ""
sys.pypy_set_resource_warning(True)
msg = fn()
-
-#f1 = open("/tmp/test", "w+")
-#import pdb; pdb.set_trace()
-print msg
-#close(f1)
-
assert msg.startswith("WARNING: unclosed file: https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy resource_warning: (antocuni, florinpapa): make the test better by checking the output with a regexp
Author: Antonio Cuni
Branch: resource_warning
Changeset: r83560:b20b7e966a24
Date: 2016-04-07 11:04 +0200
http://bitbucket.org/pypy/pypy/changeset/b20b7e966a24/
Log:(antocuni, florinpapa): make the test better by checking the output
with a regexp
diff --git a/pypy/module/_file/test/test_file.py
b/pypy/module/_file/test/test_file.py
--- a/pypy/module/_file/test/test_file.py
+++ b/pypy/module/_file/test/test_file.py
@@ -1,5 +1,6 @@
from __future__ import with_statement
import py, os, errno
+from pypy.interpreter.gateway import interp2app, unwrap_spec
def getfile(space):
return space.appexec([], """():
@@ -17,6 +18,17 @@
cls.w_temppath = cls.space.wrap(
str(py.test.ensuretemp("fileimpl").join("foo.txt")))
cls.w_file = getfile(cls.space)
+#
+# the following function is used e.g. in test_resource_warning
+@unwrap_spec(regex=str, s=str)
+def regex_search(space, regex, s):
+import re
+import textwrap
+regex = textwrap.dedent(regex).strip()
+m = re.search(regex, s)
+m = bool(m)
+return space.wrap(m)
+cls.w_regex_search = cls.space.wrap(interp2app(regex_search))
def test_simple(self):
f = self.file(self.temppath, "w")
@@ -256,6 +268,7 @@
def test_resource_warning(self):
import os, gc, sys, cStringIO
+import re
if '__pypy__' not in sys.builtin_module_names:
skip("pypy specific test")
def fn():
@@ -275,8 +288,13 @@
assert fn() == ""
sys.pypy_set_resource_warning(True)
msg = fn()
-assert msg.startswith("WARNING: unclosed file:
+Created at \(most recent call last\):
+ File ".*", line .*, in test_resource_warning
+ File ".*", line .*, in fn
+ File ".*", line .*, in anonymous
+""", msg)
finally:
sys.pypy_set_resource_warning(False)
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy resource_warning: (antocuni, florinpapa): make the test even better
Author: Antonio Cuni
Branch: resource_warning
Changeset: r83561:a2e023d51866
Date: 2016-04-07 11:10 +0200
http://bitbucket.org/pypy/pypy/changeset/a2e023d51866/
Log:(antocuni, florinpapa): make the test even better
diff --git a/pypy/module/_file/test/test_file.py
b/pypy/module/_file/test/test_file.py
--- a/pypy/module/_file/test/test_file.py
+++ b/pypy/module/_file/test/test_file.py
@@ -271,8 +271,10 @@
import re
if '__pypy__' not in sys.builtin_module_names:
skip("pypy specific test")
-def fn():
+def fn(flag1, flag2):
+sys.pypy_set_resource_warning(flag1)
f = self.file(self.temppath, 'w')
+sys.pypy_set_resource_warning(flag2)
g = cStringIO.StringIO()
preverr = sys.stderr
try:
@@ -281,22 +283,27 @@
gc.collect() # force __del__ to be called
finally:
sys.stderr = preverr
+sys.pypy_set_resource_warning(False)
return g.getvalue()
-try:
-sys.pypy_set_resource_warning(False)
-assert fn() == ""
-sys.pypy_set_resource_warning(True)
-msg = fn()
-assert self.regex_search(r"""
-WARNING: unclosed file:
-Created at \(most recent call last\):
- File ".*", line .*, in test_resource_warning
- File ".*", line .*, in fn
- File ".*", line .*, in anonymous
-""", msg)
-finally:
-sys.pypy_set_resource_warning(False)
+# check with resource_warning disabled
+assert fn(False, False) == ""
+#
+# check with resource_warning enabled
+msg = fn(True, True)
+assert self.regex_search(r"""
+WARNING: unclosed file:
+Created at \(most recent call last\):
+ File ".*", line .*, in test_resource_warning
+ File ".*", line .*, in fn
+ File ".*", line .*, in anonymous
+""", msg)
+#
+# check with resource_warning enabled in the destructor BUT with a
+# file which was created when resource_warning was disabled
+msg = fn(False, True)
+assert self.regex_search("WARNING: unclosed file: ", msg)
+assert "Created at" not in msg
def test_truncate(self):
f = self.file(self.temppath, "w")
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy resource_warning: enable resource-tracking for socket objects
Author: Antonio Cuni
Branch: resource_warning
Changeset: r83568:777b0214d3a8
Date: 2016-04-07 16:08 +0200
http://bitbucket.org/pypy/pypy/changeset/777b0214d3a8/
Log:enable resource-tracking for socket objects
diff --git a/pypy/interpreter/test/test_app_main.py
b/pypy/interpreter/test/test_app_main.py
--- a/pypy/interpreter/test/test_app_main.py
+++ b/pypy/interpreter/test/test_app_main.py
@@ -1044,4 +1044,3 @@
# assert it did not crash
finally:
sys.path[:] = old_sys_path
-
diff --git a/pypy/module/_file/test/test_file.py
b/pypy/module/_file/test/test_file.py
--- a/pypy/module/_file/test/test_file.py
+++ b/pypy/module/_file/test/test_file.py
@@ -11,6 +11,16 @@
return file
""")
+# the following function is used e.g. in test_resource_warning
+@unwrap_spec(regex=str, s=str)
+def regex_search(space, regex, s):
+import re
+import textwrap
+regex = textwrap.dedent(regex).strip()
+m = re.search(regex, s)
+m = bool(m)
+return space.wrap(m)
+
class AppTestFile(object):
spaceconfig = dict(usemodules=("_file",))
@@ -18,16 +28,6 @@
cls.w_temppath = cls.space.wrap(
str(py.test.ensuretemp("fileimpl").join("foo.txt")))
cls.w_file = getfile(cls.space)
-#
-# the following function is used e.g. in test_resource_warning
-@unwrap_spec(regex=str, s=str)
-def regex_search(space, regex, s):
-import re
-import textwrap
-regex = textwrap.dedent(regex).strip()
-m = re.search(regex, s)
-m = bool(m)
-return space.wrap(m)
cls.w_regex_search = cls.space.wrap(interp2app(regex_search))
def test_simple(self):
diff --git a/pypy/module/_socket/interp_socket.py
b/pypy/module/_socket/interp_socket.py
--- a/pypy/module/_socket/interp_socket.py
+++ b/pypy/module/_socket/interp_socket.py
@@ -154,9 +154,28 @@
class W_Socket(W_Root):
+w_tb = None # String representation of the traceback at creation time
+
def __init__(self, space, sock):
+self.space = space
self.sock = sock
register_socket(space, sock)
+if self.space.sys.track_resources:
+self.w_tb = self.space.format_traceback()
+
+def __del__(self):
+is_open = self.sock.fd >= 0
+if is_open and self.space.sys.track_resources:
+self.enqueue_for_destruction(self.space, W_Socket.destructor,
+ '__del__ method of ')
+
+def destructor(self):
+assert isinstance(self, W_Socket)
+if self.space.sys.track_resources:
+w_repr = self.space.repr(self)
+str_repr = self.space.str_w(w_repr)
+w_msg = self.space.wrap("WARNING: unclosed " + str_repr)
+self.space.resource_warning(w_msg, self.w_tb)
def get_type_w(self, space):
return space.wrap(self.sock.type)
diff --git a/pypy/module/_socket/test/test_sock_app.py
b/pypy/module/_socket/test/test_sock_app.py
--- a/pypy/module/_socket/test/test_sock_app.py
+++ b/pypy/module/_socket/test/test_sock_app.py
@@ -1,6 +1,8 @@
import sys, os
import py
from pypy.tool.pytest.objspace import gettestobjspace
+from pypy.interpreter.gateway import interp2app
+from pypy.module._file.test.test_file import regex_search
from rpython.tool.udir import udir
from rpython.rlib import rsocket
from rpython.rtyper.lltypesystem import lltype, rffi
@@ -314,6 +316,7 @@
def setup_class(cls):
cls.space = space
cls.w_udir = space.wrap(str(udir))
+cls.w_regex_search = space.wrap(interp2app(regex_search))
def teardown_class(cls):
if not cls.runappdirect:
@@ -402,6 +405,64 @@
if os.name != 'nt':
raises(OSError, os.close, fileno)
+def test_socket_track_resources(self):
+import _socket, os, gc, sys, cStringIO
+s = _socket.socket(_socket.AF_INET, _socket.SOCK_STREAM, 0)
+fileno = s.fileno()
+assert s.fileno() >= 0
+s.close()
+assert s.fileno() < 0
+s.close()
+if os.name != 'nt':
+raises(OSError, os.close, fileno)
+
+def test_track_resources(self):
+import os, gc, sys, cStringIO
+import _socket
+if '__pypy__' not in sys.builtin_module_names:
+skip("pypy specific test")
+#
+def fn(flag1, flag2, do_close=False):
+sys.pypy_set_track_resources(flag1)
+mysock = _socket.socket(_socket.AF_INET, _socket.SOCK_STREAM, 0)
+sys.pypy_set_track_resources(flag2)
+buf = cStringIO.StringIO()
+preverr = sys.stderr
+try:
+sys.stderr = buf
+if do_close:
+mysock.close()
+del mysock
+gc.collect() # force __del__ to be called
+finally:
+sys.stderr = preverr
+sys.pypy
[pypy-commit] pypy resource_warning: improve the test and check that we don't get the warning if we explicitly close the file
Author: Antonio Cuni
Branch: resource_warning
Changeset: r83569:3649155837b6
Date: 2016-04-07 16:09 +0200
http://bitbucket.org/pypy/pypy/changeset/3649155837b6/
Log:improve the test and check that we don't get the warning if we
explicitly close the file
diff --git a/pypy/module/_file/test/test_file.py
b/pypy/module/_file/test/test_file.py
--- a/pypy/module/_file/test/test_file.py
+++ b/pypy/module/_file/test/test_file.py
@@ -270,24 +270,29 @@
import os, gc, sys, cStringIO
if '__pypy__' not in sys.builtin_module_names:
skip("pypy specific test")
-def fn(flag1, flag2):
+def fn(flag1, flag2, do_close=False):
sys.pypy_set_track_resources(flag1)
f = self.file(self.temppath, 'w')
sys.pypy_set_track_resources(flag2)
-g = cStringIO.StringIO()
+buf = cStringIO.StringIO()
preverr = sys.stderr
try:
-sys.stderr = g
+sys.stderr = buf
+if do_close:
+f.close()
del f
gc.collect() # force __del__ to be called
finally:
sys.stderr = preverr
sys.pypy_set_track_resources(False)
-return g.getvalue()
+return buf.getvalue()
# check with track_resources disabled
assert fn(False, False) == ""
#
+# check that we don't get the warning if we actually close the file
+assert fn(False, False, do_close=True) == ""
+#
# check with track_resources enabled
msg = fn(True, True)
assert self.regex_search(r"""
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy resource_warning: don't import re, it's not needed
Author: Antonio Cuni
Branch: resource_warning
Changeset: r83567:17a9af6e11a3
Date: 2016-04-07 15:46 +0200
http://bitbucket.org/pypy/pypy/changeset/17a9af6e11a3/
Log:don't import re, it's not needed
diff --git a/pypy/module/_file/test/test_file.py
b/pypy/module/_file/test/test_file.py
--- a/pypy/module/_file/test/test_file.py
+++ b/pypy/module/_file/test/test_file.py
@@ -268,7 +268,6 @@
def test_track_resources(self):
import os, gc, sys, cStringIO
-import re
if '__pypy__' not in sys.builtin_module_names:
skip("pypy specific test")
def fn(flag1, flag2):
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy resource_warning: failing test and corresponding fix
Author: Antonio Cuni
Branch: resource_warning
Changeset: r83577:9ba84d112325
Date: 2016-04-07 21:54 +0200
http://bitbucket.org/pypy/pypy/changeset/9ba84d112325/
Log:failing test and corresponding fix
diff --git a/pypy/interpreter/baseobjspace.py b/pypy/interpreter/baseobjspace.py
--- a/pypy/interpreter/baseobjspace.py
+++ b/pypy/interpreter/baseobjspace.py
@@ -1751,11 +1751,20 @@
""")
def format_traceback(self):
-return self.appexec([],
- """():
-import traceback
-return "".join(traceback.format_stack())
-""")
+# we need to disable track_resources before calling the traceback
+# module. Else, it tries to open more files to format the traceback,
+# the file constructor will call space.format_traceback etc., in an
+# inifite recursion
+flag = self.sys.track_resources
+self.sys.track_resources = False
+try:
+return self.appexec([],
+ """():
+import traceback
+return "".join(traceback.format_stack())
+""")
+finally:
+self.sys.track_resources = flag
class AppExecCache(SpaceCache):
diff --git a/pypy/module/_file/test/test_file.py
b/pypy/module/_file/test/test_file.py
--- a/pypy/module/_file/test/test_file.py
+++ b/pypy/module/_file/test/test_file.py
@@ -309,6 +309,27 @@
assert self.regex_search("WARNING: unclosed file: ", msg)
assert "Created at" not in msg
+def test_track_resources_dont_crash(self):
+import os, gc, sys, cStringIO
+if '__pypy__' not in sys.builtin_module_names:
+skip("pypy specific test")
+#
+# try hard to create a code object whose co_filename points to an
+# EXISTING file, so that traceback.py tries to open it when formatting
+# the stacktrace
+f = open(self.temppath, 'w')
+f.close()
+co = compile('open("%s")' % self.temppath, self.temppath, 'exec')
+sys.pypy_set_track_resources(True)
+try:
+# this exec used to fail, because space.format_traceback tried to
+# recurively open a file, causing an infinite recursion. For the
+# purpose of this test, it is enough that it actually finishes
+# without errors
+exec co
+finally:
+sys.pypy_set_track_resources(False)
+
def test_truncate(self):
f = self.file(self.temppath, "w")
f.write("foo")
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy resource_warning: don't show the 'anonymous' frame corresponding to the appexec when calling format_traceback
Author: Antonio Cuni Branch: resource_warning Changeset: r83578:005ca768df30 Date: 2016-04-07 22:18 +0200 http://bitbucket.org/pypy/pypy/changeset/005ca768df30/ Log:don't show the 'anonymous' frame corresponding to the appexec when calling format_traceback diff --git a/pypy/interpreter/baseobjspace.py b/pypy/interpreter/baseobjspace.py --- a/pypy/interpreter/baseobjspace.py +++ b/pypy/interpreter/baseobjspace.py @@ -1760,8 +1760,11 @@ try: return self.appexec([], """(): -import traceback -return "".join(traceback.format_stack()) +import sys, traceback +# the "1" is because we don't want to show THIS code +# object in the traceback +f = sys._getframe(1) +return "".join(traceback.format_stack(f)) """) finally: self.sys.track_resources = flag diff --git a/pypy/interpreter/test/test_objspace.py b/pypy/interpreter/test/test_objspace.py --- a/pypy/interpreter/test/test_objspace.py +++ b/pypy/interpreter/test/test_objspace.py @@ -427,3 +427,28 @@ space.finish() # assert that we reach this point without getting interrupted # by the OperationError(NameError) + +def test_format_traceback(self): +from pypy.tool.pytest.objspace import maketestobjspace +from pypy.interpreter.gateway import interp2app +# +def format_traceback(space): +return space.format_traceback() +# +space = maketestobjspace() +w_format_traceback = space.wrap(interp2app(format_traceback)) +w_tb = space.appexec([w_format_traceback], """(format_traceback): +def foo(): +return bar() +def bar(): +return format_traceback() +return foo() +""") +tb = space.str_w(w_tb) +expected = '\n'.join([ +' File "?", line 6, in anonymous', # this is the appexec code object +' File "?", line 3, in foo', +' File "?", line 5, in bar', +'' +]) +assert tb == expected ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy resource_warning: exit if you don't specify the correct -X option; in theory you should continue and put it in sys._Xoptions, but since it's not implemented it's better to just exit
Author: Antonio Cuni Branch: resource_warning Changeset: r83580:5de70e252eef Date: 2016-04-07 23:00 +0200 http://bitbucket.org/pypy/pypy/changeset/5de70e252eef/ Log:exit if you don't specify the correct -X option; in theory you should continue and put it in sys._Xoptions, but since it's not implemented it's better to just exit for now diff --git a/pypy/interpreter/app_main.py b/pypy/interpreter/app_main.py --- a/pypy/interpreter/app_main.py +++ b/pypy/interpreter/app_main.py @@ -26,6 +26,7 @@ file : program read from script file - : program read from stdin (default; interactive mode if a tty) arg ...: arguments passed to program in sys.argv[1:] + PyPy options and arguments: --info : print translation information about this PyPy executable -X track-resources : track the creation of files and sockets and display @@ -230,10 +231,9 @@ if Xparam == 'track-resources': sys.pypy_set_track_resources(True) else: -print >> sys.stderr print >> sys.stderr, 'usage: %s -X [options]' % (get_sys_executable(),) print >> sys.stderr, '[options] can be: track-resources' -print >> sys.stderr +raise SystemExit class CommandLineError(Exception): pass ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy resource_warning: don't crash if we enable track_resources at the early startup (in particular, if we pass the -X track-resources option). It is a bit impossible to write an app-lev
Author: Antonio Cuni Branch: resource_warning Changeset: r83579:e1647e262594 Date: 2016-04-07 21:54 + http://bitbucket.org/pypy/pypy/changeset/e1647e262594/ Log:don't crash if we enable track_resources at the early startup (in particular, if we pass the -X track-resources option). It is a bit impossible to write an app-level test for it because we always have a bottom frame in that case diff --git a/pypy/interpreter/baseobjspace.py b/pypy/interpreter/baseobjspace.py --- a/pypy/interpreter/baseobjspace.py +++ b/pypy/interpreter/baseobjspace.py @@ -1763,7 +1763,12 @@ import sys, traceback # the "1" is because we don't want to show THIS code # object in the traceback -f = sys._getframe(1) +try: +f = sys._getframe(1) +except ValueError: +# this happens if you call format_traceback at the very beginning +# of startup, when there is no bottom code object +return '' return "".join(traceback.format_stack(f)) """) finally: ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy fix-struct-unpack-Q: a failing test and the fix: on CPython, struct.unpack('Q', ...) tries hard to return an int, while PyPy always returned a long
Author: Antonio Cuni
Branch: fix-struct-unpack-Q
Changeset: r87960:0fc23c94e459
Date: 2016-10-27 18:08 +0200
http://bitbucket.org/pypy/pypy/changeset/0fc23c94e459/
Log:a failing test and the fix: on CPython, struct.unpack('Q',...) tries
hard to return an int, while PyPy always returned a long
diff --git a/pypy/module/struct/formatiterator.py
b/pypy/module/struct/formatiterator.py
--- a/pypy/module/struct/formatiterator.py
+++ b/pypy/module/struct/formatiterator.py
@@ -148,7 +148,18 @@
@specialize.argtype(1)
def appendobj(self, value):
-self.result_w.append(self.space.wrap(value))
+from rpython.rlib.rarithmetic import r_uint, maxint, intmask
+if isinstance(value, r_uint):
+# native-size uint: try to wrap it inside a native int if it fits,
+# as CPython does
+if value <= maxint:
+w_value = self.space.wrap(intmask(value))
+else:
+w_value = self.space.wrap(value)
+else:
+w_value = self.space.wrap(value)
+#
+self.result_w.append(w_value)
def get_pos(self):
return self.pos
diff --git a/pypy/module/struct/test/test_struct.py
b/pypy/module/struct/test/test_struct.py
--- a/pypy/module/struct/test/test_struct.py
+++ b/pypy/module/struct/test/test_struct.py
@@ -431,6 +431,17 @@
def test_overflow(self):
raises(self.struct.error, self.struct.pack, 'i', 1<<65)
+def test_unpack_Q(self):
+import sys
+buf = self.struct.pack('Q', sys.maxint)
+obj, = self.struct.unpack('Q', buf)
+assert obj == sys.maxint
+assert type(obj) is int
+#
+buf = self.struct.pack('Q', sys.maxint+1)
+obj, = self.struct.unpack('Q', buf)
+assert obj == sys.maxint+1
+assert type(obj) is long
class AppTestStructBuffer(object):
spaceconfig = dict(usemodules=['struct', '__pypy__'])
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy fix-struct-unpack-Q: rpython fix
Author: Antonio Cuni Branch: fix-struct-unpack-Q Changeset: r87975:adfab214048f Date: 2016-10-28 18:35 +0200 http://bitbucket.org/pypy/pypy/changeset/adfab214048f/ Log:rpython fix diff --git a/pypy/module/struct/formatiterator.py b/pypy/module/struct/formatiterator.py --- a/pypy/module/struct/formatiterator.py +++ b/pypy/module/struct/formatiterator.py @@ -149,7 +149,7 @@ @specialize.argtype(1) def appendobj(self, value): -if isinstance(value, (r_uint, r_ulonglong)): +if isinstance(value, r_uint) or isinstance(value, r_ulonglong): # unsigned int: space.wrap would wrap it inside a long, but # CPython tries hard to return an int, if it fits if value <= maxint: ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy fix-struct-unpack-Q: generalize the test and the fix: it's not only about Q, but about all the unsigned types; this also fix it on 32bit
Author: Antonio Cuni
Branch: fix-struct-unpack-Q
Changeset: r87974:cced34dd6dde
Date: 2016-10-28 18:33 +0200
http://bitbucket.org/pypy/pypy/changeset/cced34dd6dde/
Log:generalize the test and the fix: it's not only about Q, but about
all the unsigned types; this also fix it on 32bit
diff --git a/pypy/module/struct/formatiterator.py
b/pypy/module/struct/formatiterator.py
--- a/pypy/module/struct/formatiterator.py
+++ b/pypy/module/struct/formatiterator.py
@@ -1,3 +1,4 @@
+from rpython.rlib.rarithmetic import r_uint, r_ulonglong, maxint, intmask
from rpython.rlib import jit
from rpython.rlib.objectmodel import specialize
from rpython.rlib.rstring import StringBuilder
@@ -148,15 +149,15 @@
@specialize.argtype(1)
def appendobj(self, value):
-from rpython.rlib.rarithmetic import r_uint, maxint, intmask
-if isinstance(value, r_uint):
-# native-size uint: try to wrap it inside a native int if it fits,
-# as CPython does
+if isinstance(value, (r_uint, r_ulonglong)):
+# unsigned int: space.wrap would wrap it inside a long, but
+# CPython tries hard to return an int, if it fits
if value <= maxint:
w_value = self.space.wrap(intmask(value))
else:
w_value = self.space.wrap(value)
else:
+# generic type, just use space.wrap
w_value = self.space.wrap(value)
#
self.result_w.append(w_value)
diff --git a/pypy/module/struct/test/test_struct.py
b/pypy/module/struct/test/test_struct.py
--- a/pypy/module/struct/test/test_struct.py
+++ b/pypy/module/struct/test/test_struct.py
@@ -431,17 +431,20 @@
def test_overflow(self):
raises(self.struct.error, self.struct.pack, 'i', 1<<65)
-def test_unpack_Q(self):
+def test_unpack_unsigned(self):
import sys
-buf = self.struct.pack('Q', sys.maxint)
-obj, = self.struct.unpack('Q', buf)
-assert obj == sys.maxint
-assert type(obj) is int
+for fmt in 'ILQ':
+# check that we return an int, if it fits
+buf = self.struct.pack(fmt, 42)
+val, = self.struct.unpack(fmt, buf)
+assert val == 42
+assert type(val) is int
#
+# check that we return a long, if it doesn't fit into an int
buf = self.struct.pack('Q', sys.maxint+1)
-obj, = self.struct.unpack('Q', buf)
-assert obj == sys.maxint+1
-assert type(obj) is long
+val, = self.struct.unpack('Q', buf)
+assert val == sys.maxint+1
+assert type(val) is long
class AppTestStructBuffer(object):
spaceconfig = dict(usemodules=['struct', '__pypy__'])
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy fix-struct-unpack-Q: make sure that we return an int even when we use 'q' on 32bit
Author: Antonio Cuni Branch: fix-struct-unpack-Q Changeset: r87999:08ad056ff621 Date: 2016-10-31 15:29 +0100 http://bitbucket.org/pypy/pypy/changeset/08ad056ff621/ Log:make sure that we return an int even when we use 'q' on 32bit diff --git a/pypy/module/struct/formatiterator.py b/pypy/module/struct/formatiterator.py --- a/pypy/module/struct/formatiterator.py +++ b/pypy/module/struct/formatiterator.py @@ -1,4 +1,5 @@ -from rpython.rlib.rarithmetic import r_uint, r_ulonglong, maxint, intmask +from rpython.rlib.rarithmetic import (r_uint, r_ulonglong, r_longlong, + maxint, intmask) from rpython.rlib import jit from rpython.rlib.objectmodel import specialize from rpython.rlib.rstring import StringBuilder @@ -149,13 +150,15 @@ @specialize.argtype(1) def appendobj(self, value): -if isinstance(value, r_uint) or isinstance(value, r_ulonglong): -# unsigned int: space.wrap would wrap it inside a long, but -# CPython tries hard to return an int, if it fits -if value <= maxint: -w_value = self.space.wrap(intmask(value)) -else: -w_value = self.space.wrap(value) +# CPython tries hard to return int objects whenever it can, but +# space.wrap returns a long if we pass a r_uint, r_ulonglong or +# r_longlong. So, we need special care in those cases. +is_unsigned = (isinstance(value, r_uint) or + isinstance(value, r_ulonglong)) +if is_unsigned and value <= maxint: +w_value = self.space.wrap(intmask(value)) +elif isinstance(value, r_longlong) and -maxint-1 <= value <= maxint: +w_value = self.space.wrap(intmask(value)) else: # generic type, just use space.wrap w_value = self.space.wrap(value) diff --git a/pypy/module/struct/test/test_struct.py b/pypy/module/struct/test/test_struct.py --- a/pypy/module/struct/test/test_struct.py +++ b/pypy/module/struct/test/test_struct.py @@ -431,9 +431,9 @@ def test_overflow(self): raises(self.struct.error, self.struct.pack, 'i', 1<<65) -def test_unpack_unsigned(self): +def test_unpack_fits_into_int(self): import sys -for fmt in 'ILQ': +for fmt in 'ILQq': # check that we return an int, if it fits buf = self.struct.pack(fmt, 42) val, = self.struct.unpack(fmt, buf) ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy default: merge branch fix-struct-unpack-Q: make sure that we return an int whenever we can, instead of a long
Author: Antonio Cuni
Branch:
Changeset: r88015:6b78c0c00c1c
Date: 2016-11-01 10:47 +0100
http://bitbucket.org/pypy/pypy/changeset/6b78c0c00c1c/
Log:merge branch fix-struct-unpack-Q: make sure that we return an int
whenever we can, instead of a long
diff --git a/pypy/module/struct/formatiterator.py
b/pypy/module/struct/formatiterator.py
--- a/pypy/module/struct/formatiterator.py
+++ b/pypy/module/struct/formatiterator.py
@@ -1,3 +1,5 @@
+from rpython.rlib.rarithmetic import (r_uint, r_ulonglong, r_longlong,
+ maxint, intmask)
from rpython.rlib import jit
from rpython.rlib.objectmodel import specialize
from rpython.rlib.rstring import StringBuilder
@@ -148,7 +150,20 @@
@specialize.argtype(1)
def appendobj(self, value):
-self.result_w.append(self.space.wrap(value))
+# CPython tries hard to return int objects whenever it can, but
+# space.wrap returns a long if we pass a r_uint, r_ulonglong or
+# r_longlong. So, we need special care in those cases.
+is_unsigned = (isinstance(value, r_uint) or
+ isinstance(value, r_ulonglong))
+if is_unsigned and value <= maxint:
+w_value = self.space.wrap(intmask(value))
+elif isinstance(value, r_longlong) and -maxint-1 <= value <= maxint:
+w_value = self.space.wrap(intmask(value))
+else:
+# generic type, just use space.wrap
+w_value = self.space.wrap(value)
+#
+self.result_w.append(w_value)
def get_pos(self):
return self.pos
diff --git a/pypy/module/struct/test/test_struct.py
b/pypy/module/struct/test/test_struct.py
--- a/pypy/module/struct/test/test_struct.py
+++ b/pypy/module/struct/test/test_struct.py
@@ -431,6 +431,20 @@
def test_overflow(self):
raises(self.struct.error, self.struct.pack, 'i', 1<<65)
+def test_unpack_fits_into_int(self):
+import sys
+for fmt in 'ILQq':
+# check that we return an int, if it fits
+buf = self.struct.pack(fmt, 42)
+val, = self.struct.unpack(fmt, buf)
+assert val == 42
+assert type(val) is int
+#
+# check that we return a long, if it doesn't fit into an int
+buf = self.struct.pack('Q', sys.maxint+1)
+val, = self.struct.unpack('Q', buf)
+assert val == sys.maxint+1
+assert type(val) is long
class AppTestStructBuffer(object):
spaceconfig = dict(usemodules=['struct', '__pypy__'])
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy default: document this branch
Author: Antonio Cuni Branch: Changeset: r88062:c8fcf9b35525 Date: 2016-11-02 11:33 +0100 http://bitbucket.org/pypy/pypy/changeset/c8fcf9b35525/ Log:document this branch diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst --- a/pypy/doc/whatsnew-head.rst +++ b/pypy/doc/whatsnew-head.rst @@ -80,3 +80,10 @@ Improve support for new buffer interface in cpyext, bf_getbuffer on built-in types still missing + +.. branch: fix-struct-unpack-Q + +Improve compatibility with CPython in the ``struct`` module. In particular, +``struct.unpack`` now returns an ``int`` whenever the returned value fits, +while previously it always returned a ``long`` for certains format codes such +as ``Q`` (and also ``I``, ``L`` and ``q`` on 32 bit) ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: introduce rawstorage.str_storage_getitem{, _unaligned}, which works as rawstorage.raw_storage_* but operates on RPython strings instead of raw buffers: since strings
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80668:84c49eed1754
Date: 2015-05-15 22:38 +0200
http://bitbucket.org/pypy/pypy/changeset/84c49eed1754/
Log:introduce rawstorage.str_storage_getitem{,_unaligned}, which works
as rawstorage.raw_storage_* but operates on RPython strings instead
of raw buffers: since strings are immutable, we provide only the
functions for reading out of them, not writing
diff --git a/rpython/rlib/rawstorage.py b/rpython/rlib/rawstorage.py
--- a/rpython/rlib/rawstorage.py
+++ b/rpython/rlib/rawstorage.py
@@ -1,10 +1,13 @@
-from rpython.rlib.objectmodel import we_are_translated
+from rpython.rlib.objectmodel import we_are_translated, keepalive_until_here
from rpython.rtyper.extregistry import ExtRegistryEntry
from rpython.rtyper.lltypesystem import lltype, rffi, llmemory
from rpython.annotator import model as annmodel
from rpython.rtyper.llannotation import lltype_to_annotation
+from rpython.rlib import rgc
from rpython.rlib.rgc import lltype_is_gc
from rpython.rlib.objectmodel import specialize
+from rpython.rtyper.lltypesystem.rstr import STR, _get_raw_str_buf
+from rpython.rtyper.annlowlevel import llstr
RAW_STORAGE = rffi.CCHARP.TO
RAW_STORAGE_PTR = rffi.CCHARP
@@ -40,6 +43,29 @@
def free_raw_storage(storage, track_allocation=True):
lltype.free(storage, flavor='raw', track_allocation=track_allocation)
+
+@rgc.no_collect
+def str_storage_getitem(TP, s, index):
+lls = llstr(s)
+# from here, no GC operations can happen
+buf = _get_raw_str_buf(STR, lls, 0)
+storage = rffi.cast(RAW_STORAGE_PTR, buf)
+res = raw_storage_getitem(TP, storage, index)
+# end of "no GC" section
+keepalive_until_here(lls)
+return res
+
+@rgc.no_collect
+def str_storage_getitem_unaligned(TP, s, index):
+lls = llstr(s)
+# from here, no GC operations can happen
+buf = _get_raw_str_buf(STR, lls, 0)
+storage = rffi.cast(RAW_STORAGE_PTR, buf)
+res = raw_storage_getitem_unaligned(TP, storage, index)
+# end of "no GC" section
+keepalive_until_here(lls)
+return res
+
#
#
# Support for possibly-unaligned accesses
diff --git a/rpython/rlib/test/test_rawstorage.py
b/rpython/rlib/test/test_rawstorage.py
--- a/rpython/rlib/test/test_rawstorage.py
+++ b/rpython/rlib/test/test_rawstorage.py
@@ -4,7 +4,8 @@
from rpython.rlib import rawstorage
from rpython.rlib.rawstorage import alloc_raw_storage, free_raw_storage,\
raw_storage_setitem, raw_storage_getitem, AlignmentError,\
- raw_storage_setitem_unaligned, raw_storage_getitem_unaligned
+ raw_storage_setitem_unaligned, raw_storage_getitem_unaligned,\
+ str_storage_getitem, str_storage_getitem_unaligned
from rpython.rtyper.test.tool import BaseRtypingTest
from rpython.translator.c.test.test_genc import compile
@@ -32,6 +33,26 @@
assert res == 3.14
free_raw_storage(r)
+def test_untranslated_str_storage():
+import struct
+buf = struct.pack('@lld', 42, 43, 123.0)
+size = struct.calcsize('@l')
+res = str_storage_getitem(lltype.Signed, buf, 0)
+assert res == 42
+res = str_storage_getitem(lltype.Signed, buf, size)
+assert res == 43
+res = str_storage_getitem(lltype.Float, buf, size*2)
+assert res == 123.0
+
+def test_untranslated_str_storage_unaligned(monkeypatch):
+import struct
+monkeypatch.setattr(rawstorage, 'misaligned_is_fine', False)
+buf = 'foo' + struct.pack('@ll', 42, 43)
+size = struct.calcsize('@l')
+res = str_storage_getitem_unaligned(lltype.Signed, buf, 3)
+assert res == 42
+res = str_storage_getitem_unaligned(lltype.Signed, buf, size+3)
+assert res == 43
class TestRawStorage(BaseRtypingTest):
@@ -46,6 +67,21 @@
x = self.interpret(f, [1<<30])
assert x == 1 << 30
+def test_str_storage_int(self):
+import struct
+buf = struct.pack('@ll', 42, 43)
+size = struct.calcsize('@l')
+
+def f(i):
+res = str_storage_getitem(lltype.Signed, buf, i)
+return res
+
+x = self.interpret(f, [0])
+assert x == 42
+x = self.interpret(f, [8])
+assert x == 43
+
+
def test_storage_float_unaligned(self, monkeypatch):
def f(v):
r = alloc_raw_storage(24)
diff --git a/rpython/rtyper/lltypesystem/rstr.py
b/rpython/rtyper/lltypesystem/rstr.py
--- a/rpython/rtyper/lltypesystem/rstr.py
+++ b/rpython/rtyper/lltypesystem/rstr.py
@@ -60,6 +60,13 @@
@signature(types.any(), types.any(), types.int(), returns=types.any())
@specialize.arg(0)
def _get_raw_buf(TP, src, ofs):
+"""
+WARNING: dragons ahead.
+Return the address of the internal char* buffer of the low level
+string. The return value is valid as long as no GC operation occur, so
+you must ensure that it will be used inside a "GC safe" section, for
+example by mar
[pypy-commit] pypy faster-rstruct: a branch where to improve the performance of rstruct
Author: Antonio Cuni Branch: faster-rstruct Changeset: r80667:d9628e7cafa0 Date: 2015-11-13 17:20 +0100 http://bitbucket.org/pypy/pypy/changeset/d9628e7cafa0/ Log:a branch where to improve the performance of rstruct ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: use the fast raw_storage way to read doubles and floats when rstruct is using native formats
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80669:b790b4f590e3
Date: 2015-11-13 17:18 +0100
http://bitbucket.org/pypy/pypy/changeset/b790b4f590e3/
Log:use the fast raw_storage way to read doubles and floats when rstruct
is using native formats
diff --git a/rpython/rlib/rawstorage.py b/rpython/rlib/rawstorage.py
--- a/rpython/rlib/rawstorage.py
+++ b/rpython/rlib/rawstorage.py
@@ -45,6 +45,7 @@
@rgc.no_collect
+@specialize.ll()
def str_storage_getitem(TP, s, index):
lls = llstr(s)
# from here, no GC operations can happen
@@ -56,6 +57,7 @@
return res
@rgc.no_collect
+@specialize.ll()
def str_storage_getitem_unaligned(TP, s, index):
lls = llstr(s)
# from here, no GC operations can happen
diff --git a/rpython/rlib/rstruct/nativefmttable.py
b/rpython/rlib/rstruct/nativefmttable.py
--- a/rpython/rlib/rstruct/nativefmttable.py
+++ b/rpython/rlib/rstruct/nativefmttable.py
@@ -10,6 +10,7 @@
from rpython.rlib.rstruct import standardfmttable as std
from rpython.rlib.rstruct.error import StructError
from rpython.rlib.unroll import unrolling_iterable
+from rpython.rlib.rawstorage import str_storage_getitem
from rpython.rtyper.lltypesystem import lltype, rffi
from rpython.rtyper.tool import rffi_platform
from rpython.translator.tool.cbuild import ExternalCompilationInfo
@@ -27,9 +28,6 @@
#
-double_buf = lltype.malloc(rffi.DOUBLEP.TO, 1, flavor='raw', immortal=True)
-float_buf = lltype.malloc(rffi.FLOATP.TO, 1, flavor='raw', immortal=True)
-
range_8_unroll = unrolling_iterable(list(reversed(range(8
range_4_unroll = unrolling_iterable(list(reversed(range(4
@@ -48,10 +46,7 @@
@specialize.argtype(0)
def unpack_double(fmtiter):
input = fmtiter.read(sizeof_double)
-p = rffi.cast(rffi.CCHARP, double_buf)
-for i in range(sizeof_double):
-p[i] = input[i]
-doubleval = double_buf[0]
+doubleval = str_storage_getitem(rffi.DOUBLE, input, 0)
fmtiter.appendobj(doubleval)
def pack_float(fmtiter):
@@ -71,11 +66,8 @@
@specialize.argtype(0)
def unpack_float(fmtiter):
input = fmtiter.read(sizeof_float)
-p = rffi.cast(rffi.CCHARP, float_buf)
-for i in range(sizeof_float):
-p[i] = input[i]
-floatval = float_buf[0]
-doubleval = float(floatval)
+floatval = str_storage_getitem(rffi.FLOAT, input, 0)
+doubleval = float(floatval) # convert from r_singlefloat to rpython's float
fmtiter.appendobj(doubleval)
#
diff --git a/rpython/rlib/rstruct/test/test_runpack.py
b/rpython/rlib/rstruct/test/test_runpack.py
--- a/rpython/rlib/rstruct/test/test_runpack.py
+++ b/rpython/rlib/rstruct/test/test_runpack.py
@@ -37,3 +37,21 @@
return runpack(">d", "testtest")
assert fn() == struct.unpack(">d", "testtest")[0]
assert self.interpret(fn, []) == struct.unpack(">d", "testtest")[0]
+
+def test_native_floats(self):
+"""
+Check the 'd' and 'f' format characters on native packing.
+"""
+d_data = struct.pack("d", 12.34)
+f_data = struct.pack("f", 12.34)
+def fn():
+d = runpack("@d", d_data)
+f = runpack("@f", f_data)
+return d, f
+#
+res = self.interpret(fn, [])
+d = res.item0
+f = res.item1 # convert from r_singlefloat
+assert d == 12.34 # no precision lost
+assert f != 12.34 # precision lost
+assert abs(f - 12.34) < 1E-6
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: use the fast raw_storage unpacking also for integer types, when possible
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80670:9972a7f270d5
Date: 2015-11-13 19:01 +0100
http://bitbucket.org/pypy/pypy/changeset/9972a7f270d5/
Log:use the fast raw_storage unpacking also for integer types, when
possible
diff --git a/rpython/rlib/rstruct/nativefmttable.py
b/rpython/rlib/rstruct/nativefmttable.py
--- a/rpython/rlib/rstruct/nativefmttable.py
+++ b/rpython/rlib/rstruct/nativefmttable.py
@@ -8,6 +8,7 @@
from rpython.rlib.objectmodel import specialize
from rpython.rlib.rarithmetic import r_singlefloat, widen
from rpython.rlib.rstruct import standardfmttable as std
+from rpython.rlib.rstruct.standardfmttable import native_is_bigendian
from rpython.rlib.rstruct.error import StructError
from rpython.rlib.unroll import unrolling_iterable
from rpython.rlib.rawstorage import str_storage_getitem
@@ -16,8 +17,6 @@
from rpython.translator.tool.cbuild import ExternalCompilationInfo
-native_is_bigendian = struct.pack("=i", 1) == struct.pack(">i", 1)
-
native_fmttable = {
'x': std.standard_fmttable['x'],
'c': std.standard_fmttable['c'],
diff --git a/rpython/rlib/rstruct/standardfmttable.py
b/rpython/rlib/rstruct/standardfmttable.py
--- a/rpython/rlib/rstruct/standardfmttable.py
+++ b/rpython/rlib/rstruct/standardfmttable.py
@@ -12,7 +12,11 @@
from rpython.rlib.rstruct import ieee
from rpython.rlib.rstruct.error import StructError, StructOverflowError
from rpython.rlib.unroll import unrolling_iterable
+from rpython.rlib.rawstorage import str_storage_getitem
+from rpython.rlib import rarithmetic
+from rpython.rtyper.lltypesystem import rffi
+native_is_bigendian = struct.pack("=i", 1) == struct.pack(">i", 1)
def pack_pad(fmtiter, count):
fmtiter.result.append_multiple_char('\x00', count)
@@ -162,6 +166,15 @@
#
+def get_rffi_int_type(size, signed):
+for TYPE in rffi.platform.numbertype_to_rclass:
+if (rffi.sizeof(TYPE) == size and
+rarithmetic.is_signed_integer_type(TYPE) == signed):
+return TYPE
+raise KeyError("Cannot find an int type size=%d, signed=%d" % (size,
signed))
+
+UNPACK_ALLOW_RAW_STORAGE = True
+
def make_int_unpacker(size, signed, _memo={}):
try:
return _memo[size, signed]
@@ -180,13 +193,20 @@
else:
inttype = r_ulonglong
unroll_range_size = unrolling_iterable(range(size))
+TYPE = get_rffi_int_type(size, signed)
@specialize.argtype(0)
def unpack_int(fmtiter):
intvalue = inttype(0)
s = fmtiter.read(size)
idx = 0
-if fmtiter.bigendian:
+if UNPACK_ALLOW_RAW_STORAGE and fmtiter.bigendian ==
native_is_bigendian:
+# fast path, using the native raw_storage
+intvalue = str_storage_getitem(TYPE, s, 0)
+if not signed and size < native_int_size:
+intvalue = rarithmetic.intmask(intvalue)
+intvalue = inttype(intvalue)
+elif fmtiter.bigendian:
for i in unroll_range_size:
x = ord(s[idx])
if signed and i == 0 and x >= 128:
diff --git a/rpython/rlib/rstruct/test/test_runpack.py
b/rpython/rlib/rstruct/test/test_runpack.py
--- a/rpython/rlib/rstruct/test/test_runpack.py
+++ b/rpython/rlib/rstruct/test/test_runpack.py
@@ -1,5 +1,6 @@
from rpython.rtyper.test.tool import BaseRtypingTest
from rpython.rlib.rstruct.runpack import runpack
+from rpython.rlib.rstruct import standardfmttable
from rpython.rlib.rarithmetic import LONG_BIT
import struct
@@ -55,3 +56,37 @@
assert d == 12.34 # no precision lost
assert f != 12.34 # precision lost
assert abs(f - 12.34) < 1E-6
+
+def test_unpack_standard_little(self):
+def unpack(fmt, data):
+def fn():
+return runpack(fmt, data)
+return self.interpret(fn, [])
+#
+assert unpack("i", 'ABCD') == 0x41424344
+assert unpack(">i", '\xff\xff\xff\xfd') == -3
+assert unpack(">i", '\x80\x00\x00\x00') == -2147483648
+assert unpack(">I", '\x81BCD') == 0x81424344
+assert unpack(">q", 'ABCDEFGH') == 0x4142434445464748
+assert unpack(">q", '\xbeMLKJIHH') == -0x41B2B3B4B5B6B7B8
+assert unpack(">Q", '\x81BCDEFGH') == 0x8142434445464748
+
+def test_unpack_standard_no_raw_storage(self, monkeypatch):
+monkeypatch.setattr(standardfmttable, 'UNPACK_ALLOW_RAW_STORAGE',
False)
+self.test_unpack_standard_little()
+self.test_unpack_standard_big()
+
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: hg merge default
Author: Antonio Cuni Branch: faster-rstruct Changeset: r80671:ec2db01088b1 Date: 2015-11-14 02:08 +0100 http://bitbucket.org/pypy/pypy/changeset/ec2db01088b1/ Log:hg merge default diff --git a/pypy/module/cpyext/pystrtod.py b/pypy/module/cpyext/pystrtod.py --- a/pypy/module/cpyext/pystrtod.py +++ b/pypy/module/cpyext/pystrtod.py @@ -5,6 +5,7 @@ from rpython.rlib import rdtoa from rpython.rlib import rfloat from rpython.rlib import rposix, jit +from rpython.rlib.rarithmetic import intmask from rpython.rtyper.lltypesystem import lltype from rpython.rtyper.lltypesystem import rffi @@ -112,7 +113,9 @@ NULL if the conversion failed. The caller is responsible for freeing the returned string by calling PyMem_Free(). """ -buffer, rtype = rfloat.double_to_string(val, format_code, precision, flags) +buffer, rtype = rfloat.double_to_string(val, format_code, +intmask(precision), +intmask(flags)) if ptype != lltype.nullptr(rffi.INTP.TO): ptype[0] = rffi.cast(rffi.INT, DOUBLE_TO_STRING_TYPES_MAP[rtype]) bufp = rffi.str2charp(buffer) diff --git a/rpython/jit/backend/llsupport/rewrite.py b/rpython/jit/backend/llsupport/rewrite.py --- a/rpython/jit/backend/llsupport/rewrite.py +++ b/rpython/jit/backend/llsupport/rewrite.py @@ -120,13 +120,14 @@ # this case means between CALLs or unknown-size mallocs. # operations = self.remove_bridge_exception(operations) -self._source_operations = operations +self._changed_op = None for i in range(len(operations)): -self._current_position = i op = operations[i] assert op.get_forwarded() is None if op.getopnum() == rop.DEBUG_MERGE_POINT: continue +if op is self._changed_op: +op = self._changed_op_to # -- GETFIELD_GC -- if op.getopnum() in (rop.GETFIELD_GC_I, rop.GETFIELD_GC_F, rop.GETFIELD_GC_R): @@ -213,11 +214,10 @@ self.emit_op(op1) lst = op.getfailargs()[:] lst[i] = op1 -operations = self._source_operations -assert operations[self._current_position + 1] is op newop = op.copy_and_change(opnum) newop.setfailargs(lst) -operations[self._current_position + 1] = newop +self._changed_op = op +self._changed_op_to = newop # -- ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: fix
Author: Antonio Cuni Branch: faster-rstruct Changeset: r80673:e7449f376b4e Date: 2015-11-14 13:08 + http://bitbucket.org/pypy/pypy/changeset/e7449f376b4e/ Log:fix diff --git a/rpython/rtyper/lltypesystem/rbytearray.py b/rpython/rtyper/lltypesystem/rbytearray.py --- a/rpython/rtyper/lltypesystem/rbytearray.py +++ b/rpython/rtyper/lltypesystem/rbytearray.py @@ -8,10 +8,10 @@ def mallocbytearray(size): return lltype.malloc(BYTEARRAY, size) -_, _, copy_bytearray_contents = rstr._new_copy_contents_fun(BYTEARRAY, BYTEARRAY, +_, _, _, copy_bytearray_contents = rstr._new_copy_contents_fun(BYTEARRAY, BYTEARRAY, lltype.Char, 'bytearray') -_, _, copy_bytearray_contents_from_str = rstr._new_copy_contents_fun(rstr.STR, +_, _, _, copy_bytearray_contents_from_str = rstr._new_copy_contents_fun(rstr.STR, BYTEARRAY, lltype.Char, 'bytearray_from_str') ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: IN-PROGRESS: we would like this test to pass, failing so far
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80725:6fbe291f9f9b
Date: 2015-11-17 11:24 +0100
http://bitbucket.org/pypy/pypy/changeset/6fbe291f9f9b/
Log:IN-PROGRESS: we would like this test to pass, failing so far
diff --git a/rpython/jit/metainterp/test/test_rawmem.py
b/rpython/jit/metainterp/test/test_rawmem.py
--- a/rpython/jit/metainterp/test/test_rawmem.py
+++ b/rpython/jit/metainterp/test/test_rawmem.py
@@ -1,7 +1,8 @@
from rpython.jit.metainterp.test.support import LLJitMixin
from rpython.rtyper.lltypesystem import lltype, rffi
from rpython.rlib.rawstorage import (alloc_raw_storage, raw_storage_setitem,
- free_raw_storage, raw_storage_getitem)
+ free_raw_storage, raw_storage_getitem,
+ str_storage_getitem)
class RawMemTests(object):
@@ -105,6 +106,21 @@
res = self.interp_operations(f, [])
assert res == ord('X')
+def test_str_storage_int(self):
+import struct
+data = struct.pack('q', 42)
+def f():
+res = str_storage_getitem(lltype.Signed, data, 0)
+return res
+res = self.interp_operations(f, [])
+assert res == 42
+import pdb;pdb.set_trace()
+self.check_operations_history({'call_i': 1, 'guard_no_exception': 1,
+ 'call_n': 1,
+ 'raw_store': 1, 'raw_load_i': 1,
+ 'finish': 1})
+self.metainterp.staticdata.stats.check_resops({'finish': 1},
omit_finish=False)
+
class TestRawMem(RawMemTests, LLJitMixin):
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: move str_storage_getitem into its own module, as it's no longer based on rawstorage. Refactor, and re-implement based on casting. Add JIT support for force_cast betw
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80736:2294ad1228f5
Date: 2015-11-18 07:21 +0100
http://bitbucket.org/pypy/pypy/changeset/2294ad1228f5/
Log:move str_storage_getitem into its own module, as it's no longer
based on rawstorage. Refactor, and re-implement based on casting.
Add JIT support for force_cast between GC objects and the specific
case of getinteriorfield we need. Add enough support to the llgraph
backend to run the tests
diff --git a/rpython/jit/backend/llgraph/runner.py
b/rpython/jit/backend/llgraph/runner.py
--- a/rpython/jit/backend/llgraph/runner.py
+++ b/rpython/jit/backend/llgraph/runner.py
@@ -638,9 +638,18 @@
return array.getlength()
def bh_getarrayitem_gc(self, a, index, descr):
-a = support.cast_arg(lltype.Ptr(descr.A), a)
+assert index >= 0
+if descr.A is descr.OUTERA:
+a = support.cast_arg(lltype.Ptr(descr.A), a)
+else:
+# we use rffi.cast instead of support.cast_arg because the types
+# might not be "compatible" enough from the lltype point of
+# view. In particular, this happens when we use
+# str_storage_getitem, in which an rpy_string is casted to
+# rpy_string_as_Signed (or similar)
+a = rffi.cast(lltype.Ptr(descr.OUTERA), a)
+a = getattr(a, descr.OUTERA._arrayfld)
array = a._obj
-assert index >= 0
return support.cast_result(descr.A.OF, array.getitem(index))
bh_getarrayitem_gc_pure_i = bh_getarrayitem_gc
diff --git a/rpython/jit/codewriter/jtransform.py
b/rpython/jit/codewriter/jtransform.py
--- a/rpython/jit/codewriter/jtransform.py
+++ b/rpython/jit/codewriter/jtransform.py
@@ -1008,12 +1008,11 @@
return SpaceOperation('getarrayitem_gc_i',
[op.args[0], v_index, bytearraydescr],
op.result)
-else:
+elif op.result.concretetype is lltype.Void:
+return
+elif isinstance(op.args[0].concretetype.TO, lltype.GcArray):
+# special-case 1: GcArray of Struct
v_inst, v_index, c_field = op.args
-if op.result.concretetype is lltype.Void:
-return
-# only GcArray of Struct supported
-assert isinstance(v_inst.concretetype.TO, lltype.GcArray)
STRUCT = v_inst.concretetype.TO.OF
assert isinstance(STRUCT, lltype.Struct)
descr = self.cpu.interiorfielddescrof(v_inst.concretetype.TO,
@@ -1022,6 +1021,16 @@
kind = getkind(op.result.concretetype)[0]
return SpaceOperation('getinteriorfield_gc_%s' % kind, args,
op.result)
+elif isinstance(op.args[0].concretetype.TO, lltype.GcStruct):
+# special-case 2: GcStruct with Array field
+v_inst, c_field, v_index = op.args
+STRUCT = v_inst.concretetype.TO
+ARRAY = getattr(STRUCT, c_field.value)
+assert isinstance(ARRAY, lltype.Array)
+arraydescr = self.cpu.arraydescrof(STRUCT)
+return SpaceOperation('getarrayitem_gc_i',
+ [op.args[0], v_index, arraydescr],
+ op.result)
def rewrite_op_setinteriorfield(self, op):
assert len(op.args) == 4
@@ -1130,10 +1139,13 @@
def rewrite_op_force_cast(self, op):
v_arg = op.args[0]
v_result = op.result
-assert not self._is_gc(v_arg)
-
if v_arg.concretetype == v_result.concretetype:
return
+elif self._is_gc(v_arg) and self._is_gc(v_result):
+# cast from GC to GC is always fine
+return
+else:
+assert not self._is_gc(v_arg)
float_arg = v_arg.concretetype in [lltype.Float, lltype.SingleFloat]
float_res = v_result.concretetype in [lltype.Float, lltype.SingleFloat]
diff --git a/rpython/jit/metainterp/test/test_rawmem.py
b/rpython/jit/metainterp/test/test_rawmem.py
--- a/rpython/jit/metainterp/test/test_rawmem.py
+++ b/rpython/jit/metainterp/test/test_rawmem.py
@@ -108,18 +108,17 @@
def test_str_storage_int(self):
import struct
-data = struct.pack('q', 42)
+data = struct.pack('qq', 42, 100)
def f():
-res = str_storage_getitem(lltype.Signed, data, 0)
-return res
+a = str_storage_getitem(lltype.Signed, data, 0)
+b = str_storage_getitem(lltype.Signed, data, 8)
+return a+b
res = self.interp_operations(f, [])
-assert res == 42
-import pdb;pdb.set_trace()
-self.check_operations_history({'call_i': 1, 'guard_no_exception': 1,
- 'call_n': 1,
- 'raw_store': 1, 'raw_load_i': 1,
+assert res == 142
+self.check_operati
[pypy-commit] pypy faster-rstruct: move strstorage tests into their own file, and refactor to avoid duplication
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80737:f92d64d0ed7e
Date: 2015-11-18 07:54 +0100
http://bitbucket.org/pypy/pypy/changeset/f92d64d0ed7e/
Log:move strstorage tests into their own file, and refactor to avoid
duplication
diff --git a/rpython/rlib/rawstorage.py b/rpython/rlib/rawstorage.py
--- a/rpython/rlib/rawstorage.py
+++ b/rpython/rlib/rawstorage.py
@@ -6,7 +6,6 @@
from rpython.rlib import rgc
from rpython.rlib.rgc import lltype_is_gc
from rpython.rlib.objectmodel import specialize
-from rpython.rlib.strstorage import str_storage_getitem
RAW_STORAGE = rffi.CCHARP.TO
RAW_STORAGE_PTR = rffi.CCHARP
diff --git a/rpython/rlib/test/test_rawstorage.py
b/rpython/rlib/test/test_rawstorage.py
--- a/rpython/rlib/test/test_rawstorage.py
+++ b/rpython/rlib/test/test_rawstorage.py
@@ -4,8 +4,7 @@
from rpython.rlib import rawstorage
from rpython.rlib.rawstorage import alloc_raw_storage, free_raw_storage,\
raw_storage_setitem, raw_storage_getitem, AlignmentError,\
- raw_storage_setitem_unaligned, raw_storage_getitem_unaligned,\
- str_storage_getitem
+ raw_storage_setitem_unaligned, raw_storage_getitem_unaligned
from rpython.rtyper.test.tool import BaseRtypingTest
from rpython.translator.c.test.test_genc import compile
@@ -33,17 +32,6 @@
assert res == 3.14
free_raw_storage(r)
-def test_untranslated_str_storage():
-import struct
-buf = struct.pack('@lld', 42, 43, 123.0)
-size = struct.calcsize('@l')
-res = str_storage_getitem(lltype.Signed, buf, 0)
-assert res == 42
-res = str_storage_getitem(lltype.Signed, buf, size)
-assert res == 43
-res = str_storage_getitem(lltype.Float, buf, size*2)
-assert res == 123.0
-
class TestRawStorage(BaseRtypingTest):
def test_storage_int(self):
@@ -57,21 +45,6 @@
x = self.interpret(f, [1<<30])
assert x == 1 << 30
-def test_str_storage_int(self):
-import struct
-buf = struct.pack('@ll', 42, 43)
-size = struct.calcsize('@l')
-
-def f(i):
-res = str_storage_getitem(lltype.Signed, buf, i)
-return res
-
-x = self.interpret(f, [0])
-assert x == 42
-x = self.interpret(f, [8])
-assert x == 43
-
-
def test_storage_float_unaligned(self, monkeypatch):
def f(v):
r = alloc_raw_storage(24)
diff --git a/rpython/rlib/test/test_strstorage.py
b/rpython/rlib/test/test_strstorage.py
new file mode 100644
--- /dev/null
+++ b/rpython/rlib/test/test_strstorage.py
@@ -0,0 +1,33 @@
+import py
+import struct
+from rpython.rtyper.lltypesystem import lltype
+from rpython.rlib.strstorage import str_storage_getitem
+from rpython.rtyper.test.tool import BaseRtypingTest
+
+class BaseStrStorageTest:
+
+def test_signed(self):
+buf = struct.pack('@ll', 42, 43)
+size = struct.calcsize('@l')
+assert self.str_storage_getitem(lltype.Signed, buf, 0) == 42
+assert self.str_storage_getitem(lltype.Signed, buf, size) == 43
+
+def test_float(self):
+buf = struct.pack('@dd', 12.3, 45.6)
+size = struct.calcsize('@d')
+assert self.str_storage_getitem(lltype.Float, buf, 0) == 12.3
+assert self.str_storage_getitem(lltype.Float, buf, size) == 45.6
+
+
+class TestDirect(BaseStrStorageTest):
+
+def str_storage_getitem(self, TYPE, buf, offset):
+return str_storage_getitem(TYPE, buf, offset)
+
+
+class TestRTyping(BaseStrStorageTest, BaseRtypingTest):
+
+def str_storage_getitem(self, TYPE, buf, offset):
+def fn(offset):
+return str_storage_getitem(TYPE, buf, offset)
+return self.interpret(fn, [offset])
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: re-use BaseStrStorage to implement JIT tests; test_float is failing, as expected
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80738:35a943788bea
Date: 2015-11-18 08:16 +0100
http://bitbucket.org/pypy/pypy/changeset/35a943788bea/
Log:re-use BaseStrStorage to implement JIT tests; test_float is failing,
as expected
diff --git a/rpython/jit/metainterp/test/test_rawmem.py
b/rpython/jit/metainterp/test/test_rawmem.py
--- a/rpython/jit/metainterp/test/test_rawmem.py
+++ b/rpython/jit/metainterp/test/test_rawmem.py
@@ -1,8 +1,7 @@
from rpython.jit.metainterp.test.support import LLJitMixin
from rpython.rtyper.lltypesystem import lltype, rffi
from rpython.rlib.rawstorage import (alloc_raw_storage, raw_storage_setitem,
- free_raw_storage, raw_storage_getitem,
- str_storage_getitem)
+ free_raw_storage, raw_storage_getitem)
class RawMemTests(object):
@@ -106,20 +105,6 @@
res = self.interp_operations(f, [])
assert res == ord('X')
-def test_str_storage_int(self):
-import struct
-data = struct.pack('qq', 42, 100)
-def f():
-a = str_storage_getitem(lltype.Signed, data, 0)
-b = str_storage_getitem(lltype.Signed, data, 8)
-return a+b
-res = self.interp_operations(f, [])
-assert res == 142
-self.check_operations_history({'getarrayitem_gc_i': 2,
- 'int_add': 1,
- 'finish': 1})
-
-
class TestRawMem(RawMemTests, LLJitMixin):
diff --git a/rpython/jit/metainterp/test/test_strstorage.py
b/rpython/jit/metainterp/test/test_strstorage.py
new file mode 100644
--- /dev/null
+++ b/rpython/jit/metainterp/test/test_strstorage.py
@@ -0,0 +1,17 @@
+import py
+from rpython.rlib.strstorage import str_storage_getitem
+from rpython.rlib.test.test_strstorage import BaseStrStorageTest
+from rpython.jit.metainterp.test.support import LLJitMixin
+
+class TestStrStorage(BaseStrStorageTest, LLJitMixin):
+
+# for the individual tests see
+# > ../../../rlib/test/test_strstorage.py
+
+def str_storage_getitem(self, TYPE, buf, offset):
+def f():
+return str_storage_getitem(TYPE, buf, offset)
+res = self.interp_operations(f, [])
+self.check_operations_history({'getarrayitem_gc_i': 1,
+ 'finish': 1})
+return res
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: support str_storage_getitem(lltype.Float, ...)
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80740:b5ebdf19edb4
Date: 2015-11-18 08:27 +0100
http://bitbucket.org/pypy/pypy/changeset/b5ebdf19edb4/
Log:support str_storage_getitem(lltype.Float, ...)
diff --git a/rpython/jit/codewriter/jtransform.py
b/rpython/jit/codewriter/jtransform.py
--- a/rpython/jit/codewriter/jtransform.py
+++ b/rpython/jit/codewriter/jtransform.py
@@ -1028,7 +1028,9 @@
ARRAY = getattr(STRUCT, c_field.value)
assert isinstance(ARRAY, lltype.Array)
arraydescr = self.cpu.arraydescrof(STRUCT)
-return SpaceOperation('getarrayitem_gc_i',
+kind = getkind(op.result.concretetype)[0]
+assert kind in ('i', 'f')
+return SpaceOperation('getarrayitem_gc_%s' % kind,
[op.args[0], v_index, arraydescr],
op.result)
diff --git a/rpython/jit/metainterp/test/test_strstorage.py
b/rpython/jit/metainterp/test/test_strstorage.py
--- a/rpython/jit/metainterp/test/test_strstorage.py
+++ b/rpython/jit/metainterp/test/test_strstorage.py
@@ -1,6 +1,7 @@
import py
from rpython.rlib.strstorage import str_storage_getitem
from rpython.rlib.test.test_strstorage import BaseStrStorageTest
+from rpython.jit.metainterp.history import getkind
from rpython.jit.metainterp.test.support import LLJitMixin
class TestStrStorage(BaseStrStorageTest, LLJitMixin):
@@ -12,6 +13,8 @@
def f():
return str_storage_getitem(TYPE, buf, offset)
res = self.interp_operations(f, [])
-self.check_operations_history({'getarrayitem_gc_i': 1,
+#
+kind = getkind(TYPE)[0] # 'i' or 'f'
+self.check_operations_history({'getarrayitem_gc_%s' % kind: 1,
'finish': 1})
return res
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: aargh, forgot to add the default case. Thanks arigato
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80741:86622dc58965
Date: 2015-11-18 08:27 +0100
http://bitbucket.org/pypy/pypy/changeset/86622dc58965/
Log:aargh, forgot to add the default case. Thanks arigato
diff --git a/rpython/jit/codewriter/jtransform.py
b/rpython/jit/codewriter/jtransform.py
--- a/rpython/jit/codewriter/jtransform.py
+++ b/rpython/jit/codewriter/jtransform.py
@@ -1033,6 +1033,8 @@
return SpaceOperation('getarrayitem_gc_%s' % kind,
[op.args[0], v_index, arraydescr],
op.result)
+else:
+assert False, 'not supported'
def rewrite_op_setinteriorfield(self, op):
assert len(op.args) == 4
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: add a passing test for singlefloats
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80745:85c3b26dea8a
Date: 2015-11-18 09:29 +0100
http://bitbucket.org/pypy/pypy/changeset/85c3b26dea8a/
Log:add a passing test for singlefloats
diff --git a/rpython/jit/metainterp/test/test_strstorage.py
b/rpython/jit/metainterp/test/test_strstorage.py
--- a/rpython/jit/metainterp/test/test_strstorage.py
+++ b/rpython/jit/metainterp/test/test_strstorage.py
@@ -1,6 +1,8 @@
import py
+from rpython.rtyper.lltypesystem import lltype
from rpython.rlib.strstorage import str_storage_getitem
from rpython.rlib.test.test_strstorage import BaseStrStorageTest
+from rpython.jit.codewriter import longlong
from rpython.jit.metainterp.history import getkind
from rpython.jit.metainterp.test.support import LLJitMixin
@@ -12,9 +14,15 @@
def str_storage_getitem(self, TYPE, buf, offset):
def f():
return str_storage_getitem(TYPE, buf, offset)
-res = self.interp_operations(f, [])
+res = self.interp_operations(f, [], supports_singlefloats=True)
#
kind = getkind(TYPE)[0] # 'i' or 'f'
self.check_operations_history({'getarrayitem_gc_%s' % kind: 1,
'finish': 1})
+#
+if TYPE == lltype.SingleFloat:
+# interp_operations returns the int version of r_singlefloat, but
+# our tests expects to receive an r_singlefloat: let's convert it
+# back!
+return longlong.int2singlefloat(res)
return res
diff --git a/rpython/rlib/test/test_strstorage.py
b/rpython/rlib/test/test_strstorage.py
--- a/rpython/rlib/test/test_strstorage.py
+++ b/rpython/rlib/test/test_strstorage.py
@@ -2,6 +2,7 @@
import struct
from rpython.rtyper.lltypesystem import lltype, rffi
from rpython.rlib.strstorage import str_storage_getitem
+from rpython.rlib.rarithmetic import r_singlefloat
from rpython.rtyper.test.tool import BaseRtypingTest
class BaseStrStorageTest:
@@ -26,6 +27,15 @@
assert self.str_storage_getitem(lltype.Float, buf, 0) == 12.3
assert self.str_storage_getitem(lltype.Float, buf, size) == 45.6
+def test_singlefloat(self):
+buf = struct.pack('@ff', 12.3, 45.6)
+size = struct.calcsize('@f')
+x = self.str_storage_getitem(lltype.SingleFloat, buf, 0)
+assert x == r_singlefloat(12.3)
+x = self.str_storage_getitem(lltype.SingleFloat, buf, size)
+assert x == r_singlefloat(45.6)
+
+
class TestDirect(BaseStrStorageTest):
def str_storage_getitem(self, TYPE, buf, offset):
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: add a passing test for shorts
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80744:7f76146cee2a
Date: 2015-11-18 09:00 +0100
http://bitbucket.org/pypy/pypy/changeset/7f76146cee2a/
Log:add a passing test for shorts
diff --git a/rpython/rlib/test/test_strstorage.py
b/rpython/rlib/test/test_strstorage.py
--- a/rpython/rlib/test/test_strstorage.py
+++ b/rpython/rlib/test/test_strstorage.py
@@ -1,6 +1,6 @@
import py
import struct
-from rpython.rtyper.lltypesystem import lltype
+from rpython.rtyper.lltypesystem import lltype, rffi
from rpython.rlib.strstorage import str_storage_getitem
from rpython.rtyper.test.tool import BaseRtypingTest
@@ -12,13 +12,20 @@
assert self.str_storage_getitem(lltype.Signed, buf, 0) == 42
assert self.str_storage_getitem(lltype.Signed, buf, size) == 43
+def test_short(self):
+buf = struct.pack('@hh', 42, 43)
+size = struct.calcsize('@h')
+x = self.str_storage_getitem(rffi.SHORT, buf, 0)
+assert int(x) == 42
+x = self.str_storage_getitem(rffi.SHORT, buf, size)
+assert int(x) == 43
+
def test_float(self):
buf = struct.pack('@dd', 12.3, 45.6)
size = struct.calcsize('@d')
assert self.str_storage_getitem(lltype.Float, buf, 0) == 12.3
assert self.str_storage_getitem(lltype.Float, buf, size) == 45.6
-
class TestDirect(BaseStrStorageTest):
def str_storage_getitem(self, TYPE, buf, offset):
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: restore rawstorage.py as it is on default
Author: Antonio Cuni Branch: faster-rstruct Changeset: r80742:2b3a2c2f3640 Date: 2015-11-18 08:32 +0100 http://bitbucket.org/pypy/pypy/changeset/2b3a2c2f3640/ Log:restore rawstorage.py as it is on default diff --git a/rpython/rlib/rawstorage.py b/rpython/rlib/rawstorage.py --- a/rpython/rlib/rawstorage.py +++ b/rpython/rlib/rawstorage.py @@ -1,9 +1,8 @@ -from rpython.rlib.objectmodel import we_are_translated, keepalive_until_here +from rpython.rlib.objectmodel import we_are_translated from rpython.rtyper.extregistry import ExtRegistryEntry from rpython.rtyper.lltypesystem import lltype, rffi, llmemory from rpython.annotator import model as annmodel from rpython.rtyper.llannotation import lltype_to_annotation -from rpython.rlib import rgc from rpython.rlib.rgc import lltype_is_gc from rpython.rlib.objectmodel import specialize @@ -41,7 +40,6 @@ def free_raw_storage(storage, track_allocation=True): lltype.free(storage, flavor='raw', track_allocation=track_allocation) - # # # Support for possibly-unaligned accesses ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: run the strstorage tests also on top of the x86 backed
Author: Antonio Cuni Branch: faster-rstruct Changeset: r80746:d528df864b73 Date: 2015-11-18 09:30 +0100 http://bitbucket.org/pypy/pypy/changeset/d528df864b73/ Log:run the strstorage tests also on top of the x86 backed diff --git a/rpython/jit/backend/x86/test/test_strstorage.py b/rpython/jit/backend/x86/test/test_strstorage.py new file mode 100644 --- /dev/null +++ b/rpython/jit/backend/x86/test/test_strstorage.py @@ -0,0 +1,8 @@ +from rpython.jit.backend.x86.test.test_basic import Jit386Mixin +from rpython.jit.metainterp.test.test_strstorage import TestStrStorage as _TestStrStorage + + +class TestStrStorage(Jit386Mixin, _TestStrStorage): +# for the individual tests see +# > ../../../metainterp/test/test_strstorage.py +pass ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: we no longer need @rgc.nocollect
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80743:5f6a7b8eaf84
Date: 2015-11-18 08:59 +0100
http://bitbucket.org/pypy/pypy/changeset/5f6a7b8eaf84/
Log:we no longer need @rgc.nocollect
diff --git a/rpython/rlib/strstorage.py b/rpython/rlib/strstorage.py
--- a/rpython/rlib/strstorage.py
+++ b/rpython/rlib/strstorage.py
@@ -31,7 +31,6 @@
from rpython.rtyper.lltypesystem.rstr import STR, _get_raw_str_buf
from rpython.rtyper.annlowlevel import llstr
from rpython.rlib.objectmodel import specialize
-from rpython.rlib import rgc
@specialize.memo()
def rpy_string_as_type(TP):
@@ -45,7 +44,6 @@
('chars', lltype.Array(TP, hints={'immutable':
True})))
return STR_AS_TP
-@rgc.no_collect
@specialize.ll()
def str_storage_getitem(TP, s, index):
STR_AS_TP = rpy_string_as_type(TP)
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: fix imports
Author: Antonio Cuni Branch: faster-rstruct Changeset: r80749:4c362a658da5 Date: 2015-11-18 10:01 +0100 http://bitbucket.org/pypy/pypy/changeset/4c362a658da5/ Log:fix imports diff --git a/rpython/rlib/rstruct/nativefmttable.py b/rpython/rlib/rstruct/nativefmttable.py --- a/rpython/rlib/rstruct/nativefmttable.py +++ b/rpython/rlib/rstruct/nativefmttable.py @@ -11,7 +11,7 @@ from rpython.rlib.rstruct.standardfmttable import native_is_bigendian from rpython.rlib.rstruct.error import StructError from rpython.rlib.unroll import unrolling_iterable -from rpython.rlib.rawstorage import str_storage_getitem +from rpython.rlib.strstorage import str_storage_getitem from rpython.rtyper.lltypesystem import lltype, rffi from rpython.rtyper.tool import rffi_platform from rpython.translator.tool.cbuild import ExternalCompilationInfo diff --git a/rpython/rlib/rstruct/standardfmttable.py b/rpython/rlib/rstruct/standardfmttable.py --- a/rpython/rlib/rstruct/standardfmttable.py +++ b/rpython/rlib/rstruct/standardfmttable.py @@ -12,7 +12,7 @@ from rpython.rlib.rstruct import ieee from rpython.rlib.rstruct.error import StructError, StructOverflowError from rpython.rlib.unroll import unrolling_iterable -from rpython.rlib.rawstorage import str_storage_getitem +from rpython.rlib.strstorage import str_storage_getitem from rpython.rlib import rarithmetic from rpython.rtyper.lltypesystem import rffi ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: fix the optimizer, which gets confused when we pass a virtual string to getarrayitem_gc
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80762:4fb804c3e5fe
Date: 2015-11-18 17:33 +0100
http://bitbucket.org/pypy/pypy/changeset/4fb804c3e5fe/
Log:fix the optimizer, which gets confused when we pass a virtual string
to getarrayitem_gc
diff --git a/rpython/jit/metainterp/optimizeopt/heap.py
b/rpython/jit/metainterp/optimizeopt/heap.py
--- a/rpython/jit/metainterp/optimizeopt/heap.py
+++ b/rpython/jit/metainterp/optimizeopt/heap.py
@@ -535,10 +535,14 @@
cf.do_setfield(self, op)
def optimize_GETARRAYITEM_GC_I(self, op):
+# When using str_storage_getitem we op.getarg(0) is a string, NOT an
+# array. That's why we need to check for .is_array() before
+# remembering the reads. There is no point in remembering it, as the
+# box will be forced anyway by the optimizer
arrayinfo = self.ensure_ptr_info_arg0(op)
indexb = self.getintbound(op.getarg(1))
cf = None
-if indexb.is_constant():
+if indexb.is_constant() and arrayinfo.is_array():
index = indexb.getint()
arrayinfo.getlenbound(None).make_gt_const(index)
# use the cache on (arraydescr, index), which is a constant
@@ -555,7 +559,7 @@
self.make_nonnull(op.getarg(0))
self.emit_operation(op)
# the remember the result of reading the array item
-if cf is not None:
+if cf is not None and arrayinfo.is_array():
arrayinfo.setitem(op.getdescr(), indexb.getint(),
self.get_box_replacement(op.getarg(0)),
self.get_box_replacement(op), cf,
diff --git a/rpython/jit/metainterp/optimizeopt/info.py
b/rpython/jit/metainterp/optimizeopt/info.py
--- a/rpython/jit/metainterp/optimizeopt/info.py
+++ b/rpython/jit/metainterp/optimizeopt/info.py
@@ -49,6 +49,9 @@
def is_null(self):
return False
+def is_array(self):
+return False
+
def force_at_the_end_of_preamble(self, op, optforce, rec):
if not self.is_virtual():
return optforce.get_box_replacement(op)
@@ -478,6 +481,9 @@
self.lenbound = intutils.ConstIntBound(size)
self._clear = clear
+def is_array(self):
+return True
+
def getlenbound(self, mode):
from rpython.jit.metainterp.optimizeopt import intutils
@@ -716,6 +722,9 @@
def is_virtual(self):
return False
+def is_array(self):
+return True
+
def get_known_class(self, cpu):
if not self._const.nonnull():
return None
diff --git a/rpython/jit/metainterp/optimizeopt/virtualize.py
b/rpython/jit/metainterp/optimizeopt/virtualize.py
--- a/rpython/jit/metainterp/optimizeopt/virtualize.py
+++ b/rpython/jit/metainterp/optimizeopt/virtualize.py
@@ -271,8 +271,11 @@
self.emit_operation(op)
def optimize_GETARRAYITEM_GC_I(self, op):
+# When using str_storage_getitem we op.getarg(0) is a string, NOT an
+# array, hence the check for is_array(). If it's a virtual but not an
+# array (i.e., if it's a string) it will be forced.
opinfo = self.getptrinfo(op.getarg(0))
-if opinfo and opinfo.is_virtual():
+if opinfo and opinfo.is_virtual() and opinfo.is_array():
indexbox = self.get_constant_box(op.getarg(1))
if indexbox is not None:
item = opinfo.getitem(op.getdescr(), indexbox.getint())
diff --git a/rpython/jit/metainterp/test/test_strstorage.py
b/rpython/jit/metainterp/test/test_strstorage.py
--- a/rpython/jit/metainterp/test/test_strstorage.py
+++ b/rpython/jit/metainterp/test/test_strstorage.py
@@ -1,5 +1,6 @@
import py
-from rpython.rtyper.lltypesystem import lltype
+import struct
+from rpython.rtyper.lltypesystem import lltype, rffi
from rpython.rlib.strstorage import str_storage_getitem
from rpython.rlib.test.test_strstorage import BaseStrStorageTest
from rpython.jit.codewriter import longlong
@@ -26,3 +27,20 @@
# back!
return longlong.int2singlefloat(res)
return res
+
+
+def test_force_virtual_str_storage(self):
+size = rffi.sizeof(lltype.Signed)
+def f(val):
+x = chr(val) + '\x00'*(size-1)
+return str_storage_getitem(lltype.Signed, x, 0)
+res = self.interp_operations(f, [42], supports_singlefloats=True)
+assert res == 42
+self.check_operations_history({
+'newstr': 1, # str forcing
+'strsetitem': 1, # str forcing
+'call_pure_r': 1, # str forcing (copystrcontent)
+'guard_no_exception': 1, # str forcing
+'getarrayitem_gc_i': 1, # str_storage_getitem
+'finish': 1
+})
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: reverse the logic to distinguish virtual arrays and strings, and most importantly remove the weird ConstPtrInfo.is_array()==True
Author: Antonio Cuni Branch: faster-rstruct Changeset: r80784:d8b23469f91f Date: 2015-11-20 00:36 +0100 http://bitbucket.org/pypy/pypy/changeset/d8b23469f91f/ Log:reverse the logic to distinguish virtual arrays and strings, and most importantly remove the weird ConstPtrInfo.is_array()==True diff --git a/rpython/jit/metainterp/optimizeopt/heap.py b/rpython/jit/metainterp/optimizeopt/heap.py --- a/rpython/jit/metainterp/optimizeopt/heap.py +++ b/rpython/jit/metainterp/optimizeopt/heap.py @@ -536,13 +536,12 @@ def optimize_GETARRAYITEM_GC_I(self, op): # When using str_storage_getitem we op.getarg(0) is a string, NOT an -# array. That's why we need to check for .is_array() before -# remembering the reads. There is no point in remembering it, as the -# box will be forced anyway by the optimizer +# array. In that case, we cannot cache the getarrayitem as if it were +# an array, obviously arrayinfo = self.ensure_ptr_info_arg0(op) indexb = self.getintbound(op.getarg(1)) cf = None -if indexb.is_constant() and arrayinfo.is_array(): +if indexb.is_constant() and not arrayinfo.is_vstring(): index = indexb.getint() arrayinfo.getlenbound(None).make_gt_const(index) # use the cache on (arraydescr, index), which is a constant @@ -559,7 +558,7 @@ self.make_nonnull(op.getarg(0)) self.emit_operation(op) # the remember the result of reading the array item -if cf is not None and arrayinfo.is_array(): +if cf is not None and not arrayinfo.is_vstring(): arrayinfo.setitem(op.getdescr(), indexb.getint(), self.get_box_replacement(op.getarg(0)), self.get_box_replacement(op), cf, diff --git a/rpython/jit/metainterp/optimizeopt/info.py b/rpython/jit/metainterp/optimizeopt/info.py --- a/rpython/jit/metainterp/optimizeopt/info.py +++ b/rpython/jit/metainterp/optimizeopt/info.py @@ -24,6 +24,9 @@ def is_virtual(self): return False +def is_vstring(self): +return False + def is_precise(self): return False @@ -49,9 +52,6 @@ def is_null(self): return False -def is_array(self): -return False - def force_at_the_end_of_preamble(self, op, optforce, rec): if not self.is_virtual(): return optforce.get_box_replacement(op) @@ -481,9 +481,6 @@ self.lenbound = intutils.ConstIntBound(size) self._clear = clear -def is_array(self): -return True - def getlenbound(self, mode): from rpython.jit.metainterp.optimizeopt import intutils @@ -722,9 +719,6 @@ def is_virtual(self): return False -def is_array(self): -return True - def get_known_class(self, cpu): if not self._const.nonnull(): return None diff --git a/rpython/jit/metainterp/optimizeopt/virtualize.py b/rpython/jit/metainterp/optimizeopt/virtualize.py --- a/rpython/jit/metainterp/optimizeopt/virtualize.py +++ b/rpython/jit/metainterp/optimizeopt/virtualize.py @@ -272,10 +272,9 @@ def optimize_GETARRAYITEM_GC_I(self, op): # When using str_storage_getitem we op.getarg(0) is a string, NOT an -# array, hence the check for is_array(). If it's a virtual but not an -# array (i.e., if it's a string) it will be forced. +# array, hence the check. In that case, it will be forced opinfo = self.getptrinfo(op.getarg(0)) -if opinfo and opinfo.is_virtual() and opinfo.is_array(): +if opinfo and opinfo.is_virtual() and not opinfo.is_vstring(): indexbox = self.get_constant_box(op.getarg(1)) if indexbox is not None: item = opinfo.getitem(op.getdescr(), indexbox.getint()) diff --git a/rpython/jit/metainterp/optimizeopt/vstring.py b/rpython/jit/metainterp/optimizeopt/vstring.py --- a/rpython/jit/metainterp/optimizeopt/vstring.py +++ b/rpython/jit/metainterp/optimizeopt/vstring.py @@ -62,6 +62,9 @@ self.mode = mode self.length = length +def is_vstring(self): +return True + def getlenbound(self, mode): from rpython.jit.metainterp.optimizeopt import intutils ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: add a fast-path to unpack_double, which reads directly from the input string instead of taking a slice. There are still some rough edges, but I commit this anyway so
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80785:7cf0a0cef3b5
Date: 2015-11-20 01:01 +0100
http://bitbucket.org/pypy/pypy/changeset/7cf0a0cef3b5/
Log:add a fast-path to unpack_double, which reads directly from the
input string instead of taking a slice. There are still some rough
edges, but I commit this anyway so I can translate while I sleep :)
diff --git a/pypy/module/struct/formatiterator.py
b/pypy/module/struct/formatiterator.py
--- a/pypy/module/struct/formatiterator.py
+++ b/pypy/module/struct/formatiterator.py
@@ -149,3 +149,17 @@
@specialize.argtype(1)
def appendobj(self, value):
self.result_w.append(self.space.wrap(value))
+
+def is_aligned(self, size):
+return self.pos % size == 0
+
+def get_buffer(self):
+# XXX: if self.buf is something different that StringBuffer, this has
+# the effect to copy the whole string at each unpack!
+return self.buf.as_str()
+
+def get_pos(self):
+return self.pos
+
+def advance(self, size):
+self.read(size) # XXX, could avoid taking the slice
diff --git a/rpython/rlib/rstruct/nativefmttable.py
b/rpython/rlib/rstruct/nativefmttable.py
--- a/rpython/rlib/rstruct/nativefmttable.py
+++ b/rpython/rlib/rstruct/nativefmttable.py
@@ -44,8 +44,16 @@
@specialize.argtype(0)
def unpack_double(fmtiter):
-input = fmtiter.read(sizeof_double)
-doubleval = str_storage_getitem(rffi.DOUBLE, input, 0)
+if fmtiter.is_aligned(sizeof_double):
+# fast path
+input = fmtiter.get_buffer()
+pos = fmtiter.get_pos()
+doubleval = str_storage_getitem(rffi.DOUBLE, input, pos)
+fmtiter.advance(sizeof_double)
+else:
+# slow path, take the slice
+input = fmtiter.read(sizeof_double)
+doubleval = str_storage_getitem(rffi.DOUBLE, input, 0)
fmtiter.appendobj(doubleval)
def pack_float(fmtiter):
diff --git a/rpython/rlib/rstruct/runpack.py b/rpython/rlib/rstruct/runpack.py
--- a/rpython/rlib/rstruct/runpack.py
+++ b/rpython/rlib/rstruct/runpack.py
@@ -38,6 +38,18 @@
def appendobj(self, value):
self.value = value
+
+def is_aligned(self, size):
+return self.mr.inputpos % size == 0
+
+def get_buffer(self):
+return self.mr.input
+
+def get_pos(self):
+return self.mr.inputpos
+
+def advance(self, size):
+self.read(size) # XXX, could avoid taking the slice
ReaderForPos.__name__ = 'ReaderForPos%d' % pos
return ReaderForPos
diff --git a/rpython/rlib/rstruct/test/test_runpack.py
b/rpython/rlib/rstruct/test/test_runpack.py
--- a/rpython/rlib/rstruct/test/test_runpack.py
+++ b/rpython/rlib/rstruct/test/test_runpack.py
@@ -43,11 +43,9 @@
"""
Check the 'd' and 'f' format characters on native packing.
"""
-d_data = struct.pack("d", 12.34)
-f_data = struct.pack("f", 12.34)
+d_data = struct.pack("df", 12.34, 12.34)
def fn():
-d = runpack("@d", d_data)
-f = runpack("@f", f_data)
+d, f = runpack("@df", d_data)
return d, f
#
res = self.interpret(fn, [])
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: make sure that struct.unpack_from uses the fast path as well
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80808:f2275be59406
Date: 2015-11-20 17:37 +0100
http://bitbucket.org/pypy/pypy/changeset/f2275be59406/
Log:make sure that struct.unpack_from uses the fast path as well
diff --git a/pypy/module/struct/formatiterator.py
b/pypy/module/struct/formatiterator.py
--- a/pypy/module/struct/formatiterator.py
+++ b/pypy/module/struct/formatiterator.py
@@ -154,9 +154,8 @@
return self.pos
def get_buffer_as_string_maybe(self):
-# XXX: if self.buf is something different that StringBuffer, this has
-# the effect to copy the whole string at each unpack!
-return self.buf.as_str()
+string, pos = self.buf.as_str_and_offset_maybe()
+return string, pos+self.pos
def skip(self, size):
self.read(size) # XXX, could avoid taking the slice
diff --git a/pypy/module/struct/test/test_struct.py
b/pypy/module/struct/test/test_struct.py
--- a/pypy/module/struct/test/test_struct.py
+++ b/pypy/module/struct/test/test_struct.py
@@ -462,3 +462,29 @@
assert self.struct.unpack_from("ii", b, 2) == (17, 42)
b[:sz] = self.struct.pack("ii", 18, 43)
assert self.struct.unpack_from("ii", b) == (18, 43)
+
+
+class AppTestFastPath(object):
+spaceconfig = dict(usemodules=['struct', '__pypy__'])
+
+def setup_class(cls):
+from rpython.rlib.rstruct import standardfmttable
+standardfmttable.ALLOW_SLOWPATH = False
+#
+cls.w_struct = cls.space.appexec([], """():
+import struct
+return struct
+""")
+cls.w_bytebuffer = cls.space.appexec([], """():
+import __pypy__
+return __pypy__.bytebuffer
+""")
+
+def teardown_class(cls):
+from rpython.rlib.rstruct import standardfmttable
+standardfmttable.ALLOW_SLOWPATH = True
+
+def test_unpack_from(self):
+buf = self.struct.pack("iii", 0, 42, 43)
+offset = self.struct.calcsize("i")
+assert self.struct.unpack_from("ii", buf, offset) == (42, 43)
diff --git a/rpython/rlib/buffer.py b/rpython/rlib/buffer.py
--- a/rpython/rlib/buffer.py
+++ b/rpython/rlib/buffer.py
@@ -22,6 +22,14 @@
# May be overridden.
return self.getslice(0, self.getlength(), 1, self.getlength())
+def as_str_and_offset_maybe(self):
+"""
+If the buffer is backed by a string, return a pair (string, offset),
where
+offset is the offset inside the string where the buffer start.
+Else, return (None, 0).
+"""
+return None, 0
+
def getitem(self, index):
"Returns the index'th character in the buffer."
raise NotImplementedError # Must be overriden. No bounds checks.
@@ -66,6 +74,9 @@
def as_str(self):
return self.value
+def as_str_and_offset_maybe(self):
+return self.value, 0
+
def getitem(self, index):
return self.value[index]
@@ -99,6 +110,12 @@
else:
return 0
+def as_str_and_offset_maybe(self):
+string, offset = self.buffer.as_str_and_offset_maybe()
+if string is not None:
+return string, offset+self.offset
+return None, 0
+
def getitem(self, index):
return self.buffer.getitem(self.offset + index)
diff --git a/rpython/rlib/rstruct/standardfmttable.py
b/rpython/rlib/rstruct/standardfmttable.py
--- a/rpython/rlib/rstruct/standardfmttable.py
+++ b/rpython/rlib/rstruct/standardfmttable.py
@@ -130,7 +130,8 @@
#
-ALLOW_FASTPATH = True # set to False by TestNoFastPath
+USE_FASTPATH = True# set to False by some tests
+ALLOW_SLOWPATH = True # set to False by some tests
class CannotUnpack(Exception):
pass
@@ -139,7 +140,7 @@
def unpack_fastpath(TYPE, fmtiter):
size = rffi.sizeof(TYPE)
strbuf, pos = fmtiter.get_buffer_as_string_maybe()
-if pos % size != 0 or strbuf is None or not ALLOW_FASTPATH:
+if strbuf is None or pos % size != 0 or not USE_FASTPATH:
raise CannotUnpack
fmtiter.skip(size)
return str_storage_getitem(TYPE, strbuf, pos)
@@ -226,6 +227,9 @@
if unpack_int_fastpath_maybe(fmtiter):
return
# slow path
+if not ALLOW_SLOWPATH:
+# we enter here only on some tests
+raise ValueError("fastpath not taken :(")
intvalue = inttype(0)
s = fmtiter.read(size)
idx = 0
diff --git a/rpython/rlib/rstruct/test/test_runpack.py
b/rpython/rlib/rstruct/test/test_runpack.py
--- a/rpython/rlib/rstruct/test/test_runpack.py
+++ b/rpython/rlib/rstruct/test/test_runpack.py
@@ -94,7 +94,7 @@
class TestNoFastPath(TestRStruct):
def setup_method(self, meth):
-standardfmttable.ALLOW_FASTPATH = False
+standardfmttable.USE_FASTPATH = False
def teardown_method(self, meth):
-standardfmttable.ALLOW_FASTPATH = True
+
[pypy-commit] pypy faster-rstruct: use the str_storage_getitem fastpath to unpack ints and floats by reading them directly from within the buffer, is it's properly aligned. Add tests which disallow th
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80805:7b525c75f558
Date: 2015-11-20 16:21 +0100
http://bitbucket.org/pypy/pypy/changeset/7b525c75f558/
Log:use the str_storage_getitem fastpath to unpack ints and floats by
reading them directly from within the buffer, is it's properly
aligned. Add tests which disallow the fast path so that we can
easily test also the slow paths
diff --git a/rpython/rlib/rstruct/nativefmttable.py
b/rpython/rlib/rstruct/nativefmttable.py
--- a/rpython/rlib/rstruct/nativefmttable.py
+++ b/rpython/rlib/rstruct/nativefmttable.py
@@ -8,7 +8,8 @@
from rpython.rlib.objectmodel import specialize
from rpython.rlib.rarithmetic import r_singlefloat, widen
from rpython.rlib.rstruct import standardfmttable as std
-from rpython.rlib.rstruct.standardfmttable import native_is_bigendian
+from rpython.rlib.rstruct.standardfmttable import (native_is_bigendian,
unpack_fastpath,
+ CannotUnpack)
from rpython.rlib.rstruct.error import StructError
from rpython.rlib.unroll import unrolling_iterable
from rpython.rlib.strstorage import str_storage_getitem
@@ -44,16 +45,13 @@
@specialize.argtype(0)
def unpack_double(fmtiter):
-if fmtiter.is_aligned(sizeof_double):
-# fast path
-input = fmtiter.get_buffer()
-pos = fmtiter.get_pos()
-doubleval = str_storage_getitem(rffi.DOUBLE, input, pos)
-fmtiter.advance(sizeof_double)
-else:
+try:
+doubleval = unpack_fastpath(rffi.DOUBLE, fmtiter)
+except CannotUnpack:
# slow path, take the slice
input = fmtiter.read(sizeof_double)
doubleval = str_storage_getitem(rffi.DOUBLE, input, 0)
+#
fmtiter.appendobj(doubleval)
def pack_float(fmtiter):
@@ -72,8 +70,11 @@
@specialize.argtype(0)
def unpack_float(fmtiter):
-input = fmtiter.read(sizeof_float)
-floatval = str_storage_getitem(rffi.FLOAT, input, 0)
+try:
+floatval = unpack_fastpath(rffi.FLOAT, fmtiter)
+except CannotUnpack:
+input = fmtiter.read(sizeof_float)
+floatval = str_storage_getitem(rffi.FLOAT, input, 0)
doubleval = float(floatval) # convert from r_singlefloat to rpython's float
fmtiter.appendobj(doubleval)
diff --git a/rpython/rlib/rstruct/runpack.py b/rpython/rlib/rstruct/runpack.py
--- a/rpython/rlib/rstruct/runpack.py
+++ b/rpython/rlib/rstruct/runpack.py
@@ -39,16 +39,13 @@
def appendobj(self, value):
self.value = value
-def is_aligned(self, size):
-return self.mr.inputpos % size == 0
-
-def get_buffer(self):
-return self.mr.input
-
def get_pos(self):
return self.mr.inputpos
-def advance(self, size):
+def get_buffer_as_string_maybe(self):
+return self.mr.input
+
+def skip(self, size):
self.read(size) # XXX, could avoid taking the slice
ReaderForPos.__name__ = 'ReaderForPos%d' % pos
return ReaderForPos
diff --git a/rpython/rlib/rstruct/standardfmttable.py
b/rpython/rlib/rstruct/standardfmttable.py
--- a/rpython/rlib/rstruct/standardfmttable.py
+++ b/rpython/rlib/rstruct/standardfmttable.py
@@ -130,6 +130,21 @@
#
+ALLOW_FASTPATH = True # set to False by TestNoFastPath
+
+class CannotUnpack(Exception):
+pass
+
+@specialize.arg(0)
+def unpack_fastpath(TYPE, fmtiter):
+size = rffi.sizeof(TYPE)
+pos = fmtiter.get_pos()
+strbuf = fmtiter.get_buffer_as_string_maybe()
+if pos % size != 0 or strbuf is None or not ALLOW_FASTPATH:
+raise CannotUnpack
+fmtiter.skip(size)
+return str_storage_getitem(TYPE, strbuf, pos)
+
@specialize.argtype(0)
def unpack_pad(fmtiter, count):
fmtiter.read(count)
@@ -173,8 +188,6 @@
return TYPE
raise KeyError("Cannot find an int type size=%d, signed=%d" % (size,
signed))
-UNPACK_ALLOW_RAW_STORAGE = True
-
def make_int_unpacker(size, signed, _memo={}):
try:
return _memo[size, signed]
@@ -196,17 +209,28 @@
TYPE = get_rffi_int_type(size, signed)
@specialize.argtype(0)
+def unpack_int_fastpath_maybe(fmtiter):
+if fmtiter.bigendian != native_is_bigendian:
+return False
+try:
+intvalue = unpack_fastpath(TYPE, fmtiter)
+except CannotUnpack:
+return False
+if not signed and size < native_int_size:
+intvalue = rarithmetic.intmask(intvalue)
+intvalue = inttype(intvalue)
+fmtiter.appendobj(intvalue)
+return True
+
+@specialize.argtype(0)
def unpack_int(fmtiter):
+if unpack_int_fastpath_maybe(fmtiter):
+return
+# slow path
intvalue = inttype(0)
s = fmtiter.read(size)
idx = 0
-if UNPACK_ALLOW_RAW_STORAGE and fmtiter.bigendian ==
native_is_bigendian:
-# fast pa
[pypy-commit] pypy faster-rstruct: adapt the applevel struct to the new interface for the fast path
Author: Antonio Cuni Branch: faster-rstruct Changeset: r80806:913f469cb07b Date: 2015-11-20 16:27 +0100 http://bitbucket.org/pypy/pypy/changeset/913f469cb07b/ Log:adapt the applevel struct to the new interface for the fast path diff --git a/pypy/module/struct/formatiterator.py b/pypy/module/struct/formatiterator.py --- a/pypy/module/struct/formatiterator.py +++ b/pypy/module/struct/formatiterator.py @@ -150,16 +150,13 @@ def appendobj(self, value): self.result_w.append(self.space.wrap(value)) -def is_aligned(self, size): -return self.pos % size == 0 +def get_pos(self): +return self.pos -def get_buffer(self): +def get_buffer_as_string_maybe(self): # XXX: if self.buf is something different that StringBuffer, this has # the effect to copy the whole string at each unpack! return self.buf.as_str() -def get_pos(self): -return self.pos - -def advance(self, size): +def skip(self, size): self.read(size) # XXX, could avoid taking the slice ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: try hard to run the rstruct code when calling runpack before the translation
Author: Antonio Cuni Branch: faster-rstruct Changeset: r80804:5c33c457e1ac Date: 2015-11-20 15:41 +0100 http://bitbucket.org/pypy/pypy/changeset/5c33c457e1ac/ Log:try hard to run the rstruct code when calling runpack before the translation diff --git a/rpython/rlib/rstruct/runpack.py b/rpython/rlib/rstruct/runpack.py --- a/rpython/rlib/rstruct/runpack.py +++ b/rpython/rlib/rstruct/runpack.py @@ -100,13 +100,6 @@ exec source.compile() in miniglobals self.unpack = miniglobals['unpack'] # override not-rpython version -def unpack(self, s): -# NOT_RPYTHON -res = unpack(self.fmt, s) -if len(res) == 1: -return res[0] -return res - def _freeze_(self): assert self.formats self._create_unpacking_func() @@ -115,6 +108,7 @@ def create_unpacker(unpack_str): fmtiter = FrozenUnpackIterator(unpack_str) fmtiter.interpret(unpack_str) +assert fmtiter._freeze_() return fmtiter create_unpacker._annspecialcase_ = 'specialize:memo' ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: change the inteface: now get_buffer_as_string_maybe also return the position inside the string: this will allow to support sub-buffers
Author: Antonio Cuni Branch: faster-rstruct Changeset: r80807:c4eb83d8e88f Date: 2015-11-20 16:52 +0100 http://bitbucket.org/pypy/pypy/changeset/c4eb83d8e88f/ Log:change the inteface: now get_buffer_as_string_maybe also return the position inside the string: this will allow to support sub-buffers diff --git a/rpython/rlib/rstruct/runpack.py b/rpython/rlib/rstruct/runpack.py --- a/rpython/rlib/rstruct/runpack.py +++ b/rpython/rlib/rstruct/runpack.py @@ -39,11 +39,8 @@ def appendobj(self, value): self.value = value -def get_pos(self): -return self.mr.inputpos - def get_buffer_as_string_maybe(self): -return self.mr.input +return self.mr.input, self.mr.inputpos def skip(self, size): self.read(size) # XXX, could avoid taking the slice diff --git a/rpython/rlib/rstruct/standardfmttable.py b/rpython/rlib/rstruct/standardfmttable.py --- a/rpython/rlib/rstruct/standardfmttable.py +++ b/rpython/rlib/rstruct/standardfmttable.py @@ -138,8 +138,7 @@ @specialize.arg(0) def unpack_fastpath(TYPE, fmtiter): size = rffi.sizeof(TYPE) -pos = fmtiter.get_pos() -strbuf = fmtiter.get_buffer_as_string_maybe() +strbuf, pos = fmtiter.get_buffer_as_string_maybe() if pos % size != 0 or strbuf is None or not ALLOW_FASTPATH: raise CannotUnpack fmtiter.skip(size) ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: translation fix; I didn't find a better way to express specialize.arg0_and_argtype1
Author: Antonio Cuni Branch: faster-rstruct Changeset: r80811:ec447791f752 Date: 2015-11-20 18:34 +0100 http://bitbucket.org/pypy/pypy/changeset/ec447791f752/ Log:translation fix; I didn't find a better way to express specialize.arg0_and_argtype1 diff --git a/rpython/rlib/rstruct/nativefmttable.py b/rpython/rlib/rstruct/nativefmttable.py --- a/rpython/rlib/rstruct/nativefmttable.py +++ b/rpython/rlib/rstruct/nativefmttable.py @@ -46,7 +46,7 @@ @specialize.argtype(0) def unpack_double(fmtiter): try: -doubleval = unpack_fastpath(rffi.DOUBLE, fmtiter) +doubleval = unpack_fastpath(rffi.DOUBLE)(fmtiter) except CannotUnpack: # slow path, take the slice input = fmtiter.read(sizeof_double) @@ -71,7 +71,7 @@ @specialize.argtype(0) def unpack_float(fmtiter): try: -floatval = unpack_fastpath(rffi.FLOAT, fmtiter) +floatval = unpack_fastpath(rffi.FLOAT)(fmtiter) except CannotUnpack: input = fmtiter.read(sizeof_float) floatval = str_storage_getitem(rffi.FLOAT, input, 0) diff --git a/rpython/rlib/rstruct/standardfmttable.py b/rpython/rlib/rstruct/standardfmttable.py --- a/rpython/rlib/rstruct/standardfmttable.py +++ b/rpython/rlib/rstruct/standardfmttable.py @@ -136,14 +136,17 @@ class CannotUnpack(Exception): pass -@specialize.arg(0) -def unpack_fastpath(TYPE, fmtiter): -size = rffi.sizeof(TYPE) -strbuf, pos = fmtiter.get_buffer_as_string_maybe() -if strbuf is None or pos % size != 0 or not USE_FASTPATH: -raise CannotUnpack -fmtiter.skip(size) -return str_storage_getitem(TYPE, strbuf, pos) +@specialize.memo() +def unpack_fastpath(TYPE): +@specialize.argtype(0) +def do_unpack_fastpath(fmtiter): +size = rffi.sizeof(TYPE) +strbuf, pos = fmtiter.get_buffer_as_string_maybe() +if strbuf is None or pos % size != 0 or not USE_FASTPATH: +raise CannotUnpack +fmtiter.skip(size) +return str_storage_getitem(TYPE, strbuf, pos) +return do_unpack_fastpath @specialize.argtype(0) def unpack_pad(fmtiter, count): @@ -213,7 +216,7 @@ if fmtiter.bigendian != native_is_bigendian: return False try: -intvalue = unpack_fastpath(TYPE, fmtiter) +intvalue = unpack_fastpath(TYPE)(fmtiter) except CannotUnpack: return False if not signed and size < native_int_size: ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: there is no distinction between native and standard float/double formats: the Python struct module requires that the format is ieee754 in both cases. However, in cas
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80814:8f216a415954
Date: 2015-11-21 11:34 +0100
http://bitbucket.org/pypy/pypy/changeset/8f216a415954/
Log:there is no distinction between native and standard float/double
formats: the Python struct module requires that the format is
ieee754 in both cases. However, in case our platform DOES support
ieee754, we can directly convert the value from the memory. The very
slow path defined in ieee.py is used only for obscure platforms
which do not use native ieee754 floats, or in case we require a
differend endianess
diff --git a/rpython/rlib/rstruct/nativefmttable.py
b/rpython/rlib/rstruct/nativefmttable.py
--- a/rpython/rlib/rstruct/nativefmttable.py
+++ b/rpython/rlib/rstruct/nativefmttable.py
@@ -8,8 +8,7 @@
from rpython.rlib.objectmodel import specialize
from rpython.rlib.rarithmetic import r_singlefloat, widen
from rpython.rlib.rstruct import standardfmttable as std
-from rpython.rlib.rstruct.standardfmttable import (native_is_bigendian,
unpack_fastpath,
- CannotUnpack)
+from rpython.rlib.rstruct.standardfmttable import native_is_bigendian
from rpython.rlib.rstruct.error import StructError
from rpython.rlib.unroll import unrolling_iterable
from rpython.rlib.strstorage import str_storage_getitem
@@ -43,16 +42,6 @@
fmtiter.result.append(chr(value & 0xff))
value >>= 8
-@specialize.argtype(0)
-def unpack_double(fmtiter):
-try:
-doubleval = unpack_fastpath(rffi.DOUBLE)(fmtiter)
-except CannotUnpack:
-# slow path, take the slice
-input = fmtiter.read(sizeof_double)
-doubleval = str_storage_getitem(rffi.DOUBLE, input, 0)
-#
-fmtiter.appendobj(doubleval)
def pack_float(fmtiter):
doubleval = fmtiter.accept_float_arg()
@@ -68,16 +57,6 @@
fmtiter.result.append(chr(value & 0xff))
value >>= 8
-@specialize.argtype(0)
-def unpack_float(fmtiter):
-try:
-floatval = unpack_fastpath(rffi.FLOAT)(fmtiter)
-except CannotUnpack:
-input = fmtiter.read(sizeof_float)
-floatval = str_storage_getitem(rffi.FLOAT, input, 0)
-doubleval = float(floatval) # convert from r_singlefloat to rpython's float
-fmtiter.appendobj(doubleval)
-
#
#
# Use rffi_platform to get the native sizes and alignments from the C compiler
@@ -134,10 +113,10 @@
if fmtchar == 'f':
pack = pack_float
-unpack = unpack_float
+unpack = std.unpack_float
elif fmtchar == 'd':
pack = pack_double
-unpack = unpack_double
+unpack = std.unpack_double
elif fmtchar == '?':
pack = std.pack_bool
unpack = std.unpack_bool
diff --git a/rpython/rlib/rstruct/standardfmttable.py
b/rpython/rlib/rstruct/standardfmttable.py
--- a/rpython/rlib/rstruct/standardfmttable.py
+++ b/rpython/rlib/rstruct/standardfmttable.py
@@ -17,6 +17,7 @@
from rpython.rtyper.lltypesystem import rffi
native_is_bigendian = struct.pack("=i", 1) == struct.pack(">i", 1)
+native_is_ieee754 = float.__getformat__('double').startswith('IEEE')
def pack_pad(fmtiter, count):
fmtiter.result.append_multiple_char('\x00', count)
@@ -175,12 +176,27 @@
end = count
fmtiter.appendobj(data[1:end])
-def make_float_unpacker(size):
+def make_ieee_unpacker(TYPE):
@specialize.argtype(0)
-def unpacker(fmtiter):
-data = fmtiter.read(size)
-fmtiter.appendobj(ieee.unpack_float(data, fmtiter.bigendian))
-return unpacker
+def unpack_ieee(fmtiter):
+size = rffi.sizeof(TYPE)
+if fmtiter.bigendian != native_is_bigendian or not native_is_ieee754:
+# fallback to the very slow unpacking code in ieee.py
+data = fmtiter.read(size)
+fmtiter.appendobj(ieee.unpack_float(data, fmtiter.bigendian))
+return
+try:
+# fast path
+val = unpack_fastpath(TYPE)(fmtiter)
+except CannotUnpack:
+# slow path, take the slice
+input = fmtiter.read(size)
+val = str_storage_getitem(TYPE, input, 0)
+fmtiter.appendobj(float(val))
+return unpack_ieee
+
+unpack_double = make_ieee_unpacker(rffi.DOUBLE)
+unpack_float = make_ieee_unpacker(rffi.FLOAT)
#
@@ -267,9 +283,9 @@
'p':{ 'size' : 1, 'pack' : pack_pascal, 'unpack' : unpack_pascal,
'needcount' : True },
'f':{ 'size' : 4, 'pack' : make_float_packer(4),
-'unpack' : make_float_unpacker(4)},
+'unpack' : unpack_float},
'd':{ 'size' : 8, 'pack' : make_float_packer(8),
-'unpack' : make_float_unpacker(8)},
+'unpack' : unpack_double},
'?':{ 'siz
[pypy-commit] pypy faster-rstruct: blindly try to make this test passing on big-endian machines
Author: Antonio Cuni Branch: faster-rstruct Changeset: r80816:8846f2974ae5 Date: 2015-11-21 15:51 +0100 http://bitbucket.org/pypy/pypy/changeset/8846f2974ae5/ Log:blindly try to make this test passing on big-endian machines diff --git a/rpython/jit/metainterp/test/test_strstorage.py b/rpython/jit/metainterp/test/test_strstorage.py --- a/rpython/jit/metainterp/test/test_strstorage.py +++ b/rpython/jit/metainterp/test/test_strstorage.py @@ -1,4 +1,5 @@ import py +import sys import struct from rpython.rtyper.lltypesystem import lltype, rffi from rpython.rlib.strstorage import str_storage_getitem @@ -30,9 +31,13 @@ def test_force_virtual_str_storage(self): +byteorder = sys.byteorder size = rffi.sizeof(lltype.Signed) def f(val): -x = chr(val) + '\x00'*(size-1) +if byteorder == 'little': +x = chr(val) + '\x00'*(size-1) +else: +x = '\x00'*(size-1) + chr(val) return str_storage_getitem(lltype.Signed, x, 0) res = self.interp_operations(f, [42], supports_singlefloats=True) assert res == 42 ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: hg merge default-at-the-latest-green-revision
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80815:a256226d8bce
Date: 2015-11-21 15:48 +0100
http://bitbucket.org/pypy/pypy/changeset/a256226d8bce/
Log:hg merge default-at-the-latest-green-revision
diff --git a/.hgtags b/.hgtags
--- a/.hgtags
+++ b/.hgtags
@@ -17,3 +17,4 @@
295ee98b69288471b0fcf2e0ede82ce5209eb90b release-2.6.0
f3ad1e1e1d6215e20d34bb65ab85ff9188c9f559 release-2.6.1
850edf14b2c75573720f59e95767335fb1affe55 release-4.0.0
+5f8302b8bf9f53056e40426f10c72151564e5b19 release-4.0.1
diff --git a/LICENSE b/LICENSE
--- a/LICENSE
+++ b/LICENSE
@@ -56,14 +56,15 @@
Anders Chrigstrom
Eric van Riet Paap
Wim Lavrijsen
+ Richard Plangger
Richard Emslie
Alexander Schremmer
Dan Villiom Podlaski Christiansen
Lukas Diekmann
Sven Hager
Anders Lehmann
+ Remi Meier
Aurelien Campeas
- Remi Meier
Niklaus Haldimann
Camillo Bruni
Laura Creighton
@@ -87,7 +88,6 @@
Ludovic Aubry
Jacob Hallen
Jason Creighton
- Richard Plangger
Alex Martelli
Michal Bendowski
stian
@@ -200,9 +200,12 @@
Alex Perry
Vincent Legoll
Alan McIntyre
+ Spenser Bauman
Alexander Sedov
Attila Gobi
Christopher Pope
+ Devin Jeanpierre
+ Vaibhav Sood
Christian Tismer
Marc Abramowitz
Dan Stromberg
@@ -234,6 +237,7 @@
Lutz Paelike
Lucio Torre
Lars Wassermann
+ Philipp Rustemeuer
Henrik Vendelbo
Dan Buch
Miguel de Val Borro
@@ -244,6 +248,7 @@
Martin Blais
Lene Wagner
Tomo Cocoa
+ Kim Jin Su
Toni Mattis
Lucas Stadler
Julian Berman
@@ -253,6 +258,7 @@
Anna Katrina Dominguez
William Leslie
Bobby Impollonia
+ Faye Zhao
t...@eistee.fritz.box
Andrew Thompson
Yusei Tahara
@@ -283,6 +289,7 @@
shoma hosaka
Daniel Neuhäuser
Ben Mather
+ Niclas Olofsson
halgari
Boglarka Vezer
Chris Pressey
@@ -309,13 +316,16 @@
Stefan Marr
jiaaro
Mads Kiilerich
+ Richard Lancaster
opassembler.py
Antony Lee
+ Yaroslav Fedevych
Jim Hunziker
Markus Unterwaditzer
Even Wiik Thomassen
jbs
squeaky
+ Zearin
soareschen
Kurt Griffiths
Mike Bayer
@@ -327,6 +337,7 @@
Anna Ravencroft
Andrey Churin
Dan Crosta
+ Tobias Diaz
Julien Phalip
Roman Podoliaka
Dan Loewenherz
diff --git a/lib_pypy/cffi.egg-info/PKG-INFO b/lib_pypy/cffi.egg-info/PKG-INFO
--- a/lib_pypy/cffi.egg-info/PKG-INFO
+++ b/lib_pypy/cffi.egg-info/PKG-INFO
@@ -1,6 +1,6 @@
Metadata-Version: 1.1
Name: cffi
-Version: 1.3.0
+Version: 1.3.1
Summary: Foreign Function Interface for Python calling C code.
Home-page: http://cffi.readthedocs.org
Author: Armin Rigo, Maciej Fijalkowski
diff --git a/lib_pypy/cffi/__init__.py b/lib_pypy/cffi/__init__.py
--- a/lib_pypy/cffi/__init__.py
+++ b/lib_pypy/cffi/__init__.py
@@ -4,8 +4,8 @@
from .api import FFI, CDefError, FFIError
from .ffiplatform import VerificationError, VerificationMissing
-__version__ = "1.3.0"
-__version_info__ = (1, 3, 0)
+__version__ = "1.3.1"
+__version_info__ = (1, 3, 1)
# The verifier module file names are based on the CRC32 of a string that
# contains the following version number. It may be older than __version__
diff --git a/lib_pypy/cffi/cparser.py b/lib_pypy/cffi/cparser.py
--- a/lib_pypy/cffi/cparser.py
+++ b/lib_pypy/cffi/cparser.py
@@ -62,7 +62,8 @@
if csource.startswith('*', endpos):
parts.append('('); closing += ')'
level = 0
-for i in xrange(endpos, len(csource)):
+i = endpos
+while i < len(csource):
c = csource[i]
if c == '(':
level += 1
@@ -73,6 +74,7 @@
elif c in ',;=':
if level == 0:
break
+i += 1
csource = csource[endpos:i] + closing + csource[i:]
#print repr(''.join(parts)+csource)
parts.append(csource)
diff --git a/lib_pypy/cffi/model.py b/lib_pypy/cffi/model.py
--- a/lib_pypy/cffi/model.py
+++ b/lib_pypy/cffi/model.py
@@ -514,12 +514,17 @@
if self.baseinttype is not None:
return self.baseinttype.get_cached_btype(ffi, finishlist)
#
+from . import api
if self.enumvalues:
smallest_value = min(self.enumvalues)
largest_value = max(self.enumvalues)
else:
-smallest_value = 0
-largest_value = 0
+import warnings
+warnings.warn("%r has no values explicitly defined; next version "
+ "will refuse to guess which integer type it is "
+ "meant to be (unsigned/signed, int/long)"
+ % self._get_c_name())
+smallest_value = largest_value = 0
if smallest_value < 0: # needs a signed type
sign = 1
candidate1 = PrimitiveType("int")
diff --git a/pypy/doc/contributor.rst b/pypy/doc/contributor.rst
--- a/pypy/doc/contributor.rst
+++ b/pypy/doc/contributor.rst
@@ -26,15 +26,15 @@
Anders Chrigstro
[pypy-commit] pypy faster-rstruct: sanity check to ensure that the cast we do is actually valid. Currently it fails e.g. on win32 and 32bit ARM
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80869:0a7af3c17db0
Date: 2015-11-23 18:35 +0100
http://bitbucket.org/pypy/pypy/changeset/0a7af3c17db0/
Log:sanity check to ensure that the cast we do is actually valid.
Currently it fails e.g. on win32 and 32bit ARM
diff --git a/rpython/rlib/strstorage.py b/rpython/rlib/strstorage.py
--- a/rpython/rlib/strstorage.py
+++ b/rpython/rlib/strstorage.py
@@ -32,6 +32,18 @@
from rpython.rtyper.annlowlevel import llstr
from rpython.rlib.objectmodel import specialize
+def compute_offsetof(TP, field):
+"""
+NOT_RPYTHON
+"""
+obj = lltype.malloc(TP, 0)
+baseadr = llmemory.cast_ptr_to_adr(obj)
+offset = llmemory.offsetof(TP, field)
+interioradr = baseadr + offset
+return (llmemory.cast_adr_to_int(interioradr, 'forced') -
+llmemory.cast_adr_to_int(baseadr, 'forced'))
+
+
@specialize.memo()
def rpy_string_as_type(TP):
# sanity check that STR is actually what we think it is
@@ -42,6 +54,8 @@
STR_AS_TP = lltype.GcStruct('rpy_string_as_%s' % TP,
('hash', lltype.Signed),
('chars', lltype.Array(TP, hints={'immutable':
True})))
+# sanity check
+assert compute_offsetof(STR, 'chars') == compute_offsetof(STR_AS_TP,
'chars')
return STR_AS_TP
@specialize.ll()
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: fix rstruct.unpack q and Q on systems which do not support str_storage_getitem, such as win32 and probably ARM 32 bit
Author: Antonio Cuni Branch: faster-rstruct Changeset: r80870:f856371c182b Date: 2015-11-23 20:56 +0100 http://bitbucket.org/pypy/pypy/changeset/f856371c182b/ Log:fix rstruct.unpack q and Q on systems which do not support str_storage_getitem, such as win32 and probably ARM 32 bit diff --git a/rpython/rlib/rstruct/standardfmttable.py b/rpython/rlib/rstruct/standardfmttable.py --- a/rpython/rlib/rstruct/standardfmttable.py +++ b/rpython/rlib/rstruct/standardfmttable.py @@ -12,7 +12,7 @@ from rpython.rlib.rstruct import ieee from rpython.rlib.rstruct.error import StructError, StructOverflowError from rpython.rlib.unroll import unrolling_iterable -from rpython.rlib.strstorage import str_storage_getitem +from rpython.rlib.strstorage import str_storage_getitem, str_storage_supported from rpython.rlib import rarithmetic from rpython.rtyper.lltypesystem import rffi @@ -229,7 +229,7 @@ @specialize.argtype(0) def unpack_int_fastpath_maybe(fmtiter): -if fmtiter.bigendian != native_is_bigendian: +if fmtiter.bigendian != native_is_bigendian or not str_storage_supported(TYPE): return False try: intvalue = unpack_fastpath(TYPE)(fmtiter) diff --git a/rpython/rlib/strstorage.py b/rpython/rlib/strstorage.py --- a/rpython/rlib/strstorage.py +++ b/rpython/rlib/strstorage.py @@ -58,6 +58,15 @@ assert compute_offsetof(STR, 'chars') == compute_offsetof(STR_AS_TP, 'chars') return STR_AS_TP +@specialize.memo() +def str_storage_supported(TP): +try: +rpy_string_as_type(TP) +except AssertionError: +return False +else: +return True + @specialize.ll() def str_storage_getitem(TP, s, index): STR_AS_TP = rpy_string_as_type(TP) ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: fix runpack('d') on win32 and 32bit ARM. I tried to keep the fix and the implementation as least invasive as possible, because it will go away as soon as we have gc_
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80871:59a4b706d487
Date: 2015-11-23 21:37 +0100
http://bitbucket.org/pypy/pypy/changeset/59a4b706d487/
Log:fix runpack('d') on win32 and 32bit ARM. I tried to keep the fix and
the implementation as least invasive as possible, because it will go
away as soon as we have gc_load; but in the meantime, this fixes it
with reasonable performance
diff --git a/rpython/rlib/rstruct/standardfmttable.py
b/rpython/rlib/rstruct/standardfmttable.py
--- a/rpython/rlib/rstruct/standardfmttable.py
+++ b/rpython/rlib/rstruct/standardfmttable.py
@@ -185,6 +185,13 @@
data = fmtiter.read(size)
fmtiter.appendobj(ieee.unpack_float(data, fmtiter.bigendian))
return
+if not str_storage_supported(TYPE):
+# this happens e.g. on win32 and ARM32: we cannot read the string
+# content as an array of doubles because it's not properly
+# aligned. But we can read a longlong and convert to float
+assert TYPE == rffi.DOUBLE
+assert rffi.sizeof(TYPE) == 8
+return unpack_longlong2float(fmtiter)
try:
# fast path
val = unpack_fastpath(TYPE)(fmtiter)
@@ -195,6 +202,16 @@
fmtiter.appendobj(float(val))
return unpack_ieee
+@specialize.argtype(0)
+def unpack_longlong2float(fmtiter):
+from rpython.rlib.rstruct.runpack import runpack
+from rpython.rlib.longlong2float import longlong2float
+s = fmtiter.read(8)
+llval = runpack('q', s) # this is a bit recursive, I know
+doubleval = longlong2float(llval)
+fmtiter.appendobj(doubleval)
+
+
unpack_double = make_ieee_unpacker(rffi.DOUBLE)
unpack_float = make_ieee_unpacker(rffi.FLOAT)
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: skip this test on some platforms
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80872:783361f5f31e
Date: 2015-11-23 21:40 +0100
http://bitbucket.org/pypy/pypy/changeset/783361f5f31e/
Log:skip this test on some platforms
diff --git a/rpython/rlib/test/test_strstorage.py
b/rpython/rlib/test/test_strstorage.py
--- a/rpython/rlib/test/test_strstorage.py
+++ b/rpython/rlib/test/test_strstorage.py
@@ -1,7 +1,7 @@
import py
import struct
from rpython.rtyper.lltypesystem import lltype, rffi
-from rpython.rlib.strstorage import str_storage_getitem
+from rpython.rlib.strstorage import str_storage_getitem, str_storage_supported
from rpython.rlib.rarithmetic import r_singlefloat
from rpython.rtyper.test.tool import BaseRtypingTest
@@ -22,6 +22,8 @@
assert int(x) == 43
def test_float(self):
+if not str_storage_supported(lltype.Float):
+py.test.skip('str_storage_getitem(lltype.Float) not supported on
this machine')
buf = struct.pack('@dd', 12.3, 45.6)
size = struct.calcsize('@d')
assert self.str_storage_getitem(lltype.Float, buf, 0) == 12.3
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: merge heads
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80875:c9ab4be2f15b
Date: 2015-11-24 00:59 +0100
http://bitbucket.org/pypy/pypy/changeset/c9ab4be2f15b/
Log:merge heads
diff --git a/rpython/rlib/rstruct/standardfmttable.py
b/rpython/rlib/rstruct/standardfmttable.py
--- a/rpython/rlib/rstruct/standardfmttable.py
+++ b/rpython/rlib/rstruct/standardfmttable.py
@@ -12,7 +12,7 @@
from rpython.rlib.rstruct import ieee
from rpython.rlib.rstruct.error import StructError, StructOverflowError
from rpython.rlib.unroll import unrolling_iterable
-from rpython.rlib.strstorage import str_storage_getitem
+from rpython.rlib.strstorage import str_storage_getitem, str_storage_supported
from rpython.rlib import rarithmetic
from rpython.rtyper.lltypesystem import rffi
@@ -185,6 +185,13 @@
data = fmtiter.read(size)
fmtiter.appendobj(ieee.unpack_float(data, fmtiter.bigendian))
return
+if not str_storage_supported(TYPE):
+# this happens e.g. on win32 and ARM32: we cannot read the string
+# content as an array of doubles because it's not properly
+# aligned. But we can read a longlong and convert to float
+assert TYPE == rffi.DOUBLE
+assert rffi.sizeof(TYPE) == 8
+return unpack_longlong2float(fmtiter)
try:
# fast path
val = unpack_fastpath(TYPE)(fmtiter)
@@ -195,6 +202,16 @@
fmtiter.appendobj(float(val))
return unpack_ieee
+@specialize.argtype(0)
+def unpack_longlong2float(fmtiter):
+from rpython.rlib.rstruct.runpack import runpack
+from rpython.rlib.longlong2float import longlong2float
+s = fmtiter.read(8)
+llval = runpack('q', s) # this is a bit recursive, I know
+doubleval = longlong2float(llval)
+fmtiter.appendobj(doubleval)
+
+
unpack_double = make_ieee_unpacker(rffi.DOUBLE)
unpack_float = make_ieee_unpacker(rffi.FLOAT)
@@ -229,7 +246,7 @@
@specialize.argtype(0)
def unpack_int_fastpath_maybe(fmtiter):
-if fmtiter.bigendian != native_is_bigendian:
+if fmtiter.bigendian != native_is_bigendian or not
str_storage_supported(TYPE):
return False
try:
intvalue = unpack_fastpath(TYPE)(fmtiter)
diff --git a/rpython/rlib/strstorage.py b/rpython/rlib/strstorage.py
--- a/rpython/rlib/strstorage.py
+++ b/rpython/rlib/strstorage.py
@@ -32,6 +32,18 @@
from rpython.rtyper.annlowlevel import llstr
from rpython.rlib.objectmodel import specialize
+def compute_offsetof(TP, field):
+"""
+NOT_RPYTHON
+"""
+obj = lltype.malloc(TP, 0)
+baseadr = llmemory.cast_ptr_to_adr(obj)
+offset = llmemory.offsetof(TP, field)
+interioradr = baseadr + offset
+return (llmemory.cast_adr_to_int(interioradr, 'forced') -
+llmemory.cast_adr_to_int(baseadr, 'forced'))
+
+
@specialize.memo()
def rpy_string_as_type(TP):
# sanity check that STR is actually what we think it is
@@ -42,8 +54,19 @@
STR_AS_TP = lltype.GcStruct('rpy_string_as_%s' % TP,
('hash', lltype.Signed),
('chars', lltype.Array(TP, hints={'immutable':
True})))
+# sanity check
+assert compute_offsetof(STR, 'chars') == compute_offsetof(STR_AS_TP,
'chars')
return STR_AS_TP
+@specialize.memo()
+def str_storage_supported(TP):
+try:
+rpy_string_as_type(TP)
+except AssertionError:
+return False
+else:
+return True
+
@specialize.ll()
def str_storage_getitem(TP, s, index):
STR_AS_TP = rpy_string_as_type(TP)
diff --git a/rpython/rlib/test/test_strstorage.py
b/rpython/rlib/test/test_strstorage.py
--- a/rpython/rlib/test/test_strstorage.py
+++ b/rpython/rlib/test/test_strstorage.py
@@ -1,7 +1,7 @@
import py
import struct
from rpython.rtyper.lltypesystem import lltype, rffi
-from rpython.rlib.strstorage import str_storage_getitem
+from rpython.rlib.strstorage import str_storage_getitem, str_storage_supported
from rpython.rlib.rarithmetic import r_singlefloat
from rpython.rtyper.test.tool import BaseRtypingTest
@@ -22,6 +22,8 @@
assert int(x) == 43
def test_float(self):
+if not str_storage_supported(lltype.Float):
+py.test.skip('str_storage_getitem(lltype.Float) not supported on
this machine')
buf = struct.pack('@dd', 12.3, 45.6)
size = struct.calcsize('@d')
assert self.str_storage_getitem(lltype.Float, buf, 0) == 12.3
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: fix test_pypy_c/test_struct, to reflect the new optimization
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80874:c32ced64c29f
Date: 2015-11-24 00:58 +0100
http://bitbucket.org/pypy/pypy/changeset/c32ced64c29f/
Log:fix test_pypy_c/test_struct, to reflect the new optimization
diff --git a/pypy/module/pypyjit/test_pypy_c/test_struct.py
b/pypy/module/pypyjit/test_pypy_c/test_struct.py
--- a/pypy/module/pypyjit/test_pypy_c/test_struct.py
+++ b/pypy/module/pypyjit/test_pypy_c/test_struct.py
@@ -19,7 +19,8 @@
import struct
i = 1
while i < n:
-x = struct.unpack("i", struct.pack("i", i))[0] # ID: struct
+buf = struct.pack("i", i) # ID: pack
+x = struct.unpack("i", buf)[0] # ID: unpack
i += x / i
return i
@@ -29,7 +30,7 @@
loop, = log.loops_by_filename(self.filepath)
# This could, of course stand some improvement, to remove all these
# arithmatic ops, but we've removed all the core overhead.
-assert loop.match_by_id("struct", """
+assert loop.match_by_id("pack", """
guard_not_invalidated(descr=...)
# struct.pack
%s
@@ -40,17 +41,22 @@
i17 = int_and(i16, 255)
i19 = int_rshift(i16, 8)
i20 = int_and(i19, 255)
+""" % extra)
+# the newstr and the strsetitems are because the string is forced,
+# which is in turn because the optimizer doesn't know how to handle a
+# getarrayitem_gc_i on a virtual string. It could be improved, but it
+# is also true that in real life cases struct.unpack is called on
+# strings which come from the outside, so it's a minor issue.
+assert loop.match_by_id("unpack", """
# struct.unpack
-i22 = int_lshift(i14, 8)
-i23 = int_or(i11, i22)
-i25 = int_lshift(i17, 16)
-i26 = int_or(i23, i25)
-i28 = int_ge(i20, 128)
-guard_false(i28, descr=...)
-i30 = int_lshift(i20, 24)
-i31 = int_or(i26, i30)
-""" % extra)
+p88 = newstr(4)
+strsetitem(p88, 0, i11)
+strsetitem(p88, 1, i14)
+strsetitem(p88, 2, i17)
+strsetitem(p88, 3, i20)
+i91 = getarrayitem_gc_i(p88, 0, descr=)
+""")
def test_struct_object(self):
def main(n):
@@ -58,7 +64,8 @@
s = struct.Struct("i")
i = 1
while i < n:
-x = s.unpack(s.pack(i))[0] # ID: struct
+buf = s.pack(i) # ID: pack
+x = s.unpack(buf)[0] # ID: unpack
i += x / i
return i
@@ -66,7 +73,7 @@
assert log.result == main(1000)
loop, = log.loops_by_filename(self.filepath)
-assert loop.match_by_id('struct', """
+assert loop.match_by_id('pack', """
guard_not_invalidated(descr=...)
# struct.pack
%s
@@ -77,14 +84,14 @@
i17 = int_and(i16, 255)
i19 = int_rshift(i16, 8)
i20 = int_and(i19, 255)
+""" % extra)
+assert loop.match_by_id('unpack', """
# struct.unpack
-i22 = int_lshift(i14, 8)
-i23 = int_or(i11, i22)
-i25 = int_lshift(i17, 16)
-i26 = int_or(i23, i25)
-i28 = int_ge(i20, 128)
-guard_false(i28, descr=...)
-i30 = int_lshift(i20, 24)
-i31 = int_or(i26, i30)
-""" % extra)
+p88 = newstr(4)
+strsetitem(p88, 0, i11)
+strsetitem(p88, 1, i14)
+strsetitem(p88, 2, i17)
+strsetitem(p88, 3, i20)
+i91 = getarrayitem_gc_i(p88, 0, descr=)
+""")
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: in case of cross-compilation we need to delay the comparison of offsets until compile time (where FieldOffset is a symbolic correctly handled by the C compiler). How
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80898:89a3ecaf0b2d
Date: 2015-11-24 18:05 +0100
http://bitbucket.org/pypy/pypy/changeset/89a3ecaf0b2d/
Log:in case of cross-compilation we need to delay the comparison of
offsets until compile time (where FieldOffset is a symbolic
correctly handled by the C compiler). However, we need to introduce
a bit of extra machinery to make it working also in direct tests and
on top of llinterp
diff --git a/rpython/rlib/strstorage.py b/rpython/rlib/strstorage.py
--- a/rpython/rlib/strstorage.py
+++ b/rpython/rlib/strstorage.py
@@ -30,22 +30,10 @@
from rpython.rtyper.lltypesystem import lltype, rffi, llmemory
from rpython.rtyper.lltypesystem.rstr import STR, _get_raw_str_buf
from rpython.rtyper.annlowlevel import llstr
-from rpython.rlib.objectmodel import specialize
-
-def compute_offsetof(TP, field):
-"""
-NOT_RPYTHON
-"""
-obj = lltype.malloc(TP, 0)
-baseadr = llmemory.cast_ptr_to_adr(obj)
-offset = llmemory.offsetof(TP, field)
-interioradr = baseadr + offset
-return (llmemory.cast_adr_to_int(interioradr, 'forced') -
-llmemory.cast_adr_to_int(baseadr, 'forced'))
-
+from rpython.rlib.objectmodel import specialize, we_are_translated
@specialize.memo()
-def rpy_string_as_type(TP):
+def _rpy_string_as_type(TP):
# sanity check that STR is actually what we think it is
assert STR._flds == {
'hash': lltype.Signed,
@@ -54,23 +42,32 @@
STR_AS_TP = lltype.GcStruct('rpy_string_as_%s' % TP,
('hash', lltype.Signed),
('chars', lltype.Array(TP, hints={'immutable':
True})))
-# sanity check
-assert compute_offsetof(STR, 'chars') == compute_offsetof(STR_AS_TP,
'chars')
return STR_AS_TP
-@specialize.memo()
+@specialize.arg(0)
def str_storage_supported(TP):
-try:
-rpy_string_as_type(TP)
-except AssertionError:
-return False
-else:
-return True
+STR_AS_TP = _rpy_string_as_type(TP)
+return _offsetof_chars(STR) == _offsetof_chars(STR_AS_TP)
@specialize.ll()
def str_storage_getitem(TP, s, index):
-STR_AS_TP = rpy_string_as_type(TP)
+assert str_storage_supported(TP) # sanity check
+STR_AS_TP = _rpy_string_as_type(TP)
lls = llstr(s)
str_as_tp = rffi.cast(lltype.Ptr(STR_AS_TP), lls)
index = index / rffi.sizeof(TP)
return str_as_tp.chars[index]
+
+@specialize.arg(0)
+def _offsetof_chars(TP):
+# we need to be careful in presence of cross-compilation, because we want
+# to defer the actual check at compile-time on the target machine, NOT at
+# translation time on the host machine. However, we still need to support
+# direct tests, so if we are not translated, we immediately force the
+# result into an actual int.
+offset = llmemory.offsetof(TP, 'chars')
+if we_are_translated():
+return offset # symbolic
+else:
+return offset.force_as_int()
+return offset
diff --git a/rpython/rlib/test/test_strstorage.py
b/rpython/rlib/test/test_strstorage.py
--- a/rpython/rlib/test/test_strstorage.py
+++ b/rpython/rlib/test/test_strstorage.py
@@ -1,12 +1,29 @@
import py
+import sys
+import platform
import struct
from rpython.rtyper.lltypesystem import lltype, rffi
from rpython.rlib.strstorage import str_storage_getitem, str_storage_supported
from rpython.rlib.rarithmetic import r_singlefloat
from rpython.rtyper.test.tool import BaseRtypingTest
+IS_WIN32 = (sys.platform == 'win32')
+IS_ARM = (platform.machine().startswith('arm'))
+IS_32BIT = (sys.maxint == 2147483647)
+
class BaseStrStorageTest:
+def test_str_getitem_supported(self):
+# on win32 or ARM32, we expect LONGLONG and DOUBLE to be NOT supported
+# by str_storage_getitem
+if IS_WIN32 or (IS_ARM and IS_32BIT):
+expected = False
+else:
+expected = True
+#
+assert self.str_storage_supported(rffi.LONGLONG) == expected
+assert self.str_storage_supported(rffi.DOUBLE) == expected
+
def test_signed(self):
buf = struct.pack('@ll', 42, 43)
size = struct.calcsize('@l')
@@ -40,11 +57,18 @@
class TestDirect(BaseStrStorageTest):
+def str_storage_supported(self, TYPE):
+return str_storage_supported(TYPE)
+
def str_storage_getitem(self, TYPE, buf, offset):
return str_storage_getitem(TYPE, buf, offset)
+class TestRTyping(BaseStrStorageTest, BaseRtypingTest):
-class TestRTyping(BaseStrStorageTest, BaseRtypingTest):
+def str_storage_supported(self, TYPE):
+def fn():
+return str_storage_supported(TYPE)
+return self.interpret(fn, [])
def str_storage_getitem(self, TYPE, buf, offset):
def fn(offset):
diff --git a/rpython/rtyper/lltypesystem/llmemory.py
b/rpython/rtyper/lltypesystem/llmemory.py
--- a/rpython/rtyper/lltypesystem/llmemory.py
+++ b/rpy
[pypy-commit] pypy faster-rstruct: fix
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80899:82d45be90c37
Date: 2015-11-24 18:09 +0100
http://bitbucket.org/pypy/pypy/changeset/82d45be90c37/
Log:fix
diff --git a/rpython/jit/metainterp/test/test_strstorage.py
b/rpython/jit/metainterp/test/test_strstorage.py
--- a/rpython/jit/metainterp/test/test_strstorage.py
+++ b/rpython/jit/metainterp/test/test_strstorage.py
@@ -29,6 +29,8 @@
return longlong.int2singlefloat(res)
return res
+def str_storage_supported(self, TYPE):
+py.test.skip('this is not a JIT test')
def test_force_virtual_str_storage(self):
byteorder = sys.byteorder
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: hg merge default
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80900:ce2d724dad21
Date: 2015-11-24 18:12 +0100
http://bitbucket.org/pypy/pypy/changeset/ce2d724dad21/
Log:hg merge default
diff too long, truncating to 2000 out of 2119 lines
diff --git a/lib_pypy/greenlet.py b/lib_pypy/greenlet.py
--- a/lib_pypy/greenlet.py
+++ b/lib_pypy/greenlet.py
@@ -88,9 +88,19 @@
#
try:
unbound_method = getattr(_continulet, methodname)
+_tls.leaving = current
args, kwds = unbound_method(current, *baseargs, to=target)
-finally:
_tls.current = current
+except:
+_tls.current = current
+if hasattr(_tls, 'trace'):
+_run_trace_callback('throw')
+_tls.leaving = None
+raise
+else:
+if hasattr(_tls, 'trace'):
+_run_trace_callback('switch')
+_tls.leaving = None
#
if kwds:
if args:
@@ -122,6 +132,34 @@
return f.f_back.f_back.f_back # go past start(), __switch(), switch()
#
+# Recent additions
+
+GREENLET_USE_GC = True
+GREENLET_USE_TRACING = True
+
+def gettrace():
+return getattr(_tls, 'trace', None)
+
+def settrace(callback):
+try:
+prev = _tls.trace
+del _tls.trace
+except AttributeError:
+prev = None
+if callback is not None:
+_tls.trace = callback
+return prev
+
+def _run_trace_callback(event):
+try:
+_tls.trace(event, (_tls.leaving, _tls.current))
+except:
+# In case of exceptions trace function is removed
+if hasattr(_tls, 'trace'):
+del _tls.trace
+raise
+
+#
# Internal stuff
try:
@@ -143,22 +181,32 @@
_tls.current = gmain
def _greenlet_start(greenlet, args):
-args, kwds = args
-_tls.current = greenlet
try:
-res = greenlet.run(*args, **kwds)
-except GreenletExit, e:
-res = e
+args, kwds = args
+_tls.current = greenlet
+try:
+if hasattr(_tls, 'trace'):
+_run_trace_callback('switch')
+res = greenlet.run(*args, **kwds)
+except GreenletExit, e:
+res = e
+finally:
+_continuation.permute(greenlet, greenlet.parent)
+return ((res,), None)
finally:
-_continuation.permute(greenlet, greenlet.parent)
-return ((res,), None)
+_tls.leaving = greenlet
def _greenlet_throw(greenlet, exc, value, tb):
-_tls.current = greenlet
try:
-raise exc, value, tb
-except GreenletExit, e:
-res = e
+_tls.current = greenlet
+try:
+if hasattr(_tls, 'trace'):
+_run_trace_callback('throw')
+raise exc, value, tb
+except GreenletExit, e:
+res = e
+finally:
+_continuation.permute(greenlet, greenlet.parent)
+return ((res,), None)
finally:
-_continuation.permute(greenlet, greenlet.parent)
-return ((res,), None)
+_tls.leaving = greenlet
diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -1,7 +1,18 @@
=
-What's new in PyPy 4.0.+
+What's new in PyPy 4.1.+
=
.. this is a revision shortly after release-4.0.1
.. startrev: 4b5c840d0da2
+.. branch: numpy-1.10
+
+Fix tests to run cleanly with -A and start to fix micronumpy for upstream numpy
+which is now 1.10.2
+
+.. branch: osx-flat-namespace
+
+Fix the cpyext tests on OSX by linking with -flat_namespace
+
+.. branch: anntype
+Refactor and improve exception analysis in the annotator.
diff --git a/pypy/goal/targetpypystandalone.py
b/pypy/goal/targetpypystandalone.py
--- a/pypy/goal/targetpypystandalone.py
+++ b/pypy/goal/targetpypystandalone.py
@@ -302,7 +302,7 @@
def hack_for_cffi_modules(self, driver):
# HACKHACKHACK
-# ugly hack to modify target goal from compile_c to build_cffi_imports
+# ugly hack to modify target goal from compile_* to build_cffi_imports
# this should probably get cleaned up and merged with driver.create_exe
from rpython.translator.driver import taskdef
import types
@@ -316,7 +316,8 @@
name = name.new(ext='exe')
return name
-@taskdef(['compile_c'], "Create cffi bindings for modules")
+compile_goal, = driver.backend_select_goals(['compile'])
+@taskdef([compile_goal], "Create cffi bindings for modules")
def task_build_cffi_imports(self):
from pypy.tool.build_cffi_imports import
create_cffi_import_libraries
''' Use cffi to compile cffi interfaces to modules'''
@@ -335,7 +336,7 @@
# if failures, they
[pypy-commit] pypy faster-rstruct: dismiss the offsetof approach, as it's too complicate to make it working reliably in presence of cross-compilation. Simply disallow str_storage_getitem for types >=
Author: Antonio Cuni Branch: faster-rstruct Changeset: r80901:02f4fd70c0bb Date: 2015-11-24 18:44 +0100 http://bitbucket.org/pypy/pypy/changeset/02f4fd70c0bb/ Log:dismiss the offsetof approach, as it's too complicate to make it working reliably in presence of cross-compilation. Simply disallow str_storage_getitem for types >= lltype.Signed diff --git a/rpython/rlib/strstorage.py b/rpython/rlib/strstorage.py --- a/rpython/rlib/strstorage.py +++ b/rpython/rlib/strstorage.py @@ -46,8 +46,11 @@ @specialize.arg(0) def str_storage_supported(TP): -STR_AS_TP = _rpy_string_as_type(TP) -return _offsetof_chars(STR) == _offsetof_chars(STR_AS_TP) +# on some architectures (e.g. win32 and arm32) an array of longlongs needs +# to be aligned at 8 bytes boundaries, so we cannot safely cast from STR +# to STR_AS_TP. In that case, we str_storage_getitem is simply not +# supported +return rffi.sizeof(TP) <= rffi.sizeof(lltype.Signed) @specialize.ll() def str_storage_getitem(TP, s, index): @@ -57,17 +60,3 @@ str_as_tp = rffi.cast(lltype.Ptr(STR_AS_TP), lls) index = index / rffi.sizeof(TP) return str_as_tp.chars[index] - -@specialize.arg(0) -def _offsetof_chars(TP): -# we need to be careful in presence of cross-compilation, because we want -# to defer the actual check at compile-time on the target machine, NOT at -# translation time on the host machine. However, we still need to support -# direct tests, so if we are not translated, we immediately force the -# result into an actual int. -offset = llmemory.offsetof(TP, 'chars') -if we_are_translated(): -return offset # symbolic -else: -return offset.force_as_int() -return offset ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: this is now unsupported on linux x86_32
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80902:a70875ac8f0e
Date: 2015-11-24 18:46 +0100
http://bitbucket.org/pypy/pypy/changeset/a70875ac8f0e/
Log:this is now unsupported on linux x86_32
diff --git a/rpython/rlib/test/test_strstorage.py
b/rpython/rlib/test/test_strstorage.py
--- a/rpython/rlib/test/test_strstorage.py
+++ b/rpython/rlib/test/test_strstorage.py
@@ -1,22 +1,17 @@
import py
import sys
-import platform
import struct
from rpython.rtyper.lltypesystem import lltype, rffi
from rpython.rlib.strstorage import str_storage_getitem, str_storage_supported
from rpython.rlib.rarithmetic import r_singlefloat
from rpython.rtyper.test.tool import BaseRtypingTest
-IS_WIN32 = (sys.platform == 'win32')
-IS_ARM = (platform.machine().startswith('arm'))
IS_32BIT = (sys.maxint == 2147483647)
class BaseStrStorageTest:
def test_str_getitem_supported(self):
-# on win32 or ARM32, we expect LONGLONG and DOUBLE to be NOT supported
-# by str_storage_getitem
-if IS_WIN32 or (IS_ARM and IS_32BIT):
+if IS_32BIT:
expected = False
else:
expected = True
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: hg merge default
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80903:56589ed1776a
Date: 2015-11-24 18:57 +0100
http://bitbucket.org/pypy/pypy/changeset/56589ed1776a/
Log:hg merge default
diff --git a/lib_pypy/datetime.py b/lib_pypy/datetime.py
--- a/lib_pypy/datetime.py
+++ b/lib_pypy/datetime.py
@@ -17,6 +17,7 @@
"""
from __future__ import division
+import numbers as _numbers
import time as _time
import math as _math
import struct as _struct
@@ -271,15 +272,17 @@
def _check_int_field(value):
if isinstance(value, int):
-return value
+return int(value)
if not isinstance(value, float):
try:
value = value.__int__()
except AttributeError:
pass
else:
-if isinstance(value, (int, long)):
-return value
+if isinstance(value, int):
+return int(value)
+elif isinstance(value, long):
+return int(long(value))
raise TypeError('__int__ method should return an integer')
raise TypeError('an integer is required')
raise TypeError('integer argument expected, got float')
@@ -468,7 +471,7 @@
d = days
assert isinstance(daysecondsfrac, float)
assert abs(daysecondsfrac) <= 1.0
-assert isinstance(d, (int, long))
+assert isinstance(d, _numbers.Integral)
assert abs(s) <= 24 * 3600
# days isn't referenced again before redefinition
@@ -484,11 +487,11 @@
assert isinstance(secondsfrac, float)
assert abs(secondsfrac) <= 2.0
-assert isinstance(seconds, (int, long))
+assert isinstance(seconds, _numbers.Integral)
days, seconds = divmod(seconds, 24*3600)
d += days
s += int(seconds)# can't overflow
-assert isinstance(s, int)
+assert isinstance(s, _numbers.Integral)
assert abs(s) <= 2 * 24 * 3600
# seconds isn't referenced again before redefinition
@@ -510,8 +513,8 @@
d += days
s += int(seconds)
microseconds = _round(microseconds + usdouble)
-assert isinstance(s, int)
-assert isinstance(microseconds, int)
+assert isinstance(s, _numbers.Integral)
+assert isinstance(microseconds, _numbers.Integral)
assert abs(s) <= 3 * 24 * 3600
assert abs(microseconds) < 3.1e6
@@ -521,9 +524,9 @@
days, s = divmod(s, 24*3600)
d += days
-assert isinstance(d, (int, long))
-assert isinstance(s, int) and 0 <= s < 24*3600
-assert isinstance(us, int) and 0 <= us < 100
+assert isinstance(d, _numbers.Integral)
+assert isinstance(s, _numbers.Integral) and 0 <= s < 24*3600
+assert isinstance(us, _numbers.Integral) and 0 <= us < 100
if abs(d) > 9:
raise OverflowError("timedelta # of days is too large: %d" % d)
@@ -1510,7 +1513,7 @@
converter = _time.localtime if tz is None else _time.gmtime
-if isinstance(timestamp, int):
+if isinstance(timestamp, _numbers.Integral):
us = 0
else:
t_full = timestamp
@@ -1535,7 +1538,7 @@
@classmethod
def utcfromtimestamp(cls, t):
"Construct a UTC datetime from a POSIX timestamp (like time.time())."
-if isinstance(t, int):
+if isinstance(t, _numbers.Integral):
us = 0
else:
t_full = t
diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -15,4 +15,10 @@
Fix the cpyext tests on OSX by linking with -flat_namespace
.. branch: anntype
+
Refactor and improve exception analysis in the annotator.
+
+.. branch: posita/2193-datetime-timedelta-integrals
+
+Fix issue #2193. ``isinstance(..., int)`` => ``isinstance(...,
numbers.Integral)``
+to allow for alternate ``int``-like implementations (e.g.,
``future.types.newint``)
diff --git a/pypy/module/test_lib_pypy/test_datetime.py
b/pypy/module/test_lib_pypy/test_datetime.py
--- a/pypy/module/test_lib_pypy/test_datetime.py
+++ b/pypy/module/test_lib_pypy/test_datetime.py
@@ -170,14 +170,23 @@
self.value = value
def __int__(self):
return self.value
+class SubInt(int): pass
+class SubLong(long): pass
+dt10 = datetime.datetime(10, 10, 10, 10, 10, 10, 10)
for xx in [10L,
decimal.Decimal(10),
decimal.Decimal('10.9'),
Number(10),
- Number(10L)]:
-assert datetime.datetime(10, 10, 10, 10, 10, 10, 10) == \
- datetime.datetime(xx, xx, xx, xx, xx, xx, xx)
+ Number(10L),
+ SubInt(10),
+ SubLong(10),
+ Number(SubInt(10)),
+ Number(SubLong(10))]:
+dtxx = d
[pypy-commit] pypy faster-rstruct: temporarily disable micronumpy as it causes translation to fail
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80904:ffe06132bdbc
Date: 2015-11-24 19:00 +0100
http://bitbucket.org/pypy/pypy/changeset/ffe06132bdbc/
Log:temporarily disable micronumpy as it causes translation to fail
diff --git a/pypy/config/pypyoption.py b/pypy/config/pypyoption.py
--- a/pypy/config/pypyoption.py
+++ b/pypy/config/pypyoption.py
@@ -39,6 +39,8 @@
"_csv", "cppyy", "_pypyjson"
])
+working_modules.remove('micronumpy') # XXX temporary
+
if ((sys.platform.startswith('linux') or sys.platform == 'darwin')
and os.uname()[4] == 'x86_64' and sys.maxint > 2**32):
# it's not enough that we get x86_64
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: try to disable this as well
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80906:74414c3c9204
Date: 2015-11-24 19:21 +0100
http://bitbucket.org/pypy/pypy/changeset/74414c3c9204/
Log:try to disable this as well
diff --git a/pypy/config/pypyoption.py b/pypy/config/pypyoption.py
--- a/pypy/config/pypyoption.py
+++ b/pypy/config/pypyoption.py
@@ -40,6 +40,7 @@
])
working_modules.remove('micronumpy') # XXX temporary
+working_modules.remove('cpyext')
if ((sys.platform.startswith('linux') or sys.platform == 'darwin')
and os.uname()[4] == 'x86_64' and sys.maxint > 2**32):
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: try harder to disable cpyext
Author: Antonio Cuni Branch: faster-rstruct Changeset: r80909:1bf434a1141d Date: 2015-11-24 20:39 +0100 http://bitbucket.org/pypy/pypy/changeset/1bf434a1141d/ Log:try harder to disable cpyext diff --git a/pypy/config/pypyoption.py b/pypy/config/pypyoption.py --- a/pypy/config/pypyoption.py +++ b/pypy/config/pypyoption.py @@ -335,6 +335,8 @@ def enable_allworkingmodules(config): +config.objspace.usemodules.cpyext = False # XXX + modules = working_modules.copy() if config.translation.sandbox: modules = default_modules @@ -344,6 +346,7 @@ config.objspace.usemodules.suggest(**dict.fromkeys(modules, True)) + def enable_translationmodules(config): modules = translation_modules modules = [name for name in modules if name not in essential_modules] ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: re-enable cpyext and micronumpy. The failure due to micronumpy seems to be random
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80918:7f3a328c275d
Date: 2015-11-25 00:55 +0100
http://bitbucket.org/pypy/pypy/changeset/7f3a328c275d/
Log:re-enable cpyext and micronumpy. The failure due to micronumpy seems
to be random
diff --git a/pypy/config/pypyoption.py b/pypy/config/pypyoption.py
--- a/pypy/config/pypyoption.py
+++ b/pypy/config/pypyoption.py
@@ -39,9 +39,6 @@
"_csv", "cppyy", "_pypyjson"
])
-working_modules.remove('micronumpy') # XXX temporary
-working_modules.remove('cpyext')
-
if ((sys.platform.startswith('linux') or sys.platform == 'darwin')
and os.uname()[4] == 'x86_64' and sys.maxint > 2**32):
# it's not enough that we get x86_64
@@ -335,8 +332,6 @@
def enable_allworkingmodules(config):
-config.objspace.usemodules.cpyext = False # XXX
-
modules = working_modules.copy()
if config.translation.sandbox:
modules = default_modules
@@ -346,7 +341,6 @@
config.objspace.usemodules.suggest(**dict.fromkeys(modules, True))
-
def enable_translationmodules(config):
modules = translation_modules
modules = [name for name in modules if name not in essential_modules]
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: hg merge default
Author: Antonio Cuni
Branch: faster-rstruct
Changeset: r80919:246229cd7f49
Date: 2015-11-25 00:57 +0100
http://bitbucket.org/pypy/pypy/changeset/246229cd7f49/
Log:hg merge default
diff --git a/lib_pypy/cffi/api.py b/lib_pypy/cffi/api.py
--- a/lib_pypy/cffi/api.py
+++ b/lib_pypy/cffi/api.py
@@ -72,6 +72,7 @@
self._cdefsources = []
self._included_ffis = []
self._windows_unicode = None
+self._init_once_cache = {}
if hasattr(backend, 'set_ffi'):
backend.set_ffi(self)
for name in backend.__dict__:
@@ -598,6 +599,30 @@
return recompile(self, module_name, source, tmpdir=tmpdir,
source_extension=source_extension, **kwds)
+def init_once(self, func, tag):
+# Read _init_once_cache[tag], which is either (False, lock) if
+# we're calling the function now in some thread, or (True, result).
+# Don't call setdefault() in most cases, to avoid allocating and
+# immediately freeing a lock; but still use setdefaut() to avoid
+# races.
+try:
+x = self._init_once_cache[tag]
+except KeyError:
+x = self._init_once_cache.setdefault(tag, (False, allocate_lock()))
+# Common case: we got (True, result), so we return the result.
+if x[0]:
+return x[1]
+# Else, it's a lock. Acquire it to serialize the following tests.
+with x[1]:
+# Read again from _init_once_cache the current status.
+x = self._init_once_cache[tag]
+if x[0]:
+return x[1]
+# Call the function and store the result back.
+result = func()
+self._init_once_cache[tag] = (True, result)
+return result
+
def _load_backend_lib(backend, name, flags):
if name is None:
diff --git a/lib_pypy/cffi/parse_c_type.h b/lib_pypy/cffi/parse_c_type.h
--- a/lib_pypy/cffi/parse_c_type.h
+++ b/lib_pypy/cffi/parse_c_type.h
@@ -1,5 +1,6 @@
-/* See doc/misc/parse_c_type.rst in the source of CFFI for more information */
+/* This part is from file 'cffi/parse_c_type.h'. It is copied at the
+ beginning of C sources generated by CFFI's ffi.set_source(). */
typedef void *_cffi_opcode_t;
diff --git a/lib_pypy/datetime.py b/lib_pypy/datetime.py
--- a/lib_pypy/datetime.py
+++ b/lib_pypy/datetime.py
@@ -17,11 +17,12 @@
"""
from __future__ import division
-import numbers as _numbers
import time as _time
import math as _math
import struct as _struct
+_SENTINEL = object()
+
def _cmp(x, y):
return 0 if x == y else 1 if x > y else -1
@@ -32,6 +33,8 @@
MAXYEAR =
_MINYEARFMT = 1900
+_MAX_DELTA_DAYS = 9
+
# Utility functions, adapted from Python's Demo/classes/Dates.py, which
# also assumes the current Gregorian calendar indefinitely extended in
# both directions. Difference: Dates.py calls January 1 of year 0 day
@@ -96,6 +99,15 @@
# pasting together 25 4-year cycles.
assert _DI100Y == 25 * _DI4Y - 1
+_US_PER_US = 1
+_US_PER_MS = 1000
+_US_PER_SECOND = 100
+_US_PER_MINUTE = 6000
+_SECONDS_PER_DAY = 24 * 3600
+_US_PER_HOUR = 36
+_US_PER_DAY = 864
+_US_PER_WEEK = 6048
+
def _ord2ymd(n):
"ordinal -> (year, month, day), considering 01-Jan-0001 as day 1."
@@ -417,6 +429,30 @@
self.hour, self.minute, self.second = hour, minute, second
self.microsecond = microsecond
+def _accum(tag, sofar, num, factor, leftover):
+if isinstance(num, (int, long)):
+prod = num * factor
+rsum = sofar + prod
+return rsum, leftover
+if isinstance(num, float):
+fracpart, intpart = _math.modf(num)
+prod = int(intpart) * factor
+rsum = sofar + prod
+if fracpart == 0.0:
+return rsum, leftover
+assert isinstance(factor, (int, long))
+fracpart, intpart = _math.modf(factor * fracpart)
+rsum += int(intpart)
+return rsum, leftover + fracpart
+raise TypeError("unsupported type for timedelta %s component: %s" %
+(tag, type(num)))
+
+def _normalize_pair(hi, lo, factor):
+if lo < 0 or lo >= factor:
+inc, lo = divmod(lo, factor)
+hi += inc
+return hi, lo
+
class timedelta(object):
"""Represent the difference between two datetime objects.
@@ -436,100 +472,42 @@
"""
__slots__ = '_days', '_seconds', '_microseconds', '_hashcode'
-def __new__(cls, days=0, seconds=0, microseconds=0,
-milliseconds=0, minutes=0, hours=0, weeks=0):
-# Doing this efficiently and accurately in C is going to be difficult
-# and error-prone, due to ubiquitous overflow possibilities, and that
-# C double doesn't have enough bits of precision to represent
-# microseconds over 10K years faithfully. The code here tries to make
-# explicit where go-fast assumptions can be relied on, in order to
-# gui
[pypy-commit] pypy faster-rstruct: partial backout of 89a3ecaf0b2d, we no longer need to compare field offsets before translation
Author: Antonio Cuni Branch: faster-rstruct Changeset: r80934:909baa7fdb8e Date: 2015-11-25 11:09 +0100 http://bitbucket.org/pypy/pypy/changeset/909baa7fdb8e/ Log:partial backout of 89a3ecaf0b2d, we no longer need to compare field offsets before translation diff --git a/rpython/rtyper/lltypesystem/llmemory.py b/rpython/rtyper/lltypesystem/llmemory.py --- a/rpython/rtyper/lltypesystem/llmemory.py +++ b/rpython/rtyper/lltypesystem/llmemory.py @@ -214,20 +214,6 @@ dst = cast_adr_to_ptr(dstadr, PTR) _reccopy(src, dst) -def force_as_int(self): -""" -NOT_RPYTHON -Return the actual offset as an integer. -""" -if self.TYPE._arrayfld: -obj = lltype.malloc(self.TYPE, 0) -else: -obj = lltype.malloc(self.TYPE) -baseadr = cast_ptr_to_adr(obj) -interioradr = baseadr + self -return cast_adr_to_int(interioradr, 'forced') - cast_adr_to_int(baseadr, 'forced') - - class CompositeOffset(AddressOffset): diff --git a/rpython/rtyper/lltypesystem/opimpl.py b/rpython/rtyper/lltypesystem/opimpl.py --- a/rpython/rtyper/lltypesystem/opimpl.py +++ b/rpython/rtyper/lltypesystem/opimpl.py @@ -103,13 +103,6 @@ def op_int_eq(x, y): -# so far we support only == and only for FieldOffset. In theory, we could -# generalize support for direct comparisons of AddressOffsets -if isinstance(x, llmemory.FieldOffset): -x = x.force_as_int() -if isinstance(y, llmemory.FieldOffset): -y = y.force_as_int() -# if not isinstance(x, (int, long)): from rpython.rtyper.lltypesystem import llgroup assert isinstance(x, llgroup.CombinedSymbolic), ( ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: close merged branch
Author: Antonio Cuni Branch: faster-rstruct Changeset: r80936:ac9b475d8326 Date: 2015-11-25 11:20 +0100 http://bitbucket.org/pypy/pypy/changeset/ac9b475d8326/ Log:close merged branch ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy faster-rstruct: pro-actively fix test_whatsnews
Author: Antonio Cuni Branch: faster-rstruct Changeset: r80935:4dfc49434925 Date: 2015-11-25 11:19 +0100 http://bitbucket.org/pypy/pypy/changeset/4dfc49434925/ Log:pro-actively fix test_whatsnews diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst --- a/pypy/doc/whatsnew-head.rst +++ b/pypy/doc/whatsnew-head.rst @@ -22,3 +22,10 @@ Fix issue #2193. ``isinstance(..., int)`` => ``isinstance(..., numbers.Integral)`` to allow for alternate ``int``-like implementations (e.g., ``future.types.newint``) + +.. branch: faster-rstruct + +Improve the performace of struct.unpack, which now directly reads inside the +string buffer and directly casts the bytes to the appropriate type, when +allowed. Unpacking of floats and doubles is about 15 times faster now, while +for integer types it's up to ~50% faster for 64bit integers. ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy default: Merge the faster-rstruct branch, which improves the performace of
Author: Antonio Cuni
Branch:
Changeset: r80937:cdc2cea532d1
Date: 2015-11-25 11:22 +0100
http://bitbucket.org/pypy/pypy/changeset/cdc2cea532d1/
Log:Merge the faster-rstruct branch, which improves the performace of
struct.unpack, which now directly reads inside the string buffer and
directly casts the bytes to the appropriate type, when allowed.
Unpacking of floats and doubles is about 15 times faster now, while
for integer types it's up to ~50% faster for 64bit integers.
This is done by the introduction of the more general
rlib.str_storage.str_storage_getitem, which allows to read arbitrary
types from the internal buffer of RPython strings. Right now this is
done by casting the internal array, but eventually we should switch
to use llop.gc_load, once it's implemented :)
diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -22,3 +22,10 @@
Fix issue #2193. ``isinstance(..., int)`` => ``isinstance(...,
numbers.Integral)``
to allow for alternate ``int``-like implementations (e.g.,
``future.types.newint``)
+
+.. branch: faster-rstruct
+
+Improve the performace of struct.unpack, which now directly reads inside the
+string buffer and directly casts the bytes to the appropriate type, when
+allowed. Unpacking of floats and doubles is about 15 times faster now, while
+for integer types it's up to ~50% faster for 64bit integers.
diff --git a/pypy/module/pypyjit/test_pypy_c/test_struct.py
b/pypy/module/pypyjit/test_pypy_c/test_struct.py
--- a/pypy/module/pypyjit/test_pypy_c/test_struct.py
+++ b/pypy/module/pypyjit/test_pypy_c/test_struct.py
@@ -19,7 +19,8 @@
import struct
i = 1
while i < n:
-x = struct.unpack("i", struct.pack("i", i))[0] # ID: struct
+buf = struct.pack("i", i) # ID: pack
+x = struct.unpack("i", buf)[0] # ID: unpack
i += x / i
return i
@@ -29,7 +30,7 @@
loop, = log.loops_by_filename(self.filepath)
# This could, of course stand some improvement, to remove all these
# arithmatic ops, but we've removed all the core overhead.
-assert loop.match_by_id("struct", """
+assert loop.match_by_id("pack", """
guard_not_invalidated(descr=...)
# struct.pack
%s
@@ -40,17 +41,22 @@
i17 = int_and(i16, 255)
i19 = int_rshift(i16, 8)
i20 = int_and(i19, 255)
+""" % extra)
+# the newstr and the strsetitems are because the string is forced,
+# which is in turn because the optimizer doesn't know how to handle a
+# getarrayitem_gc_i on a virtual string. It could be improved, but it
+# is also true that in real life cases struct.unpack is called on
+# strings which come from the outside, so it's a minor issue.
+assert loop.match_by_id("unpack", """
# struct.unpack
-i22 = int_lshift(i14, 8)
-i23 = int_or(i11, i22)
-i25 = int_lshift(i17, 16)
-i26 = int_or(i23, i25)
-i28 = int_ge(i20, 128)
-guard_false(i28, descr=...)
-i30 = int_lshift(i20, 24)
-i31 = int_or(i26, i30)
-""" % extra)
+p88 = newstr(4)
+strsetitem(p88, 0, i11)
+strsetitem(p88, 1, i14)
+strsetitem(p88, 2, i17)
+strsetitem(p88, 3, i20)
+i91 = getarrayitem_gc_i(p88, 0, descr=)
+""")
def test_struct_object(self):
def main(n):
@@ -58,7 +64,8 @@
s = struct.Struct("i")
i = 1
while i < n:
-x = s.unpack(s.pack(i))[0] # ID: struct
+buf = s.pack(i) # ID: pack
+x = s.unpack(buf)[0] # ID: unpack
i += x / i
return i
@@ -66,7 +73,7 @@
assert log.result == main(1000)
loop, = log.loops_by_filename(self.filepath)
-assert loop.match_by_id('struct', """
+assert loop.match_by_id('pack', """
guard_not_invalidated(descr=...)
# struct.pack
%s
@@ -77,14 +84,14 @@
i17 = int_and(i16, 255)
i19 = int_rshift(i16, 8)
i20 = int_and(i19, 255)
+""" % extra)
+assert loop.match_by_id('unpack', """
# struct.unpack
-i22 = int_lshift(i14, 8)
-i23 = int_or(i11, i22)
-i25 = int_lshift(i17, 16)
-i26 = int_or(i23, i25)
-i28 = int_ge(i20, 128)
-guard_false(i28, descr=...)
-i30 = int_lshift(i20, 24)
-i31 = int_or(i26, i30)
-""" % extra)
+p88 = newstr(4)
+strsetitem(p88, 0, i11)
+strsetitem(p88, 1, i14)
+strsetitem(p88
[pypy-commit] pypy default: improve comment, as per fijal's request
Author: Antonio Cuni Branch: Changeset: r80938:fdfae75d80b9 Date: 2015-11-25 11:47 +0100 http://bitbucket.org/pypy/pypy/changeset/fdfae75d80b9/ Log:improve comment, as per fijal's request diff --git a/rpython/jit/metainterp/optimizeopt/heap.py b/rpython/jit/metainterp/optimizeopt/heap.py --- a/rpython/jit/metainterp/optimizeopt/heap.py +++ b/rpython/jit/metainterp/optimizeopt/heap.py @@ -535,9 +535,12 @@ cf.do_setfield(self, op) def optimize_GETARRAYITEM_GC_I(self, op): -# When using str_storage_getitem we op.getarg(0) is a string, NOT an -# array. In that case, we cannot cache the getarrayitem as if it were -# an array, obviously +# When using str_storage_getitem it might happen that op.getarg(0) is +# a virtual string, NOT an array. In that case, we cannot cache the +# getarrayitem as if it were an array, obviously. In theory we could +# improve by writing special code to interpter the buffer of the +# virtual string as if it were an array, but it looks complicate, +# fragile and not worth it. arrayinfo = self.ensure_ptr_info_arg0(op) indexb = self.getintbound(op.getarg(1)) cf = None ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] this is a test
This is a test to see if the bitbucket hook can send emails to the pypy-commit mailing list ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] this is a test
This is a test to see if the bitbucket hook can send emails to the pypy-commit mailing list ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] this is another test
This is a test to see if the bitbucket hook can send emails to the pypy-commit mailing list ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] buildbot default: ignore the file with the password
Author: Antonio Cuni Branch: Changeset: r973:cb78fa21e8ed Date: 2015-12-16 15:41 +0100 http://bitbucket.org/pypy/buildbot/changeset/cb78fa21e8ed/ Log:ignore the file with the password diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -7,6 +7,7 @@ # master/slaveinfo.py contains the passwords, so it should never be tracked master/slaveinfo.py +bbhook/smtp.password # ignore pidfiles and all the logs twistd.pid ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy default: extract libpypy-c.so too now that it's needed
Author: Antonio Cuni
Branch:
Changeset: r76435:d0354c8dfeb8
Date: 2015-03-17 17:17 +0100
http://bitbucket.org/pypy/pypy/changeset/d0354c8dfeb8/
Log:extract libpypy-c.so too now that it's needed
diff --git a/pypy/goal/getnightly.py b/pypy/goal/getnightly.py
--- a/pypy/goal/getnightly.py
+++ b/pypy/goal/getnightly.py
@@ -7,7 +7,7 @@
if sys.platform.startswith('linux'):
arch = 'linux'
cmd = 'wget "%s"'
-tar = "tar -x -v --wildcards --strip-components=2 -f %s '*/bin/pypy'"
+tar = "tar -x -v --wildcards --strip-components=2 -f %s '*/bin/pypy'
'*/bin/libpypy-c.so'"
if os.uname()[-1].startswith('arm'):
arch += '-armhf-raspbian'
elif sys.platform.startswith('darwin'):
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy default: fix issue #2000, which was caused by 2e4e36c84077: partially revert it, by keeping the AST transformation but also allow the possibility to pass a tuple to it
Author: Antonio Cuni
Branch:
Changeset: r76436:9159bcc2b0d0
Date: 2015-03-17 17:20 +0100
http://bitbucket.org/pypy/pypy/changeset/9159bcc2b0d0/
Log:fix issue #2000, which was caused by 2e4e36c84077: partially revert
it, by keeping the AST transformation but also allow the possibility
to pass a tuple to it
diff --git a/pypy/interpreter/pyopcode.py b/pypy/interpreter/pyopcode.py
--- a/pypy/interpreter/pyopcode.py
+++ b/pypy/interpreter/pyopcode.py
@@ -1619,6 +1619,13 @@
def prepare_exec(f, prog, globals, locals, compile_flags, builtin,
codetype):
"""Manipulate parameters to exec statement to (codeobject, dict, dict).
"""
+if (globals is None and locals is None and
+isinstance(prog, tuple) and
+(len(prog) == 2 or len(prog) == 3)):
+globals = prog[1]
+if len(prog) == 3:
+locals = prog[2]
+prog = prog[0]
if globals is None:
globals = f.f_globals
if locals is None:
diff --git a/pypy/interpreter/test/test_exec.py
b/pypy/interpreter/test/test_exec.py
--- a/pypy/interpreter/test/test_exec.py
+++ b/pypy/interpreter/test/test_exec.py
@@ -262,3 +262,11 @@
"""]
for c in code:
compile(c, "", "exec")
+
+def test_exec_tuple(self):
+# note: this is VERY different than testing exec("a = 42", d), because
+# this specific case is handled specially by the AST compiler
+d = {}
+x = ("a = 42", d)
+exec x
+assert d['a'] == 42
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy vmprof: disable vmprof around fork(), and re-enable it only in the parent process
Author: Antonio Cuni
Branch: vmprof
Changeset: r76540:78914547f680
Date: 2015-03-24 19:01 +0100
http://bitbucket.org/pypy/pypy/changeset/78914547f680/
Log:disable vmprof around fork(), and re-enable it only in the parent
process
diff --git a/pypy/module/_vmprof/src/vmprof.c b/pypy/module/_vmprof/src/vmprof.c
--- a/pypy/module/_vmprof/src/vmprof.c
+++ b/pypy/module/_vmprof/src/vmprof.c
@@ -26,6 +26,7 @@
#include
#include
#include
+#include
#define UNW_LOCAL_ONLY
#include
@@ -44,6 +45,8 @@
void* vmprof_mainloop_func;
static ptrdiff_t mainloop_sp_offset;
static vmprof_get_virtual_ip_t mainloop_get_virtual_ip;
+static long last_period_usec = 0;
+static int atfork_hook_installed = 0;
/* *
@@ -275,6 +278,7 @@
static int install_sigprof_timer(long period_usec) {
static struct itimerval timer;
+last_period_usec = period_usec;
timer.it_interval.tv_sec = 0;
timer.it_interval.tv_usec = period_usec;
timer.it_value = timer.it_interval;
@@ -295,6 +299,34 @@
return 0;
}
+static void atfork_disable_timer(void) {
+remove_sigprof_timer();
+}
+
+static void atfork_enable_timer(void) {
+install_sigprof_timer(last_period_usec);
+}
+
+static int install_pthread_atfork_hooks(void) {
+/* this is needed to prevent the problems described there:
+ - http://code.google.com/p/gperftools/issues/detail?id=278
+ - http://lists.debian.org/debian-glibc/2010/03/msg00161.html
+
+TL;DR: if the RSS of the process is large enough, the clone() syscall
+will be interrupted by the SIGPROF before it can complete, then
+retried, interrupted again and so on, in an endless loop. The
+solution is to disable the timer around the fork, and re-enable it
+only inside the parent.
+*/
+if (atfork_hook_installed)
+return 0;
+int ret = pthread_atfork(atfork_disable_timer, atfork_enable_timer, NULL);
+if (ret != 0)
+return -1;
+atfork_hook_installed = 1;
+return 0;
+}
+
/* *
* public API
* *
@@ -321,6 +353,9 @@
if (install_sigprof_timer(period_usec) == -1) {
return -1;
}
+if (install_pthread_atfork_hooks() == -1) {
+return -1;
+}
return 0;
}
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] extradoc extradoc: abstract for my EP talk
Author: Antonio Cuni Branch: extradoc Changeset: r5526:92dbb3dfe86f Date: 2015-04-13 18:08 +0200 http://bitbucket.org/pypy/extradoc/changeset/92dbb3dfe86f/ Log:abstract for my EP talk diff --git a/talk/ep2015/performance-abstract.txt b/talk/ep2015/performance-abstract.txt new file mode 100644 --- /dev/null +++ b/talk/ep2015/performance-abstract.txt @@ -0,0 +1,20 @@ +== +PyPy performance (not) for dummies +== + + +Abstract +- + +PyPy is the fastest Python interpreter around, and its JIT can optimize most +of your Python programs without problems. However, there are techniques to +improve the performances even further and squeeze the most out of PyPy. In +this talk we will see: + +- the general principles behind the PyPy JIT + +- how to profile programs to find the bottlenecks + +- how to examine the code generated by the JIT + +- how to write JIT-friendly programs ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] extradoc extradoc: tweaks
Author: Antonio Cuni Branch: extradoc Changeset: r5528:5bb398128d9a Date: 2015-04-14 12:30 + http://bitbucket.org/pypy/extradoc/changeset/5bb398128d9a/ Log:tweaks diff --git a/talk/ep2015/performance-abstract.txt b/talk/ep2015/performance-abstract.txt --- a/talk/ep2015/performance-abstract.txt +++ b/talk/ep2015/performance-abstract.txt @@ -1,6 +1,10 @@ - -Python performance (not) for dummies - +== +Python and PyPy performance (not) for dummies +== + +[[The talk will be given by Antonio Cuni and Maciej Fijalkowski, +both long time PyPy core developers and expert in the area of +Python performance.]] In this talk we would like to have a short introduction on how Python programs are compiled and executed, with a special attention towards ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] extradoc extradoc: more slides
Author: Antonio Cuni Branch: extradoc Changeset: r5531:9b1bfdbddfa5 Date: 2015-04-16 17:52 +0200 http://bitbucket.org/pypy/extradoc/changeset/9b1bfdbddfa5/ Log:more slides diff --git a/talk/pycon-italy-2015/jit-overview.odg b/talk/pycon-italy-2015/jit-overview.odg new file mode 100644 index ..43aec7c16da3c72bec87a6be52ce41969f4db8c7 GIT binary patch [cut] diff --git a/talk/pycon-italy-2015/jit-overview1.pdf b/talk/pycon-italy-2015/jit-overview1.pdf new file mode 100644 index ..6b511590edd06e9deb157eb02cecb7cf64c61926 GIT binary patch [cut] diff --git a/talk/pycon-italy-2015/jit-overview2.pdf b/talk/pycon-italy-2015/jit-overview2.pdf new file mode 100644 index ..c07d0e3948403a7c42951a65ba023311d4739ac8 GIT binary patch [cut] diff --git a/talk/pycon-italy-2015/jit-overview3.pdf b/talk/pycon-italy-2015/jit-overview3.pdf new file mode 100644 index ..277b2067bc2daf79f7696edaabd22614182e6bfa GIT binary patch [cut] diff --git a/talk/pycon-italy-2015/talk.rst b/talk/pycon-italy-2015/talk.rst --- a/talk/pycon-italy-2015/talk.rst +++ b/talk/pycon-italy-2015/talk.rst @@ -102,6 +102,30 @@ :scale: 47% +The JIT + + +.. image:: jit-overview1.pdf + :scale: 50% + + +The JIT + + +.. image:: jit-overview2.pdf + :scale: 50% + + +The JIT + + +.. image:: jit-overview3.pdf + :scale: 50% + + + + + Contacts, Q&A -- ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] extradoc extradoc: merge
Author: Antonio Cuni Branch: extradoc Changeset: r5532:2403b934d742 Date: 2015-04-16 17:52 +0200 http://bitbucket.org/pypy/extradoc/changeset/2403b934d742/ Log:merge diff --git a/talk/ep2015/performance-abstract.txt b/talk/ep2015/performance-abstract.txt --- a/talk/ep2015/performance-abstract.txt +++ b/talk/ep2015/performance-abstract.txt @@ -1,20 +1,25 @@ -== -PyPy performance (not) for dummies -== +== +Python and PyPy performance (not) for dummies +== +[[The talk will be given by Antonio Cuni and Maciej Fijalkowski, +both long time PyPy core developers and expert in the area of +Python performance.]] -Abstract -- +In this talk we would like to have a short introduction on how Python +programs are compiled and executed, with a special attention towards +just in time compilation done by PyPy. PyPy is the most advanced Python +interpreter around and while it should generally just speed up your programs +there is a wide range of performance that you can get out of PyPy, ranging from +slightly faster than CPython to C speeds, depending on how you write your +programs. -PyPy is the fastest Python interpreter around, and its JIT can optimize most -of your Python programs without problems. However, there are techniques to -improve the performances even further and squeeze the most out of PyPy. In -this talk we will see: +We will split the talk in two parts. In the first part we will explain +how things work and what can and what cannot be optimized as well as describe +the basic heuristics of JIT compiler and optimizer. In the next part we will +do a survey of existing tools for looking at performance of Python programs +with specific focus on PyPy. -- the general principles behind the PyPy JIT - -- how to profile programs to find the bottlenecks - -- how to examine the code generated by the JIT - -- how to write JIT-friendly programs +As a result of this talk, an audience member should be better equipped with +tools how to write new software and improve existing software with performance +in mind. ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] extradoc extradoc: first part of the slides
Author: Antonio Cuni Branch: extradoc Changeset: r5530:5d0633b2f4f8 Date: 2015-04-16 17:29 +0200 http://bitbucket.org/pypy/extradoc/changeset/5d0633b2f4f8/ Log:first part of the slides diff --git a/talk/pycon-italy-2015/speed.png b/talk/pycon-italy-2015/speed.png new file mode 100644 index ..e2372b9e0d5fc3fe7711b27d641d2d9706185fd3 GIT binary patch [cut] diff --git a/talk/pycon-italy-2015/talk.rst b/talk/pycon-italy-2015/talk.rst --- a/talk/pycon-italy-2015/talk.rst +++ b/talk/pycon-italy-2015/talk.rst @@ -16,10 +16,90 @@ - http://antocuni.eu -PyPy status - +What is PyPy +--- -bla bla +- Alternative, fast Python implementation + +- Performance: JIT compiler, advanced GC + +- STM: goodbye GIL + +- PyPy 2.5.1 (2.7.8) + +- Py3k as usual in progress (3.2.5 out, 3.3 in development) + +- http://pypy.org + + +STM +--- + +- pypy-stm-2.5.1 is out + + * 64 bit Linux only + +- no GIL! + +- 25-40% slowdown for single core programs + + * still ``7*0.75 = 5.25x`` faster than CPython :) + +- parallelism up to 4 threads + +- concurrency slow-but-correct by default + + * compared to fast-but-buggy by using threads + +- conflict detection + +- TransactionQueue: parallelize your program without using threads! + + +Extension modules +-- + +- CFFI: stable, mature and widely used + + * psycopg2cffi, lxml-cffi, pysdl2-cffi, etc. + + * should be used even for CPython-only projects! + +- numpy: + + * support for linalg + + * support for pure Python, JIT friendly ufuncs + + * object dtype in-progress + +- scipy: see next slide :) + + +Pymetabiosis +-- + +- embed CPython in PyPy + +- import and use CPython modules in PyPy + +- ALPHA status + +- slow when passing arbitrary objects + +- but fast for numpy arrays + +- matplotlib and scipy works + +- https://github.com/rguillebert/pymetabiosis + + + +Speed: 7x faster than CPython +--- + +.. image:: speed.png + :scale: 47% Contacts, Q&A ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] extradoc extradoc: template for my talk
Author: Antonio Cuni
Branch: extradoc
Changeset: r5529:6ba8055f6c90
Date: 2015-04-16 16:50 +0200
http://bitbucket.org/pypy/extradoc/changeset/6ba8055f6c90/
Log:template for my talk
diff --git a/talk/pycon-italy-2015/Makefile b/talk/pycon-italy-2015/Makefile
new file mode 100644
--- /dev/null
+++ b/talk/pycon-italy-2015/Makefile
@@ -0,0 +1,18 @@
+# you can find rst2beamer.py here:
+# http://codespeak.net/svn/user/antocuni/bin/rst2beamer.py
+
+# WARNING: to work, it needs this patch for docutils
+#
https://sourceforge.net/tracker/?func=detail&atid=422032&aid=1459707&group_id=38414
+
+talk.pdf: talk.rst author.latex title.latex stylesheet.latex
+ python `which rst2beamer.py` --stylesheet=stylesheet.latex
--documentoptions=14pt talk.rst talk.latex || exit
+ #/home/antocuni/.virtualenvs/rst2beamer/bin/python `which
rst2beamer.py` --stylesheet=stylesheet.latex --documentoptions=14pt talk.rst
talk.latex || exit
+ sed 's/\\date{}/\\input{author.latex}/' -i talk.latex || exit
+ #sed 's/\\maketitle/\\input{title.latex}/' -i talk.latex || exit
+ pdflatex talk.latex || exit
+
+view: talk.pdf
+ evince talk.pdf &
+
+xpdf: talk.pdf
+ xpdf talk.pdf &
diff --git a/talk/pycon-italy-2015/author.latex
b/talk/pycon-italy-2015/author.latex
new file mode 100644
--- /dev/null
+++ b/talk/pycon-italy-2015/author.latex
@@ -0,0 +1,8 @@
+\definecolor{rrblitbackground}{rgb}{0.0, 0.0, 0.0}
+
+\title[PyPy JIT]{PyPy JIT (not) for dummies}
+\author[antocuni]
+{Antonio Cuni}
+
+\institute{PyCon Sei}
+\date{April 17, 2015}
diff --git a/talk/pycon-italy-2015/beamerdefs.txt
b/talk/pycon-italy-2015/beamerdefs.txt
new file mode 100644
--- /dev/null
+++ b/talk/pycon-italy-2015/beamerdefs.txt
@@ -0,0 +1,108 @@
+.. colors
+.. ===
+
+.. role:: green
+.. role:: red
+
+
+.. general useful commands
+.. ===
+
+.. |pause| raw:: latex
+
+ \pause
+
+.. |small| raw:: latex
+
+ {\small
+
+.. |end_small| raw:: latex
+
+ }
+
+.. |scriptsize| raw:: latex
+
+ {\scriptsize
+
+.. |end_scriptsize| raw:: latex
+
+ }
+
+.. |strike<| raw:: latex
+
+ \sout{
+
+.. closed bracket
+.. ===
+
+.. |>| raw:: latex
+
+ }
+
+
+.. example block
+.. ===
+
+.. |example<| raw:: latex
+
+ \begin{exampleblock}{
+
+
+.. |end_example| raw:: latex
+
+ \end{exampleblock}
+
+
+
+.. alert block
+.. ===
+
+.. |alert<| raw:: latex
+
+ \begin{alertblock}{
+
+
+.. |end_alert| raw:: latex
+
+ \end{alertblock}
+
+
+
+.. columns
+.. ===
+
+.. |column1| raw:: latex
+
+ \begin{columns}
+ \begin{column}{0.45\textwidth}
+
+.. |column2| raw:: latex
+
+ \end{column}
+ \begin{column}{0.45\textwidth}
+
+
+.. |end_columns| raw:: latex
+
+ \end{column}
+ \end{columns}
+
+
+
+.. |snake| image:: ../../img/py-web-new.png
+ :scale: 15%
+
+
+
+.. nested blocks
+.. ===
+
+.. |nested| raw:: latex
+
+ \begin{columns}
+ \begin{column}{0.85\textwidth}
+
+.. |end_nested| raw:: latex
+
+ \end{column}
+ \end{columns}
diff --git a/talk/pycon-italy-2015/stylesheet.latex
b/talk/pycon-italy-2015/stylesheet.latex
new file mode 100644
--- /dev/null
+++ b/talk/pycon-italy-2015/stylesheet.latex
@@ -0,0 +1,10 @@
+\usetheme{Boadilla}
+\setbeamercovered{transparent}
+\setbeamertemplate{navigation symbols}{}
+
+\definecolor{darkgreen}{rgb}{0, 0.5, 0.0}
+\newcommand{\docutilsrolegreen}[1]{\color{darkgreen}#1\normalcolor}
+\newcommand{\docutilsrolered}[1]{\color{red}#1\normalcolor}
+
+\newcommand{\green}[1]{\color{darkgreen}#1\normalcolor}
+\newcommand{\red}[1]{\color{red}#1\normalcolor}
diff --git a/talk/pycon-italy-2015/talk.pdf.info
b/talk/pycon-italy-2015/talk.pdf.info
new file mode 100644
--- /dev/null
+++ b/talk/pycon-italy-2015/talk.pdf.info
@@ -0,0 +1,11 @@
+AvailableTransitions=[Crossfade]
+TransitionDuration = 100
+EstimatedDuration = 45*60 # in seconds
+MinutesOnly = True
+
+PageProps = {
+1: {
+'reset': FirstTimeOnly,
+'progress': False,
+},
+}
diff --git a/talk/pycon-italy-2015/talk.rst b/talk/pycon-italy-2015/talk.rst
new file mode 100644
--- /dev/null
+++ b/talk/pycon-italy-2015/talk.rst
@@ -0,0 +1,41 @@
+.. include:: beamerdefs.txt
+
+
+PyPy JIT (not) for dummies
+
+
+About me
+-
+
+- PyPy core dev
+
+- ``pdb++``, ``fancycompleter``, ...
+
+- Consultant, trainer
+
+- http://antocuni.eu
+
+
+PyPy status
+
+
+bla bla
+
+
+Contacts, Q&A
+--
+
+- http://pypy.org
+
+- http://morepypy.blogspot.com/
+
+- twitter: @antocuni
+
+- Available for consultancy & training:
+
+ * http://antocuni.eu
+
+ * i...@antocuni.eu
+
+- Any question?
+
diff --git a/talk/pycon-italy-2015/title.latex
b/talk/pycon-italy-2015/title.latex
ne
[pypy-commit] extradoc extradoc: more slides
Author: Antonio Cuni Branch: extradoc Changeset: r5533:5ff800bfd872 Date: 2015-04-17 00:40 +0200 http://bitbucket.org/pypy/extradoc/changeset/5ff800bfd872/ Log:more slides diff --git a/talk/pycon-italy-2015/talk.rst b/talk/pycon-italy-2015/talk.rst --- a/talk/pycon-italy-2015/talk.rst +++ b/talk/pycon-italy-2015/talk.rst @@ -121,7 +121,102 @@ .. image:: jit-overview3.pdf :scale: 50% - + + +JIT overview +- + +- Tracing JIT + + * detect and compile "hot" loops + + * (although not only loops) + +- **Specialization** + +- Precompute as much as possible + +- Constant propagation + +- Aggressive inlining + + +Specialization (1) +--- + +- ``obj.foo()`` + +- which code is executed? (SIMPLIFIED) + + * lookup ``foo`` in obj.__dict__ + + * lookup ``foo`` in obj.__class__ + + * lookup ``foo`` in obj.__bases__[0], etc. + + * finally, execute ``foo`` + +- without JIT, you need to do these steps again and again + +- Precompute the lookup? + + +Specialization (2) + + +- pretend and assume that ``obj.__class__`` IS constant + + * "promotion" + +- guard + + * check our assumption: if it's false, bail out + +- now we can directly jump to ``foo`` code + + * ...unless ``foo`` is in ``obj.__dict__``: GUARD! + + * ...unless ``foo.__class__.__dict__`` changed: GUARD! + +- Too many guard failures? + + * Compile some more assembler! + +- guards are cheap + + * out-of-line guards even more + + +Specialization (3) +- + +- who decides what to promote/specialize for? + + * we, the PyPy devs :) + + * heuristics + +- instance attributes are never promoted + +- class attributes are promoted by default (with some exceptions) + +- module attributes (i.e., globals) as well + +- bytecode constants + + +Specialization trade-offs +-- + +- Too much specialization + + * guards fails often + + * explosion of assembler + +- Not enough specialization + + * inefficient code ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] extradoc extradoc: introduce a real world example
Author: Antonio Cuni
Branch: extradoc
Changeset: r5534:14123b4ae075
Date: 2015-04-17 12:55 +0200
http://bitbucket.org/pypy/extradoc/changeset/14123b4ae075/
Log:introduce a real world example
diff --git a/talk/pycon-italy-2015/src/decode0.py
b/talk/pycon-italy-2015/src/decode0.py
new file mode 100644
--- /dev/null
+++ b/talk/pycon-italy-2015/src/decode0.py
@@ -0,0 +1,25 @@
+import sys
+import struct
+
+P1 = '\x0c\x00\x00\x00"\x00\x00\x00\x07\x00'
+P2 = '\x15\x00\x00\x00+\x00\x00\x00\x08\x00'
+
+PLIST = [P1, P2] * 2000
+
+def read_x(p):
+return struct.unpack_from('l', p, 0)[0]
+
+def read_y(p):
+return struct.unpack_from('l', p, 4)[0]
+
+def read_color(p):
+return struct.unpack_from('i', p, 8)[0]
+
+def main():
+res = 0
+for p in PLIST:
+x = read_x(p)
+res += x
+print res
+
+main()
diff --git a/talk/pycon-italy-2015/src/decode1.py
b/talk/pycon-italy-2015/src/decode1.py
new file mode 100644
--- /dev/null
+++ b/talk/pycon-italy-2015/src/decode1.py
@@ -0,0 +1,41 @@
+import sys
+import struct
+
+P1 = '\x0c\x00\x00\x00"\x00\x00\x00\x07\x00'
+P2 = '\x15\x00\x00\x00+\x00\x00\x00\x08\x00'
+
+PLIST = [P1, P2] * 2000
+
+class Field(object):
+
+def __init__(self, fmt, offset):
+self.fmt = fmt
+self.offset = offset
+
+
+class Message(object):
+
+def __init__(self, name, fields):
+self._name = name
+self._fields = fields
+
+def read(self, buf, name):
+f = self._fields[name]
+return struct.unpack_from(f.fmt, buf, f.offset)[0]
+
+
+Point = Message('Point', {
+'x': Field('l', 0),
+'y': Field('l', 4),
+'color': Field('i', 8)
+})
+
+
+def main():
+res = 0
+for p in PLIST:
+x = Point.read(p, 'x')
+res += x
+print res
+
+main()
diff --git a/talk/pycon-italy-2015/talk.rst b/talk/pycon-italy-2015/talk.rst
--- a/talk/pycon-italy-2015/talk.rst
+++ b/talk/pycon-italy-2015/talk.rst
@@ -219,6 +219,220 @@
* inefficient code
+Virtuals
+
+
+- Remove unnecessary allocations
+
+- Remove unnecessary load/store
+
+- "Abstraction for free"
+
+
+Example
+
+
+- Real world example
+
+- Decoding binary messages
+
+- Messages: strings of bytes
+
+|small|
+|example<| |small| Point |end_small| |>|
+
+.. sourcecode:: C
+
+struct Point {
+long x;
+long y;
+int color;
+}
+
+|end_example|
+|end_small|
+
+
+
+Example: low-level solution
+
+
+|scriptsize|
+|example<| |small| decode0.py |end_small| |>|
+
+.. sourcecode:: python
+
+P1 = '\x0c\x00\x00\x00"\x00\x00\x00\x07\x00\x00\x00'
+P2 = '\x15\x00\x00\x00+\x00\x00\x00\x08\x00\x00\x00'
+
+PLIST = [P1, P2] * 2000
+
+def read_x(p):
+return struct.unpack_from('l', p, 0)[0]
+
+def main():
+res = 0
+for p in PLIST:
+x = read_x(p)
+res += x
+print res
+
+|end_example|
+|end_scriptsize|
+
+Example: low-level solution
+
+
+|scriptsize|
+|example<| |small| decode0.py trace |end_small| |>|
+
+.. sourcecode:: python
+
+debug_merge_point(1, 1, ' #0 LOAD_GLOBAL')
+debug_merge_point(1, 1, ' #3 LOOKUP_METHOD')
+debug_merge_point(1, 1, ' #6 LOAD_CONST')
+debug_merge_point(1, 1, ' #9 LOAD_FAST')
+debug_merge_point(1, 1, ' #12 LOAD_CONST')
+debug_merge_point(1, 1, ' #15 CALL_METHOD')
++606: i91 = strlen(p88)
++609: i92 = int_lt(i91, 4)
+guard_false(i92, descr=)
++618: i93 = strgetitem(p88, 0)
++622: i94 = strgetitem(p88, 1)
++632: i95 = int_lshift(i94, 8)
++635: i96 = int_or(i93, i95)
++637: i97 = strgetitem(p88, 2)
++653: i98 = int_lshift(i97, 16)
++656: i99 = int_or(i96, i98)
++658: i100 = strgetitem(p88, 3)
++662: i101 = int_ge(i100, 128)
+guard_false(i101, descr=)
++674: i102 = int_lshift(i100, 24)
++677: i103 = int_or(i99, i102)
+
+|end_example|
+|end_scriptsize|
+
+Example: better API
+-
+
+|scriptsize|
+|example<| |small| decode1.py |end_small| |>|
+
+.. sourcecode:: python
+
+class Field(object):
+def __init__(self, fmt, offset):
+self.fmt = fmt; self.offset = offset
+
+class Message(object):
+def __init__(self, name, fields):
+self._name = name; self._fields = fields
+
+def read(self, buf, name):
+f = self._fields[name]
+return struct.unpack_from(f.fmt, buf, f.offset)[0]
+
+Point = Message('Point', {'x': Field('l', 0),
+ 'y': Field('l', 8),
+ 'color': Field('i', 16)})
+
+def main():
+res = 0
+for p in PLIST:
+x = Point.read(p, 'x')
+res += x
+print res
+
+|end_example|
+|end_scriptsize|
+
+Example: better API
+
+
+|scriptsize|
+|example<| |small| decode1.py trace (1) |end_small| |>|
+
+.. sourcecode:: python
+
+debug_merge_point(1, 1, '
[pypy-commit] extradoc extradoc: add pdf
Author: Antonio Cuni Branch: extradoc Changeset: r5537:0769e28550ef Date: 2015-04-17 13:19 +0200 http://bitbucket.org/pypy/extradoc/changeset/0769e28550ef/ Log:add pdf diff --git a/talk/pycon-italy-2015/talk.pdf b/talk/pycon-italy-2015/talk.pdf new file mode 100644 index ..1492bcaa865783996c4f06a4afc7eabcea54249b GIT binary patch [cut] ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] extradoc extradoc: more slides
Author: Antonio Cuni
Branch: extradoc
Changeset: r5535:60dd04fca106
Date: 2015-04-17 13:10 +0200
http://bitbucket.org/pypy/extradoc/changeset/60dd04fca106/
Log:more slides
diff --git a/talk/pycon-italy-2015/src/decode2.py
b/talk/pycon-italy-2015/src/decode2.py
new file mode 100644
--- /dev/null
+++ b/talk/pycon-italy-2015/src/decode2.py
@@ -0,0 +1,42 @@
+import sys
+import struct
+
+P1 = '\x0c\x00\x00\x00"\x00\x00\x00\x07\x00'
+P2 = '\x15\x00\x00\x00+\x00\x00\x00\x08\x00'
+
+PLIST = [P1, P2] * 2000
+
+class Field(object):
+
+def __init__(self, fmt, offset):
+self.fmt = fmt
+self.offset = offset
+
+def Message(name, fields):
+class M(object):
+def read(self, buf, name):
+f = getattr(self, name)
+return struct.unpack_from(f.fmt, buf, f.offset)[0]
+
+for fname, f in fields.iteritems():
+setattr(M, fname, f)
+
+M.__name__ = name
+return M()
+
+
+Point = Message('Point', {
+'x': Field('l', 0),
+'y': Field('l', 4),
+'color': Field('i', 8)
+})
+
+
+def main():
+res = 0
+for p in PLIST:
+x = Point.read(p, 'x')
+res += x
+print res
+
+main()
diff --git a/talk/pycon-italy-2015/src/decode3.py
b/talk/pycon-italy-2015/src/decode3.py
new file mode 100644
--- /dev/null
+++ b/talk/pycon-italy-2015/src/decode3.py
@@ -0,0 +1,32 @@
+import sys
+import struct
+
+P1 = '\x0c\x00\x00\x00"\x00\x00\x00\x07\x00'
+P2 = '\x15\x00\x00\x00+\x00\x00\x00\x08\x00'
+
+PLIST = [P1, P2] * 2000
+
+class Field(object):
+def __init__(self, fmt, offset):
+self.fmt = fmt
+self.offset = offset
+
+def __get__(self, obj, cls):
+return struct.unpack_from(self.fmt, obj._buf, self.offset)[0]
+
+class Point(object):
+def __init__(self, buf):
+self._buf = buf
+
+x = Field('l', 0)
+y = Field('l', 4)
+color = Field('h', 8)
+
+def main():
+res = 0
+for p in PLIST:
+p = Point(p)
+res += p.x
+print res
+
+main()
diff --git a/talk/pycon-italy-2015/talk.rst b/talk/pycon-italy-2015/talk.rst
--- a/talk/pycon-italy-2015/talk.rst
+++ b/talk/pycon-italy-2015/talk.rst
@@ -244,9 +244,9 @@
.. sourcecode:: C
struct Point {
-long x;
-long y;
-int color;
+int x;
+int y;
+short color;
}
|end_example|
@@ -433,6 +433,121 @@
|end_example|
|end_scriptsize|
+Example: faster API
+-
+
+|scriptsize|
+|example<| |small| decode2.py |end_small| |>|
+
+.. sourcecode:: python
+
+def Message(name, fields):
+class M(object):
+def read(self, buf, name):
+f = getattr(self, name)
+return struct.unpack_from(f.fmt, buf, f.offset)[0]
+
+for fname, f in fields.iteritems():
+setattr(M, fname, f)
+
+M.__name__ = name
+return M()
+
+Point = Message('Point', {
+'x': Field('l', 0),
+'y': Field('l', 4),
+'color': Field('i', 8)
+})
+
+ ...
+ x = Point.read(p, 'x')
+ ...
+
+|end_example|
+|end_scriptsize|
+
+Example: faster API
+
+
+|scriptsize|
+|example<| |small| decode2.py trace (3) |end_small| |>|
+
+.. sourcecode:: python
+
+debug_merge_point(1, 1, ' #36 CALL_METHOD')
++670: i104 = strlen(p101)
++673: i105 = int_lt(i104, 4)
+guard_false(i105, descr=)
++682: i106 = strgetitem(p101, 0)
++686: i107 = strgetitem(p101, 1)
++696: i108 = int_lshift(i107, 8)
++699: i109 = int_or(i106, i108)
++701: i110 = strgetitem(p101, 2)
++717: i111 = int_lshift(i110, 16)
++720: i112 = int_or(i109, i111)
++722: i113 = strgetitem(p101, 3)
++726: i114 = int_ge(i113, 128)
+guard_false(i114, descr=)
++738: i115 = int_lshift(i113, 24)
++741: i116 = int_or(i112, i115)
+
+|end_example|
+|end_scriptsize|
+
+What happened?
+---
+
+- dict lookups inside classes are specialized
+
+- decode1.py
+
+ * ``fields`` is "normal data" and expected to change
+
+ * one JIT code for **all** possible messages
+
+- decode2.py
+
+ * ``fields`` is expected to be constant
+
+ * one JIT code for **each** message
+
+- Behaviour is the same, different performance
+
+
+Example: even better API :)
+-
+
+|scriptsize|
+|example<| |small| decode3.py |end_small| |>|
+
+.. sourcecode:: python
+
+class Field(object):
+def __init__(self, fmt, offset):
+self.fmt = fmt
+self.offset = offset
+
+def __get__(self, obj, cls):
+return struct.unpack_from(self.fmt, obj._buf, self.offset)[0]
+
+class Point(object):
+def __init__(self, buf):
+self._buf = buf
+
+x = Field('l', 0)
+y = Field('l', 4)
+color = Field('h', 8)
+
+def main():
+res = 0
+for p in PLIST:
+p = Point(p)
+res += p.x
+print r
[pypy-commit] extradoc extradoc: expand virtuals
Author: Antonio Cuni Branch: extradoc Changeset: r5536:673c2a4346a1 Date: 2015-04-17 13:13 +0200 http://bitbucket.org/pypy/extradoc/changeset/673c2a4346a1/ Log:expand virtuals diff --git a/talk/pycon-italy-2015/talk.rst b/talk/pycon-italy-2015/talk.rst --- a/talk/pycon-italy-2015/talk.rst +++ b/talk/pycon-italy-2015/talk.rst @@ -226,7 +226,18 @@ - Remove unnecessary load/store -- "Abstraction for free" +|small| +|example<| |small| virtuals.py |end_small| |>| + +.. sourcecode:: python + +res = 0 +while res < 1: +obj = Foo(x, y, z) +res += obj.x + +|end_example| +|end_small| Example ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pyrepl default: fix this bug, which has been around unnoticed forever
Author: Antonio Cuni
Branch:
Changeset: r258:cef132a7e464
Date: 2014-08-02 10:15 +0100
http://bitbucket.org/pypy/pyrepl/changeset/cef132a7e464/
Log:fix this bug, which has been around unnoticed forever
diff --git a/pyrepl/completing_reader.py b/pyrepl/completing_reader.py
--- a/pyrepl/completing_reader.py
+++ b/pyrepl/completing_reader.py
@@ -40,9 +40,8 @@
STRIPCOLOR_REGEX = re.compile(r"\x1B\[([0-9]{1,3}(;[0-9]{1,2})?)?[m|K]")
-
def stripcolor(s):
-return STRIPCOLOR_REGEX.regexp.sub('', s)
+return STRIPCOLOR_REGEX.sub('', s)
def real_len(s):
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] extradoc extradoc: add myself
Author: Antonio Cuni Branch: extradoc Changeset: r5420:523d967e3386 Date: 2014-10-03 18:35 +0200 http://bitbucket.org/pypy/extradoc/changeset/523d967e3386/ Log:add myself diff --git a/sprintinfo/warsaw-2014/people.txt b/sprintinfo/warsaw-2014/people.txt --- a/sprintinfo/warsaw-2014/people.txt +++ b/sprintinfo/warsaw-2014/people.txt @@ -14,4 +14,5 @@ Romain Guillebert19/10-26-10??? Manuel Jacob 20/10-26/10? (shared hotel room?) Kostia Lopuhin +Antonio Cuni 20/10-26/10ibis Reduta http://www.ibis.com/gb/hotel-7125-ibis-warszawa-reduta/index.shtml == === ___ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] extradoc extradoc: copy all the stuff needed to produce the pdf, expand the slides
Author: Antonio Cuni
Branch: extradoc
Changeset: r5542:0e6f6ebd4d39
Date: 2015-07-17 15:46 +0200
http://bitbucket.org/pypy/extradoc/changeset/0e6f6ebd4d39/
Log:copy all the stuff needed to produce the pdf, expand the slides
diff --git a/talk/ep2015/performance/Makefile b/talk/ep2015/performance/Makefile
new file mode 100644
--- /dev/null
+++ b/talk/ep2015/performance/Makefile
@@ -0,0 +1,18 @@
+# you can find rst2beamer.py here:
+# http://codespeak.net/svn/user/antocuni/bin/rst2beamer.py
+
+# WARNING: to work, it needs this patch for docutils
+#
https://sourceforge.net/tracker/?func=detail&atid=422032&aid=1459707&group_id=38414
+
+talk.pdf: talk.rst author.latex stylesheet.latex
+ python `which rst2beamer.py` --stylesheet=stylesheet.latex
--documentoptions=14pt talk.rst talk.latex || exit
+ #/home/antocuni/.virtualenvs/rst2beamer/bin/python `which
rst2beamer.py` --stylesheet=stylesheet.latex --documentoptions=14pt talk.rst
talk.latex || exit
+ sed 's/\\date{}/\\input{author.latex}/' -i talk.latex || exit
+ #sed 's/\\maketitle/\\input{title.latex}/' -i talk.latex || exit
+ pdflatex talk.latex || exit
+
+view: talk.pdf
+ evince talk.pdf &
+
+xpdf: talk.pdf
+ xpdf talk.pdf &
diff --git a/talk/ep2015/performance/author.latex
b/talk/ep2015/performance/author.latex
new file mode 100644
--- /dev/null
+++ b/talk/ep2015/performance/author.latex
@@ -0,0 +1,8 @@
+\definecolor{rrblitbackground}{rgb}{0.0, 0.0, 0.0}
+
+\title[Python and PyPy performance]{Python and PyPy performance\\(not) for
dummies}
+\author[antocuni,fijal]
+{Antonio Cuni and Maciej Fijalkowski}
+
+\institute{EuroPython 2015}
+\date{July 21, 2015}
diff --git a/talk/ep2015/performance/beamerdefs.txt
b/talk/ep2015/performance/beamerdefs.txt
new file mode 100644
--- /dev/null
+++ b/talk/ep2015/performance/beamerdefs.txt
@@ -0,0 +1,108 @@
+.. colors
+.. ===
+
+.. role:: green
+.. role:: red
+
+
+.. general useful commands
+.. ===
+
+.. |pause| raw:: latex
+
+ \pause
+
+.. |small| raw:: latex
+
+ {\small
+
+.. |end_small| raw:: latex
+
+ }
+
+.. |scriptsize| raw:: latex
+
+ {\scriptsize
+
+.. |end_scriptsize| raw:: latex
+
+ }
+
+.. |strike<| raw:: latex
+
+ \sout{
+
+.. closed bracket
+.. ===
+
+.. |>| raw:: latex
+
+ }
+
+
+.. example block
+.. ===
+
+.. |example<| raw:: latex
+
+ \begin{exampleblock}{
+
+
+.. |end_example| raw:: latex
+
+ \end{exampleblock}
+
+
+
+.. alert block
+.. ===
+
+.. |alert<| raw:: latex
+
+ \begin{alertblock}{
+
+
+.. |end_alert| raw:: latex
+
+ \end{alertblock}
+
+
+
+.. columns
+.. ===
+
+.. |column1| raw:: latex
+
+ \begin{columns}
+ \begin{column}{0.45\textwidth}
+
+.. |column2| raw:: latex
+
+ \end{column}
+ \begin{column}{0.45\textwidth}
+
+
+.. |end_columns| raw:: latex
+
+ \end{column}
+ \end{columns}
+
+
+
+.. |snake| image:: ../../img/py-web-new.png
+ :scale: 15%
+
+
+
+.. nested blocks
+.. ===
+
+.. |nested| raw:: latex
+
+ \begin{columns}
+ \begin{column}{0.85\textwidth}
+
+.. |end_nested| raw:: latex
+
+ \end{column}
+ \end{columns}
diff --git a/talk/ep2015/performance/jit-overview1.pdf
b/talk/ep2015/performance/jit-overview1.pdf
new file mode 100644
index
..6b511590edd06e9deb157eb02cecb7cf64c61926
GIT binary patch
[cut]
diff --git a/talk/ep2015/performance/jit-overview2.pdf
b/talk/ep2015/performance/jit-overview2.pdf
new file mode 100644
index
..c07d0e3948403a7c42951a65ba023311d4739ac8
GIT binary patch
[cut]
diff --git a/talk/ep2015/performance/jit-overview3.pdf
b/talk/ep2015/performance/jit-overview3.pdf
new file mode 100644
index
..277b2067bc2daf79f7696edaabd22614182e6bfa
GIT binary patch
[cut]
diff --git a/talk/ep2015/performance/speed.png
b/talk/ep2015/performance/speed.png
new file mode 100644
index
..e980452b50ec2cd6bcc1787cb5596bf106ddcd99
GIT binary patch
[cut]
diff --git a/talk/ep2015/performance/stylesheet.latex
b/talk/ep2015/performance/stylesheet.latex
new file mode 100644
--- /dev/null
+++ b/talk/ep2015/performance/stylesheet.latex
@@ -0,0 +1,10 @@
+\usetheme{Boadilla}
+\setbeamercovered{transparent}
+\setbeamertemplate{navigation symbols}{}
+
+\definecolor{darkgreen}{rgb}{0, 0.5, 0.0}
+\newcommand{\docutilsrolegreen}[1]{\color{darkgreen}#1\normalcolor}
+\newcommand{\docutilsrolered}[1]{\color{red}#1\normalcolor}
+
+\newcommand{\green}[1]{\color{darkgreen}#1\normalcolor}
+\newcommand{\red}[1]{\color{red}#1\normalcolor}
diff --git a/talk/ep2015/performance/talk.rst b/talk/ep2015/performance/talk.rst
--- a/talk/ep2015/performance/talk.rst
+++ b/talk/ep2015/performance/tal
