Author: Wim Lavrijsen <wlavrij...@lbl.gov>
Branch: cppyy-packaging
Changeset: r94403:60b72b97202f
Date: 2018-04-19 10:47 -0700
http://bitbucket.org/pypy/pypy/changeset/60b72b97202f/
Log: merge default into branch
diff too long, truncating to 2000 out of 19728 lines
diff --git a/LICENSE b/LICENSE
--- a/LICENSE
+++ b/LICENSE
@@ -6,36 +6,36 @@
Except when otherwise stated (look for LICENSE files in directories or
information at the beginning of each file) all software and documentation in
the 'rpython', 'pypy', 'ctype_configure', 'dotviewer', 'demo', 'lib_pypy',
-'py', and '_pytest' directories is licensed as follows:
+'py', and '_pytest' directories is licensed as follows:
The MIT License
- Permission is hereby granted, free of charge, to any person
- obtaining a copy of this software and associated documentation
- files (the "Software"), to deal in the Software without
- restriction, including without limitation the rights to use,
- copy, modify, merge, publish, distribute, sublicense, and/or
- sell copies of the Software, and to permit persons to whom the
+ Permission is hereby granted, free of charge, to any person
+ obtaining a copy of this software and associated documentation
+ files (the "Software"), to deal in the Software without
+ restriction, including without limitation the rights to use,
+ copy, modify, merge, publish, distribute, sublicense, and/or
+ sell copies of the Software, and to permit persons to whom the
Software is furnished to do so, subject to the following conditions:
- The above copyright notice and this permission notice shall be included
+ The above copyright notice and this permission notice shall be included
in all copies or substantial portions of the Software.
- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
- OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY,
- FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+ OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.
PyPy Copyright holders 2003-2018
------------------------------------
+--------------------------------
Except when otherwise stated (look for LICENSE files or information at
the beginning of each file) the files in the 'pypy' directory are each
-copyrighted by one or more of the following people and organizations:
+copyrighted by one or more of the following people and organizations:
Armin Rigo
Maciej Fijalkowski
@@ -89,13 +89,13 @@
Niko Matsakis
Alexander Hesse
Ludovic Aubry
+ stian
Jacob Hallen
Jason Creighton
Mark Young
Alex Martelli
Spenser Bauman
Michal Bendowski
- stian
Jan de Mooij
Tyler Wade
Vincent Legoll
@@ -123,10 +123,10 @@
Wenzhu Man
Konstantin Lopuhin
John Witulski
+ Jeremy Thurgood
Greg Price
Ivan Sichmann Freitas
Dario Bertini
- Jeremy Thurgood
Mark Pearse
Simon Cross
Tobias Pape
@@ -145,18 +145,19 @@
Adrian Kuhn
tav
Georg Brandl
+ Joannah Nanjekye
Bert Freudenberg
Stian Andreassen
Wanja Saatkamp
Mike Blume
- Joannah Nanjekye
Gerald Klix
Oscar Nierstrasz
Rami Chowdhury
Stefan H. Muller
+ Dodan Mihai
Tim Felgentreff
Eugene Oden
- Dodan Mihai
+ Colin Valliant
Jeff Terrace
Henry Mason
Vasily Kuznetsov
@@ -225,12 +226,14 @@
Vaibhav Sood
Reuben Cummings
Attila Gobi
+ Floris Bruynooghe
Christopher Pope
Tristan Arthur
Christian Tismer
Dan Stromberg
Carl Meyer
Florin Papa
+ Arianna Avanzini
Jens-Uwe Mager
Valentina Mukhamedzhanova
Stefano Parmesan
@@ -250,9 +253,11 @@
Alejandro J. Cura
Vladimir Kryachko
Gabriel
+ Thomas Hisch
Mark Williams
Kunal Grover
Nathan Taylor
+ Barry Hart
Travis Francis Athougies
Yasir Suhail
Sergey Kishchenko
@@ -260,6 +265,7 @@
Lutz Paelike
Ian Foote
Philipp Rustemeuer
+ Logan Chien
Catalin Gabriel Manciu
Jacob Oscarson
Ryan Gonzalez
@@ -295,7 +301,6 @@
Akira Li
Gustavo Niemeyer
Rafał Gałczyński
- Logan Chien
Lucas Stadler
roberto@goyle
Matt Bogosian
@@ -308,6 +313,7 @@
Anna Katrina Dominguez
Kim Jin Su
Amber Brown
+ Miro Hrončok
Anthony Sottile
Nate Bragg
Ben Darnell
@@ -315,7 +321,6 @@
Godefroid Chappelle
Julian Berman
Michael Hudson-Doyle
- Floris Bruynooghe
Stephan Busemann
Dan Colish
timo
@@ -357,6 +362,7 @@
Michael Chermside
Anna Ravencroft
remarkablerocket
+ Pauli Virtanen
Petre Vijiac
Berker Peksag
Christian Muirhead
@@ -381,6 +387,7 @@
Graham Markall
Dan Loewenherz
werat
+ Andrew Stepanov
Niclas Olofsson
Chris Pressey
Tobias Diaz
@@ -395,14 +402,14 @@
m...@funkyhat.org
Stefan Marr
- Heinrich-Heine University, Germany
+ Heinrich-Heine University, Germany
Open End AB (formerly AB Strakt), Sweden
- merlinux GmbH, Germany
- tismerysoft GmbH, Germany
- Logilab Paris, France
- DFKI GmbH, Germany
+ merlinux GmbH, Germany
+ tismerysoft GmbH, Germany
+ Logilab Paris, France
+ DFKI GmbH, Germany
Impara, Germany
- Change Maker, Sweden
+ Change Maker, Sweden
University of California Berkeley, USA
Google Inc.
King's College London
@@ -410,14 +417,14 @@
The PyPy Logo as used by http://speed.pypy.org and others was created
by Samuel Reis and is distributed on terms of Creative Commons Share Alike
License.
-
-License for 'lib-python/2.7'
-============================
+
+License for 'lib-python/2.7, lib-python/3'
+==========================================
Except when otherwise stated (look for LICENSE files or copyright/license
-information at the beginning of each file) the files in the 'lib-python/2.7'
+information at the beginning of each file) the files in the 'lib-python'
directory are all copyrighted by the Python Software Foundation and licensed
-under the terms that you can find here: https://docs.python.org/2/license.html
+under the terms that you can find here: https://docs.python.org/3/license.html
License for 'pypy/module/unicodedata/'
======================================
@@ -441,9 +448,9 @@
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
-
+
http://www.apache.org/licenses/LICENSE-2.0
-
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
diff --git a/README.rst b/README.rst
--- a/README.rst
+++ b/README.rst
@@ -4,42 +4,40 @@
Welcome to PyPy!
-PyPy is both an implementation of the Python programming language, and
-an extensive compiler framework for dynamic language implementations.
-You can build self-contained Python implementations which execute
-independently from CPython.
+PyPy is an interperter that implements the Python programming language, based
+on the RPython compiler framework for dynamic language implementations.
-The home page is:
+The home page for the interpreter is:
http://pypy.org/
-If you want to help developing PyPy, this document might help you:
+If you want to help developing PyPy, this documentation might help you:
http://doc.pypy.org/
-It will also point you to the rest of the documentation which is generated
-from files in the pypy/doc directory within the source repositories. Enjoy
-and send us feedback!
+More documentation about the RPython framework can be found here
- the pypy-dev team <pypy-...@python.org>
+ http://rpython.readthedocs.io
+The source for the documentation is in the pypy/doc directory
+
+Using PyPy instead of CPython
+=============================
+
+Please read the information at http://pypy.org to find the correct way to
+download and use PyPy as an alternative to CPython.
Building
========
-First switch to or download the correct branch. The basic choices are
-``default`` for Python 2.7 and, for Python 3.X, the corresponding py3.X
-branch (e.g. ``py3.5``).
+Building PyPy is not the recommended way to obtain the PyPy alternative python
+interpreter. It is time-consuming and requires significant computing resources.
+More information can be found here
-Build with:
+ http://doc.pypy.org/en/latest/build.html
-.. code-block:: console
+Enjoy and send us feedback!
- $ rpython/bin/rpython -Ojit pypy/goal/targetpypystandalone.py
+ the pypy-dev team <pypy-...@python.org>
-This ends up with a ``pypy-c`` or ``pypy3-c`` binary in the main pypy
-directory. We suggest to use virtualenv with the resulting
-pypy-c/pypy3-c as the interpreter; you can find more details about
-various installation schemes here:
- http://doc.pypy.org/en/latest/install.html
diff --git a/pypy/module/test_lib_pypy/pyrepl/__init__.py
b/extra_tests/test_pyrepl/__init__.py
rename from pypy/module/test_lib_pypy/pyrepl/__init__.py
rename to extra_tests/test_pyrepl/__init__.py
--- a/pypy/module/test_lib_pypy/pyrepl/__init__.py
+++ b/extra_tests/test_pyrepl/__init__.py
@@ -1,3 +1,1 @@
-import sys
-import lib_pypy.pyrepl
-sys.modules['pyrepl'] = sys.modules['lib_pypy.pyrepl']
+
diff --git a/pypy/module/test_lib_pypy/pyrepl/infrastructure.py
b/extra_tests/test_pyrepl/infrastructure.py
rename from pypy/module/test_lib_pypy/pyrepl/infrastructure.py
rename to extra_tests/test_pyrepl/infrastructure.py
diff --git a/pypy/module/test_lib_pypy/pyrepl/test_basic.py
b/extra_tests/test_pyrepl/test_basic.py
rename from pypy/module/test_lib_pypy/pyrepl/test_basic.py
rename to extra_tests/test_pyrepl/test_basic.py
diff --git a/pypy/module/test_lib_pypy/pyrepl/test_bugs.py
b/extra_tests/test_pyrepl/test_bugs.py
rename from pypy/module/test_lib_pypy/pyrepl/test_bugs.py
rename to extra_tests/test_pyrepl/test_bugs.py
diff --git a/extra_tests/test_pyrepl/test_functional.py
b/extra_tests/test_pyrepl/test_functional.py
new file mode 100644
--- /dev/null
+++ b/extra_tests/test_pyrepl/test_functional.py
@@ -0,0 +1,28 @@
+# Copyright 2000-2007 Michael Hudson-Doyle <mica...@gmail.com>
+# Maciek Fijalkowski
+# License: MIT
+# some functional tests, to see if this is really working
+
+import pytest
+import sys
+
+
+@pytest.fixture()
+def child():
+ try:
+ import pexpect
+ except ImportError:
+ pytest.skip("no pexpect module")
+ except SyntaxError:
+ pytest.skip('pexpect wont work on py3k')
+ child = pexpect.spawn(sys.executable, ['-S'], timeout=10)
+ child.logfile = sys.stdout
+ child.sendline('from pyrepl.python_reader import main')
+ child.sendline('main()')
+ return child
+
+
+def test_basic(child):
+ child.sendline('a = 3')
+ child.sendline('a')
+ child.expect('3')
diff --git a/pypy/module/test_lib_pypy/pyrepl/test_keymap.py
b/extra_tests/test_pyrepl/test_keymap.py
rename from pypy/module/test_lib_pypy/pyrepl/test_keymap.py
rename to extra_tests/test_pyrepl/test_keymap.py
diff --git a/pypy/module/test_lib_pypy/pyrepl/test_reader.py
b/extra_tests/test_pyrepl/test_reader.py
rename from pypy/module/test_lib_pypy/pyrepl/test_reader.py
rename to extra_tests/test_pyrepl/test_reader.py
diff --git a/pypy/module/test_lib_pypy/pyrepl/test_readline.py
b/extra_tests/test_pyrepl/test_readline.py
rename from pypy/module/test_lib_pypy/pyrepl/test_readline.py
rename to extra_tests/test_pyrepl/test_readline.py
diff --git a/pypy/module/test_lib_pypy/pyrepl/test_wishes.py
b/extra_tests/test_pyrepl/test_wishes.py
rename from pypy/module/test_lib_pypy/pyrepl/test_wishes.py
rename to extra_tests/test_pyrepl/test_wishes.py
diff --git a/get_externals.py b/get_externals.py
new file mode 100644
--- /dev/null
+++ b/get_externals.py
@@ -0,0 +1,69 @@
+'''Get external dependencies for building PyPy
+they will end up in the platform.host().basepath, something like
repo-root/external
+'''
+
+from __future__ import print_function
+
+import argparse
+import os
+import zipfile
+from subprocess import Popen, PIPE
+from rpython.translator.platform import host
+
+def runcmd(cmd, verbose):
+ stdout = stderr = ''
+ report = False
+ try:
+ p = Popen(cmd, stdout=PIPE, stderr=PIPE)
+ stdout, stderr = p.communicate()
+ if p.wait() != 0 or verbose:
+ report = True
+ except Exception as e:
+ stderr = str(e) + '\n' + stderr
+ report = True
+ if report:
+ print('running "%s" returned\n%s\n%s' % (' '.join(cmd), stdout,
stderr))
+ if stderr:
+ raise RuntimeError(stderr)
+
+def checkout_repo(dest='externals', org='pypy', branch='default',
verbose=False):
+ url = 'https://bitbucket.org/{}/externals'.format(org)
+ if not os.path.exists(dest):
+ cmd = ['hg','clone',url,dest]
+ runcmd(cmd, verbose)
+ cmd = ['hg','-R', dest, 'update',branch]
+ runcmd(cmd, verbose)
+
+def extract_zip(externals_dir, zip_path):
+ with zipfile.ZipFile(os.fspath(zip_path)) as zf:
+ zf.extractall(os.fspath(externals_dir))
+ return externals_dir / zf.namelist()[0].split('/')[0]
+
+def parse_args():
+ p = argparse.ArgumentParser()
+ p.add_argument('-v', '--verbose', action='store_true')
+ p.add_argument('-O', '--organization',
+ help='Organization owning the deps repos', default='pypy')
+ p.add_argument('-e', '--externals', default=host.externals,
+ help='directory in which to store dependencies',
+ )
+ p.add_argument('-b', '--branch', default=host.externals_branch,
+ help='branch to check out',
+ )
+ p.add_argument('-p', '--platform', default=None,
+ help='someday support cross-compilation, ignore for now',
+ )
+ return p.parse_args()
+
+
+def main():
+ args = parse_args()
+ checkout_repo(
+ dest=args.externals,
+ org=args.organization,
+ branch=args.branch,
+ verbose=args.verbose,
+ )
+
+if __name__ == '__main__':
+ main()
diff --git a/lib-python/2.7/re.py b/lib-python/2.7/re.py
--- a/lib-python/2.7/re.py
+++ b/lib-python/2.7/re.py
@@ -225,7 +225,7 @@
_pattern_type = type(sre_compile.compile("", 0))
-_MAXCACHE = 100
+_MAXCACHE = 1000
def _compile(*key):
# internal: compile pattern
diff --git a/lib-python/2.7/test/test_eof.py b/lib-python/2.7/test/test_eof.py
--- a/lib-python/2.7/test/test_eof.py
+++ b/lib-python/2.7/test/test_eof.py
@@ -5,7 +5,7 @@
class EOFTestCase(unittest.TestCase):
def test_EOFC(self):
- expect = "EOL while scanning string literal (<string>, line 1)"
+ expect = "end of line (EOL) while scanning string literal (<string>,
line 1)"
try:
eval("""'this is a test\
""")
@@ -15,7 +15,7 @@
raise test_support.TestFailed
def test_EOFS(self):
- expect = ("EOF while scanning triple-quoted string literal "
+ expect = ("end of file (EOF) while scanning triple-quoted string
literal "
"(<string>, line 1)")
try:
eval("""'''this is a test""")
diff --git a/lib-python/2.7/test/test_generators.py
b/lib-python/2.7/test/test_generators.py
--- a/lib-python/2.7/test/test_generators.py
+++ b/lib-python/2.7/test/test_generators.py
@@ -398,7 +398,10 @@
0
>>> type(i.gi_frame)
<type 'frame'>
->>> i.gi_running = 42
+
+PyPy prints "readonly attribute 'gi_running'" so ignore the exception detail
+
+>>> i.gi_running = 42 # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
TypeError: readonly attribute
diff --git a/lib-python/2.7/test/test_genexps.py
b/lib-python/2.7/test/test_genexps.py
--- a/lib-python/2.7/test/test_genexps.py
+++ b/lib-python/2.7/test/test_genexps.py
@@ -87,7 +87,7 @@
>>> dict(a = i for i in xrange(10))
Traceback (most recent call last):
...
- SyntaxError: invalid syntax
+ SyntaxError: invalid syntax (expected ')')
Verify that parenthesis are required when used as a keyword argument value
diff --git a/lib-python/2.7/test/test_traceback.py
b/lib-python/2.7/test/test_traceback.py
--- a/lib-python/2.7/test/test_traceback.py
+++ b/lib-python/2.7/test/test_traceback.py
@@ -123,10 +123,7 @@
self.assertEqual(len(err), 4)
self.assertEqual(err[1].strip(), "print(2)")
self.assertIn("^", err[2])
- if check_impl_detail():
- self.assertEqual(err[1].find("p"), err[2].find("^"))
- if check_impl_detail(pypy=True):
- self.assertEqual(err[1].find("2)") + 1, err[2].find("^"))
+ self.assertEqual(err[1].find("p"), err[2].find("^"))
def test_base_exception(self):
# Test that exceptions derived from BaseException are formatted right
diff --git a/lib-python/2.7/threading.py b/lib-python/2.7/threading.py
--- a/lib-python/2.7/threading.py
+++ b/lib-python/2.7/threading.py
@@ -351,6 +351,21 @@
# forward-compatibility reasons we do the same.
waiter.acquire()
gotit = True
+ except AttributeError:
+ # someone patched the 'waiter' class, probably.
+ # Fall back to the standard CPython logic.
+ # See the CPython lib for the comments about it...
+ endtime = _time() + timeout
+ delay = 0.0005 # 500 us -> initial delay of 1 ms
+ while True:
+ gotit = waiter.acquire(0)
+ if gotit:
+ break
+ remaining = endtime - _time()
+ if remaining <= 0:
+ break
+ delay = min(delay * 2, remaining, .05)
+ _sleep(delay)
else:
gotit = waiter.acquire(False)
if not gotit:
diff --git a/lib_pypy/_pypy_testcapi.py b/lib_pypy/_pypy_testcapi.py
--- a/lib_pypy/_pypy_testcapi.py
+++ b/lib_pypy/_pypy_testcapi.py
@@ -2,7 +2,7 @@
import tempfile, binascii
-def get_hashed_dir(cfile):
+def _get_hashed_filename(cfile):
with open(cfile,'r') as fid:
content = fid.read()
# from cffi's Verifier()
@@ -21,10 +21,28 @@
username = os.environ['USERNAME'] #windows
except KeyError:
username = os.getuid()
- output_dir = tempfile.gettempdir() + os.path.sep + 'tmp_%s_%s%s' % (
+ return tempfile.gettempdir() + os.path.sep + 'testcapi_%s_%s%s' % (
username, k1, k2)
- if not os.path.exists(output_dir):
+
+def get_hashed_dir(cfile):
+ hashed_fn = _get_hashed_filename(cfile)
+ try:
+ with open(hashed_fn) as f:
+ dirname = f.read(1024)
+ except IOError:
+ dirname = ''
+ tmpdir = tempfile.gettempdir()
+ if (not dirname or '/' in dirname or '\\' in dirname or '\x00' in dirname
+ or not os.path.isdir(os.path.join(tmpdir, dirname))):
+ dirname = binascii.hexlify(os.urandom(8))
+ if not isinstance(dirname, str): # Python 3
+ dirname = dirname.decode('ascii')
+ dirname = 'testcapi_' + dirname
+ output_dir = os.path.join(tmpdir, dirname)
+ try:
os.mkdir(output_dir)
+ except OSError:
+ pass
return output_dir
@@ -34,13 +52,12 @@
return ext
-def compile_shared(csource, modulename, output_dir=None):
+def compile_shared(csource, modulename, output_dir):
"""Compile '_testcapi.c' or '_ctypes_test.c' into an extension module,
and import it.
"""
thisdir = os.path.dirname(__file__)
- if output_dir is None:
- output_dir = tempfile.mkdtemp()
+ assert output_dir is not None
from distutils.ccompiler import new_compiler
@@ -85,4 +102,16 @@
# Now import the newly created library, it will replace the original
# module in sys.modules
fp, filename, description = imp.find_module(modulename, path=[output_dir])
- imp.load_module(modulename, fp, filename, description)
+ with fp:
+ imp.load_module(modulename, fp, filename, description)
+
+ # If everything went fine up to now, write the name of this new
+ # directory to 'hashed_fn', for future processes (and to avoid a
+ # growing number of temporary directories that are not completely
+ # obvious to clean up on Windows)
+ hashed_fn = _get_hashed_filename(os.path.join(thisdir, csource))
+ try:
+ with open(hashed_fn, 'w') as f:
+ f.write(os.path.basename(output_dir))
+ except IOError:
+ pass
diff --git a/lib_pypy/_sqlite3.py b/lib_pypy/_sqlite3.py
--- a/lib_pypy/_sqlite3.py
+++ b/lib_pypy/_sqlite3.py
@@ -153,9 +153,10 @@
factory = Connection if not factory else factory
# an sqlite3 db seems to be around 100 KiB at least (doesn't matter if
# backed by :memory: or a file)
+ res = factory(database, timeout, detect_types, isolation_level,
+ check_same_thread, factory, cached_statements)
add_memory_pressure(100 * 1024)
- return factory(database, timeout, detect_types, isolation_level,
- check_same_thread, factory, cached_statements)
+ return res
def _unicode_text_factory(x):
diff --git a/lib_pypy/cffi.egg-info/PKG-INFO b/lib_pypy/cffi.egg-info/PKG-INFO
--- a/lib_pypy/cffi.egg-info/PKG-INFO
+++ b/lib_pypy/cffi.egg-info/PKG-INFO
@@ -1,6 +1,6 @@
Metadata-Version: 1.1
Name: cffi
-Version: 1.11.4
+Version: 1.11.5
Summary: Foreign Function Interface for Python calling C code.
Home-page: http://cffi.readthedocs.org
Author: Armin Rigo, Maciej Fijalkowski
diff --git a/lib_pypy/cffi/__init__.py b/lib_pypy/cffi/__init__.py
--- a/lib_pypy/cffi/__init__.py
+++ b/lib_pypy/cffi/__init__.py
@@ -4,8 +4,8 @@
from .api import FFI
from .error import CDefError, FFIError, VerificationError, VerificationMissing
-__version__ = "1.11.4"
-__version_info__ = (1, 11, 4)
+__version__ = "1.11.5"
+__version_info__ = (1, 11, 5)
# The verifier module file names are based on the CRC32 of a string that
# contains the following version number. It may be older than __version__
diff --git a/lib_pypy/cffi/_embedding.h b/lib_pypy/cffi/_embedding.h
--- a/lib_pypy/cffi/_embedding.h
+++ b/lib_pypy/cffi/_embedding.h
@@ -146,32 +146,6 @@
PyGILState_STATE state;
PyObject *pycode=NULL, *global_dict=NULL, *x;
-#if PY_MAJOR_VERSION >= 3
- /* see comments in _cffi_carefully_make_gil() about the
- Python2/Python3 difference
- */
-#else
- /* Acquire the GIL. We have no threadstate here. If Python is
- already initialized, it is possible that there is already one
- existing for this thread, but it is not made current now.
- */
- PyEval_AcquireLock();
-
- _cffi_py_initialize();
-
- /* The Py_InitializeEx() sometimes made a threadstate for us, but
- not always. Indeed Py_InitializeEx() could be called and do
- nothing. So do we have a threadstate, or not? We don't know,
- but we can replace it with NULL in all cases.
- */
- (void)PyThreadState_Swap(NULL);
-
- /* Now we can release the GIL and re-acquire immediately using the
- logic of PyGILState(), which handles making or installing the
- correct threadstate.
- */
- PyEval_ReleaseLock();
-#endif
state = PyGILState_Ensure();
/* Call the initxxx() function from the present module. It will
@@ -247,7 +221,7 @@
if (f != NULL && f != Py_None) {
PyFile_WriteString("\nFrom: " _CFFI_MODULE_NAME
- "\ncompiled with cffi version: 1.11.4"
+ "\ncompiled with cffi version: 1.11.5"
"\n_cffi_backend module: ", f);
modules = PyImport_GetModuleDict();
mod = PyDict_GetItemString(modules, "_cffi_backend");
@@ -278,16 +252,14 @@
that we don't hold the GIL before (if it exists), and we don't
hold it afterwards.
- What it really does is completely different in Python 2 and
- Python 3.
+ (What it really does used to be completely different in Python 2
+ and Python 3, with the Python 2 solution avoiding the spin-lock
+ around the Py_InitializeEx() call. However, after recent changes
+ to CPython 2.7 (issue #358) it no longer works. So we use the
+ Python 3 solution everywhere.)
- Python 2
- ========
-
- Initialize the GIL, without initializing the rest of Python,
- by calling PyEval_InitThreads().
-
- PyEval_InitThreads() must not be called concurrently at all.
+ This initializes Python by calling Py_InitializeEx().
+ Important: this must not be called concurrently at all.
So we use a global variable as a simple spin lock. This global
variable must be from 'libpythonX.Y.so', not from this
cffi-based extension module, because it must be shared from
@@ -297,18 +269,6 @@
string "ENDMARKER". We change it temporarily to point to the
next character in that string. (Yes, I know it's REALLY
obscure.)
-
- Python 3
- ========
-
- In Python 3, PyEval_InitThreads() cannot be called before
- Py_InitializeEx() any more. So this function calls
- Py_InitializeEx() first. It uses the same obscure logic to
- make sure we never call it concurrently.
-
- Arguably, this is less good on the spinlock, because
- Py_InitializeEx() takes much longer to run than
- PyEval_InitThreads(). But I didn't find a way around it.
*/
#ifdef WITH_THREAD
@@ -332,8 +292,7 @@
}
#endif
-#if PY_MAJOR_VERSION >= 3
- /* Python 3: call Py_InitializeEx() */
+ /* call Py_InitializeEx() */
{
PyGILState_STATE state = PyGILState_UNLOCKED;
if (!Py_IsInitialized())
@@ -344,17 +303,6 @@
PyEval_InitThreads();
PyGILState_Release(state);
}
-#else
- /* Python 2: call PyEval_InitThreads() */
-# ifdef WITH_THREAD
- if (!PyEval_ThreadsInitialized()) {
- PyEval_InitThreads(); /* makes the GIL */
- PyEval_ReleaseLock(); /* then release it */
- }
- /* else: there is already a GIL, but we still needed to do the
- spinlock dance to make sure that we see it as fully ready */
-# endif
-#endif
#ifdef WITH_THREAD
/* release the lock */
diff --git a/lib_pypy/cffi/api.py b/lib_pypy/cffi/api.py
--- a/lib_pypy/cffi/api.py
+++ b/lib_pypy/cffi/api.py
@@ -143,6 +143,13 @@
self._libraries.append(lib)
return lib
+ def dlclose(self, lib):
+ """Close a library obtained with ffi.dlopen(). After this call,
+ access to functions or variables from the library will fail
+ (possibly with a segmentation fault).
+ """
+ type(lib).__cffi_close__(lib)
+
def _typeof_locked(self, cdecl):
# call me with the lock!
key = cdecl
@@ -898,6 +905,9 @@
return addressof_var(name)
raise AttributeError("cffi library has no function or "
"global variable named '%s'" % (name,))
+ def __cffi_close__(self):
+ backendlib.close_lib()
+ self.__dict__.clear()
#
if libname is not None:
try:
diff --git a/lib_pypy/cffi/model.py b/lib_pypy/cffi/model.py
--- a/lib_pypy/cffi/model.py
+++ b/lib_pypy/cffi/model.py
@@ -352,21 +352,20 @@
self.fldquals = fldquals
self.build_c_name_with_marker()
- def has_anonymous_struct_fields(self):
- if self.fldtypes is None:
- return False
- for name, type in zip(self.fldnames, self.fldtypes):
- if name == '' and isinstance(type, StructOrUnion):
- return True
- return False
+ def anonymous_struct_fields(self):
+ if self.fldtypes is not None:
+ for name, type in zip(self.fldnames, self.fldtypes):
+ if name == '' and isinstance(type, StructOrUnion):
+ yield type
- def enumfields(self):
+ def enumfields(self, expand_anonymous_struct_union=True):
fldquals = self.fldquals
if fldquals is None:
fldquals = (0,) * len(self.fldnames)
for name, type, bitsize, quals in zip(self.fldnames, self.fldtypes,
self.fldbitsize, fldquals):
- if name == '' and isinstance(type, StructOrUnion):
+ if (name == '' and isinstance(type, StructOrUnion)
+ and expand_anonymous_struct_union):
# nested anonymous struct/union
for result in type.enumfields():
yield result
diff --git a/lib_pypy/cffi/recompiler.py b/lib_pypy/cffi/recompiler.py
--- a/lib_pypy/cffi/recompiler.py
+++ b/lib_pypy/cffi/recompiler.py
@@ -836,6 +836,10 @@
def _struct_collecttype(self, tp):
self._do_collect_type(tp)
+ if self.target_is_python:
+ # also requires nested anon struct/unions in ABI mode, recursively
+ for fldtype in tp.anonymous_struct_fields():
+ self._struct_collecttype(fldtype)
def _struct_decl(self, tp, cname, approxname):
if tp.fldtypes is None:
@@ -884,7 +888,7 @@
named_ptr not in self.ffi._parser._included_declarations)):
if tp.fldtypes is None:
pass # opaque
- elif tp.partial or tp.has_anonymous_struct_fields():
+ elif tp.partial or any(tp.anonymous_struct_fields()):
pass # field layout obtained silently from the C compiler
else:
flags.append("_CFFI_F_CHECK_FIELDS")
@@ -896,7 +900,8 @@
flags = '|'.join(flags) or '0'
c_fields = []
if reason_for_not_expanding is None:
- enumfields = list(tp.enumfields())
+ expand_anonymous_struct_union = not self.target_is_python
+ enumfields = list(tp.enumfields(expand_anonymous_struct_union))
for fldname, fldtype, fbitsize, fqual in enumfields:
fldtype = self._field_type(tp, fldname, fldtype)
self._check_not_opaque(fldtype,
diff --git a/lib_pypy/cffi/setuptools_ext.py b/lib_pypy/cffi/setuptools_ext.py
--- a/lib_pypy/cffi/setuptools_ext.py
+++ b/lib_pypy/cffi/setuptools_ext.py
@@ -81,8 +81,13 @@
it doesn't so far, creating troubles. That's why we check
for "not hasattr(sys, 'gettotalrefcount')" (the 2.7 compatible equivalent
of 'd' not in sys.abiflags). (http://bugs.python.org/issue28401)
+
+ On Windows, it's better not to use py_limited_api until issue #355
+ can be resolved (by having virtualenv copy PYTHON3.DLL). See also
+ the start of _cffi_include.h.
"""
- if 'py_limited_api' not in kwds and not hasattr(sys, 'gettotalrefcount'):
+ if ('py_limited_api' not in kwds and not hasattr(sys, 'gettotalrefcount')
+ and sys.platform != 'win32'):
import setuptools
try:
setuptools_major_version =
int(setuptools.__version__.partition('.')[0])
@@ -143,8 +148,8 @@
def _add_py_module(dist, ffi, module_name):
from distutils.dir_util import mkpath
- from distutils.command.build_py import build_py
- from distutils.command.build_ext import build_ext
+ from setuptools.command.build_py import build_py
+ from setuptools.command.build_ext import build_ext
from distutils import log
from cffi import recompiler
@@ -164,6 +169,17 @@
generate_mod(os.path.join(self.build_lib, *module_path))
dist.cmdclass['build_py'] = build_py_make_mod
+ # distutils and setuptools have no notion I could find of a
+ # generated python module. If we don't add module_name to
+ # dist.py_modules, then things mostly work but there are some
+ # combination of options (--root and --record) that will miss
+ # the module. So we add it here, which gives a few apparently
+ # harmless warnings about not finding the file outside the
+ # build directory.
+ if dist.py_modules is None:
+ dist.py_modules = []
+ dist.py_modules.append(module_name)
+
# the following is only for "build_ext -i"
base_class_2 = dist.cmdclass.get('build_ext', build_ext)
class build_ext_make_mod(base_class_2):
diff --git a/lib_pypy/datetime.py b/lib_pypy/datetime.py
--- a/lib_pypy/datetime.py
+++ b/lib_pypy/datetime.py
@@ -1415,9 +1415,14 @@
self.__setstate(year, month)
self._hashcode = -1
return self
- year, month, day = _check_date_fields(year, month, day)
- hour, minute, second, microsecond = _check_time_fields(
- hour, minute, second, microsecond)
+ elif isinstance(year, tuple) and len(year) == 7:
+ # Used by internal functions where the arguments are guaranteed to
+ # be valid.
+ year, month, day, hour, minute, second, microsecond = year
+ else:
+ year, month, day = _check_date_fields(year, month, day)
+ hour, minute, second, microsecond = _check_time_fields(
+ hour, minute, second, microsecond)
_check_tzinfo_arg(tzinfo)
self = dateinterop.__new__(cls)
self._year = year
@@ -1491,7 +1496,7 @@
us = 0
y, m, d, hh, mm, ss, weekday, jday, dst = converter(timestamp)
ss = min(ss, 59) # clamp out leap seconds if the platform has them
- return cls(y, m, d, hh, mm, ss, us, tzinfo)
+ return cls((y, m, d, hh, mm, ss, us), tzinfo=tzinfo)
@classmethod
def now(cls, tz=None):
@@ -1800,7 +1805,7 @@
return diff and 1 or 0
def _add_timedelta(self, other, factor):
- y, m, d, hh, mm, ss, us = _normalize_datetime(
+ result = _normalize_datetime(
self._year,
self._month,
self._day + other.days * factor,
@@ -1808,7 +1813,7 @@
self._minute,
self._second + other.seconds * factor,
self._microsecond + other.microseconds * factor)
- return datetime(y, m, d, hh, mm, ss, us, tzinfo=self._tzinfo)
+ return datetime(result, tzinfo=self._tzinfo)
def __add__(self, other):
"Add a datetime and a timedelta."
diff --git a/lib_pypy/dbm.py b/lib_pypy/dbm.py
--- a/lib_pypy/dbm.py
+++ b/lib_pypy/dbm.py
@@ -157,7 +157,14 @@
def open(filename, flag='r', mode=0666):
"open a DBM database"
if not isinstance(filename, str):
- raise TypeError("expected string")
+ if sys.version_info < (3,) and isinstance(filename, unicode):
+ # unlike CPython we'll encode 'filename' with filesystemencoding
+ # instead of defaultencoding, because that seems like a far
+ # better idea. But I'm also open for saying that we should
+ # rather go for bug-to-bug compatibility instead.
+ filename = filename.encode(sys.getfilesystemencoding())
+ else:
+ raise TypeError("expected string")
openflag = 0
diff --git a/lib_pypy/greenlet.egg-info b/lib_pypy/greenlet.egg-info
--- a/lib_pypy/greenlet.egg-info
+++ b/lib_pypy/greenlet.egg-info
@@ -1,6 +1,6 @@
Metadata-Version: 1.0
Name: greenlet
-Version: 0.4.12
+Version: 0.4.13
Summary: Lightweight in-process concurrent programming
Home-page: https://github.com/python-greenlet/greenlet
Author: Ralf Schmitt (for CPython), PyPy team
diff --git a/lib_pypy/greenlet.py b/lib_pypy/greenlet.py
--- a/lib_pypy/greenlet.py
+++ b/lib_pypy/greenlet.py
@@ -1,7 +1,7 @@
import sys
import _continuation
-__version__ = "0.4.12"
+__version__ = "0.4.13"
# ____________________________________________________________
# Exceptions
diff --git a/pypy/doc/contributor.rst b/pypy/doc/contributor.rst
--- a/pypy/doc/contributor.rst
+++ b/pypy/doc/contributor.rst
@@ -56,13 +56,13 @@
Niko Matsakis
Alexander Hesse
Ludovic Aubry
+ stian
Jacob Hallen
Jason Creighton
Mark Young
Alex Martelli
Spenser Bauman
Michal Bendowski
- stian
Jan de Mooij
Tyler Wade
Vincent Legoll
@@ -90,10 +90,10 @@
Wenzhu Man
Konstantin Lopuhin
John Witulski
+ Jeremy Thurgood
Greg Price
Ivan Sichmann Freitas
Dario Bertini
- Jeremy Thurgood
Mark Pearse
Simon Cross
Tobias Pape
@@ -112,18 +112,19 @@
Adrian Kuhn
tav
Georg Brandl
+ Joannah Nanjekye
Bert Freudenberg
Stian Andreassen
Wanja Saatkamp
Mike Blume
- Joannah Nanjekye
Gerald Klix
Oscar Nierstrasz
Rami Chowdhury
Stefan H. Muller
+ Dodan Mihai
Tim Felgentreff
Eugene Oden
- Dodan Mihai
+ Colin Valliant
Jeff Terrace
Henry Mason
Vasily Kuznetsov
@@ -192,12 +193,14 @@
Vaibhav Sood
Reuben Cummings
Attila Gobi
+ Floris Bruynooghe
Christopher Pope
Tristan Arthur
Christian Tismer
Dan Stromberg
Carl Meyer
Florin Papa
+ Arianna Avanzini
Jens-Uwe Mager
Valentina Mukhamedzhanova
Stefano Parmesan
@@ -221,6 +224,7 @@
Mark Williams
Kunal Grover
Nathan Taylor
+ Barry Hart
Travis Francis Athougies
Yasir Suhail
Sergey Kishchenko
@@ -228,6 +232,7 @@
Lutz Paelike
Ian Foote
Philipp Rustemeuer
+ Logan Chien
Catalin Gabriel Manciu
Jacob Oscarson
Ryan Gonzalez
@@ -263,7 +268,6 @@
Akira Li
Gustavo Niemeyer
Rafał Gałczyński
- Logan Chien
Lucas Stadler
roberto@goyle
Matt Bogosian
@@ -276,6 +280,7 @@
Anna Katrina Dominguez
Kim Jin Su
Amber Brown
+ Miro Hrončok
Anthony Sottile
Nate Bragg
Ben Darnell
@@ -283,7 +288,6 @@
Godefroid Chappelle
Julian Berman
Michael Hudson-Doyle
- Floris Bruynooghe
Stephan Busemann
Dan Colish
timo
@@ -325,6 +329,7 @@
Michael Chermside
Anna Ravencroft
remarkablerocket
+ Pauli Virtanen
Petre Vijiac
Berker Peksag
Christian Muirhead
@@ -349,6 +354,7 @@
Graham Markall
Dan Loewenherz
werat
+ Andrew Stepanov
Niclas Olofsson
Chris Pressey
Tobias Diaz
diff --git a/pypy/doc/cpython_differences.rst b/pypy/doc/cpython_differences.rst
--- a/pypy/doc/cpython_differences.rst
+++ b/pypy/doc/cpython_differences.rst
@@ -10,89 +10,6 @@
PyPy.
-.. _extension-modules:
-
-Extension modules
------------------
-
-List of extension modules that we support:
-
-* Supported as built-in modules (in :source:`pypy/module/`):
-
- __builtin__
- :doc:`__pypy__ <__pypy__-module>`
- _ast
- _codecs
- _collections
- :doc:`_continuation <stackless>`
- :doc:`_ffi <discussion/ctypes-implementation>`
- _hashlib
- _io
- _locale
- _lsprof
- _md5
- :doc:`_minimal_curses <config/objspace.usemodules._minimal_curses>`
- _multiprocessing
- _random
- :doc:`_rawffi <discussion/ctypes-implementation>`
- _sha
- _socket
- _sre
- _ssl
- _warnings
- _weakref
- _winreg
- array
- binascii
- bz2
- cStringIO
- cmath
- `cpyext`_
- crypt
- errno
- exceptions
- fcntl
- gc
- imp
- itertools
- marshal
- math
- mmap
- operator
- parser
- posix
- pyexpat
- select
- signal
- struct
- symbol
- sys
- termios
- thread
- time
- token
- unicodedata
- zipimport
- zlib
-
- When translated on Windows, a few Unix-only modules are skipped,
- and the following module is built instead:
-
- _winreg
-
-* Supported by being rewritten in pure Python (possibly using ``cffi``):
- see the :source:`lib_pypy/` directory. Examples of modules that we
- support this way: ``ctypes``, ``cPickle``, ``cmath``, ``dbm``,
``datetime``...
- Note that some modules are both in there and in the list above;
- by default, the built-in module is used (but can be disabled
- at translation time).
-
-The extension modules (i.e. modules written in C, in the standard CPython)
-that are neither mentioned above nor in :source:`lib_pypy/` are not available
in PyPy.
-(You may have a chance to use them anyway with `cpyext`_.)
-
-.. _cpyext:
http://morepypy.blogspot.com/2010/04/using-cpython-extension-modules-with.html
-
Differences related to garbage collection strategies
----------------------------------------------------
@@ -554,7 +471,101 @@
versions of PyPy may have to rename the arguments if CPython starts
accepting them too.
+* PyPy3: ``distutils`` has been enhanced to allow finding ``VsDevCmd.bat`` in
the
+ directory pointed to by the ``VS%0.f0COMNTOOLS`` (typically
``VS140COMNTOOLS``)
+ environment variable. CPython searches for ``vcvarsall.bat`` somewhere
**above**
+ that value.
+
+* SyntaxError_ s try harder to give details about the cause of the failure, so
+ the error messages are not the same as in CPython
+
+
+.. _extension-modules:
+
+Extension modules
+-----------------
+
+List of extension modules that we support:
+
+* Supported as built-in modules (in :source:`pypy/module/`):
+
+ __builtin__
+ :doc:`__pypy__ <__pypy__-module>`
+ _ast
+ _codecs
+ _collections
+ :doc:`_continuation <stackless>`
+ :doc:`_ffi <discussion/ctypes-implementation>`
+ _hashlib
+ _io
+ _locale
+ _lsprof
+ _md5
+ :doc:`_minimal_curses <config/objspace.usemodules._minimal_curses>`
+ _multiprocessing
+ _random
+ :doc:`_rawffi <discussion/ctypes-implementation>`
+ _sha
+ _socket
+ _sre
+ _ssl
+ _warnings
+ _weakref
+ _winreg
+ array
+ binascii
+ bz2
+ cStringIO
+ cmath
+ `cpyext`_
+ crypt
+ errno
+ exceptions
+ fcntl
+ gc
+ imp
+ itertools
+ marshal
+ math
+ mmap
+ operator
+ parser
+ posix
+ pyexpat
+ select
+ signal
+ struct
+ symbol
+ sys
+ termios
+ thread
+ time
+ token
+ unicodedata
+ zipimport
+ zlib
+
+ When translated on Windows, a few Unix-only modules are skipped,
+ and the following module is built instead:
+
+ _winreg
+
+* Supported by being rewritten in pure Python (possibly using ``cffi``):
+ see the :source:`lib_pypy/` directory. Examples of modules that we
+ support this way: ``ctypes``, ``cPickle``, ``cmath``, ``dbm``,
``datetime``...
+ Note that some modules are both in there and in the list above;
+ by default, the built-in module is used (but can be disabled
+ at translation time).
+
+The extension modules (i.e. modules written in C, in the standard CPython)
+that are neither mentioned above nor in :source:`lib_pypy/` are not available
in PyPy.
+(You may have a chance to use them anyway with `cpyext`_.)
+
+.. _cpyext:
http://morepypy.blogspot.com/2010/04/using-cpython-extension-modules-with.html
+
+
.. _`is ignored in PyPy`: http://bugs.python.org/issue14621
.. _`little point`:
http://events.ccc.de/congress/2012/Fahrplan/events/5152.en.html
.. _`#2072`: https://bitbucket.org/pypy/pypy/issue/2072/
.. _`issue #2653`: https://bitbucket.org/pypy/pypy/issues/2653/
+.. _SyntaxError:
https://morepypy.blogspot.co.il/2018/04/improving-syntaxerror-in-pypy.html
diff --git a/pypy/doc/gc_info.rst b/pypy/doc/gc_info.rst
--- a/pypy/doc/gc_info.rst
+++ b/pypy/doc/gc_info.rst
@@ -1,17 +1,297 @@
-Garbage collector configuration
-===============================
+Garbage collector documentation and configuration
+=================================================
+
+Incminimark
+-----------
+
+PyPy's default garbage collector is called incminimark - it's an incremental,
+generational moving collector. Here we hope to explain a bit how it works
+and how it can be tuned to suit the workload.
+
+Incminimark first allocates objects in so called *nursery* - place for young
+objects, where allocation is very cheap, being just a pointer bump. The nursery
+size is a very crucial variable - depending on your workload (one or many
+processes) and cache sizes you might want to experiment with it via
+*PYPY_GC_NURSERY* environment variable. When the nursery is full, there is
+performed a minor collection. Freed objects are no longer referencable and
+just die, just by not being referenced any more; on the other hand, objects
+found to still be alive must survive and are copied from the nursery
+to the old generation. Either to arenas, which are collections
+of objects of the same size, or directly allocated with malloc if they're
+larger. (A third category, the very large objects, are initially allocated
+outside the nursery and never move.)
+
+Since Incminimark is an incremental GC, the major collection is incremental,
+meaning there should not be any pauses longer than 1ms.
+
+
+Fragmentation
+-------------
+
+Before we discuss issues of "fragmentation", we need a bit of precision.
+There are two kinds of related but distinct issues:
+
+* If the program allocates a lot of memory, and then frees it all by
+ dropping all references to it, then we might expect to see the RSS
+ to drop. (RSS = Resident Set Size on Linux, as seen by "top"; it is an
+ approximation of the actual memory usage from the OS's point of view.)
+ This might not occur: the RSS may remain at its highest value. This
+ issue is more precisely caused by the process not returning "free"
+ memory to the OS. We call this case "unreturned memory".
+
+* After doing the above, if the RSS didn't go down, then at least future
+ allocations should not cause the RSS to grow more. That is, the process
+ should reuse unreturned memory as long as it has got some left. If this
+ does not occur, the RSS grows even larger and we have real fragmentation
+ issues.
+
+
+gc.get_stats
+------------
+
+There is a special function in the ``gc`` module called
+``get_stats(memory_pressure=False)``.
+
+``memory_pressure`` controls whether or not to report memory pressure from
+objects allocated outside of the GC, which requires walking the entire heap,
+so it's disabled by default due to its cost. Enable it when debugging
+mysterious memory disappearance.
+
+Example call looks like that::
+
+ >>> gc.get_stats(True)
+ Total memory consumed:
+ GC used: 4.2MB (peak: 4.2MB)
+ in arenas: 763.7kB
+ rawmalloced: 383.1kB
+ nursery: 3.1MB
+ raw assembler used: 0.0kB
+ memory pressure: 0.0kB
+ -----------------------------
+ Total: 4.2MB
+
+ Total memory allocated:
+ GC allocated: 4.5MB (peak: 4.5MB)
+ in arenas: 763.7kB
+ rawmalloced: 383.1kB
+ nursery: 3.1MB
+ raw assembler allocated: 0.0kB
+ memory pressure: 0.0kB
+ -----------------------------
+ Total: 4.5MB
+
+In this particular case, which is just at startup, GC consumes relatively
+little memory and there is even less unused, but allocated memory. In case
+there is a lot of unreturned memory or actual fragmentation, the "allocated"
+can be much higher than "used". Generally speaking, "peak" will more closely
+resemble the actual memory consumed as reported by RSS. Indeed, returning
+memory to the OS is a hard and not solved problem. In PyPy, it occurs only if
+an arena is entirely free---a contiguous block of 64 pages of 4 or 8 KB each.
+It is also rare for the "rawmalloced" category, at least for common system
+implementations of ``malloc()``.
+
+The details of various fields:
+
+* GC in arenas - small old objects held in arenas. If the amount "allocated"
+ is much higher than the amount "used", we have unreturned memory. It is
+ possible but unlikely that we have internal fragmentation here. However,
+ this unreturned memory cannot be reused for any ``malloc()``, including the
+ memory from the "rawmalloced" section.
+
+* GC rawmalloced - large objects allocated with malloc. This is gives the
+ current (first block of text) and peak (second block of text) memory
+ allocated with ``malloc()``. The amount of unreturned memory or
+ fragmentation caused by ``malloc()`` cannot easily be reported. Usually
+ you can guess there is some if the RSS is much larger than the total
+ memory reported for "GC allocated", but do keep in mind that this total
+ does not include malloc'ed memory not known to PyPy's GC at all. If you
+ guess there is some, consider using `jemalloc`_ as opposed to system malloc.
+
+.. _`jemalloc`: http://jemalloc.net/
+
+* nursery - amount of memory allocated for nursery, fixed at startup,
+ controlled via an environment variable
+
+* raw assembler allocated - amount of assembler memory that JIT feels
+ responsible for
+
+* memory pressure, if asked for - amount of memory we think got allocated
+ via external malloc (eg loading cert store in SSL contexts) that is kept
+ alive by GC objects, but not accounted in the GC
+
+
+GC Hooks
+--------
+
+GC hooks are user-defined functions which are called whenever a specific GC
+event occur, and can be used to monitor GC activity and pauses. You can
+install the hooks by setting the following attributes:
+
+``gc.hook.on_gc_minor``
+ Called whenever a minor collection occurs. It corresponds to
+ ``gc-minor`` sections inside ``PYPYLOG``.
+
+``gc.hook.on_gc_collect_step``
+ Called whenever an incremental step of a major collection occurs. It
+ corresponds to ``gc-collect-step`` sections inside ``PYPYLOG``.
+
+``gc.hook.on_gc_collect``
+ Called after the last incremental step, when a major collection is fully
+ done. It corresponds to ``gc-collect-done`` sections inside ``PYPYLOG``.
+
+To uninstall a hook, simply set the corresponding attribute to ``None``. To
+install all hooks at once, you can call ``gc.hooks.set(obj)``, which will look
+for methods ``on_gc_*`` on ``obj``. To uninstall all the hooks at once, you
+can call ``gc.hooks.reset()``.
+
+The functions called by the hooks receive a single ``stats`` argument, which
+contains various statistics about the event.
+
+Note that PyPy cannot call the hooks immediately after a GC event, but it has
+to wait until it reaches a point in which the interpreter is in a known state
+and calling user-defined code is harmless. It might happen that multiple
+events occur before the hook is invoked: in this case, you can inspect the
+value ``stats.count`` to know how many times the event occured since the last
+time the hook was called. Similarly, ``stats.duration`` contains the
+**total** time spent by the GC for this specific event since the last time the
+hook was called.
+
+On the other hand, all the other fields of the ``stats`` object are relative
+only to the **last** event of the series.
+
+The attributes for ``GcMinorStats`` are:
+
+``count``
+ The number of minor collections occured since the last hook call.
+
+``duration``
+ The total time spent inside minor collections since the last hook
+ call. See below for more information on the unit.
+
+``duration_min``
+ The duration of the fastest minor collection since the last hook call.
+
+``duration_max``
+ The duration of the slowest minor collection since the last hook call.
+
+ ``total_memory_used``
+ The amount of memory used at the end of the minor collection, in
+ bytes. This include the memory used in arenas (for GC-managed memory) and
+ raw-malloced memory (e.g., the content of numpy arrays).
+
+``pinned_objects``
+ the number of pinned objects.
+
+
+The attributes for ``GcCollectStepStats`` are:
+
+``count``, ``duration``, ``duration_min``, ``duration_max``
+ See above.
+
+``oldstate``, ``newstate``
+ Integers which indicate the state of the GC before and after the step.
+
+The value of ``oldstate`` and ``newstate`` is one of these constants, defined
+inside ``gc.GcCollectStepStats``: ``STATE_SCANNING``, ``STATE_MARKING``,
+``STATE_SWEEPING``, ``STATE_FINALIZING``. It is possible to get a string
+representation of it by indexing the ``GC_STATS`` tuple.
+
+
+The attributes for ``GcCollectStats`` are:
+
+``count``
+ See above.
+
+``num_major_collects``
+ The total number of major collections which have been done since the
+ start. Contrarily to ``count``, this is an always-growing counter and it's
+ not reset between invocations.
+
+``arenas_count_before``, ``arenas_count_after``
+ Number of arenas used before and after the major collection.
+
+``arenas_bytes``
+ Total number of bytes used by GC-managed objects.
+
+``rawmalloc_bytes_before``, ``rawmalloc_bytes_after``
+ Total number of bytes used by raw-malloced objects, before and after the
+ major collection.
+
+Note that ``GcCollectStats`` has **not** got a ``duration`` field. This is
+because all the GC work is done inside ``gc-collect-step``:
+``gc-collect-done`` is used only to give additional stats, but doesn't do any
+actual work.
+
+A note about the ``duration`` field: depending on the architecture and
+operating system, PyPy uses different ways to read timestamps, so ``duration``
+is expressed in varying units. It is possible to know which by calling
+``__pypy__.debug_get_timestamp_unit()``, which can be one of the following
+values:
+
+``tsc``
+ The default on ``x86`` machines: timestamps are expressed in CPU ticks, as
+ read by the `Time Stamp Counter`_.
+
+``ns``
+ Timestamps are expressed in nanoseconds.
+
+``QueryPerformanceCounter``
+ On Windows, in case for some reason ``tsc`` is not available: timestamps
+ are read using the win API ``QueryPerformanceCounter()``.
+
+
+Unfortunately, there does not seem to be a reliable standard way for
+converting ``tsc`` ticks into nanoseconds, although in practice on modern CPUs
+it is enough to divide the ticks by the maximum nominal frequency of the CPU.
+For this reason, PyPy gives the raw value, and leaves the job of doing the
+conversion to external libraries.
+
+Here is an example of GC hooks in use::
+
+ import sys
+ import gc
+
+ class MyHooks(object):
+ done = False
+
+ def on_gc_minor(self, stats):
+ print 'gc-minor: count = %02d, duration = %d' %
(stats.count,
+
stats.duration)
+
+ def on_gc_collect_step(self, stats):
+ old = gc.GcCollectStepStats.GC_STATES[stats.oldstate]
+ new = gc.GcCollectStepStats.GC_STATES[stats.newstate]
+ print 'gc-collect-step: %s --> %s' % (old, new)
+ print ' count = %02d, duration = %d' %
(stats.count,
+
stats.duration)
+
+ def on_gc_collect(self, stats):
+ print 'gc-collect-done: count = %02d' % stats.count
+ self.done = True
+
+ hooks = MyHooks()
+ gc.hooks.set(hooks)
+
+ # simulate some GC activity
+ lst = []
+ while not hooks.done:
+ lst = [lst, 1, 2, 3]
+
+
+.. _`Time Stamp Counter`: https://en.wikipedia.org/wiki/Time_Stamp_Counter
+
.. _minimark-environment-variables:
-Minimark
---------
+Environment variables
+---------------------
PyPy's default ``incminimark`` garbage collector is configurable through
several environment variables:
``PYPY_GC_NURSERY``
The nursery size.
- Defaults to 1/2 of your cache or ``4M``.
+ Defaults to 1/2 of your last-level cache, or ``4M`` if unknown.
Small values (like 1 or 1KB) are useful for debugging.
``PYPY_GC_NURSERY_DEBUG``
diff --git a/pypy/doc/how-to-release.rst b/pypy/doc/how-to-release.rst
--- a/pypy/doc/how-to-release.rst
+++ b/pypy/doc/how-to-release.rst
@@ -40,6 +40,8 @@
sure things are ported back to the trunk and to the branch as
necessary.
+* Make sure the RPython builds on the buildbot pass with no failures
+
* Maybe bump the SOABI number in module/imp/importing. This has many
implications, so make sure the PyPy community agrees to the change.
diff --git a/pypy/doc/index-of-release-notes.rst
b/pypy/doc/index-of-release-notes.rst
--- a/pypy/doc/index-of-release-notes.rst
+++ b/pypy/doc/index-of-release-notes.rst
@@ -6,6 +6,7 @@
.. toctree::
+ release-v6.0.0.rst
release-v5.10.1.rst
release-v5.10.0.rst
release-v5.9.0.rst
diff --git a/pypy/doc/index-of-whatsnew.rst b/pypy/doc/index-of-whatsnew.rst
--- a/pypy/doc/index-of-whatsnew.rst
+++ b/pypy/doc/index-of-whatsnew.rst
@@ -7,6 +7,8 @@
.. toctree::
whatsnew-head.rst
+ whatsnew-pypy2-6.0.0.rst
+ whatsnew-pypy2-5.10.0.rst
whatsnew-pypy2-5.10.0.rst
whatsnew-pypy2-5.9.0.rst
whatsnew-pypy2-5.8.0.rst
diff --git a/pypy/doc/install.rst b/pypy/doc/install.rst
--- a/pypy/doc/install.rst
+++ b/pypy/doc/install.rst
@@ -17,13 +17,18 @@
~~~~~~~~~~~~~~~~~~~~~~~~~
The quickest way to start using PyPy is to download a prebuilt binary for your
-OS and architecture. You can either use the `most recent release`_ or one of
-our `development nightly build`_. Please note that the nightly builds are not
+OS and architecture. You may be able to use either use the
+`most recent release`_ or one of our `development nightly build`_. These
+builds depend on dynamically linked libraries that may not be available on your
+OS. See the section about `Linux binaries` for more info and alternatives that
+may work on your system.
+
+Please note that the nightly builds are not
guaranteed to be as stable as official releases, use them at your own risk.
.. _most recent release: http://pypy.org/download.html
.. _development nightly build: http://buildbot.pypy.org/nightly/trunk/
-
+.. _Linux binaries:
http://pypy.org/download.html#linux-binaries-and-common-distributions
Installing PyPy
~~~~~~~~~~~~~~~
@@ -69,9 +74,9 @@
~~~~~~~~~~~~~~~~~~~~~~~~~~~
It is often convenient to run pypy inside a virtualenv. To do this
-you need a recent version of virtualenv -- 1.6.1 or greater. You can
+you need a version of virtualenv -- 1.6.1 or greater. You can
then install PyPy both from a precompiled tarball or from a mercurial
-checkout::
+checkout after translation::
# from a tarball
$ virtualenv -p /opt/pypy-xxx/bin/pypy my-pypy-env
diff --git a/pypy/doc/project-ideas.rst b/pypy/doc/project-ideas.rst
--- a/pypy/doc/project-ideas.rst
+++ b/pypy/doc/project-ideas.rst
@@ -1,26 +1,41 @@
Potential Project List
======================
-Google Summer of Code 2017
---------------------------
+Getting involved
+----------------
-PyPy is generally open to new ideas for Google Summer of Code. We are happy to
accept good ideas around the PyPy ecosystem. If you need more information about
the ideas we propose for this year please join us on irc, channel #pypy
(freenode). If you are unsure, but still think that you can make a valuable
contribution to PyPy, dont hesitate to contact us on #pypy or on our mailing
list.
-
+We are happy to discuss ideas around the PyPy ecosystem.
+If you are interested in palying with RPython or PyPy, or have a new idea not
+mentioned here please join us on irc, channel #pypy (freenode). If you are
unsure,
+but still think that you can make a valuable contribution to PyPy, dont
+hesitate to contact us on #pypy or on our mailing list. Here are some ideas
+to get you thinking:
* **Optimize PyPy Memory Usage**: Sometimes PyPy consumes more memory than
CPython.
- Two examples: 1) PyPy seems to allocate and keep alive more strings when
importing a big Python modules.
- 2) The base interpreter size (cold VM started from a console) of PyPy is
bigger than the one of CPython.
- The general procedure of this project is: Run both CPython and PyPy of the
same Python version and
- compare the memory usage (using Massif or other tools).
+ Two examples: 1) PyPy seems to allocate and keep alive more strings when
+ importing a big Python modules. 2) The base interpreter size (cold VM
started
+ from a console) of PyPy is bigger than the one of CPython. The general
+ procedure of this project is: Run both CPython and PyPy of the same Python
+ version and compare the memory usage (using Massif or other tools).
If PyPy consumes a lot more memory then find and resolve the issue.
-* **VMProf + memory profiler**: vmprof by now has a memory profiler that can
be used already. We want extend it with more features and resolve some current
limitations.
+* **VMProf + memory profiler**: vmprof is a statistical memory profiler. We
+ want extend it with new features and resolve some current limitations.
-* **VMProf visualisations**: vmprof just shows a flame graph of the
statistical profile and some more information about specific call sites. It
would be very interesting to experiment with different information (such as
memory, or even information generated by our jit compiler).
+* **VMProf visualisations**: vmprof shows a flame graph of the statistical
+ profile and some more information about specific call sites. It would be
+ very interesting to experiment with different information (such as memory,
+ or even information generated by our jit compiler).
-* **Explicit typing in RPython**: PyPy wants to have better ways to specify
the signature and class attribute types in RPython. See more information about
this topic below on this page.
+* **Explicit typing in RPython**: PyPy wants to have better ways to specify
+ the signature and class attribute types in RPython. See more information
+ about this topic below on this page.
-* **Virtual Reality (VR) visualisations for vmprof**: This is a very open
topic with lots of freedom to explore data visualisation for profiles. No VR
hardware would be needed for this project. Either universities provide such
hardware or in any other case we potentially can lend the VR hardware setup.
+* **Virtual Reality (VR) visualisations for vmprof**: This is a very open
+ topic with lots of freedom to explore data visualisation for profiles. No
+ VR hardware would be needed for this project. Either universities provide
+ such hardware or in any other case we potentially can lend the VR hardware
+ setup.
Simple tasks for newcomers
--------------------------
@@ -34,6 +49,11 @@
* Implement AF_XXX packet types of sockets:
https://bitbucket.org/pypy/pypy/issue/1942/support-for-af_xxx-sockets
+* Help with documentation. One task would be to document rpython configuration
+ options currently listed only on :doc:`this site <configuration>` also on the
+ RPython_ documentation site.
+
+.. _RPython: http://rpython.readthedocs.io
Mid-to-large tasks
------------------
@@ -201,7 +221,9 @@
Introduce new benchmarks
------------------------
-We're usually happy to introduce new benchmarks. Please consult us
+Our benchmark runner_ is showing its age. We should merge with the `CPython
site`_
+
+Additionally, we're usually happy to introduce new benchmarks. Please consult
us
before, but in general something that's real-world python code
and is not already represented is welcome. We need at least a standalone
script that can run without parameters. Example ideas (benchmarks need
@@ -209,6 +231,8 @@
* `hg`
+.. _runner: http://speed.pypy.org
+.. _`CPython site`: https://speed.python.org/
======================================
Make more python modules pypy-friendly
@@ -238,15 +262,6 @@
using more pypy-friendly technologies, e.g. cffi. Here is a partial list of
good work that needs to be finished:
-**matplotlib** https://github.com/matplotlib/matplotlib
-
- Status: using the matplotlib branch of PyPy and the tkagg-cffi branch of
- matplotlib from https://github.com/mattip/matplotlib/tree/tkagg-cffi, the
- tkagg backend can function.
-
- TODO: the matplotlib branch passes numpy arrays by value (copying all the
- data), this proof-of-concept needs help to become completely compliant
-
**wxPython** https://bitbucket.org/amauryfa/wxpython-cffi
Status: A project by a PyPy developer to adapt the Phoenix sip build
system to cffi
diff --git a/pypy/doc/release-v6.0.0.rst b/pypy/doc/release-v6.0.0.rst
new file mode 100644
--- /dev/null
+++ b/pypy/doc/release-v6.0.0.rst
@@ -0,0 +1,109 @@
+======================================
+PyPy2.7 and PyPy3.5 v6.0 dual release
+======================================
+
+The PyPy team is proud to release both PyPy2.7 v6.0 (an interpreter supporting
+Python 2.7 syntax), and a PyPy3.5 v6.0 (an interpreter supporting Python
+3.5 syntax). The two releases are both based on much the same codebase, thus
+the dual release.
+
+This release is a feature release following our previous 5.10 incremental
+release in late December 2017. Our C-API compatability layer ``cpyext`` is
+now much faster (see the `blog post`_) as well as more complete. We have made
+many other improvements in speed and CPython compatibility. Since the changes
+affect the included python development header files, all c-extension modules
must
+be recompiled for this version.
+
+First-time python users are often stumped by silly typos and emissions when
+getting started writing code. We have improved our parser to emit more friendly
+`syntax errors`_, making PyPy not only faster but more friendly.
+
+The Windows PyPy3.5 release is still considered beta-quality. There are open
+issues with unicode handling especially around system calls and c-extensions.
+
+The Matplotlib TkAgg backend now works with PyPy, as do pygame and pygobject_.
+
+As always, this release is 100% compatible with the previous one and fixed
+several issues and bugs raised by the growing community of PyPy users.
+We strongly recommend updating.
+
+We updated the cffi module included in PyPy to version 1.11.5
+
+The utf8 branch that changes internal representation of unicode to utf8 did not
+make it into the release, so there is still more goodness coming. We also
+began working on a Python3.6 implementation, help is welcome.
+
+You can download the v6.0 releases here:
+
+ http://pypy.org/download.html
+
+We would like to thank our donors for the continued support of the PyPy
+project. If PyPy is not quite good enough for your needs, we are available for
+direct consulting work.
+
+We would also like to thank our contributors and encourage new people to join
+the project. PyPy has many layers and we need help with all of them: `PyPy`_
+and `RPython`_ documentation improvements, tweaking popular `modules`_ to run
+on pypy, or general `help`_ with making RPython's JIT even better.
+
+.. _`PyPy`: index.html
+.. _`RPython`: https://rpython.readthedocs.org
+.. _`modules`: project-ideas.html#make-more-python-modules-pypy-friendly
+.. _`help`: project-ideas.html
+.. _`blog post`:
https://morepypy.blogspot.it/2017/10/cape-of-good-hope-for-pypy-hello-from.html
+.. _pygobject: https://lazka.github.io/posts/2018-04_pypy-pygobject/index.html
+.. _`syntax errors`:
https://morepypy.blogspot.com/2018/04/improving-syntaxerror-in-pypy.html
+
+What is PyPy?
+=============
+
+PyPy is a very compliant Python interpreter, almost a drop-in replacement for
+CPython 2.7 and CPython 3.5. It's fast (`PyPy and CPython 2.7.x`_ performance
comparison)
+due to its integrated tracing JIT compiler.
+
+We also welcome developers of other `dynamic languages`_ to see what RPython
+can do for them.
+
+The PyPy release supports:
+
+ * **x86** machines on most common operating systems
+ (Linux 32/64 bits, Mac OS X 64 bits, Windows 32 bits, OpenBSD, FreeBSD)
+
+ * newer **ARM** hardware (ARMv6 or ARMv7, with VFPv3) running Linux,
+
+ * big- and little-endian variants of **PPC64** running Linux,
+
+ * **s390x** running Linux
+
+.. _`PyPy and CPython 2.7.x`: http://speed.pypy.org
+.. _`dynamic languages`: http://rpython.readthedocs.io/en/latest/examples.html
+
+Changelog
+=========
+
+* Speed up C-API method calls, and make most Py*_Check calls C macros
+* Speed up C-API slot method calls
+* Enable TkAgg backend support for matplotlib
+* support ``hastzinfo`` and ``tzinfo`` in the C-API ``PyDateTime*`` structures
+* datetime.h is now more similar to CPython
+* We now support ``PyUnicode_AsUTF{16,32}String``, ``_PyLong_AsByteArray``,
+ ``_PyLong_AsByteArrayO``,
+* PyPy3.5 on Windows is compiled with the Microsoft Visual Compiler v14, like
+ CPython
+* Fix performance of attribute lookup when more than 80 attributes are used
+* Improve performance on passing built-in types to C-API C code
+* Improve the performance of datetime and timedelta by skipping the consistency
+ checks of the datetime values (they are correct by construction)
+* Improve handling of ``bigint`` s, including fixing ``int_divmod``
+* Improve reporting of GC statistics
+* Accept unicode filenames in ``dbm.open()``
+* Improve RPython support for half-floats
+* Added missing attributes to C-API ``instancemethod`` on pypy3
+* Store error state in thread-local storage for C-API.
+* Fix JIT bugs exposed in the sre module
+* Improve speed of Python parser, improve ParseError messages slightly
+* Handle JIT hooks more efficiently
+
+We also refactored many parts of the JIT bridge optimizations, as well as
cpyext
+internals, and together with new contributors fixed issues, added new
+documentation, and cleaned up the codebase.
diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -1,16 +1,20 @@
-===========================
-What's new in PyPy2.7 5.10+
-===========================
+==========================
+What's new in PyPy2.7 6.0+
+==========================
-.. this is a revision shortly after release-pypy2.7-v5.10.0
-.. startrev: 6b024edd9d12
+.. this is a revision shortly after release-pypy-6.0.0
+.. startrev: f22145c34985
-.. branch: cpyext-avoid-roundtrip
-Big refactoring of some cpyext code, which avoids a lot of nonsense when
-calling C from Python and vice-versa: the result is a big speedup in
-function/method calls, up to 6 times faster.
+.. branch: issue2752
-.. branch: cpyext-datetime2
+Fix a rare GC bug that was introduced more than one year ago, but was
+not diagnosed before issue #2752.
-Support ``tzinfo`` field on C-API datetime objects, fixes latest pandas HEAD
+.. branch: gc-hooks
+
+Introduce GC hooks, as documented in doc/gc_info.rst
+
+.. branch: gc-hook-better-timestamp
+
+Improve GC hooks
diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-pypy2-6.0.0.rst
copy from pypy/doc/whatsnew-head.rst
copy to pypy/doc/whatsnew-pypy2-6.0.0.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-pypy2-6.0.0.rst
@@ -14,3 +14,98 @@
.. branch: cpyext-datetime2
Support ``tzinfo`` field on C-API datetime objects, fixes latest pandas HEAD
+
+
+.. branch: mapdict-size-limit
+
+Fix a corner case of mapdict: When an instance is used like a dict (using
+``setattr`` and ``getattr``, or ``.__dict__``) and a lot of attributes are
+added, then the performance using mapdict is linear in the number of
+attributes. This is now fixed (by switching to a regular dict after 80
+attributes).
+
+
+.. branch: cpyext-faster-arg-passing
+
+When using cpyext, improve the speed of passing certain objects from PyPy to C
+code, most notably None, True, False, types, all instances of C-defined types.
+Before, a dict lookup was needed every time such an object crossed over, now it
+is just a field read.
+
+
+.. branch: 2634_datetime_timedelta_performance
+
+Improve datetime + timedelta performance.
+
+.. branch: memory-accounting
+
+Improve way to describe memory
+
+.. branch: msvc14
+
+Allow compilaiton with Visual Studio 2017 compiler suite on windows
+
+.. branch: refactor-slots
+
+Refactor cpyext slots.
+
+
+.. branch: call-loopinvariant-into-bridges
+
+Speed up branchy code that does a lot of function inlining by saving one call
+to read the TLS in most bridges.
+
_______________________________________________
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
