Author: Manuel Jacob <m...@manueljacob.de> Branch: llvm-translation-backend Changeset: r75743:6000e86e39ba Date: 2015-02-06 20:17 +0100 http://bitbucket.org/pypy/pypy/changeset/6000e86e39ba/
Log: hg merge default diff too long, truncating to 2000 out of 19187 lines diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -7,10 +7,7 @@ 9b623bc48b5950cf07184462a0e48f2c4df0d720 pypy-2.1-beta1-arm ab0dd631c22015ed88e583d9fdd4c43eebf0be21 pypy-2.1-beta1-arm 20e51c4389ed4469b66bb9d6289ce0ecfc82c4b9 release-2.3.0 -20e51c4389ed4469b66bb9d6289ce0ecfc82c4b9 release-2.3.0 -0000000000000000000000000000000000000000 release-2.3.0 394146e9bb673514c61f0150ab2013ccf78e8de7 release-2.3 32f35069a16d819b58c1b6efb17c44e3e53397b2 release-2.2=3.1 32f35069a16d819b58c1b6efb17c44e3e53397b2 release-2.3.1 -32f35069a16d819b58c1b6efb17c44e3e53397b2 release-2.2=3.1 -0000000000000000000000000000000000000000 release-2.2=3.1 +10f1b29a2bd21f837090286174a9ca030b8680b2 release-2.5.0 diff --git a/LICENSE b/LICENSE --- a/LICENSE +++ b/LICENSE @@ -42,19 +42,19 @@ Amaury Forgeot d'Arc Samuele Pedroni Alex Gaynor + Brian Kearns + Matti Picus + Philip Jenvey Michael Hudson David Schneider - Matti Picus - Brian Kearns - Philip Jenvey Holger Krekel Christian Tismer Hakan Ardo Benjamin Peterson Manuel Jacob + Ronan Lamy Anders Chrigstrom Eric van Riet Paap - Ronan Lamy Wim Lavrijsen Richard Emslie Alexander Schremmer @@ -68,9 +68,9 @@ Camillo Bruni Laura Creighton Toon Verwaest + Romain Guillebert Leonardo Santagada Seo Sanghyeon - Romain Guillebert Justin Peel Ronny Pfannschmidt David Edelsohn @@ -91,15 +91,16 @@ Michal Bendowski Jan de Mooij stian + Tyler Wade Michael Foord Stephan Diehl - Tyler Wade Stefan Schwarzer Valentino Volonghi Tomek Meka Patrick Maupin Bob Ippolito Bruno Gola + David Malcolm Jean-Paul Calderone Timo Paulssen Squeaky @@ -108,18 +109,19 @@ Marius Gedminas Martin Matusiak Konstantin Lopuhin + Wenzhu Man John Witulski - Wenzhu Man + Laurence Tratt + Ivan Sichmann Freitas Greg Price Dario Bertini Mark Pearse Simon Cross - Ivan Sichmann Freitas Andreas Stührk + Stefano Rivera Jean-Philippe St. Pierre Guido van Rossum Pavel Vinogradov - Stefano Rivera Paweł Piotr Przeradowski Paul deGrandis Ilya Osadchiy @@ -129,7 +131,6 @@ tav Taavi Burns Georg Brandl - Laurence Tratt Bert Freudenberg Stian Andreassen Wanja Saatkamp @@ -141,13 +142,12 @@ Jeremy Thurgood Rami Chowdhury Tobias Pape - David Malcolm Eugene Oden Henry Mason Vasily Kuznetsov Preston Timmons + David Ripton Jeff Terrace - David Ripton Dusty Phillips Lukas Renggli Guenter Jantzen @@ -166,13 +166,16 @@ Gintautas Miliauskas Michael Twomey Lucian Branescu Mihaila + Yichao Yu Gabriel Lavoie Olivier Dormond Jared Grubb Karl Bartel + Wouter van Heyst Brian Dorsey Victor Stinner Andrews Medina + anatoly techtonik Stuart Williams Jasper Schulz Christian Hudon @@ -182,12 +185,11 @@ Michael Cheng Justas Sadzevicius Gasper Zejn - anatoly techtonik Neil Shepperd + Stanislaw Halik Mikael Schönenberg Elmo M?ntynen Jonathan David Riehl - Stanislaw Halik Anders Qvist Corbin Simpson Chirag Jadwani @@ -196,10 +198,13 @@ Vincent Legoll Alan McIntyre Alexander Sedov + Attila Gobi Christopher Pope Christian Tismer Marc Abramowitz Dan Stromberg + Arjun Naik + Valentina Mukhamedzhanova Stefano Parmesan Alexis Daboville Jens-Uwe Mager @@ -213,8 +218,6 @@ Sylvain Thenault Nathan Taylor Vladimir Kryachko - Arjun Naik - Attila Gobi Jacek Generowicz Alejandro J. Cura Jacob Oscarson @@ -222,22 +225,23 @@ Ryan Gonzalez Ian Foote Kristjan Valur Jonsson + David Lievens Neil Blakey-Milner Lutz Paelike Lucio Torre Lars Wassermann - Valentina Mukhamedzhanova Henrik Vendelbo Dan Buch Miguel de Val Borro Artur Lisiecki Sergey Kishchenko - Yichao Yu Ignas Mikalajunas Christoph Gerum Martin Blais Lene Wagner Tomo Cocoa + Toni Mattis + Lucas Stadler roberto@goyle Yury V. Zaytsev Anna Katrina Dominguez @@ -265,23 +269,30 @@ Stephan Busemann Rafał Gałczyński Christian Muirhead + Berker Peksag James Lan shoma hosaka - Daniel Neuh?user - Matthew Miller + Daniel Neuhäuser + Ben Mather + halgari + Boglarka Vezer + Chris Pressey Buck Golemon Konrad Delong Dinu Gherman Chris Lambacher coolbutusel...@gmail.com + Jim Baker Rodrigo Araújo - Jim Baker + Nikolaos-Digenis Karagiannis James Robert Armin Ronacher Brett Cannon + Donald Stufft yrttyr aliceinwire OlivierBlanvillain + Dan Sanders Zooko Wilcox-O Hearn Tomer Chachamu Christopher Groskopf @@ -295,6 +306,7 @@ Markus Unterwaditzer Even Wiik Thomassen jbs + squeaky soareschen Kurt Griffiths Mike Bayer @@ -306,6 +318,7 @@ Anna Ravencroft Dan Crosta Julien Phalip + Roman Podoliaka Dan Loewenherz Heinrich-Heine University, Germany diff --git a/lib-python/2.7/collections.py b/lib-python/2.7/collections.py --- a/lib-python/2.7/collections.py +++ b/lib-python/2.7/collections.py @@ -17,6 +17,10 @@ except ImportError: assert '__pypy__' not in _sys.builtin_module_names newdict = lambda _ : {} +try: + from __pypy__ import reversed_dict +except ImportError: + reversed_dict = lambda d: reversed(d.keys()) try: from thread import get_ident as _get_ident @@ -29,142 +33,35 @@ ################################################################################ class OrderedDict(dict): - 'Dictionary that remembers insertion order' - # An inherited dict maps keys to values. - # The inherited dict provides __getitem__, __len__, __contains__, and get. - # The remaining methods are order-aware. - # Big-O running times for all methods are the same as regular dictionaries. + '''Dictionary that remembers insertion order. - # The internal self.__map dict maps keys to links in a doubly linked list. - # The circular doubly linked list starts and ends with a sentinel element. - # The sentinel element never gets deleted (this simplifies the algorithm). - # Each link is stored as a list of length three: [PREV, NEXT, KEY]. + In PyPy all dicts are ordered anyway. This is mostly useful as a + placeholder to mean "this dict must be ordered even on CPython". - def __init__(self, *args, **kwds): - '''Initialize an ordered dictionary. The signature is the same as - regular dictionaries, but keyword arguments are not recommended because - their insertion order is arbitrary. - - ''' - if len(args) > 1: - raise TypeError('expected at most 1 arguments, got %d' % len(args)) - try: - self.__root - except AttributeError: - self.__root = root = [] # sentinel node - root[:] = [root, root, None] - self.__map = {} - self.__update(*args, **kwds) - - def __setitem__(self, key, value, dict_setitem=dict.__setitem__): - 'od.__setitem__(i, y) <==> od[i]=y' - # Setting a new item creates a new link at the end of the linked list, - # and the inherited dictionary is updated with the new key/value pair. - if key not in self: - root = self.__root - last = root[0] - last[1] = root[0] = self.__map[key] = [last, root, key] - return dict_setitem(self, key, value) - - def __delitem__(self, key, dict_delitem=dict.__delitem__): - 'od.__delitem__(y) <==> del od[y]' - # Deleting an existing item uses self.__map to find the link which gets - # removed by updating the links in the predecessor and successor nodes. - dict_delitem(self, key) - link_prev, link_next, _ = self.__map.pop(key) - link_prev[1] = link_next # update link_prev[NEXT] - link_next[0] = link_prev # update link_next[PREV] - - def __iter__(self): - 'od.__iter__() <==> iter(od)' - # Traverse the linked list in order. - root = self.__root - curr = root[1] # start at the first node - while curr is not root: - yield curr[2] # yield the curr[KEY] - curr = curr[1] # move to next node + Known difference: iterating over an OrderedDict which is being + concurrently modified raises RuntimeError in PyPy. In CPython + instead we get some behavior that appears reasonable in some + cases but is nonsensical in other cases. This is officially + forbidden by the CPython docs, so we forbid it explicitly for now. + ''' def __reversed__(self): - 'od.__reversed__() <==> reversed(od)' - # Traverse the linked list in reverse order. - root = self.__root - curr = root[0] # start at the last node - while curr is not root: - yield curr[2] # yield the curr[KEY] - curr = curr[0] # move to previous node - - def clear(self): - 'od.clear() -> None. Remove all items from od.' - root = self.__root - root[:] = [root, root, None] - self.__map.clear() - dict.clear(self) - - # -- the following methods do not depend on the internal structure -- - - def keys(self): - 'od.keys() -> list of keys in od' - return list(self) - - def values(self): - 'od.values() -> list of values in od' - return [self[key] for key in self] - - def items(self): - 'od.items() -> list of (key, value) pairs in od' - return [(key, self[key]) for key in self] - - def iterkeys(self): - 'od.iterkeys() -> an iterator over the keys in od' - return iter(self) - - def itervalues(self): - 'od.itervalues -> an iterator over the values in od' - for k in self: - yield self[k] - - def iteritems(self): - 'od.iteritems -> an iterator over the (key, value) pairs in od' - for k in self: - yield (k, self[k]) - - update = MutableMapping.update - - __update = update # let subclasses override update without breaking __init__ - - __marker = object() - - def pop(self, key, default=__marker): - '''od.pop(k[,d]) -> v, remove specified key and return the corresponding - value. If key is not found, d is returned if given, otherwise KeyError - is raised. - - ''' - if key in self: - result = self[key] - del self[key] - return result - if default is self.__marker: - raise KeyError(key) - return default - - def setdefault(self, key, default=None): - 'od.setdefault(k[,d]) -> od.get(k,d), also set od[k]=d if k not in od' - if key in self: - return self[key] - self[key] = default - return default + return reversed_dict(self) def popitem(self, last=True): '''od.popitem() -> (k, v), return and remove a (key, value) pair. Pairs are returned in LIFO order if last is true or FIFO order if false. ''' - if not self: - raise KeyError('dictionary is empty') - key = next(reversed(self) if last else iter(self)) - value = self.pop(key) - return key, value + if last: + return dict.popitem(self) + else: + it = dict.__iter__(self) + try: + k = it.next() + except StopIteration: + raise KeyError('dictionary is empty') + return (k, self.pop(k)) def __repr__(self, _repr_running={}): 'od.__repr__() <==> repr(od)' @@ -183,8 +80,6 @@ 'Return state information for pickling' items = [[k, self[k]] for k in self] inst_dict = vars(self).copy() - for k in vars(OrderedDict()): - inst_dict.pop(k, None) if inst_dict: return (self.__class__, (items,), inst_dict) return self.__class__, (items,) @@ -193,17 +88,6 @@ 'od.copy() -> a shallow copy of od' return self.__class__(self) - @classmethod - def fromkeys(cls, iterable, value=None): - '''OD.fromkeys(S[, v]) -> New ordered dictionary with keys from S. - If not specified, the value defaults to None. - - ''' - self = cls() - for key in iterable: - self[key] = value - return self - def __eq__(self, other): '''od.__eq__(y) <==> od==y. Comparison to another OD is order-sensitive while comparison to a regular mapping is order-insensitive. diff --git a/lib-python/2.7/ctypes/test/test_frombuffer.py b/lib-python/2.7/ctypes/test/test_frombuffer.py --- a/lib-python/2.7/ctypes/test/test_frombuffer.py +++ b/lib-python/2.7/ctypes/test/test_frombuffer.py @@ -2,7 +2,6 @@ import array import gc import unittest -from ctypes.test import xfail class X(Structure): _fields_ = [("c_int", c_int)] @@ -11,7 +10,6 @@ self._init_called = True class Test(unittest.TestCase): - @xfail def test_fom_buffer(self): a = array.array("i", range(16)) x = (c_int * 16).from_buffer(a) @@ -34,10 +32,9 @@ del a; gc.collect(); gc.collect(); gc.collect() self.assertEqual(x[:], expected) - self.assertRaises(TypeError, + self.assertRaises((TypeError, ValueError), (c_char * 16).from_buffer, "a" * 16) - @xfail def test_fom_buffer_with_offset(self): a = array.array("i", range(16)) x = (c_int * 15).from_buffer(a, sizeof(c_int)) @@ -46,7 +43,6 @@ self.assertRaises(ValueError, lambda: (c_int * 16).from_buffer(a, sizeof(c_int))) self.assertRaises(ValueError, lambda: (c_int * 1).from_buffer(a, 16 * sizeof(c_int))) - @xfail def test_from_buffer_copy(self): a = array.array("i", range(16)) x = (c_int * 16).from_buffer_copy(a) @@ -71,7 +67,6 @@ x = (c_char * 16).from_buffer_copy("a" * 16) self.assertEqual(x[:], "a" * 16) - @xfail def test_fom_buffer_copy_with_offset(self): a = array.array("i", range(16)) x = (c_int * 15).from_buffer_copy(a, sizeof(c_int)) diff --git a/lib-python/2.7/test/test_collections.py b/lib-python/2.7/test/test_collections.py --- a/lib-python/2.7/test/test_collections.py +++ b/lib-python/2.7/test/test_collections.py @@ -578,7 +578,12 @@ def __repr__(self): return "MySet(%s)" % repr(list(self)) s = MySet([5,43,2,1]) - self.assertEqual(s.pop(), 1) + # changed from CPython 2.7: it was "s.pop() == 1" but I see + # nothing that guarantees a particular order here. In the + # 'all_ordered_dicts' branch of PyPy (or with OrderedDict + # instead of sets), it consistently returns 5, but this test + # should not rely on this or any other order. + self.assert_(s.pop() in [5,43,2,1]) def test_issue8750(self): empty = WithSet() @@ -1010,8 +1015,9 @@ c=3, e=5).items()), pairs) # mixed input # make sure no positional args conflict with possible kwdargs - self.assertEqual(inspect.getargspec(OrderedDict.__dict__['__init__']).args, - ['self']) + if '__init__' in OrderedDict.__dict__: # absent in PyPy + self.assertEqual(inspect.getargspec(OrderedDict.__dict__['__init__']).args, + ['self']) # Make sure that direct calls to __init__ do not clear previous contents d = OrderedDict([('a', 1), ('b', 2), ('c', 3), ('d', 44), ('e', 55)]) diff --git a/lib_pypy/_ctypes/basics.py b/lib_pypy/_ctypes/basics.py --- a/lib_pypy/_ctypes/basics.py +++ b/lib_pypy/_ctypes/basics.py @@ -83,6 +83,37 @@ def in_dll(self, dll, name): return self.from_address(dll._handle.getaddressindll(name)) + def from_buffer(self, obj, offset=0): + size = self._sizeofinstances() + buf = buffer(obj, offset, size) + if len(buf) < size: + raise ValueError( + "Buffer size too small (%d instead of at least %d bytes)" + % (len(buf) + offset, size + offset)) + raw_addr = buf._pypy_raw_address() + result = self.from_address(raw_addr) + result._ensure_objects()['ffffffff'] = obj + return result + + def from_buffer_copy(self, obj, offset=0): + size = self._sizeofinstances() + buf = buffer(obj, offset, size) + if len(buf) < size: + raise ValueError( + "Buffer size too small (%d instead of at least %d bytes)" + % (len(buf) + offset, size + offset)) + result = self() + dest = result._buffer.buffer + try: + raw_addr = buf._pypy_raw_address() + except ValueError: + _rawffi.rawstring2charp(dest, buf) + else: + from ctypes import memmove + memmove(dest, raw_addr, size) + return result + + class CArgObject(object): """ simple wrapper around buffer, just for the case of freeing it afterwards diff --git a/lib_pypy/_ctypes/structure.py b/lib_pypy/_ctypes/structure.py --- a/lib_pypy/_ctypes/structure.py +++ b/lib_pypy/_ctypes/structure.py @@ -1,3 +1,4 @@ +import sys import _rawffi from _ctypes.basics import _CData, _CDataMeta, keepalive_key,\ store_reference, ensure_objects, CArgObject @@ -178,6 +179,8 @@ instance = StructOrUnion.__new__(self) if isinstance(address, _rawffi.StructureInstance): address = address.buffer + # fix the address: turn it into as unsigned, in case it is negative + address = address & (sys.maxint * 2 + 1) instance.__dict__['_buffer'] = self._ffistruct.fromaddress(address) return instance diff --git a/lib_pypy/cffi/__init__.py b/lib_pypy/cffi/__init__.py --- a/lib_pypy/cffi/__init__.py +++ b/lib_pypy/cffi/__init__.py @@ -6,3 +6,8 @@ __version__ = "0.8.6" __version_info__ = (0, 8, 6) + +# The verifier module file names are based on the CRC32 of a string that +# contains the following version number. It may be older than __version__ +# if nothing is clearly incompatible. +__version_verifier_modules__ = "0.8.6" diff --git a/lib_pypy/cffi/api.py b/lib_pypy/cffi/api.py --- a/lib_pypy/cffi/api.py +++ b/lib_pypy/cffi/api.py @@ -69,6 +69,7 @@ self._function_caches = [] self._libraries = [] self._cdefsources = [] + self._windows_unicode = None if hasattr(backend, 'set_ffi'): backend.set_ffi(self) for name in backend.__dict__: @@ -77,6 +78,7 @@ # with self._lock: self.BVoidP = self._get_cached_btype(model.voidp_type) + self.BCharA = self._get_cached_btype(model.char_array_type) if isinstance(backend, types.ModuleType): # _cffi_backend: attach these constants to the class if not hasattr(FFI, 'NULL'): @@ -189,13 +191,16 @@ cdecl = self._typeof(cdecl) return self._backend.alignof(cdecl) - def offsetof(self, cdecl, fieldname): + def offsetof(self, cdecl, *fields_or_indexes): """Return the offset of the named field inside the given - structure, which must be given as a C type name. + structure or array, which must be given as a C type name. + You can give several field names in case of nested structures. + You can also give numeric values which correspond to array + items, in case of an array type. """ if isinstance(cdecl, basestring): cdecl = self._typeof(cdecl) - return self._backend.typeoffsetof(cdecl, fieldname)[1] + return self._typeoffsetof(cdecl, *fields_or_indexes)[1] def new(self, cdecl, init=None): """Allocate an instance according to the specified C type and @@ -264,6 +269,16 @@ """ return self._backend.buffer(cdata, size) + def from_buffer(self, python_buffer): + """Return a <cdata 'char[]'> that points to the data of the + given Python object, which must support the buffer interface. + Note that this is not meant to be used on the built-in types str, + unicode, or bytearray (you can build 'char[]' arrays explicitly) + but only on objects containing large quantities of raw data + in some other format, like 'array.array' or numpy arrays. + """ + return self._backend.from_buffer(self.BCharA, python_buffer) + def callback(self, cdecl, python_callable=None, error=None): """Return a callback object or a decorator making such a callback object. 'cdecl' must name a C function pointer type. @@ -335,9 +350,23 @@ which requires binary compatibility in the signatures. """ from .verifier import Verifier, _caller_dir_pycache + # + # If set_unicode(True) was called, insert the UNICODE and + # _UNICODE macro declarations + if self._windows_unicode: + self._apply_windows_unicode(kwargs) + # + # Set the tmpdir here, and not in Verifier.__init__: it picks + # up the caller's directory, which we want to be the caller of + # ffi.verify(), as opposed to the caller of Veritier(). tmpdir = tmpdir or _caller_dir_pycache() + # + # Make a Verifier() and use it to load the library. self.verifier = Verifier(self, source, tmpdir, **kwargs) lib = self.verifier.load_library() + # + # Save the loaded library for keep-alive purposes, even + # if the caller doesn't keep it alive itself (it should). self._libraries.append(lib) return lib @@ -356,15 +385,29 @@ with self._lock: return model.pointer_cache(self, ctype) - def addressof(self, cdata, field=None): + def addressof(self, cdata, *fields_or_indexes): """Return the address of a <cdata 'struct-or-union'>. - If 'field' is specified, return the address of this field. + If 'fields_or_indexes' are given, returns the address of that + field or array item in the structure or array, recursively in + case of nested structures. """ ctype = self._backend.typeof(cdata) - ctype, offset = self._backend.typeoffsetof(ctype, field) + if fields_or_indexes: + ctype, offset = self._typeoffsetof(ctype, *fields_or_indexes) + else: + if ctype.kind == "pointer": + raise TypeError("addressof(pointer)") + offset = 0 ctypeptr = self._pointer_to(ctype) return self._backend.rawaddressof(ctypeptr, cdata, offset) + def _typeoffsetof(self, ctype, field_or_index, *fields_or_indexes): + ctype, offset = self._backend.typeoffsetof(ctype, field_or_index) + for field1 in fields_or_indexes: + ctype, offset1 = self._backend.typeoffsetof(ctype, field1, 1) + offset += offset1 + return ctype, offset + def include(self, ffi_to_include): """Includes the typedefs, structs, unions and enums defined in another FFI instance. Usage is similar to a #include in C, @@ -387,6 +430,44 @@ def from_handle(self, x): return self._backend.from_handle(x) + def set_unicode(self, enabled_flag): + """Windows: if 'enabled_flag' is True, enable the UNICODE and + _UNICODE defines in C, and declare the types like TCHAR and LPTCSTR + to be (pointers to) wchar_t. If 'enabled_flag' is False, + declare these types to be (pointers to) plain 8-bit characters. + This is mostly for backward compatibility; you usually want True. + """ + if self._windows_unicode is not None: + raise ValueError("set_unicode() can only be called once") + enabled_flag = bool(enabled_flag) + if enabled_flag: + self.cdef("typedef wchar_t TBYTE;" + "typedef wchar_t TCHAR;" + "typedef const wchar_t *LPCTSTR;" + "typedef const wchar_t *PCTSTR;" + "typedef wchar_t *LPTSTR;" + "typedef wchar_t *PTSTR;" + "typedef TBYTE *PTBYTE;" + "typedef TCHAR *PTCHAR;") + else: + self.cdef("typedef char TBYTE;" + "typedef char TCHAR;" + "typedef const char *LPCTSTR;" + "typedef const char *PCTSTR;" + "typedef char *LPTSTR;" + "typedef char *PTSTR;" + "typedef TBYTE *PTBYTE;" + "typedef TCHAR *PTCHAR;") + self._windows_unicode = enabled_flag + + def _apply_windows_unicode(self, kwds): + defmacros = kwds.get('define_macros', ()) + if not isinstance(defmacros, (list, tuple)): + raise TypeError("'define_macros' must be a list or tuple") + defmacros = list(defmacros) + [('UNICODE', '1'), + ('_UNICODE', '1')] + kwds['define_macros'] = defmacros + def _load_backend_lib(backend, name, flags): if name is None: diff --git a/lib_pypy/cffi/backend_ctypes.py b/lib_pypy/cffi/backend_ctypes.py --- a/lib_pypy/cffi/backend_ctypes.py +++ b/lib_pypy/cffi/backend_ctypes.py @@ -169,6 +169,7 @@ class CTypesGenericPtr(CTypesData): __slots__ = ['_address', '_as_ctype_ptr'] _automatic_casts = False + kind = "pointer" @classmethod def _newp(cls, init): @@ -370,10 +371,12 @@ (CTypesPrimitive, type(source).__name__)) return source # + kind1 = kind class CTypesPrimitive(CTypesGenericPrimitive): __slots__ = ['_value'] _ctype = ctype _reftypename = '%s &' % name + kind = kind1 def __init__(self, value): self._value = value @@ -703,12 +706,13 @@ class struct_or_union(base_ctypes_class): pass struct_or_union.__name__ = '%s_%s' % (kind, name) + kind1 = kind # class CTypesStructOrUnion(CTypesBaseStructOrUnion): __slots__ = ['_blob'] _ctype = struct_or_union _reftypename = '%s &' % (name,) - _kind = kind + _kind = kind = kind1 # CTypesStructOrUnion._fix_class() return CTypesStructOrUnion @@ -994,27 +998,42 @@ def getcname(self, BType, replace_with): return BType._get_c_name(replace_with) - def typeoffsetof(self, BType, fieldname): - if fieldname is not None and issubclass(BType, CTypesGenericPtr): - BType = BType._BItem - if not issubclass(BType, CTypesBaseStructOrUnion): - raise TypeError("expected a struct or union ctype") - if fieldname is None: - return (BType, 0) - else: + def typeoffsetof(self, BType, fieldname, num=0): + if isinstance(fieldname, str): + if num == 0 and issubclass(BType, CTypesGenericPtr): + BType = BType._BItem + if not issubclass(BType, CTypesBaseStructOrUnion): + raise TypeError("expected a struct or union ctype") BField = BType._bfield_types[fieldname] if BField is Ellipsis: raise TypeError("not supported for bitfields") return (BField, BType._offsetof(fieldname)) + elif isinstance(fieldname, (int, long)): + if issubclass(BType, CTypesGenericArray): + BType = BType._CTPtr + if not issubclass(BType, CTypesGenericPtr): + raise TypeError("expected an array or ptr ctype") + BItem = BType._BItem + offset = BItem._get_size() * fieldname + if offset > sys.maxsize: + raise OverflowError + return (BItem, offset) + else: + raise TypeError(type(fieldname)) - def rawaddressof(self, BTypePtr, cdata, offset): + def rawaddressof(self, BTypePtr, cdata, offset=None): if isinstance(cdata, CTypesBaseStructOrUnion): ptr = ctypes.pointer(type(cdata)._to_ctypes(cdata)) elif isinstance(cdata, CTypesGenericPtr): + if offset is None or not issubclass(type(cdata)._BItem, + CTypesBaseStructOrUnion): + raise TypeError("unexpected cdata type") + ptr = type(cdata)._to_ctypes(cdata) + elif isinstance(cdata, CTypesGenericArray): ptr = type(cdata)._to_ctypes(cdata) else: raise TypeError("expected a <cdata 'struct-or-union'>") - if offset != 0: + if offset: ptr = ctypes.cast( ctypes.c_void_p( ctypes.cast(ptr, ctypes.c_void_p).value + offset), diff --git a/lib_pypy/cffi/commontypes.py b/lib_pypy/cffi/commontypes.py --- a/lib_pypy/cffi/commontypes.py +++ b/lib_pypy/cffi/commontypes.py @@ -29,6 +29,9 @@ result = model.PointerType(resolve_common_type(result[:-2])) elif result in model.PrimitiveType.ALL_PRIMITIVE_TYPES: result = model.PrimitiveType(result) + elif result == 'set-unicode-needed': + raise api.FFIError("The Windows type %r is only available after " + "you call ffi.set_unicode()" % (commontype,)) else: if commontype == result: raise api.FFIError("Unsupported type: %r. Please file a bug " @@ -86,8 +89,6 @@ "ULONGLONG": "unsigned long long", "WCHAR": "wchar_t", "SHORT": "short", - "TBYTE": "WCHAR", - "TCHAR": "WCHAR", "UCHAR": "unsigned char", "UINT": "unsigned int", "UINT8": "unsigned char", @@ -157,14 +158,12 @@ "LPCVOID": model.const_voidp_type, "LPCWSTR": "const WCHAR *", - "LPCTSTR": "LPCWSTR", "LPDWORD": "DWORD *", "LPHANDLE": "HANDLE *", "LPINT": "int *", "LPLONG": "long *", "LPSTR": "CHAR *", "LPWSTR": "WCHAR *", - "LPTSTR": "LPWSTR", "LPVOID": model.voidp_type, "LPWORD": "WORD *", "LRESULT": "LONG_PTR", @@ -173,7 +172,6 @@ "PBYTE": "BYTE *", "PCHAR": "CHAR *", "PCSTR": "const CHAR *", - "PCTSTR": "LPCWSTR", "PCWSTR": "const WCHAR *", "PDWORD": "DWORD *", "PDWORDLONG": "DWORDLONG *", @@ -200,9 +198,6 @@ "PSIZE_T": "SIZE_T *", "PSSIZE_T": "SSIZE_T *", "PSTR": "CHAR *", - "PTBYTE": "TBYTE *", - "PTCHAR": "TCHAR *", - "PTSTR": "LPWSTR", "PUCHAR": "UCHAR *", "PUHALF_PTR": "UHALF_PTR *", "PUINT": "UINT *", @@ -240,6 +235,15 @@ "USN": "LONGLONG", "VOID": model.void_type, "WPARAM": "UINT_PTR", + + "TBYTE": "set-unicode-needed", + "TCHAR": "set-unicode-needed", + "LPCTSTR": "set-unicode-needed", + "PCTSTR": "set-unicode-needed", + "LPTSTR": "set-unicode-needed", + "PTSTR": "set-unicode-needed", + "PTBYTE": "set-unicode-needed", + "PTCHAR": "set-unicode-needed", }) return result diff --git a/lib_pypy/cffi/cparser.py b/lib_pypy/cffi/cparser.py --- a/lib_pypy/cffi/cparser.py +++ b/lib_pypy/cffi/cparser.py @@ -1,4 +1,3 @@ - from . import api, model from .commontypes import COMMON_TYPES, resolve_common_type try: @@ -209,6 +208,8 @@ def _add_constants(self, key, val): if key in self._int_constants: + if self._int_constants[key] == val: + return # ignore identical double declarations raise api.FFIError( "multiple declarations of constant: %s" % (key,)) self._int_constants[key] = val @@ -228,12 +229,18 @@ pyvalue = int(int_str, 0) self._add_constants(key, pyvalue) + self._declare('macro ' + key, pyvalue) elif value == '...': self._declare('macro ' + key, value) else: - raise api.CDefError('only supports the syntax "#define ' - '%s ..." (literally) or "#define ' - '%s 0x1FF" for now' % (key, key)) + raise api.CDefError( + 'only supports one of the following syntax:\n' + ' #define %s ... (literally dot-dot-dot)\n' + ' #define %s NUMBER (with NUMBER an integer' + ' constant, decimal/hex/octal)\n' + 'got:\n' + ' #define %s %s' + % (key, key, key, value)) def _parse_decl(self, decl): node = decl.type @@ -460,6 +467,8 @@ elif kind == 'union': tp = model.UnionType(explicit_name, None, None, None) elif kind == 'enum': + if explicit_name == '__dotdotdot__': + raise CDefError("Enums cannot be declared with ...") tp = self._build_enum_type(explicit_name, type.values) else: raise AssertionError("kind = %r" % (kind,)) @@ -532,9 +541,24 @@ def _parse_constant(self, exprnode, partial_length_ok=False): # for now, limited to expressions that are an immediate number - # or negative number + # or positive/negative number if isinstance(exprnode, pycparser.c_ast.Constant): - return int(exprnode.value, 0) + s = exprnode.value + if s.startswith('0'): + if s.startswith('0x') or s.startswith('0X'): + return int(s, 16) + return int(s, 8) + elif '1' <= s[0] <= '9': + return int(s, 10) + elif s[0] == "'" and s[-1] == "'" and ( + len(s) == 3 or (len(s) == 4 and s[1] == "\\")): + return ord(s[-2]) + else: + raise api.CDefError("invalid constant %r" % (s,)) + # + if (isinstance(exprnode, pycparser.c_ast.UnaryOp) and + exprnode.op == '+'): + return self._parse_constant(exprnode.expr) # if (isinstance(exprnode, pycparser.c_ast.UnaryOp) and exprnode.op == '-'): diff --git a/lib_pypy/cffi/ffiplatform.py b/lib_pypy/cffi/ffiplatform.py --- a/lib_pypy/cffi/ffiplatform.py +++ b/lib_pypy/cffi/ffiplatform.py @@ -11,6 +11,9 @@ """ +LIST_OF_FILE_NAMES = ['sources', 'include_dirs', 'library_dirs', + 'extra_objects', 'depends'] + def get_extension(srcfilename, modname, sources=(), **kwds): from distutils.core import Extension allsources = [srcfilename] diff --git a/lib_pypy/cffi/model.py b/lib_pypy/cffi/model.py --- a/lib_pypy/cffi/model.py +++ b/lib_pypy/cffi/model.py @@ -235,6 +235,8 @@ BPtrItem = PointerType(self.item).get_cached_btype(ffi, finishlist) return global_cache(self, ffi, 'new_array_type', BPtrItem, self.length) +char_array_type = ArrayType(PrimitiveType('char'), None) + class StructOrUnionOrEnum(BaseTypeByIdentity): _attrs_ = ('name',) @@ -478,7 +480,7 @@ try: res = getattr(ffi._backend, funcname)(*args) except NotImplementedError as e: - raise NotImplementedError("%r: %s" % (srctype, e)) + raise NotImplementedError("%s: %r: %s" % (funcname, srctype, e)) # note that setdefault() on WeakValueDictionary is not atomic # and contains a rare bug (http://bugs.python.org/issue19542); # we have to use a lock and do it ourselves diff --git a/lib_pypy/cffi/vengine_cpy.py b/lib_pypy/cffi/vengine_cpy.py --- a/lib_pypy/cffi/vengine_cpy.py +++ b/lib_pypy/cffi/vengine_cpy.py @@ -65,7 +65,7 @@ # The following two 'chained_list_constants' items contains # the head of these two chained lists, as a string that gives the # call to do, if any. - self._chained_list_constants = ['0', '0'] + self._chained_list_constants = ['((void)lib,0)', '((void)lib,0)'] # prnt = self._prnt # first paste some standard set of lines that are mostly '#define' @@ -138,15 +138,22 @@ prnt() prnt('#endif') - def load_library(self): + def load_library(self, flags=None): # XXX review all usages of 'self' here! # import it as a new extension module + if hasattr(sys, "getdlopenflags"): + previous_flags = sys.getdlopenflags() try: + if hasattr(sys, "setdlopenflags") and flags is not None: + sys.setdlopenflags(flags) module = imp.load_dynamic(self.verifier.get_module_name(), self.verifier.modulefilename) except ImportError as e: error = "importing %r: %s" % (self.verifier.modulefilename, e) raise ffiplatform.VerificationError(error) + finally: + if hasattr(sys, "setdlopenflags"): + sys.setdlopenflags(previous_flags) # # call loading_cpy_struct() to get the struct layout inferred by # the C compiler @@ -228,7 +235,8 @@ converter = '_cffi_to_c_int' extraarg = ', %s' % tp.name else: - converter = '_cffi_to_c_%s' % (tp.name.replace(' ', '_'),) + converter = '(%s)_cffi_to_c_%s' % (tp.get_c_name(''), + tp.name.replace(' ', '_')) errvalue = '-1' # elif isinstance(tp, model.PointerType): @@ -267,8 +275,8 @@ self._prnt(' if (datasize != 0) {') self._prnt(' if (datasize < 0)') self._prnt(' %s;' % errcode) - self._prnt(' %s = alloca(datasize);' % (tovar,)) - self._prnt(' memset((void *)%s, 0, datasize);' % (tovar,)) + self._prnt(' %s = alloca((size_t)datasize);' % (tovar,)) + self._prnt(' memset((void *)%s, 0, (size_t)datasize);' % (tovar,)) self._prnt(' if (_cffi_convert_array_from_object(' '(char *)%s, _cffi_type(%d), %s) < 0)' % ( tovar, self._gettypenum(tp), fromvar)) @@ -336,7 +344,7 @@ prnt = self._prnt numargs = len(tp.args) if numargs == 0: - argname = 'no_arg' + argname = 'noarg' elif numargs == 1: argname = 'arg0' else: @@ -386,6 +394,9 @@ prnt(' Py_END_ALLOW_THREADS') prnt() # + prnt(' (void)self; /* unused */') + if numargs == 0: + prnt(' (void)noarg; /* unused */') if result_code: prnt(' return %s;' % self._convert_expr_from_c(tp.result, 'result', 'result type')) @@ -452,6 +463,7 @@ prnt('static void %s(%s *p)' % (checkfuncname, cname)) prnt('{') prnt(' /* only to generate compile-time warnings or errors */') + prnt(' (void)p;') for fname, ftype, fbitsize in tp.enumfields(): if (isinstance(ftype, model.PrimitiveType) and ftype.is_integer_type()) or fbitsize >= 0: @@ -482,6 +494,8 @@ prnt(' sizeof(((%s *)0)->%s),' % (cname, fname)) prnt(' -1') prnt(' };') + prnt(' (void)self; /* unused */') + prnt(' (void)noarg; /* unused */') prnt(' return _cffi_get_struct_layout(nums);') prnt(' /* the next line is not executed, but compiled */') prnt(' %s(0);' % (checkfuncname,)) @@ -578,7 +592,8 @@ # constants, likely declared with '#define' def _generate_cpy_const(self, is_int, name, tp=None, category='const', - vartp=None, delayed=True, size_too=False): + vartp=None, delayed=True, size_too=False, + check_value=None): prnt = self._prnt funcname = '_cffi_%s_%s' % (category, name) prnt('static int %s(PyObject *lib)' % funcname) @@ -590,6 +605,9 @@ else: assert category == 'const' # + if check_value is not None: + self._check_int_constant_value(name, check_value) + # if not is_int: if category == 'var': realexpr = '&' + name @@ -637,6 +655,27 @@ # ---------- # enums + def _check_int_constant_value(self, name, value, err_prefix=''): + prnt = self._prnt + if value <= 0: + prnt(' if ((%s) > 0 || (long)(%s) != %dL) {' % ( + name, name, value)) + else: + prnt(' if ((%s) <= 0 || (unsigned long)(%s) != %dUL) {' % ( + name, name, value)) + prnt(' char buf[64];') + prnt(' if ((%s) <= 0)' % name) + prnt(' snprintf(buf, 63, "%%ld", (long)(%s));' % name) + prnt(' else') + prnt(' snprintf(buf, 63, "%%lu", (unsigned long)(%s));' % + name) + prnt(' PyErr_Format(_cffi_VerificationError,') + prnt(' "%s%s has the real value %s, not %s",') + prnt(' "%s", "%s", buf, "%d");' % ( + err_prefix, name, value)) + prnt(' return -1;') + prnt(' }') + def _enum_funcname(self, prefix, name): # "$enum_$1" => "___D_enum____D_1" name = name.replace('$', '___D_') @@ -653,25 +692,8 @@ prnt('static int %s(PyObject *lib)' % funcname) prnt('{') for enumerator, enumvalue in zip(tp.enumerators, tp.enumvalues): - if enumvalue < 0: - prnt(' if ((%s) >= 0 || (long)(%s) != %dL) {' % ( - enumerator, enumerator, enumvalue)) - else: - prnt(' if ((%s) < 0 || (unsigned long)(%s) != %dUL) {' % ( - enumerator, enumerator, enumvalue)) - prnt(' char buf[64];') - prnt(' if ((%s) < 0)' % enumerator) - prnt(' snprintf(buf, 63, "%%ld", (long)(%s));' % enumerator) - prnt(' else') - prnt(' snprintf(buf, 63, "%%lu", (unsigned long)(%s));' % - enumerator) - prnt(' PyErr_Format(_cffi_VerificationError,') - prnt(' "enum %s: %s has the real value %s, ' - 'not %s",') - prnt(' "%s", "%s", buf, "%d");' % ( - name, enumerator, enumvalue)) - prnt(' return -1;') - prnt(' }') + self._check_int_constant_value(enumerator, enumvalue, + "enum %s: " % name) prnt(' return %s;' % self._chained_list_constants[True]) self._chained_list_constants[True] = funcname + '(lib)' prnt('}') @@ -695,8 +717,11 @@ # macros: for now only for integers def _generate_cpy_macro_decl(self, tp, name): - assert tp == '...' - self._generate_cpy_const(True, name) + if tp == '...': + check_value = None + else: + check_value = tp # an integer + self._generate_cpy_const(True, name, check_value=check_value) _generate_cpy_macro_collecttype = _generate_nothing _generate_cpy_macro_method = _generate_nothing @@ -783,6 +808,24 @@ typedef unsigned __int16 uint16_t; typedef unsigned __int32 uint32_t; typedef unsigned __int64 uint64_t; + typedef __int8 int_least8_t; + typedef __int16 int_least16_t; + typedef __int32 int_least32_t; + typedef __int64 int_least64_t; + typedef unsigned __int8 uint_least8_t; + typedef unsigned __int16 uint_least16_t; + typedef unsigned __int32 uint_least32_t; + typedef unsigned __int64 uint_least64_t; + typedef __int8 int_fast8_t; + typedef __int16 int_fast16_t; + typedef __int32 int_fast32_t; + typedef __int64 int_fast64_t; + typedef unsigned __int8 uint_fast8_t; + typedef unsigned __int16 uint_fast16_t; + typedef unsigned __int32 uint_fast32_t; + typedef unsigned __int64 uint_fast64_t; + typedef __int64 intmax_t; + typedef unsigned __int64 uintmax_t; # else # include <stdint.h> # endif @@ -828,12 +871,15 @@ PyLong_FromLongLong((long long)(x))) #define _cffi_from_c_int(x, type) \ - (((type)-1) > 0 ? /* unsigned */ \ - (sizeof(type) < sizeof(long) ? PyInt_FromLong(x) : \ - sizeof(type) == sizeof(long) ? PyLong_FromUnsignedLong(x) : \ - PyLong_FromUnsignedLongLong(x)) \ - : (sizeof(type) <= sizeof(long) ? PyInt_FromLong(x) : \ - PyLong_FromLongLong(x))) + (((type)-1) > 0 ? /* unsigned */ \ + (sizeof(type) < sizeof(long) ? \ + PyInt_FromLong((long)x) : \ + sizeof(type) == sizeof(long) ? \ + PyLong_FromUnsignedLong((unsigned long)x) : \ + PyLong_FromUnsignedLongLong((unsigned long long)x)) : \ + (sizeof(type) <= sizeof(long) ? \ + PyInt_FromLong((long)x) : \ + PyLong_FromLongLong((long long)x))) #define _cffi_to_c_int(o, type) \ (sizeof(type) == 1 ? (((type)-1) > 0 ? (type)_cffi_to_c_u8(o) \ @@ -844,7 +890,7 @@ : (type)_cffi_to_c_i32(o)) : \ sizeof(type) == 8 ? (((type)-1) > 0 ? (type)_cffi_to_c_u64(o) \ : (type)_cffi_to_c_i64(o)) : \ - (Py_FatalError("unsupported size for type " #type), 0)) + (Py_FatalError("unsupported size for type " #type), (type)0)) #define _cffi_to_c_i8 \ ((int(*)(PyObject *))_cffi_exports[1]) @@ -907,6 +953,7 @@ { PyObject *library; int was_alive = (_cffi_types != NULL); + (void)self; /* unused */ if (!PyArg_ParseTuple(args, "OOO", &_cffi_types, &_cffi_VerificationError, &library)) return NULL; diff --git a/lib_pypy/cffi/vengine_gen.py b/lib_pypy/cffi/vengine_gen.py --- a/lib_pypy/cffi/vengine_gen.py +++ b/lib_pypy/cffi/vengine_gen.py @@ -58,12 +58,12 @@ modname = self.verifier.get_module_name() prnt("void %s%s(void) { }\n" % (prefix, modname)) - def load_library(self): + def load_library(self, flags=0): # import it with the CFFI backend backend = self.ffi._backend # needs to make a path that contains '/', on Posix filename = os.path.join(os.curdir, self.verifier.modulefilename) - module = backend.load_library(filename) + module = backend.load_library(filename, flags) # # call loading_gen_struct() to get the struct layout inferred by # the C compiler @@ -235,6 +235,7 @@ prnt('static void %s(%s *p)' % (checkfuncname, cname)) prnt('{') prnt(' /* only to generate compile-time warnings or errors */') + prnt(' (void)p;') for fname, ftype, fbitsize in tp.enumfields(): if (isinstance(ftype, model.PrimitiveType) and ftype.is_integer_type()) or fbitsize >= 0: @@ -354,11 +355,20 @@ # ---------- # constants, likely declared with '#define' - def _generate_gen_const(self, is_int, name, tp=None, category='const'): + def _generate_gen_const(self, is_int, name, tp=None, category='const', + check_value=None): prnt = self._prnt funcname = '_cffi_%s_%s' % (category, name) self.export_symbols.append(funcname) - if is_int: + if check_value is not None: + assert is_int + assert category == 'const' + prnt('int %s(char *out_error)' % funcname) + prnt('{') + self._check_int_constant_value(name, check_value) + prnt(' return 0;') + prnt('}') + elif is_int: assert category == 'const' prnt('int %s(long long *out_value)' % funcname) prnt('{') @@ -367,6 +377,7 @@ prnt('}') else: assert tp is not None + assert check_value is None prnt(tp.get_c_name(' %s(void)' % funcname, name),) prnt('{') if category == 'var': @@ -383,9 +394,13 @@ _loading_gen_constant = _loaded_noop - def _load_constant(self, is_int, tp, name, module): + def _load_constant(self, is_int, tp, name, module, check_value=None): funcname = '_cffi_const_%s' % name - if is_int: + if check_value is not None: + assert is_int + self._load_known_int_constant(module, funcname) + value = check_value + elif is_int: BType = self.ffi._typeof_locked("long long*")[0] BFunc = self.ffi._typeof_locked("int(*)(long long*)")[0] function = module.load_function(BFunc, funcname) @@ -396,6 +411,7 @@ BLongLong = self.ffi._typeof_locked("long long")[0] value += (1 << (8*self.ffi.sizeof(BLongLong))) else: + assert check_value is None BFunc = self.ffi._typeof_locked(tp.get_c_name('(*)(void)', name))[0] function = module.load_function(BFunc, funcname) value = function() @@ -410,6 +426,36 @@ # ---------- # enums + def _check_int_constant_value(self, name, value): + prnt = self._prnt + if value <= 0: + prnt(' if ((%s) > 0 || (long)(%s) != %dL) {' % ( + name, name, value)) + else: + prnt(' if ((%s) <= 0 || (unsigned long)(%s) != %dUL) {' % ( + name, name, value)) + prnt(' char buf[64];') + prnt(' if ((%s) <= 0)' % name) + prnt(' sprintf(buf, "%%ld", (long)(%s));' % name) + prnt(' else') + prnt(' sprintf(buf, "%%lu", (unsigned long)(%s));' % + name) + prnt(' sprintf(out_error, "%s has the real value %s, not %s",') + prnt(' "%s", buf, "%d");' % (name[:100], value)) + prnt(' return -1;') + prnt(' }') + + def _load_known_int_constant(self, module, funcname): + BType = self.ffi._typeof_locked("char[]")[0] + BFunc = self.ffi._typeof_locked("int(*)(char*)")[0] + function = module.load_function(BFunc, funcname) + p = self.ffi.new(BType, 256) + if function(p) < 0: + error = self.ffi.string(p) + if sys.version_info >= (3,): + error = str(error, 'utf-8') + raise ffiplatform.VerificationError(error) + def _enum_funcname(self, prefix, name): # "$enum_$1" => "___D_enum____D_1" name = name.replace('$', '___D_') @@ -427,24 +473,7 @@ prnt('int %s(char *out_error)' % funcname) prnt('{') for enumerator, enumvalue in zip(tp.enumerators, tp.enumvalues): - if enumvalue < 0: - prnt(' if ((%s) >= 0 || (long)(%s) != %dL) {' % ( - enumerator, enumerator, enumvalue)) - else: - prnt(' if ((%s) < 0 || (unsigned long)(%s) != %dUL) {' % ( - enumerator, enumerator, enumvalue)) - prnt(' char buf[64];') - prnt(' if ((%s) < 0)' % enumerator) - prnt(' sprintf(buf, "%%ld", (long)(%s));' % enumerator) - prnt(' else') - prnt(' sprintf(buf, "%%lu", (unsigned long)(%s));' % - enumerator) - prnt(' sprintf(out_error,' - ' "%s has the real value %s, not %s",') - prnt(' "%s", buf, "%d");' % ( - enumerator[:100], enumvalue)) - prnt(' return -1;') - prnt(' }') + self._check_int_constant_value(enumerator, enumvalue) prnt(' return 0;') prnt('}') prnt() @@ -456,16 +485,8 @@ tp.enumvalues = tuple(enumvalues) tp.partial_resolved = True else: - BType = self.ffi._typeof_locked("char[]")[0] - BFunc = self.ffi._typeof_locked("int(*)(char*)")[0] funcname = self._enum_funcname(prefix, name) - function = module.load_function(BFunc, funcname) - p = self.ffi.new(BType, 256) - if function(p) < 0: - error = self.ffi.string(p) - if sys.version_info >= (3,): - error = str(error, 'utf-8') - raise ffiplatform.VerificationError(error) + self._load_known_int_constant(module, funcname) def _loaded_gen_enum(self, tp, name, module, library): for enumerator, enumvalue in zip(tp.enumerators, tp.enumvalues): @@ -476,13 +497,21 @@ # macros: for now only for integers def _generate_gen_macro_decl(self, tp, name): - assert tp == '...' - self._generate_gen_const(True, name) + if tp == '...': + check_value = None + else: + check_value = tp # an integer + self._generate_gen_const(True, name, check_value=check_value) _loading_gen_macro = _loaded_noop def _loaded_gen_macro(self, tp, name, module, library): - value = self._load_constant(True, tp, name, module) + if tp == '...': + check_value = None + else: + check_value = tp # an integer + value = self._load_constant(True, tp, name, module, + check_value=check_value) setattr(library, name, value) type(library)._cffi_dir.append(name) @@ -565,6 +594,24 @@ typedef unsigned __int16 uint16_t; typedef unsigned __int32 uint32_t; typedef unsigned __int64 uint64_t; + typedef __int8 int_least8_t; + typedef __int16 int_least16_t; + typedef __int32 int_least32_t; + typedef __int64 int_least64_t; + typedef unsigned __int8 uint_least8_t; + typedef unsigned __int16 uint_least16_t; + typedef unsigned __int32 uint_least32_t; + typedef unsigned __int64 uint_least64_t; + typedef __int8 int_fast8_t; + typedef __int16 int_fast16_t; + typedef __int32 int_fast32_t; + typedef __int64 int_fast64_t; + typedef unsigned __int8 uint_fast8_t; + typedef unsigned __int16 uint_fast16_t; + typedef unsigned __int32 uint_fast32_t; + typedef unsigned __int64 uint_fast64_t; + typedef __int64 intmax_t; + typedef unsigned __int64 uintmax_t; # else # include <stdint.h> # endif diff --git a/lib_pypy/cffi/verifier.py b/lib_pypy/cffi/verifier.py --- a/lib_pypy/cffi/verifier.py +++ b/lib_pypy/cffi/verifier.py @@ -1,12 +1,23 @@ -import sys, os, binascii, imp, shutil -from . import __version__ +import sys, os, binascii, shutil +from . import __version_verifier_modules__ from . import ffiplatform +if sys.version_info >= (3, 3): + import importlib.machinery + def _extension_suffixes(): + return importlib.machinery.EXTENSION_SUFFIXES[:] +else: + import imp + def _extension_suffixes(): + return [suffix for suffix, _, type in imp.get_suffixes() + if type == imp.C_EXTENSION] + class Verifier(object): def __init__(self, ffi, preamble, tmpdir=None, modulename=None, - ext_package=None, tag='', force_generic_engine=False, **kwds): + ext_package=None, tag='', force_generic_engine=False, + source_extension='.c', flags=None, relative_to=None, **kwds): self.ffi = ffi self.preamble = preamble if not modulename: @@ -14,14 +25,15 @@ vengine_class = _locate_engine_class(ffi, force_generic_engine) self._vengine = vengine_class(self) self._vengine.patch_extension_kwds(kwds) - self.kwds = kwds + self.flags = flags + self.kwds = self.make_relative_to(kwds, relative_to) # if modulename: if tag: raise TypeError("can't specify both 'modulename' and 'tag'") else: - key = '\x00'.join([sys.version[:3], __version__, preamble, - flattened_kwds] + + key = '\x00'.join([sys.version[:3], __version_verifier_modules__, + preamble, flattened_kwds] + ffi._cdefsources) if sys.version_info >= (3,): key = key.encode('utf-8') @@ -33,7 +45,7 @@ k1, k2) suffix = _get_so_suffixes()[0] self.tmpdir = tmpdir or _caller_dir_pycache() - self.sourcefilename = os.path.join(self.tmpdir, modulename + '.c') + self.sourcefilename = os.path.join(self.tmpdir, modulename + source_extension) self.modulefilename = os.path.join(self.tmpdir, modulename + suffix) self.ext_package = ext_package self._has_source = False @@ -97,6 +109,20 @@ def generates_python_module(self): return self._vengine._gen_python_module + def make_relative_to(self, kwds, relative_to): + if relative_to and os.path.dirname(relative_to): + dirname = os.path.dirname(relative_to) + kwds = kwds.copy() + for key in ffiplatform.LIST_OF_FILE_NAMES: + if key in kwds: + lst = kwds[key] + if not isinstance(lst, (list, tuple)): + raise TypeError("keyword '%s' should be a list or tuple" + % (key,)) + lst = [os.path.join(dirname, fn) for fn in lst] + kwds[key] = lst + return kwds + # ---------- def _locate_module(self): @@ -148,7 +174,10 @@ def _load_library(self): assert self._has_module - return self._vengine.load_library() + if self.flags is not None: + return self._vengine.load_library(self.flags) + else: + return self._vengine.load_library() # ____________________________________________________________ @@ -181,6 +210,9 @@ def _caller_dir_pycache(): if _TMPDIR: return _TMPDIR + result = os.environ.get('CFFI_TMPDIR') + if result: + return result filename = sys._getframe(2).f_code.co_filename return os.path.abspath(os.path.join(os.path.dirname(filename), '__pycache__')) @@ -222,11 +254,7 @@ pass def _get_so_suffixes(): - suffixes = [] - for suffix, mode, type in imp.get_suffixes(): - if type == imp.C_EXTENSION: - suffixes.append(suffix) - + suffixes = _extension_suffixes() if not suffixes: # bah, no C_EXTENSION available. Occurs on pypy without cpyext if sys.platform == 'win32': diff --git a/lib_pypy/greenlet.egg-info b/lib_pypy/greenlet.egg-info --- a/lib_pypy/greenlet.egg-info +++ b/lib_pypy/greenlet.egg-info @@ -1,6 +1,6 @@ Metadata-Version: 1.0 Name: greenlet -Version: 0.4.0 +Version: 0.4.5 Summary: Lightweight in-process concurrent programming Home-page: https://github.com/python-greenlet/greenlet Author: Ralf Schmitt (for CPython), PyPy team diff --git a/lib_pypy/greenlet.py b/lib_pypy/greenlet.py --- a/lib_pypy/greenlet.py +++ b/lib_pypy/greenlet.py @@ -1,7 +1,7 @@ import sys import _continuation -__version__ = "0.4.0" +__version__ = "0.4.5" # ____________________________________________________________ # Exceptions diff --git a/pypy/doc/build.rst b/pypy/doc/build.rst --- a/pypy/doc/build.rst +++ b/pypy/doc/build.rst @@ -47,6 +47,11 @@ Install build-time dependencies ------------------------------- +(**Note**: for some hints on how to translate the Python interpreter under +Windows, see the `windows document`_) + +.. _`windows document`: windows.html + To build PyPy on Unix using the C translation backend, you need at least a C compiler and ``make`` installed. Further, some optional modules have additional diff --git a/pypy/doc/conf.py b/pypy/doc/conf.py --- a/pypy/doc/conf.py +++ b/pypy/doc/conf.py @@ -65,9 +65,9 @@ # built documents. # # The short X.Y version. -version = '2.4' +version = '2.5' # The full version, including alpha/beta/rc tags. -release = '2.4.0' +release = '2.5.0' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/pypy/doc/contributor.rst b/pypy/doc/contributor.rst --- a/pypy/doc/contributor.rst +++ b/pypy/doc/contributor.rst @@ -12,19 +12,19 @@ Amaury Forgeot d'Arc Samuele Pedroni Alex Gaynor + Brian Kearns + Matti Picus + Philip Jenvey Michael Hudson David Schneider - Matti Picus - Brian Kearns - Philip Jenvey Holger Krekel Christian Tismer Hakan Ardo Benjamin Peterson Manuel Jacob + Ronan Lamy Anders Chrigstrom Eric van Riet Paap - Ronan Lamy Wim Lavrijsen Richard Emslie Alexander Schremmer @@ -38,9 +38,9 @@ Camillo Bruni Laura Creighton Toon Verwaest + Romain Guillebert Leonardo Santagada Seo Sanghyeon - Romain Guillebert Justin Peel Ronny Pfannschmidt David Edelsohn @@ -61,15 +61,16 @@ Michal Bendowski Jan de Mooij stian + Tyler Wade Michael Foord Stephan Diehl - Tyler Wade Stefan Schwarzer Valentino Volonghi Tomek Meka Patrick Maupin Bob Ippolito Bruno Gola + David Malcolm Jean-Paul Calderone Timo Paulssen Squeaky @@ -78,18 +79,19 @@ Marius Gedminas Martin Matusiak Konstantin Lopuhin + Wenzhu Man John Witulski - Wenzhu Man + Laurence Tratt + Ivan Sichmann Freitas Greg Price Dario Bertini Mark Pearse Simon Cross - Ivan Sichmann Freitas Andreas Stührk + Stefano Rivera Jean-Philippe St. Pierre Guido van Rossum Pavel Vinogradov - Stefano Rivera Paweł Piotr Przeradowski Paul deGrandis Ilya Osadchiy @@ -99,7 +101,6 @@ tav Taavi Burns Georg Brandl - Laurence Tratt Bert Freudenberg Stian Andreassen Wanja Saatkamp @@ -111,13 +112,12 @@ Jeremy Thurgood Rami Chowdhury Tobias Pape - David Malcolm Eugene Oden Henry Mason Vasily Kuznetsov Preston Timmons + David Ripton Jeff Terrace - David Ripton Dusty Phillips Lukas Renggli Guenter Jantzen @@ -136,13 +136,16 @@ Gintautas Miliauskas Michael Twomey Lucian Branescu Mihaila + Yichao Yu Gabriel Lavoie Olivier Dormond Jared Grubb Karl Bartel + Wouter van Heyst Brian Dorsey Victor Stinner Andrews Medina + anatoly techtonik Stuart Williams Jasper Schulz Christian Hudon @@ -152,12 +155,11 @@ Michael Cheng Justas Sadzevicius Gasper Zejn - anatoly techtonik Neil Shepperd + Stanislaw Halik Mikael Schönenberg Elmo M?ntynen Jonathan David Riehl - Stanislaw Halik Anders Qvist Corbin Simpson Chirag Jadwani @@ -166,10 +168,13 @@ Vincent Legoll Alan McIntyre Alexander Sedov + Attila Gobi Christopher Pope Christian Tismer Marc Abramowitz Dan Stromberg + Arjun Naik + Valentina Mukhamedzhanova Stefano Parmesan Alexis Daboville Jens-Uwe Mager @@ -183,8 +188,6 @@ Sylvain Thenault Nathan Taylor Vladimir Kryachko - Arjun Naik - Attila Gobi Jacek Generowicz Alejandro J. Cura Jacob Oscarson @@ -192,22 +195,23 @@ Ryan Gonzalez Ian Foote Kristjan Valur Jonsson + David Lievens Neil Blakey-Milner Lutz Paelike Lucio Torre Lars Wassermann - Valentina Mukhamedzhanova Henrik Vendelbo Dan Buch Miguel de Val Borro Artur Lisiecki Sergey Kishchenko - Yichao Yu Ignas Mikalajunas Christoph Gerum Martin Blais Lene Wagner Tomo Cocoa + Toni Mattis + Lucas Stadler roberto@goyle Yury V. Zaytsev Anna Katrina Dominguez @@ -235,23 +239,30 @@ Stephan Busemann Rafał Gałczyński Christian Muirhead + Berker Peksag James Lan shoma hosaka - Daniel Neuh?user - Matthew Miller + Daniel Neuhäuser + Ben Mather + halgari + Boglarka Vezer + Chris Pressey Buck Golemon Konrad Delong Dinu Gherman Chris Lambacher coolbutusel...@gmail.com + Jim Baker Rodrigo Araújo - Jim Baker + Nikolaos-Digenis Karagiannis James Robert Armin Ronacher Brett Cannon + Donald Stufft yrttyr aliceinwire OlivierBlanvillain + Dan Sanders Zooko Wilcox-O Hearn Tomer Chachamu Christopher Groskopf @@ -265,6 +276,7 @@ Markus Unterwaditzer Even Wiik Thomassen jbs + squeaky soareschen Kurt Griffiths Mike Bayer @@ -276,5 +288,6 @@ Anna Ravencroft Dan Crosta Julien Phalip + Roman Podoliaka Dan Loewenherz diff --git a/pypy/doc/cppyy.rst b/pypy/doc/cppyy.rst --- a/pypy/doc/cppyy.rst +++ b/pypy/doc/cppyy.rst @@ -74,7 +74,9 @@ See the :doc:`backend documentation <cppyy_backend>`. A standalone version of Reflex that also provides the dynamically loadable -backend is available for `download`_. +backend is available for `download`_. Note this is currently the only way to +get the dynamically loadable backend, so use this first. + That version, as well as any other distribution of Reflex (e.g. the one that comes with `ROOT`_, which may be part of your Linux distribution as part of the selection of scientific software) will also work for a build with the diff --git a/pypy/doc/embedding.rst b/pypy/doc/embedding.rst --- a/pypy/doc/embedding.rst +++ b/pypy/doc/embedding.rst @@ -30,12 +30,10 @@ Initialize threads. Only need to be called if there are any threads involved -.. function:: long pypy_setup_home(char* home, int verbose); +.. function:: int pypy_setup_home(char* home, int verbose); This function searches the PyPy standard library starting from the given - "PyPy home directory". It is not strictly necessary to execute it before - running Python code, but without it you will not be able to import any - non-builtin module from the standard library. The arguments are: + "PyPy home directory". The arguments are: * ``home``: NULL terminated path to an executable inside the pypy directory (can be a .so name, can be made up) @@ -84,17 +82,22 @@ const char source[] = "print 'hello from pypy'"; - int main() + int main(void) { - int res; + int res; - rpython_startup_code(); - // pypy_setup_home() is not needed in this trivial example - res = pypy_execute_source((char*)source); - if (res) { - printf("Error calling pypy_execute_source!\n"); - } - return res; + rpython_startup_code(); + res = pypy_setup_home("/opt/pypy/bin/libpypy-c.so", 1); + if (res) { + printf("Error setting pypy home!\n"); + return 1; + } + + res = pypy_execute_source((char*)source); + if (res) { + printf("Error calling pypy_execute_source!\n"); + } + return res; } If we save it as ``x.c`` now, compile it and run it (on linux) with:: diff --git a/pypy/doc/faq.rst b/pypy/doc/faq.rst --- a/pypy/doc/faq.rst +++ b/pypy/doc/faq.rst @@ -3,6 +3,13 @@ .. contents:: +See also: `Frequently ask questions about RPython.`__ + +.. __: http://rpython.readthedocs.org/en/latest/faq.html + +--------------------------- + + What is PyPy? ------------- @@ -190,6 +197,55 @@ (now-dead) object are still true about the new object. + +Would type annotations help PyPy's performance? +----------------------------------------------- + +Two examples of type annotations that are being proposed for improved +performance are `Cython types`__ and `PEP 484 - Type Hints`__. + +.. __: http://docs.cython.org/src/reference/language_basics.html#declaring-data-types +.. __: https://www.python.org/dev/peps/pep-0484/ + +**Cython types** are, by construction, similar to C declarations. For +example, a local variable or an instance attribute can be declared +``"cdef int"`` to force a machine word to be used. This changes the +usual Python semantics (e.g. no overflow checks, and errors when +trying to write other types of objects there). It gives some extra +performance, but the exact benefits are unclear: right now +(January 2015) for example we are investigating a technique that would +store machine-word integers directly on instances, giving part of the +benefits without the user-supplied ``"cdef int"``. + +**PEP 484 - Type Hints,** on the other hand, is almost entirely +useless if you're looking at performance. First, as the name implies, +they are *hints:* they must still be checked at runtime, like PEP 484 +says. Or maybe you're fine with a mode in which you get very obscure +crashes when the type annotations are wrong; but even in that case the +speed benefits would be extremely minor. + +There are several reasons for why. One of them is that annotations +are at the wrong level (e.g. a PEP 484 "int" corresponds to Python 3's +int type, which does not necessarily fits inside one machine word; +even worse, an "int" annotation allows arbitrary int subclasses). +Another is that a lot more information is needed to produce good code _______________________________________________ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit