[pypy-commit] pypy default: Add the Win64 plan in the back of my mind.
Author: Armin Rigo Branch: Changeset: r66274:422233ea1b89 Date: 2013-08-21 09:40 +0200 http://bitbucket.org/pypy/pypy/changeset/422233ea1b89/ Log:Add the Win64 plan in the back of my mind. diff --git a/pypy/doc/windows.rst b/pypy/doc/windows.rst --- a/pypy/doc/windows.rst +++ b/pypy/doc/windows.rst @@ -6,6 +6,10 @@ The following text gives some hints about how to translate the PyPy interpreter. +PyPy supports only being translated as a 32bit program, even on +64bit Windows. See at the end of this page for what is missing +for a full 64bit translation. + To build pypy-c you need a C compiler. Microsoft Visual Studio is preferred, but can also use the mingw32 port of gcc. @@ -63,7 +67,7 @@ INCLUDE, LIB and PATH (for DLLs) environment variables appropriately. Abridged method (for -Ojit builds using Visual Studio 2008) -~~ +~~~ Download the versions of all the external packages from https://bitbucket.org/pypy/pypy/downloads/local.zip @@ -112,13 +116,14 @@ nmake -f makefile.msc The sqlite3 database library - + Download http://www.sqlite.org/2013/sqlite-amalgamation-3071601.zip and extract it into a directory under the base directory. Also get http://www.sqlite.org/2013/sqlite-dll-win32-x86-3071601.zip and extract the dll into the bin directory, and the sqlite3.def into the sources directory. Now build the import library so cffi can use the header and dll:: + lib /DEF:sqlite3.def" /OUT:sqlite3.lib" copy sqlite3.lib path\to\libs @@ -206,8 +211,68 @@ March 2012, --cc is not a valid option for pytest.py. However if you set an environment variable CC to the compliter exe, testing will use it. -.. _'mingw32 build': http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Automated%20Builds +.. _`mingw32 build`: http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Automated%20Builds .. _`mingw64 build`: http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Automated%20Builds .. _`msys for mingw`: http://sourceforge.net/projects/mingw-w64/files/External%20binary%20packages%20%28Win64%20hosted%29/MSYS%20%2832-bit%29 .. _`libffi source files`: http://sourceware.org/libffi/ .. _`RPython translation toolchain`: translation.html + + +What is missing for a full 64-bit translation +- + +The main blocker is that we assume that the integer type of RPython is +large enough to (occasionally) contain a pointer value cast to an +integer. The simplest fix is to make sure that it is so, but it will +give the following incompatibility between CPython and PyPy on Win64: + +CPython: ``sys.maxint == 2**32-1, sys.maxsize == 2**64-1`` + +PyPy: ``sys.maxint == sys.maxsize == 2**64-1`` + +...and, correspondingly, PyPy supports ints up to the larger value of +sys.maxint before they are converted to ``long``. The first decision +that someone needs to make is if this incompatibility is reasonable. + +Assuming that it is, the fixes are probably not too much work if the +goal is only to get a translated PyPy executable and to run tests with +it --- and not care about running all the tests of PyPy before +translation. To do that, the only tests that you should run (and start +with) are some tests in rpython/translator/c/test/, like +``test_standalone.py`` and ``test_newgc.py``. Keep in mind that this +runs small translations, and some details may go wrong, running on top +of CPython Win64; notably, constant integer values should be allowed up +to ``2**63-1``, but any value larger than ``2**32-1`` will be considered +out of bound. To fix this, you need to explicitly wrap such large +integers e.g. in the class ``r_longlong`` of rpython.rlib.rarithmetic. +This makes the translation toolchain handle them as longlong, which +have the correct range, even though in the end it is the same type, +i.e. a 64-bit integer. + +What is really needed is to review all the C files in +rpython/translator/c/src for the word ``long``, because this means a +32-bit integer even on Win64. Replace it with ``Signed``, and check the +definition of ``Signed``: it should be equal to ``long`` on every other +platforms (so you can replace one with the other without breaking +anything on other platforms), and on Win64 it should be something like +``long long``. + +These two types have corresponding RPython types: ``rffi.LONG`` and +``lltype.Signed`` respectively. Add tests that check that integers +casted to one type or the other really have 32 and 64 bits respectively, +on Win64. + +Once these basic tests work, you need to review ``pypy/module/*/`` for +usages of ``rffi.LONG`` versus ``lltype.Signed``. Some other places +might need a similar review too, like ``rpython/rlib/``. Important: at +this point the goal would not be to run the tests in
[pypy-commit] pypy default: Mention early work (hi Christian)
Author: Armin Rigo Branch: Changeset: r66275:bc66a4bf350e Date: 2013-08-21 09:45 +0200 http://bitbucket.org/pypy/pypy/changeset/bc66a4bf350e/ Log:Mention early work (hi Christian) diff --git a/pypy/doc/windows.rst b/pypy/doc/windows.rst --- a/pypy/doc/windows.rst +++ b/pypy/doc/windows.rst @@ -268,7 +268,11 @@ this point the goal would not be to run the tests in these directories! Doing so would create more confusion to work around. Instead, the goal would be to fix some ``LONG-versus-Signed`` issues, and if necessary -make sure that the tests still run fine e.g. on Win32. +make sure that the tests still run fine e.g. on Win32. There was some +early work done notably in ``rpython/rlib/rarithmetic`` with the goal of +running all the tests on Win64, but I think by now that it's a bad idea: +again, we should only make sure that the tests work on Win32, and that +PyPy translates on Win64 and then run the (standard lib-python) tests. This should get you a translation of PyPy with ``-O2``, i.e. without the JIT. Check carefully the warnings of the C compiler at the end. I ___ pypy-commit mailing list pypy-commit@python.org http://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy default: Two new SSE 4 instructions
Author: Armin Rigo Branch: Changeset: r66276:8de0c4e1a787 Date: 2013-08-21 09:59 +0200 http://bitbucket.org/pypy/pypy/changeset/8de0c4e1a787/ Log:Two new SSE 4 instructions diff --git a/rpython/translator/c/gcc/trackgcroot.py b/rpython/translator/c/gcc/trackgcroot.py --- a/rpython/translator/c/gcc/trackgcroot.py +++ b/rpython/translator/c/gcc/trackgcroot.py @@ -489,7 +489,7 @@ 'pabs', 'pack', 'padd', 'palign', 'pand', 'pavg', 'pcmp', 'pextr', 'phadd', 'phsub', 'pinsr', 'pmadd', 'pmax', 'pmin', 'pmovmsk', 'pmul', 'por', 'psadb', 'pshuf', 'psign', 'psll', 'psra', 'psrl', -'psub', 'punpck', 'pxor', +'psub', 'punpck', 'pxor', 'pmovzx', 'pmovsx', # all vectors don't produce pointers 'v', # sign-extending moves should not produce GC pointers ___ pypy-commit mailing list pypy-commit@python.org http://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy stmgc-static-barrier: Add a XXX about that case
Author: Armin Rigo Branch: stmgc-static-barrier Changeset: r66277:a4d5e4fccf79 Date: 2013-08-21 11:26 +0200 http://bitbucket.org/pypy/pypy/changeset/a4d5e4fccf79/ Log:Add a XXX about that case diff --git a/rpython/translator/stm/writebarrier.py b/rpython/translator/stm/writebarrier.py --- a/rpython/translator/stm/writebarrier.py +++ b/rpython/translator/stm/writebarrier.py @@ -186,6 +186,8 @@ for v, cat in category.items(): if cat == 'W': category[v] = 'V' +# XXX the V2W barrier is only necessary when we're +# writing pointers, not if we're writing ints effectinfo = stmtransformer.write_analyzer.analyze( op, graphinfo=graphinfo) ___ pypy-commit mailing list pypy-commit@python.org http://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy default: Develop and use other potential ideas
Author: Armin Rigo Branch: Changeset: r66278:c6dd660901e1 Date: 2013-08-21 13:18 +0200 http://bitbucket.org/pypy/pypy/changeset/c6dd660901e1/ Log:Develop and use other potential ideas diff --git a/pypy/doc/windows.rst b/pypy/doc/windows.rst --- a/pypy/doc/windows.rst +++ b/pypy/doc/windows.rst @@ -262,10 +262,9 @@ casted to one type or the other really have 32 and 64 bits respectively, on Win64. -Once these basic tests work, you need to review ``pypy/module/*/`` for -usages of ``rffi.LONG`` versus ``lltype.Signed``. Some other places -might need a similar review too, like ``rpython/rlib/``. Important: at -this point the goal would not be to run the tests in these directories! +Once these basic tests work, you need to review ``rpython/rlib/`` for +usages of ``rffi.LONG`` versus ``lltype.Signed``. Important: at this +point the goal would not be to run the tests in these directories! Doing so would create more confusion to work around. Instead, the goal would be to fix some ``LONG-versus-Signed`` issues, and if necessary make sure that the tests still run fine e.g. on Win32. There was some @@ -274,9 +273,27 @@ again, we should only make sure that the tests work on Win32, and that PyPy translates on Win64 and then run the (standard lib-python) tests. -This should get you a translation of PyPy with ``-O2``, i.e. without the -JIT. Check carefully the warnings of the C compiler at the end. I -think that MSVC is "nice" in the sense that by default a lot of -mismatches of integer sizes are reported as warnings. +The goal here is to get a translation of PyPy with ``-O2`` with a +minimal set of modules, starting with ``--no-allworkingmodules``. Check +carefully the warnings of the C compiler at the end. I think that MSVC +is "nice" in the sense that by default a lot of mismatches of integer +sizes are reported as warnings. -This should be your first long-term goal. Happy hacking :-) +Why first try to translate when the modules ``pypy/module/*/`` may need +fixes too? The idea is that you really need to get a minimal translated +PyPy, with the minimal amount of modules (this used to be with the +``--translationmodules`` option, if it still works). Then we have a +Python interpreter, namely this minimal PyPy, which can run a full +translation and which has the "correct" setting of ``sys.maxint`` and +64-bit integers. So once we get this minimal PyPy we can use it to +translate a complete PyPy with less troubles. (We still need to review +e.g. ``rffi.LONG`` / ``lltype.Signed`` issues, obviously.) + +Alternatively, you might try to hack CPython to have ints store a 64-bit +number and ``sys.maxint`` be 2**63-1. This might be easier, and work as +long as you don't try too hard to crash it because of the precision loss +that undoubtedly occurs everywhere. Running the translation with such a +hacked CPython would give the same effect as running it on top of the +minimal PyPy described above. + +Happy hacking :-) ___ pypy-commit mailing list pypy-commit@python.org http://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy default: Add a note.
Author: Armin Rigo Branch: Changeset: r66279:1d3e8ab7f4dd Date: 2013-08-21 13:21 +0200 http://bitbucket.org/pypy/pypy/changeset/1d3e8ab7f4dd/ Log:Add a note. diff --git a/pypy/doc/windows.rst b/pypy/doc/windows.rst --- a/pypy/doc/windows.rst +++ b/pypy/doc/windows.rst @@ -294,6 +294,10 @@ long as you don't try too hard to crash it because of the precision loss that undoubtedly occurs everywhere. Running the translation with such a hacked CPython would give the same effect as running it on top of the -minimal PyPy described above. +minimal PyPy described above. (Note that it's ok to do that: once we get +a full PyPy, we can simply tell people that future translations must be +run on top of that. We end up with a strange kind of dependency, but +I believe it's ok here, as Windows executables are supposed to never be +broken by newer versions of Windows.) Happy hacking :-) ___ pypy-commit mailing list pypy-commit@python.org http://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy refactor-str-types: Fix comparison of W_BytesObject with W_StringBufferObject.
Author: Manuel Jacob Branch: refactor-str-types Changeset: r66281:a25bcb6612ef Date: 2013-08-21 15:55 +0200 http://bitbucket.org/pypy/pypy/changeset/a25bcb6612ef/ Log:Fix comparison of W_BytesObject with W_StringBufferObject. diff --git a/pypy/objspace/std/bytesobject.py b/pypy/objspace/std/bytesobject.py --- a/pypy/objspace/std/bytesobject.py +++ b/pypy/objspace/std/bytesobject.py @@ -566,31 +566,49 @@ return space.wrap(StringBuffer(self._value)) def descr_eq(self, space, w_other): +from pypy.objspace.std.strbufobject import W_StringBufferObject +if isinstance(w_other, W_StringBufferObject): +return space.newbool(self._value == w_other.force()) if not isinstance(w_other, W_BytesObject): return space.w_NotImplemented return space.newbool(self._value == w_other._value) def descr_ne(self, space, w_other): +from pypy.objspace.std.strbufobject import W_StringBufferObject +if isinstance(w_other, W_StringBufferObject): +return space.newbool(self._value != w_other.force()) if not isinstance(w_other, W_BytesObject): return space.w_NotImplemented return space.newbool(self._value != w_other._value) def descr_lt(self, space, w_other): +from pypy.objspace.std.strbufobject import W_StringBufferObject +if isinstance(w_other, W_StringBufferObject): +return space.newbool(self._value < w_other.force()) if not isinstance(w_other, W_BytesObject): return space.w_NotImplemented return space.newbool(self._value < w_other._value) def descr_le(self, space, w_other): +from pypy.objspace.std.strbufobject import W_StringBufferObject +if isinstance(w_other, W_StringBufferObject): +return space.newbool(self._value <= w_other.force()) if not isinstance(w_other, W_BytesObject): return space.w_NotImplemented return space.newbool(self._value <= w_other._value) def descr_gt(self, space, w_other): +from pypy.objspace.std.strbufobject import W_StringBufferObject +if isinstance(w_other, W_StringBufferObject): +return space.newbool(self._value > w_other.force()) if not isinstance(w_other, W_BytesObject): return space.w_NotImplemented return space.newbool(self._value > w_other._value) def descr_ge(self, space, w_other): +from pypy.objspace.std.strbufobject import W_StringBufferObject +if isinstance(w_other, W_StringBufferObject): +return space.newbool(self._value >= w_other.force()) if not isinstance(w_other, W_BytesObject): return space.w_NotImplemented return space.newbool(self._value >= w_other._value) ___ pypy-commit mailing list pypy-commit@python.org http://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy refactor-str-types: Fix.
Author: Manuel Jacob Branch: refactor-str-types Changeset: r66280:8360b8f1a263 Date: 2013-08-21 12:47 +0200 http://bitbucket.org/pypy/pypy/changeset/8360b8f1a263/ Log:Fix. diff --git a/pypy/objspace/std/bytesobject.py b/pypy/objspace/std/bytesobject.py --- a/pypy/objspace/std/bytesobject.py +++ b/pypy/objspace/std/bytesobject.py @@ -611,7 +611,7 @@ from pypy.objspace.std.strbufobject import W_StringBufferObject builder = StringBuilder() builder.append(self._value) -builder.append(w_other._value) +builder.append(self._op_val(space, w_other)) return W_StringBufferObject(builder) return self._StringMethods_descr_add(space, w_other) ___ pypy-commit mailing list pypy-commit@python.org http://mail.python.org/mailman/listinfo/pypy-commit
[pypy-commit] pypy optmodel-refactor: Progress on building new sort of bytecode - intermediate checkin (I just
Author: Maciej Fijalkowski
Branch: optmodel-refactor
Changeset: r66282:597c1b69f100
Date: 2013-08-21 17:39 +0200
http://bitbucket.org/pypy/pypy/changeset/597c1b69f100/
Log:Progress on building new sort of bytecode - intermediate checkin (I
just need a point of reference)
diff --git a/rpython/jit/codewriter/assembler.py
b/rpython/jit/codewriter/assembler.py
--- a/rpython/jit/codewriter/assembler.py
+++ b/rpython/jit/codewriter/assembler.py
@@ -36,7 +36,7 @@
self.fix_labels()
self.check_result()
if jitcode is None:
-jitcode = JitCode(ssarepr.name)
+jitcode = JitCode(ssarepr.name, self._count_jitcodes)
jitcode._ssarepr = ssarepr
self.make_jitcode(jitcode)
if self._count_jitcodes < 20:# stop if we have a lot of them
diff --git a/rpython/jit/codewriter/call.py b/rpython/jit/codewriter/call.py
--- a/rpython/jit/codewriter/call.py
+++ b/rpython/jit/codewriter/call.py
@@ -22,6 +22,7 @@
self.cpu = cpu
self.jitdrivers_sd = jitdrivers_sd
self.jitcodes = {} # map {graph: jitcode}
+self.alljitcodes = [] # list of all jitcodes
self.unfinished_graphs = []# list of graphs with pending jitcodes
self.callinfocollection = CallInfoCollection()
if hasattr(cpu, 'rtyper'): # for tests
@@ -167,8 +168,9 @@
'%s has _gctransformer_hint_close_stack_' % (graph,))
#
fnaddr, calldescr = self.get_jitcode_calldescr(graph)
-jitcode = JitCode(graph.name, fnaddr, calldescr,
+jitcode = JitCode(graph.name, len(self.jitcodes), fnaddr,
calldescr,
called_from=called_from)
+self.alljitcodes.append(jitcode)
self.jitcodes[graph] = jitcode
self.unfinished_graphs.append(graph)
return jitcode
diff --git a/rpython/jit/codewriter/jitcode.py
b/rpython/jit/codewriter/jitcode.py
--- a/rpython/jit/codewriter/jitcode.py
+++ b/rpython/jit/codewriter/jitcode.py
@@ -8,8 +8,10 @@
_empty_r = []
_empty_f = []
-def __init__(self, name, fnaddr=None, calldescr=None, called_from=None):
+def __init__(self, name, number=-1, fnaddr=None, calldescr=None,
+ called_from=None):
self.name = name
+self.number = number
self.fnaddr = fnaddr
self.calldescr = calldescr
self.is_portal = False
diff --git a/rpython/jit/metainterp/blackhole.py
b/rpython/jit/metainterp/blackhole.py
--- a/rpython/jit/metainterp/blackhole.py
+++ b/rpython/jit/metainterp/blackhole.py
@@ -1608,10 +1608,10 @@
def resume_in_blackhole(metainterp_sd, jitdriver_sd, resumedescr, deadframe,
all_virtuals=None):
-from rpython.jit.metainterp.resume import blackhole_from_resumedata
+from rpython.jit.metainterp.resume2 import blackhole_from_resumedata
#debug_start('jit-blackhole')
blackholeinterp = blackhole_from_resumedata(
-metainterp_sd.blackholeinterpbuilder,
+metainterp_sd,
jitdriver_sd,
resumedescr,
deadframe,
diff --git a/rpython/jit/metainterp/compile.py
b/rpython/jit/metainterp/compile.py
--- a/rpython/jit/metainterp/compile.py
+++ b/rpython/jit/metainterp/compile.py
@@ -133,7 +133,8 @@
[ResOperation(rop.LABEL, jumpargs, None,
descr=jitcell_token)]
try:
-optimize_trace(metainterp_sd, part, enable_opts)
+optimize_trace(metainterp_sd, metainterp.resume_bc.boxes, part,
+ enable_opts)
except InvalidLoop:
return None
target_token = part.operations[0].getdescr()
@@ -160,7 +161,7 @@
jumpargs = part.operations[-1].getarglist()
try:
-optimize_trace(metainterp_sd, part, enable_opts)
+optimize_trace(metainterp_sd, allboxes, part, enable_opts)
except InvalidLoop:
return None
@@ -212,7 +213,7 @@
orignial_label = label.clone()
assert label.getopnum() == rop.LABEL
try:
-optimize_trace(metainterp_sd, part, jitdriver_sd.warmstate.enable_opts)
+optimize_trace(metainterp_sd, allboxes, part,
jitdriver_sd.warmstate.enable_opts)
except InvalidLoop:
# Fall back on jumping to preamble
target_token = label.getdescr()
@@ -222,7 +223,8 @@
[ResOperation(rop.JUMP, inputargs[:],
None, descr=loop_jitcell_token)]
try:
-optimize_trace(metainterp_sd, part,
jitdriver_sd.warmstate.enable_opts,
+optimize_trace(metainterp_sd, allboxes, part,
+ jitdriver_sd.warmstate.enable_opts,
inline_short_preamble=False)
except InvalidLoop:
return None
@@ -630,6 +632,7 @@
def copy_all_attributes_into(self, res):
# XXX a bit ugly to have to list them all here
[pypy-commit] pypy gc-del-2: in-progress
Author: Armin Rigo Branch: gc-del-2 Changeset: r66283:c37e00e43d40 Date: 2013-08-21 19:04 +0200 http://bitbucket.org/pypy/pypy/changeset/c37e00e43d40/ Log:in-progress diff --git a/rpython/memory/gc/minimark.py b/rpython/memory/gc/minimark.py --- a/rpython/memory/gc/minimark.py +++ b/rpython/memory/gc/minimark.py @@ -321,8 +321,9 @@ # finalizers, they are made young again. self.young_objects_not_in_nursery = self.null_address_dict() # -# A list of all objects with finalizers (these are never young). -self.objects_with_finalizers = self.AddressDeque() +# 4 lists of all objects with finalizers, young or old, light or not. +self.young_objects_with_finalizers = self.AddressDeque() +self.old_objects_with_finalizers = self.AddressDeque() self.young_objects_with_light_finalizers = self.AddressStack() self.old_objects_with_light_finalizers = self.AddressStack() # @@ -526,18 +527,10 @@ totalsize = size_gc_header + size rawtotalsize = raw_malloc_usage(totalsize) # -# If the object needs a finalizer, ask for a rawmalloc. -# The following check should be constant-folded. -if needs_finalizer and not is_finalizer_light: -ll_assert(not contains_weakptr, - "'needs_finalizer' and 'contains_weakptr' both specified") -obj = self.external_malloc(typeid, 0, can_make_young=False) -self.objects_with_finalizers.append(obj) -# # If totalsize is greater than nonlarge_max (which should never be # the case in practice), ask for a rawmalloc. The following check # should be constant-folded. -elif rawtotalsize > self.nonlarge_max: +if rawtotalsize > self.nonlarge_max: ll_assert(not contains_weakptr, "'contains_weakptr' specified for a large object") obj = self.external_malloc(typeid, 0) @@ -559,14 +552,19 @@ # Build the object. llarena.arena_reserve(result, totalsize) obj = result + size_gc_header -if is_finalizer_light: -self.young_objects_with_light_finalizers.append(obj) self.init_gc_object(result, typeid, flags=0) # # If it is a weakref, record it (check constant-folded). if contains_weakptr: self.young_objects_with_weakrefs.append(obj) # +# More checks for recording, constant-folded +if needs_finalizer: +if is_finalizer_light: +self.young_objects_with_light_finalizers.append(obj) +else: +self.young_objects_with_finalizers.append(obj) +# return llmemory.cast_adr_to_ptr(obj, llmemory.GCREF) @@ -683,9 +681,9 @@ collect_and_reserve._dont_inline_ = True -def external_malloc(self, typeid, length, can_make_young=True): -"""Allocate a large object using the ArenaCollection or -raw_malloc(), possibly as an object with card marking enabled, +def external_malloc(self, typeid, length): +"""Allocate a large object using raw_malloc(), +possibly as an object with card marking enabled, if it has gc pointers in its var-sized part. 'length' should be specified as 0 if the object is not varsized. The returned object is fully initialized and zero-filled.""" @@ -722,28 +720,9 @@ self.minor_collection() self.major_collection(raw_malloc_usage(totalsize)) # -# Check if the object would fit in the ArenaCollection. -if raw_malloc_usage(totalsize) <= self.small_request_threshold: -# -# Yes. Round up 'totalsize' (it cannot overflow and it -# must remain <= self.small_request_threshold.) -totalsize = llarena.round_up_for_allocation(totalsize) -ll_assert(raw_malloc_usage(totalsize) <= - self.small_request_threshold, - "rounding up made totalsize > small_request_threshold") -# -# Allocate from the ArenaCollection and clear the memory returned. -result = self.ac.malloc(totalsize) -llmemory.raw_memclear(result, totalsize) -# -# An object allocated from ArenaCollection is always old, even -# if 'can_make_young'. The interesting case of 'can_make_young' -# is for large objects, bigger than the 'large_objects' threshold, -# which are raw-malloced but still young. -extra_flags = GCFLAG_TRACK_YOUNG_PTRS -# -else: -# No, so proceed to allocate it externally with raw_malloc(). +# (preserves the indentation of the following block) +if 1: +# Allocate the object externally with raw_malloc(). # Check if we need to introdu
[pypy-commit] pypy incremental-gc: added some failing tests
Author: Andrew Chambers
Branch: incremental-gc
Changeset: r66284:74b07286c87d
Date: 2013-08-22 13:35 +1200
http://bitbucket.org/pypy/pypy/changeset/74b07286c87d/
Log:added some failing tests
diff --git a/rpython/memory/gc/incminimark.py b/rpython/memory/gc/incminimark.py
--- a/rpython/memory/gc/incminimark.py
+++ b/rpython/memory/gc/incminimark.py
@@ -130,13 +130,24 @@
# by the incremental collection
GCFLAG_GRAY = first_gcflag << 8
-# The following flag is just an alias for the gray flag. It
-# is only used by major collections, it is set on objects
-# which are allocated during the sweeping and finalization states
-# it has different meaning outside of the sweeping state.
-# This flag should not be reset by any minor collection operation
-GCFLAG_NOSWEEP = first_gcflag << 8
+# This flag allows sweeping to be incrementalised.
+# it is set when an object would be swept, but isnt
+# because this flag was not set. The point of this
+# flag is to make sure an object has survived through
+# at least one major collection so we are sure
+# it is unreachable. It is needed because a write
+# barrier has no way of knowing which objects are truly
+# unvisited, or they were simply already reset by
+# a sweep.
+GCFLAG_CANSWEEP = first_gcflag << 9
+# Flag indicates object is old. It is needed by the
+# write barrier code so that we can track when a young
+# reference is written into a black object.
+# we must make a shadow and prevent such an object from being freed by
+# the next minor collection so that we dont get dead objects in
+# objects_to_trace during marking.
+GCFLAG_OLD = first_gcflag << 10
# States for the incremental GC
@@ -155,7 +166,7 @@
-TID_MASK= (first_gcflag << 9) - 1
+TID_MASK= (first_gcflag << 11) - 1
FORWARDSTUB = lltype.GcStruct('forwarding_stub',
@@ -1636,6 +1647,11 @@
self._visit_young_rawmalloced_object(obj)
return
#
+
+# Do this after check we are old to avoid cache misses like
+# In the comment above.
+self.header(obj).tid |= GCFLAG_OLD
+
size_gc_header = self.gcheaderbuilder.size_gc_header
if self.header(obj).tid & GCFLAG_HAS_SHADOW == 0:
#
@@ -1708,7 +1724,7 @@
hdr = self.header(obj)
if hdr.tid & GCFLAG_VISITED:
return
-hdr.tid |= GCFLAG_VISITED
+hdr.tid |= (GCFLAG_VISITED|GCFLAG_OLD)
#
# we just made 'obj' old, so we need to add it to the correct lists
added_somewhere = False
@@ -1931,6 +1947,7 @@
def free_rawmalloced_object_if_unvisited(self, obj):
if self.header(obj).tid & GCFLAG_VISITED:
+self.header(obj).tid |= GCFLAG_OLD
self.header(obj).tid &= ~(GCFLAG_VISITED|GCFLAG_GRAY) # survives
self.old_rawmalloced_objects.append(obj)
else:
diff --git a/rpython/memory/gc/test/test_direct.py
b/rpython/memory/gc/test/test_direct.py
--- a/rpython/memory/gc/test/test_direct.py
+++ b/rpython/memory/gc/test/test_direct.py
@@ -640,24 +640,149 @@
newobj1 = self.malloc(S)
newobj2 = self.malloc(S)
newobj1.x = 1337
-#newobj2.x = 1338
+newobj2.x = 1338
self.write(oldobj,'next',newobj1)
self.gc.debug_gc_step_until(incminimark.STATE_SCANNING)
#should not be cleared even though it was allocated while sweeping
assert newobj1.x == 1337
-#assert newobj2.x == 1338
+# XXX find appropriate exception type
+assert py.test.raises(RuntimeError,"newobj2.x")
-def test_new_marking_write_sweeping(self):
-
-assert False
+#def test_new_marking_write_sweeping(self):
+#
+#assert False
-def test_finalizing_new_object(self):
+#def test_finalizing_new_object(self):
# Must test an object with a finalizer
# being added just before finalizers start being called
# must test this new objects finalizer is not called
# XXX maybe cant do this in test_direct and need test_transformed
-assert False
+#assert False
+def test_young_gray_collected(self):
+from rpython.memory.gc import incminimark
+
+# Test the write barrier triggers on a young object
+# but doesnt crash when that object is collected
+
+for i in range(2):
+curobj = self.malloc(S)
+curobj.x = i
+self.stackroots.append(curobj)
+
+
+self.gc.debug_gc_step_until(incminimark.STATE_MARKING)
+
+#process one object
+self.gc.debug_gc_step()
+
+oldobj = self.stackroots[-1]
+
+newobj = self.malloc(S)
+newobj.x = 5
+# make newobj gray
+self.write(oldobj,'next',newobj)
+#the barrier should have made the object gray
+newhdr = self.gc.header(llmemory.cast_ptr_to_adr(
