Re: [pypy-dev] Pypy bugs
Hi Elefterios, On Sun, May 8, 2011 at 12:18 PM, Elefterios Stamatogiannakis est...@gmail.com wrote: You are exactly right, and i've more or less used code like the one you presented. My only problem is that i believe that pypy shouldn't have answered at my first example (In [12]:...) above that cElementTree exists in xml.etree .Or maybe it should short circuit cElementTree to point at ElementTree. To declare that something exists, and then when somebody tries to use it, to throw an exception, isn't that nice. Nothing we can do: the module cElementTree is actually written in Python also in CPython --- it is cElementTree.py --- but it import internally _elementtree. So there is no way PyPy can know that cElementTree cannot be imported before actually trying to. The workaround to write try: import cXxx; except ImportError: import xxx is the general solution. It works in other cases too (cProfile, cPickle, cStringIO, etc). And also, it works in case the version of CPython you are running on misses some C extension modules. For example you would get the exact same error as you got on PyPy if you are running on a CPython where the C module _elementree was not compiled. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] rlb.parsing left recursion support ?
Hi Gustavo, On Sun, May 8, 2011 at 9:39 AM, Gustavo Kuerten rubyco...@gmail.com wrote: I noticed that there is a test in test_pypackrat.py which seems to suggest that the parser supports left recursion. a: a : | b; I cannot find this test anywhere. Can you tell more precisely where it is from? I looked in pypy/rlib/parsing/test/test_pypackrat.py. So, does it or does it not support left recursion ? It does not, as far as I know. Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] rlb.parsing left recursion support ?
Hi Gustavo, On Sun, May 8, 2011 at 2:53 PM, Gustavo Kuerten rubyco...@gmail.com wrote: This seems to indicate that it does support left recursion. Ah, sorry. I have no real clue, so I can just answer from looking at the source code, which I'm sure you did already for longer than me. There seems to be two classes called PackratParser. You are getting the error from the one in parsing.py, but test_pypackrat uses the one in makepackrat.py. Even if you want to use the class in parsing.py, it takes an argument check_for_left_recursion which you can try setting to False. Well, it's probably obvious to you; if so, sorry not to be more useful. Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy Assembler SQRT Patch
Hi Joe, On Sat, May 7, 2011 at 6:15 AM, Joe qbpro...@gmail.com wrote: My mistake. There was a bug so it appeared as though SQRTSD wasn't being used while running the tests. It turns out the first patch I posted probably worked. After benchmarking, SQRTSD is definitely the cause of the speedup. Sorry for the confusion. Sorry, the confusion probably came from my side. I missed that running the tests with interp_operations() would not set supports_floats=True :-( That's fixed now. Thank you again for your patience. Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Allow subclassing `staticmethod` and `classmethod`
Hi Cool-RR, On Tue, May 3, 2011 at 1:38 PM, cool-RR cool...@cool-rr.com wrote: Cool. Is there gonna be a 1.5.1 release with stuff like this? Yes, very likely. Note that the 1.5 release is just one of the nightly binaries :-) Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy Assembler SQRT Patch
Hi, On Tue, May 3, 2011 at 1:20 PM, Maciej Fijalkowski fij...@gmail.com wrote: Sorry it took so long to review, long weekend and whatnot. The test for x86 backend actually fails for -1.0 on 64bit. Can you reproduce/deny? I fixed it, it was really a bug about FINISH(ConstFloat(...)). But the patch, at least as applied now, does not have the intended effect; SQRTSD is never generated because ll_math_sqrt() is called with a residual call. Joe, can you fix it? I can try to if you don't have the time. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy Assembler SQRT Patch
Hi Alex, On Tue, May 3, 2011 at 6:23 PM, Alex Gaynor alex.gay...@gmail.com wrote: Another thing is it would be useful to test that the SQRTSD is generated, I don't know a good way to do that though. In the test, you can hack genop_math_sqrt() in-place to set a flag, and test that the flag is set... And another place it should also be tested is in test_pypy_c_new.py (pypy.module.pypyjit.test_pypy_c), i.e. you should write a test that checks that in a full translated pypy-c, the code corresponding to the Python source math.sqrt(x) indeed ends up being a CALL(sqrt_wrapper) and not a CALL(ll_math_sqrt). A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] cpyext: Detecting pypy and other issues
Hi Roger, On Mon, May 2, 2011 at 6:55 AM, Roger Binns rog...@rogerbinns.com wrote: The pypy site also doesn't say where to communicate for cpyext - I assumed this mailing list was the right place. Yes. static PyObject *themodule=Py_InitModule3(...); Later in one of the methods called back from Python I call PyObject_AttrString(themodule, foo) which is crashing. If I incref themodule in init then this problem doesn't occur. Indeed, it's an edge case. What you must realize is that your module object is of course kept alive in PyPy, even if its refcount drops to zero. More precisely, in PyPy, every object accessed by the C extensions is actually accessed via a wrapper, which is declared as PyObject. A wrapper holds just the refcount and a reference to the real object, which is not a PyObject at all (it's an object that can be moved by our GC, to start with). When the refcount drops to zero we assume that the C extension doesn't need the object any more, and we free the PyObject wrapper. This may or may not cause the real object within PyPy to be freed. It may even be the case that later we want again to pass a reference to the same object to the C extension; in this case, we make a new wrapper. So the refcount of -- in your case -- the module object is never zero; it's just that as soon as the main function of your C extension module finishes, the refcount is decremented and reaches zero, so the PyObject wrapper is freed. It is arguably a bug in your module, even though in that case it is hard to write an example that triggers the bug, because it is impossible in CPython to free a loaded extension module. The point is that if it is was possible, then doing so in CPython would cause the module to see its refcount drop to zero, and be freed, which makes the pointer in the static global variable invalid. That's what I meant by saying that such issues are, somehow, showing bugs in the C extension module, although admittedly they are often bugs that cannot possibly cause an issue in practice. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] virtualenv 1.6.1 now released
Hi, If you are using virtualenv, note that PyPy 1.5 requires virtualenv 1.6.1. It has been released a few hours *after* PyPy 1.5. For those who did not find it, it's there now :-) Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] cpyext: Detecting pypy and other issues
Hi Roger, A first warning: even if you manage to compile your C extension with PyPy, the interfacing is much slower than with CPython. Keep that in mind, given that according to what you write you care enough about performance to try to squeeze a few percents. On Sun, May 1, 2011 at 12:08 PM, Roger Binns rog...@rogerbinns.com wrote: It looks like PYPY_VERSION being #defined will work but I'd rather use the official way whatever that is. That's the official way. I need the api PyRun_StringFlags as it is used to execute a Python string and attach the results to the module. If it's not already provided, then you are welcome to contribute it to PyPy in pypy/module/cpyext/. It is known that we don't have 100% coverage even of Python.h (not to mention other headers like compile.h, which will probably never be supported). Do I have to add an incref for everything that has a static pointer to it? It depends on details of exactly what you do, but it is likely. The exact rule is, like CPython, that any object is freed as soon as its reference counter goes down to zero; but unlike CPython, this does *not* include places like a reference stored in the module's dictionary. In other words, if it seems to work on CPython but not on PyPy, then it is likely that there is actually a (potentially very rare) bug in the CPython version too. For example, the object you have in the static variable could be meant to be kept alive by being also stored in the module's dictionary. If so, what if a user actually delete it from your module's dictionary? Then you get a crash on CPython too. My next problem is a list allocated with PyList_New(0). I later call PyList_GET_SIZE on it and get a SIGSEGV. I put an assertion before that of PyList_Check but that crashes too. Unfortunately I can't really do much with the pypy/gdb output: You get easier-to-read tracebacks if you run it without the JIT: pypy --jit threshold=-1. Of course in order to get the details of, say, the local variables, then you need a custom pypy built with debugging symbols. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] virtualenv 1.6.1 now released
Hi all, On Sun, May 1, 2011 at 5:22 PM, Armin Rigo ar...@tunes.org wrote: I can only guess that you have some older virtualenv or some older pypy around. For reference, I just installed now the pypy 1.5 release as the default pypy on tannit and tannit32. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] virtualenv 1.6.1 now released
Hi Holger, On Sun, May 1, 2011 at 8:07 PM, holger krekel hol...@merlinux.eu wrote: i use pypy 1.5 from opt. However, /usr/bin/virtualenv on tannit is at version 1.4.5 and maybe that is the problem. I assumed that you got virtualenv 1.6.1, because you replied to my message that was saying precisely virtualenv 1.6.1 is needed, use that version. :-) I will update it on tannit too. Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] PyPy 1.5 released
== PyPy 1.5: Catching Up == We're pleased to announce the 1.5 release of PyPy. This release updates PyPy with the features of CPython 2.7.1, including the standard library. Thus all the features of `CPython 2.6`_ and `CPython 2.7`_ are now supported. It also contains additional performance improvements. You can download it here: http://pypy.org/download.html What is PyPy? = PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7.1. It's fast (`pypy 1.5 and cpython 2.6.2`_ performance comparison) due to its integrated tracing JIT compiler. This release includes the features of CPython 2.6 and 2.7. It also includes a large number of small improvements to the tracing JIT compiler. It supports Intel machines running Linux 32/64 or Mac OS X. Windows is beta (it roughly works but a lot of small issues have not been fixed so far). Windows 64 is not yet supported. Numerous speed achievements are described on `our blog`_. Normalized speed charts comparing `pypy 1.5 and pypy 1.4`_ as well as `pypy 1.5 and cpython 2.6.2`_ are available on our benchmark website. The speed improvement over 1.4 seems to be around 25% on average. More highlights === - The largest change in PyPy's tracing JIT is adding support for `loop invariant code motion`_, which was mostly done by Håkan Ardö. This feature improves the performance of tight loops doing numerical calculations. - The CPython extension module API has been improved and now supports many more extensions. For information on which one are supported, please refer to our `compatibility wiki`_. - These changes make it possible to support `Tkinter and IDLE`_. - The `cProfile`_ profiler is now working with the JIT. However, it skews the performance in unstudied ways. Therefore it is not yet usable to analyze subtle performance problems (the same is true for CPython of course). - There is an `external fork`_ which includes an RPython version of the ``postgresql``. However, there are no prebuilt binaries for this. - Our developer documentation was moved to Sphinx and cleaned up. (click 'Dev Site' on http://pypy.org/ .) - and many small things :-) Cheers, Carl Friedrich Bolz, Laura Creighton, Antonio Cuni, Maciej Fijalkowski, Amaury Forgeot d'Arc, Alex Gaynor, Armin Rigo and the PyPy team .. _`CPython 2.6`: http://docs.python.org/dev/whatsnew/2.6.html .. _`CPython 2.7`: http://docs.python.org/dev/whatsnew/2.7.html .. _`our blog`: http://morepypy.blogspot.com .. _`pypy 1.5 and pypy 1.4`: http://bit.ly/joPhHo .. _`pypy 1.5 and cpython 2.6.2`: http://bit.ly/mbVWwJ .. _`loop invariant code motion`: http://morepypy.blogspot.com/2011/01/loop-invariant-code-motion.html .. _`compatibility wiki`: https://bitbucket.org/pypy/compatibility/wiki/Home .. _`Tkinter and IDLE`: http://morepypy.blogspot.com/2011/04/using-tkinter-and-idle-with-pypy.html .. _`cProfile`: http://docs.python.org/library/profile.html .. _`external fork`: https://bitbucket.org/alex_gaynor/pypy-postgresql ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] stackless over ctypes - threading
Hi Brett, On Mon, Apr 25, 2011 at 9:45 AM, Hart's Antler bhart...@yahoo.com wrote: I have an example of using a compiled Rpython extension module in Python using stackless (rcoroutines). From ctypes i can safely read the values of a shared-object (shared between CPython and RPython), but i can not in a thread-safe way set values on the shared object so that RPython can see them. I was wondering what is the best way to handle the thread locking. Sorry, no clue. We don't know exactly what you are doing, but I can tell you that poking around with CPython's ctypes in the GC objects of a translated PyPy will just get you randomness. It might work a bit by chance with today's version of PyPy (e.g. as long as there are not multithreading issues, as long as you are on 32-bits and not 64-bits or vice-versa, as long as you use only the default GC options, etc.). It's not something that we want to support. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] cpyext reference counting and other gc's
Hi Dima, On Mon, Apr 25, 2011 at 9:53 PM, Dima Tisnek dim...@gmail.com wrote: https://docs.google.com/document/d/1k7t-WIsfKW4tIL9i8-7Y6_9lo18wcsibyDONOF2i_l8/edit?hl=en Can you explain a bit more what are the advantages of the solution you propose, compared to what is already implemented in cpyext? Your description is far too high-level for us to know exactly what you mean. It seems that you want to replace the currently implemented solution with a very different one. I can explain it in a bit more details, but I would first like to hear what goal you are trying to achieve. Here is a quick reply based on guessing. The issue with your version is that Py_INCREF() and Py_DECREF() needs to do a slow dictionary lookup, while ours doesn't. Conversely, I believe that your version doesn't need a dictionary lookup in other cases where ours needs to. However it seems to me that if you add so much overhead to Py_INCREF() and Py_DECREF(), you loose all other speed advantages. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] Release 1.5 preparation
Hi all, We are starting to prepare the release PyPy 1.5. Note that we are doing it in the default branch of mercurial, because it's more convenient to run tests (notably codespeed's). So please, *any* development that you don't explicitly intend to be included in the 1.5 release must go to the branch post-release-1.5, which we will merge back with default after the release is done. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Problem with large allocation in test
Hi Vincent, On Tue, Apr 19, 2011 at 2:02 PM, Vincent Legoll vincent.leg...@gmail.com wrote: Wouldn't a big read only mmap from /dev/zero before the to be jumped adress allow for not really trying to allocate the big chunk of RAM ? That may be a good fix, still allowing the test for far jumps... It's not needed; mmap() can optionally take a position argument, allowing us to choose the position of the small pieces of requested memory. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Problem with large allocation in test
Hi, On Thu, Apr 21, 2011 at 12:33 PM, p...@pocketnix.org wrote: i am assuming this is the correct course of action, if i should instead not be replacing the original alloc function and just create a new alloc_with_hint function and keep/use both let me know Indeed, you should keep the original function too: the hint parameter is not used anywhere else in the source code, but it's convenient for debugging because it means that the JIT-generated code will be allocated at a known address, instead of randomly. That's this function's purpose. I'm fine if you create another function. Or if you prefer, you can change the existing function to take an optional base_addr argument. This arguments would default to NULL, and if it is NULL then the function would use the hintp as it does now. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Problem with large allocation in test
Hi Joe, On Tue, Apr 19, 2011 at 8:23 AM, Maciej Fijalkowski fij...@gmail.com wrote: Anyway, after I went into ll2ctypes.py and set far_regions to True This is a correct workaround, indeed. As you found out the problem comes from the fact that your OS reserves actual RAM from mmap() eagerly. As Maciej points out, it's useful to test e.g. jumps over more than +/-2GB. But the same result could potentially be obtained in a different way: instead of allocating a single block of 20GB, we could mmap 10 blocks of (say) 20MB for the test, but taking care of placing the blocks so that they start 2GB apart from each other. This could be done using pypy.rlib.rmmap.alloc() or a variant. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] thinking about the EuroPython sprint
Hi Laura, On Tue, Apr 12, 2011 at 12:13 AM, Laura Creighton l...@openend.se wrote: yes, I was hoping to catch Alex Gaynor for that, as he is attending EuroDjango earlier the same month. My point is that we need to decide what we are doing very, very soon. else everybody will have their plane tickets already, since we will have to announce what we are doing differently. Then what about a short week's sprint before the conference, plus the two days after? Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] translationmodules option failing
Re-hi, On Tue, Apr 12, 2011 at 12:34 PM, p...@pocketnix.org wrote: - struct, _md5, cStringIO, array])) + struct, _md5, cStringIO, array, thread])) Uh, we can't import ctypes if we have no thread module? That looks obscure and should be fixed... Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Waf benchmark
Hi Maciej, On Mon, Apr 11, 2011 at 11:53 AM, Maciej Fijalkowski fij...@gmail.com wrote: I propose the waf benchmark removal. Given that its speed is at 1 for CPython, 1 for PyPy without JIT, 1 for PyPy with JIT, it seems rather pointless indeed, at least for us. Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] translationmodules option failing
Hi, On Mon, Apr 11, 2011 at 1:44 PM, p...@pocketnix.org wrote: - struct, md5, cStringIO, array])) + struct, _md5, cStringIO, array])) Thanks! Applied. Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] File overwriting (--output flag to translate.py)
Re-Hi,
On Mon, Apr 11, 2011 at 2:45 PM, p...@pocketnix.org wrote:
+ # Ensure the file does not exisit else we fail at end of translation
+ if os.path.isfile(drv.exe_name):
+ raise ValueError('File ' + drv.exe_name+ ' already exisits
(--output)')
Sadly everyone so far has his own additional hacks to categorize
multiple translated versions. Mine is to ignore the pypy-c entirely
and copy the executable from the /tmp, after it has been produced
there. I also copy the C sources (but not the other files produced by
gcc). Anyway, my point is that the particular change you are
proposing would actually harm me, because I always have a pypy-c and
I'm fine if it gets overwritten by every translation :-)
We need to think of some better solution...
Armin
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Pre sprint beer session Sunday 24th of April
Hi Bea, On Mon, Apr 11, 2011 at 8:27 PM, Antonio Cuni anto.c...@gmail.com wrote: session at Bishops Arms pub in central Gothenburg, Sunday 24th of April, Good, it will be nice to see you again Bea! Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] thinking about the EuroPython sprint
Hi Laura, On Mon, Apr 11, 2011 at 3:40 PM, Laura Creighton l...@openend.se wrote: 2 days after the conference is not a lot of time. Do we want to rent some space to have a longer sprint? Or is it too late, people will already have booked their plane tickets, or ... As far as I'm concerned, I think a sprint should be a bit longer to be useful. Wasn't there also talk about having the sprint (or *a* sprint) done before the conference? Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Thoughts on multithreading in PyPy
Hi, On Sun, Apr 10, 2011 at 1:33 PM, p...@pocketnix.org wrote: What i am wondering about is if some of the base services provided by pypy can be moved into another thread, eg GC and JIT compilation and how much of a benfiit there would be to doing so It would be possible to move the GC or the JIT to another thread, introducing a lot of complexity but keeping it hidden to the Python programmer. You would end up with a program that can use maybe 1.2 core instead of just 1. That sounds like a lot of work for a minimal gain. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] jit.backend.test.test_random
Hi Matthew, On Sun, Apr 10, 2011 at 2:46 PM, Matthew Woodcraft matt...@woodcraft.me.uk wrote: + cpu.setup_once() Thanks, fixed. We did not notice this failure because pypy/jit/backend/x86/test/test_zll_random.py provides its own interface over calling test_random.py directly with --backend=x86. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Thoughts on multithreading in PyPy
Hi Jacob, On Sat, Apr 9, 2011 at 10:50 PM, Jacob Hallén ja...@openend.se wrote: So, in a second step, we provide for special data types that can be shared between threads. These would typically be allocated in non-movable memory, to avoid the complexity of garbage collection of memory with shared use. You can make simple fifo structures for communication between the threads and complex structures with advanced algorithms for dealing with shared access. That's where the real issue is. You can come up with some reasonable API to communicate with other threads, but precisely, they will be some API, which means that they will only work in programs written specifically to use them. Designing a new API (at the level of the Python language) is something we carefully avoided so far in PyPy; but it's possible that this issue is important enough for us to break that rule :-) What you are describing sounds similar to the multiprocessing module in CPython, which achieves the same goal using separated processes (and tons and tons of hacks), and requires the program to use a custom API. The advantage of doing it in PyPy rather than CPython is probably limited to the fact that it would be easier in PyPy (but still some work) to make sure that the multiple threads have no shared state. You still have to design some custom API. There is also another possible goal with more pypy-like goals and results, which would be to use some technique to weave a solution in the interpreter transparently for the Python programmer (so, a solution that works without requiring the Python programmer to learn another system than threads). I can imagine a Software Transactional Memory solution that would in theory work very nicely, but in practice have completely dreadful performance, because it would do large amounts of checked memory access for each bytecode. As far as I know it means that that approach does not work, but it may one day, if Hardware Transactional Memory really shows up and supports that scale. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy in the benchmarks game - yes or no?
Hi, On Tue, Apr 5, 2011 at 2:23 AM, Isaac Gouy igo...@yahoo.com wrote: A simple yes / no question. Do you want PyPy to be shown in the benchmarks game or not? Sorry for missing general knowledge of what you mean by benchmarks game. According to Google, it is probably what I know as the language shootout (shootout.alioth.debian.org). Is it the case? Assuming it is, then my position is pretty much the same as William Leslie: I have little interest if the benchmarks that PyPy runs with are written in completely non-idiomatic ways, super hand-optimized for CPython, for the reasons explained in detail by William. It would be interesting, however, if you accept versions rewritten in just plain Python. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy in the benchmarks game - yes or no?
Hi Isaac, If you're not interested in this discussion and just want a Yes or No answer to precisely the following question: should PyPy be added to the benchmarks game's site, with no changes whatsoever to any of the Python benchmarks that are there so far? then the answer is No. We are not interested in the results of PyPy on such skewed benchmarks, and publishing them would only add to the general confusion. Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy in the benchmarks game - yes or no?
Hi Dima, On Tue, Apr 5, 2011 at 9:30 PM, Dima Tisnek dim...@gmail.com wrote: A more pragmatic approach is to go through the list of benchmarks and either accept current python code for pypy or fail on purpose if python code is too un-pypy-like. Later on, hopefully, someone volunteers to rewrite problem benchmarks. Several of us already went down that road. However, as I said, the unresolved issue right now is if the maintainer (Isaac?) is willing to accept or not any new, simpler, more portable version of the benchmarks programs. If not, then that's the end of this discussion, I fear. Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] minimum system requirements, build configuration
Hi Dima, On Tue, Apr 5, 2011 at 9:52 PM, Dima Tisnek dim...@gmail.com wrote: What I mean to say is that there's gotta be a more clever way where gc thresholds depend on e.g. size of working set or rate of new allocations or something yet smarter. Yes, it does in PyPy; we do a full collection when the total amount of data reaches 1.82 times (by default) the amount of live data at the end of the previous collection, with additional tweaks to improve performance in various cases -- and one such tweak is to set the minimum threshold to 16MB (actually, now I think it is not fixed to 16MB but it depends on the amount of L2 cache). It was all reached by various benchmarks on various desktop machines, including the minimum threshold. You can see the various thresholds and their defaults at the start of pypy/rpython/memory/gc/minimark.py (and that's only if you are using minimark, our default GC). Of course all the numbers -- and even half of the algorithms -- are going to be bogus if you start to think about very different machines. That's what I meant when I said that there is open work to do, and you or anyone with an interest in the area (and corresponding hardware) is welcome to attack it. A bientôt, Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Pypy custom interpreter JIT question
Hi Andrew, On Mon, Mar 28, 2011 at 7:21 PM, Andrew Brown brow...@gmail.com wrote: When the optimizer encounters a pure function, it must compare the objects promote - promote the argument from a variable into a constant. Could this be an appropriate alternate to the @purefunction solution? Or, I'm guessing, does it just mean the name bracket_map won't change bindings, but does not impose a restriction on mutating the dictionary? One point of view on 'promote' is to mean this variable was red, but now turn it green (i.e. make it constant). It has no effect on a variable that is already green (= a constant). We have no support for considering that a dict is immutable, so it needs to be done with @purefunction. But to be effective, @purefunction must receive constant arguments; so in one or two places in the source code of PyPy you will find a construction like: x = hint(x, promote=True) # turn x into a constant some_pure_function(x) # call this pure function on x Indeed, Carl Friedrich's blog series explains it nicely, but it should also mention that when the hints described in the blog are applied not to integer but to pointers, they apply only to the pointers themselves, not on the fields of the objects they point to. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] Next sprint
Hi all, The next sprint will most probably occur in Gothenburg, April 25th to May 1st. A proper sprint announcement should follow shortly. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Runtime module loading
Hi Timothy, On Fri, Mar 25, 2011 at 4:00 PM, Timothy Baldridge tbaldri...@gmail.com wrote: I have a program written in RPython. Is there a built-in way to import other RPython modules at runtime? No, and that's not obviously fixed; for example, the GC relies on having a table of all the RPython types in the program. In order to load dynamically new RPython extension modules, we would need to do something about that. Nothing unsolvable, but there are quite a few such places around. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Thinking about the GIL
Hi, On Fri, Mar 18, 2011 at 1:44 AM, hyar...@iinet.net.au hyar...@iinet.net.au wrote: (...) Wrapping each bytecode in an STM transaction would give you an as-if-serial execution order, again with no guarantees about which order. You get transaction overheads instead of lock/unlock overheads, but some STM systems can be quite efficient for short transactions that rarely conflict. Yes, I also thought about this as one of the solutions that would fit the model of PyPy by not needing changes all over the place. However, I am unsure that the performance of STM is good enough for that application so far. Maybe I'm wrong, but I fear (a priori, with no precise experience) that it would be really too slow to wrap *all* memory reads and writes with the STM machinery. I would be interested in learning if I'm wrong, or if there are hardware solutions around the corner ready to be tried. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Pypy custom interpreter JIT question
Hi Andrew, On Fri, Mar 25, 2011 at 5:47 PM, Andrew Brown brow...@gmail.com wrote: Thanks, it does indeed work now! The next step is to have a look at the traces produced (run with PYPYLOG=jit-log-opt:logfile), and spot the obvious missing optimizations. The biggest issue seems to be the fact that the dictionary 'bracket_map' is green, but it is not enough to ensure that it is a constant dict (it could be mutated behind the JIT's back); so in the end, every trace contains reads from it. You could fix it by moving the line newpc = bracket_map[pc] to a new function to which you apply the decorator @pypy.rlib.jit.pure_function. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Pypy custom interpreter JIT question
Hi Andrew, On Wed, Mar 23, 2011 at 8:27 PM, Andrew Brown brow...@gmail.com wrote: However, the version that didn't work before still does not run correctly. It seems like I'm still left with the same problem as before. Ah. Looking more closely, it turns out to be a bug in the optimization step of the JIT which just never showed up so far :-/ Working on it, by cleaning up optimizeopt/heap.py... Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Pypy custom interpreter JIT question
Re-hi, On Thu, Mar 24, 2011 at 5:11 PM, Armin Rigo ar...@tunes.org wrote: Working on it, by cleaning up optimizeopt/heap.py... Done, at least as far as fixing the bug is concerned. Now your original version (with can_enter_jit removed) works. Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Pypy custom interpreter JIT question
Hi Andrew, On Tue, Mar 22, 2011 at 8:44 PM, Andrew Brown brow...@gmail.com wrote: https://bitbucket.org/brownan/bf-interpreter/src/c4679b354313/targetbf.py can_enter_jit() is not correct. For it to work, it must be called just before jit_merge_point(). It's wrong that there are two intermediate instructions here: pc+=1 and the pc len(program) condition. As a first attempt, you should just not call can_enter_jit() at all. Nowadays, if can_enter_jit is never called, it's done automatically for you; moreover, a misplaced can_enter_jit can give nonsensical results, as opposed to many other hints, which cannot give a result worst than terribly bad performance (like the greens/reds variable separation --- which seems correct in your example). A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] pypy fold_intadd: Changed test to reflect my optimizeopt's decision to emit int_sub(iX, -x) when x 0
Hi Anto, hi Ademan, On Sat, Mar 19, 2011 at 10:07 AM, Antonio Cuni anto.c...@gmail.com wrote: what happens when you are on 32 bit and see int_add(i0, -2147483648)? It probably works, because -(-2147483648) == -2147483648, and int_add(i0, -2147483648) and int_sub(i0, -2147483648) do the same thing. However I have no clue why this replacement is giving you anything. If anything at all I would say that replacing int_add(i0, -128) with int_sub(i0, 128) is a (very marginally) bad idea on x86 because the constant -128 fits in 8 bits but the constant 128 doesn't. Well, I suppose that it makes it easier for me, the human, to understand it is a valid reason after all. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] GC Tuning script values
Hi Chris, On Thu, Mar 17, 2011 at 4:34 PM, Chris Mulligan chris.mulli...@gmail.com wrote: Just installed PyPy on my Macbook Pro per Bob Ippolito's instructions. Very easy! Thanks for the feedback. The results make it clear that we should somehow tune the number according to the load of the machine -- picking up the right number for the load can easily make a 20% speed difference (at least on Mac OS X, but I strongly suspect the same is true on other platforms). Ideally, it should dynamically adapt its nursery size in order to minimize the cache misses. If anyone has a suggestion on how to implement that, preferably in a non-OS-specific way (e.g. by reading some x86 CPU counters), I'd welcome it :-) A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] wrong precedence of __radd__ vs list __iadd__
Hi Philip, On Sat, Mar 12, 2011 at 5:21 PM, Philip Jenvey pjen...@underboss.org wrote: Jython passes the first example. Shouldn't pypy's inplace_add__List_ANY be returning NotImplemented (or whatever the pypy equiv. would be, FailedToImplement?) instead of raising the exception? To allow the binop rules to continue. Good idea. Right now, both on CPython and on PyPy, [].__iadd__(5) raises TypeError (instead of returning NotImplemented), but [].__add__(5) already behaves differently: it raises TypeError on CPython and returns NotImplemented on PyPy, with the result that []+Bar() does call the __radd__() method on Bar(). So having the same difference in the __iadd__() method looks like a useful compromise. This is particularly true given that it already works on all sequence types different from lists, because these don't have an __iadd__() method at all. Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] wrong precedence of __radd__ vs list __iadd__
Hi, Checked in 72ce40f4803c. Greg, about the second example you reported: class C(object): def __rmul__(self, other): other *= 2 return other def __index__(self): return 3 print [1] * C() # - CPython: [1,1] PyPy: [1,1,1] This is obscure. It works that way because in that particular case, CPython first tries __mul__/__rmul__ and then tries the internal sq_repeat slot of the list (which calls C.__index__()). I think there is just no way to reproduce this behavior while remaining sane :-/ Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] tooling/sandboxing (remarks from Ikai Lan)
Hi Holger, Yes, that's what all three representatives of alternate VMs said at the panel: sandboxing at the VM level is cool and supported, but we don't want sandboxing at the language level. As far as I can tell, Ikan Lan just misunderstood it and got the opposite message. Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] wrong precedence of __radd__ vs list __iadd__
Hi Greg, On Wed, Mar 9, 2011 at 7:13 PM, Greg Price g...@quora.com wrote: The following program works in CPython, but fails in PyPy: This is (if we are positive) an internal implementation detail and (if we are negative) a bug in CPython. There is no way to define in pure Python a list-like type that would have the exact same behavior. We already have one such special case for __add__/__radd__ on subclasses of strings and unicodes, which appears to be the use case that people rely on most often, but I don't see the point in duplicating this strange behavior for lists, tuples, and so on, unless there are really several programs out there that rely on it. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Yelp RSVP
Hi Dan, On Thu, Mar 3, 2011 at 6:02 PM, Dan Roberts ademan...@gmail.com wrote: Apparently I missed the memo about RSVPing to the Yelp talk, and we're having trouble getting in the door, assuming the situation can be rectified, can someone help out? For (un?)interested readers of this mailing list, the situation was eventually solved. Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] EBNF translation error
Hi Timothy, On Fri, Mar 4, 2011 at 5:14 AM, Timothy Baldridge tbaldri...@gmail.com wrote: I hope this is the right place for this question. ---snip pypy.rpython.error.TyperError: don't know how to convert from InstanceRepr for pypy.rlib.parsing.tree.Node to PyObjRepr * PyObject Basically, it worked all right, except that you are trying to return from the main translated function an instance of Node, which the translator doesn't know how to convert back to Python level. You can only return simple types, like ints, strings, and so on. In real usage, i.e. translating a stand-alone program with pypy/translator/goal/translate.py, the main translated function needs anyway to have a specific signature, similar to the main() function in C code: takes a list of strings (argv), and returns an integer (exit code). A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Couple of questions...
Hi Luke, On Tue, Feb 22, 2011 at 3:11 PM, Luke Taylor l...@freeculture.co.uk wrote: if there are any plans to make PyPy implement Python 3.x ? There are no plans at the moment, though I imagine that such plans will emerge at one point. Indeed, we are almost done with supporting 2.7, which is on the way there. Why don't extension modules for Python with bits written in C (eg. SciPy) work with PyPy? You can't generalize like that; some of them do, thanks to the cpyext module of PyPy. But not SciPy, which is using a lot of fonctionality of CPython that is semi-internal. Moreover, cpyext is slow, so it's rather meant to support C extensions like wxWindow that are not there for performance but for functionality. We can improve performance a bit, but it's unlikely to ever get at a level where it's appropriate for NumPy or SciPy. Instead, what needs to be done there is rewrite some parts in a more PyPy-friendly way. (For NumPy there is a project at http://projects.scipy.org/numpy/wiki/NumPyRefactoring that is likely to be useful for us too.) A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] further pypy repo migrations
Hi Laura, On Thu, Feb 17, 2011 at 5:43 PM, Laura Creighton l...@openend.se wrote: I just found a surprising number (at least it surprised me) of references in my own stuff to /svn/user/arigo/hack/somethings I moved this part of the repository to http://bitbucket.org/arigo/arigo . Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] Fwd: [codespeak-ann] termination of codespeak.net services end FEB 2011
Hi all, It seems that Codespeak is going down sooner rather than later. Here is a copy of the start of holger's announce: -- Forwarded message -- From: holger krekel hol...@merlinux.eu Date: Thu, Feb 10, 2011 at 9:45 AM Subject: [codespeak-ann] termination of codespeak.net services end FEB 2011 To: codespeak-...@codespeak.net hi codespeak.net users, (sorry if you get mail twice, i wanted to make sure ...) after 8 years of operation codespeak.net services are bound to terminate, starting END OF FEBRUARY 2011 As far as PyPy is concerned, we need to move somewhere else: * the mailing lists pypy-dev, pypy-svn, pypy-z (also the history, please :-) * the remainings parts that are in Subversion: at least svn/pypy/extradoc, svn/pypy/benchmarks, svn/pypy/lang, which should all be converted to Mercurial I suppose (help needed here) * the developers web site (from pypy/doc/) * the users web site (http://pypy.org/) * the buildmaster * the issue tracker * the svn/user/xxx directory for people like me that use it a lot * something else? It's not like we weren't warned plenty of times. It still looks like a lot of work. The timing is a bit bad, because I wanted to finish fixing the 2.7 test failures in time to do a PyPy 1.5 release for PyCon, but it looks a bit more unlikely right now :-( (Not too important, we can do a pre-release any time, and should do one soon anyway.) A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Fwd: [codespeak-ann] termination of codespeak.net services end FEB 2011
Hi Laura, On Thu, Feb 10, 2011 at 12:40 PM, Laura Creighton l...@openend.se wrote: We can move things to Open End if people are interested. But there is a lot to be said for moving to a place where the whole hosting matter is somebody else's problem. I agree, but I still think that we will take your offer for *some* of the services I described. At least the buildmaster should be moved to Open End (I doubt there is an existing provider that supports customized buildbot masters like we need). Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] pypy jit
Hi, On Fri, Feb 4, 2011 at 7:44 AM, Maciej Fijalkowski fij...@gmail.com wrote: I disagree a bit. The example would be newstylization of classes. This is making your code more JIT friendly and it's not trivial to optimize this from the JIT standpoint. We know there is an issue there but we don't want to invest the large efforts to improve it because it's really the Python program's fault for using a mixture of deprecated and modern features... Sure, you have a point. I was taking a rather theoretical point of view in my previous e-mail. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Link errors while translating with VS2010 and 64bit
Hi Tasos, On Wed, Feb 2, 2011 at 1:15 AM, Tasos Vogiatzoglou tvog...@gmail.com wrote: You are right. My mistake. It was picking the 2.6 32bit python. Now is working and tests are passing A few extra notes. What is needed to make this work is three-fold: * make sure that in the C code, intptr_t is generated instead of long for lltype.Signed, and similarly uintptr_t instead of unsigned long for lltype.Unsigned. This involves mainly fixing translator/c/primitive.py to declare types using intptr_t instead of long, and probably fixing a number of other uses of long in the .py files in that directory. * make sure that in the hand-written headers translator/c/src/*.h, we use intptr_t instead of long whenever it's needed (mostly everywhere, I suppose, but not absolutely everywhere, e.g. not in calls to external C functions that are declared to take or return a long). The changes above should fix the generation of code. Now the third part is the longest, probably. You need to distinguish the two types used in RPython, which are lltype.Signed and rffi.LONG, and which are equal so far. Of course rffi.LONG should remain equivalent to the C long. This probably needs looking around in all modules and in pypy/rlib/, to make sure that we use the correct one, either lltype.Signed or rffi.LONG. Fortunately, errors resulting from this confusion will probably just give translation-time errors, so we can fix the places one after the other. But you need to carefully check the declaration of external functions that are implemented in translator/c/src/*.h, like pypy/module/signal/interp_signal.py (this one happens to use mostly rffi.INT, so it should work right now, with the exception of LONG_STRUCT that uses a lltype.Signed). A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] pypy jit
Hi, On Wed, Feb 2, 2011 at 4:48 PM, Maciej Fijalkowski fij...@gmail.com wrote: As of now there is no such guide. However, the rule of thumb is simple is better than complex. To some extend that is true; however, note that the second rule is but even on messily complicated code the JIT can do something :-) For example, translate.py was not written with the JIT in mind at all, but turns out to be twice as fast on PyPy. To get a better summary, I think that we can say that, to some extent, there is little point in spending ages tweaking your Python code to make it more JIT-friendly. If there is any Python code that would get seriously faster by some trivial tweaking, then it's somehow a bug, and we need to fix it. Of course, if you use introspection of the stack frames or even pdb (the Python debugger) all over the place, you need to expect the code to be hard to optimize for us. But I guess that it should somehow be clear. For the rest, any decision that can be resolved locally can nicely be optimized by the JIT, whatever the answer is (for example, is this '+' going to just add integers, or does it invoke some __add__() method?). A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] OSError: [Errno 10] No child processes
Hi Amaury, On Wed, Jan 12, 2011 at 2:49 AM, Paolo Giarrusso p.giarru...@gmail.com wrote: I propose that PyPy keeps reporting the error for files opened in any write mode I would also think that it's better to keep reporting errors (and for all files instead of just write mode files). In my opinion, it would, if nothing else, give users the message: we got an error when close()ing this file automatically, but you should really close it explicitly yourself in the first place. Maybe it can even be written in that sense in the error message. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] OSError: [Errno 10] No child processes
Hi, On Thu, Jan 13, 2011 at 1:25 PM, Maciej Fijalkowski fij...@gmail.com wrote: close()ing this file automatically, but you should really close it explicitly yourself in the first place. Maybe it can even be written in that sense in the error message. How about a link to differencies between pypy and cpython, especially about closing files? Yes please :-) Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] OSError: [Errno 10] No child processes
Hi Amaury, On Tue, Jan 11, 2011 at 4:41 PM, Amaury Forgeot d'Arc amaur...@gmail.com wrote: It's probably related to our file.__del__ which does not silence errors raised by close(). Indeed, it seems to be the case. Arnd, to answer the original question: there is a file that you are not explicitly closing, and when its __del__ method is (later) invoked, it raises an OSError which is printed and ignored. The real problem you have there is that you should close your files (in this case, an fdopen file, so I guess it's actually a pipe returned by os.popen*()). The secondary problem that we have is what to do when file.__del__ gets an exception from calling close(). Is it ok to just ignore it? FWIW in CPython, file.__del__ also prints an error message if close() raises an exception. So from that point of view there is nothing to fix in PyPy, apart maybe making the error message a bit more explicit (CPython prints close failed in file object destructor:...). A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] 1.4.1 threading
Hi Paolo, Although I'm sure that your post is very interesting with the right mindset, I have a problem understanding the connection with Python. I know that it's exactly what you are complaining about, i.e. that Python developers appear not to care much about this deep Java/C++ issue. However, if you mean to do anything about it, you need to first understand yourself why it is so -- and the following reason is by far not enough: As a matter of fact, this work is unsurprisingly often totally ignored in many discussions about Python's memory model. I'm not surprised because it's complex stuff, and understanding it doesn't matter for today's Python programming. The most obvious misunderstanding, in my opinion, is that in Java or C++ it's about fields reads and writes, whereas in Python it's about any operation on any built-in type -- which can be for example reading or writing attributes, or insert()ing in a list, or doing setdefault() on a dictionary, or... any complex operation. This means that you cannot *at all* reduce the problem to field reads and field writes. As long as the discussion at the level of Java or C++ is only about fields, it's going to be ignored as mostly uninteresting by the Python folks. On the other hand, the links you posted about nonblockinghashmap are indeed interesting in this context. From a Python point of view, what is left to discuss is whether these hash maps offer enough consistency behavior to be usable on Python's default memory model. Or maybe it's not that interesting any more now that PyPy tends not to use a lot of dictionaries any more (the attributes of objects are not in a hash map, internally). I am myself just expressing vague, uninformed opinions again. But the final point is that this really needs someone motivated to experiment with PyPy (or CPython), and as long as no-one shows up, it will mostly be just moving air around, i.e. wasting time to discuss this too much in depth. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] pypy GC on large objects Re: funding/popularity?
Hi René, On Thu, Dec 23, 2010 at 8:30 PM, Dima Tisnek dim...@gmail.com wrote: Basically collecting this is hard: dict(a=range(9**9)) I think you missed the point of my original email. I was talking about GC-referenced objects that hold a reference to a large piece of memory allocated outside the GC. There is none here, and any GC (including PyPy's) will do a correct job in collecting this. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Building PyPy on Windows
Hi Seung, On Thu, Dec 23, 2010 at 6:51 AM, Seung Soo, sungs...@gmail.com wrote: An interesting article came up on python planet: http://www.redmountainsw.com/wordpress/archives/building-pypy-on-windows I think the official documentation needs to state also how to build it with the missing modules (expat etc.). http://codespeak.net/pypy/dist/pypy/doc/windows.html has some hints, but is not complete as far as I can tell (e.g. it doesn't say how to arrange for the .h files to be found). Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] pypy GC on large objects Re: funding/popularity?
Hi René, On Thu, Dec 23, 2010 at 2:33 PM, René Dudfield ren...@gmail.com wrote: I think this is a case where the object returned by ctypes.create_string_buffer() could use a correct __sizeof__ method return value. If pypy supported that, then the GC's could support extensions, and 'opaque' data structures in C too a little more nicely. I think you are confusing levels. There is no way the GC can call some app-level Python method to get information about the objects it frees (and when would it even call it?). Remember that our GC is written at a level where it works for any interpreter for any language, not just Python. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] funding/popularity?
Hi Paolo, On Thu, Dec 23, 2010 at 3:09 PM, Paolo Giarrusso p.giarru...@gmail.com wrote: From your description, I guess that what you describe as raw memory is not accounted in the stats triggering GC. That would explain the behavior you describe, and suggest an easy fix. Indeed, one could wrap the raw-memory allocator to create and update such stats; then the heap-overflow check could consider them to decide whether to trigger GC. The details of the modified heap-overflow check would probably not be entirely trivial, but still doable. Absolutely. I will write it down in extradoc/planning/ for now. Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] long.__itemsize__
Hi, On Tue, Dec 21, 2010 at 9:58 AM, Arnd Rechenburg arnd.rechenb...@tomtom.com wrote: In my code I need something like long.__itemsize__ in Python. I suppose the question is why? This is supposed to be the size in bytes occupied by one element of the type, and one element of long happens to be 15 bits, so that's why long.__itemsize__ is 2. It has no other meaning. If you are interested in an estimate of how many bytes some actual object takes, we have some functions in the 'gc' module, but even that is incomplete -- there are cases were it's hard to come up with a definitive answer, like objects based on mapdicts. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] long.__itemsize__
Hi René, On Tue, Dec 21, 2010 at 1:24 PM, René Dudfield ren...@gmail.com wrote: sys.getsizeof(obj) Ah, thank you, didn't know about it. It's a 2.6 feature only though. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Interpretor for vectorized langugage
Hi Alex, On Thu, Dec 16, 2010 at 8:28 PM, Alex Gaynor alex.gay...@gmail.com wrote: Regarding this - I was thinking about haveing a + b - c create a bytecode that would be executed using small interpreter with a jit-merge-point and a hint can be vectorized. That seems like a pretty big special case, why not work at the larger idea of cross-loop optimizations? We have cross-loop optimizations. Unless I'm misunderstanding you, we have already done it. That's how for example map(f, somelist) gets JITted: it is a small loop that contains a call to the normal loop of the interpreter for f(). This is JITted as the loop from map() containing inline code from the interpreter loop. I am a bit unsure however what is being discussed here, because fijal's comment is very terse and contains hidden issues, like when to create the bytecode (at runtime? but how can it be a green constant then? And what is the hint can be vectorized?). A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Can't run PyPy due to missing libbz2.so.1.0
Hi Gary, On Fri, Dec 17, 2010 at 5:08 PM, Gary Robinson gary...@me.com wrote: As a quick experiment, I tried temporarily renaming it to /lib/libbz2.so.1.0 but that didn't change the result. It's strange, what I would expect on a system is that there is a symlink from libbz2.so.1.0 to libbz2.so.1.0.0. Adding it should be enough -- did you run ldconfig after trying to rename? It is needed to refresh Linux's caches. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Interpretor for vectorized langugage
Hi, On Wed, Dec 15, 2010 at 6:21 PM, René Dudfield ren...@gmail.com wrote: Is pypy suitable for writing interpretor of vectorized language like Matlab, R etc which vector and matrix are first class objects? This includes matrix shape inference, and efficient linear algebra code generation. have you seen numpy/scipy? The first aspect is simply if RPython would be suitable for writing an interpreter for, say, Matlab. The answer is probably yes: PyPy would be suitable for such dynamic languages, giving you a JIT compiler for free. I don't really know how complex the core of these languages are, but I suspect not too much. Of course you are then going to hit the same problems that Ademan tries to solve for numpy/scipy, notably how to implement at least the basic linear algebra operations in such a way that the JIT can improve them. There are various goals there, e.g. to turn Python (or Matlab) code like A+B+C, adding three matrices together, into one matrix operation instead of two (as it is now: (A+B)+C). This is all a bit experimental so far. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] import errors when calling pypy
Hi Arnd, On Thu, Dec 16, 2010 at 12:58 PM, Arnd Rechenburg arnd.rechenb...@tomtom.com wrote: I have problems to start pypy. It is explained (or rather, at least hinted) here: http://pypy.org/download.html#installing In more words, though, the important point is that you need to *not* split up the zip or tbz2 archive that you are getting. PyPy looks for its libraries starting from the location of the pypy executable. You can put the uncompressed constant of the archive anywhere, but don't move pypy outside it. (It's fine to install a symlink to it from somewhere else.) We may rectify this situation when we have more of a clue about the directory layout that people would really expect, possibly including a Makefile to perform installation, for example. The current situation was deemed sufficient so far. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] error from function.py
Hi,
On Thu, Dec 16, 2010 at 1:35 PM, Arnd Rechenburg
arnd.rechenb...@tomtom.com wrote:
return cdll.ptr(self.name, argshapes, resshape, self._flags_)
AttributeError: 'NoneType' object has no attribute 'ptr'
That's a very indirect way of crashing because
ctypes.util.find_library('c') returned None, i.e. it did not find your
C library. I agree that this could be improved by at least reporting
a clearer error.
To understand the error, first note that it occurs in standard code
from CPython's ctypes.util module. Look in
lib-python/modified-2.5.2/ctypes/util.py. On Linux it seems to call
/sbin/ldconfig -p and look in the result for /libc.*. On my
machine it would first find /lib/libc.so.6 in this way, which is
correct. In general it all looks pretty hackish, but it's not clear
if any particular piece of code is to blame, given the way that the
system of libraries is designed, which is not really adapted for fully
dynamic lookups like needed here...
A bientôt,
Armin.
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] setrecursionlimit deprecation?
Hi Dan, On Sat, Dec 11, 2010 at 8:17 PM, Dan Stromberg drsali...@gmail.com wrote: I originally wrote the code ( http://stromberg.dnsalias.org/~strombrg/treap/ ) for CPython, where the recursion depth was too low out of the box. It sounds like on pypy, I might actually be decreasing the recursion depth. There probably should be a set recursion depth to at least... function. Or I should getrecursiondepth and inspect before increasing. PyPy throws a RuntimeError anyway when the stack is getting too big, i.e. when it uses more than N kilobytes. On the other hand, setrecursiondepth() on CPython (and previously on PyPy) is based on counting the recursion depth, which is a different measure which only approximates the real stack size. We decided that it doesn't make sense to provide both on PyPy, so we killed the one that is an approximation (and makes the JIT a little bit slower). That's all. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Trunk failing to build?
Hi, On Sat, Dec 11, 2010 at 5:46 AM, Benjamin Peterson benja...@python.org wrote: Yep. Hopefully fixed in r79965. Sorry, my fault. Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] setrecursionlimit deprecation?
Hi Benjamin, On Sat, Dec 11, 2010 at 6:10 PM, Benjamin Peterson benja...@python.org wrote: FWIW, I disagree with the deprecation. I think should just be a no-op. But then people are going to complain that their app seems to hang up on pypy, and neither they nor we have any clue what is going on --- until we figure out that they used sys.setrecursionlimit() to put a bound on recursion and catch the RuntimeError. That's at least the reason for which I suggested that calling sys.setrecursionlimit() should at least print a warning. Now I agree that maybe the message of the warning is not the clearest one. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] r79938 - in pypy/trunk/pypy: module/posix module/posix/test rpython/module translator/c/test
Hi, On Fri, Dec 10, 2010 at 2:45 AM, Benjamin Peterson benja...@python.org wrote: 2010/12/9 Maciej Fijalkowski fij...@gmail.com: Do we *really* need those functions available on RPython level? Even if we don't, I'd say the consistency is nice, rather than having ll implementations all over creation. Agreed. Consistency is probably better than having pypy/module/posix be half written in the way it is now and half written in a lltype-specific way. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] Leysin Winter sprint: Jan 16-22
= PyPy Leysin Winter Sprint (16-22nd January 2011) = As announced on http://morepypy.blogspot.com/2010/12/leysin-winter-sprint.html, the next PyPy sprint will be in Leysin, Switzerland, for the seventh time. This is a fully public sprint: newcomers and topics other than those proposed below are welcome. -- Goals and topics of the sprint -- * Now that we have released 1.4, and plan to release 1.4.1 soon (possibly before the sprint), the sprint itself is going to be mainly working on fixing issues reported by various users. Of course this does not prevent people from showing up with a more precise interest in mind. If there are newcomers, we will gladly give introduction talks. * We will also work on polishing and merging the long-standing branches that are around, which could eventually lead to the next PyPy release. These branches are notably: - fast-forward (Python 2.7 support, by Benjamin, Amaury, and others) - jit-unroll-loops (improve JITting of smaller loops, by Hakan) - arm-backend (a JIT backend for ARM, by David) - jitypes2 (fast ctypes calls with the JIT, by Antonio). * And as usual, the main side goal is to have fun in winter sports :-) We can take a day off for ski. --- Exact times --- The work days should be 16-22 January 2011. People may arrive on the 15th already and/or leave on the 23rd. --- Location Accomodation --- Leysin, Switzerland, same place as before. Let me refresh your memory: both the sprint venue and the lodging will be in a very spacious pair of chalets built specifically for bed breakfast: http://www.ermina.ch/. The place has a good ADSL Internet connexion with wireless installed. You can of course arrange your own lodging anywhere (as long as you are in Leysin, you cannot be more than a 15 minutes walk away from the sprint venue), but I definitely recommend lodging there too -- you won't find a better view anywhere else (though you probably won't get much worse ones easily, either :-) Please *confirm* that you are coming so that we can adjust the reservations as appropriate. The rate so far has been around 60 CHF a night all included in 2-person rooms, with breakfast. There are larger rooms too (less expensive) and maybe the possibility to get a single room if you really want to. Please register by svn: http://codespeak.net/svn/pypy/extradoc/sprintinfo/leysin-winter-2011/people.txt or on the pypy-sprint mailing list if you do not yet have check-in rights: http://codespeak.net/mailman/listinfo/pypy-sprint You need a Swiss-to-(insert country here) power adapter. There will be some Swiss-to-EU adapters around -- bring a EU-format power strip if you have one. = Armin Rigo ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] os.major? os.minor?
Hi Dan, On Thu, Dec 9, 2010 at 5:25 PM, Dan Stromberg drsali...@gmail.com wrote: What would be the best way to add os.major and os.minor to the os module? It's amazing the kind of stuff there is in the 'os' module that I've never heard about before. :-) They appear to be C macros on some systems, so I'm guessing most FFI's aren't going to see them without a wrapper. Indeed. See for example pypy.rlib._rsocket_rffi.FD_ZERO, which is also typically a macro. There is a bit of workaround code to write, I'm afraid. This is mostly to support direct testing; if the goal is only to have it work after translation, then it does not matter that it is actually a macro. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] os.major? os.minor?
Re-hi, I have opened an issue about the missing functions from the 'os' module: https://codespeak.net/issue/pypy-dev/issue605 I will probably get around to implement a part of them in the next few days. If you plan to do more than os.major() and os.minor(), let's coordinate there. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] os.major? os.minor?
Hi, Sorry for the series of mails. I just wanted to add that after looking more at the code, it's fine if you implement major() and minor() the same way as the other posix functions (see e.g. r79941 adding os.fchdir()), calling the major() and minor() functions in C. If you add tests in the same places I did in r79941, then it should work even if the functions are actually macros. One place where it would not work is in translated code running on the llinterp; the other place might be the JIT trying to get the address of the function. (If that's the case, we should fix that a bit more generally.) A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Windows source link
Hi Seung, On Fri, Dec 3, 2010 at 11:19 AM, Seung Soo, sungs...@gmail.com wrote: http://pypy.org/download/pypy-1.4-src.zip Down for everyone or is it just me? Works for me. Try again? A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] pyopengl on pypy
Hi Renaud, On Wed, Dec 1, 2010 at 2:21 PM, renaud blanch rndbl...@gmail.com wrote: We'd need to find a mechanism in PyPy that would give us that direct memory-pointer access to be able to use it. Note: a compacting garbage collector (or anything else that can move memory locations) will cause problems there, so we may need to find a way to signal PyPy not to move a given object, and to use contiguous data-arrays for their storage. Indeed, this part from Mike Fletcher got the cause right, but the solution we recommend is different. It's kind of impossible in PyPy to mark objects as non-moving (pin them, as the garbage collection language goes). Instead you need to just use the official ctypes API, create_string_buffer(). It's equivalent, but requires one extra copy of the data. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] test_all.py -A behavior
Hi Dan, On Sat, Nov 27, 2010 at 5:47 AM, Dan Roberts ademan...@gmail.com wrote: I'm having trouble running the applevel tests on a translated pypy-c (without jit). I have this problem with recent trunk and my psycopg2 compatibility branch. Translating both with: python translate.py targetpypystandalone.py --withmod-cpyext and running tests with: translator/goal/pypy-c test_all.py -A We run these tests nightly, and they seem to pass. You'll have to come to IRC to discuss this, or else post much more details about your problem -- e.g. does it work in a plain trunk without your compatibility branch? A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] r79480 - pypy/trunk/pypy/jit/tool
Hi Anto, On Thu, Nov 25, 2010 at 9:24 AM, Maciej Fijalkowski fij...@gmail.com wrote: For the same reason why we don't put extradoc in checkout I guess. I tend to agree with fijal that this file should be in extradoc/somepath... Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] r79382 - pypy/trunk/pypy/config
Hi, On Wed, Nov 24, 2010 at 1:19 AM, exar...@twistedmatrix.com wrote: I guess it should just be another log category, like jit-summary. Then you can get the effect you want by setting (in the shell) PYPYLOG to jit-summary:-. This or an env variable. The PYPYLOG solution I described *is* an environment variable. What do you mean? A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Memory consumption of pypy-c translate.py
Hi, (Answering here in addition to on IRC, for reference.) On Wed, Nov 10, 2010 at 9:53 PM, Maciej Fijalkowski fij...@gmail.com wrote: * 31M of mmaped code blocks for assembler * about 150M I can account for that is jit resume data * about 150M of the rest I can account for. Where is everything else? Can GC trash some memory (like a lot). Everything else is memory that has been freed, but not returned to the OS. (You get the same effect with CPython 2.5.) That's a bit unexpected, because we call the C-level free() on it, but it is still not returned to the OS; it seems to be caused by only a few mallocs() that occur from time to time and stay alive for long (e.g. a few 4KB pages needed by the GC for its various AddressStacks). I verified it in a custom C program that does the same calls to malloc() and free(). If at the end we force free() on the remaining few 4KB blocks, then the total memory use suddenly drops from (in this example) more than 200MB to almost 0. This seems to be the main difference between the way pypy-c and CPython 2.6 call the C-level malloc() and free(). Unsure how to fix it. (If the same program then asks for more memory again, it will reuse this memory -- the libc malloc() at least ensures this, of course.) A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] gdbm
Hi, On Tue, Nov 16, 2010 at 5:27 PM, Dan Stromberg drsali...@gmail.com wrote: So, if I understand correctly you are saying that we should rename our dbm.py to bsdb.py, and write a new dbm.py which can use either bsdb or gdbm? I think it's anydbm that can use whatever among dbm, bsddb, gdbm and dumbdbm, as it sees fit. TTBoMK, it's not until python 3.x that dbm becomes a sort of unifying module hierarchy. Yes, in Python 2.x, dbm.py is very specifically an interface to the Unix dbm library (see e.g. man dbm_open). At the level of C, the gdbm interface is some kind of extension of that. It's not related to bsddb, which has a very different interface. Would sharing based on inheritance or a more functional approach be preferred? I think either is fine (or even not sharing at all if it turns out to be too much of a mess in practice). A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Release 1.4
Hi Phyo, On Fri, Nov 12, 2010 at 9:18 PM, Phyo Arkar phyo.arkarl...@gmail.com wrote: Is python-mysql gonna work in 1.4 ? i cant wait to run my webserver via pypy-jit Yes, you can use the CPython C extension module. It's a feature that already worked more or less since PyPy 1.3. See explanations in http://morepypy.blogspot.com/2010/06/pypy-13-released.html . Note that _mysql.c first needs a small patch in order to work, which is here: http://codespeak.net/svn/pypy/trunk/pypy/module/cpyext/patches/mysqldb.patch . Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] Release 1.4
Hi all, We are preparing for the next release of PyPy. A branch was already created http://codespeak.net/svn/pypy/release/1.4.x, twice, but it may get thrown away and re-created again. I have 3 things that I would like to fix before the release: 1. Mac OS/X 64. We have there an issue with asmgcc not finding the gc roots. 2. Try to understand if there is a leak in pypy-c-jit or not. The issue is that it consumes quite a lot of memory and fijal could not, so far, really figure out where it comes from. 3. Optionally, depending on 2, implement freeing old JIT-generated assembler and supporting objects. For 3, I have in mind the following lightweight plan: put an age counter on each TreeLoop object, reset it to zero whenever we run the corresponding code, and create an extension of the asmgcc root finder that also resets to zero the counter of loops found in the stack. After every GC collection we increment the age counter. This should ensure that the age of any live loop never goes past 1; on the other hand, loops whose age reach some limit (like 20 or so) have not been run for a long time, and are freed. If we do it right it would free all supporting data structures, and with a __del__ somewhere we could also free the assembler itself (which may only account for 25% of ram usage). A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] pypy_g_action_dispatcher
Hi Hakan, On Mon, Nov 1, 2010 at 10:13 PM, Hakan Ardo ha...@debian.org wrote: after 1016420 iterations of the loop below, pypy_g_action_dispatcher() is called for the first time. After that it is called every 100 iterations or so. Is this intentional? No, strange. Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] Next sprint: Düsseldorf, end of O ctober
Hi all! Düsseldorf PyPy sprint October 25th-31st 2010 = The next PyPy sprint will be held in the Computer Science department of Heinrich-Heine Universität Düsseldorf from the 25th to the 31st of October 2010. This is a fully public sprint, everyone is welcome to join us. Topics and goals Open. (to be defined more precisely) Location The sprint will take place in a seminar room of the computer science department. It is in the building 25.12 of the university campus. For travel instructions see http://stups.cs.uni-duesseldorf.de/anreise/esbahn.php Registration If you'd like to come, please subscribe to the `pypy-sprint mailing list`_ and drop a note about your interests and post any questions. More organisational information will be send to that list. We'll keep a list of `people`_ which we'll update (which you can do so yourself if you have codespeak commit rights). .. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint .. _`people`: http://codespeak.net/pypy/extradoc/sprintinfo/ddorf2010/people.txt ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Question on the future of RPython
Hi Maciej, On Thu, Sep 30, 2010 at 10:51 AM, Maciej Fijalkowski fij...@gmail.com wrote: Other big issues are about RPython itself. Do we want RPython to be python3 compatible? How? No, I'm pretty sure that even if we want to support python3 at some point, RPython will remain what it is now, and translate.py will remain a python2 tool. Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Question on the future of RPython
Hi Paolo, On Thu, Sep 30, 2010 at 8:33 AM, Paolo Giarrusso p.giarru...@gmail.com wrote: My proposal, here, would be a virtual guard, (...) Yes, this proposal makes sense. It's an optimization that is definitely done in regular JITs, and we have a to-do task about it in http://codespeak.net/svn/pypy/extradoc/planning/jit.txt (where they are called out-of-line guards). Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Fwd: Question on the future of RPython
Hi Terrence, I think that what you are describing is found in informal discussions about LLVM/HLVM, and more formally in the plans for Unladen Swallow at http://code.google.com/p/unladen-swallow/wiki/ProjectPlan . See in particular the section about Feedback-Directed Optimization. Maybe you want to discuss these ideas with the Unladen Swallow guys instead :-) Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy JIT C extensions, greenlet
Hi, On Wed, Sep 29, 2010 at 11:39 PM, Andrew Francis andrewfr_...@yahoo.com wrote: I talked briefly with Armin at EuroPython about Stackless and JIT. I poke around pypy-dev and use stackless.py. However I am very interested in learning about how the stackless transform works and how pypy works so I could help. I have been at Stackless for about five years now and I wouldn't mind spending a year learning pypy. Maybe I should expand on an idea posted on #pypy by fijal. He mentioned that he would like to try to support Stackless in PyPy without using the stackless transform, just by using the same low-level stack hacks that are done by greenlet.c and optionally by Stackless Python. This means that there would be two different approaches we can consider to support Stackless in PyPy: stackless transform (done) C-level stack switching approach from Stackless Python 1approach from Stackless Python 2 10-20% speed penalty in the no speed penalty whole interpreter JIT support needed JIT supports comes for free (missing so far) fully portable needs a little bit of assembler some issues to integrate with easy to integrate with C code non-PyPy C code tasklet-switching Python code tasklet-switching Python code becomes a single loop inbecomes N loops with residual calls machine codeto switch() functions (potentially very good) (less good) As you can see from the above summary, the main issue with the 2nd approach would be that Python tasklet-switching loops do not turn into a form that is as efficient as what we would get in the 1st approach. Nevertheless it is an interesting approach because it makes basic JIT support and integration easier. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy JIT C extensions, greenlet
Re-hi, I forgot to mention that the improvement over the stackless transform approach for PyPy might be to not apply the stackless transform on the interpreter, but replace it with resuming C code using the blackhole interpreter that we have anyway in the JIT. Also, any final long-term approach that anyone should at least consider if taking all of this seriously might be a mix of the two approaches, similar to Stackless Python 3 which combines both Stackless Python 1 and Stackless Python 2 features. Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] External RPython mailing list
Hi, On Sun, Sep 12, 2010 at 9:58 PM, Bea During b...@changemaker.nu wrote: Maybe we should be clear in our documentation somewhere on where we stand regarding RPython What about renaming it first? There is at least one other project that uses the name RPython. What about something like InterpPy or InterpPython to make it clear that it's supposed to be used to write interpreters? It doesn't sound terrific but I don't really care -- so, comments welcome, but please no infinite discussion on the pros and cons of various names. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy JIT C extensions, greenlet
Hi Andy, On Mon, Sep 13, 2010 at 7:21 AM, Andy angelf...@yahoo.com wrote: Does that mean PyPy will not work with greenlet/gevent/etc? Sorry if I wasn't clear. PyPy contains greenlet support (since 2005-6). It's part of the same package that we call pypy-stackless. Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] External RPython mailing list
Hi, On Mon, Sep 13, 2010 at 10:11 AM, Maciej Fijalkowski fij...@gmail.com wrote: I don't think it's hideable. Sorry, I wasn't clear. I'm not really trying to hide it. But I'm also not really trying to push it forward (which seems to be what creating a website for it would do). Armin ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy JIT C extensions, greenlet
Hi, On Mon, Sep 13, 2010 at 10:20 AM, Antonio Cuni anto.c...@gmail.com wrote: yes, but it must also be said that at the moment, pypy-stackless and pypy-jit do not work together. Oups, sorry. I missed the word JIT in the original message of this thread :-( Sorry for the confusion. To answer the original question: it would be nice if someone would show up and help contribute JIT support for Stackless builds of PyPy. I think that the status is that no-one of us is ready to invest a lot of time there, but we can definitely give pointers and get people started and follow their progress. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Question on the future of RPython
Hi, On Mon, Sep 6, 2010 at 8:27 PM, Saravanan Shanmugham Is there a wish list of RPython enhancements somewhere that the PyPy team might be considering? Stuff that would benefit RPython users in general. I feel like I am repeating myself so that's my last mail to this thread. There are no enhancements we are considering to benefit other RPython users because *there* *are* *no* *other* *RPython* *users.* There is only us and RPython suits us just fine for the purpose for which it was designed. Again, feel free to make a fork or a branch of PyPy and try to develop a version of RPython that is more suited to writing general programs in. I don't know if there is a wish list of what is missing, but certainly I haven't given it much thoughts myself. Personally, I think that writing RPython programs is kind of fun, but in a perverse way -- if I could just write plain Python that was as fast or mostly as fast, it would be perfect. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
