Re: [pypy-dev] Problem with large allocation in test
On Thu, Apr 21, 2011 at 12:21:48PM +0200, Armin Rigo wrote: Hi Vincent, On Tue, Apr 19, 2011 at 2:02 PM, Vincent Legoll vincent.leg...@gmail.com wrote: Wouldn't a big read only mmap from /dev/zero before the to be jumped adress allow for not really trying to allocate the big chunk of RAM ? That may be a good fix, still allowing the test for far jumps... It's not needed; mmap() can optionally take a position argument, allowing us to choose the position of the small pieces of requested memory. i have been working on a patch based around this behavior of mmap, currently mmap as called in this test does not allow you to specify the base_addr hint however there is an alloc function as used by the jit that allows you to specify this address indirectly (by setting hint.pos in the same class). it appears only the jit is calling this function and as such i have been working on a patch that i hope to have ready in the next couple of days. i wouldn't mind some feedback on how i am trying to do it i am cloning the alloc function and adding a new parameter to specify a base addr, then ripping out the hint object as i have been unable to find anything that references it directly (with a quick grep, will test to be sure) after there is done i will migrate the test from mmap to the new alloc function and provide hints that satisfy the spacing requirements one this is all working i had intended to dump the Alic function and replace it with the new alloc function and update the callers, from my grepping of the source it appears like it is called once or twice in the jit and should be easy to update i am assuming this is the correct course of action, if i should instead not be replacing the original alloc function and just create a new alloc_with_hint function and keep/use both let me know this has been an intresting way to get fammilar with pypy ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] project infrastructure issues
On Thu, Apr 14, 2011 at 05:47:39PM +, holger krekel wrote: Hey all, now that pypy's codespeak subversion usage is basically gone i'd like to push for remaining issues related to the pypy infrastructure: - apache/website - buildbot/master - roundup/issue tracker - mailman/mailing lists pypy-dev/commits/z Which of them shall we try to move elsewhere? My preliminary suggestion: - website - readthedocs? or other site - buildbot - python.org? or other site - issue tracker - bitbucket issue tracker - mailing lists - google groups or python.org or other site The other site could be a host that i anyway need to have for remaining codespeak and merlinux stuff and which thus is somewhat guaranteed to work mail- and otherwise. Other people could get admin access as well, of course. Hi I dont mind hosting the website and mailing lists and would be more than willing to grab a cheap dedicated box or vps somewhere to do so. i already manage a number of boxes for my own private use and adding another wont be much more work i should be able to provide some build slaves without issue. these would be on a home dsl plan and may be down for a day or two per year. that said i can easily cache the output on a publicly available server. KVM or linux containers are avalible on the host and i was planning to run at least ubuntu and fedora as build slaves in a container with $GB of ram allocated to them (64bit) ssh access would be avalible. at the very least i am willing to put in some admin time for the pypy project. if there is alternate hardware avalible and you just need someone to man them or set them up let me know Questions: * with Apache is this a standard cgi setup or a wsgi or mod_python setup for any dynamic parts of the site * if there are any other oddities or special setups let me know - Da_Blitz ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] File overwriting (--output flag to translate.py)
On Mon, Apr 11, 2011 at 09:27:09PM +0200, Armin Rigo wrote:
Sadly everyone so far has his own additional hacks to categorize
multiple translated versions. Mine is to ignore the pypy-c entirely
and copy the executable from the /tmp, after it has been produced
there. I also copy the C sources (but not the other files produced by
gcc). Anyway, my point is that the particular change you are
proposing would actually harm me, because I always have a pypy-c and
I'm fine if it gets overwritten by every translation :-)
We need to think of some better solution...
Armin
here is an updated version that specifically checks if the destination
specified by the --output flag is the same as the running interpreter,
this matches the error i was having more closely and should hopefully
not interfere with other users work flows
of course i am assuming here that you are not relying on the pdb
shell as a notification to copy the file over, if you do feel free to
ignore this updated patch
- Da_Blitz
-
Double check to ensure we are not overwriting the current interpreter
--- a/pypy/translator/goal/translate.py
+++ b/pypy/translator/goal/translate.py
@@ -285,6 +285,10 @@
elif drv.exe_name is None and '__name__' in targetspec_dic:
drv.exe_name = targetspec_dic['__name__'] + '-%(backend)s'
+# Double check to ensure we are not overwriting the current interpreter
+if os.path.realpath(drv.exe_name) == sys.executable:
+raise ValueError('File ' + drv.exe_name+ ' already exisits
(--output)')
+
goals = translateconfig.goals
try:
drv.proceed(goals)
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] translationmodules option failing
On Mon, Apr 11, 2011 at 09:19:22PM +0200, Armin Rigo wrote: Hi, On Mon, Apr 11, 2011 at 1:44 PM, p...@pocketnix.org wrote: - struct, md5, cStringIO, array])) + struct, _md5, cStringIO, array])) Thanks! Applied. Armin seems i missed the ctypes dependence on the threading module (i always passed --thread to translation.py) while the first pypy-c would compile correctly, attempting to translate a second pypy interpreter would fail with an exception related to ctypes with the patch below executing ./pypy-c ./translate.py -O jit ./targetpypystandalone.py --translationmodules will work without passing any more arguments in case anyone is interested i included the last few lines of the traceback to the end of the email from the second run that fails - struct, _md5, cStringIO, array])) + struct, _md5, cStringIO, array, thread])) -- File /home/dablitz/code/pypy/pypy/rpython/lltypesystem/ll2ctypes.py, line 247, in get_ctypes_type cls = build_new_ctypes_type(T, delayed_builders) File /home/dablitz/code/pypy/pypy/rpython/lltypesystem/ll2ctypes.py, line 284, in build_new_ctypes_type _setup_ctypes_cache() File /home/dablitz/code/pypy/pypy/rpython/lltypesystem/ll2ctypes.py, line 92, in _setup_ctypes_cache lltype.Signed: ctypes.c_long, AttributeError: 'NoneType' object has no attribute 'c_long' ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] translationmodules option failing
Hi just tried to bootstrap myself in the quickest way possible via the --translationmodules option (cmdline below) and encountered an issue with the md5 module which appears to be renamed _md5. the patch below corrects this. while there was an md5 directory present in my tree it only contained untracked pyc files and as such i have also removed this - ./translate.py -O jit --thread --make-jobs=9 ./targetpypystandalone.py --translationmodules - --- a/pypy/config/pypyoption.py +++ b/pypy/config/pypyoption.py @@ -39,7 +39,7 @@ translation_modules = default_modules.copy() translation_modules.update(dict.fromkeys( [fcntl, rctime, select, signal, _rawffi, zlib, - struct, md5, cStringIO, array])) + struct, _md5, cStringIO, array])) - ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] benchunix.py fixes
Hi
bench-unix.py did not work so i did q quick repair job and enhancement
on a small bit of the code, included as 3 patches below. i seem to
recall someone mentioning that this was depreciated however as a new
user it was tempting to use it as it was the first thing i saw and is
a nice quick test for getting a rough guide of the relative
performance of several versions of pypy i have generated
currently contemplating a rewrite of the code so let me know if there
is something better or if someone else is working on something
Add newer python implementations and benchmark system default interpreter
--- a/pypy/translator/goal/bench-unix.py
+++ b/pypy/translator/goal/bench-unix.py
@@ -102,7 +102,7 @@
ref_rich, ref_stone = None, None
#for exe in '/usr/local/bin/python2.5 python2.4 python2.3'.split():
-for exe in 'python2.4 python2.3'.split():
+for exe in 'python2.7 python2.6 python2.4 python2.3 python'.split():
v = os.popen(exe + ' -c import sys;print sys.version.split()[0]').rea
if not v:
continue
--
Discard missing interpreters
--- a/pypy/translator/goal/bench-unix.py
+++ b/pypy/translator/goal/bench-unix.py
@@ -102,7 +102,13 @@
ref_rich, ref_stone = None, None
#for exe in '/usr/local/bin/python2.5 python2.4 python2.3'.split():
-for exe in 'python2.4 python2.3'.split():
+for exe in 'python2.7 python2.6 python2.4 python2.3 python'.split():
+path = os.environ.get(PATH, )
+path = [x + os.sep + exe for x in path.split(os.pathsep) if exe in os.l
+if len(path) 0:
+exe = path[0]
+else:
+continue
v = os.popen(exe + ' -c import sys;print sys.version.split()[0]').rea
if not v:
continue
--
Fix off by one
--- a/pypy/translator/goal/bench-unix.py
+++ b/pypy/translator/goal/bench-unix.py
@@ -85,7 +85,7 @@
if os.path.isdir(exe) or exe.endswith('.jar'):
continue
try:
-exes.append( (exe.split('-')[2], exe) )
+exes.append( (exe.split('-')[1], exe) )
except:
pass#skip filenames without version number
exes.sort()
--
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] File overwriting (--output flag to translate.py)
Hi again
i have been compiling a bunch of different pypy instances with different
levels of optimizations and features and found that if i run pypy-c
from the current directory and dont specify a new output filename it
will attempt to and fail to overwrite pypy-c due to the file being in
use. unfortunately the exception generated is in the shutil lib from
mem and the error message/exception does not give away immediately what
the cause is which can lead to some frustration on some of the longer
compiles ;)
its quick and dirty and i don't mind if it gets changed at all. not
100% sure what the correct way to report an error and abandon is other
than what everything else does (uncaught exception to a pdb shell)
--- a/pypy/translator/goal/translate.py
+++ b/pypy/translator/goal/translate.py
@@ -285,6 +285,10 @@
elif drv.exe_name is None and '__name__' in targetspec_dic:
drv.exe_name = targetspec_dic['__name__'] + '-%(backend)s'
+# Ensure the file does not exisit else we fail at end of translation
+if os.path.isfile(drv.exe_name):
+raise ValueError('File ' + drv.exe_name+ ' already exisits
(--output)')
+
goals = translateconfig.goals
try:
drv.proceed(goals)
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Thoughts on multithreading in PyPy
What i am wondering about is if some of the base services provided by pypy can be moved into another thread, eg GC and JIT compilation and how much of a benfiit there would be to doing so at least with moving the gc to another thread i would think that doing so would provide some insight into concurrent access to python objects ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Implementing CAS
On Tue, Mar 22, 2011 at 03:41:19PM -0500, Timothy Baldridge wrote: After my last discussion about getting some multi-threading primitives, I took a look at how locks are implemented in PyPy. PyPy currently uses OS level mutexes...which is okay for most uses, but in my case, I need super fast spinlocks, and for that I need a CAS operation. What I'm looking for is a bit of direction on how to go about implementing this function (shown in C syntax): int cas(*expected, *new, **location) This function looks at the contents of the data pointed at by **location. If the contents == *expected, the contents are replaced with *new and the function returns true. Otherwise the function returns false. There are several issues I see with implementing this in python: 1) **location must be pointer to a object pointer, is this even possible in PyPy? 2) I'd rather not call an FFI C function for this, when this entire function could be inlined with just a few lines of assembly (CAS is a single instruction on x86). So could this be created as a C function that is simply inserted into the generated C code, so that GCC could inline it at will? I'm just brainstorming here. No matter how you dice it, CAS is pretty much a prerequisite to any sort of multi-threaded programming these days. That is unless you want to spend thousands of clock cycles in context switches. Timothy -- “One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.” (Robert Firth) ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev Dont know if this would be helpful at all but i have been wondering if RCU's would be useful at all http://en.wikipedia.org/wiki/Read-copy-update (userspace libary at http://lttng.org/urcu ) from what i ahve read up on them i thought they would be a nice match to python and pypy ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] google should build a pypy phone or someone
On Tue, Feb 15, 2011 at 03:06:07PM +0100, Carl Friedrich Bolz wrote: On 02/15/2011 12:51 PM, Laura Creighton wrote: http://www.freedominput.com/freedom-accessories/freedom-pro-keyboard which works with Android. which makes the whole idea of being able to sit down and program whenever the mood strikes you, even when your laptop is far away or out of power more attractive, since you can touchtype. But that doesn't make me want to start programming in java. Especially for the sort of quick and dirty calculations I use python -c for. I am sure you know of this then: http://code.google.com/p/android-scripting/ It's an app for Android that lets you script the phone with various dynamic languages (including Python). Actually I think that PyPy could/should be made to work with this thing. It would not quite make PyPy a first class VM on the phone, but would already give you access to a number of phone-specific features. the Scripting eniroment uses an enviroment varible to indicate a socket to connect to, once connected you make JSON calls over the socket to access the platform API's that are done via a java proxy. porting pypy to run in the enviroment would be as simple as compiling an arm port and importing the (all python) proxy libary i have a couple of scripts running on my phone for utility functions (checking my site is up and such) the API is a bit limited and i think that pypy running on the native VM would be more flexible for creating full programs (cant acceses many GUI componenets from this restricted enviroment) ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
