Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On Tue, May 13, 2008 at 09:23:02PM -0400, Tom Pinckney wrote: - Why not use MPI? It's cross platform, cross language and very widely - supported already. And there're Python bindings already. MPI requires far more setup than the pyprocessing module does; it's by no means plug play. --titus -- C. Titus Brown, [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Nick Coghlan schrieb: Alex Martelli wrote: On Wed, May 21, 2008 at 1:53 PM, Raymond Hettinger [EMAIL PROTECTED] wrote: Putting this functionality in 2.6/3.0 would provide a really nice incentive to update from Py2.5. It would be a sad lost opportunity if this module had to wait another couple years. I feel essentially the same way: it WOULD be sad to waste this excellent opportunity, so I second the suggestion to put pyprocessing in the library right now. Thirded (I think I'm contradicting what I posted earlier in the thread, but I've had more of a chance to think about it and 18 months really is quite a long time to wait for this functionality...) This seems to be enough support to put an entry into PEP 361, which I've just done. Personally, I'd also find it useful to have such a thing in the stdlib, since at the moment multiple processes are really the way to go for parallelism in Python, and Python needs to stress this fact. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Writing portable ctypes modules is really hard - significantly harder than writing portable C code (although writing non-portable ctypes code is apparently easier than writing non-portable C code). I would say that writing portable C code is hard as well, aren't there just more tools that help? The C compiler in particular. It already gets symbolic constants and struct layouts right, something that ctypes can't do (because it doesn't use header files). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On Thu, 22 May 2008, Martin v. Löwis wrote: I would say that writing portable C code is hard as well, aren't there just more tools that help? The C compiler in particular. It already gets symbolic constants and struct layouts right, something that ctypes can't do (because it doesn't use header files). I don't think it makes much sense to talk about The C Compiler when you are discussing matters of portability. Just my $0.02. -- Cheers, Leif___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
% make cc -c -DNDEBUG -O -I. -IInclude -I./Include -DPy_BUILD_CORE -o Modules/python.o ./Modules/python.c /usr/include/sys/feature_tests.h, line 353: #error: Compiler or options invalid for pre-UNIX 03 X/Open applications and pre-2001 POSIX applications cc: acomp failed for ./Modules/python.c which cc? It works fine for me, with cc: Sun C 5.9 SunOS_sparc Patch 124867-01 2007/07/12 (it just complains about not supporting -OPT:Olimit) *** Error code 2 make: Fatal error: Command failed for target `Modules/python.o' % So maybe Python just doesn't run on Solaris with the Sun C compiler. Certainly doesn't build out of the box. Apparently, there were some intermediate releases of SunPRO which had this problem; in these cases, setting -xc99=%none may help. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Could it be a solution to build libffi with gcc, then configure Python with '--with-system--ffi' and compile with the sun compiler, or would this introduce the dependencies on libgcc or whatever again that Ulrich wanted to avoid? It depends on the processor and the code, but chances are high that this would indeed create a dependency on libgcc. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Martin v. Löwis schrieb: I can neither find recvfd in my man pages nor in my header files in /usr/include on Linux (Ubuntu 8.04 i686). I assume recvfd and sendfd aren't syscalls but the proposed names for the functions. Yes (and no). The system call is sendmsg, with a cmsg_type of SCM_RIGHTS. I don't think there is a plan to add library functions with that name. There is at least a patch: http://bugs.python.org/issue1194378 Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Brett Cannon wrote: On Mon, May 19, 2008 at 2:03 AM, Ulrich Berning [EMAIL PROTECTED] wrote: Gregory P. Smith wrote: On Fri, May 16, 2008 at 1:32 AM, Ulrich Berning [EMAIL PROTECTED] wrote: As long as the ctypes extension doesn't build on major Un*x platforms (AIX, HP-UX), I don't like to see ctypes dependend modules included into the stdlib. Please keep the stdlib as portable as possible. Nice in theory but ctypes already works on at least the top 3 popular platforms. Lets not hold Python's stdlib back because nobody who uses IBM and HP proprietary stuff has contributed the necessary support. Making nice libraries available for other platforms is a good way to encourage people to either pitch in and add support or consider their platform choices in the future. -gps It's not my platform choice, it's the choice of our customers. I'm not using these platforms just for fun (in fact it isn't fun compared to Linux or Windows). If porting libffi to AIX, HP-UX, IRIX, Solaris... (especially using vendor compilers) would be an easy job, I'm sure it would have been done already. Well, ctypes isn't simple. =) If more and more essential packages depend on ctypes, we should make a clear statement, that Python isn't supported any longer on platform/compiler combinations where libffi/ctypes doesn't build. This would give me arguments to drop support of our software on those platforms. You are mixing the stdlib in with the language in terms of what is required for Python to work, which I think is unfair. Just because some part of the stdlib isn't portable to some OS does not mean Python is not supported on that platform. If you can run a pure Python module that does not depend on any C extension, then that platform has the support needed to run Python. Everything else is extra (which is why we have modules in the stdlib only available on specific platforms). -Brett I don't think it is unfair. If the development team decides one day to reimplement essential extensions like math, _socket, select, _ssl, pwd, grp, time, _locale, zlib... based on ctypes because it may be much easier to maintain python modules instead of dealing with complicated C code, Python will become pretty useless. It's like a cool radio without the chance to get any batteries for it, pretty useless. Platform specific modules are documented as such and nobody would expect a _winreg module on AIX or HP-UX. As said before, PyOpenGL is an example of an extension that moved from C code to Python/ctypes, luckily we don't use it, but what if the maintainers of MySQL-Python or cx_Oracle decide to move to ctypes. Having the ctypes extension in the stdlib doesn't imply it runs on any platform where python runs. Extension writers should keep this in mind when they decide to use ctypes. They should document, that their extension depends on ctypes and therefore doesn't run on platforms where ctypes doesn't work. Ulli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Ulrich Having the ctypes extension in the stdlib doesn't imply it runs Ulrich on any platform where python runs. Maybe the presence of a functioning ctypes (can|might|should|will) become the operational definition of Python runs on platform X. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On Wed, May 21, 2008 at 10:11:05AM -0500, [EMAIL PROTECTED] wrote: Maybe the presence of a functioning ctypes (can|might|should|will) become the operational definition of Python runs on platform X. It is not black-or-white, runs or doesn't. PythonD, e.g., runs on DOS, can use sockets (from WatTCP library), but certainly cannot do multithreading or multitasking. So the wording should be Python supports platform X with the following limitations: ... Oleg. -- Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
[EMAIL PROTECTED] wrote: Ulrich Having the ctypes extension in the stdlib doesn't imply it runs Ulrich on any platform where python runs. Maybe the presence of a functioning ctypes (can|might|should|will) become the operational definition of Python runs on platform X. And what about platforms like the JVM or CLR? Incidentally there were a small but vocal group of Pythonistas who were (are?) certain that IronPython is not Python because it doesn't have [all of...] the C extensions. Michael Foord Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
James Y Knight wrote: On May 21, 2008, at 11:26 AM, Michael Foord wrote: And what about platforms like the JVM or CLR? Incidentally there were a small but vocal group of Pythonistas who were (are?) certain that IronPython is not Python because it doesn't have [all of...] the C extensions. It seems likely to be easier to port ctypes to Jython/IronPython than it is to port 500 C extensions to Jython. So more software using ctype seems like an improvement for those communities. Yes, both of them have their own FFIs. Seo Sanghyeon started a port of ctypes for IronPython that should be revived by 'someone'. :-) Michael Foord James ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On May 21, 2008, at 11:26 AM, Michael Foord wrote: And what about platforms like the JVM or CLR? Incidentally there were a small but vocal group of Pythonistas who were (are?) certain that IronPython is not Python because it doesn't have [all of...] the C extensions. It seems likely to be easier to port ctypes to Jython/IronPython than it is to port 500 C extensions to Jython. So more software using ctype seems like an improvement for those communities. James ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Skip Maybe the presence of a functioning ctypes (can|might|should|will) Skip become the operational definition of Python runs on platform X. Michael And what about platforms like the JVM or CLR? Sorry, allow me to rephrase: Maybe the presence of a functioning ctypes (can|might|should|will) become the operational definition of CPython runs on platform X. Michael Incidentally there were a small but vocal group of Pythonistas Michael who were (are?) certain that IronPython is not Python because Michael it doesn't have [all of...] the C extensions. From my perspective IronPython isn't Python because it doesn't run on my platforms (Solaris Mac). ;-) (At least not easily enough for an old codger like me to install.) Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Bill Janssen schrieb: What happens on those platforms where ctypes doesn't work? Does the module fail to import, either because it isn't present, or because it can't load the libffi library? Or does it fail silently in some way? It doesn't compile, and Python's setup.py script removes if afterwards IIRC so it cannot be imported. In some cases it compiles, but the ctypes unittests segfault (I have seen this some days ago on HP-UX PA with gcc). The cause for the segfault is either a ctypes bug or a libffi bug. Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
A.M. Kuchling schrieb: On Mon, May 19, 2008 at 08:50:39PM +0200, Thomas Heller wrote: Myself I would rather spend my energy to make ctypes more portable, within my skills and the platforms I have access to. Someone could run Solaris x86 inside a hosted virtual machine and make it available to the Python developers. Is it possible to find similar hosting for HP-UX and AIX? Or might IBM or HP be willing to donate a low-end machine to the PSF for porting use? I have a vmware appliance installed that runs solaris 10 update 2, gcc and the sun compiler are installed. As expected (?), ctypes works fine when compiled with gcc, but fails to build with the siun compiler. There is also a solaris buildbot running on a sparc machine. Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Ulrich Berning schrieb: If porting libffi to AIX, HP-UX, IRIX, Solaris... (especially using vendor compilers) would be an easy job, I'm sure it would have been done already. If more and more essential packages depend on ctypes, we should make a clear statement, that Python isn't supported any longer on platform/compiler combinations where libffi/ctypes doesn't build. This would give me arguments to drop support of our software on those platforms. Ulrich, if *you* have access to these platforms and want to help a good start which hopefully wouldn't take too much time would be to compile the current libffi release [1], run the test suite, and report the results. Compile with gcc, for a start, and note that the testsuite requires dejagnu (the PyObjC folks have written a dejagnu replacement in Python for testing libffi, but I haven't tried that). [1] http://sourceware.org/libffi/ -- Thanks, Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Michael Foord schrieb: James Y Knight wrote: On May 21, 2008, at 11:26 AM, Michael Foord wrote: And what about platforms like the JVM or CLR? Incidentally there were a small but vocal group of Pythonistas who were (are?) certain that IronPython is not Python because it doesn't have [all of...] the C extensions. It seems likely to be easier to port ctypes to Jython/IronPython than it is to port 500 C extensions to Jython. So more software using ctype seems like an improvement for those communities. Yes, both of them have their own FFIs. Seo Sanghyeon started a port of ctypes for IronPython that should be revived by 'someone'. :-) OT: There is also some noise about a ctypes-js implementation for JavaScript; although I do not know how far they are. Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
[EMAIL PROTECTED] wrote: Skip Maybe the presence of a functioning ctypes (can|might|should|will) Skip become the operational definition of Python runs on platform X. Michael And what about platforms like the JVM or CLR? Sorry, allow me to rephrase: Maybe the presence of a functioning ctypes (can|might|should|will) become the operational definition of CPython runs on platform X. Michael Incidentally there were a small but vocal group of Pythonistas Michael who were (are?) certain that IronPython is not Python because Michael it doesn't have [all of...] the C extensions. From my perspective IronPython isn't Python because it doesn't run on my platforms (Solaris Mac). ;-) (At least not easily enough for an old codger like me to install.) IronPython runs very well on the Mac. Installing Mono is pretty much 'one click' these days and it *comes* with IronPython (and has done for a while). I'm writing IronPython in Action on the Mac. Solaris I don't care about... :-) Michael Foord Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
As said before, PyOpenGL is an example of an extension that moved from C code to Python/ctypes, luckily we don't use it, but what if the maintainers of MySQL-Python or cx_Oracle decide to move to ctypes. Having the ctypes extension in the stdlib doesn't imply it runs on any platform where python runs. Extension writers should keep this in mind when they decide to use ctypes. They should document, that their extension depends on ctypes and therefore doesn't run on platforms where ctypes doesn't work. Plus, even if ctypes works, the code might be incorrect, because they had been assuming structure layouts and symbolic constants that have just a different definition on some other platform, causing the extension module to crash. Writing portable ctypes modules is really hard - significantly harder than writing portable C code (although writing non-portable ctypes code is apparently easier than writing non-portable C code). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Thomas Heller schrieb: A.M. Kuchling schrieb: On Mon, May 19, 2008 at 08:50:39PM +0200, Thomas Heller wrote: Myself I would rather spend my energy to make ctypes more portable, within my skills and the platforms I have access to. Someone could run Solaris x86 inside a hosted virtual machine and make it available to the Python developers. Is it possible to find similar hosting for HP-UX and AIX? Or might IBM or HP be willing to donate a low-end machine to the PSF for porting use? I have a vmware appliance installed that runs solaris 10 update 2, gcc and the sun compiler are installed. As expected (?), ctypes works fine when compiled with gcc, but fails to build with the siun compiler. There is also a solaris buildbot running on a sparc machine. I just downloaded the Python 2.5.2 source tar, and tried to build it on a Solaris 11 machine with the SunPro 8 compiler (Sun CC 5.5), and failed: % ./configure [...] creating Modules/Setup creating Modules/Setup.local creating Makefile % make cc -c -DNDEBUG -O -I. -IInclude -I./Include -DPy_BUILD_CORE -o Modules/python.o ./Modules/python.c /usr/include/sys/feature_tests.h, line 353: #error: Compiler or options invalid for pre-UNIX 03 X/Open applications and pre-2001 POSIX applications cc: acomp failed for ./Modules/python.c *** Error code 2 make: Fatal error: Command failed for target `Modules/python.o' % So maybe Python just doesn't run on Solaris with the Sun C compiler. Certainly doesn't build out of the box. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On Wed, 21 May 2008 20:57:33 +0200, \Martin v. Löwis\ [EMAIL PROTECTED] wrote: As said before, PyOpenGL is an example of an extension that moved from C code to Python/ctypes, luckily we don't use it, but what if the maintainers of MySQL-Python or cx_Oracle decide to move to ctypes. Having the ctypes extension in the stdlib doesn't imply it runs on any platform where python runs. Extension writers should keep this in mind when they decide to use ctypes. They should document, that their extension depends on ctypes and therefore doesn't run on platforms where ctypes doesn't work. Plus, even if ctypes works, the code might be incorrect, because they had been assuming structure layouts and symbolic constants that have just a different definition on some other platform, causing the extension module to crash. Writing portable ctypes modules is really hard - significantly harder than writing portable C code (although writing non-portable ctypes code is apparently easier than writing non-portable C code). True. There's some room for improvement in ctypes here, fortunately. For example, PyPy has some tools which resolve the particular problem you're talking about; the library is even available separately and can (and probably should) be used by anyone writing a ctypes module. Sample usage and installation instructions available here: http://codespeak.net/~fijal/configure.html Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On May 21, 2008, at 3:58 PM, Jean-Paul Calderone wrote: Plus, even if ctypes works, the code might be incorrect, because they had been assuming structure layouts and symbolic constants that have just a different definition on some other platform, causing the extension module to crash. Writing portable ctypes modules is really hard - significantly harder than writing portable C code (although writing non-portable ctypes code is apparently easier than writing non-portable C code). True. There's some room for improvement in ctypes here, fortunately. For example, PyPy has some tools which resolve the particular problem you're talking about; the library is even available separately and can (and probably should) be used by anyone writing a ctypes module. Sample usage and installation instructions available here: http://codespeak.net/~fijal/configure.html The csvn subversion bindings use ctypesgen to grovel header files: http://code.google.com/p/ctypesgen/ There's also ctypeslib, which uses gcc-xml to parse the headers instead of a pure python C parser as ctypesgen does: http://svn.python.org/projects/ctypes/trunk/ctypeslib/ I don't really know how all these tools compare to eachother. It sure would be nice if someone could compare them, and figure out if one can be blessed and included with python so that doing the right thing is easy. James ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
This thread has diverged a bit from the original topic. I suggest going ahead and adding pyprocessing to the library. IMO, its functionality is going to be an essential capability as more and more computers ship with multiple processors. At this point, the basic API for pyprocessing seems well thought-out and somewhat stable. Over time, I expect the implementation will get tweaked in a number of ways including support more platforms as developers figure-out that they like the idea enough to write some platform dependent patches. Putting this functionality in 2.6/3.0 would provide a really nice incentive to update from Py2.5. It would be a sad lost opportunity if this module had to wait another couple years. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
The csvn subversion bindings use ctypesgen to grovel header files: http://code.google.com/p/ctypesgen/ Thanks for the pointer. Looks nice. I've used ctypeslib before, but it takes a bit of infrastructure building (gcc-xml) before it works. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On Wed, May 21, 2008 at 1:53 PM, Raymond Hettinger [EMAIL PROTECTED] wrote: This thread has diverged a bit from the original topic. I suggest going ahead and adding pyprocessing to the library. IMO, its functionality is going to be an essential capability as more and more computers ship with multiple processors. At this point, the basic API for pyprocessing seems well thought-out and somewhat stable. Over time, I expect the implementation will get tweaked in a number of ways including support more platforms as developers figure-out that they like the idea enough to write some platform dependent patches. Putting this functionality in 2.6/3.0 would provide a really nice incentive to update from Py2.5. It would be a sad lost opportunity if this module had to wait another couple years. I feel essentially the same way: it WOULD be sad to waste this excellent opportunity, so I second the suggestion to put pyprocessing in the library right now. Alex ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Martin v. Löwis wrote: IIRC, it chokes on the attempt to compile assembler code that has C preprocessor macros in it (can't test it right now). Could the build process be modified to run the C preprocessor over the assembly language first? -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Alex Martelli wrote: On Wed, May 21, 2008 at 1:53 PM, Raymond Hettinger [EMAIL PROTECTED] wrote: Putting this functionality in 2.6/3.0 would provide a really nice incentive to update from Py2.5. It would be a sad lost opportunity if this module had to wait another couple years. I feel essentially the same way: it WOULD be sad to waste this excellent opportunity, so I second the suggestion to put pyprocessing in the library right now. Thirded (I think I'm contradicting what I posted earlier in the thread, but I've had more of a chance to think about it and 18 months really is quite a long time to wait for this functionality...) Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On Mon, May 19, 2008 at 06:13:11PM -0700, Bill Janssen wrote: And these are all SYSV derivatives, aren't they? So perhaps it's some common fix for all three? This reminds of a Tim-ism: == Just for the record, on AIX, the following C program: Oh no you don't! I followed AIX threads for the first year it came out, but eventually decided there was no future in investing time in baffling discussions that usually ended with oh, never mind -- turns out it's a bug 0.9 wink. Vladimir Marangozov and Tim Peters, 23 Jun 1998 == 10 years later, things don't seem to be much different. For ctypes it looks like libffi often fails to compile when not using GCC; http://bugs.python.org/issue1637120 is an AIX bug report. For the curses module, I've noticed that HP-UX seems to require including additional header files or defining _XOPEN_SOURCE to make curses.h work. So I don't think there's a common problem with all these minority platforms, and we really would need developers on all of them. --amk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Bill Janssen wrote: And these are all SYSV derivatives, aren't they? So perhaps it's some common fix for all three? I've never personally had the pleasure(?!) of having to get stuff working on all of AIX/HP-UX/Tru64/Solaris, but what little dealings I've had with people who have suggests that the differences between these platforms make the discrepancies between different Linux distributions look utterly trivial by comparison. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
A.M. Kuchling [EMAIL PROTECTED] wrote: On Mon, May 19, 2008 at 08:50:39PM +0200, Thomas Heller wrote: Myself I would rather spend my energy to make ctypes more portable, within my skills and the platforms I have access to. Someone could run Solaris x86 inside a hosted virtual machine and make it available to the Python developers. Is it possible to find similar hosting for HP-UX and AIX? Or might IBM or HP be willing to donate a low-end machine to the PSF for porting use? You can test HPUX and VMS with a (free) HP Testdrive account (along with a bunch of Linuxes and *BSDs) on various hardware platforms. I used it extensively when doing portability testing of another project. http://www.testdrive.hp.com/ Charles -- --- Charles Cazabon GPL'ed software available at: http://pyropus.ca/software/ --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Charles Cazabon schrieb: A.M. Kuchling [EMAIL PROTECTED] wrote: On Mon, May 19, 2008 at 08:50:39PM +0200, Thomas Heller wrote: Myself I would rather spend my energy to make ctypes more portable, within my skills and the platforms I have access to. Someone could run Solaris x86 inside a hosted virtual machine and make it available to the Python developers. Is it possible to find similar hosting for HP-UX and AIX? Or might IBM or HP be willing to donate a low-end machine to the PSF for porting use? You can test HPUX and VMS with a (free) HP Testdrive account (along with a bunch of Linuxes and *BSDs) on various hardware platforms. I used it extensively when doing portability testing of another project. http://www.testdrive.hp.com/ Charles I'm doing that. From time to time, these systems are a pain to use IMO. A buildbot is s much better... -- Thanks, Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On Mon, 19 May 2008, Bill Janssen wrote: On Mon, May 19, 2008 at 5:15 PM, Bill Janssen [EMAIL PROTECTED] wrote: If you can run a pure Python module that does not depend on any C extension, then that platform has the support needed to run Python. This is certainly a point of view. One that many end-users wouldn't understand :-). Perhaps, but it's clear-cut. Is OS X not properly supported because it can't run the _winreg module? I just don't like labeling a platform as unsupported just because ctypes doesn't compile on it. I assume _winreg runs everywhere Python is said to run, and there's a Windows registry to examine, so I think it's a different kettle of fish. ctypes doesn't run everywhere Python is said to run, and there are dynamic libraries to call into. I think it would be great if we could get some AIX, HP-UX, and Solaris boxes for Thomas to work on. What would motivate IBM, H-P, and Sun to donate such gear? Perhaps each of the companies have an office somewhere that could help with this resource allocation? For instance, talking to the AIX Collaboration Center of IBM ([EMAIL PROTECTED])? And these are all SYSV derivatives, aren't they? So perhaps it's some common fix for all three? I just spent a semester in Advanced Systems Programming in Unix/C, and all we did was port old software to solaris, hpux, irix, and friends (I don't think we used AIX, but I believe he has one running). Most of these were virtual machines, but they adequately represented the annoyances present in their respective architectures. I've cc'd my professor, hopefully he will let me know if there are any tips he has for setting up a VM. -- Cheers, Leif ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Steve Holden wrote: The more libraries that use ctypes to call into native functionality, the more important it becomes to have ctypes run, even if only to implement platform-specific functionality on the given platforms. I would like being able to call a wide range of native libraries to be a Python claim on all platforms, even when the native libraries are platform-proprietary. Yes, I'd like to see this happen, too. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Bill Janssen schrieb: Hmm, perhaps the ctypes documentation could use a more prominent warning that it may not be available on some Unix platforms (HP-UX, AIX, IRIX), and that it may require the use of GCC rather than the vendor compiler on others (Solaris). At the moment, I suspect some projects may be switching to using it without realising the implications for cross-platform portability. I think it's a tad more problematic. As other modules, both in and out of the standard library, move to use ctypes, it implies that *Python* isn't supported on those combinations. Personally, that's fine with me (as long as there's a workaround for Solaris!), but I think that Ulrich is right in saying this should be more prominent. I won't object if anyone adds this notice to the Python docs, so please go ahead. A table of platforms (on the wiki?) where ctypes builds/works or does not may also be helpful. What happens on those platforms where ctypes doesn't work? Does the module fail to import, either because it isn't present, or because it can't load the libffi library? Or does it fail silently in some way? Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
r.m.oudkerk schrieb: Now that socket.fromfd() and socket._dup() is available on Windows in 2.6 and 3.0 (after a patch from me) some of the code could be removed. I would also like to see recvfd() and sendfd() get added to the socket module -- these functions allow the transfer of file descriptors between processes over a Unix domain socket. I can neither find recvfd in my man pages nor in my header files in /usr/include on Linux (Ubuntu 8.04 i686). I assume recvfd and sendfd aren't syscalls but the proposed names for the functions. (1) a Connection type (plus PipeConnection on windows) (2) a process shared lock/semaphore type (3) Win32 functions/constants to allow use of named pipes (4) A few other Win32 functions/constants (5) A wrapper making PyObject_AsWriteBuffer() available from Python I suppose (4) and perhaps (3) could sensibly be added/merged with _subprocess.c. (5) could also be moved somewhere else. Why do you want to put the named pipes into the _subprocess module? IMHO the socket module is more appropriate for communication channels like name pipes. (Off-topic but I think that the way that subprocess.Popen.__del__() keeps the object alive if the process has not finished is a little distasteful. The __del__() method and the _cleanup() function are completely unecessary on Windows since Windows does not have zombie processes. On Unix __del__() could simply save the pid to the _active list and _cleanup() could be rewritten to use os.waitpid().) Interesting idea. The approach could safe us some trouble. I'm always +1 when it comes to removing ugly hacks. Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Ted Leung wrote: On May 19, 2008, at 7:47 PM, Steve Holden wrote: OK, I know people in Sun and possibly H-P I could ask, and I may have an arm or two to twist to get to IBM. But worst-case, what budget would the infrastructure committee need for the hardware and (more importantly) if the hardware materialized, would there be manpower to maintain the platforms as Python supported? You can ask me as far as Sun goes. There are probably a variety of options: 1) Get the open source version of VirtualBox and use it to run a virtualized OpenSolaris 2) I can see if we can get folks access to a box running Solaris 3) I can ask and see if there are people at Sun who would be willing to jump in and help We actually have a couple of Solaris buildbots already - as I understand it, the issue there isn't Solaris as such, it's being able to use the Sun compiler instead of GCC to compile ctypes/libffi. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On May 20, 2008, at 2:03 PM, Nick Coghlan wrote: We actually have a couple of Solaris buildbots already - as I understand it, the issue there isn't Solaris as such, it's being able to use the Sun compiler instead of GCC to compile ctypes/libffi. Does that mean that we need access to the Sun compiler or that the Sun compiler has bugs which prevent ctypes from compiling? Ted ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Ted Leung wrote: On May 20, 2008, at 2:03 PM, Nick Coghlan wrote: We actually have a couple of Solaris buildbots already - as I understand it, the issue there isn't Solaris as such, it's being able to use the Sun compiler instead of GCC to compile ctypes/libffi. Does that mean that we need access to the Sun compiler or that the Sun compiler has bugs which prevent ctypes from compiling? I don't think anyone's mentioned Solaris in this context, but there are claims that libffi doesn't build for AIX and HP-UX. regards Steve -- Steve Holden+1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Does that mean that we need access to the Sun compiler or that the Sun compiler has bugs which prevent ctypes from compiling? I don't think anyone's mentioned Solaris in this context, but there are claims that libffi doesn't build for AIX and HP-UX. I think there was also an issue with using the Sun compiler to build ctypes and/or libffi. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Christian Heimes wrote: I assume recvfd and sendfd aren't syscalls but the proposed names for the functions. Yes, the functionality in question is accessed through the sendmsg() and recvmsg() system calls. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
I can neither find recvfd in my man pages nor in my header files in /usr/include on Linux (Ubuntu 8.04 i686). I assume recvfd and sendfd aren't syscalls but the proposed names for the functions. Yes (and no). The system call is sendmsg, with a cmsg_type of SCM_RIGHTS. I don't think there is a plan to add library functions with that name. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Nick Coghlan wrote: Ulrich Berning wrote: More and more people tend to say Runs on Un*x when they really mean Tested on Linux. Un*x is not Linux. Hmm, perhaps that would be why there are Solaris, FreeBSD and Tru64 machines amongst the main Python buildbots, to go along with the assorted OS X, Windows and Linux boxes - and as far as I know test_ctypes runs quite happily on all of them. On the specific problems with AIX, HP-UX and ctypes, was it ctypes itself that was failing to build, or the underlying libffi? Cheers, Nick. On HP-UX-11.00, HP ANSI C++ B3910B A.03.73, Python-2.5.2, I get configure: error: libffi has not been ported to hppa2.0w-hp-hpux11.00. On AIX-4.3.3, C for AIX Compiler Version 6, Python-2.5.2, I get build/temp.aix-4.3-2.5/libffi/include/ffi.h, line 123.4: 1506-205 (S) #error no 64-bit data type supported On Solaris-10/x86, Sun C 5.8 Patch 121016-07 2007/10/03, Python-2.5.2, I get build/temp.solaris-2.10-i86pc-2.5/libffi/include/ffitarget.h, line 64: undefined symbol: FFI_DEFAULT_ABI On Solaris-8/sparc, Sun C 5.8 2005/10/13, Python-2.5.2, I get build/temp.solaris-2.8-sun4u-2.5/libffi/include/ffi.h, line 225: syntax error before or at: __attribute__ On IRIX-6.5, gcc-3.4.4, Python-2.5.2, ffi_closure is undefined, because only the old O32 binary format is supported, not the new N32/N64 format. I'm trying to use the vendor specific compilers whenever possible, because using gcc puts in additional dependencies (libgcc), I want to avoid, and even if I could live with these dependencies, it's not easy to get/build the 'right' gcc version, if your software also depends on other big packages like Qt and PyQt. I'm not using these platforms for my own pleasure (in fact, I would be happy if these platforms would disappear from the market), but as long as our customers use these platforms, we want to promise our software runs on those platforms. I have no problem with the fact that ctypes doesn't build on those platforms because I don't use it, but if more and more essential packages depend on ctypes, I'm running into trouble. PyOpenGL is an example of an extension, that moved completely from C-Source (SWIG generated) to ctypes usage. Ulli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Gregory P. Smith wrote: On Fri, May 16, 2008 at 1:32 AM, Ulrich Berning [EMAIL PROTECTED] wrote: As long as the ctypes extension doesn't build on major Un*x platforms (AIX, HP-UX), I don't like to see ctypes dependend modules included into the stdlib. Please keep the stdlib as portable as possible. Nice in theory but ctypes already works on at least the top 3 popular platforms. Lets not hold Python's stdlib back because nobody who uses IBM and HP proprietary stuff has contributed the necessary support. Making nice libraries available for other platforms is a good way to encourage people to either pitch in and add support or consider their platform choices in the future. -gps It's not my platform choice, it's the choice of our customers. I'm not using these platforms just for fun (in fact it isn't fun compared to Linux or Windows). If porting libffi to AIX, HP-UX, IRIX, Solaris... (especially using vendor compilers) would be an easy job, I'm sure it would have been done already. If more and more essential packages depend on ctypes, we should make a clear statement, that Python isn't supported any longer on platform/compiler combinations where libffi/ctypes doesn't build. This would give me arguments to drop support of our software on those platforms. Ulli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Ulrich Berning wrote: I'm trying to use the vendor specific compilers whenever possible, because using gcc puts in additional dependencies (libgcc), I want to avoid, and even if I could live with these dependencies, it's not easy to get/build the 'right' gcc version, if your software also depends on other big packages like Qt and PyQt. I'm not using these platforms for my own pleasure (in fact, I would be happy if these platforms would disappear from the market), but as long as our customers use these platforms, we want to promise our software runs on those platforms. I have no problem with the fact that ctypes doesn't build on those platforms because I don't use it, but if more and more essential packages depend on ctypes, I'm running into trouble. PyOpenGL is an example of an extension, that moved completely from C-Source (SWIG generated) to ctypes usage. Hmm, perhaps the ctypes documentation could use a more prominent warning that it may not be available on some Unix platforms (HP-UX, AIX, IRIX), and that it may require the use of GCC rather than the vendor compiler on others (Solaris). At the moment, I suspect some projects may be switching to using it without realising the implications for cross-platform portability. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Hmm, perhaps the ctypes documentation could use a more prominent warning that it may not be available on some Unix platforms (HP-UX, AIX, IRIX), and that it may require the use of GCC rather than the vendor compiler on others (Solaris). At the moment, I suspect some projects may be switching to using it without realising the implications for cross-platform portability. I think it's a tad more problematic. As other modules, both in and out of the standard library, move to use ctypes, it implies that *Python* isn't supported on those combinations. Personally, that's fine with me (as long as there's a workaround for Solaris!), but I think that Ulrich is right in saying this should be more prominent. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 A.M. Kuchling wrote: | Python development doesn't seem to have any volunteers who use AIX or | HP-UX and can fix bugs on these platforms. Searching bugs.python.org, | I find about 20 open AIX issues, 4 or 5 HP-UX issues, 6 IRIX bugs, and | about 15 Solaris bugs; but I don't know if any of the developers here | use these platforms. (There is a Solaris buildbot, at least.) I develop under Solaris 10 x86. My environment is 64 bits, but my python deployment/development in 32 bits. I could try 64 bits if asked for. I can look at any Solaris related issue. Just ask. - -- Jesus Cea Avion _/_/ _/_/_/_/_/_/ [EMAIL PROTECTED] - http://www.jcea.es/ _/_/_/_/ _/_/_/_/ _/_/ jabber / xmpp:[EMAIL PROTECTED] _/_/_/_/ _/_/_/_/_/ ~ _/_/ _/_/_/_/ _/_/ _/_/ Things are not so easy _/_/ _/_/_/_/ _/_/_/_/ _/_/ My name is Dump, Core Dump _/_/_/_/_/_/ _/_/ _/_/ El amor es poner tu felicidad en la felicidad de otro - Leibniz -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBSDGz2Jlgi5GaxT1NAQIx6wP+NudycEV/nkaCEGLUwnj5BP19urVURedj x6Udb690/Hon/fv6tkplIw5pLramdv3hH8KKsxWr+kzrp4iHUZ7JPyItC0DBXImQ 3wRrV7Yhu1nVmFV2qm5Kdbu8+x7fCiWnzESLzQmIjEKQd0dlO2vStXUNmndoJ21h pDwIW0bUriA= =jbHH -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Bill Janssen schrieb: Hmm, perhaps the ctypes documentation could use a more prominent warning that it may not be available on some Unix platforms (HP-UX, AIX, IRIX), and that it may require the use of GCC rather than the vendor compiler on others (Solaris). At the moment, I suspect some projects may be switching to using it without realising the implications for cross-platform portability. I think it's a tad more problematic. As other modules, both in and out of the standard library, move to use ctypes, it implies that *Python* isn't supported on those combinations. Personally, that's fine with me (as long as there's a workaround for Solaris!), but I think that Ulrich is right in saying this should be more prominent. I won't object if anyone adds this notice to the Python docs, so please go ahead. A table of platforms (on the wiki?) where ctypes builds/works or does not may also be helpful. Myself I would rather spend my energy to make ctypes more portable, within my skills and the platforms I have access to. Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
If you can run a pure Python module that does not depend on any C extension, then that platform has the support needed to run Python. This is certainly a point of view. One that many end-users wouldn't understand :-). Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On Mon, May 19, 2008 at 5:15 PM, Bill Janssen [EMAIL PROTECTED] wrote: If you can run a pure Python module that does not depend on any C extension, then that platform has the support needed to run Python. This is certainly a point of view. One that many end-users wouldn't understand :-). Perhaps, but it's clear-cut. Is OS X not properly supported because it can't run the _winreg module? I just don't like labeling a platform as unsupported just because ctypes doesn't compile on it. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On Mon, May 19, 2008 at 5:15 PM, Bill Janssen [EMAIL PROTECTED] wrote: If you can run a pure Python module that does not depend on any C extension, then that platform has the support needed to run Python. This is certainly a point of view. One that many end-users wouldn't understand :-). Perhaps, but it's clear-cut. Is OS X not properly supported because it can't run the _winreg module? I just don't like labeling a platform as unsupported just because ctypes doesn't compile on it. I assume _winreg runs everywhere Python is said to run, and there's a Windows registry to examine, so I think it's a different kettle of fish. ctypes doesn't run everywhere Python is said to run, and there are dynamic libraries to call into. I think it would be great if we could get some AIX, HP-UX, and Solaris boxes for Thomas to work on. What would motivate IBM, H-P, and Sun to donate such gear? Perhaps each of the companies have an office somewhere that could help with this resource allocation? For instance, talking to the AIX Collaboration Center of IBM ([EMAIL PROTECTED])? And these are all SYSV derivatives, aren't they? So perhaps it's some common fix for all three? Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Bill Janssen wrote: On Mon, May 19, 2008 at 5:15 PM, Bill Janssen [EMAIL PROTECTED] wrote: If you can run a pure Python module that does not depend on any C extension, then that platform has the support needed to run Python. This is certainly a point of view. One that many end-users wouldn't understand :-). Perhaps, but it's clear-cut. Is OS X not properly supported because it can't run the _winreg module? I just don't like labeling a platform as unsupported just because ctypes doesn't compile on it. I assume _winreg runs everywhere Python is said to run, and there's a Windows registry to examine, so I think it's a different kettle of fish. ctypes doesn't run everywhere Python is said to run, and there are dynamic libraries to call into. I think it would be great if we could get some AIX, HP-UX, and Solaris boxes for Thomas to work on. What would motivate IBM, H-P, and Sun to donate such gear? Perhaps each of the companies have an office somewhere that could help with this resource allocation? For instance, talking to the AIX Collaboration Center of IBM ([EMAIL PROTECTED])? And these are all SYSV derivatives, aren't they? So perhaps it's some common fix for all three? OK, I know people in Sun and possibly H-P I could ask, and I may have an arm or two to twist to get to IBM. But worst-case, what budget would the infrastructure committee need for the hardware and (more importantly) if the hardware materialized, would there be manpower to maintain the platforms as Python supported? The more libraries that use ctypes to call into native functionality, the more important it becomes to have ctypes run, even if only to implement platform-specific functionality on the given platforms. I would like being able to call a wide range of native libraries to be a Python claim on all platforms, even when the native libraries are platform-proprietary. regards Steve -- Steve Holden+1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Nick Craig-Wood wrote: Jesse Noller [EMAIL PROTECTED] wrote: I am looking for any questions, concerns or benchmarks python-dev has regarding the possible inclusion of the pyprocessing module to the standard library - preferably in the 2.6 timeline. In March, I began working on the PEP for the inclusion of the pyprocessing (processing) module into the python standard library[1]. The original email to the stdlib-sig can be found here, it includes a basic overview of the module: http://mail.python.org/pipermail/stdlib-sig/2008-March/000129.html The processing module mirrors/mimics the API of the threading module - and with simple import/subclassing changes depending on the code, allows you to leverage multi core machines via an underlying forking mechanism. The module also supports the sharing of data across groups of networked machines - a feature obviously not part of the core threading module, but useful in a distributed environment. I think processing looks interesting and useful, especially since it works on Windows as well as Un*x. However I'd like to see a review of the security - anything which can run across networks of machines has security implications and I didn't see these spelt out in the documentation. Networked running should certainly be disabled by default and need explicitly enabling by the user - I'd hate for a new version of python to come with a remote exploit by default... As long as the ctypes extension doesn't build on major Un*x platforms (AIX, HP-UX), I don't like to see ctypes dependend modules included into the stdlib. Please keep the stdlib as portable as possible. More and more people tend to say Runs on Un*x when they really mean Tested on Linux. Un*x is not Linux. Ulli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Ulrich Berning schrieb: Nick Craig-Wood wrote: Jesse Noller [EMAIL PROTECTED] wrote: I am looking for any questions, concerns or benchmarks python-dev has regarding the possible inclusion of the pyprocessing module to the standard library - preferably in the 2.6 timeline. In March, I began working on the PEP for the inclusion of the pyprocessing (processing) module into the python standard library[1]. The original email to the stdlib-sig can be found here, it includes a basic overview of the module: http://mail.python.org/pipermail/stdlib-sig/2008-March/000129.html The processing module mirrors/mimics the API of the threading module - and with simple import/subclassing changes depending on the code, allows you to leverage multi core machines via an underlying forking mechanism. The module also supports the sharing of data across groups of networked machines - a feature obviously not part of the core threading module, but useful in a distributed environment. I think processing looks interesting and useful, especially since it works on Windows as well as Un*x. However I'd like to see a review of the security - anything which can run across networks of machines has security implications and I didn't see these spelt out in the documentation. Networked running should certainly be disabled by default and need explicitly enabling by the user - I'd hate for a new version of python to come with a remote exploit by default... As long as the ctypes extension doesn't build on major Un*x platforms (AIX, HP-UX), I don't like to see ctypes dependend modules included into the stdlib. Please keep the stdlib as portable as possible. More and more people tend to say Runs on Un*x when they really mean Tested on Linux. Un*x is not Linux. Hm, I've never looked at the processing module. Does it depend on ctypes? Anyway, the trunk version of ctypes is a lot more portable than the release25-maint version. I have once tried to build the trunk on HP-UX machines, and, IIRC, it did build on IA64 and PA machines. Of course only with GCC, not with the vendor compilers. Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On Fri, May 16, 2008 at 11:22 AM, Thomas Heller [EMAIL PROTECTED] wrote: Ulrich Berning schrieb: Nick Craig-Wood wrote: Jesse Noller [EMAIL PROTECTED] wrote: I am looking for any questions, concerns or benchmarks python-dev has regarding the possible inclusion of the pyprocessing module to the standard library - preferably in the 2.6 timeline. In March, I began working on the PEP for the inclusion of the pyprocessing (processing) module into the python standard library[1]. The original email to the stdlib-sig can be found here, it includes a basic overview of the module: http://mail.python.org/pipermail/stdlib-sig/2008-March/000129.html The processing module mirrors/mimics the API of the threading module - and with simple import/subclassing changes depending on the code, allows you to leverage multi core machines via an underlying forking mechanism. The module also supports the sharing of data across groups of networked machines - a feature obviously not part of the core threading module, but useful in a distributed environment. I think processing looks interesting and useful, especially since it works on Windows as well as Un*x. However I'd like to see a review of the security - anything which can run across networks of machines has security implications and I didn't see these spelt out in the documentation. Networked running should certainly be disabled by default and need explicitly enabling by the user - I'd hate for a new version of python to come with a remote exploit by default... As long as the ctypes extension doesn't build on major Un*x platforms (AIX, HP-UX), I don't like to see ctypes dependend modules included into the stdlib. Please keep the stdlib as portable as possible. More and more people tend to say Runs on Un*x when they really mean Tested on Linux. Un*x is not Linux. Hm, I've never looked at the processing module. Does it depend on ctypes? Anyway, the trunk version of ctypes is a lot more portable than the release25-maint version. I have once tried to build the trunk on HP-UX machines, and, IIRC, it did build on IA64 and PA machines. Of course only with GCC, not with the vendor compilers. Thomas Yes, it requires ctypes. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
2008/5/13 Jesse Noller [EMAIL PROTECTED]: I am looking for any questions, concerns or benchmarks python-dev has regarding the possible inclusion of the pyprocessing module to the standard library - preferably in the 2.6 timeline. In March, I began +1 to include this module in the library, if... - it's included in 2.7/3.1 after a stabilization period. - it's stop reinventing the wheel and starts using the already-in-stdlib infrastructure (this is from Martin's post). - a champion appears willing to work with it and maintain it afterwards (which is the opinion of the module owner about this?) I think that including in the stdlib a threading-like-API module is a clear win. Of course, this module won't solve all the problems or necessities of multiprocessing world, but it's an included easy to use start (in the same spirit that SimpleHTTPServer it's not an Apache, but it's very useful if your problem area fits its limitations). Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
[Facundo] I think that including in the stdlib a threading-like-API module is a clear win. I also think this is vital and should not wait for Py2.7. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Apart from (1) and (2) which I think are unavoidable, nearly all the C code is Windows specific. I assume you don't want to bring in the whole of pywin32 into stdlib... I wouldn't structure it the same way as pywin32, but yes, I have pondered exposing all of Win32 in a module, as part of the standard library. This has somewhat lost relevance with ctypes. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On Fri, May 16, 2008 at 1:32 AM, Ulrich Berning [EMAIL PROTECTED] wrote: As long as the ctypes extension doesn't build on major Un*x platforms (AIX, HP-UX), I don't like to see ctypes dependend modules included into the stdlib. Please keep the stdlib as portable as possible. Nice in theory but ctypes already works on at least the top 3 popular platforms. Lets not hold Python's stdlib back because nobody who uses IBM and HP proprietary stuff has contributed the necessary support. Making nice libraries available for other platforms is a good way to encourage people to either pitch in and add support or consider their platform choices in the future. -gps ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
-1 on multicore - multiprocess or multiprocessing are a fine names. cores are irrelevant. Thanks you for saying that. I'm constantly amazed how people think multi-core systems are a new thing, conceptually, when they are really just a different packaging for multi-processor systems (perhaps with NUMA consequences that the operating system, but not application, needs to deal with). So -1 on multicore from me as well, despise marketing (typo intentional :-) Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Gregory P. Smith wrote: -1 on multicore - multiprocess or multiprocessing are a fine names. cores are irrelevant. systems have multiple cpus real or virtual regardless of how many dies, sockets and cores there are. +0.5 on inclusion. that means i am happy if it does but don't think it needs to make it into 2.6/3.0. leave inclusion for 2.7/3.1. its easy for people to install from an external source for now if they want it. This is a nice summary of my opinion as well. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Gregory P. Smith schrieb: +0.5 on inclusion. that means i am happy if it does but don't think it needs to make it into 2.6/3.0. leave inclusion for 2.7/3.1. its easy for people to install from an external source for now if they want it. I'm still -0.5 on inclusion *NOW*, but +1 on inclusion in 2.7/3.1. IMHO pyprocessing isn't mature enough - the author has labeled it as 'beta' - and the proposal came too late. It's going to take a while to give the package a security audit and integrate the C code into Python. Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Jesse Noller wrote: I am looking for any questions, concerns or benchmarks python-dev has regarding the possible inclusion of the pyprocessing module to the standard library - preferably in the 2.6 timeline. I think for inclusion in 2.6 it's to late. For 3.0, it's definitely too late - the PEP acceptance deadline was a year ago (IIRC). As I am trying to finish up the PEP, I want to see if I can address any questions or include any other useful data (including benchmarks) in the PEP prior to publishing it. I am also intending to include basic benchmarks for both the processing module against the threading module as a comparison. I'm worried whether it's stable, what user base it has, whether users (other than the authors) are lobbying for inclusion. Statistically, it seems to be not ready yet: it is not even a year old, and has not reached version 1.0 yet. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Martin v. Löwis schrieb: I'm worried whether it's stable, what user base it has, whether users (other than the authors) are lobbying for inclusion. Statistically, it seems to be not ready yet: it is not even a year old, and has not reached version 1.0 yet. I'm on Martin's side here. Although I like to see some sort of multi processing mechanism in Python 'cause I need it for lots of projects I'm against the inclusion of pyprocessing in 2.6 and 3.0. The project isn't old and mature enough and it has some competitors like pp (parallel processing). On the one hand the inclusion of a package gives it an unfair advantage over similar packages. On the other hand it slows down future development because a new feature release must be synced with Python releases about every 1.5 years. -0.5 from me Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Christian Heimes wrote: Martin v. Löwis schrieb: I'm worried whether it's stable, what user base it has, whether users (other than the authors) are lobbying for inclusion. Statistically, it seems to be not ready yet: it is not even a year old, and has not reached version 1.0 yet. I'm on Martin's side here. Although I like to see some sort of multi processing mechanism in Python 'cause I need it for lots of projects I'm against the inclusion of pyprocessing in 2.6 and 3.0. The project isn't old and mature enough and it has some competitors like pp (parallel processing). On the one hand the inclusion of a package gives it an unfair advantage over similar packages. On the other hand it slows down future development because a new feature release must be synced with Python releases about every 1.5 years. -0.5 from me It also isn't something which needs to be done *right now*. Leaving this for 3.x/2.7 seems like a much better idea to me. With the continuing rise of multi-processor desktop machines, the parallel processing approaches that are out there should see a fair amount of use and active development over the next 18 months. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On Wed, May 14, 2008 at 6:58 AM, Nick Craig-Wood [EMAIL PROTECTED] wrote: Jesse Noller [EMAIL PROTECTED] wrote: I am looking for any questions, concerns or benchmarks python-dev has regarding the possible inclusion of the pyprocessing module to the standard library - preferably in the 2.6 timeline. In March, I began working on the PEP for the inclusion of the pyprocessing (processing) module into the python standard library[1]. The original email to the stdlib-sig can be found here, it includes a basic overview of the module: http://mail.python.org/pipermail/stdlib-sig/2008-March/000129.html The processing module mirrors/mimics the API of the threading module - and with simple import/subclassing changes depending on the code, allows you to leverage multi core machines via an underlying forking mechanism. The module also supports the sharing of data across groups of networked machines - a feature obviously not part of the core threading module, but useful in a distributed environment. I think processing looks interesting and useful, especially since it works on Windows as well as Un*x. However I'd like to see a review of the security - anything which can run across networks of machines has security implications and I didn't see these spelt out in the documentation. Networked running should certainly be disabled by default and need explicitly enabling by the user - I'd hate for a new version of python to come with a remote exploit by default... -- Nick Craig-Wood [EMAIL PROTECTED] -- http://www.craig-wood.com/nick See the Manager documentation: http://pyprocessing.berlios.de/doc/manager-objects.html And the Listener/Client documentation: http://pyprocessing.berlios.de/doc/connection-ref.html Remote access is not implicit - it is explicit - you must spawn a Manager/Client instance and tell it to use the network instead of it being always networked. I'll add a security audit to the list of open issues though - that's a good point. -jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On 2008-05-14 14:15, Jesse Noller wrote: On Wed, May 14, 2008 at 5:45 AM, Christian Heimes [EMAIL PROTECTED] wrote: Martin v. Löwis schrieb: I'm worried whether it's stable, what user base it has, whether users (other than the authors) are lobbying for inclusion. Statistically, it seems to be not ready yet: it is not even a year old, and has not reached version 1.0 yet. I'm on Martin's side here. Although I like to see some sort of multi processing mechanism in Python 'cause I need it for lots of projects I'm against the inclusion of pyprocessing in 2.6 and 3.0. The project isn't old and mature enough and it has some competitors like pp (parallel processing). On the one hand the inclusion of a package gives it an unfair advantage over similar packages. On the other hand it slows down future development because a new feature release must be synced with Python releases about every 1.5 years. -0.5 from me Christian I said this in reply to Martin - but the competitors (in my mind) are not as compelling due to the alternative paradigm for application construction they propose. The processing module is an easy win for us if included. Personally - I don't see how inclusion in the stdlib would slow down development - yes, you have to stick with the same release cycle as python-core, but if the module is feature complete and provides a stable API as it stands I don't see following python-core timelines as overly onerous. The module itself doesn't change that frequently - the last release in April was a bugfix release and API consistency change (the API would have to be locked for inclusion obviously - targeting a 2.7/3.1 release may be advantageous to achieve this). Why don't you start a parallel-sig and then hash this out with other distributed computing users ? You could then reach a decision by the time 2.7 is scheduled for release and then add the chosen module to the stdlib. The API of the processing module does look simple and nice, but parallel processing is a minefield - esp. when it comes to handling error situations (e.g. a worker failing, network going down, fail-over, etc.). What I'm missing with the processing module is a way to spawn processes on clusters (rather than just on a single machine). In the scientific world, MPI is the standard API of choice for doing parallel processing, so if we're after standards, supporting MPI would seem to be more attractive than the processing module. http://pypi.python.org/pypi/mpi4py In the enterprise world, you often find CORBA based solutions. http://omniorb.sourceforge.net/ And then, of course, you have a gazillion specialized solutions such as PyRO: http://pyro.sourceforge.net/ OTOH, perhaps the stdlib should just include entry-level support for some form of parallel processing, in which case processing does look attractive. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 14 2008) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On Wed, May 14, 2008 at 11:46 AM, M.-A. Lemburg [EMAIL PROTECTED] wrote: On 2008-05-14 14:15, Jesse Noller wrote: On Wed, May 14, 2008 at 5:45 AM, Christian Heimes [EMAIL PROTECTED] wrote: Martin v. Löwis schrieb: I'm worried whether it's stable, what user base it has, whether users (other than the authors) are lobbying for inclusion. Statistically, it seems to be not ready yet: it is not even a year old, and has not reached version 1.0 yet. I'm on Martin's side here. Although I like to see some sort of multi processing mechanism in Python 'cause I need it for lots of projects I'm against the inclusion of pyprocessing in 2.6 and 3.0. The project isn't old and mature enough and it has some competitors like pp (parallel processing). On the one hand the inclusion of a package gives it an unfair advantage over similar packages. On the other hand it slows down future development because a new feature release must be synced with Python releases about every 1.5 years. -0.5 from me Christian I said this in reply to Martin - but the competitors (in my mind) are not as compelling due to the alternative paradigm for application construction they propose. The processing module is an easy win for us if included. Personally - I don't see how inclusion in the stdlib would slow down development - yes, you have to stick with the same release cycle as python-core, but if the module is feature complete and provides a stable API as it stands I don't see following python-core timelines as overly onerous. The module itself doesn't change that frequently - the last release in April was a bugfix release and API consistency change (the API would have to be locked for inclusion obviously - targeting a 2.7/3.1 release may be advantageous to achieve this). Why don't you start a parallel-sig and then hash this out with other distributed computing users ? You could then reach a decision by the time 2.7 is scheduled for release and then add the chosen module to the stdlib. The API of the processing module does look simple and nice, but parallel processing is a minefield - esp. when it comes to handling error situations (e.g. a worker failing, network going down, fail-over, etc.). What I'm missing with the processing module is a way to spawn processes on clusters (rather than just on a single machine). In the scientific world, MPI is the standard API of choice for doing parallel processing, so if we're after standards, supporting MPI would seem to be more attractive than the processing module. http://pypi.python.org/pypi/mpi4py In the enterprise world, you often find CORBA based solutions. http://omniorb.sourceforge.net/ And then, of course, you have a gazillion specialized solutions such as PyRO: http://pyro.sourceforge.net/ OTOH, perhaps the stdlib should just include entry-level support for some form of parallel processing, in which case processing does look attractive. -- Marc-Andre Lemburg eGenix.com Thanks for bringing up something I was going to mention - I am not attempting to solve the distributed computing problem with this proposal - you are right in mentioning there's a variety of technologies out there for achieving true loosely-coupled distributed computing, including all of those which you pointed out. I am proposing exactly what you mentioned: Entry level parallel processing. The fact that the processing module does have remote capabilities is a bonus: Not core to the proposal. While in a perfect world - a system might exist which truly insulates programmers from the difference between local concurrency and distributed systems - the two are really different problems. My concern is the taking advantage of the 8 core machine sitting under my desk (or the 10 or so I have in the lab)- the processing module allows me to do that - easily. The module is basic enough to be flexible in other technologies to use with it for highly distributed systems but it is also simple enough to act as an entry point for those people just starting out in the domain. Think of it like the difference between asyncore and Twisted. I could easily see more loosely-coupled-highly-distributed tools being built on top of the basics it has provided. -jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On Wed, May 14, 2008 at 05:46:25PM +0200, M.-A. Lemburg wrote: What I'm missing with the processing module is a way to spawn processes on clusters (rather than just on a single machine). In the scientific world, MPI is the standard API of choice for doing parallel processing, so if we're after standards, supporting MPI would seem to be more attractive than the processing module. Think of the processing module as an alternative to the threading module, not as an alternative to MPI. In Python, multi-threading can be extremely slow. The processing module gives you a way to convert from using multiple threads to using multiple processes. If it made people feel better, maybe it should be called threading2 instead of multiprocessing. The word processing seems to make people think of parallel processing and clusters, which is missing the point. Anyway, I would love to see the processing module included in the standard library. -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868 pgp42e26gDfkd.pgp Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On Wed, May 14, 2008 at 08:06:15AM -0400, Jesse Noller wrote: Overwhelmingly, many of the python programmers I spoke to are looking for a solution that does not require the alteration of a known programming paradigm (i.e: threads) to allow them to take advantage of systems which are not getting faster - instead, they are getting wider. Simply put due to the GIL - pure python applications can not take advantage of these machines which are now more common than not without switching to an alternative interpreter - which for many - myself included - is not an attractive option. On Newegg, there are currently 15 single core processors listed, but there are 57 dual core processors and 52 quad core processors. By the time Python 2.6 comes out, it will be hard to buy a new computer without multiple cores. In my opinion, the sooner Python has a nice simple library for inter-process communication, the better. -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868 pgp4U92STA7VZ.pgp Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
In the scientific world, MPI is the standard API of choice for doing parallel processing, so if we're after standards, supporting MPI would seem to be more attractive than the processing module. http://pypi.python.org/pypi/mpi4py Of course, for MPI, pyprocessing's main functionality (starting new activities) isn't needed - you use the vendor's mpirun binary, which will create as many processes as you wish, following a policy that was set up by the cluster administration, or that you chose in a product-specific manner (e.g. what nodes to involve in the job). If my task was high-performance computing, I would indeed use MPI and ignore pyprocessing. In the enterprise world, you often find CORBA based solutions. http://omniorb.sourceforge.net/ Same here: I would prefer CORBA of pyprocessing when I want a componentized, distributed application. And then, of course, you have a gazillion specialized solutions such as PyRO: http://pyro.sourceforge.net/ I personally would not use that library, although I know others are very fond of it. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
I really do feel that inclusion of this library offers us the best of both worlds - it gives us (as a community) an easy answer to those people who would dismiss python due to the GIL and it also allows users to easily implement their applications. I really feel that you can get the best of both worlds even without inclusion of the module in the standard library. It should be fairly easy to install, assuming pre-compiled packages are available. For inclusion into Python, a number of things need to be changed, in particular with respect to the C code. It duplicates a lot of code from the socket and os modules, and should (IMO) be merged into these, rather than being separate - or perhaps merged into the subprocess module. Why isn't it possible to implement all this in pure Python, on top of the subprocess module? If there are limitations in Python that make it impossible to implement such functionality in pure Python - then *those* limitations should be overcome, rather than including the code wholesale. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On May 14, 2008, at 12:32 PM, Andrew McNabb wrote: Think of the processing module as an alternative to the threading module, not as an alternative to MPI. In Python, multi-threading can be extremely slow. The processing module gives you a way to convert from using multiple threads to using multiple processes. If it made people feel better, maybe it should be called threading2 instead of multiprocessing. The word processing seems to make people think of parallel processing and clusters, which is missing the point. Anyway, I would love to see the processing module included in the standard library. Is the goal of the pyprocessing module to be exactly drop in compatible with threading, Queue and friends? I guess the idea would be that if my problem is compute bound I'd use pyprocessing and if it was I/O bound I might just use the existing threading library? Can I write a program using only threading and Queue interfaces for inter-thread communication and just change my import statements and have my program work? Currently, it looks like the pyprocessing.Queue interface is slightly different than than Queue.Queue for example (no task_done() etc). Perhaps a stdlib version of pyprocessing could be simplified down to not try to be a cross-machine computing environment and just be a same- machine threading replacement? This would make the maintenance easier and reduce confusion with users about how they should do cross-machine multiprocessing. By the way, a thread-safe simple dict in the style of Queue would be extremely helpful in writing multi-threaded programs, whether using the threading or pyprocessing modules. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Andrew McNabb [EMAIL PROTECTED] wrote: Think of the processing module as an alternative to the threading module, not as an alternative to MPI. In Python, multi-threading can be extremely slow. The processing module gives you a way to convert from using multiple threads to using multiple processes. Indeed; pyprocessing was exactly what I wanted, and worked exactly as it said it did. It had essentially no learning curve at all because of its implementation of essentially the same interface as the threading module. It's remoting capabilities are almost surplus to requirements; if I wanted that, I might use MPI or something else. If it made people feel better, maybe it should be called threading2 instead of multiprocessing. The word processing seems to make people think of parallel processing and clusters, which is missing the point. threading is to threads as processing is to processes; that's why it was named processing. But the choice of name shouldn't affect the decision as to whether it should be included or not. Anyway, I would love to see the processing module included in the standard library. I would as well; I'm using it in a current project, and can see opportunities for it to be useful in other projects. My only note is that based on my experience, the code needs a little cleanup for portability reasons before it is included with the Python standard library -- it worked fine as-is on Linux, but needed some hackery to get running on a slightly elderly Solaris 8 box. I didn't test it on anything else. Charles -- --- Charles Cazabon GPL'ed software available at: http://pyropus.ca/software/ --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
M.-A. Lemburg wrote: The API of the processing module does look simple and nice, but parallel processing is a minefield - esp. when it comes to handling error situations (e.g. a worker failing, network going down, fail-over, etc.). What I'm missing with the processing module is a way to spawn processes on clusters (rather than just on a single machine). Perhaps one-size-fits-all isn't the right approach here. I think there's room for more than one module -- a simple one for people who just want to spawn some extra processes on the same CPU to take advantage of multiple cores, and a fancier one (maybe based on MPI) for people who want grid-computing style distribution with error handling, fault tolerance, etc. (I didn't set out to justify that paragraph, btw -- it just happened!) -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Andrew McNabb wrote: If it made people feel better, maybe it should be called threading2 instead of multiprocessing. I think that errs in the other direction, making it sound like just another way of doing single-process threading, which it's not. Maybe multicore would help give the right impression? (Still allows for networking -- nobody says all the cores have to be on the same machine.:-) -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Charles Cazabon wrote: threading is to threads as processing is to processes; that's why it was named processing. Unfortunately, the word processing is already used in the field of computing with a very general meaning -- any kind of data transfomation at all can be, and is, referred to as processing. So the intended meaning fails to come across. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Greg Maybe multicore would help give the right impression? multiproc, multiprocess? Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
At 12:19 PM 5/15/2008 +1200, Greg Ewing wrote: Andrew McNabb wrote: If it made people feel better, maybe it should be called threading2 instead of multiprocessing. I think that errs in the other direction, making it sound like just another way of doing single-process threading, which it's not. Maybe multicore would help give the right impression? Sounds like a marketing win to me, since it directly addresses the python doesn't do multicore meme. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On Wed, May 14, 2008 at 6:48 PM, Phillip J. Eby [EMAIL PROTECTED] wrote: At 12:19 PM 5/15/2008 +1200, Greg Ewing wrote: Andrew McNabb wrote: If it made people feel better, maybe it should be called threading2 instead of multiprocessing. I think that errs in the other direction, making it sound like just another way of doing single-process threading, which it's not. Maybe multicore would help give the right impression? Sounds like a marketing win to me, since it directly addresses the python doesn't do multicore meme. -1 on multicore - multiprocess or multiprocessing are a fine names. cores are irrelevant. systems have multiple cpus real or virtual regardless of how many dies, sockets and cores there are. +0.5 on inclusion. that means i am happy if it does but don't think it needs to make it into 2.6/3.0. leave inclusion for 2.7/3.1. its easy for people to install from an external source for now if they want it. -gps ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Addition of pyprocessing module to standard lib.
I am looking for any questions, concerns or benchmarks python-dev has regarding the possible inclusion of the pyprocessing module to the standard library - preferably in the 2.6 timeline. In March, I began working on the PEP for the inclusion of the pyprocessing (processing) module into the python standard library[1]. The original email to the stdlib-sig can be found here, it includes a basic overview of the module: http://mail.python.org/pipermail/stdlib-sig/2008-March/000129.html The processing module mirrors/mimics the API of the threading module - and with simple import/subclassing changes depending on the code, allows you to leverage multi core machines via an underlying forking mechanism. The module also supports the sharing of data across groups of networked machines - a feature obviously not part of the core threading module, but useful in a distributed environment. As I am trying to finish up the PEP, I want to see if I can address any questions or include any other useful data (including benchmarks) in the PEP prior to publishing it. I am also intending to include basic benchmarks for both the processing module against the threading module as a comparison. -Jesse [1] Processing page: http://pyprocessing.berlios.de/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Jesse Noller wrote: I am looking for any questions, concerns or benchmarks python-dev has regarding the possible inclusion of the pyprocessing module to the standard library Sounds good, but I'd suggest giving a more specific name than processing, which is so generic as to be meaningless. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
On May 13, 2008, at 8:57 PM, Greg Ewing [EMAIL PROTECTED] wrote: Jesse Noller wrote: I am looking for any questions, concerns or benchmarks python-dev has regarding the possible inclusion of the pyprocessing module to the standard library Sounds good, but I'd suggest giving a more specific name than processing, which is so generic as to be meaningless. -- Greg Any suggestions? Subprocess is taken and concurrent might be presumptuous unless it was added as concurrent.processing or some like permutation. -Jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Why not use MPI? It's cross platform, cross language and very widely supported already. And there're Python bindings already. On May 13, 2008, at 8:52 PM, Jesse Noller wrote: I am looking for any questions, concerns or benchmarks python-dev has regarding the possible inclusion of the pyprocessing module to the standard library - preferably in the 2.6 timeline. In March, I began working on the PEP for the inclusion of the pyprocessing (processing) module into the python standard library[1]. The original email to the stdlib-sig can be found here, it includes a basic overview of the module: http://mail.python.org/pipermail/stdlib-sig/2008-March/000129.html The processing module mirrors/mimics the API of the threading module - and with simple import/subclassing changes depending on the code, allows you to leverage multi core machines via an underlying forking mechanism. The module also supports the sharing of data across groups of networked machines - a feature obviously not part of the core threading module, but useful in a distributed environment. As I am trying to finish up the PEP, I want to see if I can address any questions or include any other useful data (including benchmarks) in the PEP prior to publishing it. I am also intending to include basic benchmarks for both the processing module against the threading module as a comparison. -Jesse [1] Processing page: http://pyprocessing.berlios.de/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/thomaspinckney3%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
The inclusion of the processing module does not exclude the potential to also use or one day include MPI bindings. The goal is to add a module with a known API and semantics which allows programmer using cPython to easily take advantage of multple cores (and as a side benefit, machines). In theory - one could also use MPI within an application using the processing module as a communications/message passing system. -Jesse On May 13, 2008, at 9:23 PM, Tom Pinckney [EMAIL PROTECTED] wrote: Why not use MPI? It's cross platform, cross language and very widely supported already. And there're Python bindings already. On May 13, 2008, at 8:52 PM, Jesse Noller wrote: I am looking for any questions, concerns or benchmarks python-dev has regarding the possible inclusion of the pyprocessing module to the standard library - preferably in the 2.6 timeline. In March, I began working on the PEP for the inclusion of the pyprocessing (processing) module into the python standard library[1]. The original email to the stdlib-sig can be found here, it includes a basic overview of the module: http://mail.python.org/pipermail/stdlib-sig/2008-March/000129.html The processing module mirrors/mimics the API of the threading module - and with simple import/subclassing changes depending on the code, allows you to leverage multi core machines via an underlying forking mechanism. The module also supports the sharing of data across groups of networked machines - a feature obviously not part of the core threading module, but useful in a distributed environment. As I am trying to finish up the PEP, I want to see if I can address any questions or include any other useful data (including benchmarks) in the PEP prior to publishing it. I am also intending to include basic benchmarks for both the processing module against the threading module as a comparison. -Jesse [1] Processing page: http://pyprocessing.berlios.de/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/thomaspinckney3%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
multiprocessing Greg Ewing wrote: Jesse Noller wrote: I am looking for any questions, concerns or benchmarks python-dev has regarding the possible inclusion of the pyprocessing module to the standard library Sounds good, but I'd suggest giving a more specific name than processing, which is so generic as to be meaningless. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Talin wrote: multiprocessing multiprocessing would work for me - it adds the concurrency implications that threading has, but 'processing' on its own would be missing. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Tom Pinckney wrote: Why not use MPI? It's cross platform, cross language and very widely supported already. And there're Python bindings already. The point of pyprocessing is that fact that the API is the same as that of the threading module - making it very easy to convert a multithreaded program to a multiprocessing one, and hence making it easy to exploit multiple processors in a CPU bound Python program, regardless of the restrictions imposed by Python's GIL. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of pyprocessing module to standard lib.
Nick Coghlan wrote: Talin wrote: multiprocessing multiprocessing would work for me Me, too. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
