[Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt

2010-01-07 Thread Yoann Padioleau
Hi, I would like to use astgen.py to generate python classes corresponding to the AST of something I have defined in a .asdl file, along the line of what is apparently done for the python AST itself. I thought astgen.py would take as an argument a .asdl file, but apparently it instead process a

[Python-Dev] Backported faster RLock to Python 2.6.

2010-01-07 Thread Johan Gill
Hi devs, the company where I work has done some work on Python, and the question is how this work, owned by the company, can be contributed to the community properly. Are there any license issues or other pitfalls we need to think about? I imagine that other companies have contributed before,

Re: [Python-Dev] Backported faster RLock to Python 2.6.

2010-01-07 Thread Lennart Regebro
On Thu, Jan 7, 2010 at 10:46, Johan Gill johan.g...@agama.tv wrote: Hi devs, the company where I work has done some work on Python, and the question is how this work, owned by the company, can be contributed to the community properly. Are there any license issues or other pitfalls we need to

Re: [Python-Dev] Backported faster RLock to Python 2.6.

2010-01-07 Thread Simon Cross
On Thu, Jan 7, 2010 at 1:12 PM, Lennart Regebro rege...@gmail.com wrote: I'm not a license lawyer, but typically your company needs to give the code to the community. Yes, it means it stops owning it. This is incorrect. The correct information is at http://www.python.org/psf/contrib/. Schiavo

[Python-Dev] test_ctypes failure on AIX 5.3 using python-2.6.2 and libffi-3.0.9

2010-01-07 Thread swamy sangamesh
Hi All, I built the python-2.6.2 with the latest libffi-3.0.9 in AIX 5.3 using xlc compiler. When i try to run the ctypes test cases, two failures are seen in test_bitfields. *test_ints (ctypes.test.test_bitfields.C_Test) ... FAIL test_shorts (ctypes.test.test_bitfields.C_Test) ... FAIL* I have

Re: [Python-Dev] Backported faster RLock to Python 2.6.

2010-01-07 Thread Nick Coghlan
Lennart Regebro wrote: On Thu, Jan 7, 2010 at 10:46, Johan Gill johan.g...@agama.tv wrote: Hi devs, the company where I work has done some work on Python, and the question is how this work, owned by the company, can be contributed to the community properly. Are there any license issues or

Re: [Python-Dev] Backported faster RLock to Python 2.6.

2010-01-07 Thread Lennart Regebro
On Thu, Jan 7, 2010 at 13:23, Nick Coghlan ncogh...@gmail.com wrote: As Simon pointed out, while some organisations do work that way, the PSF isn't one of them. The PSF only requires that the code be contributed under a license that then allows us to turn around and redistribute it under a

Re: [Python-Dev] Backported faster RLock to Python 2.6.

2010-01-07 Thread Michael Foord
On 07/01/2010 13:11, Lennart Regebro wrote: On Thu, Jan 7, 2010 at 13:23, Nick Coghlanncogh...@gmail.com wrote: As Simon pointed out, while some organisations do work that way, the PSF isn't one of them. The PSF only requires that the code be contributed under a license that then allows

Re: [Python-Dev] GIL required for _all_ Python calls?

2010-01-07 Thread Stefan Behnel
Guido van Rossum, 07.01.2010 05:29: A better rule would be you may access the memory buffer in a PyString or PyUnicode object with the GIL released as long as you own a reference to the string object. Everything else is out of bounds (or not worth the bother). Is that a yes regarding the OP's

Re: [Python-Dev] Backported faster RLock to Python 2.6.

2010-01-07 Thread Lennart Regebro
On Thu, Jan 7, 2010 at 14:15, Michael Foord fuzzy...@voidspace.org.uk wrote: (i.e. copyright and ownership are legal terms that don't necessarily mean anything *practical* in these situations.) OK, fair enough. :-) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33

Re: [Python-Dev] GIL required for _all_ Python calls?

2010-01-07 Thread Antoine Pitrou
MRAB python at mrabarnett.plus.com writes: I know that it needs to have the GIL during memory-management calls, but does it for calls like Py_UNICODE_TOLOWER or PyErr_SetString? Is there an easy way to find out? There is no easy way to do so. The only safe way is to examine all the functions

Re: [Python-Dev] Question over splitting unittest into a package

2010-01-07 Thread Olemis Lang
On Mon, Jan 4, 2010 at 9:24 AM, Olemis Lang ole...@gmail.com wrote: On Thu, Dec 31, 2009 at 10:30 AM, Martin (gzlist) gzl...@googlemail.com wrote: Thanks for the quick response. On 30/12/2009, Benjamin Peterson benja...@python.org wrote: but maybe a discussion could start about a new,

Re: [Python-Dev] GIL required for _all_ Python calls?

2010-01-07 Thread Martin v. Löwis
A better rule would be you may access the memory buffer in a PyString or PyUnicode object with the GIL released as long as you own a reference to the string object. Everything else is out of bounds (or not worth the bother). Is that a yes regarding the OP's original question about releasing

Re: [Python-Dev] GIL required for _all_ Python calls?

2010-01-07 Thread Martin v. Löwis
I've been wondering whether it's possible to release the GIL in the regex engine during matching. I don't think that's possible. The regex engine can also operate on objects whose representation may move in memory when you don't hold the GIL (e.g. buffers that get mutated). Even if they stay in

Re: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt

2010-01-07 Thread Martin v. Löwis
I would like to use astgen.py to generate python classes corresponding to the AST of something I have defined in a .asdl file, along the line of what is apparently done for the python AST itself. I thought astgen.py would take as an argument a .asdl file, but apparently it instead process a

Re: [Python-Dev] GIL required for _all_ Python calls?

2010-01-07 Thread James Y Knight
On Jan 7, 2010, at 3:27 PM, Martin v. Löwis wrote: I've been wondering whether it's possible to release the GIL in the regex engine during matching. I don't think that's possible. The regex engine can also operate on objects whose representation may move in memory when you don't hold the GIL

Re: [Python-Dev] GIL required for _all_ Python calls?

2010-01-07 Thread Martin v. Löwis
I've been wondering whether it's possible to release the GIL in the regex engine during matching. I don't think that's possible. The regex engine can also operate on objects whose representation may move in memory when you don't hold the GIL (e.g. buffers that get mutated). Even if they stay

Re: [Python-Dev] Backported faster RLock to Python 2.6.

2010-01-07 Thread Nick Coghlan
Johan Gill wrote: Yes, it is the new RLock implementation. If I understood this correctly, we should make a patch against trunk if anything should be contributed. Yep. Do you mean that we wouldn't need the paperwork for backporting the original patch committed to py3k? Whether or not a

Re: [Python-Dev] GIL required for _all_ Python calls?

2010-01-07 Thread Antoine Pitrou
Martin v. Löwis martin at v.loewis.de writes: I don't think that's possible. The regex engine can also operate on objects whose representation may move in memory when you don't hold the GIL (e.g. buffers that get mutated). Why is it a problem? If we get a buffer through the new buffer API,

Re: [Python-Dev] GIL required for _all_ Python calls?

2010-01-07 Thread Martin v. Löwis
I've been wondering whether it's possible to release the GIL in the regex engine during matching. Ok, here is another problem: SRE_OP_REPEAT uses PyObject_MALLOC, which requires the GIL (it then also may call PyErr_NoMemory, which also requires the GIL). Regards, Martin

Re: [Python-Dev] Backported faster RLock to Python 2.6.

2010-01-07 Thread Johan Gill
On 01/07/2010 01:23 PM, Nick Coghlan wrote: As Simon pointed out, while some organisations do work that way, the PSF isn't one of them. The PSF only requires that the code be contributed under a license that then allows us to turn around and redistribute it under a different open source license

Re: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt

2010-01-07 Thread Yoann Padioleau
On Jan 7, 2010, at 12:31 PM, Martin v. Löwis wrote: I would like to use astgen.py to generate python classes corresponding to the AST of something I have defined in a .asdl file, along the line of what is apparently done for the python AST itself. I thought astgen.py would take as an

Re: [Python-Dev] GIL required for _all_ Python calls?

2010-01-07 Thread Martin v. Löwis
I don't think that's possible. The regex engine can also operate on objects whose representation may move in memory when you don't hold the GIL (e.g. buffers that get mutated). Why is it a problem? If we get a buffer through the new buffer API, the object should ensure that the

Re: [Python-Dev] relation between Python.asdl and Tools/compiler/ast.txt

2010-01-07 Thread Martin v. Löwis
astgen.py is not used to process asdl files; ast.txt lives right next to astgen.py. Instead, the asdl file is processed by Parser/asdl_c.py. Yes, I know that. That's why I asked about the relation between ast.txt and Python.adsl. If internally the parser uses the .adsl, but expose as a

[Python-Dev] Improve open() to support reading file starting with an unicode BOM

2010-01-07 Thread Victor Stinner
Hi, Builtin open() function is unable to open an UTF-16/32 file starting with a BOM if the encoding is not specified (raise an unicode error). For an UTF-8 file starting with a BOM, read()/readline() returns also the BOM whereas the BOM should be ignored. See recent issues related to reading

Re: [Python-Dev] Improve open() to support reading file starting with an unicode BOM

2010-01-07 Thread Guido van Rossum
I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy talk. And for the other two, perhaps it would make more sense to have a separate encoding-guessing function that takes a binary stream and returns a text stream wrapping it with the proper encoding? --Guido On Thu, Jan 7, 2010

Re: [Python-Dev] Improve open() to support reading file starting with an unicode BOM

2010-01-07 Thread MRAB
Guido van Rossum wrote: I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy talk. And for the other two, perhaps it would make more sense to have a separate encoding-guessing function that takes a binary stream and returns a text stream wrapping it with the proper encoding?

Re: [Python-Dev] --enabled-shared broken on freebsd5?

2010-01-07 Thread Nicholas Bastin
I think this problem probably needs to move over to distutils-sig, as it doesn't seem to be specific to the way that Python itself uses distutils. distutils.command.build_ext tests for Py_ENABLE_SHARED on linux and solaris and automatically adds '.' to the library_dirs, and I suspect it just

Re: [Python-Dev] Improve open() to support reading file starting with an unicode BOM

2010-01-07 Thread Glyph Lefkowitz
On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner victor.stin...@haypocalc.com wrote: Hi, Builtin open() function is unable to open an UTF-16/32 file starting with a BOM if the encoding is not specified (raise an unicode error). For an

Re: [Python-Dev] --enabled-shared broken on freebsd5?

2010-01-07 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Nicholas Bastin wrote: I think this problem probably needs to move over to distutils-sig, as it doesn't seem to be specific to the way that Python itself uses distutils. distutils.command.build_ext tests for Py_ENABLE_SHARED on linux and solaris

Re: [Python-Dev] Improve open() to support reading file starting with an unicode BOM

2010-01-07 Thread Guido van Rossum
On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz gl...@twistedmatrix.com wrote: On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner victor.stin...@haypocalc.com wrote: Hi, Builtin open() function is unable to open an UTF-16/32 file starting

Re: [Python-Dev] Improve open() to support reading file starting with an unicode BOM

2010-01-07 Thread Stephen J. Turnbull
Guido van Rossum writes: I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy talk. That doesn't stop many applications from doing it. Python should perhapswink,nudge not produce UTF-8 + BOM without a disclaimer of indemnification against all resulting damage, signed in

Re: [Python-Dev] Improve open() to support reading file starting with an unicode BOM

2010-01-07 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Guido van Rossum wrote: On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz gl...@twistedmatrix.com wrote: On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: On Thu, Jan 7, 2010 at 4:10 PM, Victor Stinner victor.stin...@haypocalc.com wrote: Hi,

Re: [Python-Dev] Improve open() to support reading file starting with an unicode BOM

2010-01-07 Thread Glyph Lefkowitz
On Jan 7, 2010, at 11:21 PM, Guido van Rossum wrote: On Thu, Jan 7, 2010 at 7:34 PM, Glyph Lefkowitz gl...@twistedmatrix.com wrote: On Jan 7, 2010, at 7:52 PM, Guido van Rossum wrote: I'm a little hesitant about this. First of all, UTF-8 + BOM is crazy talk. And for the other two,