Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Greg Ewing
Nick Coghlan wrote: Personally, I *like* CPython fitting into the simple-and-portable niche in the Python interpreter space. Me, too! I like that I can read the CPython source and understand what it's doing most of the time. Please don't screw that up by attempting to perform heroic

Re: [Python-Dev] PEP 393 review

2011-08-30 Thread Martin v. Löwis
I don't compare ASCII and ISO-8859-1 decoders. I was asking if decoding b'abc' from ISO-8859-1 is faster than decoding b'ab\xff' from ISO-8859-1, and if yes: why? No, that makes no difference. Your patch replaces PyUnicode_New(size, 255) ... memcpy(), by PyUnicode_FromUCS1(). You

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Eli Bendersky
On Tue, Aug 30, 2011 at 08:57, Greg Ewing greg.ew...@canterbury.ac.nzwrote: Nick Coghlan wrote: Personally, I *like* CPython fitting into the simple-and-portable niche in the Python interpreter space. Me, too! I like that I can read the CPython source and understand what it's doing most

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Nick Coghlan
On Tue, Aug 30, 2011 at 4:22 PM, Eli Bendersky eli...@gmail.com wrote: On Tue, Aug 30, 2011 at 08:57, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Following this argument to the extreme, the bytecode evaluation code of CPython can be simplified quite a bit. Lose 2x performance but gain a lot

Re: [Python-Dev] PEP 393 review

2011-08-30 Thread Martin v. Löwis
This looks very nice. Is 3.3 a wide build? (how about a narrow build?) It's a wide build. For reference, I also attach 64-bit narrow build results, and 32-bit results (wide, narrow, and PEP 393). Savings are much smaller in narrow builds (larger on 32-bit systems than on 64-bit systems). (is

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Stefan Behnel
Nick Coghlan, 30.08.2011 02:00: On Tue, Aug 30, 2011 at 7:14 AM, Antoine Pitrou wrote: On Mon, 29 Aug 2011 11:33:14 -0700 stefan brunthaler wrote: * The optimized dispatch routine has a changed instruction format (word-sized instead of bytecodes) that allows for regular instruction decoding

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Mark Shannon
Nick Coghlan wrote: On Tue, Aug 30, 2011 at 7:14 AM, Antoine Pitrou solip...@pitrou.net wrote: On Mon, 29 Aug 2011 11:33:14 -0700 stefan brunthaler s.bruntha...@uci.edu wrote: * The optimized dispatch routine has a changed instruction format (word-sized instead of bytecodes) that allows for

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Mark Shannon
Martin v. Löwis wrote: So, the two big issues aside, is there any interest in incorporating these optimizations in Python 3? The question really is whether this is an all-or-nothing deal. If you could identify smaller parts that can be applied independently, interest would be higher. Also,

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Martin v. Löwis
Although any such patch should discuss how it compares with Cesare's work on wpython. Personally, I *like* CPython fitting into the simple-and-portable niche in the Python interpreter space. Changing the bytecode width wouldn't make the interpreter more complex. No, but I think Stefan is

Re: [Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

2011-08-30 Thread Martin v. Löwis
You might be reading more into that statement than I meant. You have to supply Pyrex/Cython versions of the C declarations, either hand-written or generated by a tool. But you write them based on the advertised C API -- you don't have to manually expand macros, work out the low-level layout

Re: [Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

2011-08-30 Thread Stefan Behnel
Martin v. Löwis, 30.08.2011 10:46: You might be reading more into that statement than I meant. You have to supply Pyrex/Cython versions of the C declarations, either hand-written or generated by a tool. But you write them based on the advertised C API -- you don't have to manually expand macros,

Re: [Python-Dev] Planned PEP status changes

2011-08-30 Thread Nick Coghlan
On Sat, Aug 27, 2011 at 2:35 AM, Brett Cannon br...@python.org wrote: On Tue, Aug 23, 2011 at 19:42, Nick Coghlan ncogh...@gmail.com wrote: Unless I hear any objections, I plan to adjust the current PEP statuses as follows some time this weekend: Move from Accepted to Finished:    389  

Re: [Python-Dev] PEP 393 review

2011-08-30 Thread Antoine Pitrou
By the way, I don't know if you're working on it, but StringIO seems a bit broken right now. test_memoryio crashes here: test_newline_cr (test.test_memoryio.CStringIOTest) ... Fatal Python error: Segmentation fault Current thread 0x7f3f6353b700: File

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Antoine Pitrou
On Tue, 30 Aug 2011 13:29:59 +1000 Nick Coghlan ncogh...@gmail.com wrote: Anecdotal, non-reproducible performance figures are *not* the way to go about serious optimisation efforts. What about anecdotal *and* reproducible performance figures? :) I may be half-joking, but we already have a set

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Nick Coghlan
On Tue, Aug 30, 2011 at 9:38 PM, Antoine Pitrou solip...@pitrou.net wrote: On Tue, 30 Aug 2011 13:29:59 +1000 Nick Coghlan ncogh...@gmail.com wrote: Anecdotal, non-reproducible performance figures are *not* the way to go about serious optimisation efforts. What about anecdotal *and*

Re: [Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

2011-08-30 Thread Vinay Sajip
Meador Inge meadori at gmail.com writes: 1. http://bugs.python.org/issue9041 I raised a question about this patch (in the issue tracker). 2. http://bugs.python.org/issue9651 3. http://bugs.python.org/issue11241 I presume, since Amaury has commit rights, that he could commit these.

Re: [Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

2011-08-30 Thread Vlad Riscutia
I also have some patches sitting on the tracker for some time: http://bugs.python.org/issue12764 http://bugs.python.org/issue11835 http://bugs.python.org/issue12528 which also fixes http://bugs.python.org/issue6069 and http://bugs.python.org/issue11920 http://bugs.python.org/issue6068 which also

Re: [Python-Dev] cpython: Remove display options (--name, etc.) from the Distribution class.

2011-08-30 Thread Antoine Pitrou
On Tue, 30 Aug 2011 16:22:14 +0200 eric.araujo python-check...@python.org wrote: http://hg.python.org/cpython/rev/af0bcccb935b changeset: 72127:af0bcccb935b user:Éric Araujo mer...@netwok.org date:Tue Aug 30 00:55:02 2011 +0200 summary: Remove display options (--name,

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread stefan brunthaler
Changing the bytecode width wouldn't make the interpreter more complex. No, but I think Stefan is proposing to add a *second* byte code format, in addition to the one that remains there. That would certainly be an increase in complexity. Yes, indeed I have a more straightforward instruction

Re: [Python-Dev] PyPI went down

2011-08-30 Thread Oleg Broytman
On Tue, Aug 30, 2011 at 07:30:01PM +0400, Oleg Broytman wrote: PyPI went down More information: ports 80 and 443 are open, the servers performs SSL handshake but timeouts on HTTP requests (with or without SSL). Oleg. -- Oleg Broytmanhttp://phdru.name/

Re: [Python-Dev] PyPI went down

2011-08-30 Thread Oleg Broytman
It is back up. I am very sorry for the fuss. Oleg. -- Oleg Broytmanhttp://phdru.name/p...@phdru.name Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] cpython: Remove display options (--name, etc.) from the Distribution class.

2011-08-30 Thread Éric Araujo
Hi, Le 30/08/2011 17:20, Antoine Pitrou a écrit : On Tue, 30 Aug 2011 16:22:14 +0200 eric.araujo python-check...@python.org wrote: As a side effect, the Distribution class no longer accepts a 'url' key in its *attrs* argument: it has to be 'home-page' to be recognized as a valid metadata

[Python-Dev] PyPI went down

2011-08-30 Thread Oleg Broytman
Hello! I released the first package of two and PyPI went down while I was preparing to release the second. I hope it wasn't me? Oleg. -- Oleg Broytmanhttp://phdru.name/p...@phdru.name Programmers don't die, they just GOSUB without RETURN.

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Guido van Rossum
Stefan, have you shared a pointer to your code yet? Is it open source? It sounds like people are definitely interested and it would make sense to let them experiment with your code and review it. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev

Re: [Python-Dev] PyPI went down

2011-08-30 Thread Martin v. Löwis
I released the first package of two and PyPI went down while I was preparing to release the second. I hope it wasn't me? A few minutes ago, it was responding very slowly, and I found out that Postgres consumes all time. I haven't put energy into investigating what was causing this -

Re: [Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

2011-08-30 Thread Martin v. Löwis
I can understand how that works when building a CPython extension. But what about creating Jython/IronPython modules with Cython? At what point get the header files considered there? I had written a bit about this here: http://thread.gmane.org/gmane.comp.python.devel/126340/focus=126419

Re: [Python-Dev] PyPI went down

2011-08-30 Thread Thomas Wouters
On Tue, Aug 30, 2011 at 18:46, Martin v. Löwis mar...@v.loewis.de wrote: I released the first package of two and PyPI went down while I was preparing to release the second. I hope it wasn't me? A few minutes ago, it was responding very slowly, and I found out that Postgres consumes all

Re: [Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

2011-08-30 Thread Guido van Rossum
On Tue, Aug 30, 2011 at 9:49 AM, Martin v. Löwis mar...@v.loewis.de wrote: I can understand how that works when building a CPython extension. But what about creating Jython/IronPython modules with Cython? At what point get the header files considered there? I had written a bit about this

Re: [Python-Dev] PEP 393 Summer of Code Project

2011-08-30 Thread Stephen J. Turnbull
Antoine Pitrou writes: On Mon, 29 Aug 2011 12:43:24 +0900 Stephen J. Turnbull step...@xemacs.org wrote: Since when can s[0] represent a code point outside the BMP, for s a Unicode string in a narrow build? Remember, the UCS-2/narrow vs. UCS-4/wide distinction is *not* about

Re: [Python-Dev] PyPI went down

2011-08-30 Thread Martin v. Löwis
Looks like the issue keeps popping up. It was slow to respond earlier today, and I keep getting complaints about it (including now.) Somebody is mirroring the site with wget. I have null-routed them. Regards, Martin ___ Python-Dev mailing list

Re: [Python-Dev] PEP 393 Summer of Code Project

2011-08-30 Thread Antoine Pitrou
The problem with a narrow build (whether for space efficiency in CPython or for platform compatibility in Jython and IronPython) is not that we have no UTF-16 codecs. It's that array ops aren't UTF-16 conformant. Sorry, what is a conformant UTF-16 array op? Thanks Antoine.

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread stefan brunthaler
On Tue, Aug 30, 2011 at 09:42, Guido van Rossum gu...@python.org wrote: Stefan, have you shared a pointer to your code yet? Is it open source? I have no shared code repository, but could create one (is there any pydev preferred provider?). I have all the copyrights on the code, and I would like

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Antoine Pitrou
On Tue, 30 Aug 2011 08:27:13 -0700 stefan brunthaler ste...@brunthaler.net wrote: Changing the bytecode width wouldn't make the interpreter more complex. No, but I think Stefan is proposing to add a *second* byte code format, in addition to the one that remains there. That would certainly

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread stefan brunthaler
Do you really need it to match a machine word? Or is, say, a 16-bit format sufficient. Hm, technically no, but practically it makes more sense, as (at least for x86 architectures) having opargs and opcodes in half-words can be efficiently expressed in assembly. On 64bit architectures, I could

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Guido van Rossum
On Tue, Aug 30, 2011 at 10:50 AM, stefan brunthaler ste...@brunthaler.net wrote: Do you really need it to match a machine word? Or is, say, a 16-bit format sufficient. Hm, technically no, but practically it makes more sense, as (at least for x86 architectures) having opargs and opcodes in

Re: [Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

2011-08-30 Thread Terry Reedy
On 8/30/2011 1:05 PM, Guido van Rossum wrote: I see. So there is potential for error there. To elaborate, with CPython it looks pretty solid, at least for functions and constants (does it do structs?). You must manually declare the name and signature of a function, and Pyrex/Cython emits C

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread stefan brunthaler
Do I sense that the bytecode format is no longer platform-independent? That will need a bit of discussion. I bet there are some things around that depend on that. Hm, I haven't really thought about that in detail and for longer, I ran it on PowerPC 970 and Intel Atom i7 without problems (the

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Guido van Rossum
On Tue, Aug 30, 2011 at 11:23 AM, stefan brunthaler ste...@brunthaler.net wrote: Do I sense that the bytecode format is no longer platform-independent? That will need a bit of discussion. I bet there are some things around that depend on that. Hm, I haven't really thought about that in detail

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread stefan brunthaler
Um, I'm sorry, but that reply sounds incredibly naive, like you're not really sure what the on-disk format for .pyc files is or why it would matter. You're not even answering the question, except indirectly -- it seems that you've never even thought about the possibility of generating a .pyc

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Terry Reedy
On 8/30/2011 1:23 PM, stefan brunthaler wrote: (I just wanted to know if there is substantial interest so that it eventually pays off to find and fix the remaining bugs) It is the nature of our development process that there usually can be no guarantee of acceptance of future code. The

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Guido van Rossum
On Tue, Aug 30, 2011 at 11:34 AM, stefan brunthaler ste...@brunthaler.net wrote: Um, I'm sorry, but that reply sounds incredibly naive, like you're not really sure what the on-disk format for .pyc files is or why it would matter. You're not even answering the question, except indirectly -- it

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Georg Brandl
Am 30.08.2011 20:34, schrieb stefan brunthaler: Um, I'm sorry, but that reply sounds incredibly naive, like you're not really sure what the on-disk format for .pyc files is or why it would matter. You're not even answering the question, except indirectly -- it seems that you've never even

Re: [Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

2011-08-30 Thread Stefan Behnel
Guido van Rossum, 30.08.2011 19:05: On Tue, Aug 30, 2011 at 9:49 AM, Martin v. Löwis wrote: I can understand how that works when building a CPython extension. But what about creating Jython/IronPython modules with Cython? At what point get the header files considered there? I had written a

Re: [Python-Dev] Software Transactional Memory for Python

2011-08-30 Thread Armin Rigo
Re-hi, 2011/8/29 Armin Rigo ar...@tunes.org: The problem is that many locks are actually acquired implicitely. For example, `print` to a buffered stream will acquire the fileobject's mutex. Indeed. (...) I suspect that I need to do a more thorough review of the stdlib (...) I found a

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread stefan brunthaler
Ok, there there's something else you haven't told us. Are you saying that the original (old) bytecode is still used (and hence written to and read from .pyc files)? Short answer: yes. Long answer: I added an invocation counter to the code object and keep interpreting in the usual Python

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Benjamin Peterson
2011/8/30 stefan brunthaler ste...@brunthaler.net: I will remove my development commentaries and create a private repository at bitbucket for you* to take an early look like Georg (and more or less Terry, too) suggested. Is that a good way for most of you? (I would then give access to whomever

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread stefan brunthaler
On Tue, Aug 30, 2011 at 13:42, Benjamin Peterson benja...@python.org wrote: 2011/8/30 stefan brunthaler ste...@brunthaler.net: I will remove my development commentaries and create a private repository at bitbucket for you* to take an early look like Georg (and more or less Terry, too)

Re: [Python-Dev] Software Transactional Memory for Python

2011-08-30 Thread Yury Selivanov
Maybe it'd be better to put 'atomic' in the threading module? On 2011-08-30, at 4:02 PM, Armin Rigo wrote: Re-hi, 2011/8/29 Armin Rigo ar...@tunes.org: The problem is that many locks are actually acquired implicitely. For example, `print` to a buffered stream will acquire the fileobject's

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Gregory P. Smith
On Tue, Aug 30, 2011 at 1:54 PM, Benjamin Peterson benja...@python.orgwrote: 2011/8/30 stefan brunthaler ste...@brunthaler.net: On Tue, Aug 30, 2011 at 13:42, Benjamin Peterson benja...@python.org wrote: 2011/8/30 stefan brunthaler ste...@brunthaler.net: I will remove my development

[Python-Dev] Coding guidelines for os.walk filter

2011-08-30 Thread Jacek Pliszka
Hi! I would like to get some opinion on possible os.walk improvement. For the sake of simplicity let's assume I would like to skip all .svn and tmp directories. Current solution looks like this: for t in os.walk(somedir): t[1][:]=set(t[1])-{'.svn','tmp'} ... do something This is a very

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Jesse Noller
On Aug 30, 2011, at 9:05 AM, Nick Coghlan ncogh...@gmail.com wrote: On Tue, Aug 30, 2011 at 9:38 PM, Antoine Pitrou solip...@pitrou.net wrote: On Tue, 30 Aug 2011 13:29:59 +1000 Nick Coghlan ncogh...@gmail.com wrote: Anecdotal, non-reproducible performance figures are *not* the way to go

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Nick Coghlan
On Wed, Aug 31, 2011 at 3:23 AM, stefan brunthaler ste...@brunthaler.net wrote: On Tue, Aug 30, 2011 at 09:42, Guido van Rossum gu...@python.org wrote: Stefan, have you shared a pointer to your code yet? Is it open source? I have no shared code repository, but could create one (is there any

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Nick Coghlan
On Wed, Aug 31, 2011 at 9:21 AM, Jesse Noller jnol...@gmail.com wrote: Discussion of speed.python.org should happen on the mailing list for that project if possible. Hah, that's how out of the loop I am on that front - I didn't even know there *was* a mailing list for it :) Subscribed!

Re: [Python-Dev] Coding guidelines for os.walk filter

2011-08-30 Thread Michael Urman
for t in os.walk(somedir):    t[1][:]=set(t[1])-{'.svn','tmp'}    ... do something This is a very clever hack but... it relies on internal implementation of os.walk This doesn't appear to be an internal implementation detail; this is documented behavior.

Re: [Python-Dev] PEP 393 Summer of Code Project

2011-08-30 Thread Stephen J. Turnbull
Antoine Pitrou writes: Sorry, what is a conformant UTF-16 array op? For starters, one that doesn't ever return lone surrogates, but rather interprets surrogate pairs as Unicode code points as in UTF-16. (This is not a Unicode standard definition, it's intended to be suggestive of why many app

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Terry Reedy
On 8/30/2011 2:12 PM, Guido van Rossum wrote: On Tue, Aug 30, 2011 at 10:50 AM, stefan brunthaler ste...@brunthaler.net wrote: Do you really need it to match a machine word? Or is, say, a 16-bit format sufficient. Hm, technically no, but practically it makes more sense, as (at least for x86

Re: [Python-Dev] PEP 393 Summer of Code Project

2011-08-30 Thread Guido van Rossum
On Tue, Aug 30, 2011 at 7:55 PM, Stephen J. Turnbull step...@xemacs.org wrote: Antoine Pitrou writes:   Sorry, what is a conformant UTF-16 array op? For starters, one that doesn't ever return lone surrogates, but rather interprets surrogate pairs as Unicode code points as in UTF-16.  (This

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Cesare Di Mauro
2011/8/30 Antoine Pitrou solip...@pitrou.net Changing the bytecode width wouldn't make the interpreter more complex. It depends on the kind of changes. :) WPython introduced a very different intermediate code representation that required a big change on the peepholer optimizer on 1.0 alpha

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Cesare Di Mauro
2011/8/30 Nick Coghlan ncogh...@gmail.com Yeah, it's definitely a trade-off - the point I was trying to make is that there *is* a trade-off being made between complexity and speed. I think the computed-gotos stuff struck a nice balance - the macro-fu involved means that you can still

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Cesare Di Mauro
2011/8/30 stefan brunthaler ste...@brunthaler.net Yes, indeed I have a more straightforward instruction format to allow for more efficient decoding. Just going from bytecode size to word-code size without changing the instruction format is going to require 8 (or word-size) times more memory

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Cesare Di Mauro
2011/8/30 stefan brunthaler ste...@brunthaler.net Do I sense that the bytecode format is no longer platform-independent? That will need a bit of discussion. I bet there are some things around that depend on that. Hm, I haven't really thought about that in detail and for longer, I ran it

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Cesare Di Mauro
2011/8/31 Terry Reedy tjre...@udel.edu I find myself more comfortable with the Cesare Di Mauro's idea of expanding to 16 bits as the code unit. His basic idea was using 2, 4, or 6 bytes instead of 1, 3, or 6. It can be expanded to longer than 6 bytes opcodes, if needed. The format is

Re: [Python-Dev] PEP 393 Summer of Code Project

2011-08-30 Thread Stephen J. Turnbull
Guido van Rossum writes: On Tue, Aug 30, 2011 at 7:55 PM, Stephen J. Turnbull step...@xemacs.org wrote: For starters, one that doesn't ever return lone surrogates, but rather interprets surrogate pairs as Unicode code points as in UTF-16.  (This is not a Unicode standard definition,