Re: [Python-Dev] Software Transactional Memory for Python

2011-08-30 Thread Armin Rigo
Hi, On Tue, Aug 30, 2011 at 11:33 PM, Yury Selivanov wrote: > Maybe it'd be better to put 'atomic' in the threading module? 'threading' is pure Python. But anyway the consensus is to not have 'atomic' at all in the stdlib, which means it is in its own 3rd-party extension module. Armin ___

Re: [Python-Dev] PEP 393 Summer of Code Project

2011-08-30 Thread Stephen J. Turnbull
Guido van Rossum writes: > On Tue, Aug 30, 2011 at 7:55 PM, Stephen J. Turnbull > wrote: > > For starters, one that doesn't ever return lone surrogates, but rather > > interprets surrogate pairs as Unicode code points as in UTF-16.  (This > > is not a Unicode standard definition, it's inten

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Cesare Di Mauro
2011/8/31 Terry Reedy > I find myself more comfortable with the Cesare Di Mauro's idea of expanding > to 16 bits as the code unit. His basic idea was using 2, 4, or 6 bytes > instead of 1, 3, or 6. > It can be expanded to longer than 6 bytes opcodes, if needed. The format is designed to be flexi

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Cesare Di Mauro
2011/8/30 stefan brunthaler > > Do I sense that the bytecode format is no longer platform-independent? > > That will need a bit of discussion. I bet there are some things around > > that depend on that. > > > Hm, I haven't really thought about that in detail and for longer, I > ran it on PowerPC

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Cesare Di Mauro
2011/8/30 stefan brunthaler > Yes, indeed I have a more straightforward instruction format to allow > for more efficient decoding. Just going from bytecode size to > word-code size without changing the instruction format is going to > require 8 (or word-size) times more memory on a 64bit system.

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Cesare Di Mauro
2011/8/30 Nick Coghlan > > Yeah, it's definitely a trade-off - the point I was trying to make is > that there *is* a trade-off being made between complexity and speed. > > I think the computed-gotos stuff struck a nice balance - the macro-fu > involved means that you can still understand what the

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Cesare Di Mauro
2011/8/30 Antoine Pitrou > Changing the bytecode width wouldn't make the interpreter more complex. It depends on the kind of changes. :) WPython introduced a very different "intermediate code" representation that required a big change on the peepholer optimizer on 1.0 alpha version. On 1.1 fi

Re: [Python-Dev] PEP 393 Summer of Code Project

2011-08-30 Thread Guido van Rossum
On Tue, Aug 30, 2011 at 7:55 PM, Stephen J. Turnbull wrote: > Antoine Pitrou writes: > >  > Sorry, what is a conformant UTF-16 array op? > > For starters, one that doesn't ever return lone surrogates, but rather > interprets surrogate pairs as Unicode code points as in UTF-16.  (This > is not a Un

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Terry Reedy
On 8/30/2011 2:12 PM, Guido van Rossum wrote: On Tue, Aug 30, 2011 at 10:50 AM, stefan brunthaler wrote: Do you really need it to match a machine word? Or is, say, a 16-bit format sufficient. Hm, technically no, but practically it makes more sense, as (at least for x86 architectures) having

Re: [Python-Dev] PEP 393 Summer of Code Project

2011-08-30 Thread Stephen J. Turnbull
Antoine Pitrou writes: > Sorry, what is a conformant UTF-16 array op? For starters, one that doesn't ever return lone surrogates, but rather interprets surrogate pairs as Unicode code points as in UTF-16. (This is not a Unicode standard definition, it's intended to be suggestive of why many app

Re: [Python-Dev] Coding guidelines for os.walk filter

2011-08-30 Thread Michael Urman
> for t in os.walk(somedir): >    t[1][:]=set(t[1])-{'.svn','tmp'} >    ... do something > > This is a very clever hack but... it relies on internal implementation > of os.walk This doesn't appear to be an internal implementation detail; this is documented behavior. http://docs.python.org/dev/

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Nick Coghlan
On Wed, Aug 31, 2011 at 9:21 AM, Jesse Noller wrote: > Discussion of speed.python.org should happen on the mailing list for that > project if possible. Hah, that's how out of the loop I am on that front - I didn't even know there *was* a mailing list for it :) Subscribed! Cheers, Nick. P.S. F

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Nick Coghlan
On Wed, Aug 31, 2011 at 3:23 AM, stefan brunthaler wrote: > On Tue, Aug 30, 2011 at 09:42, Guido van Rossum wrote: >> Stefan, have you shared a pointer to your code yet? Is it open source? >> > I have no shared code repository, but could create one (is there any > pydev preferred provider?). I ha

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Jesse Noller
On Aug 30, 2011, at 9:05 AM, Nick Coghlan wrote: > On Tue, Aug 30, 2011 at 9:38 PM, Antoine Pitrou wrote: >> On Tue, 30 Aug 2011 13:29:59 +1000 >> Nick Coghlan wrote: >>> >>> Anecdotal, non-reproducible performance figures are *not* the way to >>> go about serious optimisation efforts. >> >

[Python-Dev] Coding guidelines for os.walk filter

2011-08-30 Thread Jacek Pliszka
Hi! I would like to get some opinion on possible os.walk improvement. For the sake of simplicity let's assume I would like to skip all .svn and tmp directories. Current solution looks like this: for t in os.walk(somedir): t[1][:]=set(t[1])-{'.svn','tmp'} ... do something This is a very

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Gregory P. Smith
On Tue, Aug 30, 2011 at 1:54 PM, Benjamin Peterson wrote: > 2011/8/30 stefan brunthaler : > > On Tue, Aug 30, 2011 at 13:42, Benjamin Peterson > wrote: > >> 2011/8/30 stefan brunthaler : > >>> I will remove my development commentaries and create a private > >>> repository at bitbucket for you* to

Re: [Python-Dev] Software Transactional Memory for Python

2011-08-30 Thread Yury Selivanov
Maybe it'd be better to put 'atomic' in the threading module? On 2011-08-30, at 4:02 PM, Armin Rigo wrote: > Re-hi, > > 2011/8/29 Armin Rigo : >>> The problem is that many locks are actually acquired implicitely. >>> For example, `print` to a buffered stream will acquire the fileobject's >>> mu

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Benjamin Peterson
2011/8/30 stefan brunthaler : > On Tue, Aug 30, 2011 at 13:42, Benjamin Peterson wrote: >> 2011/8/30 stefan brunthaler : >>> I will remove my development commentaries and create a private >>> repository at bitbucket for you* to take an early look like Georg (and >>> more or less Terry, too) sugges

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread stefan brunthaler
On Tue, Aug 30, 2011 at 13:42, Benjamin Peterson wrote: > 2011/8/30 stefan brunthaler : >> I will remove my development commentaries and create a private >> repository at bitbucket for you* to take an early look like Georg (and >> more or less Terry, too) suggested. Is that a good way for most of

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Benjamin Peterson
2011/8/30 stefan brunthaler : > I will remove my development commentaries and create a private > repository at bitbucket for you* to take an early look like Georg (and > more or less Terry, too) suggested. Is that a good way for most of > you? (I would then give access to whomever wants to take a l

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread stefan brunthaler
> Ok, there there's something else you haven't told us. Are you saying > that the original (old) bytecode is still used (and hence written to > and read from .pyc files)? > Short answer: yes. Long answer: I added an invocation counter to the code object and keep interpreting in the usual Python int

Re: [Python-Dev] Software Transactional Memory for Python

2011-08-30 Thread Armin Rigo
Re-hi, 2011/8/29 Armin Rigo : >> The problem is that many locks are actually acquired implicitely. >> For example, `print` to a buffered stream will acquire the fileobject's >> mutex. > > Indeed. > (...) > I suspect that I need to do a more thorough review of the stdlib (...) I found a solution

Re: [Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

2011-08-30 Thread Stefan Behnel
Guido van Rossum, 30.08.2011 19:05: On Tue, Aug 30, 2011 at 9:49 AM, "Martin v. Löwis" wrote: I can understand how that works when building a CPython extension. But what about creating Jython/IronPython modules with Cython? At what point get the header files considered there? I had written a b

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Georg Brandl
Am 30.08.2011 20:34, schrieb stefan brunthaler: >> Um, I'm sorry, but that reply sounds incredibly naive, like you're not >> really sure what the on-disk format for .pyc files is or why it would >> matter. You're not even answering the question, except indirectly -- >> it seems that you've never ev

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Guido van Rossum
On Tue, Aug 30, 2011 at 11:34 AM, stefan brunthaler wrote: >> Um, I'm sorry, but that reply sounds incredibly naive, like you're not >> really sure what the on-disk format for .pyc files is or why it would >> matter. You're not even answering the question, except indirectly -- >> it seems that you

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Terry Reedy
On 8/30/2011 1:23 PM, stefan brunthaler wrote: (I just wanted to know if there is substantial interest so that > it eventually pays off to find and fix the remaining bugs) It is the nature of our development process that there usually can be no guarantee of acceptance of future code. The rath

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread stefan brunthaler
> Um, I'm sorry, but that reply sounds incredibly naive, like you're not > really sure what the on-disk format for .pyc files is or why it would > matter. You're not even answering the question, except indirectly -- > it seems that you've never even thought about the possibility of > generating a .

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Guido van Rossum
On Tue, Aug 30, 2011 at 11:23 AM, stefan brunthaler wrote: >> Do I sense that the bytecode format is no longer platform-independent? >> That will need a bit of discussion. I bet there are some things around >> that depend on that. >> > Hm, I haven't really thought about that in detail and for long

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread stefan brunthaler
> Do I sense that the bytecode format is no longer platform-independent? > That will need a bit of discussion. I bet there are some things around > that depend on that. > Hm, I haven't really thought about that in detail and for longer, I ran it on PowerPC 970 and Intel Atom & i7 without problems (

Re: [Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

2011-08-30 Thread Terry Reedy
On 8/30/2011 1:05 PM, Guido van Rossum wrote: I see. So there is potential for error there. To elaborate, with CPython it looks pretty solid, at least for functions and constants (does it do structs?). You must manually declare the name and signature of a function, and Pyrex/Cython emits C cod

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Guido van Rossum
On Tue, Aug 30, 2011 at 10:50 AM, stefan brunthaler wrote: >> Do you really need it to match a machine word? Or is, say, a 16-bit >> format sufficient. >> > Hm, technically no, but practically it makes more sense, as (at least > for x86 architectures) having opargs and opcodes in half-words can be

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread stefan brunthaler
> Do you really need it to match a machine word? Or is, say, a 16-bit > format sufficient. > Hm, technically no, but practically it makes more sense, as (at least for x86 architectures) having opargs and opcodes in half-words can be efficiently expressed in assembly. On 64bit architectures, I could

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Antoine Pitrou
On Tue, 30 Aug 2011 08:27:13 -0700 stefan brunthaler wrote: > >> Changing the bytecode width wouldn't make the interpreter more complex. > > > > No, but I think Stefan is proposing to add a *second* byte code format, > > in addition to the one that remains there. That would certainly be an > > inc

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread stefan brunthaler
On Tue, Aug 30, 2011 at 09:42, Guido van Rossum wrote: > Stefan, have you shared a pointer to your code yet? Is it open source? > I have no shared code repository, but could create one (is there any pydev preferred provider?). I have all the copyrights on the code, and I would like to open-source

Re: [Python-Dev] PEP 393 Summer of Code Project

2011-08-30 Thread Antoine Pitrou
> The problem with a narrow build (whether for space efficiency in > CPython or for platform compatibility in Jython and IronPython) is not > that we have no UTF-16 codecs. It's that array ops aren't UTF-16 > conformant. Sorry, what is a conformant UTF-16 array op? Thanks Antoine. __

Re: [Python-Dev] PyPI went down

2011-08-30 Thread Martin v. Löwis
> Looks like the issue keeps popping up. It was slow to respond earlier > today, and I keep getting complaints about it (including now.) Somebody is mirroring the site with wget. I have null-routed them. Regards, Martin ___ Python-Dev mailing list Pytho

Re: [Python-Dev] PEP 393 Summer of Code Project

2011-08-30 Thread Stephen J. Turnbull
Antoine Pitrou writes: > On Mon, 29 Aug 2011 12:43:24 +0900 > "Stephen J. Turnbull" wrote: > > > > Since when can s[0] represent a code point outside the BMP, for s a > > Unicode string in a narrow build? > > > > Remember, the UCS-2/narrow vs. UCS-4/wide distinction is *not* about > > wh

Re: [Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

2011-08-30 Thread Guido van Rossum
On Tue, Aug 30, 2011 at 9:49 AM, "Martin v. Löwis" wrote: >>> I can understand how that works when building a CPython extension. >>> But what about creating Jython/IronPython modules with Cython? >>> At what point get the header files considered there? >> >> I had written a bit about this here: >>

Re: [Python-Dev] PyPI went down

2011-08-30 Thread Thomas Wouters
On Tue, Aug 30, 2011 at 18:46, "Martin v. Löwis" wrote: > >I released the first package of two and PyPI went down while I was > > preparing to release the second. I hope it wasn't me? > > A few minutes ago, it was responding very slowly, and I found out that > Postgres consumes all time. I ha

Re: [Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

2011-08-30 Thread Martin v. Löwis
>> I can understand how that works when building a CPython extension. >> But what about creating Jython/IronPython modules with Cython? >> At what point get the header files considered there? > > I had written a bit about this here: > > http://thread.gmane.org/gmane.comp.python.devel/126340/focus

Re: [Python-Dev] PyPI went down

2011-08-30 Thread Martin v. Löwis
>I released the first package of two and PyPI went down while I was > preparing to release the second. I hope it wasn't me? A few minutes ago, it was responding very slowly, and I found out that Postgres consumes all time. I haven't put energy into investigating what was causing this - apparen

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Guido van Rossum
Stefan, have you shared a pointer to your code yet? Is it open source? It sounds like people are definitely interested and it would make sense to let them experiment with your code and review it. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev

[Python-Dev] PyPI went down

2011-08-30 Thread Oleg Broytman
Hello! I released the first package of two and PyPI went down while I was preparing to release the second. I hope it wasn't me? Oleg. -- Oleg Broytmanhttp://phdru.name/p...@phdru.name Programmers don't die, they just GOSUB without RETURN. __

Re: [Python-Dev] cpython: Remove display options (--name, etc.) from the Distribution class.

2011-08-30 Thread Éric Araujo
Hi, Le 30/08/2011 17:20, Antoine Pitrou a écrit : > On Tue, 30 Aug 2011 16:22:14 +0200 > eric.araujo wrote: >> As a side effect, the Distribution class no longer accepts a 'url' key >> in its *attrs* argument: it has to be 'home-page' to be recognized as a >> valid metadata field and passed down

Re: [Python-Dev] PyPI went down

2011-08-30 Thread Oleg Broytman
It is back up. I am very sorry for the fuss. Oleg. -- Oleg Broytmanhttp://phdru.name/p...@phdru.name Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org http://

Re: [Python-Dev] PyPI went down

2011-08-30 Thread Oleg Broytman
On Tue, Aug 30, 2011 at 07:30:01PM +0400, Oleg Broytman wrote: >PyPI went down More information: ports 80 and 443 are open, the servers performs SSL handshake but timeouts on HTTP requests (with or without SSL). Oleg. -- Oleg Broytmanhttp://phdru.name/p...@phd

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread stefan brunthaler
>> Changing the bytecode width wouldn't make the interpreter more complex. > > No, but I think Stefan is proposing to add a *second* byte code format, > in addition to the one that remains there. That would certainly be an > increase in complexity. > Yes, indeed I have a more straightforward instru

Re: [Python-Dev] cpython: Remove display options (--name, etc.) from the Distribution class.

2011-08-30 Thread Antoine Pitrou
On Tue, 30 Aug 2011 16:22:14 +0200 eric.araujo wrote: > http://hg.python.org/cpython/rev/af0bcccb935b > changeset: 72127:af0bcccb935b > user:Éric Araujo > date:Tue Aug 30 00:55:02 2011 +0200 > summary: > Remove display options (--name, etc.) from the Distribution class. > >

Re: [Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

2011-08-30 Thread Vlad Riscutia
I also have some patches sitting on the tracker for some time: http://bugs.python.org/issue12764 http://bugs.python.org/issue11835 http://bugs.python.org/issue12528 which also fixes http://bugs.python.org/issue6069 and http://bugs.python.org/issue11920 http://bugs.python.org/issue6068 which also f

Re: [Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

2011-08-30 Thread Vinay Sajip
Meador Inge gmail.com> writes: > 1. http://bugs.python.org/issue9041 I raised a question about this patch (in the issue tracker). > 2. http://bugs.python.org/issue9651 > 3. http://bugs.python.org/issue11241 I presume, since Amaury has commit rights, that he could commit these. Regards, Vin

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Nick Coghlan
On Tue, Aug 30, 2011 at 9:38 PM, Antoine Pitrou wrote: > On Tue, 30 Aug 2011 13:29:59 +1000 > Nick Coghlan wrote: >> >> Anecdotal, non-reproducible performance figures are *not* the way to >> go about serious optimisation efforts. > > What about anecdotal *and* reproducible performance figures? :

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Antoine Pitrou
On Tue, 30 Aug 2011 13:29:59 +1000 Nick Coghlan wrote: > > Anecdotal, non-reproducible performance figures are *not* the way to > go about serious optimisation efforts. What about anecdotal *and* reproducible performance figures? :) I may be half-joking, but we already have a set of py3k-compati

Re: [Python-Dev] PEP 393 review

2011-08-30 Thread Antoine Pitrou
By the way, I don't know if you're working on it, but StringIO seems a bit broken right now. test_memoryio crashes here: test_newline_cr (test.test_memoryio.CStringIOTest) ... Fatal Python error: Segmentation fault Current thread 0x7f3f6353b700: File "/home/antoine/cpython/pep-393/Lib/tes

Re: [Python-Dev] Planned PEP status changes

2011-08-30 Thread Nick Coghlan
On Sat, Aug 27, 2011 at 2:35 AM, Brett Cannon wrote: > On Tue, Aug 23, 2011 at 19:42, Nick Coghlan wrote: >> Unless I hear any objections, I plan to adjust the current PEP >> statuses as follows some time this weekend: >> >> Move from Accepted to Finished: >> >>    389  argparse - New Command Lin

Re: [Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

2011-08-30 Thread Stefan Behnel
"Martin v. Löwis", 30.08.2011 10:46: You might be reading more into that statement than I meant. You have to supply Pyrex/Cython versions of the C declarations, either hand-written or generated by a tool. But you write them based on the advertised C API -- you don't have to manually expand macros

Re: [Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

2011-08-30 Thread Martin v. Löwis
> You might be reading more into that statement than I meant. > You have to supply Pyrex/Cython versions of the C declarations, > either hand-written or generated by a tool. But you write them > based on the advertised C API -- you don't have to manually > expand macros, work out the low-level layo

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Martin v. Löwis
>> Although any such patch should discuss how it compares with Cesare's >> work on wpython. >> Personally, I *like* CPython fitting into the "simple-and-portable" >> niche in the Python interpreter space. > > Changing the bytecode width wouldn't make the interpreter more complex. No, but I think

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Mark Shannon
Martin v. Löwis wrote: So, the two big issues aside, is there any interest in incorporating these optimizations in Python 3? The question really is whether this is an all-or-nothing deal. If you could identify smaller parts that can be applied independently, interest would be higher. Also, I'd

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Mark Shannon
Nick Coghlan wrote: On Tue, Aug 30, 2011 at 7:14 AM, Antoine Pitrou wrote: On Mon, 29 Aug 2011 11:33:14 -0700 stefan brunthaler wrote: * The optimized dispatch routine has a changed instruction format (word-sized instead of bytecodes) that allows for regular instruction decoding (without the

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Stefan Behnel
Nick Coghlan, 30.08.2011 02:00: On Tue, Aug 30, 2011 at 7:14 AM, Antoine Pitrou wrote: On Mon, 29 Aug 2011 11:33:14 -0700 stefan brunthaler wrote: * The optimized dispatch routine has a changed instruction format (word-sized instead of bytecodes) that allows for regular instruction decoding (wi

Re: [Python-Dev] PEP 393 review

2011-08-30 Thread Martin v. Löwis
> This looks very nice. Is 3.3 a wide build? (how about a narrow build?) It's a wide build. For reference, I also attach 64-bit narrow build results, and 32-bit results (wide, narrow, and PEP 393). Savings are much smaller in narrow builds (larger on 32-bit systems than on 64-bit systems). > (is

Re: [Python-Dev] Python 3 optimizations continued...

2011-08-30 Thread Nick Coghlan
On Tue, Aug 30, 2011 at 4:22 PM, Eli Bendersky wrote: > On Tue, Aug 30, 2011 at 08:57, Greg Ewing > wrote: > Following this argument to the extreme, the bytecode evaluation code of > CPython can be simplified quite a bit. Lose 2x performance but gain a lot of > readability. Does that sound like a