Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Nick Coghlan 
Brett Cannon wrote:
> I think the question the PEP is posing is "will python-dev accept the
> Unladen Swallow work to add an LLVM JIT into the py3k trunk?" My
> personal answer is "yes" as it's a nice speed improvement for code that
> runs more than once, all while being an optional enhancement for those
> that don't want it (whether it is memory, startup, or C++ dependency).
> 
> And as for the whole Hg branch thing, this is to merge all of this into
> the main branch and not relegate it to some side branch.

Count me as another +1 for a py3k-jit branch that gets merged to the
py3k main line of development once all the blocking bugs are fixed. I
see this as very similar to the fairly long lived python-ast compiler
branch that eventually landed on the trunk for 2.5.

It would also act as a rather nice carrot for any developers asking
"What's in it for me?" in migrating from 2.x to 3.x :)

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
---
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Nick Coghlan 
Barry Warsaw wrote:
> On Jan 21, 2010, at 04:07 PM, Jeffrey Yasskin wrote:
> 
>> I could imagine a problem if Python+LLVM link in one libstdc++, and an
>> extension module links in a different one, even if no C++ objects are
>> passed across the boundary. Does that cause problems in practice? We'd
>> have the same problems as from having two different C runtimes in the
>> same application.
> 
> Yes, I think this could cause problems.
>

But I believe they would be the same set of problems that PEP 384
addresses for cross-version compatibility. Because the linker only sees
the C symbols, it shouldn't be possible for the C++ specific parts of
the runtimes to interfere with each other.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
---
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread [email protected] 

On Jan 21, 2010, at 11:32 PM, Reid Kleckner wrote:

> On Thu, Jan 21, 2010 at 5:07 PM, David Malcolm  wrote:
>> To what extent would it be possible to use (conditionally) use full
>> ahead-of-time compilation as well as JIT?
> 
> It would be possible to do this, but it doesn't have nearly the same
> benefits as JIT compilation, as Alex mentioned.  You could do a static
> compilation of all code objects in a .pyc to LLVM IR and compile that
> to a .so that you load at runtime, but it just eliminates the
> interpreter overhead.  That is significant, and I think someone should
> try it, but I think there are far more wins to be had using feedback.

Wouldn't it be cool if you could run your app (say, with a torture test data 
set), let the JIT do its magic, then say:

"Yes, that's what I want, now save the JIT info as it was in that run 
and use it as the JIT profile every time you run the app but turn off any JIT 
profiling during those runs."

Get rid of the compiler overhead (with .so trick from quoted message) , 
and the runtime overhead, but keep the JIT-ness that it already figured out was 
best.

S


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread [email protected] 

On Jan 21, 2010, at 11:32 PM, Chris Bergstresser wrote:

> On Thu, Jan 21, 2010 at 9:49 PM, Tres Seaver  wrote:
>   Generally, that's not going to be the case.  But the broader
> point--that you've no longer got an especially good idea of what's
> taking time to run in your program--is still very valid.

I'm sure someone's given it a clever name that I don't know but it's kind of 
the profiling Heisenbug -- the information you need to optimize disappears when 
you turn on the JIT optimizer.

S

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread skip 

Reid> You could do a static compilation of all code objects in a .pyc to
Reid> LLVM IR and compile that to a .so that you load at runtime, but it
Reid> just eliminates the interpreter overhead.

OTOH, it would solve the problem some people have distributing their
proprietary apps as .py or .pyc files...  And it could have significant
performance benefits if type annotation was used (and was accurate).

Skip

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposed downstream change to site.py in Fedora (sys.defaultencoding)

2010-01-22 Thread Michael Foord 

On 21/01/2010 21:21, "Martin v. Löwis" wrote:

Where the default *file system encoding* is used (i.e. text files are
written or read without specifying an encoding)
 

I think you misunderstand the notion of the *file system encoding*.
It is *not* a "file encoding", but the file *system* encoding, i.e.
the encoding for file *names*, not for file *content*.

It was used on Windows for Windows 95; it is not used anymore on Windows
(although it's still used on Unix).
   


Ok, I'm just using the wrong terminology. I'm aware that mbcs is used 
for filename encoding on Windows (right?). The encoding I'm talking 
about is the encoding that Python uses to decode a file (or encode a 
string) when you do the following in Python 3:


text = open(filename).read()
open(filename, 'w').write(some_string)

It isn't the default encoding (always utf-8 by default in Python 3 
apparently), it isn't the file system encoding which is the system 
encoding used for file names. What is the correct terminology for this 
platform dependent encoding that Python uses here?


The important point is that it is platform dependent - so if you ship 
and use text files with your Python application and don't specify an 
encoding then it will work fine on some platforms and blow up or use the 
wrong encoding on other platforms.




I think there are way too many specific cases where Python 3 will encode
implicitly to get a complete list from the memory. If you really are
after a complete list, you'll need to perform a thorough code review.
For a few examples where some kind of default encoding is applied,
consider XML and the dbm interfaces.
   

Ok. Understood.

Thanks

Michael Foord



Regards,
Martin
   



--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of 
your employer, to release me from all obligations and waivers arising from any 
and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, 
clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and 
acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your 
employer, its partners, licensors, agents and assigns, in perpetuity, without 
prejudice to my ongoing rights and privileges. You further represent that you 
have the authority to release me from any BOGUS AGREEMENTS on behalf of your 
employer.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyCon Keynote

2010-01-22 Thread Justin Lilly 
On Wed, Jan 13, 2010 at 1:51 PM, Guido van Rossum  wrote:

> Please mail me topics you'd like to hear me talk about in my keynote
> at PyCon this year.
>
> --
> --Guido van Rossum (python.org/~guido)
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/justin%40justinlilly.com
>

Which pieces (and to what degree) the community has influenced your vision
of python. ie: You were initially against feature X, but the community
showed you that it really was a good thing.

 Something to that effect would be interesting.

 -justin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Michael Foord 

On 22/01/2010 11:45, [email protected] wrote:

On Jan 21, 2010, at 11:32 PM, Reid Kleckner wrote:

   

On Thu, Jan 21, 2010 at 5:07 PM, David Malcolm  wrote:
 

To what extent would it be possible to use (conditionally) use full
ahead-of-time compilation as well as JIT?
   

It would be possible to do this, but it doesn't have nearly the same
benefits as JIT compilation, as Alex mentioned.  You could do a static
compilation of all code objects in a .pyc to LLVM IR and compile that
to a .so that you load at runtime, but it just eliminates the
interpreter overhead.  That is significant, and I think someone should
try it, but I think there are far more wins to be had using feedback.
 

Wouldn't it be cool if you could run your app (say, with a torture test data 
set), let the JIT do its magic, then say:

"Yes, that's what I want, now save the JIT info as it was in that run and 
use it as the JIT profile every time you run the app but turn off any JIT profiling 
during those runs."

Get rid of the compiler overhead (with .so trick from quoted message) , 
and the runtime overhead, but keep the JIT-ness that it already figured out was 
best.
   


The .NET framework allows a similar process - a tool called ngen can be 
run against compiled .NET bytecode, generating and caching the native 
code images. When your compiled code is run the precompiled native code 
is used automatically without the JIT needing to be invoked.


It is provided as a separate tool (and often invoked by application 
installers) rather than allowing the native code to be distributed 
because the results can be system specific.


Michael Foord

S


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
   



--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of 
your employer, to release me from all obligations and waivers arising from any 
and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, 
clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and 
acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your 
employer, its partners, licensors, agents and assigns, in perpetuity, without 
prejudice to my ongoing rights and privileges. You further represent that you 
have the authority to release me from any BOGUS AGREEMENTS on behalf of your 
employer.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyCon Keynote

2010-01-22 Thread Gerard Flanagan 

Guido van Rossum wrote:

Please mail me topics you'd like to hear me talk about in my keynote
at PyCon this year.



You might like to comment on the rumored links between the Chinese 
Government and the Python Softwamnnn..


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposed downstream change to site.py in Fedora (sys.defaultencoding)

2010-01-22 Thread Karen Tracey 
On Fri, Jan 22, 2010 at 7:38 AM, Michael Foord wrote:

> On 21/01/2010 21:21, "Martin v. Löwis" wrote:
>
>> Where the default *file system encoding* is used (i.e. text files are
>>> written or read without specifying an encoding)
>>>
>>>
>> I think you misunderstand the notion of the *file system encoding*.
>> It is *not* a "file encoding", but the file *system* encoding, i.e.
>> the encoding for file *names*, not for file *content*.
>>
>> It was used on Windows for Windows 95; it is not used anymore on Windows
>> (although it's still used on Unix).
>>
>>
>
> Ok, I'm just using the wrong terminology. I'm aware that mbcs is used for
> filename encoding on Windows (right?). The encoding I'm talking about is the
> encoding that Python uses to decode a file (or encode a string) when you do
> the following in Python 3:
>
>text = open(filename).read()
>open(filename, 'w').write(some_string)
>
> It isn't the default encoding (always utf-8 by default in Python 3
> apparently), it isn't the file system encoding which is the system encoding
> used for file names. What is the correct terminology for this platform
> dependent encoding that Python uses here?
>
>
The doc here:
http://docs.python.org/3.1/library/functions.html?highlight=open#open just
calls it default encoding and clarifies that is "whatever
locale.getpreferredencoding() returns".

The important point is that it is platform dependent - so if you ship and
> use text files with your Python application and don't specify an encoding
> then it will work fine on some platforms and blow up or use the wrong
> encoding on other platforms.
>
>
Yes. If you ship text files with your Python application, then you'd best
take care to know the encoding when you create them and specify it as the
encoding to use when you open the file for reading by your application.

Karen
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposed downstream change to site.py in Fedora (sys.defaultencoding)

2010-01-22 Thread Michael Foord 

On 22/01/2010 14:18, Karen Tracey wrote:
On Fri, Jan 22, 2010 at 7:38 AM, Michael Foord 
mailto:[email protected]>> wrote:


On 21/01/2010 21:21, "Martin v. Löwis" wrote:

Where the default *file system encoding* is used (i.e.
text files are
written or read without specifying an encoding)

I think you misunderstand the notion of the *file system
encoding*.
It is *not* a "file encoding", but the file *system* encoding,
i.e.
the encoding for file *names*, not for file *content*.

It was used on Windows for Windows 95; it is not used anymore
on Windows
(although it's still used on Unix).


Ok, I'm just using the wrong terminology. I'm aware that mbcs is
used for filename encoding on Windows (right?). The encoding I'm
talking about is the encoding that Python uses to decode a file
(or encode a string) when you do the following in Python 3:

   text = open(filename).read()
   open(filename, 'w').write(some_string)

It isn't the default encoding (always utf-8 by default in Python 3
apparently), it isn't the file system encoding which is the system
encoding used for file names. What is the correct terminology for
this platform dependent encoding that Python uses here?


The doc here: 
http://docs.python.org/3.1/library/functions.html?highlight=open#open 
just calls it default encoding and clarifies that is "whatever 
locale.getpreferredencoding() returns".




Heh, so we have two different encoding mechanisms both called "default 
encoding". One is always utf-8 in Python 3 and one is platform 
dependent... Great. :-)


Michael



The important point is that it is platform dependent - so if you
ship and use text files with your Python application and don't
specify an encoding then it will work fine on some platforms and
blow up or use the wrong encoding on other platforms.


Yes. If you ship text files with your Python application, then you'd 
best take care to know the encoding when you create them and specify 
it as the encoding to use when you open the file for reading by your 
application.


Karen


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
   



--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your 
employer, to release me from all obligations and waivers arising from any and all 
NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, 
confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS 
AGREEMENTS") that I have entered into with your employer, its partners, licensors, 
agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. 
You further represent that you have the authority to release me from any BOGUS AGREEMENTS 
on behalf of your employer.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposed downstream change to site.py in Fedora (sys.defaultencoding)

2010-01-22 Thread Karen Tracey 
On Fri, Jan 22, 2010 at 9:22 AM, Michael Foord wrote:

>  On 22/01/2010 14:18, Karen Tracey wrote:
>
>
> The doc here:
> http://docs.python.org/3.1/library/functions.html?highlight=open#open just
> calls it default encoding and clarifies that is "whatever
> locale.getpreferredencoding() returns".
>
>
> Heh, so we have two different encoding mechanisms both called "default
> encoding". One is always utf-8 in Python 3 and one is platform dependent...
> Great. :-)
>

Well to be fair the the doc is describing the default value for the
parameter named encoding, so its reference to "default encoding" should not
be extrapolated to mean the default encoding for Python 3 overall. I worded
that sentence above poorly. But yes, it is easy to get confused here.

Karen
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposed downstream change to site.py in Fedora (sys.defaultencoding)

2010-01-22 Thread Antoine Pitrou 
Michael Foord  voidspace.org.uk> writes:
> 
> Heh, so we have two different encoding mechanisms both called "default
> encoding". One is always utf-8 in Python 3 and one is platform
> dependent... Great. 

The former is merely internal though. Also, if you grep for the "s#" and "s*"
argument type codes in the Modules directory, you'll see that they don't get a
lot of use in py3k.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposed downstream change to site.py in Fedora (sys.defaultencoding)

2010-01-22 Thread Michael Foord 

On 22/01/2010 14:33, Antoine Pitrou wrote:

Michael Foord  voidspace.org.uk>  writes:
   

Heh, so we have two different encoding mechanisms both called "default
encoding". One is always utf-8 in Python 3 and one is platform
dependent... Great.
 

The former is merely internal though. Also, if you grep for the "s#" and "s*"
argument type codes in the Modules directory, you'll see that they don't get a
lot of use in py3k.

   
Right, but this 'internal' one is the encoding that was being referred 
to as *the default encoding* earlier in this thread. What is the correct 
terminology for *the other default encoding* that I am referring to?


All the best,

Michael



Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
   



--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of 
your employer, to release me from all obligations and waivers arising from any 
and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, 
clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and 
acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your 
employer, its partners, licensors, agents and assigns, in perpetuity, without 
prejudice to my ongoing rights and privileges. You further represent that you 
have the authority to release me from any BOGUS AGREEMENTS on behalf of your 
employer.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread John Arbash Meinel 
[email protected] wrote:
> On Jan 21, 2010, at 11:32 PM, Chris Bergstresser wrote:
> 
>> On Thu, Jan 21, 2010 at 9:49 PM, Tres Seaver  wrote:
>>   Generally, that's not going to be the case.  But the broader
>> point--that you've no longer got an especially good idea of what's
>> taking time to run in your program--is still very valid.
> 
> I'm sure someone's given it a clever name that I don't know but it's kind of 
> the profiling Heisenbug -- the information you need to optimize disappears 
> when you turn on the JIT optimizer.
> 
> S

I would assume that part of the concern is not being able to get
per-line profiling out of the JIT'd code. Personally, I'd rather see one
big "and I called a JIT function that took XX seconds" rather than
getting nothing.

At the moment, we have some small issues with cProfile in that it
doesn't attribute time to extension functions particularly well.

For example, I've never seen a Pyrex "__init__" function show up in
timing, the time spent always gets assigned to the calling function. So
if I want to see it, I set up a 'create_foo(*args, **kwargs)' function
that just does return Foo(*args, **kwargs).

I don't remember the other bits I've run into. But certainly I would say
that giving some sort of real-world profiling is better than having it
drop back to interpreted code. You could always run --profile -j never
if you really wanted to get the non-JIT'd code.

John
=:->

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyCon Keynote

2010-01-22 Thread Chris McDonough 

Guido van Rossum wrote:

Please mail me topics you'd like to hear me talk about in my keynote
at PyCon this year.



Can you tell us where Uncle Timmy has been and when he'll be back?


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyCon Keynote

2010-01-22 Thread Barry Warsaw 
On Jan 22, 2010, at 10:06 AM, Chris McDonough wrote:

>Can you tell us where Uncle Timmy has been and when he'll be back?

He's given up bags of ham for walls of chocolate.

-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Summary of Python tracker Issues

2010-01-22 Thread Python tracker 

ACTIVITY SUMMARY (01/15/10 - 01/22/10)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2558 open (+37) / 17021 closed (+13) / 19579 total (+50)

Open issues with patches:  1042

Average duration of open issues: 708 days.
Median duration of open issues: 466 days.

Open Issues Breakdown
   open  2524 (+37)
pending33 ( +0)

Issues Created Or Reopened (52)
___

Bytes alias for 2.6  01/17/10
   http://bugs.python.org/issue1865reopened flox
  
   patch   

"Suggest a change" link  01/18/10
   http://bugs.python.org/issue3341reopened techtonik   
  
   

test_xmlrpc fails with non-ascii path01/15/10
   http://bugs.python.org/issue7708created  flox
  
   patch, buildbot 

the msg parameters of PyUnit assertions, such as assertEqual, sh 01/15/10
CLOSED http://bugs.python.org/issue7709created  Phlip   
  
   

Inconsistent Exception for int() conversion  01/15/10
   http://bugs.python.org/issue7710created  flox
  
   

csv error name incorrect 01/16/10
CLOSED http://bugs.python.org/issue7711created  kitterma
  
   

Add a context manager to change cwd in test.test_support 01/16/10
   http://bugs.python.org/issue7712created  ezio.melotti
  
   patch, patch

implement ability to disable automatic search path additions 01/16/10
   http://bugs.python.org/issue7713created  grobian 
  
   

configure GCC version detection fix for Darwin   01/16/10
   http://bugs.python.org/issue7714created  grobian 
  
   patch, needs review 

Allow use of GNU arch on Darwin  01/16/10
   http://bugs.python.org/issue7715created  grobian 
  
   patch, needs review 

IPv6 detection, don't assume existence of /usr/xpg4/bin/grep 01/16/10
   http://bugs.python.org/issue7716created  grobian 
  
   patch, needs review 

Compilation fixes for IRIX   01/16/10
   http://bugs.python.org/issue7717created  grobian 
  
   patch, needs review 

Build shared libpythonX.Y.so on IRIX 01/16/10
CLOSED http://bugs.python.org/issue7718created  grobian 
  
   patch   

distutils: ignore .nfs files 01/16/10
   http://bugs.python.org/issue7719created  grobian 
  
   patch, needs review 

Errors in tests and C implementation of raw FileIO   01/16/10
   http://bugs.python.org/issue7720created  pakal   
  
   

Code in xrange documentation does not work   01/16/10
   http://bugs.python.org/issue7721created  mm  
  
   

calendar.{HTMLCalendar,TextCalendar}.formatyear have incorrect d 01/17/10
CLOSED http://bugs.python.org/issue7722created  wathiede
  
   

sqlite only accept buffer() for BLOB objects(input/output)  01/17/10
   http://bugs.python.org/issue7723created  flox
  
   

setup.py ignores SDK root on OSX 01/17/10
   http://bugs.python.org/issue7724created  ronaldoussoren

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Stefan Behnel 
John Arbash Meinel, 22.01.2010 15:56:
> For example, I've never seen a Pyrex "__init__" function show up in
> timing, the time spent always gets assigned to the calling function. So
> if I want to see it, I set up a 'create_foo(*args, **kwargs)' function
> that just does return Foo(*args, **kwargs).

Note that this has been fixed in Cython. When you profile Cython code in
cProfile, you will get profiling information of all Cython implemented
functions, even for C calls inside of the Cython module itself.

Stefan

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Stefan Behnel 
Collin Winter, 21.01.2010 21:14:
> Cython is a super-set of Python, and files annotated for maximum
> Cython performance are no longer valid Python code, and will not run
> on any other Python implementation.

Except when you use plain Python annotations, in which case you get the
speed of Cython for regular Python code.

http://wiki.cython.org/pure

There's also the possibility to put the annotations into a separate .pxd
file next to the .py file, which then stays pure Python as well.

Anyway, I certainly see the advantages that a JIT compiler has over static
ahead-of-time compilation.

Stefan

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Tony Nelson 
On 10-01-22 02:53:21, Collin Winter wrote:
> On Thu, Jan 21, 2010 at 11:37 PM, Glyph Lefkowitz
>  wrote:
> >
> > On Jan 21, 2010, at 6:48 PM, Collin Winter wrote:
 ...
> > There's been a recent thread on our mailing list about a patch that
> > dramatically reduces the memory footprint of multiprocess
> > concurrency by separating reference counts from objects. ...

Currently, CPython gets a performance advantage from having reference 
counts hot in the cache when the referenced object is used.  There is 
still the write pressure from updating the counts.  With separate 
reference counts, an extra cache line must be loaded from memory (it is 
unlikely to be in the cache unless the program is trivial).  I see from 
the referenced posting that this is a 10% speed hit (the poster 
attributes the hit to extra instructions).

Perhaps the speed and memory hits could be minimized by only doing this 
for some objects?  Only objects that are fully shared (such as read-
only data) benefit from this change.  I don't know but shared objects 
may already be treated separately.

 ...
> The data I've seen comes from
> http://groups.google.com/group/comp.lang.python/msg/c18b671f2c4fef9e:
 ...

-- 

TonyN.:'   
  '  

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Collin Winter 
Hey Tony,

On Fri, Jan 22, 2010 at 10:11 AM, Tony Nelson
 wrote:
> On 10-01-22 02:53:21, Collin Winter wrote:
>> On Thu, Jan 21, 2010 at 11:37 PM, Glyph Lefkowitz
>>  wrote:
>> >
>> > On Jan 21, 2010, at 6:48 PM, Collin Winter wrote:
>  ...
>> > There's been a recent thread on our mailing list about a patch that
>> > dramatically reduces the memory footprint of multiprocess
>> > concurrency by separating reference counts from objects. ...
>
> Currently, CPython gets a performance advantage from having reference
> counts hot in the cache when the referenced object is used.  There is
> still the write pressure from updating the counts.  With separate
> reference counts, an extra cache line must be loaded from memory (it is
> unlikely to be in the cache unless the program is trivial).  I see from
> the referenced posting that this is a 10% speed hit (the poster
> attributes the hit to extra instructions).
>
> Perhaps the speed and memory hits could be minimized by only doing this
> for some objects?  Only objects that are fully shared (such as read-
> only data) benefit from this change.  I don't know but shared objects
> may already be treated separately.

One option that we discussed was to create a ./configure flag to
toggle between inline refcounts and separate refcounts. Advanced users
that care about the memory usage of multiprocess concurrency could
compile their own CPython binary to enable this space optimization at
the cost of some performance.

On the other hand, once we get enough performance out of the JIT that
python-dev is willing to take a 10% hit, then I'd say we should just
turn the space optimization on by default. In the meantime, though, a
configure flag would be a useful intermediate point for a number of
people.

Collin Winter
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposed downstream change to site.py in Fedora (sys.defaultencoding)

2010-01-22 Thread M.-A. Lemburg 
Karen Tracey wrote:
> On Fri, Jan 22, 2010 at 7:38 AM, Michael Foord 
> wrote:
> 
>> On 21/01/2010 21:21, "Martin v. Löwis" wrote:
>>
>>> Where the default *file system encoding* is used (i.e. text files are
 written or read without specifying an encoding)


>>> I think you misunderstand the notion of the *file system encoding*.
>>> It is *not* a "file encoding", but the file *system* encoding, i.e.
>>> the encoding for file *names*, not for file *content*.
>>>
>>> It was used on Windows for Windows 95; it is not used anymore on Windows
>>> (although it's still used on Unix).
>>>
>>>
>>
>> Ok, I'm just using the wrong terminology. I'm aware that mbcs is used for
>> filename encoding on Windows (right?). The encoding I'm talking about is the
>> encoding that Python uses to decode a file (or encode a string) when you do
>> the following in Python 3:
>>
>>text = open(filename).read()
>>open(filename, 'w').write(some_string)
>>
>> It isn't the default encoding (always utf-8 by default in Python 3
>> apparently), it isn't the file system encoding which is the system encoding
>> used for file names. What is the correct terminology for this platform
>> dependent encoding that Python uses here?
>>
>>
> The doc here:
> http://docs.python.org/3.1/library/functions.html?highlight=open#open just
> calls it default encoding and clarifies that is "whatever
> locale.getpreferredencoding() returns".

... which is a pretty poor guess, since the locale setting
is usually set to what the user wants to see in user interfaces,
not what the application want to use as file content.

As example take XML files: These will almost always use UTF-8 as
encoding. If you use the above approach to write them and happen
to work on a system that is set to use Latin-1 or CP1525 as
preferred encoding, you'll get garbage in your XML file.

> The important point is that it is platform dependent - so if you ship and
>> use text files with your Python application and don't specify an encoding
>> then it will work fine on some platforms and blow up or use the wrong
>> encoding on other platforms.
>>
>>
> Yes. If you ship text files with your Python application, then you'd best
> take care to know the encoding when you create them and specify it as the
> encoding to use when you open the file for reading by your application.

Right.

Applications should always provide a well-defined encoding
for use with files - they know best what encoding to use and
they also know best when to apply guessing and which set
of encodings to use as basis for such guessing.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 22 2010)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Collin Winter 
Hey Jake,

On Thu, Jan 21, 2010 at 10:48 AM, Jake McGuire  wrote:
> On Thu, Jan 21, 2010 at 10:19 AM, Reid Kleckner  wrote:
>> On Thu, Jan 21, 2010 at 12:27 PM, Jake McGuire  wrote:
>>> On Wed, Jan 20, 2010 at 2:27 PM, Collin Winter  
>>> wrote:
 Profiling
 -

 Unladen Swallow integrates with oProfile 0.9.4 and newer [#oprofile]_ to 
 support
 assembly-level profiling on Linux systems. This means that oProfile will
 correctly symbolize JIT-compiled functions in its reports.
>>>
>>> Do the current python profiling tools (profile/cProfile/pstats) still
>>> work with Unladen Swallow?
>>
>> Sort of.  They disable the use of JITed code, so they don't quite work
>> the way you would want them to.  Checking tstate->c_tracefunc every
>> line generated too much code.  They still give you a rough idea of
>> where your application hotspots are, though, which I think is
>> acceptable.
>
> Hmm.  So cProfile doesn't break, but it causes code to run under a
> completely different execution model so the numbers it produces are
> not connected to reality?
>
> We've found the call graph and associated execution time information
> from cProfile to be extremely useful for understanding performance
> issues and tracking down regressions.  Giving that up would be a huge
> blow.

FWIW, cProfile's call graph information is still perfectly accurate,
but you're right: turning on cProfile does trigger execution under a
different codepath. That's regrettable, but instrumentation-based
profiling is always going to introduce skew into your numbers. That's
why we opted to improve oProfile, since we believe sampling-based
profiling to be a better model.

Profiling was problematic to support in machine code because in
Python, you can turn profiling on from user code at arbitrary points.
To correctly support that, we would need to add lots of hooks to the
generated code to check whether profiling is enabled, and if so, call
out to the profiler. Those "is profiling enabled now?" checks are
(almost) always going to be false, which means we spend cycles for no
real benefit.

Can YouTube use oProfile for profiling, or is instrumented profiling
critical? oProfile does have its downsides for profiling user code:
you see all the C-language support functions, not just the pure-Python
functions. That extra data might be useful, but it's probably more
information than most people want. YouTube might want it, though.

Assuming YouTube can't use oProfile as-is, there are some options:
- Write a script around oProfile's reporting tool to strip out all C
functions from the report. Enhance oProfile to fix any deficiencies
compared to cProfile's reporting.
- Develop a sampling profiler for Python that only samples pure-Python
functions, ignoring C code (but including JIT-compiled Python code).
- Add the necessary profiling hooks to JITted code to better support
cProfile, but add a command-line flag (something explicit like -O3)
that removes the hooks and activates the current behaviour (or
something even more restrictive, possibly).
- Initially compile Python code without the hooks, but have a
trip-wire set to detect the installation of profiling hooks. When
profiling hooks are installed, purge all machine code from the system
and recompile all hot functions to include the profiling hooks.

Thoughts?

Collin Winter
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Alex Gaynor 
On Fri, Jan 22, 2010 at 1:07 PM, Collin Winter  wrote:
> Hey Jake,
>
> On Thu, Jan 21, 2010 at 10:48 AM, Jake McGuire  wrote:
>> On Thu, Jan 21, 2010 at 10:19 AM, Reid Kleckner  wrote:
>>> On Thu, Jan 21, 2010 at 12:27 PM, Jake McGuire  wrote:
 On Wed, Jan 20, 2010 at 2:27 PM, Collin Winter  
 wrote:
> Profiling
> -
>
> Unladen Swallow integrates with oProfile 0.9.4 and newer [#oprofile]_ to 
> support
> assembly-level profiling on Linux systems. This means that oProfile will
> correctly symbolize JIT-compiled functions in its reports.

 Do the current python profiling tools (profile/cProfile/pstats) still
 work with Unladen Swallow?
>>>
>>> Sort of.  They disable the use of JITed code, so they don't quite work
>>> the way you would want them to.  Checking tstate->c_tracefunc every
>>> line generated too much code.  They still give you a rough idea of
>>> where your application hotspots are, though, which I think is
>>> acceptable.
>>
>> Hmm.  So cProfile doesn't break, but it causes code to run under a
>> completely different execution model so the numbers it produces are
>> not connected to reality?
>>
>> We've found the call graph and associated execution time information
>> from cProfile to be extremely useful for understanding performance
>> issues and tracking down regressions.  Giving that up would be a huge
>> blow.
>
> FWIW, cProfile's call graph information is still perfectly accurate,
> but you're right: turning on cProfile does trigger execution under a
> different codepath. That's regrettable, but instrumentation-based
> profiling is always going to introduce skew into your numbers. That's
> why we opted to improve oProfile, since we believe sampling-based
> profiling to be a better model.
>
> Profiling was problematic to support in machine code because in
> Python, you can turn profiling on from user code at arbitrary points.
> To correctly support that, we would need to add lots of hooks to the
> generated code to check whether profiling is enabled, and if so, call
> out to the profiler. Those "is profiling enabled now?" checks are
> (almost) always going to be false, which means we spend cycles for no
> real benefit.
>
> Can YouTube use oProfile for profiling, or is instrumented profiling
> critical? oProfile does have its downsides for profiling user code:
> you see all the C-language support functions, not just the pure-Python
> functions. That extra data might be useful, but it's probably more
> information than most people want. YouTube might want it, though.
>
> Assuming YouTube can't use oProfile as-is, there are some options:
> - Write a script around oProfile's reporting tool to strip out all C
> functions from the report. Enhance oProfile to fix any deficiencies
> compared to cProfile's reporting.
> - Develop a sampling profiler for Python that only samples pure-Python
> functions, ignoring C code (but including JIT-compiled Python code).
> - Add the necessary profiling hooks to JITted code to better support
> cProfile, but add a command-line flag (something explicit like -O3)
> that removes the hooks and activates the current behaviour (or
> something even more restrictive, possibly).
> - Initially compile Python code without the hooks, but have a
> trip-wire set to detect the installation of profiling hooks. When
> profiling hooks are installed, purge all machine code from the system
> and recompile all hot functions to include the profiling hooks.
>
> Thoughts?
>
> Collin Winter
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/alex.gaynor%40gmail.com
>

What about making profiling something more tied to the core VM.  So
profiling is either enabled or disabled for the course of the run of
the application, not something that can be enabled or disabled
arbitrarily.  This way there's no overhead in JIT compiled code
without profiling, and profiling has no worse overhead than it would
in the VM loop.

It's a slightly different semantic to profiling, but I wonder whether
there's really any value to the other way?

Alex

-- 
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread John Arbash Meinel 
Collin Winter wrote:
> Hey Jake,
> 
...

>> Hmm.  So cProfile doesn't break, but it causes code to run under a
>> completely different execution model so the numbers it produces are
>> not connected to reality?
>>
>> We've found the call graph and associated execution time information
>> from cProfile to be extremely useful for understanding performance
>> issues and tracking down regressions.  Giving that up would be a huge
>> blow.
> 
> FWIW, cProfile's call graph information is still perfectly accurate,
> but you're right: turning on cProfile does trigger execution under a
> different codepath. That's regrettable, but instrumentation-based
> profiling is always going to introduce skew into your numbers. That's
> why we opted to improve oProfile, since we believe sampling-based
> profiling to be a better model.
> 
> Profiling was problematic to support in machine code because in
> Python, you can turn profiling on from user code at arbitrary points.
> To correctly support that, we would need to add lots of hooks to the
> generated code to check whether profiling is enabled, and if so, call
> out to the profiler. Those "is profiling enabled now?" checks are
> (almost) always going to be false, which means we spend cycles for no
> real benefit.

At least my point was that I'd rather the machine code generated never
called out to the profiler, but that at least calling the machine code
itself would be profiled. That would show larger-than-minimal hot spots,
but at least it wouldn't change the actual hotspots.


> 
> Can YouTube use oProfile for profiling, or is instrumented profiling
> critical? oProfile does have its downsides for profiling user code:
> you see all the C-language support functions, not just the pure-Python
> functions. That extra data might be useful, but it's probably more
> information than most people want. YouTube might want it, though.
> 

Does oprofile actually give you much of the python state? When I've
tried that sort of profiling, it seems to tell me the C function that
the VM is in, rather than the python functions being executed. Knowing
that PyDict_GetItem is being called 20M times doesn't help me a lot if
it doesn't tell me that it is being called in

 def foo(d):
for x in d:
  for y in d:
 if x != y:
   assert d[x] != d[y]

(Or whatever foolish function you do that in)

I'd certainly like to know that 'foo()' was the one to attribute the 20M
calls to PyDict_GetItem.

Googling to search for oProfile python just gives me Unladen Swallow
mentions of making oprofile work... :)

> Assuming YouTube can't use oProfile as-is, there are some options:
> - Write a script around oProfile's reporting tool to strip out all C
> functions from the report. Enhance oProfile to fix any deficiencies
> compared to cProfile's reporting.
> - Develop a sampling profiler for Python that only samples pure-Python
> functions, ignoring C code (but including JIT-compiled Python code).
> - Add the necessary profiling hooks to JITted code to better support
> cProfile, but add a command-line flag (something explicit like -O3)
> that removes the hooks and activates the current behaviour (or
> something even more restrictive, possibly).
> - Initially compile Python code without the hooks, but have a
> trip-wire set to detect the installation of profiling hooks. When
> profiling hooks are installed, purge all machine code from the system
> and recompile all hot functions to include the profiling hooks.
> 
> Thoughts?
> 
> Collin Winter

John
=:->
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bazaar branches available (again) on Launchpad

2010-01-22 Thread Georg Brandl 
Am 21.01.2010 04:33, schrieb Ben Finney:
> Georg Brandl  writes:
> 
>> But I've no intention to restrict feature releases to "every 18-24
>> months". What now?
> 
> Now we take further discussion to the ‘python-ideas’ forum.

Whoever "we" is.  You can be assured I've not planned any further
discussion on this matter.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposed downstream change to site.py in Fedora (sys.defaultencoding)

2010-01-22 Thread Martin v. Löwis 
> Ok, I'm just using the wrong terminology. I'm aware that mbcs is used
> for filename encoding on Windows (right?).

Not anymore, no.

> The encoding I'm talking
> about is the encoding that Python uses to decode a file (or encode a
> string) when you do the following in Python 3:
> 
> text = open(filename).read()
> open(filename, 'w').write(some_string)
> 
> It isn't the default encoding (always utf-8 by default in Python 3
> apparently), it isn't the file system encoding which is the system
> encoding used for file names. What is the correct terminology for this
> platform dependent encoding that Python uses here?

I don't think there is an official term for it. I (alternatingly)
call it the locale's encoding, or the user's preferred encoding.
It's the encoding that the user (directly or indirectly) has asked
applications to use for data in absence of a more specific
requirement.

> The important point is that it is platform dependent - so if you ship
> and use text files with your Python application and don't specify an
> encoding then it will work fine on some platforms and blow up or use the
> wrong encoding on other platforms.

Correct. Also, data output on some platform will be unreadable on some
other platform.

Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Enhancing the shutil module

2010-01-22 Thread Sridhar Ratnakumar 

On 1/17/2010 2:09 PM, Tarek Ziadé wrote:

>  Distribute has
>  some utility code to handle zip/tar archives. So does PyPM. This is because
>  the `tarfile` and `zipfile` modules do not "just work" due to several
>  issues.
>
>  Seehttp://gist.github.com/279606
>
>  Take note of the following in the above code:
>
>1) _ensure_read_write_access
>2) *File.is_valid
>3) ZippedFile.extract ... issue 6510
>4) ZippedFile.extract ... issue 6609
>5) TarredFile.extract ... issue 6584

Looks like some of these are already fixed (I just looked quickly in
the bug tracker though)


Only one of them was fixed (issue 6510), but the rest were either 
rejected or still open:


  http://bugs.python.org/issue6609 - catch special file errors
  http://bugs.python.org/issue6584 - catch CRC errors in gzip
  http://bugs.python.org/issue6196 - set permissions during extract


If its not already done and pending, it would be great if you could
refactor your fixes into patches for the remaining issues for each one
of those modules


I'd be happy to do so. This is what I am considering:

1) Patch Lib/zipfile.py to fix issue6609
2) Patch Lib/gzip.py and Lib/tarfile.py to fix issue6584
3) Patch Lib/tarfile.py to fix issue6196

I am hoping that (1) and (2) will get accepted. But not (3) - in which 
case, should this go as a workarond (_ensure_read_write_access in 
http://gist.github.com/279606) in the new, say, shutil.extract function?


-srid
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Martin v. Löwis 
> I'm not an expert here, but this is why we've tried to avoid exposing
> C++ functions into the public API. As long as the public API stays C,
> we only require that LLVM and Python are compiled with the same
> compiler, right?

Possibly not. You may end up loading two different C++ runtimes into
address space (e.g. the g++ one and the SunPRO one). If your OS has
a global single-level symbol space (such as traditional SysV, or Linux),
and if then the C++ libraries also have common function names but give
different meanings to the same function, you lose.

In practice (and in the past), this didn't really cause that much
problems in itself, as the C++RTs where likely to use different name
mangling, so they wouldn't have overlapping function names.

What the situation is today can best be determined by testing.

> I could imagine a problem if Python+LLVM link in one libstdc++, and an
> extension module links in a different one, even if no C++ objects are
> passed across the boundary. Does that cause problems in practice? We'd
> have the same problems as from having two different C runtimes in the
> same application.

This typically doesn't happen, on Unix, as the CRT is provided by the
system vendor, and even third-party C compilers still link to the system
library (which is possible because the system vendor also includes
header files with the system). For C++, having a standard ABI is not yet
that common (although Linux has established one which at least the Intel
compiler complies with).

Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread James Abbatiello 
On Thu, Jan 21, 2010 at 5:24 PM, Collin Winter  wrote:
> On Thu, Jan 21, 2010 at 12:56 PM, Paul Moore  wrote:
>> Windows compatibility is a big deal to me. And IMHO, it's a great
>> strength of Python at the moment that it has solid Windows support. I
>> would be strongly *against* this PEP if it was going to be Unix or
>> Linux only. As it is, I have concerns that Windows could suffer from
>> the common "none of the developers use Windows, but we do our best"
>> problem. I'm hoping that having U-S integrated into the core will mean
>> that there will be more Windows developers able to contribute and
>> alleviate that problem.
>
> One of our contributors, James Abbatiello (cc'd), has done a bang-up
> job of making Unladen Swallow work on Windows. My understanding from
> his last update is that Unladen Swallow works well on Windows, but he
> can comment further as to the precise state of Windows support and any
> remaining challenges faced on that platform, if any.

Unladen Swallow currently works well on Windows, compiling with MSVC.
The CPython regression test suite passes.  It has been a while since
I've run Unladen's collection of 3rd party tests but things looked
good there as well.  All the tests that were working under CPython on
Windows were also working under Unladen.  Some of the tests have Unix
assumptions in them that cause them not to work on Windows.  I've
tried to get some of these fixed upstream but I haven't yet tried to
get newer versions into the Unladen SVN repo.

Keeping things working on Windows hasn't been that difficult.  Most of
the effort recently has been in simply updating the project files to
include any new source files.  Occasionally there will be a problem
with LLVM due to tracking LLVM's trunk which may be broken on Windows
at the time a new version is pulled in.  As the plan is to work with
released versions of LLVM after 2.7 I don't anticipate this being a
problem in the future.

I'm using Visual Studio Express which doesn't have the ability to
compile PGO or x86-64 binaries so those are untested.  I've tried to
keep the project files correct for those targets so hopefully it
shouldn't be too much work for somebody with the full version of
Visual Studio to get them working.  The project files probably need a
bit of other work.  Currently LLVM has to be built in a separate step
before Unladen Swallow can be compiled.  It would be nice to automate
this and properly track dependencies but I have not attempted to do
this.  I'd be happy to work with someone who has experience with the
CPython project files to get this into shape.

Overall I don't think that merging Unladen Swallow will unduly
increase the burden of Windows support.  Most of the code that has
been added is fairly platform independent.  Any Windows-specific
complications with, for example, execution of generated machine code
are handled by LLVM and are invisible at this level.

-- 
James Abbatiello
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Martin v. Löwis 
> But I believe they would be the same set of problems that PEP 384
> addresses for cross-version compatibility. Because the linker only sees
> the C symbols, it shouldn't be possible for the C++ specific parts of
> the runtimes to interfere with each other.

Unfortunately, on systems with a global symbol space (and RTLD_GLOBAL
loading), symbols in shared libraries loaded from unrelated contexts
may conflict.

This is one of the motivations to rename the init functions in 3.x.

Regards,
Martin

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Martin v. Löwis 
> It is provided as a separate tool (and often invoked by application
> installers) rather than allowing the native code to be distributed
> because the results can be system specific.

Actually, they have now changed the implementation: it is now a service
which will do the ngen compilation in background, and the command line
tool will just queue requests for the service (unless you ask the tool
to do it synchronously).

I believe they did that because otherwise installers that had a lot of
.NET code would run incredibly long just because of ngen taking forever.
The ngen queue survives a reboot, and proceeds only if the system is
otherwise IDLE - so on an Itanium 1 system, it sometimes took about a
day until it caught up.

Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Enhancing the shutil module

2010-01-22 Thread Tarek Ziadé 
On Fri, Jan 22, 2010 at 11:17 PM, Sridhar Ratnakumar
 wrote:
[..]
> 3) Patch Lib/tarfile.py to fix issue6196
>
> I am hoping that (1) and (2) will get accepted. But not (3) - in which case,
> should this go as a workarond (_ensure_read_write_access in
> http://gist.github.com/279606) in the new, say, shutil.extract function?

I agree with Lars conclusions on this issue, I don't think this
permission feature fits the sdtlib because its a very specific use
case.

Now, for shutil.unpack_archive, we could add a callable option (like
the progress_filter one)
that would be called with each newly extracted file, so one may apply
any process on it.

Regards,
Tarek

-- 
Tarek Ziadé | http://ziade.org
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-01-22 Thread Michael Foord 

On 22/01/2010 22:43, "Martin v. Löwis" wrote:

It is provided as a separate tool (and often invoked by application
installers) rather than allowing the native code to be distributed
because the results can be system specific.
 

Actually, they have now changed the implementation: it is now a service
which will do the ngen compilation in background, and the command line
tool will just queue requests for the service (unless you ask the tool
to do it synchronously).

I believe they did that because otherwise installers that had a lot of
.NET code would run incredibly long just because of ngen taking forever.
The ngen queue survives a reboot, and proceeds only if the system is
otherwise IDLE - so on an Itanium 1 system, it sometimes took about a
day until it caught up.
   
You're right of course. I remember terrible problems at Resolver Systems 
trying to make an installer behave as if ngen was synchronous - because 
for compiled IronPython applications the loading time is about 3 times 
longer if ngen'ing hasn't completed. Wix 2 would let us call ngen 
synchronously (or somehow pretend it was synchronous) but with Wix 3 we 
had to give up and just warn the user that first run could sometimes 
take a 'bit' longer to start...


Michael



Regards,
Martin
   



--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of 
your employer, to release me from all obligations and waivers arising from any 
and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, 
clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and 
acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your 
employer, its partners, licensors, agents and assigns, in perpetuity, without 
prejudice to my ongoing rights and privileges. You further represent that you 
have the authority to release me from any BOGUS AGREEMENTS on behalf of your 
employer.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Enhancing the shutil module

2010-01-22 Thread Sridhar Ratnakumar 

On 1/22/2010 2:44 PM, Tarek Ziadé wrote:

On Fri, Jan 22, 2010 at 11:17 PM, Sridhar Ratnakumar
  wrote:
[..]

>  3) Patch Lib/tarfile.py to fix issue6196
>
>  I am hoping that (1) and (2) will get accepted. But not (3) - in which case,
>  should this go as a workarond (_ensure_read_write_access in
>  http://gist.github.com/279606) in the new, say, shutil.extract function?

I agree with Lars conclusions on this issue, I don't think this
permission feature fits the sdtlib because its a very specific use
case.


Do you think anyone would want to use `shutil.unpack_archive` but 
*retain* the u-x,u-r permissions? I presume that one would - in most 
cases - want to *process* the unpacked archive .. especially when doing 
all this by writing a script.


To give an example, what if when Distribute uses `shutil.unpack_archive` 
to unpack a sdist from PyPI (the author generated the archive with 
u-r,u-x set on files/directoreis -- I've seen this happening before), 
the subsequent "python setup.py install" would fail due to permission issue?



Now, for shutil.unpack_archive, we could add a callable option (like
the progress_filter one)
that would be called with each newly extracted file, so one may apply
any process on it.


How about having an extra argument that would fix the permission? 
(Fixing the permission is only applicable for tarfile, not zipfile, 
hence even our callable will become specific to tarfile).


  >>> shutil.unpack_archive("/tmp/foo.tgz", "/tmp", make_readable=True)
  >>> help(shutil.unpack_archive)
  [...]
  If `make_readable` is True, then files with limited permission (cannot
  be read) will be fixed to make them readable by the current user.

-srid
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposed downstream change to site.py in Fedora (sys.defaultencoding)

2010-01-22 Thread Scott Dial 
On 1/22/2010 2:04 PM, M.-A. Lemburg wrote:
> Karen Tracey wrote:
>> On Fri, Jan 22, 2010 at 7:38 AM, Michael Foord 
>> wrote:
>>> The encoding I'm talking about is the
>>> encoding that Python uses to decode a file (or encode a string) when you do
>>> the following in Python 3:
>>>
>>>text = open(filename).read()
>>>open(filename, 'w').write(some_string)
>>>
>>> It isn't the default encoding (always utf-8 by default in Python 3
>>> apparently), it isn't the file system encoding which is the system encoding
>>> used for file names. What is the correct terminology for this platform
>>> dependent encoding that Python uses here?
>>>
>>>
>> The doc here:
>> http://docs.python.org/3.1/library/functions.html?highlight=open#open just
>> calls it default encoding and clarifies that is "whatever
>> locale.getpreferredencoding() returns".
> 
> ... which is a pretty poor guess, since the locale setting
> is usually set to what the user wants to see in user interfaces,
> not what the application want to use as file content.
[...]
> Applications should always provide a well-defined encoding
> for use with files - they know best what encoding to use and
> they also know best when to apply guessing and which set
> of encodings to use as basis for such guessing.

This all begs the question: why is there a default? and why is the
default a guess?

I have to admit that I was completely oblivious to this potential
pitfall, and mostly that's because in the most common case, I am working
with ASCII files. It's just serendipity that most systems specify (if
not require) the locale encoding be an ASCII superset.

"""
locale.getpreferredencoding([do_setlocale])

Return the encoding used for text data, according to user
preferences. User preferences are expressed differently on different
systems, and might not be available programmatically on some systems, so
this function only returns a guess.
"""

I already know that this suggestion will not get any following because,
for most people, it just works. However: "In the face of ambiguity,
refuse the temptation to guess." Would it really be that unfortunate to
force everyone to reconsider what they are doing when they open() files?

-- 
Scott Dial
[email protected]
[email protected]
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Enhancing the shutil module

2010-01-22 Thread Tarek Ziadé 
[...]
> To give an example, what if when Distribute uses `shutil.unpack_archive` to
> unpack a sdist from PyPI (the author generated the archive with u-r,u-x set
> on files/directoreis -- I've seen this happening before), the subsequent
> "python setup.py install" would fail due to permission issue?

That's an issue to be fixed by the author of the archive imho.  IOW,
if someone produces an archive that can't be used as-is, fixing it
yourself afterwards can be a good workaround I guess in the
ActiveState packaging system, but it's still a workaround on a
specific bug introduced by the author (not by the tar system)

[..]
> How about having an extra argument that would fix the permission? (Fixing
> the permission is only applicable for tarfile, not zipfile, hence even our
> callable will become specific to tarfile).
>
>  >>> shutil.unpack_archive("/tmp/foo.tgz", "/tmp", make_readable=True)
>  >>> help(shutil.unpack_archive)
>  [...]
>  If `make_readable` is True, then files with limited permission (cannot
>  be read) will be fixed to make them readable by the current user.

That's still the same specific use case. Out of curiosity, how many
archives have you had the
problem with ?

If like I think, this is quite rare, I am -1 to introduce it that way
because you could handle with a generic callable as mentioned.

Regards
Tarek

-- 
Tarek Ziadé | http://ziade.org
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Enhancing the shutil module

2010-01-22 Thread Sridhar Ratnakumar 

On 1/22/2010 3:19 PM, Tarek Ziadé wrote:

[..]

>  How about having an extra argument that would fix the permission? (Fixing
>  the permission is only applicable for tarfile, not zipfile, hence even our
>  callable will become specific to tarfile).
>
>>>>  shutil.unpack_archive("/tmp/foo.tgz", "/tmp", make_readable=True)
>>>>  help(shutil.unpack_archive)
>[...]
>If `make_readable` is True, then files with limited permission (cannot
>be read) will be fixed to make them readable by the current user.

That's still the same specific use case. Out of curiosity, how many
archives have you had the
problem with ?

If like I think, this is quite rare, I am -1 to introduce it that way
because you could handle with a generic callable as mentioned.


Yes, it is quite rare (I can recall maybe 2-4 packages having this 
problem). I see what you mean now. Since it is not a popular issue, 
adding this to stdlib may not be a good idea.


On 1/22/2010 2:44 PM, Tarek Ziadé wrote:
> Now, for shutil.unpack_archive, we could add a callable option (like
> the progress_filter one)
> that would be called with each newly extracted file, so one may apply
> any process on it.

Will this callable recieve TarInfo objects if the filetype is tarfile? 
What would it receive otherwise? How can `_ensure_read_write_access` be 
implemented using this callable? I cannot think of a design for this. 
(In that case, maybe then the permission setting would happen on the 
actual extracted files post-extraction).


-srid
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Enhancing the shutil module

2010-01-22 Thread Tarek Ziadé 
On Sat, Jan 23, 2010 at 1:15 AM, Sridhar Ratnakumar
 wrote:
[..]
> Will this callable recieve TarInfo objects if the filetype is tarfile? What
> would it receive otherwise? How can `_ensure_read_write_access` be
> implemented using this callable? I cannot think of a design for this. (In
> that case, maybe then the permission setting would happen on the actual
> extracted files post-extraction).

Yes, I was thinking of a post-process on the file freshly created on the system

Tarek
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Enhancing the shutil module

2010-01-22 Thread Sridhar Ratnakumar 

On 1/22/2010 4:38 PM, Tarek Ziadé wrote:

On Sat, Jan 23, 2010 at 1:15 AM, Sridhar Ratnakumar
  wrote:
[..]

>  Will this callable recieve TarInfo objects if the filetype is tarfile? What
>  would it receive otherwise? How can `_ensure_read_write_access` be
>  implemented using this callable? I cannot think of a design for this. (In
>  that case, maybe then the permission setting would happen on the actual
>  extracted files post-extraction).

Yes, I was thinking of a post-process on the file freshly created on the system


Or use os.walk. :)

-srid
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposed downstream change to site.py in Fedora (sys.defaultencoding)

2010-01-22 Thread Martin v. Löwis 
> This all begs the question: why is there a default? and why is the
> default a guess?
> 
> I have to admit that I was completely oblivious to this potential
> pitfall, and mostly that's because in the most common case, I am working
> with ASCII files.

You answered your own question: it is this reason why there is a
default for the IO encoding.

> It's just serendipity that most systems specify (if
> not require) the locale encoding be an ASCII superset.

No, it's not. It is deliberate that the locale's encoding is
an ASCII superset. On systems where it isn't, users are typically
well aware that they are not using ASCII. On the systems where
it is, users get completely oblivious of the entire issue.

> I already know that this suggestion will not get any following because,
> for most people, it just works. However: "In the face of ambiguity,
> refuse the temptation to guess." Would it really be that unfortunate to
> force everyone to reconsider what they are doing when they open() files?

Yes, definitely. It is this very reasoning that caused Python 2.x to
use ASCII as the default encoding (when mixing strings and unicode),
and, for the entire lifetime of 2.x, has caused endless pain for
developers, which simply fail to understand the notion of encodings
in the first place. The majority of developers is unable to get it
right, in particular if their native language is English. These
developers just hate Unicode. They google for solutions, and come
up with all kinds of proposals which are all wrong (such as reloading
the sys module to get back sys.setdefaultencoding, to then set it
to UTF-8).

So for the limited case of text IO, Python 3.x now makes a guess.
However, this guess is not in the face of ambiguity: it is the
locale that the user (or his administrator) has selected, which
identifies the language that they speak and the character encoding
they use for text. So if Python also uses that encoding, it's not
really an ambiguous guess.

Regards,
Martin

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com