Re: [Python-Dev] Summary of 2 years of Python fuzzing

[Neal Norwitz]

On Mon, Jan 25, 2010 at 2:34 PM, Victor Stinner
victor.stin...@haypocalc.com wrote:

 Interaction with the Python developers
 ==

 I open an issue for each bug found in CPython. I describe how to reproduce it
 and try to write a patch. I have learn to always write an unit test, useful to
 reproduce the bug, and it makes Python commiters happy :-)

Yes! :-)

 The reaction depends on the impacted component, the severity of the bug, the
 complexity of the code reproducing the bug, and the quality of my bug report
 :-) The answer was always quick for core components. But some modules are
 maintained by everyone, which means nobody, like imageop, audioop or
 cProfile/hotshot. Having a module maitainer, like Guilherme Polo aka gpolo for
 Tkiner, does really help!

 It looks like fuzzing bugs are not always appreciated by developers, maybe
 because they are always borderline cases (not realist).

As many others have said, I greatly appreciate your work in this area.
 Even if obscure, all of the crashes are important to fix.  Stability
has been a big selling point of Python and will likely remain so in
the future.  While it may not be important in some environments, it's
critical in others.  We should strive to do the best job we can and
Fusil makes it easier to be more robust.

 Sometimes, even if I write a patch, an unit test, explain the problem and the
 solution, I don't get any comment. It doesn't motivate me to continue fuzzing
 :-/

Yeah, I know this can be frustrating.  As Terry mentioned later in
this post, many of your patches have been committed which is very
impressive.  The outstanding ones are at most about a month old.  I
would expect most of those to move along as people find the time to
address them.

If you keep creating more high quality patches (e.g., with unit
tests), you could be on your way to becoming a Python committer which
would help solve this problem.

I definitely hope you continue to find and fix problems in Python.  It
helps everyone who uses Python even those who will never know to thank
you.  Who knows, someone might even write a book about Fusil someday
about a topic as obscure as Beautiful Testing. :-)

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Summary of 2 years of Python fuzzing

[Ben Finney]

Neal Norwitz nnorw...@gmail.com writes:

 On Mon, Jan 25, 2010 at 2:34 PM, Victor Stinner
 victor.stin...@haypocalc.com wrote:

Along with others expressed here, you have my warm thanks, Victor, for
your continuing efforts at fuzz testing Python and especially for
careful reporting of the discovered bugs.

 I definitely hope you continue to find and fix problems in Python. It
 helps everyone who uses Python even those who will never know to thank
 you. Who knows, someone might even write a book about Fusil someday
 about a topic as obscure as Beautiful Testing. :-)

The art of good testing is slowly getting the recognition and
appreciation it deserves, which we can hope will result in testing
becoming a more central part of the programming practice.

Your suggested title is already taken, though, for exactly this purpose.
The book “Beautiful Testing”, published by O'Reilly, might help
URL:http://oreilly.com/catalog/9780596159825.

-- 
 \  “When I was a little kid we had a sand box. It was a quicksand |
  `\   box. I was an only child... eventually.” —Steven Wright |
_o__)  |
Ben Finney

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[Glenn Linderman]

On approximately 1/26/2010 7:35 PM, came the following characters from 
the keyboard of David Lyon:

Glen wrote:

   

So let's say that the .zip file was dropped onto the Desktop or start
menu.  It would have an icon, then.
 

It would have an icon. But nothing to identify it as a python
application.

One problem is that in many places, users are trained specifically
under windows to *never* run anything in a zip file. As it might
contain a virus and bring down the whole company network.

That is not a culture that a lot of people want to undo

   

But having a __main__.py file in a zip file is hardly a clear
and obvious way (to outside people) that it is a python
application.
   

So let's further say that the .zip file was named .py, instead, but was
a .zip internally.
 


So this cures the icon too, maybe you realized that.



That would seem to go a long ways toward making the facility user
friendly, at least on Windows, which is where your complaint about icons
was based, and the only change to Python would be to recognize that if a
.py contains a .zip signature, then process it the same way as this
mysterious, not mentioned in the What's New .zip execution facility.
 

That is perfectly true to say that.

It is *very* clever.

So it begs the question then, why not use that as a form of application
packaging?

The only one downside (from an opinion perspective) is having an
external distribution format, like .egg, does make it easier to
see that the application is not yet installed.
   


A ZIP named .py doesn't need to be installed it can just run in 
place.  And as a couple other subthreads noted, this is already 
functional in Python 2.6/3.1 today.


Of course, if you want to create a ZIP named .py package that is an 
application installer, you could do that, too.  It might be handy for 
the case where not everything in the application can be a .py, .pyc, or 
.pyo... shared libraries cannot be run from the ZIP files as a 
documented limitation, it seems... so if those are needed, then an 
installer is needed too.  But it does seem that a ZIP name .py installer 
package could carry other payloads in the ZIP file, and those payloads 
could be extracted a places in the right places.




Double click them and they get filed in the right place

What platform files things in the right place when you double click them?
 

Windows and Linux do. When you double click an application installer. They
will go away and start the appropriate application installer. I am
refering to files like .msi's or .rpm's respectively.
   


You are now describing something different... something that you click 
on, that causes _other_ things to get filed in the right place.  But one 
has to first create the platform-specific thing to click on, in addition 
to the code you want to run.  Unless, of course, it can be a 
Python-specific-platform-independent thing.  Here the Windows technique 
of associations, also mentioned in another subthread, do make it easier 
to actually _click_ on the thing and have the right thing happen, 
whereas Unix/Linux must have the #! line, and that could vary between 
different distributions.  The command  python application_installer.py 
could be cross-platform, though.




But overall, I think this discussion is going in the right direction.

I hope Tarek can pick up on it and fnd a way to work it into distutils if
he has time as it does need to have a home to live.
   


Without reference to distutils, it seems the pieces are:

1) a way to decide what to include in the package
2) code that knows where to put what is included, on one or more platforms
3) the process to create the ZIP file that includes 1  2, and call it 
an appropriate name.py


3 looks easy, once 1  2 are figured out.  distutils might provide the 
foundation for 1.  2 sounds like something a distutils application might 
create.  I'm not sure that distutils is in the business of building 
installer programs, my understanding is that it is in the business of 
providing a standard way of recording and interpreting bills of 
materials and maybe dependencies, but that is based only on reading 
discussions here, not reading documentation.  I haven't had a chance to 
read all the module documentation since coming to python.


--
Glenn -- http://nevcal.com/
===
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Glenn Linderman]

On approximately 1/26/2010 7:50 PM, came the following characters from 
the keyboard of Cameron Simpson:

My point was that I look on python builtins like list and dict as highly
optimised, highly efficient facilities. That means that I expect a list
to be very very much like a linear array as one would find in C-like
languages, with realloc() managment behind the scenes to handle the
resizing requirements on append/extend.
   


As a newcomer to python, I must say that I wouldn't expect a list to be 
like an array.  I'd expect it more to be like a list... many 
implementations of lists (linked lists, in particular) make it O(1) to 
add to the front or back.  An array can be used to represent a list, but 
there are known inefficiencies that result when doing so, one of which 
the Subject patch is working to address.  I guess I would have expected 
something called an array to be more like an array, rather than 
something called a list.  But one has to read the documentation to find 
out what things really mean, in a new environment.


--
Glenn -- http://nevcal.com/
===
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[David Lyon]

 Glen wrote:

 So let's further say that the .zip file was named .py, instead, but was
 a .zip internally.


 So this cures the icon too, maybe you realized that.

It takes it to 80% cured.

Being the purist that I am I still long for the day when I
can see a python package in my file manager with a proper
icon. Icons only cost $400 to get done professionally at
a graphic artist. That's roughly the same as a round of
drinks at a python conference.

I'd like to see a pythonic egg icon. I know from experience
graphic artists can knock together a nice set pretty easily.

 A ZIP named .py doesn't need to be installed it can just run in
 place.  And as a couple other subthreads noted, this is already
 functional in Python 2.6/3.1 today.

It's not that I forget or didn't understand, it's that the inplace
concept doesn't sit nicely in a formal windows environment. You have
to accept my word that there is a 'tradition', started by Microsofts
'guidelines' over 20 years ago about where windows users must put their
programs.

I can tell you about places where I have contracted where the
desktops are cleaned of any inplace files. If I spent a
week doing an application, I would want it to last on a system
well into the next day..

To get around that, we simply put the programs where microsoft
tell us and the data similarly and everything is sweet.

 Of course, if you want to create a ZIP named .py package that is an
 application installer, you could do that, too.  It might be handy for
 the case where not everything in the application can be a .py, .pyc, or
 .pyo... shared libraries cannot be run from the ZIP files as a
 documented limitation, it seems... so if those are needed, then an
 installer is needed too.

Yes now I *really* *like* that idea...

 But it does seem that a ZIP name .py installer
 package could carry other payloads in the ZIP file, and those payloads
 could be extracted a places in the right places.

Yes. If the user wishes, why not. Good idea

 Double click them ...
 Here the Windows technique
 of associations, also mentioned in another subthread, do make it easier
 to actually _click_ on the thing and have the right thing happen,
 whereas Unix/Linux must have the #! line,

If you are without an X11 window manager such as gnome or kde and are
command line bound that is true. If you have Gnome/KDE then they
both give you the association capability under Linux/Unix.

 Without reference to distutils, it seems the pieces are:

 1) a way to decide what to include in the package
 2) code that knows where to put what is included, on one or more platforms
 3) the process to create the ZIP file that includes 1  2, and call it
 an appropriate name.py

Yes.

 3 looks easy, once 1  2 are figured out.  distutils might provide the
 foundation for 1.  2 sounds like something a distutils application might
 create.  I'm not sure that distutils is in the business of building
 installer programs, my understanding is that it is in the business of
 providing a standard way of recording and interpreting bills of
 materials and maybe dependencies, but that is based only on reading
 discussions here, not reading documentation.  I haven't had a chance to
 read all the module documentation since coming to python.

Well, the package building technology within python is quite old
at the moment.

According to the Pycon-convention program Tarek is talking about
doing a revamp of distutils. I can't tell you what the exact plan
is so you'll have to follow that one yourself.

In the last one year, I've done some work on coming up with a
more modern system but only have had time to do sporadic work.

Here's where I hack about..: http://bitbucket.org/djlyon/

Right now I'm kindof busy working on a package test robot
for pypi along the lines of cpants.

But I'd love to see some of this come together so you
would be more than welcome to join in and help on my
community project shown above. I have to keep my postings
to a minimum here, because (rightly) I've been told to
code and not blab, blab, blab all the time. But I'm happy
because every now and again something I say gets picked
up and put into action.

Best Regards

David





___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[Michael Foord]

On 27/01/2010 11:21, David Lyon wrote:

Glen wrote:
   
   

So let's further say that the .zip file was named .py, instead, but was
a .zip internally.

 

So this cures the icon too, maybe you realized that.
 

It takes it to 80% cured.

Being the purist that I am I still long for the day when I
can see a python package in my file manager with a proper
icon. Icons only cost $400 to get done professionally at
a graphic artist. That's roughly the same as a round of
drinks at a python conference.
   

If you install Python packages from


I'd like to see a pythonic egg icon. I know from experience
graphic artists can knock together a nice set pretty easily.

   

A ZIP named .py doesn't need to be installed it can just run in
place.  And as a couple other subthreads noted, this is already
functional in Python 2.6/3.1 today.
 

It's not that I forget or didn't understand, it's that the inplace
concept doesn't sit nicely in a formal windows environment. You have
to accept my word that there is a 'tradition', started by Microsofts
'guidelines' over 20 years ago about where windows users must put their
programs.
   


But a module or a package isn't a program. If a Python programmer wants 
to create an application that is properly 'installed' on Windows then 
the *right* thing to do is to create an installer - and that uses 
infrastructure not provided by a language, but that is built into 
Windows. Tools like Wix are used to build Windows installers, Python 
does not need to do anything specific to support this. Java, C#, C++ and 
none of the other major programming languages support this *in the 
language* because it isn't the right place to do this.



I can tell you about places where I have contracted where the
desktops are cleaned of any inplace files. If I spent a
week doing an application, I would want it to last on a system
well into the next day..

To get around that, we simply put the programs where microsoft
tell us and the data similarly and everything is sweet.

   
If the programmer is building applications for Windows then they should 
create an installer that installs into the right place. No argument 
about that. Python doesn't need to, and *shouldn't* do anything to 
accommodate that.


All the best,

Michael Foord

--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of 
your employer, to release me from all obligations and waivers arising from any 
and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, 
clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and 
acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your 
employer, its partners, licensors, agents and assigns, in perpetuity, without 
prejudice to my ongoing rights and privileges. You further represent that you 
have the authority to release me from any BOGUS AGREEMENTS on behalf of your 
employer.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[Nick Coghlan]

David Lyon wrote:
 Being the purist that I am I still long for the day when I
 can see a python package in my file manager with a proper
 icon. Icons only cost $400 to get done professionally at
 a graphic artist. That's roughly the same as a round of
 drinks at a python conference.

If a non-programmer* end user can easily tell that the applications
they're using are written in Python as opposed to (say) C++ or C# or
Java then we're doing something wrong.

It's our job as language developers to provide the tools that
application developers need to write applications and package developers
need to write packages. Packaging those applications correctly for the
myriad of potential target platforms isn't really our problem.

Having a standard way to provide the metadata that the packaging
applications need, on the other hand, is beneficial to have as part of
the standard library, since it makes it easier for an application
developer to describe their metadata once and then feed it to the
appropriate tools to create platform specific installers, but dragging
those installer creation tools themselves into the standard library is a
bad idea.

Cheers,
Nick.

*(Programmers can often pick up on toolkit and language idiosyncrasies
that let them hazard a pretty good guess as to the main languages used
in an application)

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[Nick Coghlan]

David Lyon wrote:
 Glen wrote:
 
 So let's further say that the .zip file was named .py, instead, but was
 a .zip internally.
 
 The only one thing I have to say about that is that it makes
 embedding of .py files recursive.

No it doesn't. The mechanisms involved for processing the top-level
zipfile and those for processing the .py text files within that zipfile
are completely different.

 So, it begs the question How many times can you embed a .py
 within a .py?

Zipfiles inside the outermost zipfile will be ignored no matter what you
call them.

 And then; How do you check if a .py is a .py?

The same way the interpreter does: try treating it as a sys.path entry
and see if it works. If it does, it's a zipfile or directory (you can
tell which based on the importer object you get back). If it doesn't,
it's probably a script.

 Whereas, if you just associate to a .egg or a .eag or some
 other extension suffix, it just keeps things very orthodox
 and you can have a visual queue (for users with gui-
 interfaces).

Python applications should have an icon based on the specific
application, not one based on the implementation language the author
happened to use.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyCon Keynote

[Glenn Linderman]

Yesterday, I said:

On approximately 1/25/2010 9:27 PM, came the following characters
from  the keyboard of David Lyon:

Firstly, it doesn't create create desktop shortcuts - sorry users
need those. Where do the programs go?



So let's say that the .zip file was dropped onto the Desktop or
start  menu. It would have an icon, then.


But having a __main__.py file in a zip file is hardly a clear and
obvious way (to outside people) that it is a python application.



So let's further say that the .zip file was named .py, instead, but

 was a .zip internally.


That would seem to go a long ways toward making the facility user

 friendly, at least on Windows, which is where your complaint about
 icons was based, and the only change to Python would be to recognize
 that if a .py contains a .zip signature, then process it the same way
 as this mysterious, not mentioned in the What's New .zip execution
 facility.


Actually, this doesn't require any change to Python... some 
experimentation this morning discovered that after you ZIP your 
__main__.py script (and other stuff), you can rename it to foo.py and 
say Python foo.py, and it will correctly execute from within the ZIP 
file named foo.py.




Why can't we just be like the rest of the universe and have one
icon type for packages and one icon type for applications.

Double click them and they get filed in the right place.



What platform files things in the right place when you double click
them?


This is still an open question.



--
Glenn -- http://nevcal.com/
===
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Nick Coghlan]

Cameron Simpson wrote:
 The proposed change to make pop(0) O(1) involves adding a base offset
 to the internal list implementation. Doesn't that incur a (small)
 overhead to _every_ list operation? Doesn't that weaken list as the
 as efficient as possible implementation of choice for array-like
 things?

No, the implementation is smarter than that. The existing pointer in the
PyList_* structure will continue to point to the first element of the
list while an extra pointer is added that points to the beginning of the
allocated memory. The difference between the two memory addresses will
determine the number of orphan pointer slots that exist at the
beginning of the list.

Only operations that change the length of the list will be slowed down
at all, since they will need to take the orphans into account when
deciding whether or not to resize the allocated memory.

The big practical concern is actually the memory cost of adding yet
another pointer (potentially 8 bytes of data) to every list object, even
empty ones.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[David Lyon]

 On 27/01/2010 11:21, Michael Foord wrote:

 .. If a Python programmer wants
 to create an application that is properly 'installed' on Windows then
 the *right* thing to do is to create an installer - and that uses
 infrastructure not provided by a language, but that is built into
 Windows. Tools like Wix are used to build Windows installers,

You're getting a bit confused here. Python has this capability
right now. So it is *already* part of the language and I don't
want it taken out.

Fixed up a little - yes, taken out - no.

Installers aren't built into windows.

Since Python has distutils, and it builds installers, why
shouldn't we be using that? (apart from the fact that it
is slightly broken)

 ... Java, C#, C++ and
 none of the other major programming languages support this *in the
 language* because it isn't the right place to do this.

Java is a beast. Let's not argue over that one. I've no
interest in java and what it can or can't do. It has all
sorts of stuff. Don't underestimate what it can do.

C and C++ don't because they are too old and too low level.

On the other hand, Delphi and Visual Basic and Access and
most other major application development languages do also.

Point is : Python does offer this now. Copied good or bad
from languages such as visual basic perphaps. All of those
were major languages, but perphaps not around as much right
now.

 If the programmer is building applications for Windows then they should
 create an installer that installs into the right place. No argument
 about that. Python doesn't need to, and *shouldn't* do anything to
 accommodate that.

Python is able to create the installers. It's just that as of right
now the programs that get installed don't end up in the place where
convention and Microsofts guidelines say they should go.

If you've ever used a distutils-built Python Installer, you will
know that it is actually quite good.

What lets it down, is that it is quite difficult to get the build
and collection process done in the setup.py. That is quite
difficult.

So it is all there, just slightly warped. It will be interesting
to see if there will be any windows work done in the upcoming
distutils revamp. We'll just have to wait and see.

David





___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

--- On Wed, 1/27/10, Glenn Linderman v+pyt...@g.nevcal.com wrote:
 As a newcomer to python, I must say that I wouldn't expect
 a list to be like an array.  I'd expect it more to be
 like a list... many implementations of lists (linked lists,
 in particular) make it O(1) to add to the front or
 back.  An array can be used to represent a list, but
 there are known inefficiencies that result when doing so,
 one of which the Subject patch is working to address. 
 I guess I would have expected something called an array to
 be more like an array, rather than something called a
 list.  But one has to read the documentation to find
 out what things really mean, in a new environment.
 

My concept of Python lists is that they should have at least the same 
performance characteristics as an ordinary to-do list that you make with 
pencil, paper, and an eraser.

When you complete the first task on your to-do list, you can just erase it; no 
need to recopy the whole list.   When you complete all the elements on the 
first page, throw away the paper.  As you find new tasks to complete, add them 
on the end.  When you fill up a page, get a new sheet of paper.

If you complete the first task, erase it, but then a new urgent task comes in, 
go ahead and write the new urgent task on the first line of the first page.

If, for some reason, you keep getting bombarded with tasks that are more urgent 
than the plan you originally set out for yourself, you need a more powerful 
tool than a simple pencil/paper tool list, and by analogy, you would need a 
more powerful tool than a Python list to do it electronically.

But if you are just working your way through a paper to-do list, then erasing 
elements from the top works in O(1) time with occasional compacting of paper 
when you finish a whole page of tasks.

In my mind Python's lists should have the same performance characteristics as 
the paper list (or better).


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyCon Keynote

[Xavier Morel]

On 26 Jan 2010, at 17:09 , Glenn Linderman wrote:
 
 Why can't we just be like the rest of the universe and have one
 icon type for packages and one icon type for applications.
 
 Double click them and they get filed in the right place.
 
 
 What platform files things in the right place when you double click
 them?
 
 This is still an open question.

Not quite platform-level, but I recall this coming up in the indie mac 
development scene a few months ago [1]: Mac apps are usually a single .app 
package (a file tree which looks like a single file) which is installed by 
copying it into /Applications. But that isn't mandatory by any mean, and 
apparently new users to the platform forget that step and just launch the 
application from the download folder (if the application was in a zip and got 
unzipped) or from the .dmg archive (which is basically a mounted read-only disk 
image). Or they drop the application in the doc itself, which creates a 
shortcut to the content of the dmg or to the unzipped application in 
~/downloads.

Neither of these are very good, though they do work (generally) the former 
makes a mess out of the download folder, and the latter requires having 
thousands of dmgs open which is a pain (and the user has to keep the dmg 
files). So in the line of Sparkle, for those and other reasons, some 
applications started asking if they should move themselves to the Applications 
folder when launched from out of it. Two of these are Delicious Library and The 
Hit List [2] whose developers open-sourced the application-moving code [3].

1: http://daringfireball.net/2009/09/how_should_mac_apps_be_distributed
2: http://www.potionfactory.com/node/251
3: http://github.com/potionfactory/LetsMove/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

--- On Tue, 1/26/10, Cameron Simpson c...@zip.com.au wrote:

 From: Cameron Simpson c...@zip.com.au
 | The idea that CPython should not be improved because it
 would spoil
 | programmers strikes me as a thin, even desparate
 objection.
 
 Hey, I even used the word thin to describe my concern!
 
 My point was that I look on python builtins like list and
 dict as highly
 optimised, highly efficient facilities. That means that I
 expect a list
 to be very very much like a linear array as one would find
 in C-like
 languages, with realloc() managment behind the scenes to
 handle the
 resizing requirements on append/extend.
 
 It also means that the O() cost of operations should be
 pretty obvious.
 pop(0) obviously involves shuffling everything down one
 slot.

In my previous posting, though, I mention another plausible metaphor that 
programmers would have about lists--ordinary to-do lists where you can cross 
out or erase the first item in the list in O(1) time.
 
 
 The proposed change to make pop(0) O(1) involves adding a
 base offset
 to the internal list implementation. Doesn't that incur a
 (small)
 overhead to _every_ list operation? Doesn't that weaken
 list as the
 as efficient as possible implementation of choice for
 array-like
 things?


The patch advances the pointer itself during prepops, so accesses continue to 
work as quickly as before.   The offset between the originally allocated 
pointer and the pointer to the new first element of the list only gets 
calculated and used during list_resize and list_dealloc.  List_resize gets 
called by any operation that changes the size of the list, including inserts, 
deletes, pops, etc.  

Because I have to increase the size of PyListObject, you could argue that I 
even affect the performance of access to the extent that the greater size of 
PyListObject increases the likelihood of cache misses.  I would consider that 
also to be a very thin objection, although not completely implausible. 

 
 Yes, optimisation is nice. Higher seed is nice. But there
 are
 optimisations that simple fix inefficiencies or provide a
 special case
 for a common operation, and there are those with side
 effects elsewhere
 in the implementation (i.e. the base offset this pop(0) opt
 would incur,
 which adds a (small) cost to everything).


Your general point about optimizations having tradeoffs is valid, but I am not 
making the particular tradeoff that you mention.
 
 The other aspect, mentioned elsewhere in this thread, is
 that
 programmers should know the O() cost of these operations.
 

The tutorial does mention that the current list implementation has to move all 
the elements forward when you remove off the top.

(Cameron: Quick aside, for some reason your emails keep going to my spam 
folder.  I am sure it's just my mail client being stupid, but I thought I'd let 
you know in case it's something on your side.)


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[David Lyon]

 Nick wrote:

 The only one thing I have to say about that is that it makes
 embedding of .py files recursive.

 No it doesn't. The mechanisms involved for processing the top-level
 zipfile and those for processing the .py text files within that zipfile
 are completely different.

Well, if it has all been done correctly (and I believe you when you say it
has) then it's all ready to rock and roll.

 Whereas, if you just associate to a .egg or a .eag or some
 other extension suffix, it just keeps things very orthodox
 and you can have a visual queue (for users with gui-
 interfaces).

 Python applications should have an icon based on the specific
 application, not one based on the implementation language the author
 happened to use.

It is fairly rare for that to happen on an installer program. Most
users expect some sort of generic installer icon. They cover 98%
of cases.

Of course, once the application is installed, the (installed) program
then has it's own icon. But I'm not talking about that here. I think
that's what you are referring to.

What I am saying is that a Python Package deserves to have a
nice icon. Be it a library or an Application.

I'm a big fan of having 'an egg icon' because eggs are out there
in developer land. They are simple to comprehend, and if somebody
sees an egg icon, they are going to instantly make an association
back to the fact that it belongs to the python language.

Having done all this, you get shell support which allows the
user to install the package simply by right-mouse or
double-clicking.

We even get to install a python package directly from our
browser using this scheme. I don't understand why anybody
wouldn't want that.

See a package on pypi, click to download it, get prompted
to install it. It could all be just so simple.

David



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Daniel Stutzbach]

On Wed, Jan 27, 2010 at 7:13 AM, Steve Howell showel...@yahoo.com wrote:

 My concept of Python lists is that they should have at least the same
 performance characteristics as an ordinary to-do list that you make with
 pencil, paper, and an eraser.

 When you complete the first task on your to-do list, you can just erase it;
 no need to recopy the whole list.


I don't think your analogy works, unless you recopy your to-do lists
whenever you complete a task in the middle of the list. ;-)

--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC http://stutzbachenterprises.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[Nick Coghlan]

David Lyon wrote:
 Installers aren't built into windows.

Umm, yeah they are. msi's don't install themselves.

 Since Python has distutils, and it builds installers, why
 shouldn't we be using that? (apart from the fact that it
 is slightly broken)

I believe you're confused about what distutils is for. It generates
platform independent metadata that can be used to create platform
specific installers. It also generates some platform independent Python
specific installation formats that are useful for developers, but don't
really play well with the native platform installation mechanisms.

Yes, there are some bdist_* commands that let you build something
vaguely resembling a platform specific installer, but those are so far
from being full fledged installer development tools it really isn't funny.

 What lets it down, is that it is quite difficult to get the build
 and collection process done in the setup.py. That is quite
 difficult.
 
 So it is all there, just slightly warped. It will be interesting
 to see if there will be any windows work done in the upcoming
 distutils revamp. We'll just have to wait and see.

Perhaps we should take it out and make it an external project rather
than having something that only kind-of-sort-of almost works in the
standard library.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

--- On Wed, 1/27/10, Daniel Stutzbach dan...@stutzbachenterprises.com wrote:

 From: Daniel Stutzbach dan...@stutzbachenterprises.com
 Subject: Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
 To: Steve Howell showel...@yahoo.com
 Cc: python-dev@python.org
 Date: Wednesday, January 27, 2010, 5:32 AM
 On Wed, Jan 27,
 2010 at 7:13 AM, Steve Howell showel...@yahoo.com
 wrote:
 
 My concept of Python lists is that they should have at
 least the same performance characteristics as an ordinary
 to-do list that you make with pencil, paper, and an eraser.
 
 
 
 When you complete the first task on your to-do list, you
 can just erase it; no need to recopy the whole list.   
 
 I don't think your analogy works, unless you recopy
 your to-do lists whenever you complete a task in the middle
 of the list. ;-)
 

The bunch of stickies on my desk, and scribbled notes on the back of envelopes, 
etc. does indeed suggest a jumbled data structure that I would never want to 
reproduce electronically! :)  


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Stefan Behnel]

Glenn Linderman, 27.01.2010 10:13:
 As a newcomer to python, I must say that I wouldn't expect a list to be
 like an array.  I'd expect it more to be like a list... many
 implementations of lists (linked lists, in particular) make it O(1) to
 add to the front or back.  An array can be used to represent a list, but
 there are known inefficiencies that result when doing so

Performance-wise, there are a lot more inefficiencies in linked lists for
the most common use cases than in arrays. It's hinting to see how common the

List l = new ArrayList()

idiom is in Java (plus generics, obviously). I can't remember seeing any
other kind of initialisation in ages.

That's a huge difference between Java and Python, BTW. Python optimises for
common use cases to keep you from thinking too much about implementation
details, whereas Java just leaves you alone with all possible solutions for
all possible use cases and forces you to choose the right one at each
single code line you write.

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Antoine Pitrou]

Nick Coghlan ncoghlan at gmail.com writes:
 
 The big practical concern is actually the memory cost of adding yet
 another pointer (potentially 8 bytes of data) to every list object, even
 empty ones.

It needn't be, actually. Why?
You only need to store this other pointer when you have an orphaned area in the
first place. But then you can store the pointer *in* the orphaned area :-)

So let's say for example:
- when ob_size = 0, there are no orphans
- when ob_size  0, the pointer to the orphaned area is given by 
  (PyObject **) self-ob_items[-1]

This makes a couple of micro-operations slightly slower (you need to take
ob_size's absolute value to get the length (*)); and it kills code which uses
Py_SIZE() on lists. But PyList_GET_SIZE() exists for a reason.

(*) or you could use ob_size's MSB, and then a simple binary AND gets you the
length; or even ob_items's LSB, since the pointer will always be aligned on more
than 1 byte

(and, yes, this is a bit insane)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Antoine Pitrou]

Daniel Stutzbach daniel at stutzbachenterprises.com writes:
 
 
 I don't think your analogy works, unless you recopy your to-do lists whenever
 you complete a task in the middle of the list. 

I think his analogy suggests that his to-do list is a doubly-linked list ;)
Or perhaps an array with lazy deletion (so that he loses O(1) random access).

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[Lennart Regebro]

On Wed, Jan 27, 2010 at 14:23, David Lyon david.l...@pythontest.org wrote:
 We even get to install a python package directly from our
 browser using this scheme. I don't understand why anybody
 wouldn't want that.

So, go ahead, make it happen. No one is stopping you.
-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[Michael Foord]

On 27/01/2010 13:04, David Lyon wrote:

On 27/01/2010 11:21, Michael Foord wrote:
 
   

.. If a Python programmer wants
to create an application that is properly 'installed' on Windows then
the *right* thing to do is to create an installer - and that uses
infrastructure not provided by a language, but that is built into
Windows. Tools like Wix are used to build Windows installers,
 

You're getting a bit confused here. Python has this capability
right now. So it is *already* part of the language and I don't
want it taken out.
   


No - you're confusing python package installation (should be installed 
into the Python system) with application installation (should be 
installed into Program Files).


Distutils supports the former, it doesn't (and shouldn't) support the 
latter. I think a bit part of the problem with your discussion of 
package management on Windows is that you don't separate these two 
*very* different use cases.



Fixed up a little - yes, taken out - no.

Installers aren't built into windows.

   
The infrastructure for building and using msi installers are part of 
Windows and the Windows development environment. For example:


http://msdn.microsoft.com/en-us/library/aa370834%28VS.85%29.aspx



Since Python has distutils, and it builds installers, why
shouldn't we be using that? (apart from the fact that it
is slightly broken)

   


Because this is the wrong way to build installers for Windows 
applications (at least not what distutils is intended for). Have you 
ever built and distributed Windows applications?


Michael

--
http://www.ironpythoninaction.com/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[William Dode]

Hi (as a simple user),

I'd like to know why you didn't followed the same way as V8 Javascript, 
or the opposite, why for V8 they didn't choose llvm ?

I imagine that startup time and memory was also critical for V8.

thanks


-- 
William Dodé - http://flibuste.net
Informaticien Indépendant

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

--- On Tue, 1/26/10, Guido van Rossum gu...@python.org wrote:

 From: Guido van Rossum gu...@python.org
 Subject: Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
 To: Steve Howell showel...@yahoo.com
 Cc: Nick Coghlan ncogh...@gmail.com, python-dev@python.org
 Date: Tuesday, January 26, 2010, 12:57 PM
 On Tue, Jan 26, 2010 at 12:46 PM,
 Steve Howell showel...@yahoo.com
 wrote:
  It seems to me that the goal of keeping lists
 super-compact from a memory standpoint is contradicted by
 the code in list_resize that optimistically preallocates
 extra memory on appends.
 
 Ah, but that applies for *large* lists. Adding 8 bytes to
 each list
 object applies to *all* lists. I betcha that many programs
 use many
 tiny lists.
 

I think that some of the large programs that you mention with many tiny lists 
have some subset of lists still in memory only due to the fact that they cannot 
be garbage collected due to dangling references.

One of the ways to eliminate dangling references is to aggressively delete 
objects after you are done processing them.

Right now the Python programmer looking to aggressively delete elements from 
the top of a list has to consider the tradeoff that the operation takes O(N) 
time and would possibly churn his memory caches with the O(N) memmove 
operation.  In some cases, the Python programmer would only have himself to 
blame for not using a deque in the first place.  But maybe he's a maintenance 
programmer, so it's not his fault, and maybe the code he inherits uses lists in 
a pervasive way that makes it hard to swap in deque after the fact.  What 
advice do you give him?

Of course, this scenario is mostly speculative.  A concrete example of a large 
Python program that keeps lots of tiny lists around would probably shed more 
light on the matter.



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[David Lyon]

Hi Nick,

 I believe you're confused about what distutils is for.

Tell me..

 It generates
 platform independent metadata that can be used to create platform
 specific installers. It also generates some platform independent Python
 specific installation formats that are useful for developers, but don't
 really play well with the native platform installation mechanisms.

Well I can understand that.. no problem.

 Yes, there are some bdist_* commands that let you build something
 vaguely resembling a platform specific installer, but those are so far
 from being full fledged installer development tools it really isn't funny.

I think we've strayed from the topic in so far as there still is
a need in many cases to pack an application into a zip box and
ship it out.

Think how useful it would be to ship Django or Plone from a zip box...

 Perhaps we should take it out and make it an external project rather
 than having something that only kind-of-sort-of almost works in the
 standard library.

My $20 says not enough good PEPs, developers, directions or decisions.

I doubt distutils will be replaced. Too much time and effort has been
spent on bandaids to keep it running.

Pypi has some 8700+ packages that depend on wholly on a bandaid supply.

Nobody clever goes and pulls a bandage off a mummy until they're sure
that the head and legs won't come off too at the same time. The
situation is that the information neccessary to make a proper
decision really isn't at hand.

For me, I would want to do a lot more analysis of pypi packages
and what their structural shape is before going to the next step.

Seriously Nick, you can't stop people putting on more bandaids on
distutils.

More importantly, there are too many people out there who would
notice if a big change was made..

These are the major reasons why significant direction change is
difficult. Lack of information on the state of pypi packages.

Hope that answers

David





David


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[P.J. Eby]

At 01:08 AM 1/27/2010 -0800, Glenn Linderman wrote:
Of course, if you want to create a ZIP named .py package that is an 
application installer, you could do that, too.  It might be handy 
for the case where not everything in the application can be a .py, 
.pyc, or .pyo... shared libraries cannot be run from the ZIP files 
as a documented limitation, it seems... so if those are needed, then 
an installer is needed too.


Or you just include a .egg subdirectory in the zipfile (using 
easy_install --always-unzip), and throw the pkg_resources module in 
the root of the zipfile.  The shared libraries are then transparently 
unzipped to a cache directory at import time.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[Lennart Regebro]

On Wed, Jan 27, 2010 at 16:24, David Lyon david.l...@pythontest.org wrote:
 Think how useful it would be to ship Django or Plone from a zip box...

Plone has installers, and is very easily installed on Windows, for
those who want to try it.

For real production installations you don't want that kind of
installations for Plone or Django. And since Django is not an end-user
product I fail to see how that would be useful to have an installer
like that for it.

 I doubt distutils will be replaced. Too much time and effort has been
 spent on bandaids to keep it running.

Somehow everything you ever write ends up saying that distutils should
be scrapped and something else should be rewritten instead. Also, you
are apparently the only one in the world who understands how it should
be made.

OK. Fine. Lets say that you are right.

The get going, stop spending time on complaining and implement it the
right way instead. Honestly.

Less talk, more hockey.
-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

--- On Wed, 1/27/10, John Arbash Meinel john.arbash.mei...@gmail.com wrote:

 From: John Arbash Meinel john.arbash.mei...@gmail.com
 Subject: Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
 To: Steve Howell showel...@yahoo.com
 Cc: Guido van Rossum gu...@python.org, Nick Coghlan 
 ncogh...@gmail.com, python-dev@python.org
 Date: Wednesday, January 27, 2010, 7:45 AM
 
  Right now the Python programmer looking to
 aggressively delete elements from the top of a list has to
 consider the tradeoff that the operation takes O(N) time and
 would possibly churn his memory caches with the O(N) memmove
 operation.  In some cases, the Python programmer would
 only have himself to blame for not using a deque in the
 first place.  But maybe he's a maintenance programmer,
 so it's not his fault, and maybe the code he inherits uses
 lists in a pervasive way that makes it hard to swap in deque
 after the fact.  What advice do you give him?
  
 
 Or he could just set them to None.

Fair enough, but that's still wasteful of memory, keeping around a bunch of 
None elements because you can't inexpensively delete them.  I concede that you 
can break the dangling references, though, and that's often where large 
programs waste a lot of memory, so your point is well taken.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)

[David Lyon]

 On 27/01/2010 13:04, Michael Foord wrote:

 Installers aren't built into windows.


 The infrastructure for building and using msi installers are part of
 Windows and the Windows development environment. For example:

  http://msdn.microsoft.com/en-us/library/aa370834%28VS.85%29.aspx

Michael, of course.

And you don't get msdn or the windows-sdk in the retail version of Windows.

It has to be installed separately, over the top of windows

Therefore, the ability to generate installers is *not* built in
to the 'standard' windows versions as they ship out to the world.

ok - windows has some installers on the CD. Printer drivers
and device-drivers. In the i386 directory there's an installer
to install windows from the CD...

so if you including those then I guess you got me...

yes, the windows CD has (device and a windows) installers..

As for being able to generate installers with Retail-Windows,
it's not possible till you download the SDK and some extra
tools. Where I work, you couldn't install them because you
worldn't have the privelages. Then you'd see I'm right..

:-)

 Since Python has distutils, and it builds installers, why
 shouldn't we be using that? (apart from the fact that it
 is slightly broken)

 Because this is the wrong way to build installers for Windows
 applications (at least not what distutils is intended for).

Whatever.. I give up defending distutils..

 Have you ever built and distributed Windows applications?

I do it as a day job.

I started at Windows 2.0 and Visual Basic 1 about 20 years
ago.Then with other different programming languages. C,
VB, Delphi, TCL, Perl, Java and Python.

The install tools that I know are NSIS, Wise, InnoSetup
and perphaps a few others if I thought back.

So when it comes to installation tools, I've done enough
to know that sticking a python app in a zip box is good
and that without the Microsoft SDK or some derivative
tool, your chances of building a windows installer
are pretty low.

Take care

David



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Daniel Stutzbach]

On Wed, Jan 27, 2010 at 9:55 AM, Steve Howell showel...@yahoo.com wrote:

 Fair enough, but that's still wasteful of memory, keeping around a bunch of
 None elements because you can't inexpensively delete them.


Even if there are many references to it, there is only one None element.

--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC http://stutzbachenterprises.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Summary of 2 years of Python fuzzing

[Chris Bergstresser]

On Wed, Jan 27, 2010 at 2:54 AM, Ben Finney ben+pyt...@benfinney.id.au wrote:
 Neal Norwitz nnorw...@gmail.com writes:
 I definitely hope you continue to find and fix problems in Python. It
 helps everyone who uses Python even those who will never know to thank
 you. Who knows, someone might even write a book about Fusil someday
 about a topic as obscure as Beautiful Testing. :-)

 Your suggested title is already taken, though, for exactly this purpose.
 The book “Beautiful Testing”, published by O'Reilly, might help
 URL:http://oreilly.com/catalog/9780596159825.

   I suspect Neal already knows that, since he cowrote chapter 9
Beautiful is Better than Ugly.

-- Chris
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Virgil Dupras]

On Wed, Jan 27, 2010 at 4:55 PM, Steve Howell showel...@yahoo.com wrote:
 --- On Wed, 1/27/10, John Arbash Meinel john.arbash.mei...@gmail.com wrote:

 From: John Arbash Meinel john.arbash.mei...@gmail.com
 Subject: Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
 To: Steve Howell showel...@yahoo.com
 Cc: Guido van Rossum gu...@python.org, Nick Coghlan 
 ncogh...@gmail.com, python-dev@python.org
 Date: Wednesday, January 27, 2010, 7:45 AM

  Right now the Python programmer looking to
 aggressively delete elements from the top of a list has to
 consider the tradeoff that the operation takes O(N) time and
 would possibly churn his memory caches with the O(N) memmove
 operation.  In some cases, the Python programmer would
 only have himself to blame for not using a deque in the
 first place.  But maybe he's a maintenance programmer,
 so it's not his fault, and maybe the code he inherits uses
 lists in a pervasive way that makes it hard to swap in deque
 after the fact.  What advice do you give him?
 

 Or he could just set them to None.

 Fair enough, but that's still wasteful of memory, keeping around a bunch of 
 None elements because you can't inexpensively delete them.  I concede that 
 you can break the dangling references, though, and that's often where large 
 programs waste a lot of memory, so your point is well taken.


Why is this thread still going? It seems to me that the case for this
change is very weak. Lists, like dicts and tuples, are used
*everywhere* and in *vast* quantities. Making them grow by 4 or 8
bytes each for such a corner case can't be an option.

I'm sure your new list class has a lot of uses, but it should be an
external class. If it stays close in behavior to the lists' behavior,
then we could even add a notice in pop()'s documentation that
recommends to use your new class in case they want a painless way to
replace list usage (to make the life of those poor developers
maintaining other people's code easier), maybe even add it in stdlib's
collections unit.

--
Virgil Dupras
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[David Malcolm]

On Thu, 2010-01-21 at 14:46 -0800, Jeffrey Yasskin wrote:
 On Thu, Jan 21, 2010 at 10:09 AM, Hanno Schlichting ha...@hannosch.eu wrote:
  I'm a relative outsider to core development (I'm just a Plone release
  manager), but'll allow myself a couple of questions. Feel free to
  ignore them, if you think they are not relevant at this point :-) I'd
  note that I'm generally enthusiastic and supportive of the proposal :)
  As a data point, I can add that all tests of Zope 2.12 / Plone 4.0 and
  their dependency set run fine under Unladen Swallow.
 
 Hi, thanks for checking that!
 
  On Wed, Jan 20, 2010 at 11:27 PM, Collin Winter collinwin...@google.com 
  wrote:
  We have chosen to reuse a set of existing compiler libraries called LLVM
  [#llvm]_ for code generation and code optimization.
 
  Would it be prudent to ask for more information about the llvm
  project? Especially in terms of its non-code related aspects. I can
  try to hunt down this information myself, but as a complete outsider
  to the llvm project this takes much longer, compared to someone who
  has interacted with the project as closely as you have.
 
  Questions like:
 

[snip]

  Managing LLVM Releases, C++ API Changes
  ---
 
  LLVM is released regularly every six months. This means that LLVM may be
  released two or three times during the course of development of a CPython 
  3.x
  release. Each LLVM release brings newer and more powerful optimizations,
  improved platform support and more sophisticated code generation.
 
  How does the support and maintenance policy of llvm releases look
  like? If a Python version is pegged to a specific llvm release, it
  needs to be able to rely on critical bug fixes and security fixes to
  be made for that release for a rather prolonged time. How does this
  match the llvm policies given their frequent time based releases?
 
 LLVM doesn't currently do dot releases. So, once 2.7 is released,
 it's very unlikely there would be a 2.6.1. They do make release
 branches, and they've said they're open to dot releases if someone
 else does them, so if we need a patch release for some issue we could
 make it ourselves. I recognize that's not ideal, but I also expect
 that we'll be able to work around LLVM bugs with changes in Python,
 rather than needing to change LLVM.

[snip]

(I don't think the following has specifically been asked yet, though
this thread has become large)

As a downstream distributor of Python, a major pain point for me is when
Python embeds a copy of a library's source code, rather than linking
against a system library (zlib, libffi and expat spring to mind): if
bugs (e.g. security issues) arise in a library, I have to go chasing
down all of the embedded copies of the library, rather than having
dynamic linking deal with it for me.

So I have some concerns about having a copy of LLVM embedded in Python's
source tree, which I believe other distributors of Python would echo;
our rough preference ordering is:

   dynamic linking  static linking  source code copy

I would like CPython to be faster, and if it means dynamically linking
against the system LLVM, that's probably OK (though I have some C++
concerns already discussed elsewhere in this thread).  If it means
statically linking, or worse, having a separate copy of the LLVM source
as an implementation detail of CPython, that would be painful.

I see that the u-s developers have been run into issues in LLVM itself,
and fixed them (bravo!), and seem to have done a good job of sending
those fixes back to LLVM for inclusion. [1]

Some questions for the U-S devs:
  - will it be possible to dynamically link against the system LLVM?
(the PEP currently seems to speak of statically linking against it)
  - does the PEP anticipate that the Python source tree will start
embedding a copy of the LLVM source tree?
  - if so, what can be done to mitigate the risk of drift from upstream?
(this is the motivation for some of the following questions)
  - to what extent do you anticipate further changes needed in LLVM for
U-S? (given the work you've already put in, I expect the answer is
probably a lot, but we can't know what those will be yet)
  - do you anticipate all of these changes being accepted by the
upstream LLVM maintainers?
  - to what extent would these changes be likely to break API and/or ABI
compat with other users of LLVM (i.e. would a downstream distributor of
CPython be able to simply apply the necessary patches to the system LLVM
in order to track? if they did so, would it require a recompilation of
all of the other users of the system LLVM?)
  - if Python needed to make a dot-release of LLVM, would LLVM allow
Python to increment the SONAME version identifying the ABI within the
DSO (.so) files, and guarantee not to reuse that SONAME version? (so
that automated ABI dependency tracking in e.g. RPM can identify the ABI
incompatibilities without being stomped on by a future upstream LLVM
release)
  - 

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

--- On Wed, 1/27/10, Daniel Stutzbach dan...@stutzbachenterprises.com wrote:

 From: Daniel Stutzbach dan...@stutzbachenterprises.com
 Subject: Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
 To: Steve Howell showel...@yahoo.com
 Cc: John Arbash Meinel john.arbash.mei...@gmail.com, python-dev@python.org
 Date: Wednesday, January 27, 2010, 8:20 AM
 On Wed, Jan 27,
 2010 at 9:55 AM, Steve Howell showel...@yahoo.com
 wrote:
 
 
 Fair enough, but that's still wasteful of memory,
 keeping around a bunch of None elements because you
 can't inexpensively delete them.  
 
 Even if there are many references to it, there is only one
 None element.
 

I should have been more precise and said the pointers to None, which could be 
reclaimed.  But that's a pretty minor savings--I concede on the greater point, 
you do have the alternative to break dangling references with None, so the 
expense of avoiding remove operations is only local to list itself.






___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

--- On Wed, 1/27/10, Virgil Dupras hs...@hardcoded.net wrote:
 
 Why is this thread still going? It seems to me that the
 case for this
 change is very weak. Lists, like dicts and tuples, are
 used
 *everywhere* and in *vast* quantities. Making them grow by
 4 or 8
 bytes each for such a corner case can't be an option.
 
 I'm sure your new list class has a lot of uses, but it
 should be an
 external class. If it stays close in behavior to the lists'
 behavior,
 then we could even add a notice in pop()'s documentation
 that
 recommends to use your new class in case they want a
 painless way to
 replace list usage (to make the life of those poor
 developers
 maintaining other people's code easier), maybe even add it
 in stdlib's
 collections unit.
 


Lists are indeed used *everywhere* and *vast* quantities, which is why 
optimizations on list operations are worthwhile to pursue.

The particular optimization makes a few tradeoffs, and the consensus here 
appears to be that the ugliest tradeoff is adding the pointer to PyListObject.

There is at least some irony in opposing an optimization to a remove() 
operation on the basis of compacting memory, which isn't to say that the 
argument isn't valid.

There is also the possibility that my initial patch can be refined by somebody 
smarter than myself to eliminate the particular tradeoff.  In fact, Antoine 
Pitrou already suggested an approach, although I agree that it kind of pushes 
the boundary of sanity. :)



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

Hi William,

On Wed, Jan 27, 2010 at 7:26 AM, William Dode w...@flibuste.net wrote:
 Hi (as a simple user),

 I'd like to know why you didn't followed the same way as V8 Javascript,
 or the opposite, why for V8 they didn't choose llvm ?

 I imagine that startup time and memory was also critical for V8.

Startup time and memory usage are arguably *more* critical for a
Javascript implementation, since if you only spend a few milliseconds
executing Javascript code, but your engine takes 10-20ms to startup,
then you've lost. Also, a minimized memory profile is important if you
plan to embed your JS engine on a mobile platform, for example, or you
need to run in a heavily-multiprocessed browser on low-memory consumer
desktops and netbooks.

Among other reasons we chose LLVM, we didn't want to write code
generators for each platform we were targeting. LLVM has done this for
us. V8, on the other hand, has to implement a new code generator for
each new platform they want to target. This is non-trivial work: it
takes a long time, has a lot of finicky details, and it greatly
increases the maintenance burden on the team. We felt that requiring
python-dev to understand code generation on multiple platforms was a
distraction from what python-dev is trying to do -- develop Python. V8
still doesn't have x86-64 code generation working on Windows
(http://code.google.com/p/v8/issues/detail?id=330), so I wouldn't
underestimate the time required for that kind of project.

Thanks,
Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[skip]

David As a downstream distributor of Python, a major pain point for me
David is when Python embeds a copy of a library's source code, rather
David than linking against a system library (zlib, libffi and expat
David spring to mind): if bugs (e.g. security issues) arise in a
David library, I have to go chasing down all of the embedded copies of
David the library, rather than having dynamic linking deal with it for
David me.

The Unladen Swallow developers can correct me if I'm wrong, but I believe
the Subversion checkout holds a copy of LLVM strictly to speed development.
If the U-S folks find and fix a bug in LLVM they can shoot the fix upstream,
apply it locally, then keep moving forward without waiting for a new release
of LLVM.  Support exists now for building with an external LLVM, and I would
expect that as Unladen Swallow moves out of active development into a more
stable phase of its existence that it will probably stop embedding LLVM.

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[William Dode]

On 27-01-2010, Collin Winter wrote:
 Hi William,

 On Wed, Jan 27, 2010 at 7:26 AM, William Dode w...@flibuste.net wrote:
 Hi (as a simple user),

 I'd like to know why you didn't followed the same way as V8 Javascript,
 or the opposite, why for V8 they didn't choose llvm ?

 I imagine that startup time and memory was also critical for V8.

 Startup time and memory usage are arguably *more* critical for a
 Javascript implementation, since if you only spend a few milliseconds
 executing Javascript code, but your engine takes 10-20ms to startup,
 then you've lost. Also, a minimized memory profile is important if you
 plan to embed your JS engine on a mobile platform, for example, or you
 need to run in a heavily-multiprocessed browser on low-memory consumer
 desktops and netbooks.

 Among other reasons we chose LLVM, we didn't want to write code
 generators for each platform we were targeting. LLVM has done this for
 us. V8, on the other hand, has to implement a new code generator for
 each new platform they want to target. This is non-trivial work: it
 takes a long time, has a lot of finicky details, and it greatly
 increases the maintenance burden on the team. We felt that requiring
 python-dev to understand code generation on multiple platforms was a
 distraction from what python-dev is trying to do -- develop Python. V8
 still doesn't have x86-64 code generation working on Windows
 (http://code.google.com/p/v8/issues/detail?id=330), so I wouldn't
 underestimate the time required for that kind of project.

Thanks for this answer.

The startup time and memory comsumption are a limitation of llvm that 
their developers plan to resolve or is it only specific to the current 
python integration ? I mean the work to correct this is more on U-S or 
on llvm ?

thanks (and sorry for my english !)

-- 
William Dodé - http://flibuste.net
Informaticien Indépendant

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

Hi David,

On Wed, Jan 27, 2010 at 8:37 AM, David Malcolm dmalc...@redhat.com wrote:
[snip]
 As a downstream distributor of Python, a major pain point for me is when
 Python embeds a copy of a library's source code, rather than linking
 against a system library (zlib, libffi and expat spring to mind): if
 bugs (e.g. security issues) arise in a library, I have to go chasing
 down all of the embedded copies of the library, rather than having
 dynamic linking deal with it for me.

 So I have some concerns about having a copy of LLVM embedded in Python's
 source tree, which I believe other distributors of Python would echo;
 our rough preference ordering is:

   dynamic linking  static linking  source code copy

 I would like CPython to be faster, and if it means dynamically linking
 against the system LLVM, that's probably OK (though I have some C++
 concerns already discussed elsewhere in this thread).  If it means
 statically linking, or worse, having a separate copy of the LLVM source
 as an implementation detail of CPython, that would be painful.

We absolutely do not want CPython to include a copy of LLVM in its
source tree. Unladen Swallow has done this to make it easier to pick
up changes to LLVM's codebase as we make them, but this is not a
viable model for CPython's long-term development. As mentioned in
http://www.python.org/dev/peps/pep-3146/#managing-llvm-releases-c-api-changes,
one of our full-time engineers is tasked with fixing all critical
issues in LLVM before LLVM's 2.7 release so that CPython can simply
use that release.

We are currently statically linking against LLVM because that's what
LLVM best supports, but that's not set in stone. We can make LLVM
better support shared linking; it's one of the open issues as to
whether this is important to python-dev
(http://www.python.org/dev/peps/pep-3146/#open-issues), and it sounds
like the answer is Yes. Our priorities will adjust accordingly.

To answer the individual bullet points:

 I see that the u-s developers have been run into issues in LLVM itself,
 and fixed them (bravo!), and seem to have done a good job of sending
 those fixes back to LLVM for inclusion. [1]

 Some questions for the U-S devs:
  - will it be possible to dynamically link against the system LLVM?
 (the PEP currently seems to speak of statically linking against it)

We currently link statically, but we will fix LLVM to better support
shared linking.

  - does the PEP anticipate that the Python source tree will start
 embedding a copy of the LLVM source tree?

No, that would be a terrible idea, and we do not endorse it.

  - to what extent do you anticipate further changes needed in LLVM for
 U-S? (given the work you've already put in, I expect the answer is
 probably a lot, but we can't know what those will be yet)

We have fixed all the critical issues that our testing has exposed and
have now moved on to nice to have items that will aid future
optimizations. LLVM 2.7 will probably be released in May, so we still
have time to fix LLVM as needed.

  - do you anticipate all of these changes being accepted by the
 upstream LLVM maintainers?

Three of the Unladen Swallow committers are also LLVM committers. Our
patches have historically been accepted enthusiastically by the LLVM
maintainers, and we believe this warm relationship will continue.

  - to what extent would these changes be likely to break API and/or ABI
 compat with other users of LLVM (i.e. would a downstream distributor of
 CPython be able to simply apply the necessary patches to the system LLVM
 in order to track? if they did so, would it require a recompilation of
 all of the other users of the system LLVM?)

As mentioned in
http://www.python.org/dev/peps/pep-3146/#managing-llvm-releases-c-api-changes,
every LLVM release introduces incompatibilities to the C++ API. Our
experience has been that these API changes are easily remedied. We
recommend that CPython depend on a single version of LLVM and not try
to track LLVM trunk.

  - if Python needed to make a dot-release of LLVM, would LLVM allow
 Python to increment the SONAME version identifying the ABI within the
 DSO (.so) files, and guarantee not to reuse that SONAME version? (so
 that automated ABI dependency tracking in e.g. RPM can identify the ABI
 incompatibilities without being stomped on by a future upstream LLVM
 release)

Nick Lewycky is better-positioned to answer that.

  - is your aim to minimize the difference between upstream LLVM and the
 U-S copy of LLVM?  (yes, probably)
  - will you publish/track the differences between upstream LLVM and the
 U-S copy of LLVM somewhere? I see that you currently do this here: [2],
 though it strikes me that the canonical representation of the LLVM
 you're using is in the embedded copy in Utils/llvm, rather than e.g. a
 recipe like grab upstream release foo and apply this short list of
 patches (a distributed SCM might help here, or a tool like jhbuild [3])

We generally use LLVM trunk verbatim, but will 

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Collin Winter]

Hi William,

On Wed, Jan 27, 2010 at 11:02 AM, William Dode w...@flibuste.net wrote:
 The startup time and memory comsumption are a limitation of llvm that
 their developers plan to resolve or is it only specific to the current
 python integration ? I mean the work to correct this is more on U-S or
 on llvm ?

Part of it is LLVM, part of it is Unladen Swallow. LLVM is very
flexible, and there's a price for that. We have also found and fixed
several cases of quadratic memory usage in LLVM optimization passes,
and there may be more of those lurking around. On the Unladen Swallow
side, there are doubtless things we can do to improve our usage of
LLVM; http://code.google.com/p/unladen-swallow/issues/detail?id=68 has
most of our work on this, and there are still more ideas to implement.

Part of the issue is that Unladen Swallow is using LLVM's JIT
infrastructure in ways that it really hasn't been used before, and so
there's a fair amount of low-hanging fruit left in LLVM that no-one
has needed to pick yet.

Thanks,
Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Martin v. Löwis]

 Lists are indeed used *everywhere* and *vast* quantities, which is
 why optimizations on list operations are worthwhile to pursue.

Unfortunately, the proposed change is a pessimisation, not an
optimisation. We probably shouldn't discuss it further, at least not
until a PEP gets written by its proponents.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Martin v. Löwis]

 In my mind Python's lists should have the same performance
 characteristics as the paper list (or better).

I think you'll have to adjust your mind then. People have
proposed various data structures that *do* work efficiently as
paper lists. So if you want a paper list, use one of them, rather
than abusing the Python list for one.

Regards,
Martin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Jeffrey Yasskin]

On Wed, Jan 27, 2010 at 11:16 AM, Collin Winter collinwin...@google.com wrote:
 We absolutely do not want CPython to include a copy of LLVM in its
 source tree. Unladen Swallow has done this to make it easier to pick
 up changes to LLVM's codebase as we make them, but this is not a
 viable model for CPython's long-term development. As mentioned in
 http://www.python.org/dev/peps/pep-3146/#managing-llvm-releases-c-api-changes,
 one of our full-time engineers is tasked with fixing all critical
 issues in LLVM before LLVM's 2.7 release so that CPython can simply
 use that release.

I'm now tracking my to-do list for LLVM 2.7 in
http://code.google.com/p/unladen-swallow/issues/detail?id=131.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

--- On Wed, 1/27/10, Antoine Pitrou solip...@pitrou.net wrote:

 From: Antoine Pitrou solip...@pitrou.net
 Subject: Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
 To: python-dev@python.org
 Date: Wednesday, January 27, 2010, 6:15 AM
 Nick Coghlan ncoghlan at
 gmail.com writes:
  
  The big practical concern is actually the memory cost
 of adding yet
  another pointer (potentially 8 bytes of data) to every
 list object, even
  empty ones.
 
 It needn't be, actually. Why?
 You only need to store this other pointer when you have an
 orphaned area in the
 first place. But then you can store the pointer *in* the
 orphaned area :-)
 
 So let's say for example:
 - when ob_size = 0, there are no orphans
 - when ob_size  0, the pointer to the orphaned area is
 given by 
   (PyObject **) self-ob_items[-1]
 
 This makes a couple of micro-operations slightly slower
 (you need to take
 ob_size's absolute value to get the length (*)); and it
 kills code which uses
 Py_SIZE() on lists. But PyList_GET_SIZE() exists for a
 reason.
 
 (*) or you could use ob_size's MSB, and then a simple
 binary AND gets you the
 length; or even ob_items's LSB, since the pointer will
 always be aligned on more
 than 1 byte
 
 (and, yes, this is a bit insane)
 

A slightly more sane alternative would be to leave ob_size and ob_item alone 
with their current semantics, and then replace allocated with self-excess, 
where

  self-excess == excess_above * 256 + excess_below.

Right now self-allocated is only used in a few places:
  
  list_resize
  listextend
  listsort
  list_init (but only in an assert)
  list_sizeof

Those places would need to compute:

  allocated = self-ob_size + self-excess / 256 + self-excess % 256

The right hand side could be done with shifting/bitmasking inside a macro.

Excess_left would obviously not be allowed to exceed 256; excess_right would 
max out at PY_SIZE_MAX / 256, instead of PY_SIZE_MAX / 8 + 6, which is probably 
the right thing anyway.

Here is the current algorithm for allocating extra bytes on the right:

new_allocated = (newsize  3) + (newsize  9 ? 3 : 6);

On a 32-bit platform, excess_right would max out at 16M instead of the current 
512M.

Any method that wanted to find the originally allocated pointer would compute

  self-ob_item - (self-excess % 256)

To the extent that most of the ugly details could be encapsulated in macros, I 
do not think this would complicate the list code itself greatly, but it would 
complicate debugging.  Guido has expressed a strong desire to keep a hard 
pointer to the originally allocated chunk, especially with regard to future 
garbage collection changes.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[David Malcolm]

On Wed, 2010-01-27 at 11:34 -0800, Jeffrey Yasskin wrote:
 On Wed, Jan 27, 2010 at 11:16 AM, Collin Winter collinwin...@google.com 
 wrote:
  We absolutely do not want CPython to include a copy of LLVM in its
  source tree. Unladen Swallow has done this to make it easier to pick
  up changes to LLVM's codebase as we make them, but this is not a
  viable model for CPython's long-term development. As mentioned in
  http://www.python.org/dev/peps/pep-3146/#managing-llvm-releases-c-api-changes,
  one of our full-time engineers is tasked with fixing all critical
  issues in LLVM before LLVM's 2.7 release so that CPython can simply
  use that release.
 
 I'm now tracking my to-do list for LLVM 2.7 in
 http://code.google.com/p/unladen-swallow/issues/detail?id=131.

Many thanks for addressing these concerns!

Dave

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Summary of 2 years of Python fuzzing

[Ben Finney]

Chris Bergstresser ch...@subtlety.com writes:

 On Wed, Jan 27, 2010 at 2:54 AM, Ben Finney ben+pyt...@benfinney.id.au 
 wrote:
  Neal Norwitz nnorw...@gmail.com writes:
  Who knows, someone might even write a book about Fusil someday
  about a topic as obscure as Beautiful Testing. :-)
 
  Your suggested title is already taken, though, for exactly this
  purpose. The book “Beautiful Testing”, published by O'Reilly, might
  help URL:http://oreilly.com/catalog/9780596159825.

I suspect Neal already knows that, since he cowrote chapter 9
 Beautiful is Better than Ugly.

Ah, good.

I did suspect that might be the case, but couldn't easily find Neal's
name associated with the book. I decided to post anyway, preferring the
information to be out there at the risk of seeming to miss the joke.

Maybe one day I'll get to read the book :-

-- 
 \“Without cultural sanction, most or all of our religious |
  `\  beliefs and rituals would fall into the domain of mental |
_o__) disturbance.” —John F. Schumaker |
Ben Finney

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

[Nick Lewycky]

On 27 January 2010 08:37, David Malcolm dmalc...@redhat.com wrote:

 On Thu, 2010-01-21 at 14:46 -0800, Jeffrey Yasskin wrote:
  On Thu, Jan 21, 2010 at 10:09 AM, Hanno Schlichting ha...@hannosch.eu
 wrote:
   I'm a relative outsider to core development (I'm just a Plone release
   manager), but'll allow myself a couple of questions. Feel free to
   ignore them, if you think they are not relevant at this point :-) I'd
   note that I'm generally enthusiastic and supportive of the proposal :)
   As a data point, I can add that all tests of Zope 2.12 / Plone 4.0 and
   their dependency set run fine under Unladen Swallow.
 
  Hi, thanks for checking that!
 
   On Wed, Jan 20, 2010 at 11:27 PM, Collin Winter 
 collinwin...@google.com wrote:
   We have chosen to reuse a set of existing compiler libraries called
 LLVM
   [#llvm]_ for code generation and code optimization.
  
   Would it be prudent to ask for more information about the llvm
   project? Especially in terms of its non-code related aspects. I can
   try to hunt down this information myself, but as a complete outsider
   to the llvm project this takes much longer, compared to someone who
   has interacted with the project as closely as you have.
  
   Questions like:
  

 [snip]

   Managing LLVM Releases, C++ API Changes
   ---
  
   LLVM is released regularly every six months. This means that LLVM may
 be
   released two or three times during the course of development of a
 CPython 3.x
   release. Each LLVM release brings newer and more powerful
 optimizations,
   improved platform support and more sophisticated code generation.
  
   How does the support and maintenance policy of llvm releases look
   like? If a Python version is pegged to a specific llvm release, it
   needs to be able to rely on critical bug fixes and security fixes to
   be made for that release for a rather prolonged time. How does this
   match the llvm policies given their frequent time based releases?
 
  LLVM doesn't currently do dot releases. So, once 2.7 is released,
  it's very unlikely there would be a 2.6.1. They do make release
  branches, and they've said they're open to dot releases if someone
  else does them, so if we need a patch release for some issue we could
  make it ourselves. I recognize that's not ideal, but I also expect
  that we'll be able to work around LLVM bugs with changes in Python,
  rather than needing to change LLVM.

 [snip]

 (I don't think the following has specifically been asked yet, though
 this thread has become large)

 As a downstream distributor of Python, a major pain point for me is when
 Python embeds a copy of a library's source code, rather than linking
 against a system library (zlib, libffi and expat spring to mind): if
 bugs (e.g. security issues) arise in a library, I have to go chasing
 down all of the embedded copies of the library, rather than having
 dynamic linking deal with it for me.

 So I have some concerns about having a copy of LLVM embedded in Python's
 source tree, which I believe other distributors of Python would echo;
 our rough preference ordering is:

   dynamic linking  static linking  source code copy

 I would like CPython to be faster, and if it means dynamically linking
 against the system LLVM, that's probably OK (though I have some C++
 concerns already discussed elsewhere in this thread).  If it means
 statically linking, or worse, having a separate copy of the LLVM source
 as an implementation detail of CPython, that would be painful.

 I see that the u-s developers have been run into issues in LLVM itself,
 and fixed them (bravo!), and seem to have done a good job of sending
 those fixes back to LLVM for inclusion. [1]


Hi David, I'm unladen-swallow's resident llvm consultant, so I'll answer
the questions I can from an llvm upstream point of view.


 Some questions for the U-S devs:
  - will it be possible to dynamically link against the system LLVM?
 (the PEP currently seems to speak of statically linking against it)


Sadly, LLVM still only offers static libraries (there are two exceptions,
but they don't fit unladen-swallow). This is a something we want but
nobody's stepped up to make it happen yet.


  - does the PEP anticipate that the Python source tree will start
 embedding a copy of the LLVM source tree?
  - if so, what can be done to mitigate the risk of drift from upstream?
 (this is the motivation for some of the following questions)


Jeff and I are working to make sure that everything unladen-swallow needs is
in LLVM 2.7 when it releases.

 - to what extent do you anticipate further changes needed in LLVM for
 U-S? (given the work you've already put in, I expect the answer is
 probably a lot, but we can't know what those will be yet)


Unladen-swallow works with upstream LLVM now. The changes we're making
include refactoring interfaces and adding new optimization features which
python would make use of in the future.

 - do you anticipate all of 

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Antoine Pitrou]

Le mercredi 27 janvier 2010 à 11:49 -0800, Steve Howell a écrit :
 A slightly more sane alternative would be to leave ob_size and ob_item
 alone with their current semantics, and then replace allocated with
 self-excess, where
 
   self-excess == excess_above * 256 + excess_below.

Or we could use allocated's sign bit in order to store the flag (of
whether there is an orphaned area or not) and then store the orphaned
pointer as ob_items[-1]. Thus we don't have to limit the magnitude of
anything. And since allocated itself isn't used in any really critical
path, it doesn't slow down anything.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Mercurial migration: help needed

[Martin v. Löwis]

This is a repost from a posting I sent last August. One of the issues
(the hg-eol extension) is largely resolved, so I'm only asking for
help on the other issue.

In this thread, I'd like to collect things that ought to be done
but where Dirkjan has indicated that he would prefer if somebody else
did it.

Item 1
--
The first item is build identification. If you want to work
on this, please either provide a patch (for trunk and/or py3k), or
(if you are a committer) create a subversion branch.

It seems that Barry and I agree that for the maintenance branches,
sys.subversion should be frozen, so we need actually two sets of
patches: one that removes sys.subversion entirely, and the other that
freezes the branch to the respective one, and freezes the subversion
revision to None.

The patch should consider what Dirkjan proposes as the branching
strategy: clones to separate 2.x and 3.x, as well as for features,
and branches with the clones for releases and maintenance (see the
PEP for details).

Anybody working on this should have good knowledge of the Python source
code, Mercurial, and either autoconf or Visual Studio (preferably both).

Regards,
Martin


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

--- On Wed, 1/27/10, Antoine Pitrou solip...@pitrou.net wrote:

 From: Antoine Pitrou solip...@pitrou.net
 Subject: Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
 To: python-dev@python.org
 Date: Wednesday, January 27, 2010, 12:41 PM
 Le mercredi 27 janvier 2010 à 11:49
 -0800, Steve Howell a écrit :
  A slightly more sane alternative would be to leave
 ob_size and ob_item
  alone with their current semantics, and then replace
 allocated with
  self-excess, where
  
    self-excess == excess_above * 256
 + excess_below.
 
 Or we could use allocated's sign bit in order to store the
 flag (of
 whether there is an orphaned area or not) and then store
 the orphaned
 pointer as ob_items[-1]. Thus we don't have to limit the
 magnitude of
 anything. And since allocated itself isn't used in any
 really critical
 path, it doesn't slow down anything.
 

Ok, I will try that technique and benchmark it, as it would address one strong 
objection to the proposal. 

I think I've summarized most of the other major objections here:

http://wiki.python.org/moin/ProposalToSpeedUpListOperations


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Raymond Hettinger]

On Jan 27, 2010, at 11:38 AM, Martin v. Löwis wrote:

 Lists are indeed used *everywhere* and *vast* quantities, which is
 why optimizations on list operations are worthwhile to pursue.
 
 Unfortunately, the proposed change is a pessimisation, not an
 optimisation. We probably shouldn't discuss it further, at least not
 until a PEP gets written by its proponents.
 

I concur (on both points).

AFAICT, the performance of the proposal:

* increases space requirements by a small fixed amount

* s.append() performance slightly degraded.

* the s.insert(0, value) case isn't helped -- still O(n)

* repeated s.pop(0) have amortized O(1) but either
   needs to waste space indefinitely (likely not what
   the programmer intended by popping off values)
   or needs to do occasional memmoves (which isn't
   as good as a deque which never moves the data).
 
* the resize performance doesn't work well with the
   memory allocator -- while a series of appends can often
   resize in-place without a need to do an O(n) memmove,
   but a series of pop(0) calls doesn't have a resize in-place
   option.

What currently unknown is the effect on third-party extensions
and debugging tools that access the structure directly.
Also, am not sure if this affects psyco or the other 
implementations such as Jython which may implement
lists in terms of existing Java containers.

ISTM, the *only* benefit is that an occasional s.pop(0)
will perform better than it does now but not as well
as a deque which never has to move data).


Raymond

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] O(1) random access to deque? (Re: patch to make list.pop(0) work in O(1) time)

[Nick Coghlan]

Steve Howell wrote:
 There is also the possibility that my initial patch can be refined by
 somebody smarter than myself to eliminate the particular tradeoff.
 In fact, Antoine Pitrou already suggested an approach, although I
 agree that it kind of pushes the boundary of sanity. :)

I'm actually wondering if you could apply some of the implementation
strategies discussed here to grant O(1) random access to arbitrary
elements of a deque.

I haven't looked at the deque code in a long time, but I believe the
memory structure is already larger than that for a basic list. Reworking
the way that extra space is used may be a more fruitful strategy than
trying to convince anyone that it is worth changing the list
implementation for this corner case.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] O(1) random access to deque? (Re: patch to make list.pop(0) work in O(1) time)

[Alex Gaynor]

On Wed, Jan 27, 2010 at 4:50 PM, Nick Coghlan ncogh...@gmail.com wrote:
 Steve Howell wrote:
 There is also the possibility that my initial patch can be refined by
 somebody smarter than myself to eliminate the particular tradeoff.
 In fact, Antoine Pitrou already suggested an approach, although I
 agree that it kind of pushes the boundary of sanity. :)

 I'm actually wondering if you could apply some of the implementation
 strategies discussed here to grant O(1) random access to arbitrary
 elements of a deque.

 I haven't looked at the deque code in a long time, but I believe the
 memory structure is already larger than that for a basic list. Reworking
 the way that extra space is used may be a more fruitful strategy than
 trying to convince anyone that it is worth changing the list
 implementation for this corner case.

 Cheers,
 Nick.

 --
 Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
 ---
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 http://mail.python.org/mailman/options/python-dev/alex.gaynor%40gmail.com


I don't see how that's possible.  The linked list is a pretty well
known data structure and arbitrary lookups are O(n) in it.  Using the
unrolled-linked-list data structure python uses you can make it faster
by a constant factor, but not O(1).  There are other structures like
skip-lists that have O(log n) arbitrary lookups though.  If someone
could make an O(1) linked-list I'd love to see it :)

Alex

-- 
I disapprove of what you say, but I will defend to the death your
right to say it. -- Voltaire
The people's good is the highest law. -- Cicero
Code can always be simpler than you think, but never as simple as you
want -- Me
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] O(1) random access to deque? (Re: patch to make list.pop(0) work in O(1) time)

[Daniel Stutzbach]

On Wed, Jan 27, 2010 at 3:50 PM, Nick Coghlan ncogh...@gmail.com wrote:

 I'm actually wondering if you could apply some of the implementation
 strategies discussed here to grant O(1) random access to arbitrary
 elements of a deque.

 I haven't looked at the deque code in a long time, but I believe the
 memory structure is already larger than that for a basic list.


The memory structure is a linked list of blocks, where each block can hold
several elements.  As a linked list, the current structure would have O(n)
random access.

A few years back, I had proposed an alternative way of implementing deque,
that would allow O(1) random access:
http://mail.python.org/pipermail/python-dev/2007-November/075242.html

It's essentially identical to Steve's proposal for list, except that I made
the array circular, so element i is at position items[(start + i) % len].
That way it doesn't have to regularly allocate and deallocate memory for an
approximately-fixed-length FIFO queue (which Steve's list will need to do).

Raymond's objections are here:
http://mail.python.org/pipermail/python-dev/2007-November/075244.html

--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC http://stutzbachenterprises.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] O(1) random access to deque? (Re: patch to make list.pop(0) work in O(1) time)

[Raymond Hettinger]

Are you free to chat about this in IRC?

I completely unconvinced that there are use cases for
wanting face random access to the i-th element of a large deque
that is changing sizes on each end.  The meaning of the
20th index changes as the data moves.  It becomes pointless
to store indices to moving targets.

I know on no real-world code that every uses this.

FWIW, deque indexing for small deques is already O(1)
and somewhat fast.  You only get O(n) degradation
(with a small contant factor) on large deques.

Is this a real need or a made-up use case by someone
hell-bent on changing a core data structure?


Raymond


On Jan 27, 2010, at 1:50 PM, Nick Coghlan wrote:

 Steve Howell wrote:
 There is also the possibility that my initial patch can be refined by
 somebody smarter than myself to eliminate the particular tradeoff.
 In fact, Antoine Pitrou already suggested an approach, although I
 agree that it kind of pushes the boundary of sanity. :)
 
 I'm actually wondering if you could apply some of the implementation
 strategies discussed here to grant O(1) random access to arbitrary
 elements of a deque.
 
 I haven't looked at the deque code in a long time, but I believe the
 memory structure is already larger than that for a basic list. Reworking
 the way that extra space is used may be a more fruitful strategy than
 trying to convince anyone that it is worth changing the list
 implementation for this corner case.
 
 Cheers,
 Nick.
 
 -- 
 Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
 ---
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 http://mail.python.org/mailman/options/python-dev/raymond.hettinger%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Daniel Stutzbach]

On Wed, Jan 27, 2010 at 3:49 PM, Raymond Hettinger 
raymond.hettin...@gmail.com wrote:

 Also, am not sure if this affects psyco or the other
 implementations such as Jython which may implement
 lists in terms of existing Java containers.


Or Unladen Swallow.  I'm -1 on mucking with any of the fundamental data
structures until the Unladen Swallow patch lands (assuming it lands).

--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC http://stutzbachenterprises.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] O(1) random access to deque? (Re: patch to make list.pop(0) work in O(1) time)

[Nick Coghlan]

Alex Gaynor wrote:
 I don't see how that's possible.  The linked list is a pretty well
 known data structure and arbitrary lookups are O(n) in it.  Using the
 unrolled-linked-list data structure python uses you can make it faster
 by a constant factor, but not O(1).  There are other structures like
 skip-lists that have O(log n) arbitrary lookups though.  If someone
 could make an O(1) linked-list I'd love to see it :)

Yeah, you're right - I was completely misremembering how deque worked.
I've come across a data structure implementation similar to the one
Steve is proposing somewhere before, but deque obviously isn't it.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

--- On Wed, 1/27/10, Raymond Hettinger raymond.hettin...@gmail.com wrote:

 From: Raymond Hettinger raymond.hettin...@gmail.com
 Subject: Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
 To: Martin v. Löwis mar...@v.loewis.de
 Cc: Steve Howell showel...@yahoo.com, python-dev@python.org
 Date: Wednesday, January 27, 2010, 1:49 PM
 
 On Jan 27, 2010, at 11:38 AM, Martin v. Löwis wrote:
 
  Lists are indeed used *everywhere* and *vast*
 quantities, which is
  why optimizations on list operations are
 worthwhile to pursue.
  
  Unfortunately, the proposed change is a pessimisation,
 not an
  optimisation. We probably shouldn't discuss it
 further, at least not
  until a PEP gets written by its proponents.
  
 
 I concur (on both points).
 
 AFAICT, the performance of the proposal:
 
 * increases space requirements by a small fixed amount
 
 * s.append() performance slightly degraded.
 
 * the s.insert(0, value) case isn't helped -- still O(n)
 
 * repeated s.pop(0) have amortized O(1) but either
    needs to waste space indefinitely (likely
 not what
    the programmer intended by popping off
 values)
    or needs to do occasional memmoves (which
 isn't
    as good as a deque which never moves the
 data).
  
 * the resize performance doesn't work well with the
    memory allocator -- while a series of
 appends can often
    resize in-place without a need to do an
 O(n) memmove,
    but a series of pop(0) calls doesn't have
 a resize in-place
    option.
 
 What currently unknown is the effect on third-party
 extensions
 and debugging tools that access the structure directly.
 Also, am not sure if this affects psyco or the other 
 implementations such as Jython which may implement
 lists in terms of existing Java containers.
 
 ISTM, the *only* benefit is that an occasional s.pop(0)
 will perform better than it does now but not as well
 as a deque which never has to move data).
 
 

I agree with most of what's said above, with a few clarifications.

The speedup is not limited to pop(0); it also considerably speeds up code like 
below:

n = 80
for i in range(n):
x = lst[:10]
del lst[:10]

Understood that it is a contrived benchmark, and real code like that could be 
replaced with a technique that Nones out the used up elements and advances a 
start pointer.  

Prepends are still O(N) for most use cases, but prepends will reclaim space at 
the front if it's there in O(1).

The biggest performance tradeoff, the extra space requirement in PyListObject, 
can be eliminated, albeit with a pretty ugly hack.

Since there is a lot of discussion about tradeoffs, I would remind folks that 
asking somebody to use a deque instead of a list also forces tradeoffs; you 
lose the comfort and familiarity of lists (not to be underestimated) as well as 
some features (you can't spell d[:10] as d[:10] for example).


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Antoine Pitrou]

Raymond Hettinger raymond.hettinger at gmail.com writes:
 
 AFAICT, the performance of the proposal:
 
 * increases space requirements by a small fixed amount

Well, given my proposal (assuming it turns out ok) it doesn't.

 * s.append() performance slightly degraded.

Why?

 * the resize performance doesn't work well with the
memory allocator -- while a series of appends can often
resize in-place without a need to do an O(n) memmove,
but a series of pop(0) calls doesn't have a resize in-place
option.

Can often resize in-place is overstated. It only works if there's a chunk of
sufficient free space just after your list's storage, which is probably not the
common case (except perhaps in microbenchmarks where the list is the latest
allocated object).

 What currently unknown is the effect on third-party extensions
 and debugging tools that access the structure directly.
 Also, am not sure if this affects psyco or the other 
 implementations such as Jython which may implement
 lists in terms of existing Java containers.

Well, if psyco wants to muck with internal implementation details, it's psyco's
problem. We shouldn't refrain from making internal changes under the pretext
that it might break psyco.
Otherwise the psyco author(s) should file a PEP for inclusion in the core
interpreter ;)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Jeffrey Yasskin]

On Wed, Jan 27, 2010 at 2:14 PM, Daniel Stutzbach
dan...@stutzbachenterprises.com wrote:
 On Wed, Jan 27, 2010 at 3:49 PM, Raymond Hettinger
 raymond.hettin...@gmail.com wrote:

 Also, am not sure if this affects psyco or the other
 implementations such as Jython which may implement
 lists in terms of existing Java containers.

 Or Unladen Swallow.  I'm -1 on mucking with any of the fundamental data
 structures until the Unladen Swallow patch lands (assuming it lands).

Don't block internal changes for our sake. For the most part, we've
already made plans to make them non-issues. In any cases we haven't,
we need to make them non-issues anyway to avoid slowing down python
development in the future.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] O(1) random access to deque? (Re: patch to make list.pop(0) work in O(1) time)

[Maxim Khitrov]

On Wed, Jan 27, 2010 at 5:06 PM, Daniel Stutzbach
dan...@stutzbachenterprises.com wrote:
 On Wed, Jan 27, 2010 at 3:50 PM, Nick Coghlan ncogh...@gmail.com wrote:

 I'm actually wondering if you could apply some of the implementation
 strategies discussed here to grant O(1) random access to arbitrary
 elements of a deque.

 I haven't looked at the deque code in a long time, but I believe the
 memory structure is already larger than that for a basic list.

 The memory structure is a linked list of blocks, where each block can hold
 several elements.  As a linked list, the current structure would have O(n)
 random access.

 A few years back, I had proposed an alternative way of implementing deque,
 that would allow O(1) random access:
 http://mail.python.org/pipermail/python-dev/2007-November/075242.html

 It's essentially identical to Steve's proposal for list, except that I made
 the array circular, so element i is at position items[(start + i) % len].
 That way it doesn't have to regularly allocate and deallocate memory for an
 approximately-fixed-length FIFO queue (which Steve's list will need to do).

 Raymond's objections are here:
 http://mail.python.org/pipermail/python-dev/2007-November/075244.html

I'm curious whether something similar could be added to python as a
separate data structure, but for use with strings (bytes in 3.x)? I
recently had to implement a circular FIFO buffer for storing
characters read from a serial line. I basically needed O(1) append,
removal from the front, and lookup/slice in the middle. For example:

 x = Buffer(16)
 x.write('abcde')
 x[1:-1]
'bcd'
 x.read(2)
'ab'
 len(x)
3
 x.read()
'cde'

The backing structure was array.array('B'), which worked pretty well
for my needs. It was more memory-efficient, but the problem is that
implementing all of these operations in python was a bit of a
performance hit.

You could easily adapt this to a dynamic or fixed-size array and
change the interface for working with both ends of the data. By the
way, if you keep the array size as a power of two, you can replace the
relatively expensive modulus operations with bitwise and.

- Max
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

 From: Antoine Pitrou solip...@pitrou.net
 raymond.hettinger at gmail.com writes:
  
  AFAICT, the performance of the proposal:
  
  * increases space requirements by a small fixed
 amount
 
 Well, given my proposal (assuming it turns out ok) it
 doesn't.
 
  * s.append() performance slightly degraded.
 
 Why?
 

A slight degradation is unavoidable AFAICT.  The overhead is roughly this on 
every call:


if (self-orphans = newsize) { // SKIP ALMOST ALWAYS
needed = newsize + self-orphans;

And if it's time for a realloc, you have to pay down a little more bookkeeping:

items = self-ob_item - self-orphans;
self-ob_item = items + self-orphans;
op-orphans = 0;


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Raymond Hettinger]

On Jan 27, 2010, at 2:56 PM, Steve Howell wrote:

 
 --- On Wed, 1/27/10, Raymond Hettinger raymond.hettin...@gmail.com wrote:
 
 From: Raymond Hettinger raymond.hettin...@gmail.com
 Subject: Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
 To: Martin v. Löwis mar...@v.loewis.de
 Cc: Steve Howell showel...@yahoo.com, python-dev@python.org
 Date: Wednesday, January 27, 2010, 1:49 PM
 
 On Jan 27, 2010, at 11:38 AM, Martin v. Löwis wrote:
 
 Lists are indeed used *everywhere* and *vast*
 quantities, which is
 why optimizations on list operations are
 worthwhile to pursue.
 
 Unfortunately, the proposed change is a pessimisation,
 not an
 optimisation. We probably shouldn't discuss it
 further, at least not
 until a PEP gets written by its proponents.
 
 
 I concur (on both points).
 
 AFAICT, the performance of the proposal:
 
 * increases space requirements by a small fixed amount
 
 * s.append() performance slightly degraded.
 
 * the s.insert(0, value) case isn't helped -- still O(n)
 
 * repeated s.pop(0) have amortized O(1) but either
needs to waste space indefinitely (likely
 not what
the programmer intended by popping off
 values)
or needs to do occasional memmoves (which
 isn't
as good as a deque which never moves the
 data).
 
 * the resize performance doesn't work well with the
memory allocator -- while a series of
 appends can often
resize in-place without a need to do an
 O(n) memmove,
but a series of pop(0) calls doesn't have
 a resize in-place
option.
 
 What currently unknown is the effect on third-party
 extensions
 and debugging tools that access the structure directly.
 Also, am not sure if this affects psyco or the other 
 implementations such as Jython which may implement
 lists in terms of existing Java containers.
 
 ISTM, the *only* benefit is that an occasional s.pop(0)
 will perform better than it does now but not as well
 as a deque which never has to move data).
 
 
 
 I agree with most of what's said above, with a few clarifications.
 
 The speedup is not limited to pop(0); it also considerably speeds up code 
 like below:
 
 n = 80
 for i in range(n):
   x = lst[:10]
   del lst[:10]
 

For this code, the slicing notation is nicer than the equivalent pop 
one-at-a-time code for deques.

The underlying performance isn't a good though -- a deque would free block as 
they become available and would never call a memmove.

The use case is a bit weird.  If people needed something like this, we would 
already see it in reverse order:

   lst.reverse()   # then delete slices from the right

ISTM, that there aren't actually any use cases for left popping in the absence 
of right appending or left inserting; otherwise you could just retrieve the 
slices directly (without doing any deletes).   In cases where the list is 
growing on one end and shrinking on another, a deque wins because it frees 
memory readily and doesn't need memmoves.

 Understood that it is a contrived benchmark, and real code like that could be 
 replaced with a technique that Nones out the used up elements and advances a 
 start pointer.  

Right.


 
 Since there is a lot of discussion about tradeoffs, I would remind folks that 
 asking somebody to use a deque instead of a list also forces tradeoffs; you 
 lose the comfort and familiarity of lists (not to be underestimated) as well 
 as some features (you can't spell d[:10] as d[:10] for example).


If it were actually needed, there is no reason deques couldn't support slicing 
notation.  But then, I've never had a request for it ever (and users typically 
haven't been shy about asking for what they want).

The familiarity argument doesn't hold much water for two reasons:

* deques seem to have a nearly zero learning curve.  There have been no posts 
or reports on people being challenged by the API.  The *only* issue that has 
ever arisen is that a fair number of people have heard of a queue but not heard 
of the term, deque.

* changing the list implementation makes it harder to decide which data 
structure to use.  Currently, it is simple -- if you need to append or pop on 
the left, use a deque; otherwise, use a list.  With the proposed change, the 
performance trade-offs are harding to understand (you can do pop(0) but not 
insert(0,v) unless you've popped first but not after a resize, a deque is 
better is most cases but it is hard to describe real use cases with list.pop(0) 
would be preferable).

* the current design encourages people to use the right data structure for a 
given need.  the proposed change makes the trades-off murky and implementation 
dependent.  you have to know a lot more in order to make the right choices.  
that's not good for usability.  we want tools that are easy to use 
correctly/well.

Raymond___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 

[Python-Dev] Restore the warning about mktemp now that DeprecationWarnings are silenced

[Ezio Melotti]

I noticed that in the py3k doc 'mktemp' is marked as deprecated since 
Python 2.3 [1], but the function is still there and doesn't raise any 
warning. Looking at the source I found out that there is a warning, but 
it is commented out [2], the reason being because they are too annoying.
There was already a discussion about this on python-dev [3], but now 
that the DeprecationWarnings are silenced by default [4] the warning 
could be restored.
If the comment can be removed I will fix it in trunk/py3k and possibly 
in 2.6/3.1 too, if #7319 will be backported.

(Brett, are you planning to backport it?)

Regards,
Ezio

[1]: http://docs.python.org/dev/py3k/library/tempfile.html
[2]: http://svn.python.org/view?view=revrevision=29829
[3]: http://mail.python.org/pipermail/python-dev/2008-May/079163.html
[4]: http://bugs.python.org/issue7319

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] O(1) random access to deque? (Re: patch to make list.pop(0) work in O(1) time)

[Maciej Fijalkowski]

 FWIW, deque indexing for small deques is already O(1)
 and somewhat fast.  You only get O(n) degradation
 (with a small contant factor) on large deques.


Hi.

For small dequeues (smaller than a constant), you have to have O(1)
operations, by definition :-)

Cheers,
fijal
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] O(1) random access to deque? (Re: patch to make list.pop(0) work in O(1) time)

[Raymond Hettinger]

On Jan 27, 2010, at 3:55 PM, Maciej Fijalkowski wrote:

 
 FWIW, deque indexing for small deques is already O(1)
 and somewhat fast.  You only get O(n) degradation
 (with a small contant factor) on large deques.
 
 
 Hi.
 
 For small dequeues (smaller than a constant), you have to have O(1)
 operations, by definition :-)
 

LOL, of course that's true.
What I meant though is that for deques that fit in a single block, the lookup 
is direct (no searching).

For multi-block deques, the code traverses links (one link for every 62 
elements).
So the indexing time would be i//62 + 1.  However , if i is more than half-way 
through the deque, the search starts from the other end.  So the indexing time 
is:  min(i, len(d)-i)//2 + 1.

It's still O(n) for large deques, but the constant factor isn't large and it is 
fast for small deques, and always O(1) for accesses near either end (i.e.  d[0] 
and d[-1] are direct lookups with no searching) -- those cases are typical.

What use case requires a large deque, growing on end, shrinking on the other, 
and needs O(1) random access to elements in the middle?  Stored indicies lose 
their meaning whenever the deque mutates.


Raymond___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

--- On Wed, 1/27/10, Raymond Hettinger raymond.hettin...@gmail.com wrote:

 From: Raymond Hettinger raymond.hettin...@gmail.com

 * the current design encourages people to use
 the right data structure for a given need.  the
 proposed change makes the trades-off murky and
 implementation dependent.  

Are you saying that the current slowness of list for prepops helps people to 
choose more appropriate data structures?  Really

 you have to know a lot more
 in order to make the right choices.  that's not
 good for usability.  we want tools that are easy to use
 correctly/well.

If you want tools that are easy to use correctly, make them bug-free and 
document their behavior.  If you want tools that are easy to use well, then 
make them perform better.  I am not sure how my patch contradicts either of 
these goals.

You keep making the argument that deque is a better alternative to list in many 
situations.  I actually agree with you.  Most programming problems are best 
modelled by a queue.  I am not sure why Python lists get all the syntax sugar 
and promotion over deque, when in reality, Python lists implement a pretty 
useless data structure.  Python lists are a glorification of a C array built on 
top of a memory-upward-biased memory allocator.  As such, they optimize list 
appends (good) but fail awfully on list prepops (bad).  They are much better as 
stacks than queues, even though queues are more useful for the most common 
programming known to man--work through a work queue and delete tasks when they 
are done.

It is not surprising that Python lists are starting to show their lack of 
versatility in 2010.  They're based on 1970's technology.  Python lists are 
really just a thin encapsulation of C arrays built on top of an asymmetrical 
memory manager.

In 2010 you could improve Python lists by releasing from the constraints of 
1970s semantics.  But I am starting to think a more modern approach would be to 
take more useful data structures like deques and give them more sugar.



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] scripts, applets, and applications (Was Re: PyCon Keynote)

[Ron Adam]

David Lyon wrote:

Nick Coghlan:


I'd like to see Python 3+ be more suitable for full distributable
applications over 2.X and earlier.



Out of curiousity, have you tried experimenting with the zipfile
execution capabilities in 2.6/3.1? A major part of that was to make
multi-file Python applications nearly as easy to execute as single-file
scripts.


I haven't experimented with zipfiles very much.  I think zip files are an 
excelent answer to writting small applets that don't need to be formally 
installed. But even then there are quite a few gotcha's that the developer 
needs to be aware of such as missing/changed modules and other environment 
differences if they intend to give these files to others.




I think what Ron meant was a way for normal users to install
a python application. That capability doesn't exist in standard
python in a way that is compatable or similar to anything that
has happened in the outside world for the last decade.


I prefer to consider what the average 'naive' user might experience when it 
comes to installing software and what the somewhat below average python 
developer will expect when it comes to writing it.  Make it doable for 
those groups and then it becomes easy for the average computer user, and 
average python developer.




Yes.. and finishing with your suggestions... we can
experiment.. play with all the undocumented and unobvious
features. And really, in a sense you are right. That is
all we can do.


It's not all we can do, it is a starting point to doing even more than we 
have already done. :)


I don't expect a conclusive answer, but I think it would be very good for 
both python users (programmers) and python language developers to have a 
general overview of what the differing needs are from different user points 
of view.


This is a moving target. I believe that it has changed over the years and 
continues to change.  So the question How is python used? needs to be 
readdressed every few years.


At one time all python programs were treated as simple modules.  Then 
packages where added and it was considered good practice to write your 
application as a package and put it in the python package directory.  With 
Windows 98, we of course could modify our Python installation in any way we 
wanted.  And that worked because we could get at everything and see how it 
was all tied together. (Yes, we still can if we really want to.  But should 
we?) Python scripts were shared between python programmers and not very 
often with non programmers.  But things have changed quite a bit with the 
boundaries of where python is used expanding to just about anyone with a 
computer weather they know it or not.


So I thought an updated overview on the subject as well as some thoughts on 
the future of where things are going might be both a useful and interesting 
topic.



Some things (factors?) that come to mind:

   Differences in program types:
   modules, packages, scripts, applications, other?

   Differences dues to intended user:
   me, my associate, or someone I never met

   Computer user skills:
   skilled programmer, skilled application user, novice computer user.

   Differences due to operating environments.
   linix/unix variant, Windows, Mac, Web browser,
   Smart phone or PDA, Other


Some of the questions might be.

   What packaging formats are best for different combinations of the above?


Ok, I cheated and used one general question instead of a lot of more 
specific questions. ;-)  I'm sure the readers here can expand that out if 
they care to.


For example; a python developer interested in extending python the 
language, doesn't have the same need as a casual computer user.  A casual 
computer user shouldn't need to learn Python in order to install and use an 
application written in Python.


Can I make my efforts in developing a (module, package, script, or 
application) a lot easier if I keep certain things in mind? Are there any 
best practices I should keep in mind if I intend to distribute my work?


Are there changes that can be made to python that can make things easier 
for various combinations of the above?




Regards,
  Ron Adam















































___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Alex Gaynor]

On Wed, Jan 27, 2010 at 11:30 PM, Steve Howell showel...@yahoo.com wrote:


 --- On Wed, 1/27/10, Raymond Hettinger raymond.hettin...@gmail.com wrote:

 From: Raymond Hettinger raymond.hettin...@gmail.com

 * the current design encourages people to use
 the right data structure for a given need.  the
 proposed change makes the trades-off murky and
 implementation dependent.

 Are you saying that the current slowness of list for prepops helps people to 
 choose more appropriate data structures?  Really

 you have to know a lot more
 in order to make the right choices.  that's not
 good for usability.  we want tools that are easy to use
 correctly/well.

 If you want tools that are easy to use correctly, make them bug-free and 
 document their behavior.  If you want tools that are easy to use well, then 
 make them perform better.  I am not sure how my patch contradicts either of 
 these goals.

 You keep making the argument that deque is a better alternative to list in 
 many situations.  I actually agree with you.  Most programming problems are 
 best modelled by a queue.  I am not sure why Python lists get all the syntax 
 sugar and promotion over deque, when in reality, Python lists implement a 
 pretty useless data structure.  Python lists are a glorification of a C array 
 built on top of a memory-upward-biased memory allocator.  As such, they 
 optimize list appends (good) but fail awfully on list prepops (bad).  They 
 are much better as stacks than queues, even though queues are more useful for 
 the most common programming known to man--work through a work queue and 
 delete tasks when they are done.

 It is not surprising that Python lists are starting to show their lack of 
 versatility in 2010.  They're based on 1970's technology.  Python lists are 
 really just a thin encapsulation of C arrays built on top of an asymmetrical 
 memory manager.

 In 2010 you could improve Python lists by releasing from the constraints of 
 1970s semantics.  But I am starting to think a more modern approach would be 
 to take more useful data structures like deques and give them more sugar.



 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 http://mail.python.org/mailman/options/python-dev/alex.gaynor%40gmail.com


Python lists implement a pretty useless data structure

It's very difficult for ideas to gain traction when they contain such
useless, and obviously wrong, rhetoric.  There's an enormous body of
code out there that begs to disagree with this assertion.

Alex

-- 
I disapprove of what you say, but I will defend to the death your
right to say it. -- Voltaire
The people's good is the highest law. -- Cicero
Code can always be simpler than you think, but never as simple as you
want -- Me
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Guido van Rossum]

On Wed, Jan 27, 2010 at 8:30 PM, Steve Howell showel...@yahoo.com wrote:
 I am not sure why Python lists get all the syntax sugar and promotion over 
deque, when in reality, Python lists implement a pretty useless data 
structure.  Python lists are a glorification of a C array built on top of a 
memory-upward-biased memory allocator.  As such, they optimize list appends 
(good) but fail awfully on list prepops (bad).  They are much better as stacks 
than queues, even though queues are more useful for the most common 
programming known to man--work through a work queue and delete tasks when they 
are done.

 It is not surprising that Python lists are starting to show their lack of 
 versatility in 2010.  They're based on 1970's technology.  Python lists are 
 really just a thin encapsulation of C arrays built on top of an asymmetrical 
 memory manager.

Steve, I think you might as well stop now. I see nothing useful coming
out of pursuing this thread further.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] scripts, applets, and applications (Was Re: PyCon Keynote)

[Eric Smith]

This discussion probably belongs on the distutils list.

Ron Adam wrote:
Can I make my efforts in developing a (module, package, script, or 
application) a lot easier if I keep certain things in mind? Are there 
any best practices I should keep in mind if I intend to distribute my work?


Are there changes that can be made to python that can make things easier 
for various combinations of the above?


Describe what you are doing in metadata and use (and/or create) the 
tools which consume that metadata. Such tools understand the platform 
specific issues.


Eric.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Steve Howell]

--- On Wed, 1/27/10, Alex Gaynor alex.gay...@gmail.com wrote:

 
 Python lists implement a pretty useless data structure
 
 It's very difficult for ideas to gain traction when they
 contain such
 useless, and obviously wrong, rhetoric.  There's an
 enormous body of
 code out there that begs to disagree with this assertion.
 

The statement was meant 99% tongue in cheek.  Like probably 99.99% of Python 
programmers, I use lists all the time; that's why I want them to be more 
versatile.

There's also a 1% element of truth to the statement.  To the extent that people 
are arguing for alternative data structures to list, particularly deque, I 
wonder if there actually is some merit in discouraging the use of lists in 
favor of better alternatives.





___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] scripts, applets, and applications (Was Re: PyCon Keynote)

[Ron Adam]

Eric Smith wrote:

This discussion probably belongs on the distutils list.


Yes, the discussion should be.

I possibly should clarify that this is a list of questions that are of 
interest as a topic and did not intend for them to be answered here.


I tried not to go into specific details for that reason. My apologies for 
any confusion.


Regards,
  Ron






Ron Adam wrote:
Can I make my efforts in developing a (module, package, script, or 
application) a lot easier if I keep certain things in mind? Are there 
any best practices I should keep in mind if I intend to distribute my 
work?


Are there changes that can be made to python that can make things 
easier for various combinations of the above?


Describe what you are doing in metadata and use (and/or create) the 
tools which consume that metadata. Such tools understand the platform 
specific issues.


Eric.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Guido van Rossum]

On Wed, Jan 27, 2010 at 8:46 PM, Steve Howell showel...@yahoo.com wrote:
 The statement was meant 99% tongue in cheek.  Like probably 99.99% of Python 
 programmers, I use lists all the time; that's why I want them to be more 
 versatile.

 There's also a 1% element of truth to the statement.  To the extent that 
 people are arguing for alternative data structures to list, particularly 
 deque, I wonder if there actually is some merit in discouraging the use of 
 lists in favor of better alternatives.

Steve, whatever your goal is, you're wearing out your welcome pretty
quickly with your current set of tactics. Try something different.
This thread is full.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] patch to make list.pop(0) work in O(1) time

[Cameron Simpson]

On 27Jan2010 23:08, Nick Coghlan ncogh...@gmail.com wrote:
| Cameron Simpson wrote:
|  The proposed change to make pop(0) O(1) involves adding a base offset
|  to the internal list implementation. Doesn't that incur a (small)
|  overhead to _every_ list operation? Doesn't that weaken list as the
|  as efficient as possible implementation of choice for array-like
|  things?
| 
| No, the implementation is smarter than that. The existing pointer in the
| PyList_* structure will continue to point to the first element of the
| list while an extra pointer is added that points to the beginning of the
| allocated memory. The difference between the two memory addresses will
| determine the number of orphan pointer slots that exist at the
| beginning of the list.

Mmmm, that is smarter. Nice.

| Only operations that change the length of the list will be slowed down
| at all, since they will need to take the orphans into account when
| deciding whether or not to resize the allocated memory.

Nice again.

| The big practical concern is actually the memory cost of adding yet
| another pointer (potentially 8 bytes of data) to every list object, even
| empty ones.

Um, yes. How big is a an empty list already?
-- 
Cameron Simpson c...@zip.com.au DoD#743
http://www.cskk.ezoshosting.com/cs/

Who is the happier man, he who has braved the storm of life and lived, or he
who has stayed securely on shore and merely existed?
- Hunter S. Thompson, age seventeen
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Restore the warning about mktemp now that DeprecationWarnings are silenced

[Brett Cannon]

On Wed, Jan 27, 2010 at 15:54, Ezio Melotti ezio.melo...@gmail.com wrote:
 I noticed that in the py3k doc 'mktemp' is marked as deprecated since Python
 2.3 [1], but the function is still there and doesn't raise any warning.
 Looking at the source I found out that there is a warning, but it is
 commented out [2], the reason being because they are too annoying.
 There was already a discussion about this on python-dev [3], but now that
 the DeprecationWarnings are silenced by default [4] the warning could be
 restored.
 If the comment can be removed I will fix it in trunk/py3k and possibly in
 2.6/3.1 too, if #7319 will be backported.
 (Brett, are you planning to backport it?)

If you mean forward-port to Python 3.1, then yes I am planning to
merge it. Since Python 3.2 is not due out for nearly a year I wanted
to wait a little while to make sure that the wording in the warnings
docs made sense. But if you and others want to start actively turning
some warnings on that will simply be too annoying when running the
test suite I can do it sooner rather than later.

-Brett


 Regards,
 Ezio

 [1]: http://docs.python.org/dev/py3k/library/tempfile.html
 [2]: http://svn.python.org/view?view=revrevision=29829
 [3]: http://mail.python.org/pipermail/python-dev/2008-May/079163.html
 [4]: http://bugs.python.org/issue7319

 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 http://mail.python.org/mailman/options/python-dev/brett%40python.org

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Summary of 2 years of Python fuzzing

[Neal Norwitz]

On Wed, Jan 27, 2010 at 11:58 AM, Ben Finney ben+pyt...@benfinney.id.au wrote:
 Chris Bergstresser ch...@subtlety.com writes:

 On Wed, Jan 27, 2010 at 2:54 AM, Ben Finney ben+pyt...@benfinney.id.au 
 wrote:
  Neal Norwitz nnorw...@gmail.com writes:
  Who knows, someone might even write a book about Fusil someday
  about a topic as obscure as Beautiful Testing. :-)
 
  Your suggested title is already taken, though, for exactly this
  purpose. The book “Beautiful Testing”, published by O'Reilly, might
  help URL:http://oreilly.com/catalog/9780596159825.

    I suspect Neal already knows that, since he cowrote chapter 9
 Beautiful is Better than Ugly.

 Ah, good.

 I did suspect that might be the case, but couldn't easily find Neal's
 name associated with the book. I decided to post anyway, preferring the
 information to be out there at the risk of seeming to miss the joke.

 Maybe one day I'll get to read the book :-

Ben, thanks.  Your comment gave me a great laugh and I really
appreciated it. :-)

The chapter is about the general Python development process, including
testing, static analysis, dynamic analysis, including Fusil among
other things.

One of my points to Victor and everyone else like him is that even
though it may seem no one is listening to you or cares, you might be
surprised to find out how many people really are paying attention and
do care.  If you have something you want to do, make it happen.  One
person really can do amazing things.  You may never get recognized for
much of what you do, that doesn't make it any less important.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com