Re: Python pyc/pyo (Was: Inflation)

[Bernardo Innocenti]

Mike C. Fletcher wrote:

 My understanding is that slowdown is seen due to large number of stats 
 during module import.

I made a few experiments and it turns out that, on a slow B2, it
takes only 182ms to stat all the 2500 files in /usr/lib/python2.5.

I also analyzed what happens when we do python -c 'import gtk',
which takes as much as 1.58 seconds on B2 with warm caches.

We do very few syscalls:

 332  stat()
 608  open()
 263  read()
 135  mmap()

I would call this quite affordable and inconsistent with the very
long time it takes to import gtk.

The C application gtk-demo starts in 0.16s on the same hardware,
so the huge slowdown is either in the gtk bindings or in the
python interpreter.

-- 
   // Bernardo Innocenti
 \X/  http://www.codewiz.org/
___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel

Re: Python pyc/pyo (Was: Inflation)

[Bernardo Innocenti]

Pádraig Brady wrote:

 For example there are many duplicate py[oc] files,

Yeah, do we really have to ship pyc nd pyo files?
They're sometimes even bigger than the source code,
probably due to the full blown UTF-32 strings.

No other dynamic language needs that pre-tokenized junk.
Is Python so slow at parsing?  Has anybody measured the
hit?

Someone told me that Python 2.5 can now also read
library files from an archive, and this is supposed
to reduce startup time a lot.  (I don't see why it
should, on any decent filesystem).

-- 
   // Bernardo Innocenti
 \X/  http://www.codewiz.org/

___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel

Re: Python pyc/pyo (Was: Inflation)

[Mike C. Fletcher]

Bernardo Innocenti wrote:
...
 No other dynamic language needs that pre-tokenized junk.
 Is Python so slow at parsing?  Has anybody measured the
 hit?
   
There is some minor improvement in speed from the cached byte code.  
That said, there's no reason to *ship* them, you can run a compileall 
script on installation (doesn't help with filesystem space usage 
though).  You only need one of pyc or pyo, choose whether we run with 
optimisation on or not and use that version of the compiled cache.  IIRC 
there are flags to the Python executable that can skip generating the 
byte code if you wanted to test it.  I'm guessing you'll see a 
noticeable but not huge slowdown in startup time for activities.  
Alternately you can just try deleting all of the pyc and pyos and seeing 
if the activities slow down in starting up.  You'd also want to time the 
whole-system boot, since Sugar is written in Python as well.
 Someone told me that Python 2.5 can now also read
 library files from an archive, and this is supposed
 to reduce startup time a lot.  (I don't see why it
 should, on any decent filesystem).
   
My understanding is that slowdown is seen due to large number of stats 
during module import.  Python does run-time lookup of each module, and 
with more involved PYTHONPATHs that can be 90 or 120 stats per import 
(you have to check for py, pyc, pyo, so and pyd versions (as well as 
un-decorated names for packages) for each name in the import and paths 
of 30 directories can happen readily).  Multiply by a few thousand 
imports  in a really big application and it can start to add up.  In 
contrast, the zip importer has loaded the zip directory table already on 
the top-level resolution, so can do simple in-memory hash lookups for 
each name from that point on.  That's about the same effort required as 
a second import of the same module name.

HTH,
Mike

-- 

  Mike C. Fletcher
  Designer, VR Plumber, Coder
  http://www.vrplumber.com
  http://blog.vrplumber.com

___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel