[Python-Dev] Embedded Python startup is slow

[bruce bushby]

Hi

I have previously asked this question in python-list, however I think it
belongs here.

I'm running python 2.7.1 on an embedded Linux board and noticed it takes 1.8
seconds to execute the most simple "Hello World" script.

Platform:
cpu: 200Mhz ARM (ARM926EJ-)
kernel: 2.6.38
uClibc: 0.92.1-rc2


Using strace I found that python is attempting to open 168 non-existent
files during startup, see trace:
http://www.veritrack.co.uk/static/images/strace-hello-py.txt

I think this is normal behavior but more noticeable due to the embedded
environment.


1. Is there anything I can do at compile time to tell Python these files
don't exist and avoid trying to open them?
2. Is it possible to make python first try and open the ".pyc" and only then
look for ".py" ?



[root@vx-200 /]# cat hello.py
#!/usr/bin/python

print "Hello World!!"
[root@vx-200 /]#


[root@vx-200 /]# time ./hello.py
Hello World!!

real0m1.781s
user0m1.530s
sys 0m0.230s



Thanks
Bruce
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Embedded Python startup is slow

[Glyph Lefkowitz]

On Mar 24, 2011, at 4:06 AM, bruce bushby wrote:

> I have previously asked this question in python-list, however I think it 
> belongs here.

As the saying goes, this list is for development of python, not with python.  
So it would be appropriate to make a suggestion as to some direction for 
Python's development that would improve your situation.

That said, at one point I had similar issues, and I also thought that Python 
should have had some kind of "embedded mode" compilation options, and I never 
found anything.

> I'm running python 2.7.1 on an embedded Linux board and noticed it takes 1.8 
> seconds to execute the most simple "Hello World" script.

You will notice that very, very early on in the process (not quite the first 
thing, but almost), there is a stat for python27.zip.  You want to put 
absolutely everything in the standard library into that archive, and to put 
your shared objects into /usr/lib/python2.7/ with names like 'foo.so' (not 
'foomodule.so', as that's statted second).

When I was working with embedded python, that was pretty much the whole story: 
2 zip files (one for the stdlib, one for everything else in my application), 
one directory for extension modules.  Finally, I hacked site.py to set sys.path 
to just those three entries, so that even failed imports would not stat too 
many non-existent files.

I think it would be a great idea to have a configure flag that instructed 
python to be pre-built this way, but I don't work with embedded pythons these 
days, so I'm not likely to do it :).


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] dependency injection for testing [was Python 3.3 release schedule posted]

[Floris Bruynooghe]

On 24 March 2011 02:16, Michael Foord  wrote:
> On 24/03/2011 02:06, Jesus Cea wrote:
>>
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>>
>> On 23/03/11 20:56, Georg Brandl wrote:
>>>
>>> For 3.3, I'd like to revive the tradition of listing planned large-scale
>>> changes in the PEP.
>>
>> I want to integrate dependence injection in the stdlib, specially in
>> libs creating sockets. This would be an optional feature.
>>
>> The idea would be, for instance, that smtplib could call an optional
>> constructor to create a socket-like object, if some is provided.
>>
>> That would break the implicit dependency with sockets, reduce
>> monkey-patching need, and ease testing the libs.
>
> In general I dislike dependency injection if it's only done for testability.
> With monkey patching (if done right) you can safely test Python code without
> messing with your public interfaces just to make code testable.

Though in this case this sounds interesting to me as it would allow
using stdlib modules with e.g. "green" sockets from eventlet or gevent
without having to revert to monkey patching (which I don't mind for
testing but dislike for production).  In a way I'm always a little
disappointed when a library uses sockets out of my reach, ideally
you'd be able to use a library abstracted away from this so you can
plug in into async frameworks and other things the author didn't
foresee.

Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Embedded Python startup is slow

[Martin v. Löwis]

> 1. Is there anything I can do at compile time to tell Python these files
> don't exist and avoid trying to open them?

If you disable dynamic loading of extension modules, the number of stat
calls will go down significantly.

> 2. Is it possible to make python first try and open the ".pyc" and only
> then look for ".py" ?

If you then further reduce sys.path, and zip up the standard library
.pyc files, you get further reductions.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Dict access with double-dot (syntactic sugar)

[Jameson Quinn]

"class attrdict" is a perennial dead-end for intermediate pythonistas who
want to save 3 characters/5 keystrokes for item access. Other languages such
as javascript allow "somedict.foo" to mean the same as "somedict['foo']", so
why not python? Well, there are a number of reasons why not, beginning with
all the magic method names in python.

But saving keystrokes is still a reasonable goal.

So what about a compromise? Allow "somedict..foo", with two dots, to take
that place. It still saves 2 characters (often 4 keystrokes; and I find even
', "[", or "]" harder to type than ".").

The "foo" part would of course have to obey attribute/identifier naming
rules. So there would be no shortcut for "somedict['$#!%']". But for any
identifier-legal foo, the interpreter would just read ..foo as ['foo'].

I would not be surprised if I'm not the first person to suggest this. If so,
and there's already well-known reasons why this is a bad idea, I apologize.
But if the only reason not to is "we never did it that way before" or "it
would be too addictive, and so people would never want to use older python
versions" or "headache for tools like pylint", I think we should do it.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dict access with double-dot (syntactic sugar)

[Dirkjan Ochtman]

On Thu, Mar 24, 2011 at 12:40, Jameson Quinn  wrote:
> "class attrdict" is a perennial dead-end for intermediate pythonistas who
> want to save 3 characters/5 keystrokes for item access. Other languages such
> as javascript allow "somedict.foo" to mean the same as "somedict['foo']", so
> why not python? Well, there are a number of reasons why not, beginning with
> all the magic method names in python.

This should go on python-ideas.

Cheers,

Dirkjan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Replace useless %.100s by %s in PyErr_Format()

[Victor Stinner]

Hi,

I plan to replace all %.100s (or any other size, %\.[0-9]+s regex) by %s
in the whole source code, in all calls to PyErr_Format(). And I would
like your opinion.

When Guido added the function PyErr_Format(), 13 years ago, the function
was implemented using a buffer of 500 bytes (allocated on the stack).
The developer was responsible to limit the argument fit into a total of
500 bytes. But 3 years later (2000), PyErr_Format() was patched to use a
dynamic buffer (allocated on the heap). But since this change,
PyErr_Format() doesn't support %.100s anymore (the 100 bytes limitation
is just ignored), and it becomes useless and so no, it's no more (since
10 years) a "protection" against segmentation fault.

But I would like to know if I have to do in all branches (3.1-3.3, or
worse: 2.5-3.3), or just in 3.3? Because it may make merge harder (like
any change only done in default).

I would like to replace %.100s because there are no more reason to
truncate strings to an arbitrary length.

=> http://bugs.python.org/issue10833

---

... at the same time, Ray Allen wrote a patch to implement %.100s in
PyUnicode_FromFormatV() (so PyErr_Format() will support it too). I would
like to replace %.100s in PyErr_Format(), and then commit its patch.

http://bugs.python.org/issue7330

Victor

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Replace useless %.100s by %s in PyErr_Format()

[M.-A. Lemburg]

Victor Stinner wrote:
> Hi,
> 
> I plan to replace all %.100s (or any other size, %\.[0-9]+s regex) by %s
> in the whole source code, in all calls to PyErr_Format(). And I would
> like your opinion.
> 
> When Guido added the function PyErr_Format(), 13 years ago, the function
> was implemented using a buffer of 500 bytes (allocated on the stack).
> The developer was responsible to limit the argument fit into a total of
> 500 bytes. But 3 years later (2000), PyErr_Format() was patched to use a
> dynamic buffer (allocated on the heap). But since this change,
> PyErr_Format() doesn't support %.100s anymore (the 100 bytes limitation
> is just ignored), and it becomes useless and so no, it's no more (since
> 10 years) a "protection" against segmentation fault.
> 
> But I would like to know if I have to do in all branches (3.1-3.3, or
> worse: 2.5-3.3), or just in 3.3? Because it may make merge harder (like
> any change only done in default).
> 
> I would like to replace %.100s because there are no more reason to
> truncate strings to an arbitrary length.
> 
> => http://bugs.python.org/issue10833
> 
> ---
> 
> ... at the same time, Ray Allen wrote a patch to implement %.100s in
> PyUnicode_FromFormatV() (so PyErr_Format() will support it too). I would
> like to replace %.100s in PyErr_Format(), and then commit its patch.
> 
> http://bugs.python.org/issue7330

I think it's better to add the #7330 fix and leave the
length limitations in place.

Note that the length limitation did not only protect against
segfaults at the time when PyErr_Format() was using a fixed
size buffer and later on against miscalculations in creating
the variable sized buffer, it also protects against making the
error message text too long to be of any use or cause problems
further down the line in error processing.

BTW: Why do you think that %.100s is not supported in
PyErr_Format() in Python 2.x ? PyString_FromFormatV()
does support this. The change to use Unicode error strings
introduced the problem, since PyUnicode_FromFormatV() for
some reason ignores the precision (which is shouldn't).

That said, it's a good idea to add the #7330 fix
to at least Python 2.7 as well, since ignoring the precision
is definitely a bug. It may even be security relevant, since
it could be used for DOS attacks on servers (e.g. causing them
to write huge strings to log files instead of just a few
hundreds bytes per message), so may even need to go into Python 2.6.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 24 2011)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sprints and pushes

[Tres Seaver]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 03/23/2011 09:36 PM, Stephen J. Turnbull wrote:
> Tres Seaver writes:
> 
>  > On 03/23/2011 01:24 PM, Antoine Pitrou wrote:
>  > > On Wed, 23 Mar 2011 10:25:01 -0700
>  > > Ethan Furman  wrote:
>  > >>
>  > >> I think the use-case has been lost.  Think sprints and multiple push 
>  > >> races.
> 
> I do, can't speak for others.  So what? *sigh* ... read on.
> 
>  > > Well, keep in ming hg is a *distributed* version control system. You
>  > > don't have to push your changes right now.
> 
> s/push your changes right now/push your changes to the public repo/
> 
>  > That doesn't work so well at a sprint, where the point is to maximize
>  > the value of precious face-time to get stuff done *now*.
> 
> That's where the D in DVCS comes in.  It's a new world, friends.  All
> you need to do is bring a $50 wireless router to the sprint, and have
> some volunteer set up a shared repo for the sprinters.  Then some
> volunteer *later* runs the tests and pilots the patches into the
> public repo.  Where's the latency?

The current full test suite is punishingly expensive to run, sprint or
not.  Because of that fact, people will defer running it, and sometimes
forget.  Trying to require that people run it repeatedly during a push
race is just Canute lashing the waves.

> N.B.  The repo admin and test-running volunteers can be non-coders.
> In fact, the tests can be running concurrently (gives those non-coders
> an excuse to attend sprints!), but nobody need wait for the results.

The rhythm is still broken if devlopers don't run the tests and see them
pass.  Async is an enemy to the process here, because it encourages poor
practices.

>  > Maybe we need to chop the problem up as:
> 
> "Violence is the last refuge of the incompetent." ObRef Asimov.

Hmm, that hardly seems appropriate, even with the wink.  "Chopping"
isn't violence in any normal sense of the word when applied to
non-persons:  Chopping up a problem is no more violent than chopping
wood or onions (i.e., not at all).


Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk2LPS0ACgkQ+gerLs4ltQ5GmACeKvwnwzbOX4jRzokwnm+B0yD3
X/AAoK3kQ9e1yq1LbUcg9ERj8LquAEHg
=FEvh
-END PGP SIGNATURE-

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dict access with double-dot (syntactic sugar)

[Brian Curtin]

On Thu, Mar 24, 2011 at 06:40, Jameson Quinn wrote:

> "class attrdict" is a perennial dead-end for intermediate pythonistas who
> want to save 3 characters/5 keystrokes for item access. Other languages such
> as javascript allow "somedict.foo" to mean the same as "somedict['foo']", so
> why not python? Well, there are a number of reasons why not, beginning with
> all the magic method names in python.
>
> But saving keystrokes is still a reasonable goal.
>

Code is read far more often than it is written, so readability tends to
count more than most other metrics.

So what about a compromise? Allow "somedict..foo", with two dots, to take
> that place. It still saves 2 characters (often 4 keystrokes; and I find even
> ', "[", or "]" harder to type than ".").
>

I don't see the benefit, but maybe it'll save a few bytes in file size.
Anyone reviewing your code now has to think "does this need one or two
dots?"

Anyways, why not just do something like this:

class AttrDict(dict):
def __getattr__(self, attr):
return super(AttrDict, self).__getitem__(attr)

>>> d = AttrDict()
>>> d["a"] = 1
>>> d.a
1
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Second draft: PEP397: Python launcher for Windows

[Michael Foord]

On 24/03/2011 03:02, Mark Hammond wrote:

On 24/03/2011 1:20 PM, Michael Foord wrote:

On 24/03/2011 00:44, Dj Gilcrease wrote:

On Wed, Mar 23, 2011 at 8:14 PM, Mark
Hammond wrote:

If you guys (or anyone) would like to agree on some precise rules for
both
the location of the config file and its contents and express this as
a patch
to the PEP text, I have no problem supporting it in the 
implementations.
I'd like to insist that the format of the config file was such that 
the

GetPrivateProfileString() Windows function could be used to extract
the data
(eg, only '=' can be used to separate the name/value pair,
case-insensitive
and no support for string interpolation) as I have no interest in
writing my
own config file parser in C :)

In the user directory much like TortoiseHG adds a murcurial.ini file
to the users directory for basic globa config the launch could look
for a python.ini there and use to to add known paths or version
overrides on a user by user basis.


A single global location (for shared installs) or a single per-user
location for per-user installs would seem to be sensible if the config
file route is chosen.


My concern with that would be that an administrator may install 
Python, but the user may not have write access to that global 
location, leaving that user unable to customize the launcher.  OTOH, 
that is how things work on Unix today - such users could not change 
what /usr/bin/python points to).  Always using a per-user location 
would mean there is no opportunity to have global settings, but it is 
unclear which is the lesser of 2 evils.  Supporting both might even be 
reasonable if the rules are very straightforward.




I would look first for a local config and fallback to a global config. 
That way you can choose your evil... With a single config location for 
each it should be simple enough.


I'd still very much like to see the change to the wording of the PEP 
to describe this feature though, otherwise I fear different people 
will have different assumptions about exactly what it does and how it 
does it...




I can provide a suggested pep wording if you like.

Michael


Cheers,

Mark



--
http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Let's get PEP 380 into Python 3.3

[rndblnch]

rndblnch  gmail.com> writes:
> > Anyone without push rights for hg.python.org may want to fork the
> > mirror the bitbucket folks set up during the PyCon sprints to avoid
> > having to push the whole repository up to a separate hosting site:
> > https://bitbucket.org/mirror/cpython
> 
> Done here:
> 

Now that I have figured out how to use patch queues with bitbucket, I will
maintain greg's pep380 implementation as a patch on top of cpython here:


Testing it should be as simple as:
% hg qclone https://bitbucket.org/rndblnch/cpython-pep380
% cd cpython-pep380
% hg qpush pep380

and then the usual ./configure etc.


To test on an already existing clone of ...
% hg clone http://hg.python.org/cpython
% cd cpython
% hg update -C default

you should enable mq extension...
% hg init --mq

get the pep380 patch and apply it...
% pushd .hg/patches/
% hg pull https://bitbucket.org/rndblnch/cpython-pep380/.hg/patches
% hg update
% popd
% hg qpush pep380

and then the usual ./configure etc.


The patch is now visible here:


Hope this helps,

renaud


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sprints and pushes

[Antoine Pitrou]

On Thu, 24 Mar 2011 08:46:37 -0400
Tres Seaver  wrote:
> > 
> >  > That doesn't work so well at a sprint, where the point is to maximize
> >  > the value of precious face-time to get stuff done *now*.
> > 
> > That's where the D in DVCS comes in.  It's a new world, friends.  All
> > you need to do is bring a $50 wireless router to the sprint, and have
> > some volunteer set up a shared repo for the sprinters.  Then some
> > volunteer *later* runs the tests and pilots the patches into the
> > public repo.  Where's the latency?
> 
> The current full test suite is punishingly expensive to run, sprint or
> not.  Because of that fact, people will defer running it, and sometimes
> forget.  Trying to require that people run it repeatedly during a push
> race is just Canute lashing the waves.

Punishingly expensive?
You have to remember that Python is an entire programming language with
its standard library, used by millions of people. That its test suite
can run on 4 minutes on a modern computer actually makes it rather
"fast" IMO (and, perhaps, incomplete...).

If you have a "push race", then after merging you could just re-run the
tests that are affected by your changes (of course, if you did a change
in the interpreter core, you probably should run the whole suite again).
That's both faster and better focussed than a hypothetical "smoke test".

(that assumes you did run the test suite before doing the original
commit, of course)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dict access with double-dot (syntactic sugar)

[Jameson Quinn]

2011/3/24 Brian Curtin 

> On Thu, Mar 24, 2011 at 06:40, Jameson Quinn wrote:
>
>> "class attrdict" is a perennial dead-end for intermediate pythonistas who
>> want to save 3 characters/5 keystrokes for item access. Other languages such
>> as javascript allow "somedict.foo" to mean the same as "somedict['foo']", so
>> why not python? Well, there are a number of reasons why not, beginning with
>> all the magic method names in python.
>>
>> But saving keystrokes is still a reasonable goal.
>>
>
> Code is read far more often than it is written, so readability tends to
> count more than most other metrics.
>
> So what about a compromise? Allow "somedict..foo", with two dots, to take
>> that place. It still saves 2 characters (often 4 keystrokes; and I find even
>> ', "[", or "]" harder to type than ".").
>>
>
> I don't see the benefit, but maybe it'll save a few bytes in file size.
> Anyone reviewing your code now has to think "does this need one or two
> dots?"
>
>  Anyways, why not just do something like this:
>
> class AttrDict(dict):
> def __getattr__(self, attr):
> return super(AttrDict, self).__getitem__(attr)
>
> >>> d = AttrDict()
> >>> d["a"] = 1
> >>> d.a
> 1
>

There are a few reasons not to do it your way. For one, you could easily
forget about one of the built-in dict methods (e.g. d.get != d["get"]). For
another, if you look on the web, you'll find at least 15 different recipes
for that thing you just made, several of which have more-or-less subtle
errors waiting to get you. Furthermore, the whole point is to have this
available for built-in dicts. Say you get a dict as json - you can either
subclass your own json decoder, with all the pitfalls, or you can explicitly
pass the decoded dict to AttrDict, causing an extra object to be created and
obfuscating your code. And finally, who wants to copy that AttrDict code for
the 137th time?

As for the question of "one or two dots", it's exactly the same question you
face now with "dot or bracket", so I don't see the problem.

It's not merely a matter of saving keystrokes. To me, it would be actually
easier to read code in this style. When I'm doing things like accessing my
json data, that is essentially attribute access; why should my syntax
colorer color it the same as my UI strings?

In sum:
-Saves keystrokes
-saves bugs from miscooked recipes
-faster and less memory than any such recipe
-more-readable code
-very low-risk for old code

Jameson
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Embedded Python startup is slow

[Thomas Heller]

Am 24.03.2011 12:18, schrieb "Martin v. Löwis":

1. Is there anything I can do at compile time to tell Python these files
don't exist and avoid trying to open them?


If you disable dynamic loading of extension modules, the number of stat
calls will go down significantly.


2. Is it possible to make python first try and open the ".pyc" and only
then look for ".py" ?


If you then further reduce sys.path, and zip up the standard library
.pyc files, you get further reductions.


On my embedded ARM system (400MHz ARM926EJ-S, linux 2.6.38) Python
starts up even slower when the standard library is in a zip-file.
The effect is worse when the zip-file is compressed, but the slow-down
is still there if the zip-file is not compressed.

Thomas
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dict access with double-dot (syntactic sugar)

[Jameson Quinn]

Consider:

def fun1(argument):
print argument1

fun1(argument="spam")

def fun2(**kw):
print kw["argument"]

Why should I need quotes around "argument" in just one of those places? What
if I left them off, and there happened to be a global variable named
"argument"? Why shouldn't I be able to say:

def fun2(**kw):
print kw..argument

(in real life, there would be a try... except block in case there was no
argument, I'm just showing the simplest case here.)

Jameson

2011/3/24 Jameson Quinn 

>
>
> 2011/3/24 Brian Curtin 
>
>> On Thu, Mar 24, 2011 at 06:40, Jameson Quinn wrote:
>>
>>> "class attrdict" is a perennial dead-end for intermediate pythonistas who
>>> want to save 3 characters/5 keystrokes for item access. Other languages such
>>> as javascript allow "somedict.foo" to mean the same as "somedict['foo']", so
>>> why not python? Well, there are a number of reasons why not, beginning with
>>> all the magic method names in python.
>>>
>>> But saving keystrokes is still a reasonable goal.
>>>
>>
>> Code is read far more often than it is written, so readability tends to
>> count more than most other metrics.
>>
>> So what about a compromise? Allow "somedict..foo", with two dots, to take
>>> that place. It still saves 2 characters (often 4 keystrokes; and I find even
>>> ', "[", or "]" harder to type than ".").
>>>
>>
>> I don't see the benefit, but maybe it'll save a few bytes in file size.
>> Anyone reviewing your code now has to think "does this need one or two
>> dots?"
>>
>>  Anyways, why not just do something like this:
>>
>> class AttrDict(dict):
>> def __getattr__(self, attr):
>> return super(AttrDict, self).__getitem__(attr)
>>
>> >>> d = AttrDict()
>> >>> d["a"] = 1
>> >>> d.a
>> 1
>>
>
> There are a few reasons not to do it your way. For one, you could easily
> forget about one of the built-in dict methods (e.g. d.get != d["get"]). For
> another, if you look on the web, you'll find at least 15 different recipes
> for that thing you just made, several of which have more-or-less subtle
> errors waiting to get you. Furthermore, the whole point is to have this
> available for built-in dicts. Say you get a dict as json - you can either
> subclass your own json decoder, with all the pitfalls, or you can explicitly
> pass the decoded dict to AttrDict, causing an extra object to be created and
> obfuscating your code. And finally, who wants to copy that AttrDict code for
> the 137th time?
>
> As for the question of "one or two dots", it's exactly the same question
> you face now with "dot or bracket", so I don't see the problem.
>
> It's not merely a matter of saving keystrokes. To me, it would be actually
> easier to read code in this style. When I'm doing things like accessing my
> json data, that is essentially attribute access; why should my syntax
> colorer color it the same as my UI strings?
>
> In sum:
> -Saves keystrokes
> -saves bugs from miscooked recipes
> -faster and less memory than any such recipe
> -more-readable code
> -very low-risk for old code
>
> Jameson
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dict access with double-dot (syntactic sugar)

[Antoine Pitrou]

Can this discussion be moved to python-ideas? Thank you.



On Thu, 24 Mar 2011 09:51:59 -0600
Jameson Quinn  wrote:

> Consider:
> 
> def fun1(argument):
> print argument1
> 
> fun1(argument="spam")
> 
> def fun2(**kw):
> print kw["argument"]
> 
> Why should I need quotes around "argument" in just one of those places? What
> if I left them off, and there happened to be a global variable named
> "argument"? Why shouldn't I be able to say:
> 
> def fun2(**kw):
> print kw..argument
> 
> (in real life, there would be a try... except block in case there was no
> argument, I'm just showing the simplest case here.)
> 
> Jameson
> 
> 2011/3/24 Jameson Quinn 
> 
> >
> >
> > 2011/3/24 Brian Curtin 
> >
> >> On Thu, Mar 24, 2011 at 06:40, Jameson Quinn 
> >> wrote:
> >>
> >>> "class attrdict" is a perennial dead-end for intermediate pythonistas who
> >>> want to save 3 characters/5 keystrokes for item access. Other languages 
> >>> such
> >>> as javascript allow "somedict.foo" to mean the same as "somedict['foo']", 
> >>> so
> >>> why not python? Well, there are a number of reasons why not, beginning 
> >>> with
> >>> all the magic method names in python.
> >>>
> >>> But saving keystrokes is still a reasonable goal.
> >>>
> >>
> >> Code is read far more often than it is written, so readability tends to
> >> count more than most other metrics.
> >>
> >> So what about a compromise? Allow "somedict..foo", with two dots, to take
> >>> that place. It still saves 2 characters (often 4 keystrokes; and I find 
> >>> even
> >>> ', "[", or "]" harder to type than ".").
> >>>
> >>
> >> I don't see the benefit, but maybe it'll save a few bytes in file size.
> >> Anyone reviewing your code now has to think "does this need one or two
> >> dots?"
> >>
> >>  Anyways, why not just do something like this:
> >>
> >> class AttrDict(dict):
> >> def __getattr__(self, attr):
> >> return super(AttrDict, self).__getitem__(attr)
> >>
> >> >>> d = AttrDict()
> >> >>> d["a"] = 1
> >> >>> d.a
> >> 1
> >>
> >
> > There are a few reasons not to do it your way. For one, you could easily
> > forget about one of the built-in dict methods (e.g. d.get != d["get"]). For
> > another, if you look on the web, you'll find at least 15 different recipes
> > for that thing you just made, several of which have more-or-less subtle
> > errors waiting to get you. Furthermore, the whole point is to have this
> > available for built-in dicts. Say you get a dict as json - you can either
> > subclass your own json decoder, with all the pitfalls, or you can explicitly
> > pass the decoded dict to AttrDict, causing an extra object to be created and
> > obfuscating your code. And finally, who wants to copy that AttrDict code for
> > the 137th time?
> >
> > As for the question of "one or two dots", it's exactly the same question
> > you face now with "dot or bracket", so I don't see the problem.
> >
> > It's not merely a matter of saving keystrokes. To me, it would be actually
> > easier to read code in this style. When I'm doing things like accessing my
> > json data, that is essentially attribute access; why should my syntax
> > colorer color it the same as my UI strings?
> >
> > In sum:
> > -Saves keystrokes
> > -saves bugs from miscooked recipes
> > -faster and less memory than any such recipe
> > -more-readable code
> > -very low-risk for old code
> >
> > Jameson
> >
> 


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Embedded Python startup is slow

[bruce bushby]

My main concern was that a freshly compiled Python attempts to open 168
non-existent files before starting.

I understand that an interpreted language is probably not the best choice
for an embedded device (although it's very nice for prototyping) , Python
really should know what exists after it's just been compiledie before
any corrupting modules or other nonsense has been added.

It appears it is hard coded to open these files regardless of any
"configure" options.

On my desktop pc, when I run the most simple "Hello World"  78% of the
overall execution time is spent opening filesmost of which don't exist.

Some form of "cache" would help the startup time on the "second go" .
but arguably just a "band aid" covering a deeper problem.


Bruce





On Thu, Mar 24, 2011 at 3:23 PM, Thomas Heller  wrote:

> Am 24.03.2011 12:18, schrieb "Martin v. Löwis":
>
>  1. Is there anything I can do at compile time to tell Python these files
>>> don't exist and avoid trying to open them?
>>>
>>
>> If you disable dynamic loading of extension modules, the number of stat
>> calls will go down significantly.
>>
>>  2. Is it possible to make python first try and open the ".pyc" and only
>>> then look for ".py" ?
>>>
>>
>> If you then further reduce sys.path, and zip up the standard library
>> .pyc files, you get further reductions.
>>
>
> On my embedded ARM system (400MHz ARM926EJ-S, linux 2.6.38) Python
> starts up even slower when the standard library is in a zip-file.
> The effect is worse when the zip-file is compressed, but the slow-down
> is still there if the zip-file is not compressed.
>
> Thomas
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Embedded Python startup is slow

[Thomas Heller]

Am 24.03.2011 16:58, schrieb bruce bushby:


My main concern was that a freshly compiled Python attempts to open 168
non-existent files before starting.

I understand that an interpreted language is probably not the best
choice for an embedded device (although it's very nice for prototyping)
, Python really should know what exists after it's just been
compiledie before any corrupting modules or other nonsense has been
added.

It appears it is hard coded to open these files regardless of any
"configure" options.

On my desktop pc, when I run the most simple "Hello World"  78% of
the overall execution time is spent opening filesmost of which don't
exist.

Some form of "cache" would help the startup time on the "second go"
. but arguably just a "band aid" covering a deeper problem.


So maybe you should try to come up with a patch, to find out
if the cache helps?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dict access with double-dot (syntactic sugar)

[Brian Curtin]

On Thu, Mar 24, 2011 at 10:51, Jameson Quinn wrote:

> Consider:
>
> def fun1(argument):
> print argument1
>
> fun1(argument="spam")
>
> def fun2(**kw):
> print kw["argument"]
>
> Why should I need quotes around "argument" in just one of those places?
> What if I left them off, and there happened to be a global variable named
> "argument"? Why shouldn't I be able to say:
>
> def fun2(**kw):
> print kw..argument
>
> (in real life, there would be a try... except block in case there was no
> argument, I'm just showing the simplest case here.)
>

I can certainly see use cases for this, but none generic enough for the
standard library.

Let's take the configparser module for example: in 3.2 ConfigParser objects
can now be accessed like dictionaries. Looking at the examples in the
documentation [0], an immediate problem comes to mind:

print(config..bitbucket.org..user)
*boom*

If you're going to be doing attribute access, it's likely that you know the
name of the attribute -- you wrote the code knowing what to expect in the
first place. If you know the names of the attributes you're going to be
dealing with, why not just store them in a class, or even a class with
__slots__ defined for each attribute?

The double-dot notation will only work when you already know the key. When
iterating over keys, you're going to resort back to dict[key] or
getattr(dict, key) to get the value.

In the end, it's syntactic sugar for a limited set of applicable cases.



[0] http://docs.python.org/release/3.2/library/configparser
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sprints and pushes

[Eli Bendersky]

On Thu, Mar 24, 2011 at 16:33, Antoine Pitrou  wrote:

> On Thu, 24 Mar 2011 08:46:37 -0400
> Tres Seaver  wrote:
> > >
> > >  > That doesn't work so well at a sprint, where the point is to
> maximize
> > >  > the value of precious face-time to get stuff done *now*.
> > >
> > > That's where the D in DVCS comes in.  It's a new world, friends.  All
> > > you need to do is bring a $50 wireless router to the sprint, and have
> > > some volunteer set up a shared repo for the sprinters.  Then some
> > > volunteer *later* runs the tests and pilots the patches into the
> > > public repo.  Where's the latency?
> >
> > The current full test suite is punishingly expensive to run, sprint or
> > not.  Because of that fact, people will defer running it, and sometimes
> > forget.  Trying to require that people run it repeatedly during a push
> > race is just Canute lashing the waves.
>
> Punishingly expensive?
> You have to remember that Python is an entire programming language with
> its standard library, used by millions of people. That its test suite
> can run on 4 minutes on a modern computer actually makes it rather
> "fast" IMO (and, perhaps, incomplete...).
>

+1
Having experience running [= suffering from] multiple-hour (and sometimes
weekend-long) tests for some systems, Python's test suite feels slender.
Even surprisingly so. I often wonder how such a relatively short set of
tests can exercise a project as big and full of functionality as Python with
its whole standard library.

Eli
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Proposal for Python 3.3: dependence injection

[Jesus Cea]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi, everybody.

I want to test the dev community interest in modifying the stdlib to
ease dependence injection.

The seminal idea was in
.

A lot of stdlib modules use, deep inside, other modules to do their
work. The most obvious case would be the socket library, used by ANY
module involving the network.

This internal usage, deep inside, prevents programmers to "mess" with
the networking layer in an easy way. Some examples: transparent
encryption/compression, using alternative network implementations like
shared buffers if the traffic is local, integrating the socket in a
async event loop/greenlet/etc, network logging/sniffing, testing
(simulating a server, with no real network connection), etc. Testing
gets very easy if we can "mock" the internal socket, for instance.

So I propose to add an additional OPTIONAL parameter to quite a few
libraries. If the parameter is not present (compatible with client
current code), the library would instantiate a internal socket, just
like now. But if the parameter is present, it would inject a constructor
for the library to use instead of "socket".

Traditionally this "features" can be used in Python code using monkey
patching, but this approach is fragile (can break when upgrading), not
obvious for newbies and difficult to combine (for instance, trying to
apply multiple "monkey patches" over the socket module).

What do you think?. Should I write a PEP? (I don't think so, but your
opinion matters). I care, for instance, about how to garantee the API
coverage actually needed for the new "socket-like" object. The idea is
that your object should be "socket-like", but how much "socket-like" in
the API sense?. Do you need to implement "getpeername()" for injecting
in smtplib?.

If you agree that this is a nice idea, what other libs do you think that
could benefice of being "injected", beside "socket"?.

- -- 
Jesus Cea Avion _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
jabber / xmpp:j...@jabber.org _/_/_/_/  _/_/_/_/_/
.  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBTYtz8plgi5GaxT1NAQJCzwQAkvtCwvKDh1QcQ+Qfbhe0qrKcQdI4wMfB
g4jpCmCX3UCkTvAjU1AsnxGGwVjvYZWdL2QVZS5+sJOP5GOmMzm96PBYjWZjVovk
YHIP+HFXU0BtbXlRJMV2+bjrwZf1g8CQFngyxqkQ69WdU+FMyxmDhFbcQ7y667HT
ldkINMLSuTk=
=rrkE
-END PGP SIGNATURE-
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Benjamin Peterson]

2011/3/24 Jesus Cea :
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hi, everybody.
>
> I want to test the dev community interest in modifying the stdlib to
> ease dependence injection.

I, for one, am -1. Code shouldn't be uglified for the purposes of
testing. It's also a slippery slope. Maybe we should add parameters
for open() and io functions? What about sys and os? Those often need
to be mocked.



-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Isolate the test_source() test in test_collections

[Benjamin Peterson]

2011/3/24 raymond.hettinger :
> http://hg.python.org/cpython/rev/4f1cd92fe835
> changeset:   68901:4f1cd92fe835
> user:        Raymond Hettinger 
> date:        Thu Mar 24 09:45:43 2011 -0700
> summary:
>  Isolate the test_source() test in test_collections
>
> files:
>  Lib/test/test_collections.py |  13 -
>  1 files changed, 8 insertions(+), 5 deletions(-)
>
>
> diff --git a/Lib/test/test_collections.py b/Lib/test/test_collections.py
> --- a/Lib/test/test_collections.py
> +++ b/Lib/test/test_collections.py
> @@ -330,13 +330,16 @@
>
>     def test_source(self):
>         # verify that _source can be run through exec()
> -        tmp = namedtuple('Color', 'red green blue')
> -        self.assertNotIn('Color', globals())
> +        tmp = namedtuple('NTColor', 'red green blue')
> +        globals().pop('NTColor', None)          # remove artifacts from 
> other tests
> +        self.assertNotIn('NTColor', globals())

Shouldn't these other tests cleanup after themselves?



-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Embedded Python startup is slow

[bruce bushby]

".your Desktop PC being that MS-Windows..."  ww sis man, it's all
Linux :

I wanted to make sure I've not missed some compile trick for this sort of
thing.

".So maybe you should try to come up with a patch, to find out if the
cache helps?"
Yes, now that I know I've not ignored some obvious "everybody knows not to
do that" trick I'll look into some form of cache.although off the top of
my head I imagine it''ll be difficult to precede every open with a cache
lookupperhaps this is something better suited for the underlying
"uClibc" frameworkbut then who refreshes the cache.

Thanks for the thoughts and idea, if I find a nice solution I'll be sure to
post an update.


Bruce






On Thu, Mar 24, 2011 at 4:18 PM, James Y Knight  wrote:

> On Mar 24, 2011, at 11:58 AM, bruce bushby wrote:
> > My main concern was that a freshly compiled Python attempts to open 168
> non-existent files before starting.
> >
> > I understand that an interpreted language is probably not the best choice
> for an embedded device (although it's very nice for prototyping) , Python
> really should know what exists after it's just been compiledie before
> any corrupting modules or other nonsense has been added.
> >
> > It appears it is hard coded to open these files regardless of any
> "configure" options.
> >
> > On my desktop pc, when I run the most simple "Hello World"  78% of
> the overall execution time is spent opening filesmost of which don't
> exist.
> >
> > Some form of "cache" would help the startup time on the "second go" .
> but arguably just a "band aid" covering a deeper problem.
>
> The deeper problem on your Desktop PC being that MS-Windows' file system
> calls are horrifically expensive for no good reason? :)
>
> James
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[James Y Knight]

On Mar 24, 2011, at 12:40 PM, Jesus Cea wrote:
> I want to test the dev community interest in modifying the stdlib to
> ease dependence injection.
> 
> The seminal idea was in
> .
> 
> A lot of stdlib modules use, deep inside, other modules to do their
> work. The most obvious case would be the socket library, used by ANY
> module involving the network.

Do you know about exocet (described by the last few blog entries on 
http://washort.twistedmatrix.com/search/label/exocet), a newly developed tool 
for doing dependency injection on arbitrary modules? It seems like it would be 
very well suited for the kinds of things you want to do.

James
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Embedded Python startup is slow

[James Y Knight]

On Mar 24, 2011, at 11:58 AM, bruce bushby wrote:
> My main concern was that a freshly compiled Python attempts to open 168 
> non-existent files before starting. 
> 
> I understand that an interpreted language is probably not the best choice for 
> an embedded device (although it's very nice for prototyping) , Python really 
> should know what exists after it's just been compiledie before any 
> corrupting modules or other nonsense has been added.
> 
> It appears it is hard coded to open these files regardless of any "configure" 
> options.
> 
> On my desktop pc, when I run the most simple "Hello World"  78% of the 
> overall execution time is spent opening filesmost of which don't exist. 
> 
> Some form of "cache" would help the startup time on the "second go" . but 
> arguably just a "band aid" covering a deeper problem.

The deeper problem on your Desktop PC being that MS-Windows' file system calls 
are horrifically expensive for no good reason? :)

James
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Antoine Pitrou]

On Thu, 24 Mar 2011 11:46:42 -0500
Benjamin Peterson  wrote:

> 2011/3/24 Jesus Cea :
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA1
> >
> > Hi, everybody.
> >
> > I want to test the dev community interest in modifying the stdlib to
> > ease dependence injection.
> 
> I, for one, am -1. Code shouldn't be uglified for the purposes of
> testing. It's also a slippery slope. Maybe we should add parameters
> for open() and io functions? What about sys and os? Those often need
> to be mocked.

A non-ugly way of doing "dependency injection" is to rely on
class/instance attributes instead. e.g.:

class FTP:
socket_factory = socket.create_connection

def open(self, host, port):
self.sock = self.socket_factory(host, port)
...


Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Trimming "make quicktest"

[Brett Cannon]

On Wed, Mar 23, 2011 at 11:56, Antoine Pitrou  wrote:

> Le mercredi 23 mars 2011 à 18:51 +, Michael Foord a écrit :
> > On 23/03/2011 18:42, Antoine Pitrou wrote:
> > > On Wed, 23 Mar 2011 14:29:22 -0400
> > > David Bolen  wrote:
> > >> Nick Coghlan  writes:
> > >>
> > >>> On Thu, Mar 24, 2011 at 12:36 AM, Antoine Pitrou
>  wrote:
> >  You mean in the "-j" option itself or in "make test"?
> > >>> I was actually suggesting that -j be the *default* in regrtest
> itself,
> > >>> with an option to turn it off or force a particular number of
> > >>> processes.
> > >> Just one request - if there are changes in this direction (e.g.,
> > >> trying to make regrtest use all cores by default), please include the
> > >> ability to configure/override this for individual builders (or at
> > >> least slaves) since otherwise I won't be able to disable it.
> > > I think "-j" should remain a manual setting. I've posted a patch to
> > > enable it automatically in "make test" for convenience, but it would
> > > be enabled for neither "-m test" nor "make buildbottest".
> >
> > -j doesn't pass on several of the flags to its subprocesses (e.g.
> > warning settings I believe)
>
> It does (should):
> http://hg.python.org/cpython/file/2f4865834695/Lib/test/support.py#l1375
>

I fixed that at the sprints so yes, it works as expected (at least for the
flags one will care about).

-Brett



>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Embedded Python startup is slow

[Stefan Behnel]

bruce bushby, 24.03.2011 16:58:

My main concern was that a freshly compiled Python attempts to open 168
non-existent files before starting.

I understand that an interpreted language is probably not the best choice
for an embedded device


Well, "hello world" isn't exactly the benchmark I'd use for an embedded system.



On my desktop pc, when I run the most simple "Hello World"  78% of the
overall execution time is spent opening filesmost of which don't exist.


How did you measure that?

I'd expect that after the first startup, most of the relevant parts of the 
file system are cached by the OS, so a subsequent run should be quick as 
all file existence tests will run from memory. Sure, it's a bit of overhead 
to call into the OS, and I don't know how expensive that is on ARM-Linux, 
but given that you also claim it to be expensive on your desktop, I must 
say that I'm a bit surprised about your above claim.


Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Guido van Rossum]

On Thu, Mar 24, 2011 at 9:46 AM, Benjamin Peterson  wrote:
>> I want to test the dev community interest in modifying the stdlib to
>> ease dependence injection.
>
> I, for one, am -1. Code shouldn't be uglified for the purposes of
> testing.

Well, the philosophy of dependency injection requires exactly that.
Personally I am on the fence; I've seen is used effectively but I've
also seen code, indeed, uglified with little benefits. (Same for other
extreme testing practices like TDD.)

> It's also a slippery slope. Maybe we should add parameters
> for open() and io functions? What about sys and os? Those often need
> to be mocked.

There are existing solutions that work pretty well, so I'm thinking
there is no big problem that this solved.

But my main concern is about anything that proposed to make sweeping
changes to a large number of stdlib modules. This has almost always
resulted in reduced quality.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dict access with double-dot (syntactic sugar)

[Jameson Quinn]

OK, fair enough. People don't like this. So let me back up a step.

Clearly this is intended for using with things that you get as a dictionary,
but which really should be namespaces. The top two cases of that are parsed
json objects and **kw arguments.

I suppose that, if I cared to, I could write a decorator that passed a
wrapped object in kw arguments. Also, the whole point of **kw is that you
don't know ahead of time exactly what's in there, so the usefulness is
limited.

But definitely, it looks wrong to me to be putting the attribute names of my
json in quotes. Again, I could make a wrapper, and tinker with my json
parser (subclass it or whatever) so that it passed back by default. But I
think that this use case is common enough with json (often, you know exactly
what should be in there) that it should be built into the standard json
library.

So, what about a method on (certain?) dict objects to return the
attr-wrapped version of the dict. That is:

class AttrDict(object):
   def __init__(self, dict):
 self.__dict__ = dict

class WithAttrDict(dict):
@cached_property #what it sounds like - implementation not shown
def att(self):
return AttrDict(self)

I'd like all the dicts and sub-dicts which come from the standard json
library to be WithAttrDicts, so I can write:

parsedjson.att.spam.att.eggs #I'd love to avoid the second "att."
   #but I don't want to complicate this proposal further.

instead of

parsedjson["spam"]["eggs"].

I'd love it if there were a one-or-two-letter property name to use instead
of ".att" (".s", by analogy with english apostrophe-s?). And I'd even love
it if the built-in dictionary object had such a property, even if it had a
long name ("{'spam':'eggs'}.as_attributes.spam == 'eggs' "). That latter
thing could, in theory, break working code... but only if you're using the
same attribute name to duck-type a dictionary subclass, which I'd bet heavy
odds has not been done for the given attribute name or any reasonable
substitute.

Jameson

2011/3/24 Brian Curtin 

> On Thu, Mar 24, 2011 at 10:51, Jameson Quinn wrote:
>
>> Consider:
>>
>> def fun1(argument):
>> print argument1
>>
>> fun1(argument="spam")
>>
>> def fun2(**kw):
>> print kw["argument"]
>>
>> Why should I need quotes around "argument" in just one of those places?
>> What if I left them off, and there happened to be a global variable named
>> "argument"? Why shouldn't I be able to say:
>>
>> def fun2(**kw):
>> print kw..argument
>>
>> (in real life, there would be a try... except block in case there was no
>> argument, I'm just showing the simplest case here.)
>>
>
> I can certainly see use cases for this, but none generic enough for the
> standard library.
>
> Let's take the configparser module for example: in 3.2 ConfigParser objects
> can now be accessed like dictionaries. Looking at the examples in the
> documentation [0], an immediate problem comes to mind:
>
> print(config..bitbucket.org..user)
> *boom*
>
> If you're going to be doing attribute access, it's likely that you know the
> name of the attribute -- you wrote the code knowing what to expect in the
> first place. If you know the names of the attributes you're going to be
> dealing with, why not just store them in a class, or even a class with
> __slots__ defined for each attribute?
>
> The double-dot notation will only work when you already know the key. When
> iterating over keys, you're going to resort back to dict[key] or
> getattr(dict, key) to get the value.
>
> In the end, it's syntactic sugar for a limited set of applicable cases.
>
>
>
> [0] http://docs.python.org/release/3.2/library/configparser
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Embedded Python startup is slow

[bruce bushby]

I used the most simple "Hello World" program as a benchmark for "bare bones"
Python __initialization__...nothing more.

On my pc, the "Hello World" script obviously executes very quicklyso
trying to open 160 odd files that don't exist is negligible, but it still
happens.

 "...How did you measure that?..."
I used "strace -c hello.py"

This is not a dig at Python, I love PythonI was asking if I had missed a
compile option or implementation trick.


Bruce





On Thu, Mar 24, 2011 at 5:05 PM, Stefan Behnel  wrote:

> bruce bushby, 24.03.2011 16:58:
>
>  My main concern was that a freshly compiled Python attempts to open 168
>> non-existent files before starting.
>>
>> I understand that an interpreted language is probably not the best choice
>> for an embedded device
>>
>
> Well, "hello world" isn't exactly the benchmark I'd use for an embedded
> system.
>
>
>
>  On my desktop pc, when I run the most simple "Hello World"  78% of the
>> overall execution time is spent opening filesmost of which don't
>> exist.
>>
>
> How did you measure that?
>
> I'd expect that after the first startup, most of the relevant parts of the
> file system are cached by the OS, so a subsequent run should be quick as all
> file existence tests will run from memory. Sure, it's a bit of overhead to
> call into the OS, and I don't know how expensive that is on ARM-Linux, but
> given that you also claim it to be expensive on your desktop, I must say
> that I'm a bit surprised about your above claim.
>
> Stefan
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/bruce.bushby%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dict access with double-dot (syntactic sugar)

[Guido van Rossum]

On Thu, Mar 24, 2011 at 10:37 AM, Jameson Quinn  wrote:
> OK, fair enough. People don't like this. So let me back up a step.

> Clearly this is intended for using with things that you get as a dictionary,
> but which really should be namespaces. The top two cases of that are parsed
> json objects and **kw arguments.

If you need this for **kw arguments maybe you're not using them right;
why not name your arguments if you're going to reference them by name?

The JSON use case seems to be driven because this is the way
JavaScript does things -- they don't distinguish between dicts and
objects. I personally think that's a poor language feature: it blurs a
line that should be clear, between data (dict keys) and program
variables (attributes).

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Embedded Python startup is slow

[Stefan Behnel]

bruce bushby, 24.03.2011 18:39:

On Thu, Mar 24, 2011 at 5:05 PM, Stefan Behnel wrote:

bruce bushby, 24.03.2011 16:58:
  On my desktop pc, when I run the most simple "Hello World"  78% of the

overall execution time is spent opening filesmost of which don't
exist.


How did you measure that?


I used "strace -c hello.py"


Ok, I tried that as well, using a self-compiled CPython 3.2. On the first 
run, "open()" was at the top, on the subsequent runs, "read()" was. So it 
looks like a non-problem to me.


First run:

% time seconds  usecs/call callserrors syscall
-- --- --- - - 
100.000.26   0   677   581 open
  0.000.00   0   145   read
  0.000.00   0 1   write
   ...

Second run:

% time seconds  usecs/call callserrors syscall
-- --- --- - - 
100.000.16   0   145   read
  0.000.00   0 1   write
  0.000.00   0   677   581 open
   ...

The exact timings are likely bogus as the numbers are way too low to be 
measured accurately. The higher number of calls is partly due to the ABI 
version specific .so files that CPython 3.2 uses.




I'd expect that after the first startup, most of the relevant parts of the
file system are cached by the OS, so a subsequent run should be quick as all
file existence tests will run from memory. Sure, it's a bit of overhead to
call into the OS, and I don't know how expensive that is on ARM-Linux, but
given that you also claim it to be expensive on your desktop, I must say
that I'm a bit surprised about your above claim.


On my pc, the "Hello World" script obviously executes very quicklyso
trying to open 160 odd files that don't exist is negligible, but it still
happens.


That brings us to the question why your processes are so short-running that 
the startup time starts to matter. And, if they are so short-running, what 
prevents the disk cache (or whatever kind of storage system you use) from 
speeding up the open() requests. However, that's the right kind of question 
to discuss on python-list, not python-devel.


Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Andrew McNabb]

On Thu, Mar 24, 2011 at 10:12:18AM -0700, Guido van Rossum wrote:
> On Thu, Mar 24, 2011 at 9:46 AM, Benjamin Peterson  
> wrote:
> >> I want to test the dev community interest in modifying the stdlib to
> >> ease dependence injection.
> >
> > I, for one, am -1. Code shouldn't be uglified for the purposes of
> > testing.
> 
> Well, the philosophy of dependency injection requires exactly that.
> Personally I am on the fence; I've seen is used effectively but I've
> also seen code, indeed, uglified with little benefits. (Same for other
> extreme testing practices like TDD.)

I agree with the comments about uglification, but testing was only one
of the use cases that Jesus provided.  He also mentioned using
alternative network implementations, integrating with event loops, etc.
Another use case that I've run into before is setting socket options.

--
Andrew McNabb
http://www.mcnabbs.org/andrew/
PGP Fingerprint: 8A17 B57C 6879 1863 DE55  8012 AB4D 6098 8826 6868
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dict access with double-dot (syntactic sugar)

[Maciej Fijalkowski]

On Thu, Mar 24, 2011 at 11:50 AM, Guido van Rossum  wrote:
> On Thu, Mar 24, 2011 at 10:37 AM, Jameson Quinn  
> wrote:
>> OK, fair enough. People don't like this. So let me back up a step.
>
>> Clearly this is intended for using with things that you get as a dictionary,
>> but which really should be namespaces. The top two cases of that are parsed
>> json objects and **kw arguments.
>
> If you need this for **kw arguments maybe you're not using them right;
> why not name your arguments if you're going to reference them by name?
>
> The JSON use case seems to be driven because this is the way
> JavaScript does things -- they don't distinguish between dicts and
> objects. I personally think that's a poor language feature: it blurs a
> line that should be clear, between data (dict keys) and program
> variables (attributes).

As an almost completely unrelated note, it also makes writing fast
interpreter for JavaScript harder, because you don't know usage
patterns, while in Python it's clear when you meant objects and when
you meant dicts.

>
> --
> --Guido van Rossum (python.org/~guido)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Attributes access with dict

[Oleg Broytman]

On Thu, Mar 24, 2011 at 10:50:51AM -0700, Guido van Rossum wrote:
> The JSON use case seems to be driven because this is the way
> JavaScript does things -- they don't distinguish between dicts and
> objects.

   That's particular feature has a cure (or poison - for thos who don't
want to mix getattr and getitem): json.load/loads have object_hook
parameter that's used instead of dict. Pass DictRecord (or whatever you
call that) - and voila!

> I personally think that's a poor language feature: it blurs a
> line that should be clear, between data (dict keys) and program
> variables (attributes).

   Me too.

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dict access with double-dot (syntactic sugar)

[Jameson Quinn]

>
>
> If you need this for **kw arguments maybe you're not using them right;
> why not name your arguments if you're going to reference them by name?
>

Good point.

>
> The JSON use case seems to be driven because this is the way
> JavaScript does things -- they don't distinguish between dicts and
> objects. I personally think that's a poor language feature: it blurs a
> line that should be clear, between data (dict keys) and program
> variables (attributes).
>
> OK. So if this is a json-only issue, it can be done using a wrapper and
object_hook in the json library. I believe that, since all the magic
functions are pulled from the class, not the instance, you should even be
able to use  "if 'a' in wrapped_dict:" and "for a in wrapped_dict:" if your
wrapper class delegates __contains__ and __iter__ to the underlying dict.
Anyway, you can use hasattr for the former.

So my overarching proposal has shrunk to a suggestion that I should go and
make a attrlyjson module which does this by default. I certainly think that
I can get it right, and I know that there are several easy ways to get it
wrong, so it's worth saving people the headache.

Jameson
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dict access with double-dot (syntactic sugar)

[Bob Ippolito]

On Thu, Mar 24, 2011 at 11:46 AM, Jameson Quinn  wrote:
>>
>> If you need this for **kw arguments maybe you're not using them right;
>> why not name your arguments if you're going to reference them by name?
>
> Good point.
>>
>> The JSON use case seems to be driven because this is the way
>> JavaScript does things -- they don't distinguish between dicts and
>> objects. I personally think that's a poor language feature: it blurs a
>> line that should be clear, between data (dict keys) and program
>> variables (attributes).
>>
> OK. So if this is a json-only issue, it can be done using a wrapper and
> object_hook in the json library. I believe that, since all the magic
> functions are pulled from the class, not the instance, you should even be
> able to use  "if 'a' in wrapped_dict:" and "for a in wrapped_dict:" if your
> wrapper class delegates __contains__ and __iter__ to the underlying dict.
> Anyway, you can use hasattr for the former.
> So my overarching proposal has shrunk to a suggestion that I should go and
> make a attrlyjson module which does this by default. I certainly think that
> I can get it right, and I know that there are several easy ways to get it
> wrong, so it's worth saving people the headache.

You're correct, this is trivial with object_hook.

>>> class AttrDict(dict):
... def __getattr__(self, attr):
... try:
... return self[attr]
... except KeyError:
... raise AttributeError(attr)
...
>>> import json
>>> obj = json.loads('{"foo": {"bar": "baz"}}', object_hook=AttrDict)
{u'foo': {u'bar': u'baz'}}
>>> obj.foo.bar
u'baz'

-bob
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Michael Foord]

On 24/03/2011 17:46, Andrew McNabb wrote:

On Thu, Mar 24, 2011 at 10:12:18AM -0700, Guido van Rossum wrote:

On Thu, Mar 24, 2011 at 9:46 AM, Benjamin Peterson  wrote:

I want to test the dev community interest in modifying the stdlib to
ease dependence injection.

I, for one, am -1. Code shouldn't be uglified for the purposes of
testing.

Well, the philosophy of dependency injection requires exactly that.
Personally I am on the fence; I've seen is used effectively but I've
also seen code, indeed, uglified with little benefits. (Same for other
extreme testing practices like TDD.)

I agree with the comments about uglification, but testing was only one
of the use cases that Jesus provided.  He also mentioned using
alternative network implementations, integrating with event loops, etc.
Another use case that I've run into before is setting socket options.


And some of the use cases for this seemed pretty good from the email 
thread he linked to.


Michael


--
Andrew McNabb
http://www.mcnabbs.org/andrew/
PGP Fingerprint: 8A17 B57C 6879 1863 DE55  8012 AB4D 6098 8826 6868
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk



--
http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Attributes access with dict

[Eric Smith]

> On Thu, Mar 24, 2011 at 10:50:51AM -0700, Guido van Rossum wrote:
>> The JSON use case seems to be driven because this is the way
>> JavaScript does things -- they don't distinguish between dicts and
>> objects.
>
>That's particular feature has a cure (or poison - for thos who don't
> want to mix getattr and getitem): json.load/loads have object_hook
> parameter that's used instead of dict. Pass DictRecord (or whatever you
> call that) - and voila!
>
>> I personally think that's a poor language feature: it blurs a
>> line that should be clear, between data (dict keys) and program
>> variables (attributes).
>
>Me too.

Although we do something similar with namedtuple (instead of using a
dict), so it's not like we have a strict distinction.

Eric.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dict access with double-dot (syntactic sugar)

[Greg Ewing]

Jameson Quinn wrote:
"class attrdict" is a perennial dead-end for intermediate pythonistas 
who want to save 3 characters/5 keystrokes for item access. Other 
languages such as javascript allow "somedict.foo" to mean the same as 
"somedict['foo']", so why not python?


I think the main reason this is a dead end is these
intermediate pythonistas eventually come to realise that,
if you program pythonically, it's actually extremely
rare that you need to index a dictionary with a constant.

Either you have a mostly-fixed set of field names, in
which case you should be using a custom class instead of
a dict, or the set of keys is dynamic, in which case
you're mostly indexing with computed values. Lots of
somedict['foo'] appearing is a code smell.

So there wouldn't be enough use for a somedict..foo syntax
to justify its existence.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dict access with double-dot (syntactic sugar)

[skip]

Greg> Either you have a mostly-fixed set of field names, in which case
Greg> you should be using a custom class instead of a dict, or the set
Greg> of keys is dynamic, in which case you're mostly indexing with
Greg> computed values. Lots of somedict['foo'] appearing is a code
Greg> smell.

Unless, as I suspect, given the first message in the thread, the dict is
arriving via JSON.  Doesn't that only support the most basic builtin scalar
and container types?  You couldn't transfer class instances over JSON could
you?  As with XML-RPC I suspect your best fallback is a dictionary.

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dict access with double-dot (syntactic sugar)

[Santoso Wijaya]

I just want to chip in that, as far as syntactic sugar go, `somedict:foo`
looks better than `somedict..foo`.

2c...

~/santa


On Thu, Mar 24, 2011 at 4:40 AM, Jameson Quinn wrote:

> "class attrdict" is a perennial dead-end for intermediate pythonistas who
> want to save 3 characters/5 keystrokes for item access. Other languages such
> as javascript allow "somedict.foo" to mean the same as "somedict['foo']", so
> why not python? Well, there are a number of reasons why not, beginning with
> all the magic method names in python.
>
> But saving keystrokes is still a reasonable goal.
>
> So what about a compromise? Allow "somedict..foo", with two dots, to take
> that place. It still saves 2 characters (often 4 keystrokes; and I find even
> ', "[", or "]" harder to type than ".").
>
> The "foo" part would of course have to obey attribute/identifier naming
> rules. So there would be no shortcut for "somedict['$#!%']". But for any
> identifier-legal foo, the interpreter would just read ..foo as ['foo'].
>
> I would not be surprised if I'm not the first person to suggest this. If
> so, and there's already well-known reasons why this is a bad idea, I
> apologize. But if the only reason not to is "we never did it that way
> before" or "it would be too addictive, and so people would never want to use
> older python versions" or "headache for tools like pylint", I think we
> should do it.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/santoso.wijaya%40gmail.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dict access with double-dot (syntactic sugar)

[Greg Ewing]

Jameson Quinn wrote:


def fun2(**kw):
print kw["argument"]


Since this function effectively has a compulsory argument
called 'argument', it would be better written

  def fun2(argument, **kw):
print argument

or, if the recently-added keyword-only feature is available,

  def fun2(*, argument, **kw):
print argument

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Embedded Python startup is slow

[Santoso Wijaya]

On Thu, Mar 24, 2011 at 9:18 AM, James Y Knight  wrote:

> On Mar 24, 2011, at 11:58 AM, bruce bushby wrote:
> > My main concern was that a freshly compiled Python attempts to open 168
> non-existent files before starting.
> >
> > I understand that an interpreted language is probably not the best choice
> for an embedded device (although it's very nice for prototyping) , Python
> really should know what exists after it's just been compiledie before
> any corrupting modules or other nonsense has been added.
> >
> > It appears it is hard coded to open these files regardless of any
> "configure" options.
> >
> > On my desktop pc, when I run the most simple "Hello World"  78% of
> the overall execution time is spent opening filesmost of which don't
> exist.
> >
> > Some form of "cache" would help the startup time on the "second go" .
> but arguably just a "band aid" covering a deeper problem.
>
> The deeper problem on your Desktop PC being that MS-Windows' file system
> calls are horrifically expensive for no good reason? :)
>
>
Be that as it may, Windows is a major platform Python runs on and is
supported for. Limitations and quirky behaviors of each platform should be
taken into account, IMHO. And if expensive system calls are some of them, we
should take them into consideration.

~/santa
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dict access with double-dot (syntactic sugar)

[Greg Ewing]

Santoso Wijaya wrote:


`somedict:foo` looks better than `somedict..foo`.


Parsing ambiguity:

  if foo:bar:baz

Is that

  if (foo:bar): baz

or

  if foo: (bar:baz)

?

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dict access with double-dot (syntactic sugar)

[Jameson Quinn]

>
>
> You're correct, this is trivial with object_hook.
>
> >>> class AttrDict(dict):
> ... def __getattr__(self, attr):
> ... try:
> ... return self[attr]
> ... except KeyError:
> ... raise AttributeError(attr)
> ...
> >>> import json
> >>> obj = json.loads('{"foo": {"bar": "baz"}}', object_hook=AttrDict)
> {u'foo': {u'bar': u'baz'}}
> >>> obj.foo.bar
> u'baz'
>
> -bob
>

That's pretty good, but it does clone the dict unnecessarily. I prefer:

class AttrDict(object):
  def __init__(self, adict):
self.__dict__ = adict #a reference, not a copy
  def __getattr__(self, attr):
if hasattr(dict, attr): #built-in methods of dict...
  return getattr(self.__dict__, attr) #...are bound directly to the dict
else:
  try:
return self.__dict__[attr]
  except KeyError:
raise AttributeError(attr)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Embedded Python startup is slow

[Martin v. Löwis]

> My main concern was that a freshly compiled Python attempts to open 168
> non-existent files before starting. 

Please consider my proposals then.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Replace useless %.100s by %s in PyErr_Format()

[Martin v. Löwis]

> I would like to replace %.100s because there are no more reason to
> truncate strings to an arbitrary length.

I agree with MAL. It protects against cases with ridiculously long
parameters - say, you have a string with 1GB. You *want* to truncate
bogus text.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Replace useless %.100s by %s in PyErr_Format()

[Victor Stinner]

Le jeudi 24 mars 2011 à 13:22 +0100, M.-A. Lemburg a écrit :
> BTW: Why do you think that %.100s is not supported in
> PyErr_Format() in Python 2.x ? PyString_FromFormatV()
> does support this. The change to use Unicode error strings
> introduced the problem, since PyUnicode_FromFormatV() for
> some reason ignores the precision (which is shouldn't).

Oh... You are right, it is a regression in Python 3. We started to write
unit tests for PyBytes_FromFormat() and PyUnicode_FromFormat(), I hope
that they will improve the situation.

> That said, it's a good idea to add the #7330 fix
> to at least Python 2.7 as well, since ignoring the precision
> is definitely a bug. It may even be security relevant, since
> it could be used for DOS attacks on servers (e.g. causing them
> to write huge strings to log files instead of just a few
> hundreds bytes per message), so may even need to go into Python 2.6.

Python 2 is not affected because PyErr_Format() uses
PyString_FromFormatV() which supports precision for %s format (e.g.
%.100s truncate the string to 100 bytes).

Do you think that Python 3.1-3.3 should be fixed?

Let's begin by closing #10833 as invalid.

Victor

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Nick Coghlan]

On Fri, Mar 25, 2011 at 2:40 AM, Jesus Cea  wrote:
> What do you think?. Should I write a PEP? (I don't think so, but your
> opinion matters). I care, for instance, about how to garantee the API
> coverage actually needed for the new "socket-like" object. The idea is
> that your object should be "socket-like", but how much "socket-like" in
> the API sense?. Do you need to implement "getpeername()" for injecting
> in smtplib?.
>
> If you agree that this is a nice idea, what other libs do you think that
> could benefice of being "injected", beside "socket"?.

1. A PEP is definitely needed to thrash out how this API should work
in practice, as well as whether or not it should even be done at all

2. The level at which the dependency injection works (function
arguments, class attributes, module globals) needs to be decided

3. Alternative designs (such as standardised tools for thread-safe
monkey patching or other customisation of affected libraries) should
be considered

As an example of the last point, perhaps rather than modifying all the
*clients* of the socket module, it may make more sense to have tools
in the socket module itself to temporarily customise the socket
creation process in the current thread. The advantage of such an
approach is that it would then work for 3rd party libraries that
create sockets, without requiring any further modification.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Attributes access with dict

[Eugene Toder]

> Although we do something similar with namedtuple (instead of using a
> dict), so it's not like we have a strict distinction.

Named tuple is a convenience to avoid creating boilerplate classes (or
resorting to use dict with well-known keys). Names in named tuple are
not data, only values. In dict, names are data too.

Eugene
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Embedded Python startup is slow

[Paul Boddie]

bruce bushby wrote:
>
> My main concern was that a freshly compiled Python attempts to open 168
> non-existent files before starting.

This has been a longstanding problem with CPython and, despite assertions to 
the contrary, a significant factor on some embedded systems.

> I understand that an interpreted language is probably not the best choice
> for an embedded device (although it's very nice for prototyping) , Python
> really should know what exists after it's just been compiledie before
> any corrupting modules or other nonsense has been added.
>
> It appears it is hard coded to open these files regardless of any
> "configure" options.

You might want to look at the following Wiki page:

http://wiki.python.org/moin/EmbeddedPython

Since this topic has come up a few times before, I thought it might be time to 
collect references to it as well as to other topics that people doing 
embedded work might be interested in, along with the recurring problems that 
seem to get solved, probably all too frequently from scratch, outside the 
CPython development scene. (Just looking at an average cross-compilation 
issue in the tracker is likely to fill you with despair if you're waiting for 
a resolution in the official releases, unfortunately.)

Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API deprecations in Python 3, from a Python 2 perspective

[Arfrever Frehtes Taifersar Arahesis]

2011-03-19 03:42:54 Nick Coghlan napisał(a):
> 4. Anyone testing C extensions against the 3.2 alpha and beta releases
> must have either not used the PyCObject API in the first place, or
> else had already ported their code to use the PyCapsule API as
> necessary.

Gentoo Python maintainer had noticed build failures related to PyCObject, but
he wasn't treating these problems as a bug in Python.
https://bugs.gentoo.org/buglist.cgi?short_desc=Python%203.2%20PyCObject;short_desc_type=allwordssubstr;order=bug_id

-- 
Arfrever Frehtes Taifersar Arahesis


signature.asc
Description: This is a digitally signed message part.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Attributes access with dict

[Eric Smith]

On 3/24/2011 8:10 PM, Eugene Toder wrote:

Although we do something similar with namedtuple (instead of using a
dict), so it's not like we have a strict distinction.


Named tuple is a convenience to avoid creating boilerplate classes (or
resorting to use dict with well-known keys).


My point is that I don't see the distinction between avoiding using 
dicts with well-known keys and wanting to access a dict with attribute 
access (which is where this started). Seems the same to me.


Not that it matters much to me either way.

Eric.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Suggestion on back-porting - a getpass issue.

[Senthil Kumaran]

The issue is this:

http://bugs.python.org/issue11236
getpass.getpass does not respond to ctrl-c or ctrl-z


Python 2.5 had a behavior when a user pressed CTRL-C at the getpass
prompt, it would raise a KeyBoardInterrupt and CTRL-Z would background
it.

Python 2.6 onwards this behavior got changed and KeyBoardInterrupt
Exception is not raised and nor is CTRL-Z backgrounding behavior
observed.

Users have complained about this change in behavior and want the old
behavior back. Their reasoning for wanting 2.5 behavior seems fine.

The tracker link gives the details to the discussion.

Now, if we fix it in 3.3 (and document the change), should  it be
back-ported to other versions (3.2,3.1,2.7)?

We may end up in a state where only 2.6 has a unique behavior with
respect handling CTRL-C and CTRL-Z on getpass call.

If you have any strong opinion that it should not be back-ported,
please share.

-- 
Senthil
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Attributes access with dict

[Eugene Toder]

> My point is that I don't see the distinction between avoiding using dicts
> with well-known keys and wanting to access a dict with attribute access
> (which is where this started). Seems the same to me.

I think using dict with well-known keys is what makes people want dict
with attribute access.
My point is that this is not a very good use of dict -- normally dict
keys are also data. If they are not, something more specific should be
used, though before namedtuple (and still in many other scripting
languages) there were no good alternatives.

Eugene
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Eugene Toder]

> 2. The level at which the dependency injection works (function
> arguments, class attributes, module globals) needs to be decided

+1. The scope of parameter needs to be specified. If the scope is
global, this can be achieved pretty easily -- declare some of the
imports in particular modules to be a part of the public APIs of that
modules. Being part of the public API means allowing to replace that
module with another one with compatible implementations. In other
words, bless monkey-patching of those imports.
The advantages of this is simplicity and no uglification of the code
(no changes even). However, this should really not be used for more
than just testing. For other purposes the scope should be smaller
(e.g. class) and Antoine's approach looks like the right thing.

Eugene
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Unload a module written in C

[Victor Stinner]

Hi,

I am trying to understand why I am unable to unload my faulthandler
module (implemented in C). Antoine pointed me
_PyImport_FixupExtensionObject() comment which gave me a first clue:

   Modules which do support multiple initialization set their m_size
   field to a non-negative number (indicating the size of the
   module-specific state). They are still recorded in the extensions
   dictionary, to avoid loading shared libraries twice.

Ok, so I changed the size from -1 to 0, and so the m_free callback was
called at exit. Nice. This is thanks to PyImport_Cleanup() which clears
my module attributes.

--

But if I do

  import faulthandler
  del sys.modules['faulthandler']
  del faulthandler

the module is never unloaded. There is another secret reference in the
interpreter state: state->modules_by_index. This list is cleared at exit
(by PyInterpreterState_Clear), but not my module attributes, and some of
them are functions pointing to the module.

My module attribute are not cleared at exit because PyImport_Cleanup()
clears only modules from sys.modules, and my module is no more
referenced in sys.modules.

The workaround to unload the module is to explicitly clear its
attributes:

  import faulthandler
  del sys.modules['faulthandler']
  faulthandler.__dict__.clear()
  del faulthandler

--

Is there a bug somewhere, or do I misunderstood something important?

Note: I implemented m_traversal, but it is not revelant here (you can
consider that my module only contains functions).

Victor

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Glenn Linderman]

On 3/24/2011 4:25 PM, Nick Coghlan wrote:

As an example of the last point, perhaps rather than modifying all the
*clients*  of the socket module, it may make more sense to have tools
in the socket module itself to temporarily customise the socket
creation process in the current thread. The advantage of such an
approach is that it would then work for 3rd party libraries that
create sockets, without requiring any further modification.


Would be easier to implement that way, not requiring changes to every 
client of the socket library, but in some circles that would be called 
"action at a distance", and harder to understand.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Embedded Python startup is slow

[R. David Murray]

On Fri, 25 Mar 2011 01:20:34 +0100, Paul Boddie  wrote:
> Since this topic has come up a few times before, I thought it might be time 
> to 
> collect references to it as well as to other topics that people doing 
> embedded work might be interested in, along with the recurring problems that 
> seem to get solved, probably all too frequently from scratch, outside the 
> CPython development scene. (Just looking at an average cross-compilation 
> issue in the tracker is likely to fill you with despair if you're waiting for 
> a resolution in the official releases, unfortunately.)

Personally I think it would be a good thing for CPython to better support
both embedded systems and cross compilation in general.  I think that
many improvements that would make it better for embedded use would
also make it better for general use (such as improved startup time).
I also think we have a dearth of active core committers with experience
in cross compilation.

Paul, if this is a particular itch/knowledge area for you, maybe you
could do some spade work on the relevant issues, and see if we can get
some movement on them?  If we can get some good patches accepted from
one or more people with interest in this area, maybe we can get some
people with relevant knowledge onto the development team.

--
R. David Murray   http://www.bitdance.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sprints and pushes

[Stephen J. Turnbull]

Tres Seaver writes:

 > >  > > Well, keep in ming hg is a *distributed* version control system. You
 > >  > > don't have to push your changes right now.

 > >  > That doesn't work so well at a sprint, where the point is to maximize
 > >  > the value of precious face-time to get stuff done *now*.
 > > 
 > > That's where the D in DVCS comes in.  It's a new world, friends.  All
 > > you need to do is bring a $50 wireless router to the sprint, and have
 > > some volunteer set up a shared repo for the sprinters.  Then some
 > > volunteer *later* runs the tests and pilots the patches into the
 > > public repo.  Where's the latency?
 > 
 > The current full test suite is punishingly expensive to run, sprint
 > or not.  Because of that fact, people will defer running it, and
 > sometimes forget.  Trying to require that people run it repeatedly
 > during a push race is just Canute lashing the waves.

"Defer" is precisely the point of the proposal you are apparently
responding to, and "people forget" is why I propose a volunteer (or if
possible an automatic process, see my other post and Jesus Cea's) to
run the deferred tests.  I don't understand your point, unless it is
this:

 > The rhythm is still broken if developers don't run the tests and see them
 > pass.  Async is an enemy to the process here, because it encourages poor
 > practices.

Yes, but you can't have it both ways.  Either "the" tests (in fact, I
think people are referring to several different sets of tests by the
phrase "the tests", sometimes even in the same sentence) are valuable
and it's desirable that they always be run (I'm deliberately ignoring
cost here), or they're not, in which case they should never be run.
Now costs come back in: if the tests are valuable but too costly to
run during sprints, better late than never, IMO.  What am I missing?

Also, there's nothing here that says that developers can't run the
tests they think are relevant themselves.  But shouldn't that be their
choice if we are going to relax the requirement from the full test
suite to some subset?  After all, they're the ones closest to the
problem, they should be in the best position to decide which tests are
relevant.

 > >  > Maybe we need to chop the problem up as:
 > > 
 > > "Violence is the last refuge of the incompetent." ObRef Asimov.
 > 
 > Hmm, that hardly seems appropriate, even with the wink.  "Chopping"
 > isn't violence in any normal sense of the word when applied to
 > non-persons:  Chopping up a problem is no more violent than chopping
 > wood or onions (i.e., not at all).

Well, the referent of "violence" is the global nature of the proposal.
I don't think that one size fits all, here.  If you are going to argue
for running some tests but not others after making changes, shouldn't
there be a notion of relevance involved?  IMO "the" tests for modules
with dependents should include the tests for their dependents, for
example.  Modules that are leaves in the dependency tree presumably
can be unit tested and leave it at that.

Eg, much as I normally respect Barry's intuitions, his proposal (to
remove costly tests, without reference to the possibility of missing
something important) is IMHO absolutely the wrong criterion.  I don't
really know about Python's test suite, but in XEmacs the time-
expensive tests are mostly the ones that involve timeouts and lots of
system calls because they interact with the OS -- and those are
precisely the finicky areas where a small change can occasion an
unexpected bug.  For XEmacs, those bugs also are more likely than
average to be showstoppers for dependents, in the sense that until
they're fixed, the dependents can't be tested at all.  So the cost of
*not* running those tests is relatively high, too.

OTOH, there are a couple of expensive "sledgehammer" tests, eg, the
M17N tests that iterate over *all* the characters XEmacs knows
about.  (Seriously, once you've seen one of the 11,000 precomposed
Hangul, you've seen them all, and it's almost that bad for the 21,000
kanji.)  These could easily be omitted without harming anything else.

Yes, it would be a fair amount of work to do this kind of analysis.
That's why I propose some sort of deferred testing as an alternative
to a cost-based smoke test.

Another alternative would be to require unit testing of changed
modules only, which should be a pretty accurate heuristic for
relevance, except for modules with lots of dependencies.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggestion on back-porting - a getpass issue.

[Greg Ewing]

Senthil Kumaran wrote:


http://bugs.python.org/issue11236
getpass.getpass does not respond to ctrl-c or ctrl-z


Could this have been deliberate so that people can
put control characters in their passwords?

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggestion on back-porting - a getpass issue.

[Senthil Kumaran]

Greg Ewing wrote:
> 
> >http://bugs.python.org/issue11236
> >getpass.getpass does not respond to ctrl-c or ctrl-z
> 
> Could this have been deliberate so that people can
> put control characters in their passwords?

I don't think so. There are discussions in the internet which don't
favor use of control character in password. And anyway, it was not
intention to introduce control characters in password, for the change
in 2.6.

-- 
Senthil
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com