[Python-Dev] dictnotes.txt out of date?

[Eli Bendersky]

Hi pydev,

I was looking at the memory allocation strategy of dict, out of
curiosity, and noted that Objects/dictnotes.txt is out of date as far
as the parameters go. It says about PyDict_STARTSIZE:


* PyDict_STARTSIZE. Starting size of dict (unless an instance dict).
Currently set to 8. Must be a power of two.
New dicts have to zero-out every cell.
Increasing improves the sparseness of small dictionaries but costs
time to read in the additional cache lines if they are not already
in cache. That case is common when keyword arguments are passed.
Prior to version 3.3, PyDict_MINSIZE was used as the starting size
of a new dict.
-

Although it mentions 3.3, I find no reference to PyDict_STARTSIZE in
the code anywhere.
Also it mentions PyDict_MINSIZE, which doesn't exist any more - having
been replaced by Py_DICT_MINZISE_SPLIT and Py_DICT_COMBINED.

I don't know what else is out of date, just looked at those and they
were. Maybe it would make sense to kill dictnotes.txt, folding some of
its more important contents in to comments in dictobject.c, since the
latter has a higher chance of being maintained along with code
changes?

Eli
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] issue #15038 - Optimize python Locks on Windows

[Kristján Valur Jónsson]

I have reworked the patch, so it might be helpful to specify what exactly it is 
that you object to.  Perhaps in the defect itself.
I can add here that your worries that the previous patch defaulted to Vista 
specific features, were actually unfounded.
I've added my reasons for including vista specific support in in with the issue 
as well.  I happen to think it is a good idea and would
like to get the input from others on that particular issue.

Cheers,
K

> -Original Message-
> From: python-dev-bounces+kristjan=ccpgames@python.org
> [mailto:python-dev-bounces+kristjan=ccpgames@python.org] On
> Behalf Of Antoine Pitrou
> Sent: 13. júní 2012 00:26
> To: python-dev@python.org
> Subject: Re: [Python-Dev] issue #15038 - Optimize python Locks on Windows
> 
> 
> The review I made on your previous patch still applies.
> You shouldn't ask for feedback if you aren't willing to take it into 
> account...
> 
> Regards
> 
> Antoine.
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-
> dev/kristjan%40ccpgames.com


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] issue #15038 - Optimize python Locks on Windows

[Antoine Pitrou]

On Wed, 13 Jun 2012 13:47:28 +
Kristján Valur Jónsson  wrote:
> I have reworked the patch, so it might be helpful to specify what exactly it 
> is that you object to.  Perhaps in the defect itself.
> I can add here that your worries that the previous patch defaulted to Vista 
> specific features, were actually unfounded.

Ok, thanks. I probably have misunderstood the chain of #define's.
I will let other people comment on the other aspects of the patch.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] dictnotes.txt out of date?

[Mark Shannon]

Eli Bendersky wrote:

Hi pydev,

I was looking at the memory allocation strategy of dict, out of
curiosity, and noted that Objects/dictnotes.txt is out of date as far
as the parameters go. It says about PyDict_STARTSIZE:


* PyDict_STARTSIZE. Starting size of dict (unless an instance dict).
Currently set to 8. Must be a power of two.
New dicts have to zero-out every cell.
Increasing improves the sparseness of small dictionaries but costs
time to read in the additional cache lines if they are not already
in cache. That case is common when keyword arguments are passed.
Prior to version 3.3, PyDict_MINSIZE was used as the starting size
of a new dict.
-

Although it mentions 3.3, I find no reference to PyDict_STARTSIZE in
the code anywhere.
Also it mentions PyDict_MINSIZE, which doesn't exist any more - having
been replaced by Py_DICT_MINZISE_SPLIT and Py_DICT_COMBINED.


That's my fault. I didn't update dictnotes.txt when I changed 
PyDict_STARTSIZE to PyDict_MINSIZE_COMBINED.




I don't know what else is out of date, just looked at those and they
were. Maybe it would make sense to kill dictnotes.txt, folding some of
its more important contents in to comments in dictobject.c, since the
latter has a higher chance of being maintained along with code
changes?


I think that the parts of dictnotes.txt that just duplicate comments in
dictobject.c should be removed.
However, I think it is worth keeping dictnotes.txt as it has historical
information and results of previous experiments.

Cheers,
Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] dictnotes.txt out of date?

[Eli Bendersky]

>> I was looking at the memory allocation strategy of dict, out of
>> curiosity, and noted that Objects/dictnotes.txt is out of date as far
>> as the parameters go. It says about PyDict_STARTSIZE:
>>
>> 
>> * PyDict_STARTSIZE. Starting size of dict (unless an instance dict).
>>    Currently set to 8. Must be a power of two.
>>    New dicts have to zero-out every cell.
>>    Increasing improves the sparseness of small dictionaries but costs
>>    time to read in the additional cache lines if they are not already
>>    in cache. That case is common when keyword arguments are passed.
>>    Prior to version 3.3, PyDict_MINSIZE was used as the starting size
>>    of a new dict.
>> -
>>
>> Although it mentions 3.3, I find no reference to PyDict_STARTSIZE in
>> the code anywhere.
>> Also it mentions PyDict_MINSIZE, which doesn't exist any more - having
>> been replaced by Py_DICT_MINZISE_SPLIT and Py_DICT_COMBINED.
>
>
> That's my fault. I didn't update dictnotes.txt when I changed
> PyDict_STARTSIZE to PyDict_MINSIZE_COMBINED.

Could you update it now?

>> I don't know what else is out of date, just looked at those and they
>> were. Maybe it would make sense to kill dictnotes.txt, folding some of
>> its more important contents in to comments in dictobject.c, since the
>> latter has a higher chance of being maintained along with code
>> changes?
>
>
> I think that the parts of dictnotes.txt that just duplicate comments in
> dictobject.c should be removed.
> However, I think it is worth keeping dictnotes.txt as it has historical
> information and results of previous experiments.

Personally I think that describing the customization #defines belongs
in the source, above the relevant #defines, rather than in a separate
file. No problem with leaving historical notes and misc ruminations in
the separate .txt file, though.

Eli
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] what is happening with the regex module going into Python 3.3?

[Georg Brandl]

Am 04.06.2012 00:51, schrieb "Martin v. Löwis":
>> That last statement basically suggests that something like regex would
>> never be accepted until a CPython core developer was actually running
>> into pain with the many flaws in the re module (especially when it comes
>> to Unicode).  I disagree with that.
>>
>> Per the language summit, I think we need to just do it.  Put it in as re
>> and rename the existing re module to sre.
> 
> There are really places where "we" just doesn't work, even in a
> community project. "We" will never commit anything to revision control.
> Individual committers commit.
> 
> So if *you* want to commit it, go ahead - I think there is general
> approval for that. Take the praise when it works, and take the (likely)
> blame for when it fails in some significant way, and then work on fixing
> it.
> 
>> The issue seems to be primarily one of "who is volunteering to do it?"
> 
> I don't think anybody is, or will be for the coming years. I wish I had
> trust into MRAB to stay around and work on this for the next ten years
> (and I think the author of the regex module really needs to commit for
> that timespan, see SRE's history), but I don't. So whoever commits the
> change now is in charge, and will either have to work hard on fixing the
> problems, or will be responsible for breaking Python 3 in a serious way.

Agreed with all of the above.  (That's not a -1, but a warning.)

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [compatibility-sig] making sure importlib.machinery.SourceLoader doesn't throw an exception if bytecode is not supported by a VM

[Brett Cannon]

On Wed, Jun 13, 2012 at 1:18 AM, Nick Coghlan  wrote:

> On Wed, Jun 13, 2012 at 11:10 AM, Brett Cannon  wrote:
> >> > This does mean, though, that imp.cache_from_source() and
> >> > imp.source_from_cache() might need to be updated to raise a reasonable
> >> > exception when sys.implementation.cache_tag is set to None as I
> believe
> >> > right now it will raise a TypeError because None isn't a str. But what
> >> > to
> >> > raise instead? TypeError? EnvironmentError?
> >> NotImplementedError seems fine for me too if we don't end up using this
> >> flag.
> >
> > OK, that's 2 votes for that exception.
>
> + 1 from me as well, both for skipping any implicit reading or writing
> of the cache when cache_tag is None (IIRC, that's the use case we had
> in mind when we allowed that field to be None in the PEP 421
> discussion), and for *explicit* attempts to access the cache when the
> tag is None triggering NotImplementedError.
>
> That way people are free to use either LBYL (checking cache_tag) or
> EAFP (catching NotImplementedError).
>

I'm sold: http://bugs.python.org/issue15056 for tracking the change.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[Brett Cannon]

On Tue, Jun 12, 2012 at 2:16 PM, Terry Reedy  wrote:

> http://bugs.python.org/**issue12982 
>
> Currently, cpython requires the -O flag to *read* .pyo files as well as
> the write them. This is a nuisance to people who receive them from others,
> without the source. The originator of the issue quotes the following from
> the doc (without giving the location).
>
> "It is possible to have a file called spam.pyc (or spam.pyo when -O is
> used) without a file spam.py for the same module. This can be used to
> distribute a library of Python code in a form that is moderately hard to
> reverse engineer."
>
> There is no warning that .pyo files are viral, in a sense. The user has to
> use -O, which is a) a nuisance to remember if he has multiple scripts and
> some need it and some not, and b) makes his own .py files used with .pyo
> imports cached as .pyo, without docstrings, like it or not.
>
> Currently, the easiest workaround is to rename .pyo to .pyc and all seems
> to work fine, even with a mixture of true .pyc and renamed .pyo files. (The
> same is true with the -O flag and no renaming.) This suggests that there is
> no current reason for the restriction in that the *execution* of bytecode
> is not affected by the -O flag. (Another workaround might be a custom
> importer -- but this is not trivial, apparently.)
>

In Python 3.3 it's actually trivial.


>
> So is the import restriction either an accident or obsolete holdover?


Neither. .pyo files are actually different from .pyc files in terms of what
bytecode they may emit. Currently -O causes all asserts to be left out of
the bytecode and -OO leaves out all docstrings on top of what -O does. This
makes a difference if you are trying to introspect at the interpreter
prompt or are testing things in development and want those asserts to be
triggered if needed.


> If so, can removing it be treated as a bugfix and put into current
> releases, or should it be treated as an enhancement only for a future
> release?
>
>
The behaviour shouldn't change. There has been talk of doing even more
aggressive optimizing under -O, which once again would cause an even larger
deviation between a .pyo file and a .pyc file (e.g. allowing Python code to
hook into the peephole optimizer or an entirely new AST optimizer).


> Or is the restriction an intentional reservation of the possibility of
> making *execution* depend on the flag? Which would mean that the
> restriction should be kept and only the doc changed?


The docs should get updated to be more clear.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[R. David Murray]

On Wed, 13 Jun 2012 13:19:43 -0400, Brett Cannon  wrote:
> On Tue, Jun 12, 2012 at 2:16 PM, Terry Reedy  wrote:
> 
> > http://bugs.python.org/**issue12982 
> >
> > Currently, cpython requires the -O flag to *read* .pyo files as well as
> > the write them. This is a nuisance to people who receive them from others,
> > without the source. The originator of the issue quotes the following from
> > the doc (without giving the location).
[...]
> > So is the import restriction either an accident or obsolete holdover?
> 
> Neither. .pyo files are actually different from .pyc files in terms of what
> bytecode they may emit. Currently -O causes all asserts to be left out of
> the bytecode and -OO leaves out all docstrings on top of what -O does. This
> makes a difference if you are trying to introspect at the interpreter
> prompt or are testing things in development and want those asserts to be
> triggered if needed.
> 
> > If so, can removing it be treated as a bugfix and put into current
> > releases, or should it be treated as an enhancement only for a future
> > release?
> >
> The behaviour shouldn't change. There has been talk of doing even more
> aggressive optimizing under -O, which once again would cause an even larger
> deviation between a .pyo file and a .pyc file (e.g. allowing Python code to
> hook into the peephole optimizer or an entirely new AST optimizer).
> 
> > Or is the restriction an intentional reservation of the possibility of
> > making *execution* depend on the flag? Which would mean that the
> > restriction should be kept and only the doc changed?
> 
> The docs should get updated to be more clear.

OK, but you didn't answer the question :).  If I understand correctly,
everything you said applies to *writing* the bytecode, not reading it.

So, is there any reason to not use the .pyo file (if that's all that is
around) when -O is not specified?

The only technical reason I can see why -O should be required for a .pyo
file to be used (*if* it is the only thing around) is if it won't *run*
without the -O switch.  Is there any expectation that that will ever be
the case?

On the other hand, not wanting make any extra effort to support sourceless
distributions could be a reason as well.  But if that's the case we
should be transparent about it.

--David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[Toshio Kuratomi]

On Wed, Jun 13, 2012 at 01:58:10PM -0400, R. David Murray wrote:
> 
> OK, but you didn't answer the question :).  If I understand correctly,
> everything you said applies to *writing* the bytecode, not reading it.
> 
> So, is there any reason to not use the .pyo file (if that's all that is
> around) when -O is not specified?
> 
> The only technical reason I can see why -O should be required for a .pyo
> file to be used (*if* it is the only thing around) is if it won't *run*
> without the -O switch.  Is there any expectation that that will ever be
> the case?
> 
Yes.  For instance, if I create a .pyo with -OO it wouldn't have docstrings.
Another piece of code can legally import that and try to use the docstring
for something.  This would fail if only the .pyo was present.

Of course, it would also fail under the present behaviour since no .py or
.pyc was present to be imported.  The error that's displayed might be
clearer if we fail when attempting to read a .py/.pyc rather than failing
when the docstring is found to be missing, though.

-Toshio


pgpqk9ErpLKEV.pgp
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[Ethan Furman]

Brett Cannon wrote:

On Tue, Jun 12, 2012 at 2:16 PM, Terry Reedy wrote:


http://bugs.python.org/__issue12982 

Currently, cpython requires the -O flag to *read* .pyo files as well
as the write them. This is a nuisance to people who receive them
from others, without the source. The originator of the issue quotes
the following from the doc (without giving the location).

"It is possible to have a file called spam.pyc (or spam.pyo when -O
is used) without a file spam.py for the same module. This can be
used to distribute a library of Python code in a form that is
moderately hard to reverse engineer."

There is no warning that .pyo files are viral, in a sense. The user
has to use -O, which is a) a nuisance to remember if he has multiple
scripts and some need it and some not, and b) makes his own .py
files used with .pyo imports cached as .pyo, without docstrings,
like it or not.

Currently, the easiest workaround is to rename .pyo to .pyc and all
seems to work fine, even with a mixture of true .pyc and renamed
.pyo files. (The same is true with the -O flag and no renaming.)
This suggests that there is no current reason for the restriction in
that the *execution* of bytecode is not affected by the -O flag.
(Another workaround might be a custom importer -- but this is not
trivial, apparently.)


In Python 3.3 it's actually trivial.


So is the import restriction either an accident or obsolete holdover?


Neither. .pyo files are actually different from .pyc files in terms of 
what bytecode they may emit. Currently -O causes all asserts to be left 
out of the bytecode and -OO leaves out all docstrings on top of what -O 
does. This makes a difference if you are trying to introspect at the 
interpreter prompt or are testing things in development and want those 
asserts to be triggered if needed.


But what does this have to do with those cases where *only* the .pyo 
file is available, and we are trying to run it?  In these cases it would 
have to be in the main folder (not __pycache__) which means somebody did 
it deliberately.


~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[Antoine Pitrou]

On Wed, 13 Jun 2012 11:20:24 -0700
Toshio Kuratomi  wrote:
> On Wed, Jun 13, 2012 at 01:58:10PM -0400, R. David Murray wrote:
> > 
> > OK, but you didn't answer the question :).  If I understand correctly,
> > everything you said applies to *writing* the bytecode, not reading it.
> > 
> > So, is there any reason to not use the .pyo file (if that's all that is
> > around) when -O is not specified?
> > 
> > The only technical reason I can see why -O should be required for a .pyo
> > file to be used (*if* it is the only thing around) is if it won't *run*
> > without the -O switch.  Is there any expectation that that will ever be
> > the case?
> > 
> Yes.  For instance, if I create a .pyo with -OO it wouldn't have docstrings.
> Another piece of code can legally import that and try to use the docstring
> for something.  This would fail if only the .pyo was present.

Not only docstrings, but also asserts. I think running a pyo without -O
would be a bug.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[Ethan Furman]

Toshio Kuratomi wrote:

On Wed, Jun 13, 2012 at 01:58:10PM -0400, R. David Murray wrote:

OK, but you didn't answer the question :).  If I understand correctly,
everything you said applies to *writing* the bytecode, not reading it.

So, is there any reason to not use the .pyo file (if that's all that is
around) when -O is not specified?

The only technical reason I can see why -O should be required for a .pyo
file to be used (*if* it is the only thing around) is if it won't *run*
without the -O switch.  Is there any expectation that that will ever be
the case?


Yes.  For instance, if I create a .pyo with -OO it wouldn't have docstrings.
Another piece of code can legally import that and try to use the docstring
for something.  This would fail if only the .pyo was present.


Why should it fail?  -OO causes docstring access to return None, just as 
if a docstring had not been specified in the first place.  Any decent 
code will be checking for an undefined docstring -- after all, they are 
not rare.


~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[R. David Murray]

On Wed, 13 Jun 2012 11:20:24 -0700, Toshio Kuratomi  wrote:
> On Wed, Jun 13, 2012 at 01:58:10PM -0400, R. David Murray wrote:
> > 
> > OK, but you didn't answer the question :).  If I understand correctly,
> > everything you said applies to *writing* the bytecode, not reading it.
> > 
> > So, is there any reason to not use the .pyo file (if that's all that is
> > around) when -O is not specified?
> > 
> > The only technical reason I can see why -O should be required for a .pyo
> > file to be used (*if* it is the only thing around) is if it won't *run*
> > without the -O switch.  Is there any expectation that that will ever be
> > the case?
> > 
> Yes.  For instance, if I create a .pyo with -OO it wouldn't have docstrings.
> Another piece of code can legally import that and try to use the docstring
> for something.  This would fail if only the .pyo was present.

Yes, but that's not what I'm talking about.  I would treat code that
depends on the presence of docstrings and doesn't have a fallback for
dealing with there absence as buggy code, since anyone might decide
to run that code with -OO, and the code would fail in that case too.

I'm talking about a case where the code runs correctly with -O (or -OO),
but fails if the code from the .pyo is loaded and python is run
*without* -O (or -OO).

> Of course, it would also fail under the present behaviour since no .py or
> .pyc was present to be imported.  The error that's displayed might be
> clearer if we fail when attempting to read a .py/.pyc rather than failing
> when the docstring is found to be missing, though.

Well, right now if there is only a .pyo file and you run python without
-O, you get an import error.  The question is, is that the way we really
want it to work?

--David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[R. David Murray]

On Wed, 13 Jun 2012 20:46:50 +0200, Antoine Pitrou  wrote:
> On Wed, 13 Jun 2012 11:20:24 -0700
> Toshio Kuratomi  wrote:
> > On Wed, Jun 13, 2012 at 01:58:10PM -0400, R. David Murray wrote:
> > > 
> > > OK, but you didn't answer the question :).  If I understand correctly,
> > > everything you said applies to *writing* the bytecode, not reading it.
> > > 
> > > So, is there any reason to not use the .pyo file (if that's all that is
> > > around) when -O is not specified?
> > > 
> > > The only technical reason I can see why -O should be required for a .pyo
> > > file to be used (*if* it is the only thing around) is if it won't *run*
> > > without the -O switch.  Is there any expectation that that will ever be
> > > the case?
> > > 
> > Yes.  For instance, if I create a .pyo with -OO it wouldn't have docstrings.
> > Another piece of code can legally import that and try to use the docstring
> > for something.  This would fail if only the .pyo was present.
> 
> Not only docstrings, but also asserts. I think running a pyo without -O
> would be a bug.

Again, a program that depends on asserts is buggy.

As Ethan pointed out we are asking about the case where someone is
*deliberately* setting the .pyo file up to be run as the "normal"
case.

I'm not sure we want to support that, I just want us to be clear
about why we don't :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[Ethan Furman]

R. David Murray wrote:

On Wed, 13 Jun 2012 20:46:50 +0200, Antoine Pitrou  wrote:

On Wed, 13 Jun 2012 11:20:24 -0700
Toshio Kuratomi  wrote:

On Wed, Jun 13, 2012 at 01:58:10PM -0400, R. David Murray wrote:

OK, but you didn't answer the question :).  If I understand correctly,
everything you said applies to *writing* the bytecode, not reading it.

So, is there any reason to not use the .pyo file (if that's all that is
around) when -O is not specified?

The only technical reason I can see why -O should be required for a .pyo
file to be used (*if* it is the only thing around) is if it won't *run*
without the -O switch.  Is there any expectation that that will ever be
the case?


Yes.  For instance, if I create a .pyo with -OO it wouldn't have docstrings.
Another piece of code can legally import that and try to use the docstring
for something.  This would fail if only the .pyo was present.

Not only docstrings, but also asserts. I think running a pyo without -O
would be a bug.


Again, a program that depends on asserts is buggy.

As Ethan pointed out we are asking about the case where someone is
*deliberately* setting the .pyo file up to be run as the "normal"
case.

I'm not sure we want to support that, I just want us to be clear
about why we don't :)


Currently, the alternative to supporting this behavior is to either:

  1) require the end-user to specify -O (major nuisance)

  or

  2) have the distributor rename the .pyo file to .pyc

I think 1 is a non-starter (non-finisher? ;) but I could live with 2 -- 
after all, if someone is going to the effort of removing the .py file 
and moving the .pyo file into its place, renaming the .pyo to .pyc is 
trivial.


So the question, then, is: is option 2 better than just supporting .pyo 
files without -O when they are all that is available?


~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] dictnotes.txt out of date?

[Raymond Hettinger]

On Jun 13, 2012, at 10:35 AM, Eli Bendersky wrote:

> Did you mean to send this to the list, Raymond?


Yes.  I wanted to find-out whether someone approved changing
all the dict tunable parameters.   I thought those weren't supposed
to have changed.  PEP 412 notes that the existing parameters
were finely tuned and it did not make recommendations for changing them.

At one point, I spent a full month testing all of the tunable parameters
using dozens of popular Python applications.  The values used in Py3.2
represent the best settings for most apps.  They should not have been
changed without deep thought and substantial testing.

The reduction of the dict min size has an immediate impact on code
using multiple keyword arguments.

The reduced growth rate (from x4 to x2) negatively impacts apps that
have a dicts with a steady size but constantly changing keys
(removing an old key and adding a new one).  The lru_cache is
an example.   The reduced growth causes it to resize much more
frequently than before.

I think the tunable parameters should be switched back to what they
were before.  Tim and others spent a lot of time getting those right
and my month of detailed testing confirmed that those were excellent
choices.

The introduction of Mark's shared-key dicts was orthogonal to the
issue of correct parameter settings.  Those parameters did not have
to change and probably should not have been changed.


Raymond

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[Terry Reedy]

On 6/13/2012 2:46 PM, Antoine Pitrou wrote:


Not only docstrings, but also asserts. I think running a pyo without -O
would be a bug.


That cat is already out of the bag ;-)
People are doing that now by renaming x.pyo to x.pyc.
Brett claims that it is also easy to do in 3.3 with a custom importer.

--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[Terry Reedy]

On 6/13/2012 1:19 PM, Brett Cannon wrote:


On Tue, Jun 12, 2012 at 2:16 PM, Terry Reedy mailto:tjre...@udel.edu>> wrote:

http://bugs.python.org/__issue12982 

Currently, cpython requires the -O flag to *read* .pyo files as well
as the write them. This is a nuisance to people who receive them
from others, without the source. The originator of the issue quotes
the following from the doc (without giving the location).

"It is possible to have a file called spam.pyc (or spam.pyo when -O
is used) without a file spam.py for the same module. This can be
used to distribute a library of Python code in a form that is
moderately hard to reverse engineer."

There is no warning that .pyo files are viral, in a sense. The user
has to use -O, which is a) a nuisance to remember if he has multiple
scripts and some need it and some not, and b) makes his own .py
files used with .pyo imports cached as .pyo, without docstrings,
like it or not.

Currently, the easiest workaround is to rename .pyo to .pyc and all
seems to work fine, even with a mixture of true .pyc and renamed
.pyo files. (The same is true with the -O flag and no renaming.)
This suggests that there is no current reason for the restriction in
that the *execution* of bytecode is not affected by the -O flag.
(Another workaround might be a custom importer -- but this is not
trivial, apparently.)


In Python 3.3 it's actually trivial.


For you. Anyway, I am sure Michael of #12982 is using an earlier version.


So is the import restriction either an accident or obsolete holdover?


Neither. .pyo files are actually different from .pyc files in terms of
what bytecode they may emit. Currently -O causes all asserts to be left
out of the bytecode and -OO leaves out all docstrings on top of what -O
does. This makes a difference if you are trying to introspect at the
interpreter prompt or are testing things in development and want those
asserts to be triggered if needed.


I suggested to Michael that he should request an all-.pyc library for 
that reason.



If so, can removing it be treated as a bugfix and put into current
releases, or should it be treated as an enhancement only for a
future release?


The behaviour shouldn't change. There has been talk of doing even more
aggressive optimizing under -O, which once again would cause an even
larger deviation between a .pyo file and a .pyc file (e.g. allowing
Python code to hook into the peephole optimizer or an entirely new AST
optimizer).


Would such a change mean that *reading* a .pyo file as if it were a .pyc 
file would start failing? (It now works, by renaming the file.)
If so, would not it be better to rely on having a different magic number 
*in* the file rather than on its mutable external name?


You just said above that evading import restriction by name is now 
trivial. So what is the point of keeping it. If, in the future, there 
*are* separate execution pathways*, and we want the .pyo pathway closed 
unless -O is passed, then it seems that that could only be enforced by a 
magic number in the file.


*Unladen Swallow would have *optionally* produced cache files completely 
different from current bytecode, with a different extension. A couple of 
people have suggested using wordcode instead of bytecode. If this were 
also introduced as an option, its cache files would also need a 
different extension and magic number.



Or is the restriction an intentional reservation of the possibility
of making *execution* depend on the flag? Which would mean that the
restriction should be kept and only the doc changed?

The docs should get updated to be more clear.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Tunable parameters in dictobject.c (was dictnotes.txt out of date?)

[Mark Shannon]

Raymond Hettinger wrote:


On Jun 13, 2012, at 10:35 AM, Eli Bendersky wrote:


Did you mean to send this to the list, Raymond?



Yes.  I wanted to find-out whether someone approved changing
all the dict tunable parameters.   I thought those weren't supposed
to have changed.  PEP 412 notes that the existing parameters
were finely tuned and it did not make recommendations for changing them.


The PEP says that the current (3.2) implementation is finely tuned.
No mention is made of tunable parameters.



At one point, I spent a full month testing all of the tunable parameters
using dozens of popular Python applications.  The values used in Py3.2
represent the best settings for most apps.  


Best in terms of speed, but what about memory use?


They should not have been
changed without deep thought and substantial testing.


I have thought about it, a lot :)
I have also tested it, using the Unladen Swallow benchmarks.
Additional (realistic) tests would be appreciated.



The reduction of the dict min size has an immediate impact on code
using multiple keyword arguments.


There is no change to the minimum size for combined-table (old style)
dictionaries.



The reduced growth rate (from x4 to x2) negatively impacts apps that
have a dicts with a steady size but constantly changing keys
(removing an old key and adding a new one).  The lru_cache is
an example.   The reduced growth causes it to resize much more
frequently than before.


Resizing was probably the part of the implementation that took the most 
time.


I had gained the impression from comments on this list,
comments on the tracker and the web in general that the memory 
consumption of CPython was more of an issue than its performance.

So my goal was to minimise memory usage without any significant slowdown
(< 1% slowdown).

The problem with resizing is that you don't know when it is going to 
stop. A bigger growth factor means fewer resizes, but more of a 
potential overshoot (a larger final size than required).

So in order to reduce memory use a growth factor of x2 is required.

Split-table dicts are (to a first-order approximation) never resized
so the growth factor for them should as small as possible; x2.

Combined-table dicts are more challenging. Reducing the growth rate to 
x2 increases the number of resizes by x2 or (x2-1) *and* increases the 
number of items per resize by about 50%.
But it is not the number of resizes that matters, it is the time spent 
performing those resizes.

I went to considerable effort to improve the performance of dictresize()
so that even benchmarks that create a lot of short-lived medium-to-large 
 sized combined-table dicts do not suffer impaired performance.


It would be possible to split the growth factor into two: one for 
split-tables (which would be x2) and one for combined tables.


But which is better for combined tables, x4 or x2?
What is the relative value of speed and memory consumption?
50% less memory and 1% slower is good. 1% less memory and 50% slower is 
bad. But what about intermediate values?


I think that for combined tables a growth factor of x2 is best,
but I don't have any hard evidence to back that up.



I think the tunable parameters should be switched back to what they
were before.  Tim and others spent a lot of time getting those right
and my month of detailed testing confirmed that those were excellent
choices.


They may have been excellent choices for the previous implementation.
They are not necessarily the best for the new implementation.

The current parameters seem to be the best for the new implementation.
When Django and Twisted run on Python3, then it might be worthwhile to
do some more experimentation.



The introduction of Mark's shared-key dicts was orthogonal to the
issue of correct parameter settings.  Those parameters did not have
to change and probably should not have been changed.


Parameters for tuning code and the code itself are unlikely to be
orthogonal. While I did strive to minimise the impact of the changes on
combined-table dicts, the performance characteristics have necessarily
changed.

Cheers,
Mark.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] backporting stdlib 2.7.x from pypy to cpython

[Cameron Simpson]

On 11Jun2012 15:35, PJ Eby  wrote:
| Yes, perhaps if the list were *just* a place to cc: in or send a heads-up
| to python-dev discussions, and not to have actual list discussions per se,
| that would do the trick.

This approach has its own problems. Is the proposed list, like many lists,
restricted to accept posts only from subscribers? If that is the case,
when someone CCs the VM list, everyone honouring the CC in replies needs
to be a VM list member if they are not to get annoying bounce messages. It
is a core reason that mailing list cross posting is discouraged in many
lists; the receivers are not all members of all the cross posted lists,
and so get lots of shrapnel when they try to participate.

Personally, I am _for_ cross posting (when on topic), except for this
unfortunate issue.

| IOW, the idea is, "If you're a contributor to a non-CPython implementation,
| subscribe here to get a heads-up on Python-Dev discussions you should be
| following."  Not, "here's a list to discuss Python implementations in
| general", and definitely not a place to *actually conduct discussions* at
| all:

+1 to this, but how to keep this etiquette maintained?

| the only things ever posted there should be cc:'d from or to
| Python-Dev, or be pointers to Python-Dev threads.

And (premised on my concern above), do people wanting to CC: the VM list
for the heads-up purpose need to join it first?

Conversely, some of this discussion mentions that people don't subscribe
to python-dev; do they need to subscribe to chime in when the bat signal
goes off?

To reiterate, I'm all for the bat signal, but will there be shrapnel?

Hackish idea: suppose there were a special purpose mail forwarder, like
a write-only mailing list? It would require special Mailman hackery, but
imagine:

  - a bat-signal list, which rejected posts not from members of
python-dev

  - accepted messages get forwarded to all the relevant VM-specific lists,
but with a rewritten "From:" line of "bat-sig...@python.org" or such, and
we subscribe _that_ address to the relevant lists to allow it in.
And replies directed to "<>", as done with bounce messages, perhaps?
Or python-dev, probably better.

  - if the rewritten "From:" address is the bat-signal list itself
(pleasing), the copies sent back are dropped (special hack - drop inbound
from the list address)

This mechanical approach would get you access control to block spam to
the bat-signal and send alerts to the other lists, and send discussion
back to python-dev.

Cheers,
-- 
Cameron Simpson 

Nothing is impossible for the man who doesn't have to do it.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] TZ-aware local time

[Alexander Belopolsky]

On Tue, Jun 12, 2012 at 1:14 AM, Ben Finney  wrote:
>> To the contrary, without the POSIX timestamp model to define the
>> equivalency between the same point in time expressed using different
>> timezones, sane comparisons and arithmetic on timestamps would be
>> impossible.
>
> Why is the POSIX timestamp model the only possible model? To the
> contrary, there are many representations with different tradeoffs but
> with the common properties you name (“equivalency between the same point
> in time expressed using different timezones”).

Here is my take on this.  If datetime objects did not support
fractions of a second, the difference between naive datetime and
POSIX timestamp would be just a matter of tradeoff between human
readability and efficient storage and fast calculations.  This is very
similar to a more familiar tradeoff between decimal and binary
representation of numbers.  Binary arithmetics is more efficient in
terms of both storage and processing, so it is common to convert from
decimal to binary on input and convert back on output.  In some
applications (e.g. hand-held calculators), I/O dominates internal
processing, so implementing direct decimal arithmetics is not
uncommon.   The original designers of the datetime module chose the
"decimal" internal format.  Equivalent functionality can be
implemented using "binary" format and in fact popular mxDateTime
library is implemented that way.

(Since we do support  fractions of a second, there may be small
difference in calculations performed using datetime types and float
timestamps, but this issue has nothing to do with time zones, local
time or UTC.)

It is a common misconception that POSIX timestamps are somehow more
closely tied to UTC than broken down time values.  The opposite is
true.  at the end of this month, UTC clocks (e.g.
http://tycho.usno.navy.mil/simpletime.html) will show 2012-06-30
23:59:59, 2012-06-30 23:59:60, 2012-06-31 00:00:00, while
corresponding POSIX timestamps are 1341100799, 1341100800, 1341100800.
 Most POSIX systems will not freeze their clocks for a whole second or
move them back, but instead they will either be one second ahead until
someone (or something like the NTP daemon) causes the adjustment.

In an earlier message, Guido criticized the practice of converting
local broken down time to an integer using POSIX algorithm for
calculations.  While keeping time using local timezone is often not
the best choice, there is nothing wrong with implementing local time
arithmetics using timegm/gmtime conversion to integers.   In fact, the
results will be exactly the same as if the calculations were performed
using datetime module with tzinfo=LocalTimezone.

I think many users are confused by references to "Seconds Since the
Epoch" and think that time_t should contain an actual number of SI
seconds elapsed since a world-wide radio broadcast of
"1970-01-01T00:00:00+".  First of all, there was no such broadcast
in 1970, but the time known as UTC 1970-01-01T00:00:00+ was 35 or
so seconds earlier than time.time() seconds ago.

The POSIX standard gets away hiding the true nature of UTC by defining
the integral timestamp as "a value that *approximates* the number of
seconds that have elapsed since the Epoch."

  The choice of approximation is specified by an explicit formula:

tm_sec + tm_min*60 + tm_hour*3600 + tm_yday*86400 +
(tm_year-70)*31536000 + ((tm_year-69)/4)*86400 -
((tm_year-1)/100)*86400 + ((tm_year+299)/400)*86400

This formula can be used to pack any broken down time value in an
integer.  As long as one does not care about leap seconds, any broken
down time value can be converted to an integer and back using this
formula.  If the broken down value was in local time, the values
stored in an integer will represent local time.  If the broken down
value was in UTC, the values stored in an integer will represent UTC.
Either value will "approximate the number of seconds that have elapsed
since the Epoch" given the right definition of the Epoch and
sufficient tolerance for the approximation.

The bottom line: POSIX time stamps and datetime objects (interpreted
as UTC) implement the same timescale.  The only difference is that
POSIX timestamps can be stored using fewer bytes at the expense of not
having an obvious meaning.  While conversion from UTC to local time
may require using POSIX timestamps internally, there is no need to
expose them in datetime module interfaces.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[Steven D'Aprano]

On Wed, Jun 13, 2012 at 01:58:10PM -0400, R. David Murray wrote:

> So, is there any reason to not use the .pyo file (if that's all that is
> around) when -O is not specified?

.pyo and .pyc files have potentially different semantics. Right now, 
.pyo files don't include asserts, so that's one difference right there. 
In the future there may be more aggressive optimizations.

Good practice is to never write an assert that actually changes the 
semantics of your program, but in practice people don't write asserts 
correctly, e.g. they use them for checking user-input or function 
parameters.

So, no, we should never use .pyo files unless explicitly told to do so, 
since doing so risks breaking poorly-written but otherwise working code.



-- 
Steven
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tunable parameters in dictobject.c (was dictnotes.txt out of date?)

[Raymond Hettinger]

On Jun 13, 2012, at 2:37 PM, Mark Shannon wrote:

> I think that for combined tables a growth factor of x2 is best,
> but I don't have any hard evidence to back that up.


I believe that change should be reverted.  
You've undone work that was based on extensive testing and timings of many 
python apps.
In particular, it harms the speed of building-up all large dictionaries,
and it greatly harms apps with steady-size dictionaries with changing keys.

The previously existing parameter were well studied
and have been well-reviewed by the likes of Tim Peters.
They shouldn't be changed without deep thought and study.
Certainly, "I think a growth factor of x2 is best" is insufficient.


Raymond

P.S.  In the tracker discussion of key-sharing dict, you were asked
to NOT change the tunable parameters.  I'm not sure why you
went ahead and did it anyway.___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] backporting stdlib 2.7.x from pypy to cpython

[Stephen J. Turnbull]

Cameron Simpson writes:

 > This approach has its own problems. Is the proposed list, like many lists,
 > restricted to accept posts only from subscribers? If that is the case,
 > when someone CCs the VM list, everyone honouring the CC in replies needs
 > to be a VM list member if they are not to get annoying bounce
 > messages.

Mailman has a feature called "sibling lists", which seems to include
cross-checking membership on posts, although I'm not sure if it is
tuned to exactly this purpose.

In any cases, if the proposed list is not a discussion list, it can be
configured not to send bounce messages just because somebody honored
CC.  For example, by keying on the Reference and In-Reply-To headers,
and discarding the message if they are present (possible by ordinary
configuration of the spam filters).  For bonus points, bounce such
messages when python-dev is not present among the visible addressees.
(Might require a special Handler, but it wouldn't be a big deal to
write it because it can be installed only for the new list.)

 > +1 to this, but how to keep this etiquette maintained?

A filter on the new list, implemented as above.  It's pretty much
trivial for those with the know-how.

 > And (premised on my concern above), do people wanting to CC: the VM list
 > for the heads-up purpose need to join it first?

Probably not, but this is hardly burdensome; many people with the
background to know when to CC: may want to subscribe anyway even if
they are subscribed to python-dev, and traffic should be quite low.
If even so that's a bother, they can set their subscription to no-mail.

 > Conversely, some of this discussion mentions that people don't subscribe
 > to python-dev; do they need to subscribe to chime in when the bat signal
 > goes off?

Maybe not: I believe it's possible to post to python-dev via Gmane if
you're not subscribed.  Even if they need to be subscribed, there is a
wealth of options, including Gmane and the archives, for reading
traffic without receiving it as mail (ie, by subscribing and then
setting the no-mail flag).

 > Hackish idea: suppose there were a special purpose mail forwarder,
 > like a write-only mailing list? It would require special Mailman
 > hackery,

Not that special.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] backporting stdlib 2.7.x from pypy to cpython

[Brian Curtin]

On Jun 13, 2012 8:31 PM, "Stephen J. Turnbull"  wrote:
>
> Cameron Simpson writes:
>
>  > This approach has its own problems. Is the proposed list, like many
lists,
>  > restricted to accept posts only from subscribers? If that is the case,
>  > when someone CCs the VM list, everyone honouring the CC in replies
needs
>  > to be a VM list member if they are not to get annoying bounce
>  > messages.
>
> Mailman has a feature called "sibling lists", which seems to include
> cross-checking membership on posts, although I'm not sure if it is
> tuned to exactly this purpose.
>
> In any cases, if the proposed list is not a discussion list, it can be
> configured not to send bounce messages just because somebody honored
> CC.  For example, by keying on the Reference and In-Reply-To headers,
> and discarding the message if they are present (possible by ordinary
> configuration of the spam filters).  For bonus points, bounce such
> messages when python-dev is not present among the visible addressees.
> (Might require a special Handler, but it wouldn't be a big deal to
> write it because it can be installed only for the new list.)
>
>  > +1 to this, but how to keep this etiquette maintained?
>
> A filter on the new list, implemented as above.  It's pretty much
> trivial for those with the know-how.
>
>  > And (premised on my concern above), do people wanting to CC: the VM
list
>  > for the heads-up purpose need to join it first?
>
> Probably not, but this is hardly burdensome; many people with the
> background to know when to CC: may want to subscribe anyway even if
> they are subscribed to python-dev, and traffic should be quite low.
> If even so that's a bother, they can set their subscription to no-mail.
>
>  > Conversely, some of this discussion mentions that people don't
subscribe
>  > to python-dev; do they need to subscribe to chime in when the bat
signal
>  > goes off?
>
> Maybe not: I believe it's possible to post to python-dev via Gmane if
> you're not subscribed.  Even if they need to be subscribed, there is a
> wealth of options, including Gmane and the archives, for reading
> traffic without receiving it as mail (ie, by subscribing and then
> setting the no-mail flag).
>
>  > Hackish idea: suppose there were a special purpose mail forwarder,
>  > like a write-only mailing list? It would require special Mailman
>  > hackery,
>
> Not that special.
>

Can someone create python-dev-meta?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[Nick Coghlan]

On Thu, Jun 14, 2012 at 6:06 AM, Terry Reedy  wrote:
> On 6/13/2012 2:46 PM, Antoine Pitrou wrote:
>
>> Not only docstrings, but also asserts. I think running a pyo without -O
>> would be a bug.
>
>
> That cat is already out of the bag ;-)
> People are doing that now by renaming x.pyo to x.pyc.
> Brett claims that it is also easy to do in 3.3 with a custom importer.

Right, but by resorting to either of those approaches, people are
clearly doing something that isn't formally supported by the core.
Yes, you can do it, and most of the time it will work out OK, but any
weird glitches that result are officially *not our problem*.

The main reason this matters is that the "__debug__" flag is
*supposed* to be process global - if you check it in one place, the
answer should be correct for all Python code loaded in the process. If
you load a .pyo file into a process running without -O (or a .pyc file
into a process running *with* -O), then you have broken that
assumption. Because the compiler understands __debug__, and is
explicitly free to make optimisations based on the value of that flag
at compile time (such as throwing away unreachable branches in if
statements or applying constant folding operations), the following
code will do different things if loaded from a .pyo file instead of
.pyc:

print("__debug__ is not a builtin, it is checked at compile time")
if __debug__:
print("A .pyc file always has __debug__ == True")
else:
print("A .pyo file always has __debug__ == False")

$ ./python -c "import foo"
__debug__ is not a builtin, it is checked at compile time
A .pyc file always has __debug__ == True
$ ./python -O -c "import foo"
__debug__ is not a builtin, it is checked at compile time
A .pyo file always has __debug__ == False
$ ./python __pycache__/foo.cpython-33.pyo
__debug__ is not a builtin, it is checked at compile time
A .pyo file always has __debug__ == False
$ ./python -O __pycache__/foo.cpython-33.pyc
__debug__ is not a builtin, it is checked at compile time
A .pyc file always has __debug__ == True

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[Terry Reedy]

On 6/13/2012 8:55 PM, Steven D'Aprano wrote:

On Wed, Jun 13, 2012 at 01:58:10PM -0400, R. David Murray wrote:


So, is there any reason to not use the .pyo file (if that's all that is
around) when -O is not specified?


.pyo and .pyc files have potentially different semantics. Right now,
.pyo files don't include asserts, so that's one difference right there.
In the future there may be more aggressive optimizations.

Good practice is to never write an assert that actually changes the
semantics of your program, but in practice people don't write asserts
correctly, e.g. they use them for checking user-input or function
parameters.

So, no, we


You mean the interpreter?


should never use


Do you mean import or execute?
Current, the interpreter executes any bytecode that gets imported.


.pyo files unless explicitly told to do so,


What constitutes 'explicitly told to do so'? Currently, an 'optimized' 
file written as .pyo gets imported (and hence executed) if

1) the interpreter is started with -O
2) a custom importer ignores the absence of -O
3) someone renames x.pyo to x.pyc.


since doing so risks breaking poorly-written but otherwise working code.


Agreed, though a slightly different issue. Would you somehow disable 2) 
or 3) if not considered 'explicit' enough?


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tunable parameters in dictobject.c (was dictnotes.txt out of date?)

[Terry Reedy]

On 6/13/2012 9:15 PM, Raymond Hettinger wrote:


On Jun 13, 2012, at 2:37 PM, Mark Shannon wrote:


I think that for combined tables a growth factor of x2 is best,
but I don't have any hard evidence to back that up.


I believe that change should be reverted.
You've undone work that was based on extensive testing and timings of
many python apps.
In particular, it harms the speed of building-up all large dictionaries,
and it greatly harms apps with steady-size dictionaries with changing keys.

The previously existing parameter were well studied
and have been well-reviewed by the likes of Tim Peters.
They shouldn't be changed without deep thought and study.
Certainly, "I think a growth factor of x2 is best" is insufficient.


The use cases 'directly used dict that is possibly the primary data 
structure and possibly gigabytes in size' is sufficiently different from 
'hidden instance dicts that are typically a small, constant size (for a 
given class) but possibly repeated a million times' are sufficiently 
different that different tuning parameters might be appropriate. So I 
agree with Raymond think you should leave the parameters for the 
standard dicts alone until there is good evidence for a change. (Good == 
apparent to more than just one person ;-).


If you improved the resize speed of standard dicts, great. However, that 
does not necessarily mean that the resize factor should change.


[clipped by Raymond]

>> I had gained the impression from comments on this list,
>> comments on the tracker and the web in general that the memory
>> consumption of CPython was more of an issue than its performance.

In 16 years, I seemed to have missed that. I assure you there are many 
who feel the opposite.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[R. David Murray]

On Thu, 14 Jun 2012 11:48:08 +1000, Nick Coghlan  wrote:
> On Thu, Jun 14, 2012 at 6:06 AM, Terry Reedy  wrote:
> > On 6/13/2012 2:46 PM, Antoine Pitrou wrote:
> >
> >> Not only docstrings, but also asserts. I think running a pyo without -O
> >> would be a bug.
> >
> > That cat is already out of the bag ;-)
> > People are doing that now by renaming x.pyo to x.pyc.
> > Brett claims that it is also easy to do in 3.3 with a custom importer.
> 
> Right, but by resorting to either of those approaches, people are
> clearly doing something that isn't formally supported by the core.
> Yes, you can do it, and most of the time it will work out OK, but any
> weird glitches that result are officially *not our problem*.
> 
> The main reason this matters is that the "__debug__" flag is
> *supposed* to be process global - if you check it in one place, the

OK, the above are the two concrete reasons I have heard in this thread
for continuing the current behavior:

1) we do not wish to support running from .pyo files without -O
   being on, even if it currently happens to work

2) the __debug__ setting is supposed to be process-global

Both of these are good reasons.  IMO the issue should be closed with a
documentation fix, which could optionally include either or both of the
above motivations.

--David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 362 Third Revision

[Yury Selivanov]

Hello,

The new revision of PEP 362 has been posted:
http://www.python.org/dev/peps/pep-0362/


Summary:

1. Signature object now represents the call signature of a
function.  That said, it doesn't have 'name' and 'qualname'
attributes anymore, and can be tested for equality against
other signatures.

2. signature() function support all kinds of callables:
classes, metaclasses, methods, class- & staticmethods,
'functools.partials', and callable objects.  If a callable
object has a '__signature__' attribute it does a deepcopy
of it before return.  

3. No implicit caching to __signature__.

4. Added 'Signature.bind_partial' and 'Signature.format'
methods.

A patch with the PEP implementation is attached to the
issue 15008.  It should be ready for code review.


Thank you!



PEP: 362
Title: Function Signature Object
Version: $Revision$
Last-Modified: $Date$
Author: Brett Cannon , Jiwon Seo ,
Yury Selivanov , Larry Hastings 

Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 21-Aug-2006
Python-Version: 3.3
Post-History: 04-Jun-2012


Abstract


Python has always supported powerful introspection capabilities,
including introspecting functions and methods (for the rest of
this PEP, "function" refers to both functions and methods).  By
examining a function object you can fully reconstruct the function's
signature.  Unfortunately this information is stored in an inconvenient
manner, and is spread across a half-dozen deeply nested attributes.

This PEP proposes a new representation for function signatures.
The new representation contains all necessary information about a function
and its parameters, and makes introspection easy and straightforward.

However, this object does not replace the existing function
metadata, which is used by Python itself to execute those
functions.  The new metadata object is intended solely to make
function introspection easier for Python programmers.


Signature Object


A Signature object represents the call signature of a function and
its return annotation.  For each parameter accepted by the function
it stores a `Parameter object`_ in its ``parameters`` collection.

A Signature object has the following public attributes and methods:

* return_annotation : object
The annotation for the return type of the function if specified.
If the function has no annotation for its return type, this
attribute is not set.
* parameters : OrderedDict
An ordered mapping of parameters' names to the corresponding
Parameter objects (keyword-only arguments are in the same order
as listed in ``code.co_varnames``).
* bind(\*args, \*\*kwargs) -> BoundArguments
Creates a mapping from positional and keyword arguments to
parameters.  Raises a ``BindError`` (subclass of ``TypeError``)
if the passed arguments do not match the signature.
* bind_partial(\*args, \*\*kwargs) -> BoundArguments
Works the same way as ``bind()``, but allows the omission
of some required arguments (mimics ``functools.partial``
behavior.)
* format(...) -> str
Formats the Signature object to a string.  Optional arguments allow
for custom render functions for parameter names,
annotations and default values, along with custom separators.

Signature implements the ``__str__`` method, which fallbacks to the
``Signature.format()`` call.

It's possible to test Signatures for equality.  Two signatures
are equal when they have equal parameters and return annotations.

Changes to the Signature object, or to any of its data members,
do not affect the function itself.


Parameter Object


Python's expressive syntax means functions can accept many different
kinds of parameters with many subtle semantic differences.  We
propose a rich Parameter object designed to represent any possible
function parameter.

The structure of the Parameter object is:

* name : str
The name of the parameter as a string.
* default : object
The default value for the parameter, if specified.  If the
parameter has no default value, this attribute is not set.
* annotation : object
The annotation for the parameter if specified.  If the
parameter has no annotation, this attribute is not set.
* is_keyword_only : bool
True if the parameter is keyword-only, else False.
* is_args : bool
True if the parameter accepts variable number of arguments
(``*args``-like), else False.
* is_kwargs : bool
True if the parameter accepts variable number of keyword
arguments (``**kwargs``-like), else False.
* is_implemented : bool
True if the parameter is implemented for use.  Some platforms
implement functions but can't support specific parameters
(e.g. "mode" for ``os.mkdir``).  Passing in an unimplemented
parameter may result in the parameter being ignored,
or in NotImplementedError being raised.  It is intended that
all conditions where ``is_implemented`` may be False be
thoroughly documented.

Two para

Re: [Python-Dev] PEP 362 Third Revision

[Yury Selivanov]

On 2012-06-13, at 10:52 PM, Yury Selivanov wrote:
> 2. signature() function support all kinds of callables:
> classes, metaclasses, methods, class- & staticmethods,
> 'functools.partials', and callable objects.  If a callable
> object has a '__signature__' attribute it does a deepcopy
> of it before return.  


Properly decorated functions are also supported.

-
Yury
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[Terry Reedy]

On 6/13/2012 10:47 PM, R. David Murray wrote:

On Thu, 14 Jun 2012 11:48:08 +1000, Nick Coghlan  wrote:



Right, but by resorting to either of those approaches, people are
clearly doing something that isn't formally supported by the core.


That was not clear to me until I read your post -- the key word being 
formally (or officially). I see now that distributing a sourceless 
library as a mixture of .pyc and .pyo files is even crazier that I thought.



Yes, you can do it, and most of the time it will work out OK, but any
weird glitches that result are officially *not our problem*.

The main reason this matters is that the "__debug__" flag is
*supposed* to be process global - if you check it in one place, the


OK, the above are the two concrete reasons I have heard in this thread
for continuing the current behavior:

 1) we do not wish to support running from .pyo files without -O
being on, even if it currently happens to work

 2) the __debug__ setting is supposed to be process-global

Both of these are good reasons.  IMO the issue should be closed with a
documentation fix, which could optionally include either or both of the
above motivations.


I agree. We have gotten what we need from this thread.

--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[Steven D'Aprano]

On Wed, Jun 13, 2012 at 03:13:54PM -0400, R. David Murray wrote:

> Again, a program that depends on asserts is buggy.
> 
> As Ethan pointed out we are asking about the case where someone is
> *deliberately* setting the .pyo file up to be run as the "normal"
> case.

You can't be sure that the .pyo file is there due to *deliberate* 
choice. It may be accidental. Perhaps the end user has ignorantly 
deleted the .pyc file, but failed to delete the .pyo file. Perhaps the 
developer has merely made a mistake.

Under current behaviour, deleting the .pyc file shouldn't matter:

- if the source file is available, that will be used
- if not, a clear error is raised

Under the proposed change:

- if the source file is *newer* than the .pyo file, it will be used
- but if it is missing or older, the .pyo file is used

This opens a potential mismatch between the code I *think* is being run, 
and the actual code being run: I think the .py[c] code is running when 
the .pyo is actually running.

Realistically, we should expect that most people don't *sufficiently* 
test their apps under -O (if at all!) even if they are aware that there 
are differences in behaviour. I know I don't, and I know I should. This 
is just a matter of priority: testing without -O is a higher priority 
for me than testing with -O and -OO.

The consequence is that I may then receive a mysterious bug report that 
I can't duplicate, because the user correctly reports that they are 
running *without* -O, but unknown to anyone, they are actually running 
the .pyo file.


> I'm not sure we want to support that, I just want us to be clear
> about why we don't :)

If I receive a bug report that only occurs under -O, then I immediately 
suspect that the bug has something to do with assert.

If I receive a bug report that occurs without -O, under the proposed 
change I can't be sure with the optimized code or standard code is 
running. That adds complexity and confusion.


-- 
Steven
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[Steven D'Aprano]

On Wed, Jun 13, 2012 at 04:06:22PM -0400, Terry Reedy wrote:
> On 6/13/2012 2:46 PM, Antoine Pitrou wrote:
> 
> >Not only docstrings, but also asserts. I think running a pyo without -O
> >would be a bug.
> 
> That cat is already out of the bag ;-)
> People are doing that now by renaming x.pyo to x.pyc.
> Brett claims that it is also easy to do in 3.3 with a custom importer.

That's fine. Both steps require an overt, deliberate act, and so is 
under the control of (and the responsibilty of) the developer. It's not 
something that could happen innocently by accident.



-- 
Steven

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[Steven D'Aprano]

On Wed, Jun 13, 2012 at 09:54:30PM -0400, Terry Reedy wrote:

> >So, no, we
> 
> You mean the interpreter?

Yes.

> >should never use
> 
> Do you mean import or execute?
> Current, the interpreter executes any bytecode that gets imported.

Both.

> >.pyo files unless explicitly told to do so,
> 
> What constitutes 'explicitly told to do so'? Currently, an 'optimized' 
> file written as .pyo gets imported (and hence executed) if
> 1) the interpreter is started with -O
> 2) a custom importer ignores the absence of -O
> 3) someone renames x.pyo to x.pyc.

Any of the above are fine by me.

I oppose this one:

4) the interpreter is started without -O but there is no .pyc file.

since it can lead to a mismatch between what I (the developer) thinks is 
being run and what is actually being run (or imported).

For the avoidance of doubt, if my end-users secretly rename .pyo to .pyc 
files, that's my problem, not the Python's interpreter's problem. I 
don't expect Python to be idiot-proof.



-- 
Steven

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] #12982: Should -O be required to *read* .pyo files?

[Ethan Furman]

Steven D'Aprano wrote:

On Wed, Jun 13, 2012 at 03:13:54PM -0400, R. David Murray wrote:


Again, a program that depends on asserts is buggy.

As Ethan pointed out we are asking about the case where someone is
*deliberately* setting the .pyo file up to be run as the "normal"
case.


You can't be sure that the .pyo file is there due to *deliberate* 
choice. It may be accidental. Perhaps the end user has ignorantly 
deleted the .pyc file, but failed to delete the .pyo file. Perhaps the 
developer has merely made a mistake.


You can't just delete the .pyc file to get the .pyo file to run; 
remember in 3.x compiled files are kept in a __pycache__ folder, and if 
there is no .py file the compiled files are ignored (correct me if I'm 
wrong), so to get the either the .pyc file /or/ the .pyo file to run 
/without/ a .py file, you have to physically move the compiled file to 
where the source file should be.  It could still be accidental, but it's 
far less likely to be.




Under current behaviour, deleting the .pyc file shouldn't matter:

- if the source file is available, that will be used
- if not, a clear error is raised

Under the proposed change:

- if the source file is *newer* than the .pyo file, it will be used
- but if it is missing or older, the .pyo file is used


Again, not in 3.x.

~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 362 Third Revision

[Nick Coghlan]

On Thu, Jun 14, 2012 at 1:06 PM, Yury Selivanov  wrote:
> On 2012-06-13, at 10:52 PM, Yury Selivanov wrote:
>> 2. signature() function support all kinds of callables:
>> classes, metaclasses, methods, class- & staticmethods,
>> 'functools.partials', and callable objects.  If a callable
>> object has a '__signature__' attribute it does a deepcopy
>> of it before return.
>
>
> Properly decorated functions are also supported.

I'd like to see the "shared state" decorator from the previous thread
included, as well as a short interactive interpreter session showing
correct reporting of the signature of functools.partial instances.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 362 Third Revision

[Alexandre Zani]

On Wed, Jun 13, 2012 at 8:06 PM, Yury Selivanov  wrote:
> On 2012-06-13, at 10:52 PM, Yury Selivanov wrote:
>> 2. signature() function support all kinds of callables:
>> classes, metaclasses, methods, class- & staticmethods,
>> 'functools.partials', and callable objects.  If a callable
>> object has a '__signature__' attribute it does a deepcopy
>> of it before return.
>
>
> Properly decorated functions are also supported.
>
> -
> Yury
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/alexandre.zani%40gmail.com

This is really exciting! A couple questions/points:

Why do we look at __wrapped__ only if the object is a FunctionType?
Why not support __wrapped__ on all callables?

Why special-case functools.partial? Couldn't functools.partial just
set __signature__ itself? Is that because of inspect's dependency on
functools?

Just a thought: Do we want to include the docstring? A function's
docstring is often intimately tied to its signature. (Or at least, a
lot of us try to write docstrings that effectively describe the
function's signature)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com