Re: [Python-Dev] PEP 445 delegate
Hi, 2013/6/21 Antoine Pitrou : > I've been appointed PEP 445 delegate by Nick and Guido. I would like to > know if there are still pending changes to the PEP. If not, I expect to > give it a review in the coming days or weeks, and then make a final > pronouncement (which will probably be positive anyway). As you asked on IRC, I added a section to the PEP 445 explaining in detail the redesign of debug checks on memory allocators: http://www.python.org/dev/peps/pep-0445/#redesign-debug-checks-on-memory-allocators-as-hooks The section show also how "hooks" work in the simple "call traces" example. Hooks are just inserted between the public function (ex: PyMem_Malloc) and the inner allocator function (ex: _PyMem_RawMalloc()). Minor nit: in practice, there is only one "debug hook" (reused on 3 allocators), so we may drop the trailing "s" from the name "PyMem_SetupDebugHooks". Diff on pep-0445.txt: http://hg.python.org/peps/rev/c6199202522a Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X options to python man page.
On Thu, Jun 20, 2013 at 3:36 PM, Brett Cannon wrote: > > > > On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran > wrote: >> >> http://hg.python.org/cpython/rev/dfead0696a71 >> changeset: 84224:dfead0696a71 >> branch: 3.3 >> parent: 84221:0113247f894b >> user:Senthil Kumaran >> date:Wed Jun 19 22:19:46 2013 -0500 >> summary: >> Add -b and -X options to python man page. >> Patch contributed by Corey Brune. >> >> files: >> Misc/python.man | 22 ++ >> 1 files changed, 18 insertions(+), 4 deletions(-) >> >> >> diff --git a/Misc/python.man b/Misc/python.man >> --- a/Misc/python.man >> +++ b/Misc/python.man >> @@ -11,6 +11,9 @@ >> .B \-B >> ] >> [ >> +.B \-b >> +] >> +[ >> .B \-d >> ] >> [ >> @@ -23,14 +26,14 @@ >> .B \-i >> ] >> [ >> -.B \-m >> +.B \-m >> .I module-name >> ] >> -[ >> -.B \-q >> -] >> .br >> [ >> +.B \-q >> +] >> +[ >> .B \-O >> ] >> [ >> @@ -60,6 +63,10 @@ >> .B \-x >> ] >> [ >> +[ >> +.B \-X >> +.I option >> +] >> .B \-? >> ] >> .br >> @@ -105,6 +112,10 @@ >> .I .py[co] >> files on import. See also PYTHONDONTWRITEBYTECODE. >> .TP >> +.B \-b >> +Issue warnings about str(bytes_instance), str(bytearray_instance) >> +and comparing bytes/bytearray with str. (-bb: issue errors) >> +.TP >> .BI "\-c " command >> Specify the command to execute (see next section). >> This terminates the option list (following options are passed as >> @@ -243,6 +254,9 @@ >> field matches the line number, where zero matches all line numbers and >> is thus equivalent to an omitted line number. >> .TP >> +.BI "\-X " option >> +Set implementation specific option. > > > Should probably be "Set the implementation-specific option." Is there anyone respecting this notation? (I know pypy does not, it uses --jit and stuff) > >> >> +.TP >> .B \-x >> Skip the first line of the source. This is intended for a DOS >> specific hack only. Warning: the line numbers in error messages will >> >> -- >> Repository URL: http://hg.python.org/cpython >> >> ___ >> Python-checkins mailing list >> [email protected] >> http://mail.python.org/mailman/listinfo/python-checkins >> > > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X options to python man page.
On Sun, 23 Jun 2013 17:40:13 +0200, Maciej Fijalkowski wrote: > On Thu, Jun 20, 2013 at 3:36 PM, Brett Cannon wrote: > > On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran > > wrote: > >> .TP > >> +.BI "\-X " option > >> +Set implementation specific option. > > > > > > Should probably be "Set the implementation-specific option." > > Is there anyone respecting this notation? (I know pypy does not, it > uses --jit and stuff) CPython does. We introduced it for ourselves, it is up to other implementations whether or not to use it, or use something else. --David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod
I've backed this one out, too.
2013/6/22 Scott Dial :
> On 6/22/2013 2:17 PM, Benjamin Peterson wrote:
>> Many people have raised concerns about this change, so I've now backed it
>> out.
>
> I think that change also goes with this change:
>
> http://hg.python.org/cpython/rev/f1dc30a1be72
>
> changeset 84248:f1dc30a1be72 2.7
> Arrange structure to match the common access patterns.
>
> 1.1 --- a/Modules/_collectionsmodule.c
> 1.2 +++ b/Modules/_collectionsmodule.c
> 1.3 @@ -48,8 +48,8 @@
> 1.4
> 1.5 typedef struct BLOCK {
> 1.6 struct BLOCK *leftlink;
> 1.7 +PyObject *data[BLOCKLEN];
> 1.8 struct BLOCK *rightlink;
> 1.9 -PyObject *data[BLOCKLEN];
> 1.10 } block;
> 1.11
> 1.12 #define MAXFREEBLOCKS 10
>
> , which seems like a strange micro-optimization, at best.
>
> Based on that structure, it would seem that neither BLOCKLEN being 62
> (previous value) nor 64 (the new value) make much sense. It would seem
> best that sizeof(block) == 64, so BLOCKLEN should be (64 -
> 2*sizeof(PyObject *)). Nevertheless, I am skeptical that any tuning of
> this structure provides any meaningful performance improvement.
>
> --
> Scott Dial
> [email protected]
--
Regards,
Benjamin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] A type of out-of-range exception for %c
Currently %c formatting raises OverflowError if an argument is out of range.
>>> '%c' % 1114112
Traceback (most recent call last):
File "", line 1, in
OverflowError: %c arg not in range(0x11)
>>> '{:c}'.format(1114112)
Traceback (most recent call last):
File "", line 1, in
OverflowError: %c arg not in range(0x11)
The same for PyUnicode_Format(), PyUnicode_FromFromat() and
PyUnicode_FromFromatV() in C API.
But chr() raises ValueError.
>>> chr(1114112)
Traceback (most recent call last):
File "", line 1, in
ValueError: chr() arg not in range(0x11)
And ValueError looks more suitable, because it is not really an overflow
(limitation of the C language, or a C type). Can we change an exception
type for %c formatting from OverflowError to ValueError?
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A type of out-of-range exception for %c
On Sun, 23 Jun 2013 21:59:37 +0300
Serhiy Storchaka wrote:
> Currently %c formatting raises OverflowError if an argument is out of range.
>
> >>> '%c' % 1114112
> Traceback (most recent call last):
>File "", line 1, in
> OverflowError: %c arg not in range(0x11)
> >>> '{:c}'.format(1114112)
> Traceback (most recent call last):
>File "", line 1, in
> OverflowError: %c arg not in range(0x11)
>
> The same for PyUnicode_Format(), PyUnicode_FromFromat() and
> PyUnicode_FromFromatV() in C API.
>
> But chr() raises ValueError.
>
> >>> chr(1114112)
> Traceback (most recent call last):
>File "", line 1, in
> ValueError: chr() arg not in range(0x11)
>
> And ValueError looks more suitable, because it is not really an overflow
> (limitation of the C language, or a C type). Can we change an exception
> type for %c formatting from OverflowError to ValueError?
Well, it overflows the limited range of the "unicode character" type
(which is virtual in both Python and C, but still conceptually exists).
The most annoying thing here is that OverflowError doesn't subclass
ValueError.
Regards
Antoine.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A type of out-of-range exception for %c
On 23 cze 2013, at 21:07, Antoine Pitrou wrote: > The most annoying thing here is that OverflowError doesn't subclass > ValueError. My intuition would rather make OverflowError subclass RuntimeError. Am I wrong? That means I support Serhiy's point that %c would be less surprising raising a ValueError. Backwards compatibility is an unfortunate blocker here, I suppose. -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod
On Jun 18, 2013, at 12:00 AM, Gregory P. Smith wrote:
> Why did you do this in the 2.7 branch?
>
> It hasn't been done in 3.3 or default
I worked on the 2.7 branch first because that was the one I had loaded
and the one where I did timings and code disassembly. I intended to
post it to 3.3 and 3.4 as well over the weekend.
Ideally, it makes maintenance simpler for me if I keep the branches
the same as possible. I viewed the one-line struct transposition,
comment correction, and one-line blocklen change as being somewhat
innocuous non-algorithmic changes. The struct change fixed an unintended
cache miss for left links and the blocksize change causes the
deque_index code to compile more efficiently (using a right-shift
and bitwise-and and rather than a measurably more expensive
division and modulo calculation).
I truly wasn't expecting the Spanish Inquisition :-)
On Jun 22, 2013, at 1:43 PM, Scott Dial wrote:
> changeset 84248:f1dc30a1be72 2.7
> Arrange structure to match the common access patterns.
>
> 1.1 --- a/Modules/_collectionsmodule.c
> 1.2 +++ b/Modules/_collectionsmodule.c
> 1.3 @@ -48,8 +48,8 @@
> 1.4
> 1.5 typedef struct BLOCK {
> 1.6 struct BLOCK *leftlink;
> 1.7 +PyObject *data[BLOCKLEN];
> 1.8 struct BLOCK *rightlink;
> 1.9 -PyObject *data[BLOCKLEN];
>1.10 } block;
>1.11
>1.12 #define MAXFREEBLOCKS 10
>
> , which seems like a strange micro-optimization, at best.
Yes, this is a micro-optimization. In working on implementing
deque slicing for 3.4, I restudied the block access patterns.
On an appendleft(), popleft() or extendleft() operation, the left link is
accessed immediately before or after the leftmost entry in the data block.
The old struct arrangement can cause an unnecessary cache miss
when jumping leftward. This was something I overlooked when I
originally wrote the code almost a decade ago.
On Jun 23, 2013, at 11:38 AM, Benjamin Peterson wrote:
> I've backed this one out, too.
Really, you reverted my one-line change within 24 hours of it being posted?
I can't be on-line every day and sometimes it takes a little while to catch up
with python email.
On Jun 22, 2013, at 2:55 PM, Guido van Rossum wrote:
> Actually the data buffer is an array of pointers too, so with the
> original BLOCKLEN value of 62, sizeof(block) would be 64 times
> sizeof(PyObject *). In the Py3 version of the source there's even a
> comment explaining this right before the #define BLOCKLEN. Raymond,
> can you explain?
I also tried to fix that comment so it would stop emphasizing the blocklen
being a multiple of the cache line. Also long as there is a reasonably long
data block, it matters less whether the data block size is an exact multiple
of the cache line length (62 or 64 words of data versus a typical 8 byte
cache line).
The data block size does matter elsewhere though.
The benefit of the having the data block being a power-of-two
is that the deque_index computation can use bits shifts
rather division and modulo calculations. The benefit
of switching data block size from 62 to 64 was measurable
(a few percent) and observable in the disassembly of the code.
I experimented with one other ordering as well
(putting the data block first and the links afterwards).
That saved the scaled-index byte in the generated code
but produced no measureable speed-up.
In short, I believe the following should go in:
* The comment fix. (Incorrectly suggesting that a 64-bit
Py_ssize_t would overflow). The revised comment
is simpler, more general, and correct.
* Putting the leftlink before the data block in the structure.
The follow is up for debate:
Changing the BLOCKLEN from 62 to 64 is debatable.
It measureably improved deque_index without an
observable negative effect on the other operations.
Lastly, there was a change I just put in to Py 3.4 replacing
the memcpy() with a simple loop and replacing the
"deque->" references with local variables. Besides
giving a small speed-up, it made the code more clear
and less at the mercy of various implementations
of memcpy().
Ideally, I would like 2.7 and 3.3 to replace their use of
memcpy() as well, but the flavor of this thread suggests
that is right out.
Raymond
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod
On 6/23/2013 8:16 PM, Raymond Hettinger wrote: > Yes, this is a micro-optimization. In working on implementing > deque slicing for 3.4, I restudied the block access patterns. > On an appendleft(), popleft() or extendleft() operation, the left link is > accessed immediately before or after the leftmost entry in the data block. > The old struct arrangement can cause an unnecessary cache miss > when jumping leftward. This was something I overlooked when I > originally wrote the code almost a decade ago. A decade ago, cache lines were 64 bytes, pointers were 4 bytes, and allocations were 16 byte aligned, so there could never be a cache miss. Nowadays, cache lines are still 64 bytes but pointers are 8 bytes, and we still allocating on 16 byte alignment, so you have a 25% chance of a cache miss now. -- Scott Dial [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod
On Jun 23, 2013, at 6:52 PM, Scott Dial wrote: > Nowadays, cache lines are still 64 bytes but pointers are 8 bytes, and > we still allocating on 16 byte alignment, so you have a 25% chance of a > cache miss now. Honestly, I'm not sure what you're arguing for or against. The struct should to be reordered so that the leftmost data element and left link are positioned side-by-side, and the same goes for the rightmost element and right link. Whether the blocksize should change is debatable. It causes an unfortunate odd-size at the end (not good for the final cache line), but it does improve the speed of the deque_index() code. The former seems to make no difference in timings while the latter gives a measureable speed-up. Unfortunately, I'm losing interest in this part of the deque work. I've already invested substantial time reading papers (such as http://www.akkadia.org/drepper/cpumemory.pdf and http://www.intel.com/content/www/us/en/architecture-and-technology/64-ia-32-architectures-optimization-manual.html), analyzing the code, reading disassembly, and making timings. But it isn't worth all the second guessing (and what feels like sniping). I've worked on this code for almost a decade. As far as I can tell, none of the participants in this thread has ever previously shown any interest in the deque object. It is discouraging to have a simple parameter change and struct reordering reverted. This thread has offered zero support or encouragement for my work. The python-dev social environment appears to be degrading over time. I only have a few hours of development time each week and now I'm wasting that time responding to these emails (it may go with the territory, but it is a waste none-the-less). If it is what you guys what, then leave the code as is (with an incorrect comment, a blocklen that is unfavorable to indexing, and a struct order that doesn't exploit cache locality by following the natural access patterns in the code). I understand that the code for Py2.7 is sensitive and understand if you all prefer to leave it untouched. Raymond P.S. This has all arisen in context of my working on patch for implementing slicing for deques. In that context, the code for deque_item() and deque_rotate() will become more important than they were before. Along the way, I am re-examining all my existing code. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod
On Sun, Jun 23, 2013 at 11:37 PM, Raymond Hettinger < [email protected]> wrote: > As far as I can tell, none > of the participants in this thread has ever previously shown any interest > in the deque object. It is discouraging to have a simple parameter > change and struct reordering reverted. This thread has offered zero > support or encouragement for my work. The python-dev > social environment appears to be degrading over time. > FWIW, I was surprised to see Raymond's patch reverted in what looked like a knee-jerk reaction. In my view Raymond through his contributions has earned the presumption of validity for his commits. A risk of introducing regressions may ultimately outweigh the benefit of optimization and keeping 2.x and 3.x code in sync, but there is no need in rushing the decision. This patch could be reverted at any time before the 2.7.6 release which as far as I know has not even been scheduled. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod
Your work is great! I even agree with the changes on a coding best practices level. It's just that it belongs on the default (3.4) branch as it is an enhancement, not a bug fix. We don't do new things on release branches. I agree that can be extremely frustrating at times, knowing that code won't see the light of day for most users for years, but it is what has always kept the Python project stable. -gps On Jun 23, 2013 8:38 PM, "Raymond Hettinger" wrote: > > On Jun 23, 2013, at 6:52 PM, Scott Dial > wrote: > > Nowadays, cache lines are still 64 bytes but pointers are 8 bytes, and > we still allocating on 16 byte alignment, so you have a 25% chance of a > cache miss now. > > > Honestly, I'm not sure what you're arguing for or against. > > The struct should to be reordered so that the leftmost data element > and left link are positioned side-by-side, and the same goes for the > rightmost element and right link. > > Whether the blocksize should change is debatable. > It causes an unfortunate odd-size at the end (not good for > the final cache line), but it does improve the speed of the > deque_index() code. The former seems to make no difference > in timings while the latter gives a measureable speed-up. > > Unfortunately, I'm losing interest in this part of the deque work. > I've already invested substantial time reading papers > (such as http://www.akkadia.org/drepper/cpumemory.pdf > and > http://www.intel.com/content/www/us/en/architecture-and-technology/64-ia-32-architectures-optimization-manual.html > ), > analyzing the code, reading disassembly, and making timings. > But it isn't worth all the second guessing (and what feels like sniping). > I've worked on this code for almost a decade. As far as I can tell, none > of the participants in this thread has ever previously shown any interest > in the deque object. It is discouraging to have a simple parameter > change and struct reordering reverted. This thread has offered zero > support or encouragement for my work. The python-dev > social environment appears to be degrading over time. > I only have a few hours of development time each week > and now I'm wasting that time responding to these emails > (it may go with the territory, but it is a waste none-the-less). > > If it is what you guys what, then leave the code as is > (with an incorrect comment, a blocklen that is unfavorable > to indexing, and a struct order that doesn't exploit > cache locality by following the natural access patterns > in the code). > > I understand that the code for Py2.7 is sensitive and understand > if you all prefer to leave it untouched. > > > > Raymond > > > P.S. This has all arisen in context of my working on patch > for implementing slicing for deques. In that context, the > code for deque_item() and deque_rotate() will become > more important than they were before. Along the way, > I am re-examining all my existing code. > > > > > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod
2013/6/23 Alexander Belopolsky : > > On Sun, Jun 23, 2013 at 11:37 PM, Raymond Hettinger > wrote: >> >> As far as I can tell, none >> of the participants in this thread has ever previously shown any interest >> in the deque object. It is discouraging to have a simple parameter >> change and struct reordering reverted. This thread has offered zero >> support or encouragement for my work. The python-dev >> social environment appears to be degrading over time. > > > FWIW, I was surprised to see Raymond's patch reverted in what looked like a > knee-jerk reaction. In my view Raymond through his contributions has earned > the presumption of validity for his commits. A risk of introducing > regressions may ultimately outweigh the benefit of optimization and keeping > 2.x and 3.x code in sync, but there is no need in rushing the decision. > This patch could be reverted at any time before the 2.7.6 release which as > far as I know has not even been scheduled. The first concern about 5accb0ac8bfb was raised more than a week ago and more piled up over the course of the week. I knew Raymond wasn't offline, since he continued to commit. Since we don't have mandatory reviews before push (which more and more I think we should), post-facto reviews on python-checkins need to be acknowledged. A backout is not the end of the world; it takes seconds to reland once a conclusion has been reached. This incident should not be construed to diminish Raymond's long history of contribution or his technical ability. My second reversion (86d512e0ec66) was knee-jerk and shouldn't have happened. -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod
On 24 June 2013 13:37, Raymond Hettinger wrote: > But it isn't worth all the second guessing (and what feels like sniping). > I've worked on this code for almost a decade. As far as I can tell, none > of the participants in this thread has ever previously shown any interest > in the deque object. It is discouraging to have a simple parameter > change and struct reordering reverted. This thread has offered zero > support or encouragement for my work. The python-dev > social environment appears to be degrading over time. > I only have a few hours of development time each week > and now I'm wasting that time responding to these emails > (it may go with the territory, but it is a waste none-the-less). The problem wasn't the change in and of itself, it was: - the fact it happened in a stable branch - the fact there was no explanation, even a week after one was requested (not even a quick "I do have the data to back up this assertion, please leave the commit alone until I have time to provide it"). The mailing lists, commit history, source code comments and files like dict_notes.txt are an important part of understanding *why* various parts of CPython are the way they are. So, no, "responding to emails" is *not* a waste of anybody's time. It helps to ensure important knowledge is transferred to more than the person that was responsible for the commit. This is *especially* important for areas where one person has substantially more knowledge and experience than others (alleviating that kind of problem is one of the reasons Brett, myself and others have been embarked on a multi-year campaign to make the import system less arcane and mysterious, and why I want to make the startup code more comprehensible). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
