Re: [Python-Dev] Speeding up CPython 5-10%

[Damien George]

Hi Yury,

> An off-topic: have you ever tried hg.python.org/benchmarks
> or compare MicroPython vs CPython?  I'm curious if MicroPython
> is faster -- in that case we'll try to copy some optimization
> ideas.

I've tried a small number of those benchmarks, but not in any rigorous
way, and not enough to compare properly with CPython.  Maybe one day I
(or someone) will get to it and report results :)

One thing that makes MP fast is the use of pointer tagging and
stuffing of small integers within object pointers.  Thus integer
arithmetic below 2**30 (on 32-bit arch) requires no heap.

> Do you use opcode dictionary caching only for LOAD_GLOBAL-like
> opcodes?  Do you have an equivalent of LOAD_FAST, or you use
> dicts to store local variables?

The opcodes that have dict caching are:

LOAD_NAME
LOAD_GLOBAL
LOAD_ATTR
STORE_ATTR
LOAD_METHOD (not implemented yet in mainline repo)

For local variables we use LOAD_FAST and STORE_FAST (and DELETE_FAST).
Actually, there are 16 dedicated opcodes for loading from positions
0-15, and 16 for storing to these positions.  Eg:

LOAD_FAST_0
LOAD_FAST_1
...

Mostly this is done to save RAM, since LOAD_FAST_0 is 1 byte.

> If we change the opcode size, it will probably affect libraries
> that compose or modify code objects.  Modules like "dis" will
> also need to be updated.  And that's probably just a tip of the
> iceberg.
>
> We can still implement your approach if we add a separate
> private 'unsigned char' array to each code object, so that
> LOAD_GLOBAL can store the key offsets.  It should be a bit
> faster than my current patch, since it has one less level
> of indirection.  But this way we loose the ability to
> optimize LOAD_METHOD, simply because it requires more memory
> for its cache.  In any case, I'll experiment!

Problem with that approach (having a separate array for offset_guess)
is that how do you know where to look into that array for a given
LOAD_GLOBAL opcode?  The second LOAD_GLOBAL in your bytecode should
look into the second entry in the array, but how does it know?

I'd love to experiment implementing my original caching idea with
CPython, but no time!

Cheers,
Damien.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] FAT Python (lack of) performance

[Peter Ludemann via Python-Dev]

About benchmarks ... I've been there and it's not benchmarks that decide
whether something succeeds or fails.
(I found this old discussion
 which
mentions FIB (also TAK ,
which is rather more brutal)  ... do you recognize the language that got an
implementation that was competitive with C in performance, was vastly more
expressive, yet failed to catch on?)

OTOH, good performance is never a bad thing and sometimes is a necessity;
so I applau this work.


On 28 January 2016 at 01:39, Nick Coghlan  wrote:

> On 28 January 2016 at 18:30, INADA Naoki  wrote:
> > Please stop.
> >
> > I'm sorry about messing up this thread.
>
> Not your fault at all! This is just a particular bugbear of mine,
> since software architecture design (including appropriate programming
> language selection) is an even more poorly understood discipline than
> software development in general :)
>
> > I just wanted to represent why I'm very interested in Victor's efforts.
>
> And thanks for posting that, as it is indeed cool that the
> optimisation efforts currently being discussed may result in
> performance improvements on some of the simplified micro-benchmarks
> popular in programming language shootouts.
>
> There's no way you could have anticipated the subsequent tangential
> discussion on motives for contributing to open source projects, and
> the impact that has on what we can reasonably expect from fellow
> contributors.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/pludemann%40google.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Speeding up CPython 5-10%

[Stefan Behnel]

Yury Selivanov schrieb am 27.01.2016 um 19:25:
> tl;dr The summary is that I have a patch that improves CPython performance
> up to 5-10% on macro benchmarks.  Benchmarks results on Macbook Pro/Mac OS
> X, desktop CPU/Linux, server CPU/Linux are available at [1].  There are no
> slowdowns that I could reproduce consistently.
> 
> There are two different optimizations that yield this speedup:
> LOAD_METHOD/CALL_METHOD opcodes and per-opcode cache in ceval loop.
> 
> LOAD_METHOD & CALL_METHOD
> -
> 
> We had a lot of conversations with Victor about his PEP 509, and he sent me
> a link to his amazing compilation of notes about CPython performance [2]. 
> One optimization that he pointed out to me was LOAD/CALL_METHOD opcodes, an
> idea first originated in PyPy.
> 
> There is a patch that implements this optimization, it's tracked here:
> [3].  There are some low level details that I explained in the issue, but
> I'll go over the high level design in this email as well.
> 
> Every time you access a method attribute on an object, a BoundMethod object
> is created. It is a fairly expensive operation, despite a freelist of
> BoundMethods (so that memory allocation is generally avoided).  The idea is
> to detect what looks like a method call in the compiler, and emit a pair of
> specialized bytecodes for that.
> 
> So instead of LOAD_GLOBAL/LOAD_ATTR/CALL_FUNCTION we will have
> LOAD_GLOBAL/LOAD_METHOD/CALL_METHOD.
> 
> LOAD_METHOD looks at the object on top of the stack, and checks if the name
> resolves to a method or to a regular attribute.  If it's a method, then we
> push the unbound method object and the object to the stack.  If it's an
> attribute, we push the resolved attribute and NULL.
> 
> When CALL_METHOD looks at the stack it knows how to call the unbound method
> properly (pushing the object as a first arg), or how to call a regular
> callable.
> 
> This idea does make CPython faster around 2-4%.  And it surely doesn't make
> it slower.  I think it's a safe bet to at least implement this optimization
> in CPython 3.6.
> 
> So far, the patch only optimizes positional-only method calls. It's
> possible to optimize all kind of calls, but this will necessitate 3 more
> opcodes (explained in the issue).  We'll need to do some careful
> benchmarking to see if it's really needed.

I implemented a similar but simpler optimisation in Cython a while back:

http://blog.behnel.de/posts/faster-python-calls-in-cython-021.html

Instead of avoiding the creation of method objects, as you proposed, it
just normally calls getattr and if that returns a bound method object, it
uses inlined calling code that avoids re-packing the argument tuple.
Interestingly, I got speedups of 5-15% for some of the Python benchmarks,
but I don't quite remember which ones (at least raytrace and richards, I
think), nor do I recall the overall gain, which (I assume) is what you are
referring to with your 2-4% above. Might have been in the same order.

Stefan


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Speeding up CPython 5-10%

[Yury Selivanov]

On 2016-01-29 5:00 AM, Stefan Behnel wrote:

Yury Selivanov schrieb am 27.01.2016 um 19:25:

[..]

LOAD_METHOD looks at the object on top of the stack, and checks if the name
resolves to a method or to a regular attribute.  If it's a method, then we
push the unbound method object and the object to the stack.  If it's an
attribute, we push the resolved attribute and NULL.

When CALL_METHOD looks at the stack it knows how to call the unbound method
properly (pushing the object as a first arg), or how to call a regular
callable.

This idea does make CPython faster around 2-4%.  And it surely doesn't make
it slower.  I think it's a safe bet to at least implement this optimization
in CPython 3.6.

So far, the patch only optimizes positional-only method calls. It's
possible to optimize all kind of calls, but this will necessitate 3 more
opcodes (explained in the issue).  We'll need to do some careful
benchmarking to see if it's really needed.

I implemented a similar but simpler optimisation in Cython a while back:

http://blog.behnel.de/posts/faster-python-calls-in-cython-021.html

Instead of avoiding the creation of method objects, as you proposed, it
just normally calls getattr and if that returns a bound method object, it
uses inlined calling code that avoids re-packing the argument tuple.
Interestingly, I got speedups of 5-15% for some of the Python benchmarks,
but I don't quite remember which ones (at least raytrace and richards, I
think), nor do I recall the overall gain, which (I assume) is what you are
referring to with your 2-4% above. Might have been in the same order.


That's great!

I'm still working on the patch, but so far it looks like adding
just LOAD_METHOD/CALL_METHOD (that avoid instantiating BoundMethods)
gives us 10-15% faster method calls.

Combining them with my opcode cache makes them 30-35% faster.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Speeding up CPython 5-10%

[Yury Selivanov]

Hi Damien,

BTW I just saw (and backed!) your new Kickstarter campaign
to port MicroPython to ESP8266, good stuff!

On 2016-01-29 7:38 AM, Damien George wrote:

Hi Yury,

[..]

Do you use opcode dictionary caching only for LOAD_GLOBAL-like
opcodes?  Do you have an equivalent of LOAD_FAST, or you use
dicts to store local variables?

The opcodes that have dict caching are:

LOAD_NAME
LOAD_GLOBAL
LOAD_ATTR
STORE_ATTR
LOAD_METHOD (not implemented yet in mainline repo)

For local variables we use LOAD_FAST and STORE_FAST (and DELETE_FAST).
Actually, there are 16 dedicated opcodes for loading from positions
0-15, and 16 for storing to these positions.  Eg:

LOAD_FAST_0
LOAD_FAST_1
...

Mostly this is done to save RAM, since LOAD_FAST_0 is 1 byte.


Interesting.  This might actually make CPython slightly faster
too.  Worth trying.




If we change the opcode size, it will probably affect libraries
that compose or modify code objects.  Modules like "dis" will
also need to be updated.  And that's probably just a tip of the
iceberg.

We can still implement your approach if we add a separate
private 'unsigned char' array to each code object, so that
LOAD_GLOBAL can store the key offsets.  It should be a bit
faster than my current patch, since it has one less level
of indirection.  But this way we loose the ability to
optimize LOAD_METHOD, simply because it requires more memory
for its cache.  In any case, I'll experiment!

Problem with that approach (having a separate array for offset_guess)
is that how do you know where to look into that array for a given
LOAD_GLOBAL opcode?  The second LOAD_GLOBAL in your bytecode should
look into the second entry in the array, but how does it know?




I've changed my approach a little bit.  Now I have a simple
function [1] to initialize the cache for code objects that
are called frequently enough.

It walks through the code object's opcodes and creates the
appropriate  offset/cache tables.

Then, in ceval loop I have a couple of convenient macros
to work with the cache [2].  They use INSTR_OFFSET() macro
to locate the cache entry via the offset table initialized
by [1].

Thanks,
Yury

[1] https://github.com/1st1/cpython/blob/opcache4/Objects/codeobject.c#L167
[2] https://github.com/1st1/cpython/blob/opcache4/Python/ceval.c#L1164
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] More optimisation ideas

[francismb]

Hi,

> 
> Storing these in static data is a tradeoff between
> disk space and startup performance, and one I think it likely to be
> worthwhile.

it's really an important trade off? As far a I understand from your
email those modules are always being loaded and the final data created.
won't the space be there (on mem or disk)?

Thanks in advance!
francis


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] More optimisation ideas

[Steve Dower]

It doesn't currently end up on disk. Some tables are partially or completely 
stored on disk as Python source code (some are partially generated from simple 
rules), but others are generated by inverting those. That process takes time 
that could be avoided by storing the generated tables, and storing all of it in 
a format that doesn't require parsing, compiling and executing (such as a 
native array).

Potentially it could be a win all around if we stopped including the (larger) 
source files, but that doesn't seem like a good idea for maintaining 
portability to other implementations. The main thought is making the compiler 
binary bigger to avoid generating encoding tables at startup.

Top-posted from my Windows Phone

-Original Message-
From: "francismb" 
Sent: ‎1/‎29/‎2016 13:56
To: "python-dev@python.org" 
Subject: Re: [Python-Dev] More optimisation ideas

Hi,

> 
> Storing these in static data is a tradeoff between
> disk space and startup performance, and one I think it likely to be
> worthwhile.

it's really an important trade off? As far a I understand from your
email those modules are always being loaded and the final data created.
won't the space be there (on mem or disk)?

Thanks in advance!
francis


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Speeding up CPython 5-10%

[Steven D'Aprano]

On Wed, Jan 27, 2016 at 01:25:27PM -0500, Yury Selivanov wrote:
> Hi,
> 
> 
> tl;dr The summary is that I have a patch that improves CPython 
> performance up to 5-10% on macro benchmarks.  Benchmarks results on 
> Macbook Pro/Mac OS X, desktop CPU/Linux, server CPU/Linux are available 
> at [1].  There are no slowdowns that I could reproduce consistently.

Have you looked at Cesare Di Mauro's wpython? As far as I know, it's now 
unmaintained, and the project repo on Google Code appears to be dead (I 
get a 404), but I understand that it was significantly faster than 
CPython back in the 2.6 days.

https://wpython.googlecode.com/files/Beyond%20Bytecode%20-%20A%20Wordcode-based%20Python.pdf



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] More optimisation ideas

[Steve Dower]

Since we're all talking about making Python faster, I thought I'd drop 
some previous ideas I've had here in case (1) someone wants to actually 
do them, and (2) they really are new ideas that haven't failed in the 
past. Mostly I was thinking about startup time.


Here are the list of modules imported on clean startup on my Windows, 
US-English machine (from -v and cleaned up a bit):


import _frozen_importlib
import _imp
import sys
import '_warnings'
import '_thread'
import '_weakref'
import '_frozen_importlib_external'
import '_io'
import 'marshal'
import 'nt'
import '_thread'
import '_weakref'
import 'winreg'
import 'zipimport'
import '_codecs'
import 'codecs'
import 'encodings.aliases'
import 'encodings'
import 'encodings.mbcs'
import '_signal'
import 'encodings.utf_8'
import 'encodings.latin_1'
import '_weakrefset'
import 'abc'
import 'io'
import 'encodings.cp437'
import 'errno'
import '_stat'
import 'stat'
import 'genericpath'
import 'ntpath'
import '_collections_abc'
import 'os'
import '_sitebuiltins'
import 'sysconfig'
import '_locale'
import '_bootlocale'
import 'encodings.cp1252'
import 'site'

Obviously the easiest first thing is to remove or delay unnecessary 
imports. But a while ago I used a native profiler to trace through this 
and the most impactful modules were the encodings:


import 'encodings.mbcs'
import 'encodings.utf_8'
import 'encodings.latin_1'
import 'encodings.cp437'
import 'encodings.cp1252'

While I don't doubt that we need all of these for *some* reason, 
aliases, cp437 and cp1252 are relatively expensive modules to import. 
Mostly due to having large static dictionaries or data structures 
generated on startup.


Given this is static and mostly read-only information[1], I see no 
reason why we couldn't either generate completely static versions of 
them, or better yet compile the resulting data structures into the core 
binary.


([1]: If being able to write to some of the encoding data is used by 
some people, I vote for breaking that for 3.6 and making it read-only.)


This is probably the code snippet that bothered me the most:

### Encoding table
encoding_table=codecs.charmap_build(decoding_table)

It shows up in many of the encodings modules, and while it is not a bad 
function in itself, we are obviously generating a known data structure 
on every startup. Storing these in static data is a tradeoff between 
disk space and startup performance, and one I think it likely to be 
worthwhile.


Anyway, just an idea if someone wants to try it and see what 
improvements we can get. I'd love to do it myself, but when it actually 
comes to finding time I keep coming up short.


Cheers,
Steve


P.S. If you just want to discuss optimisation techniques or benchmarking 
in general, without specific application to CPython 3.6, there's a whole 
internet out there. Please don't make me the cause of a pointless 
centithread. :)

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How to resolve distinguishing between documentation and implementation

[Guido van Rossum]

Typically we fix this in the next feature release but not in bugfix
releases, and that's what I recommend in this case. But deciding
remains an art, not an exact science.

People who catch specific lists of exceptions based on experience are
bound to run into trouble occasionally, so I have little pity on them
when they upgrade to the next feature release (especially when the new
behavior is more logical and follows an established standard). For
bugfix releases I like to be much more conservative, since people
often don't have control over which bugfix release is installed.

On Fri, Jan 29, 2016 at 10:14 AM, Serhiy Storchaka  wrote:
> How to resolve distinguishing between documentation and implementation if
> current implementation is incorrect, but third-party code can implicitly
> depends on it?
>
> For example see issue26198. Currently buffer overflow of predefined buffer
> for "es#" and "et#" format units causes TypeError (with misleading message,
> but this is other story). The correct and *documented* exception is
> ValueError. User code can depend on current behavior, because TypeError is
> what is raised now for this type of errors, and this is what is raised for
> other types of errors. Unlikely authors of such code read the documentation,
> otherwise this issue would be reported earlier. On other hand, looks these
> format units are rarely used with predefined buffer (never in the stdlib
> since 3.5).
>
> I think it is obvious that the code in the development branch should be
> changed to produce documented and more logical exception. But what about
> bugfix releases? Changing the documentation would be misleading, changing
> the code can break existing code (unlikely, but).
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2016-01-22 - 2016-01-29)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open5381 (+27)
  closed 32615 (+30)
  total  37996 (+57)

Open issues with patches: 2359 


Issues opened (39)
==

#26182: Deprecation warnings for the future async and await keywords
http://bugs.python.org/issue26182  opened by marco.buttu

#26184: raise an error when create_module() is not defined by exec_mod
http://bugs.python.org/issue26184  opened by brett.cannon

#26185: zipfile.ZipInfo slots can raise unexpected AttributeError
http://bugs.python.org/issue26185  opened by Matthew Zipay

#26186: LazyLoader rejecting use of SourceFileLoader
http://bugs.python.org/issue26186  opened by brett.cannon

#26187: sqlite3 trace callback prints duplicate line
http://bugs.python.org/issue26187  opened by palaviv

#26188: Provide more helpful error message when `await` is called insi
http://bugs.python.org/issue26188  opened by Nicholas Chammas

#26192: python3 k1om dissociation permanence: libffi
http://bugs.python.org/issue26192  opened by mancoast

#26193: python3 k1om dissociation permanence: readelf
http://bugs.python.org/issue26193  opened by mancoast

#26194: Undefined behavior for deque.insert() when len(d) == maxlen
http://bugs.python.org/issue26194  opened by rhettinger

#26195: Windows frozen .exe multiprocessing.Queue access is denied exc
http://bugs.python.org/issue26195  opened by alex_python_org

#26198: PyArg_ParseTuple with format "et#" and "es#" detects overflow 
http://bugs.python.org/issue26198  opened by hniksic

#26200: SETREF adds unnecessary work in some cases
http://bugs.python.org/issue26200  opened by rhettinger

#26204: compiler: ignore constants used as statements? (don't emit LOA
http://bugs.python.org/issue26204  opened by haypo

#26205: Inconsistency concerning nested scopes
http://bugs.python.org/issue26205  opened by Roscoe R. Higgins

#26207: distutils msvccompiler fails due to mspdb140.dll error on debu
http://bugs.python.org/issue26207  opened by haypo

#26208: decimal C module's exceptions don't match the Python version
http://bugs.python.org/issue26208  opened by encukou

#26209: TypeError in smtpd module with string arguments
http://bugs.python.org/issue26209  opened by lorenzo.ancora

#26210: `HTMLParser.handle_data` may be invoked although `HTMLParser.r
http://bugs.python.org/issue26210  opened by Hibou57

#26212: Python with ncurses6.0 will not load _curses module on Solaris
http://bugs.python.org/issue26212  opened by jonesrw

#26213: Document BUILD_*_UNPACK opcodes
http://bugs.python.org/issue26213  opened by brett.cannon

#26214: textwrap should minimize number of breaks in extra long words
http://bugs.python.org/issue26214  opened by Tuomas Salo

#26215: Make GC_Head a compile-time option
http://bugs.python.org/issue26215  opened by yuriy_levchenko

#26216: run runtktests.py error when test tkinter
http://bugs.python.org/issue26216  opened by allensll

#26218: Set PrependPath default to true
http://bugs.python.org/issue26218  opened by Wallison Resende Santos

#26219: implement per-opcode cache in ceval
http://bugs.python.org/issue26219  opened by yselivanov

#26221: asynco run_in_executor swallows StopIteration
http://bugs.python.org/issue26221  opened by ikelly

#26222: Missing code in linux_distribution python 2.7.11
http://bugs.python.org/issue26222  opened by Rasmus Rynning Rasmussen

#26223: decimal.to_eng_string() does not implement engineering notatio
http://bugs.python.org/issue26223  opened by serge.stroobandt

#26224: Add "version added" for documentation of asyncio.timeout for d
http://bugs.python.org/issue26224  opened by Udi Oron

#26225: New misleading wording in execution model documenation
http://bugs.python.org/issue26225  opened by abarnert

#26226: Various test suite failures on Windows
http://bugs.python.org/issue26226  opened by ebarry

#26228: pty.spawn hangs on FreeBSD 9.3, 10.x
http://bugs.python.org/issue26228  opened by chris.torek

#26229: Make number serialization ES6/V8 compatible
http://bugs.python.org/issue26229  opened by anders.rundgren@gmail.com

#26231: HTTPResponse.close() should consume all remaining data in body
http://bugs.python.org/issue26231  opened by Jacky

#26233: select.epoll.poll() should avoid calling malloc() each time
http://bugs.python.org/issue26233  opened by haypo

#26234: The typing module includes 're' and 'io' in __all__
http://bugs.python.org/issue26234  opened by gvanrossum

#26235: argparse docs: Positional * argument in mutually exclusive gro
http://bugs.python.org/issue26235  opened by paul.j3

#26236: urllib2 initiate irregular call to gethostbyaddr
http://bugs.python.org/issue26236  opened by juliadolgova

#26238: httplib use wrong hostname in https request with SNI support
http://bugs.python.org/issue26238  opened by lvhancy



Most recent 15 issues with no 

[Python-Dev] How to resolve distinguishing between documentation and implementation

[Serhiy Storchaka]

How to resolve distinguishing between documentation and implementation 
if current implementation is incorrect, but third-party code can 
implicitly depends on it?


For example see issue26198. Currently buffer overflow of predefined 
buffer for "es#" and "et#" format units causes TypeError (with 
misleading message, but this is other story). The correct and 
*documented* exception is ValueError. User code can depend on current 
behavior, because TypeError is what is raised now for this type of 
errors, and this is what is raised for other types of errors. Unlikely 
authors of such code read the documentation, otherwise this issue would 
be reported earlier. On other hand, looks these format units are rarely 
used with predefined buffer (never in the stdlib since 3.5).


I think it is obvious that the code in the development branch should be 
changed to produce documented and more logical exception. But what about 
bugfix releases? Changing the documentation would be misleading, 
changing the code can break existing code (unlikely, but).


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How to resolve distinguishing between documentation and implementation

[Brett Cannon]

On Fri, 29 Jan 2016 at 10:14 Serhiy Storchaka  wrote:

> How to resolve distinguishing between documentation and implementation
> if current implementation is incorrect, but third-party code can
> implicitly depends on it?
>
> For example see issue26198. Currently buffer overflow of predefined
> buffer for "es#" and "et#" format units causes TypeError (with
> misleading message, but this is other story). The correct and
> *documented* exception is ValueError. User code can depend on current
> behavior, because TypeError is what is raised now for this type of
> errors, and this is what is raised for other types of errors. Unlikely
> authors of such code read the documentation, otherwise this issue would
> be reported earlier. On other hand, looks these format units are rarely
> used with predefined buffer (never in the stdlib since 3.5).
>
> I think it is obvious that the code in the development branch should be
> changed to produce documented and more logical exception. But what about
> bugfix releases? Changing the documentation would be misleading,
> changing the code can break existing code (unlikely, but).
>

When the potential breakage is low, I would move to the more reasonable
exception and add the appropriate note to the docs and What's New about how
to port pre-existing code by either changing the exception caught or
catching both exceptions until support < 3.6 can be dropped by the user.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com