On 03/12/2019 16:15, Mark Shannon wrote:
Hi Everyone,
I am proposing a new PEP, still in draft form, to impose a limit of one
million on various aspects of Python programs, such as the lines of code
per module.
Any thoughts or feedback?
The PEP:
https://github.com/markshannon/peps/blob/one-million/pep-1000000.rst
Cheers,
Mark.
Full text
*********
PEP: 1000000
Title: The one million limit
Author: Mark Shannon <m...@hotpy.org>
Status: Active
Type: Enhancement
Content-Type: text/x-rst
Created: 03-Dec-2019
Post-History:
Abstract
========
This PR proposes a limit of one million (1 000 000) for various aspects
of Python code and its implementation.
The Python language does not specify limits for many of its features.
Not having any limit to these values seems to enhance programmer freedom,
at least superficially, but in practice the CPython VM and other Python
virtual
machines have implicit limits or are forced to assume that the limits are
astronomical, which is expensive.
This PR lists a number of features which are to have a limit of one
million.
If a language feature is not listed but appears unlimited and must be
finite,
for physical reasons if no other, then a limit of one million should be
assumed.
Motivation
==========
There are many values that need to be represented in a virtual machine.
If no limit is specified for these values, then the representation must
either be inefficient or vulnerable to overflow.
The CPython virtual machine represents values like line numbers,
stack offsets and instruction offsets by 32 bit values. This is
inefficient, and potentially unsafe.
It is inefficient as actual values rarely need more than a dozen or so
bits to represent them.
It is unsafe as malicious or poorly generated code could cause values to
exceed 2\ :sup:`32`.
For example, line numbers are represented by 32 bit values internally.
This is inefficient, given that modules almost never exceed a few
thousand lines.
Despite being inefficent, is is still vulnerable to overflow as
it is easy for an attacker to created a module with billions of newline
characters.
Memory access is usually a limiting factor in the performance of modern
CPUs.
Better packing of data structures enhances locality and reduces memory
bandwith,
at a modest increase in ALU usage (for shifting and masking).
Being able to safely store important values in 20 bits would allow
memory savings
in several data structures including, but not limited to:
* Frame objects
* Object headers
* Code objects
There is also the potential for a more efficient instruction format,
speeding up interpreter dispatch.
Rationale
=========
Imposing a limit on values such as lines of code in a module, and the
number of local variables,
has significant advantages for ease of implementation and efficiency of
virtual machines.
If the limit is sufficiently large, there is no adverse effect on users
of the language.
By selecting a fixed but large limit for these values,
it is possible to have both safety and efficiency whilst causing no
inconvience to human programmers
and only very rare problems for code generators.
One million
-----------
The Java Virtual Machine (JVM) [1]_ specifies a limit of 2\ :sup:`16`-1
(65535) for many program
elements similar to those covered here.
This limit enables limited values to fit in 16 bits, which is a very
efficient machine representation.
However, this limit is quite easily exceeded in practice by code
generators and
the author is aware of existing Python code that already exceeds 2\
:sup:`16` lines of code.
A limit of one million fits into 20 bits which, although not as
convenient for machine representation,
is still reasonably compact. Three signed valuses in the range -1000_000
to +1000_000 can fit into a 64 bit word.
A limit of one million is small enough for efficiency advantages (only
20 bits),
but large enough not to impact users (no one has ever written a module
of one million lines).
OK, let me stop you here. If you have twenty bits of information,
you'll be fitting them into a 32-bit word anyway. Anything else will be
more or less inefficient to access, depending on your processor. You
aren't going to save anything there.
If you have plans to use the spare bits for something else, please
don't. I've seen this done in two major architectures (status flags for
both the IBM System/370 and ARM 2 and 3 architectures lived in the top
bits of the program counter), and it was acknowledged to be a major
mistake both times. Aside from limiting your expansion (Who would ever
want more than 24 bits of address space? Everyone, it turns out :-),
every access you make to that word is going to need to mask out some
bits of the word. You would take an efficiency hit on every access.
Isn't this "640K ought to be enough for anybody" again?
-------------------------------------------------------
The infamous 640K memory limit was a limit on machine usable resources.
The proposed one million limit is a limit on human generated code.
While it is possible that generated code could exceed the limit,
it is easy for a code generator to modify its output to conform.
The author has hit the 64K limit in the JVM on at least two occasions
when generating Java code.
The workarounds were relatively straightforward and
probably wouldn't have been necessary with a limit of one million
bytecodes or lines of code.
I can absolutely guarantee that this will come back and bite you.
Someone out there will be doing something more complicated than you
think is plausible, and eventually someone will hit your limits. It may
not take as long as you think, either.
--
Rhodri James *-* Kynesim Ltd
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/6BJVZ6KA3OSDP5ID2RFZM3KRGLZS6VAD/
Code of Conduct: http://python.org/psf/codeofconduct/
