[Python-Dev] Re: PEP 611 -- why limit coroutines and classes?

[Kyle Stanley]

Antoine Pitrou wrote:
> 1M concurrent database transactions?  Does that sound reasonable at
> all?  Your database administrator probably won't like you.

I agree that 1M concurrent transactions would not be reasonable for the
vast majority of database configurations, I didn't mean to specifically
imply that 1M would be something reasonable today. That's why I said "a
large number of concurrent transactions" instead of specifically saying "1M
concurrent transactions". I honestly don't know if any databases are close
to capable of handling that many transactions at once, at least not at the
present time.

But, although we may think it's ridiculous today, who's to say that won't
be an occurrence in the future? Imagine a scenario where there was a
massive super-cluster of database servers, that performed real-time update
transactions every time a single item was checked in and out of some global
inventory system.

Currently, something like this would likely have to be implemented through
batching, where x number of updates have to be queued or x amount of time
has to pass before the next transaction is started. Alternatively, each
facility or region would have its own local database that synchronizes with
the global database every so often. But, there could be a significant
advantage in having a near-perfectly synchronized global inventory system,
which would only be possible if it was updated in real-time. IMO, the max
number of concurrent transactions that the a database system can handle at
once is a very clear application of Moore's Law.

My point being is that I don't want to arbitrarily restrict how many
coroutine objects can exist at once without having a strong reason for
doing so AND having a limit that's reasonable in the long term. 1M would
have the advantage of being easy to remember, but other than that I see no
reason why that should specifically be the limit for the max number of
coroutines. As Guido mentioned at the start of the thread, a coroutine
object is "just another Python object and has no operating resources
associated with it". Other than main memory usage, there's no other
external limit to the max number of coroutine objects that can exist at
once.

Note: Although coroutines were already dropped from PEP 611, I felt that
this response was still worthwhile to write. I suspect that the topic of
"coroutine object limits" is likely to come up again in the future.


On Wed, Dec 11, 2019 at 4:46 PM Antoine Pitrou  wrote:

> On Mon, 9 Dec 2019 21:42:36 -0500
> Kyle Stanley  wrote:
>
> > > (b) Why limit coroutines? It's just another Python object and has no
> > operating resources associated with it. Perhaps your definition of
> > coroutine is different, and you are thinking of OS threads?
> >
> > This was my primary concern with the proposed PEP. At the moment, it's
> > rather trivial to create one million coroutines, and the total memory
> taken
> > up by each individual coroutine object is very minimal compared to each
> OS
> > thread.
> >
> > There's also a practical use case for having a large number of coroutine
> > objects, such as for asynchronously:
> >
> > 1) Handling a large number of concurrent clients on a continuously
> running
> > web server that receives a significant amount of traffic.
>
> Not sure how that works?  Each client has an accepted socket, which is
> bound to a local port number, and there are 65536 TCP port numbers
> available.  Unless you're using 15+ coroutines per client, you probably
> won't reach 1M coroutines that way.
>
> > 2) Sending a large number of concurrent database transactions to run on a
> > cluster of database servers.
>
> 1M concurrent database transactions?  Does that sound reasonable at
> all?  Your database administrator probably won't like you.
>
> > something like this definitely scales over time. Arbitrarily placing a
> > limit on the total number of coroutine objects doesn't make sense to me
> for
> > that reason.
>
> There are a lot of arbitrary limits inside a computer system.  You just
> aren't aware of them because you don't hit them in practice.  Claiming
> that limits shouldn't exist is just pointless.
>
> Regards
>
> Antoine.
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/O3ZODXHEIJ2SM5SZBOVJ4PIAQMSYNXEJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PYUSGK4UDMGGDJPQ6E6YHKG2O5D5DTEL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 611 -- why limit coroutines and classes?

[Kyle Stanley]

Andrew Svetlov wrote:
> > Not sure how that works?  Each client has an accepted socket, which is
> > bound to a local port number, and there are 65536 TCP port numbers
> > available.  Unless you're using 15+ coroutines per client, you probably
> > won't reach 1M coroutines that way.
>
> I'm sorry, but the accepted socket has the same local port number as
> the listening one.
> Routing is performed by (local_ip, local_port, remote_ip, remote_port)
quad.

IIRC, that combination is typically referred to as the "port address" (IP
addr + port num source, IP addr + port num destination). All four are are
required in TCP, and in UDP, the IP and source port are optional. So in
UDP, this could potentially just be (remote_ip, remote_port).

Also, it's possible to bind multiple AF_INET (or AF_INET6) sockets to a
single port address by using the SO_REUSEPORT socket option, which we
discussed recently in bpo-37228 (https://bugs.python.org/issue37228). The
only requirement is that the same UID is used for each socket bound to the
same port address (from my understanding, SO_REUSEPORT is typically used
for binding a single process to multiple listening sockets).

TL;DR: It's definitely possible to have more than one client per TCP port.

I'll admit that I've never personally seen production code that uses
anywhere near 1M coroutine objects, but we shouldn't limit users from doing
that without a good reason. At the present moment, it's rather trivial to
create 1M coroutine objects on any system with ~1.3GB+ available main
memory (see my code example in
https://mail.python.org/archives/list/python-dev@python.org/message/WYZHKRGNOT252O3BUTFNDVCIYI6WSBXZ/
).

There's also the infamous "10M" problem, of accepting 10 million concurrent
clients without significant performance issues. This is mostly theoretical
at the moment, but there's an article that explains how the problem could
be addressed by using 10M goroutines: https://goroutines.com/10m. I see no
reason why this couldn't be potentially translated into Python's
coroutines, with the usage of an optimized event loop policy such as uvloop.

But, either way, Mark Shannon removed the 1M coroutine limit from PEP 611,
due to it having the least strong argument out of all of the proposed
limits and a significant amount of push-back from the dev community.

Andrew Svetlov wrote:
> The listening socket can accept hundreds of thousands of concurrent
> client connections.
> The only thing that should be tuned for this is increasing the limit
> of file descriptors.

The default soft limit of file descriptors per process on Linux is 1024
(which can be increased), but you could exceed a per-process limitation of
file descriptors by forking child processes. I'm have no idea what the
realistic maximum limit of global FDs would be for most modern servers
though, but here's the upper bound limit on Linux kernel 5.3.13:

[aeros:~]$ cat /proc/sys/fs/file-max
9223372036854775807

My system's current hard limit of file descriptors is much lower, but is
still fairly substantial:

[aeros:~]$ ulimit -nH
524288

I recall reading somewhere that per additional 100 file descriptors, it
requires approximately 1MB of main memory. Based on that estimate, 1M FDs
would require ~10GB+. This doesn't seem unreasonable to me, especially on a
modern server. But I'd imagine the actual memory usage depends upon how
much data is being buffered at once through the pipes associated with each
FD, but I believe this can be limited through the FD option F_SETPIPE_SZ (
https://linux.die.net/man/2/fcntl).

Note: I was unable to find a credible source on the minimum memory usage
per additional FD, so clarification on that would be appreciated.

On Wed, Dec 11, 2019 at 5:06 PM Andrew Svetlov 
wrote:

> On Wed, Dec 11, 2019 at 11:52 PM Antoine Pitrou 
> wrote:
> >
> > On Mon, 9 Dec 2019 21:42:36 -0500
> > Kyle Stanley  wrote:
> >
> > >
> > > There's also a practical use case for having a large number of
> coroutine
> > > objects, such as for asynchronously:
> > >
> > > 1) Handling a large number of concurrent clients on a continuously
> running
> > > web server that receives a significant amount of traffic.
> >
> > Not sure how that works?  Each client has an accepted socket, which is
> > bound to a local port number, and there are 65536 TCP port numbers
> > available.  Unless you're using 15+ coroutines per client, you probably
> > won't reach 1M coroutines that way.
> >
>
> I'm sorry, but the accepted socket has the same local port number as
> the listening one.
> Routing is performed by (local_ip, local_port, remote_ip, remote_port)
> quad.
>
> The listening socket can accept hundreds of thousands of concurrent
> client connections.
> The only thing that should be tuned for this is increasing the limit
> of file descriptors.
>
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > 

[Python-Dev] Re: Please be more precise when commenting on PEP 611.

[Richard Damon]

I've been restraining myself from commenting in this threads, as I don't
know enough details of the implementation of Python to make reasoned
arguments on how much such limits would help in the efficiency for
running a Python program.  I will way that in my many years of
programming experience I can't think of any great cases where a language
as part of the language definition limited to 'size' of a program to
good effect, and generally such limits relegate a language into being
seen as a 'toy language'. The biggest issue is that computers are
growing more powerful every day, and programs follow in getting bigger,
so any limit that we think of as more than sufficient soon becomes too
small (No one will need more than 640k of RAM).

I COULD easily see on the other hand, the language stating that some
implementations might impose certain limits like those being mentioned,
and maybe provide lower limits of those limits, that barring system
resource limits a program can expect to be able to use. If a program
needs higher limits, it can use a less limiting implementation. I could
see a fork of the current CPython implementation happening, where one
path keeps the current lack of limits, and another branch tries imposing
the limits to see how much benefit you really get. After seeing, one of
the forks might just die off as not being needed any more.

I would see those limits being added to the specific implementation that
provide identifiable advantage to making the system better. Being done,
at least initially, in a fork, there might not need to be a lot of
discussion on exactly what limits to try imposing, as 'Python' as a base
language wouldn't be changing, just some limits in a particular branch
implementation. it still might make a lot of sense for some discussion
to occur to find out if the limits being discussed are reasonable.

Perhaps the one big disadvantage of this idea is that the person wanting
to see what these limits might be able to do for the language would need
to be more involved in seeing it happen, as there won't be the easy
opportunity to propose a somewhat vague idea, get it approved, and have
someone else stuck with the job of getting it done.

-- 
Richard Damon
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QRP32EQPAQH53NR5NGHE55MCALEBJ5J6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 611 - saturating reference counts

[Random832]

as an aside in the portion of PEP 611 proposing a limit to the number of 
classes, the proposal also mentioned reducing the size of the reference count 
field to achieve part of its object size savings.

one of the mechanisms proposed was to implement a "saturating" reference count, 
though it would be different from conventional saturating arithmetic in that 
any object that reaches the maximum refcount would become immortal, since we 
could not trust decref from there to account for all references.

something that may be worth considering or testing is: what would the cost, 
both in performance and interpreter code size, of implementing this logic on 
every single incref and decref?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7KUAXS7W245KM42Z6CUVTIAY5GNTENI5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python 3.8 error?

[Christian Tismer]

Hi all,

Sorry for the noise, I was wrong, and I retract.
I was somehow mislead and hunted a phantom.

Best - Chris


On 10.12.19 00:29, Christian Tismer wrote:
> On 09.12.19 23:26, Nick Coghlan wrote:
>>
>>
>> On Tue., 10 Dec. 2019, 5:17 am MRAB, > > wrote:
>>
>> On 2019-12-09 18:22, Christian Tismer wrote:
>> >
>> >
>> > Hi Nick,
>> >
>> > after staring long at the code, I fount something funny in
>> > typeobject.c #286 ff:
>> >
>> >
>> > static void
>> > type_mro_modified(PyTypeObject *type, PyObject *bases) {
>> >      /*
>> >         Check that all base classes or elements of the MRO of type are
>> >         able to be cached.  This function is called after the base
>> >         classes or mro of the type are altered.
>> >
>> >         Unset HAVE_VERSION_TAG and VALID_VERSION_TAG if the type
>> >         has a custom MRO that includes a type which is not officially
>> >         super type, or if the type implements its own mro() method.
>> >
>> >         Called from mro_internal, which will subsequently be called on
>> >         each subclass when their mro is recursively updated.
>> >       */
>> >      Py_ssize_t i, n;
>> >      int custom = (Py_TYPE(type) != _Type);
>> >      int unbound;
>> >      PyObject *mro_meth = NULL;
>> >      PyObject *type_mro_meth = NULL;
>> >
>> >      if (!PyType_HasFeature(type, Py_TPFLAGS_HAVE_VERSION_TAG))
>> >          return;
>> >
>> >      if (custom) {
>> >          _Py_IDENTIFIER(mro);
>> >          mro_meth = lookup_maybe_method(
>> >              (PyObject *)type, _mro, );
>> >          if (mro_meth == NULL)
>> >              goto clear;
>> >          type_mro_meth = lookup_maybe_method(
>> >              (PyObject *)_Type, _mro, );
>> >          if (type_mro_meth == NULL)
>> >              goto clear;
>> >          if (mro_meth != type_mro_meth)
>> >              goto clear;
>> >          Py_XDECREF(mro_meth);
>> >          Py_XDECREF(type_mro_meth);
>> >      }
>> >
>> >
>> > Look at the "if (custom)" clause.
>> > "mro_meth = lookup_maybe_method(" uses lookup_maybe_method which
>> > gives a borrowed reference. The same holds for "type_mro_meth".
>> >
>> > But then both are decreffed, which IMHO is not correct.
>> >
>> Look at what happens at the label "clear": it DECREFs them.
>>
>> If mro_meth != NULL or mro_meth != type_mro_meth, they'll get DECREFed
>> at "clear".
>>
>>
>> I believe Christian's point is that this entire "if (custom) {" branch
>> looks suspicious, as it assumes "lookup_maybe_method" will increment the
>> refcount on the returned object. If that assumption is incorrect, we're
>> going to get DECREFs without a preceding INCREF.
>>
>> The specific code path is also obscure enough that it's plausible the
>> test suite may not currently cover it (as it requires doing something
>> that calls "type_mro_modified" on a type with a custom metaclass).
> 
> Thanks Nick for this nice analysis. And it's exactly this codepath that
> is taken in the case of PySide: custom types all the time :-)
> 
> what-a-relief - ly y'rs -- Chris
> 
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/5MTJC47BHGZW6RKKU5JDAIFLV5P6W6JA/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 


-- 
Christian Tismer :^)   tis...@stackless.com
Software Consulting  : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam: GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776  fax +49 (30) 700143-0023



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5PBHW3HEQQ4KLRUATZ7YGX5TR3T24R4C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 611 -- why limit coroutines and classes?

[Andrew Svetlov]

On Wed, Dec 11, 2019 at 11:52 PM Antoine Pitrou  wrote:
>
> On Mon, 9 Dec 2019 21:42:36 -0500
> Kyle Stanley  wrote:
>
> >
> > There's also a practical use case for having a large number of coroutine
> > objects, such as for asynchronously:
> >
> > 1) Handling a large number of concurrent clients on a continuously running
> > web server that receives a significant amount of traffic.
>
> Not sure how that works?  Each client has an accepted socket, which is
> bound to a local port number, and there are 65536 TCP port numbers
> available.  Unless you're using 15+ coroutines per client, you probably
> won't reach 1M coroutines that way.
>

I'm sorry, but the accepted socket has the same local port number as
the listening one.
Routing is performed by (local_ip, local_port, remote_ip, remote_port) quad.

The listening socket can accept hundreds of thousands of concurrent
client connections.
The only thing that should be tuned for this is increasing the limit
of file descriptors.

> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/O3ZODXHEIJ2SM5SZBOVJ4PIAQMSYNXEJ/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Thanks,
Andrew Svetlov
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZVTAHRNWGI4ESWRT44PG3JUJLWJBYXFT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 611 -- why limit coroutines and classes?

[Chris Angelico]

On Thu, Dec 12, 2019 at 8:46 AM Antoine Pitrou  wrote:
> > 1) Handling a large number of concurrent clients on a continuously running
> > web server that receives a significant amount of traffic.
>
> Not sure how that works?  Each client has an accepted socket, which is
> bound to a local port number, and there are 65536 TCP port numbers
> available.  Unless you're using 15+ coroutines per client, you probably
> won't reach 1M coroutines that way.

Each client has a socket, yes, but they all use the same local port
number. Distinct sockets are identified by the tuple (TCP, LocalAddr,
LocalPort, RemoteAddr, RemotePort) and can quite happily duplicate on
any part of that as long as they can be distinguished by some other
part.

There will be other limits, though. On Linux (and probably most
Unix-like systems), every socket requires a file descriptor, and
you're limited to a few hundred thousand of those, I think. On
Windows, sockets aren't the same things as files, so I don't know what
the limit actually is, but there'll be one somewhere.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3CWDIR5WPMW67GUMH4743SMELDKRQCRC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 611 -- why limit coroutines and classes?

[Antoine Pitrou]

On Mon, 9 Dec 2019 21:42:36 -0500
Kyle Stanley  wrote:

> > (b) Why limit coroutines? It's just another Python object and has no  
> operating resources associated with it. Perhaps your definition of
> coroutine is different, and you are thinking of OS threads?
> 
> This was my primary concern with the proposed PEP. At the moment, it's
> rather trivial to create one million coroutines, and the total memory taken
> up by each individual coroutine object is very minimal compared to each OS
> thread.
> 
> There's also a practical use case for having a large number of coroutine
> objects, such as for asynchronously:
> 
> 1) Handling a large number of concurrent clients on a continuously running
> web server that receives a significant amount of traffic.

Not sure how that works?  Each client has an accepted socket, which is
bound to a local port number, and there are 65536 TCP port numbers
available.  Unless you're using 15+ coroutines per client, you probably
won't reach 1M coroutines that way.

> 2) Sending a large number of concurrent database transactions to run on a
> cluster of database servers.

1M concurrent database transactions?  Does that sound reasonable at
all?  Your database administrator probably won't like you.

> something like this definitely scales over time. Arbitrarily placing a
> limit on the total number of coroutine objects doesn't make sense to me for
> that reason.

There are a lot of arbitrary limits inside a computer system.  You just
aren't aware of them because you don't hit them in practice.  Claiming
that limits shouldn't exist is just pointless.

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/O3ZODXHEIJ2SM5SZBOVJ4PIAQMSYNXEJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 611: The one million limit.

[Antoine Pitrou]

On Fri, 6 Dec 2019 15:19:48 -0800
"Gregory P. Smith"  wrote:
> I'd prefer it if we stayed on topic here...

+1.  Please take rants about 30-year old CPU designs elsewhere (for
example Hacker News, Twitter, Facebook, etc.).

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7EOMLYVSSVPALMQZ67TPEO74FQBZ4YIL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 611: The one million limit.

[Antoine Pitrou]

On Fri, 6 Dec 2019 13:54:13 +
Rhodri James  wrote:
> Apologies again for commenting in the wrong place.
> 
> On 05/12/2019 16:38, Mark Shannon wrote:
> 
> > Memory access is usually a limiting factor in the performance of 
> > modern CPUs. Better packing of data structures enhances locality and>
> > reduces memory bandwith, at a modest increase in ALU usage (for 
> > shifting and masking).  
> 
> I don't think this assertion holds much water:
> 
> 1. Caching make memory access much less of a limit than you would expect.
> 2. Non-aligned memory access vary from inefficient to impossible 
> depending on the processor.
> 3. Shifting and masking isn't free, and again on some processors can be 
> very expensive.

I think your knowledge is outdated.  Shifts and masks are extremely
fast on modern CPUs, and unaligned loads are fast as well (when
served from the CPU cache).  Moreover, modern CPUs are superscalar with
many different execution units, so those instructions can be executed in
parallel with other independent instructions.

However, as soon as you load from main memory because of a cache miss,
you take a hit of several hundreds cycles.  Basically, computations are
almost free compared to the cost of memory accesses.

In any case, this will have to be judged on benchmark numbers, once
Mark (or someone else) massages the interpreter to experiment with
those runtime memory footprint reductions.

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LC6K5SQP2GUQQAARPBFAJTEQFBSTXZVF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP proposal to limit various aspects of a Python program to one million.

[Antoine Pitrou]

On Tue, 10 Dec 2019 00:59:09 +
Oscar Benjamin  wrote:
> 
> > Note that changing pytest to output fewer lines won't work as we will
> > just hit the bytecode limit instead.  
> 
> I'm not sure. I think that pytest should have some kind of limit on
> what it produces in this situation.

Agreed. I think pytest just needs to be fixed not to stupid thing.

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UUNMLZ7S5VZ2FDPJI2CYNGLIIOUM76IU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [python-committers] [RELEASE] Python 3.8.1rc1 is now available for testing

[Brett Cannon]

Jason R. Coombs wrote:
> I think I missed the announcement of the cutoff date for 3.8.1; I was hoping 
> to get
> some bug fixes in for 
> importlib.metadata.https://github.com/python/cpython/pull/17568
> These aren’t crucial bugfixes, but it would be nice not to have them linger 
> for months.

Just to be clear, the lingering will last two months, so it isn't _too_ bad. 
Plus in this instance people could use the PyPI package if they need the fixes 
faster.

Not to say getting them in sooner wouldn't be nice either.  I just wanted to 
make sure people know that the bugfix schedule is faster than it used to be.

-Brett

> Would you consider including these, especially as the code changes are 
> pre-vetted in the
> backport (released 12-01)? Or maybe only if there’s another RC for another 
> reason?
> If it’s too disruptive, that’s no big deal. Your call.
> Thanks for the release work.
> On 10 Dec, 2019, at 04:22, Łukasz Langa 
> luk...@langa.pl
> wrote:
> Python 3.8.1rc1 is the release candidate of the first maintenance release of 
> Python
> 3.8.
> The Python 3.8 series is the newest feature release of the Python language, 
> and it
> contains many new features and optimizations. You can find Python 3.8.1rc1 
> here:
> https://www.python.org/downloads/release/python-381rc1/
> Assuming no critical problems are found prior to 2019-12-16, the scheduled 
> release date
> for 3.8.1 as well as Ned Deily's birthday, no code changes are planned 
> between this
> release candidate and the final release.
> That being said, please keep in mind that this is a pre-release of 3.8.1 and 
> as such
> its main purpose is testing.
> See the “What’s New in Python 
> 3.8https://docs.python.org/3.8/whatsnew/3.8.html”
> document for more information about features included in the 3.8 series. 
> Detailed
> information about all changes made in 3.8.0 can be found in its change log.
> Maintenance releases for the 3.8 series will continue at regular bi-monthly 
> intervals,
> with 3.8.2 planned for February 2020.
> We hope you enjoy Python 3.8!
> Thanks to all of the many volunteers who help make Python Development and 
> these
> releases possible! Please consider supporting our efforts by volunteering 
> yourself or
> through organization contributions to the Python Software Foundation.
> https://www.python.org/psf/
> 
> python-committers mailing list -- 
> python-committers@python.orgmailto:python-committ...@python.org
> To unsubscribe send an email to 
> python-committers-leave@python.orgmailto:python-committers-le...@python.org
> https://mail.python.org/mailman3/lists/python-committers.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-committ...@python.org/message/I...
> Code of Conduct: https://www.python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6PW7EMRPXNDVB5JTYEZXLH2W7E32GUEX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: More feedback on PEP 611, please

[Jim J. Jewett]

David Mertz wrote:
> On Wed, Dec 11, 2019, 6:40 AM Mark Shannon m...@hotpy.org
> wrote:

> > I NEVER care about memory at all... except inasmuch as it effects speed.

In my admittedly idiosyncratic experience, memory is usually by far my most 
constrained resource.  I do care about it, even if python alone isn't using 
much.

That said, CPython has typically thrown memory at all sorts of things, 
including rounding sizes up to 64-bit.  It isn't worth absurd contortions to 
make a difference decision in only one small area.

> On the other hand, if we actually have a huge number of classes, or
> coroutines, or bytecodes, then maybe shaving a couple bits off the memory
> representation of each one could matter.

Or we could only allocate classes within a certain subset of the memory space, 
so that every instance saw savings even when just pointing to their classes.

> But that is 100% orthogonal to the hard limit idea.

That sort of per-instance savings is only available if you know you won't need 
to distinguish between more than N pointer destinations.  

It probably would take at least a working prototype to show that it would help 
in practice though, before there was remotely general support for enforcing 
limits and/or changing the binary layout.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GUWAXP4X5JVCS3NUHF7R6IH46A5FYVEE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: More feedback on PEP 611, please

[Ethan Furman]

I am totally against arbitrary limits in the CPython reference implementation 
and in the language as a whole.

--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2MRLCVCMKNXQJARWDMIVHMYDK7GEBOFA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please be more precise when commenting on PEP 611.

[Guido van Rossum]

On Wed, Dec 11, 2019 at 2:14 AM Mark Shannon  wrote:

>
> If the status quo were the result of considered decision, then it would
> of course need considerable justification.
> If, as is the case here, the status quo is a result of historical
> accident and/or implementation details, then I think a weaker
> justification is OK.
>

Whoa. The lack of limits in the status quo (no limits on various things
except indirectly, through available memory) is most definitely the result
of an intentional decision. "No arbitrary limits" was part of Python's
initial design philosophy. We didn't always succeed (parse tree depth and
call recursion depth come to mind) but that was definitely the philosophy.
It was in contrast to other languages that did have arbitrary limits (e.g.
Pascal's 255-char limit on strings, or C's machine-dependent integer size)
and in several cases the implementation went through great lengths to avoid
limits (e.g. we could have avoided a lot of dynamic memory (re)allocation
if we'd limited line lengths or file sizes).

You have an extreme need to justify why we should change now. "An infinite
number of potential optimizations" does not cut it.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EYV2CCV55UC7Z2EVG4DRTFUVH3WRHWNB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: More feedback on PEP 611, please

[Walter Dörwald]

On 11 Dec 2019, at 12:12, Mark Shannon wrote:


Hi everyone,

Thanks for all your feedback so far.

Previously I asked for "more precise" feedback, which has been 
interpreted as evidence backed feedback. That's not what I meant.

My fault for not being clearer.

Your opinions without any justifications are welcome, but I need 
precision.


The PEP states:

"""
Reference Implementation

None, as yet. This will be implemented in CPython, once the PEP has been 
accepted.

"""

A Python that implements PEP 611 might have advantages and 
disadvantages. The disadvantages we can clearly see: Things that were 
possible before will no longer be possible. But whether the PEP has any 
advantages is unknown. So an implementation might have to come first.



[...]

Almost any performance gain, even 0.1% is worth the, IMO, slight 
inconvenience of 1 million limits.
The reason I believe this is that a 0.1% speedup benefits all Python 
applications and libraries everywhere and forever, whereas the 
inconvenience will be felt by a handful of developers, very rarely.


A 0.1% performance gain means that a script that runs for an hour will 
finish 4 seconds earlier.



[...]


Servus,
   Walter
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3I6MAAGNIWPH7PHKVBKBTJUTQOGND7IX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: More feedback on PEP 611, please

[Steven D'Aprano]

On Wed, Dec 11, 2019 at 11:12:59AM +, Mark Shannon wrote:

> Another thing I would like feedback on this:
> My justification for a single limit of one million across the board is 
> to ease memorization and learning.
> Is that sufficient justification, or would differing limits be better?

Why would people need to memorize these limits?

You are claiming that hardly anyone will run into these limits. If this 
is true, then hardly anyone will need to memorise them, but will look 
them up as needed. Especially if there is an API for reporting the 
limits. The idea is to have the limits high enough that people won't run 
into them (except under odd circumstances) and so why would we memorise 
them?

Besides, it's not true that there will be "a single limit of one million 
across the board". There are many actual or potential limits that your 
PEP doesn't cover and won't change:

- runtime stack depth
- maximum number of nested parentheses ...
- maximum number of characters in a string
- or items in a list, or tuple, or dict
- maximum number of nested for loops
- maximum depth of nested functions
- maximum number of characters in a variable name
- maximum number of characters in a line of code
- maximum number of threads
- maximum number of capturing groups in a regex
- maximum exponent of a Decimal

etc. How many people need to memorise the maximum number of distinct 
finite floats, excluding NANs and INFs? (18437736874454810624: I looked 
it up.)

Some of these may be limited only by memory; some might have limits 
based on addressable memory space and others may have smaller limits. 
But only seven of them (according to your PEP) will have a limit of a 
million. I've already forgotten all seven of them, except the three 
you've decided to remove from the PEP (number of coroutines, number of 
classes, number of lines of code).

Better to pick differing limits without caring if they are the same, 
based on the need for that specific resource.


-- 
Steven
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MGYTZJ3CUVPEO442OMPL45J7OVCUOHB2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: More feedback on PEP 611, please

[Paul Moore]

On Wed, 11 Dec 2019 at 11:38, Mark Shannon  wrote:
> Your opinions without any justifications are welcome, but I need precision.
>
> For example, saying "I don't want any limits ever for anything" is
> precise, but saying "A limit of 1 million is OK provided the performance
> improvements justify it" is not.
> "A limit of 1 million is OK provided a speed up of 50% can be shown" is
> precise, if a bit of a challenge :)

Thanks for clarifying. On that basis, my view is that I'm against any
limits being imposed without demonstrated benefits. I don't care *how
much* benefit, although I would expect the impact of the limit to be
kept in line with the level of benefit. In practical terms, that means
I see this proposal as backwards. I'd prefer it if the proposal were
defined in terms of "here's a benefit we can achieve if we impose
such-and-such a limit".

I see little or no value in having a single, common, or even memorable
number for the limit. Limits should be set on a case by case basis.
Although once we have agreement on the rough level of a limit, setting
a value that is memorable (rather than one that reflects an internal
implementation detail) makes sense to me.

I see basically no justification for having limits at the *language*
level. Any limits should be solely defined as CPython implementation
details. The most the language definition should say is that
implementations are free to impose limits, even if that means that
otherwise portable code might fail on a particular implementation.

Hopefully that's precise in the way you were after. If you need
further clarification, please say.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XVPOFKQKNP3F5QZVPEVNBHAWRIVMK4FD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: More feedback on PEP 611, please

[Rhodri James]

On 11/12/2019 11:12, Mark Shannon wrote:

P.S. On the subject of tradeoffs, here's a bonus question:
What, in your opinion, increase in memory consumption is acceptable for 
a 1% improvement in speed, or vice versa?


Depends on the phase of the moon, the hardware I'm trying to run on, and 
the problem I'm trying to solve.  In some cases, 1% increase in speed 
might be a godsend I'd sacrifice my non-existent first-born for.  In 
others, no increase in memory consumption might be acceptable.


--
Rhodri James *-* Kynesim Ltd
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YIPF7EH4VRICEO5PBOL2WE7U4LR6LARR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: More feedback on PEP 611, please

[David Mertz]

On Wed, Dec 11, 2019, 6:40 AM Mark Shannon  wrote:

> P.S. On the subject of tradeoffs, here's a bonus question:
> What, in your opinion, increase in memory consumption is acceptable for a
> 1% improvement in speed, or vice versa?


I NEVER care about memory at all... except inasmuch as it effects speed.
This question is a rare opportunity to use the phrase "begs the question"
correctly.

I care about avoiding swap memory solely because swap is 100x slower than
RAM. I care about avoiding RAM because L2 cache is 25x faster than RAM. I
care about avoiding L2 because L1 is 4x faster than L2. Memory savings are
only ever about speed.

These limit proposals seem to be irrelevant to main memory. If I can no
longer have a million and one classes in memory, but only a million, the
savings a few bits for a class counter is trivial... Except when it is
POSSIBLY not trivial in L1 where the counter lives but the objects
themselves don't usually.

On the other hand, if we actually have a huge number of classes, or
coroutines, or bytecodes, then maybe shaving a couple bits off the memory
representation of each one could matter. But that is 100% orthogonal to the
hard limit idea.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DFC6J5DZBCWIGXWLODTEDRNL5ZAFHX6D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: More feedback on PEP 611, please

[David Mertz]

On Wed, Dec 11, 2019, 6:40 AM Mark Shannon  wrote:

> Another thing I would like feedback on this:
> My justification for a single limit of one million across the board is to
> ease memorization and learning.
>
Is that sufficient justification, or would differing limits be better?
>

Absolutely not!

The "everything is a million" is a cute little phrase that we could
remember. Even then, I and most people works need to check technical
references to remember what "everything" includes. And I'm sure there will
be edge cases where is not immediately clear if  is one of those
things limited, or which limit it might hot.

More real-world, different limits are relevant to different things.

Moreover, as I've said, even if there are specific gains, I STRONGLY prefer
setting limits to the least restrictive number possible to get that gain.
Maybe it's obvious you'll never need 2**53 of something, but if the
specific gains only benefits from grabbing 11 bits from a 64-bit word, make
that the limit.

If the specific benefit comes from packing 3 counts into the same word,
make the limits 2**21, not 2**20 for a "rounder" number.

Before I would move from -100 on the idea, I'd want to see a specific
benefit, not just a vague speculation that "a benefit might be possible."
Performance and cache locality is the obvious thing. It seems plausible,
but not certain.

But if you can articulate and actual security gain (I'm sceptical of that)
be specific. It's not obvious why am overflow of 32 bits is a more
fundamental attack vector than an overflow of 20 bits. If anything,
intuition would digest the opposite. But if the threat model involves
detecting overflow, doesn't 1 overflow bit and a limit of 2**31 work just
as well. Maybe not, but if not, *explain* why.


> Thanks once again,
> Mark.
>
> P.S. On the subject of tradeoffs, here's a bonus question:
> What, in your opinion, increase in memory consumption is acceptable for
> a 1% improvement in speed, or vice versa?
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/FXPYPRE7RBLJXGPDI3F24L6GJFWOSMBP/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NL5LBQAIG56DVCT7JDYTZRIFNCNRROOD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] More feedback on PEP 611, please

[Mark Shannon]

Hi everyone,

Thanks for all your feedback so far.

Previously I asked for "more precise" feedback, which has been 
interpreted as evidence backed feedback. That's not what I meant.

My fault for not being clearer.

Your opinions without any justifications are welcome, but I need precision.

For example, saying "I don't want any limits ever for anything" is 
precise, but saying "A limit of 1 million is OK provided the performance 
improvements justify it" is not.
"A limit of 1 million is OK provided a speed up of 50% can be shown" is 
precise, if a bit of a challenge :)


To start this off, here are my opinion of the tradeoff:

Almost any performance gain, even 0.1% is worth the, IMO, slight 
inconvenience of 1 million limits.
The reason I believe this is that a 0.1% speedup benefits all Python 
applications and libraries everywhere and forever, whereas the 
inconvenience will be felt by a handful of developers, very rarely.



Another thing I would like feedback on this:
My justification for a single limit of one million across the board is 
to ease memorization and learning.

Is that sufficient justification, or would differing limits be better?

Thanks once again,
Mark.

P.S. On the subject of tradeoffs, here's a bonus question:
What, in your opinion, increase in memory consumption is acceptable for 
a 1% improvement in speed, or vice versa?

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FXPYPRE7RBLJXGPDI3F24L6GJFWOSMBP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Dropping coroutines from PEP 611.

[Kyle Stanley]

With the removal of the coroutine limit, I think that PEP 611 will be a lot
easier to consider. Personally, I've gone from -1 to +0. But, with the
class limit also being rather controversial, I think the PEP would benefit
significantly from backing that up with some form of analysis on memory
usage in existing code.

For example, in addition to the existing section "Total number of classes
in a running interpreter", where it mentions a potential reduction in
overhead on a Python object from 40 bytes to 16 bytes: I think this
argument would be much stronger with an example of production code. You
could perform a rough estimate on how much memory is currently being used
on objects in the code example, and mention how much could be potentially
saved by imposing the 1M class limit.

In general though, I think adding some practical examples to the PEP and
explaining how they could benefit from the limits would make it much more
convincing. At the present moment, it might be a bit too abstract.

On Wed, Dec 11, 2019 at 5:19 AM Mark Shannon  wrote:

> Hi all,
>
> Thanks again for all your feedback on PEP 611.
> So far the biggest pushback on limits has been for the number of
> classes, line count and coroutines.
>
> Of those, I have been convinced that a limit on the number of coroutines
> would the hardest to workaround. In addition the justification for
> limiting coroutines is probably the weakest.
>
> Consequently, I'm dropping the limit on the number of coroutines from
> the PEP.
>
> Cheers,
> Mark.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/6ESJKTW4CP42KWJVSEEYJABGU73T4UPA/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AK7XRUXJMFKGR6MKYHQM4HQGI57BZFMA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Feedback on PEP 611 (so far) from the Steering Council

[Mark Shannon]

On 11/12/2019 12:04 am, Barry Warsaw wrote:

The Python Steering Council discussed PEP 611 at today’s meeting.  Here is our 
feedback so far:

* The Steering Council reserves the right to be the BDFL-Delegate for this PEP

* The PEP should clearly delineate two aspects:

   - A language generic part (i.e. applies to all implementations of Python) 
which exposes implementation limits in a way that Python code can read at 
runtime.  Think sys.getrecursionlimit(), sys.maxsize, and sys.implementation

   - An implementation-specific part where the PEP would propose specific 
limits for the CPython implementation.  Other implementations of the Python 
language would be free to adjust such limits up or down as they deem 
appropriate.

* We encourage the PEP authors and proponents to gather actual performance data 
that can be used to help us evaluate whether this PEP is a good idea or not.


I think that judging this on numbers from a handful of optimizations 
would be a mistake.


There are an infinite number of potential optimizations, so it is 
impossible to put numbers on as yet undiscovered optimization. 
Conversely it is impossible to perfectly predict what limits might 
restrict possible future applications.


Having said that, I agree that it would be useful to come up with 
example optimizations and applications that benefit or suffer from the 
limits.




By all means, continue to discuss and refine the PEP.  When the PEP author is 
ready for a pronouncement, you can email the Steering Council.


I welcome more discussion.
Ultimately this is a subjective judgement, so the more opinions we have, 
the better.




Cheers,
-Barry


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KY46EXGLKNTFMQZXKHMMYWD2GIM5PDL5/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/A6Y3PGYVNLZRYZOPNQN7L3XQUTN3IAQB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Dropping coroutines from PEP 611.

[Mark Shannon]

Hi all,

Thanks again for all your feedback on PEP 611.
So far the biggest pushback on limits has been for the number of 
classes, line count and coroutines.


Of those, I have been convinced that a limit on the number of coroutines
would the hardest to workaround. In addition the justification for 
limiting coroutines is probably the weakest.


Consequently, I'm dropping the limit on the number of coroutines from 
the PEP.


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6ESJKTW4CP42KWJVSEEYJABGU73T4UPA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please be more precise when commenting on PEP 611.

[Mark Shannon]

On 10/12/2019 7:24 am, Kyle Stanley wrote:

Chris Angelico wrote:
 > We have people who believe that a bit
 > mask will slow things down, others who claim that improved cache
 > locality will speed things up, and Mark asks us to please justify our
 > objections with numbers. But surely it's up to Mark to show numbers
 > first?

+1. While it would be helpful for criticisms of the PEP to be more 
specific and provide more of a cost-benefit analysis, the burden of 
proof for demonstrating the benefits provided are ultimately up to the 
author(s) of the PEP. We require far more justification for making a 
change to impose the limits in the first place than we do for 
maintaining the status quo.


I agree that we require justification for changing the status quo.

If the status quo were the result of considered decision, then it would 
of course need considerable justification.
If, as is the case here, the status quo is a result of historical 
accident and/or implementation details, then I think a weaker 
justification is OK.




Personally, I don't think it would be reasonable to impose any of these 
limits without some form of concrete evidence that doing so will provide 
an improvement; in any combination of efficiency, performance, and/or 
security, as the PEP suggests would happen. But, I don't think exact 
numbers are needed. (I.E. adding these limits will improve performance 
across all Python programs by an average of 22.359%!).


On Mon, Dec 9, 2019 at 7:15 PM Chris Angelico > wrote:


On Tue, Dec 10, 2019 at 10:51 AM Steven D'Aprano
mailto:st...@pearwood.info>> wrote:
 >
 > On Mon, Dec 09, 2019 at 02:12:37PM -0800, Nathaniel Smith wrote:
 > > > > On 09/12/2019 2:15 pm, Chris Angelico wrote:
 > > > You: "We should limit things. Stuff will be faster."
 > > > Others: "Really? Because bit masking is work. It'll be slower."
 >
 > I'm not an expert, but the impression I've got from various
discussions
 > on performance over the last few years is that the single biggest
 > bottleneck for CPU performance is memory locality. Cache misses
are so
 > expensive, and CPU instructions so fast, that memory locality is king
 > and the cost of bit masking is insignificant. In other words,
worrying
 > about the cost of bit masking in C code is so very 1990s.
 >
 > I could be wrong of course: I'm not an expert. And I don't think we
 > should take it for granted that this is the case, unless some
experts on
 > modern CPUs speak up and say that Mark is so obviously correct that a
 > demonstration is unnecessary.

And the speculation continues.

 > > > You: "Maybe we limit it somewhere else, whatever. It'll be
faster."
 >
 > That's a totally unfair and inaccurate representation of Mark's
 > position. The PEP doesn't say "let's put in arbitrary limits in
random
 > places for the lols", he proposed seven concrete limits and gave
reasons
 > for why he expects that they will improve memory efficiency, safety,
 > performance or all three.

*He expects that*. That's what I'm talking about. We have lots of
speculation and no evidence either way.

 > Having said that though, I think you are right that the PEP could do
 > with a bit more detail on the current status quo and existing limits,
 > and how the proposed changes will improve safety and memory use.

Exactly. Yes, I know that I massively oversimplified things in that
post. But you nonetheless acknowledge here that we are *still* quite
lacking in any actual evidence. We have people who believe that a bit
mask will slow things down, others who claim that improved cache
locality will speed things up, and Mark asks us to please justify our
objections with numbers. But surely it's up to Mark to show numbers
first?

ChrisA
___
Python-Dev mailing list -- python-dev@python.org

To unsubscribe send an email to python-dev-le...@python.org

https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/YN2BDJFPGHTEZJRDN7LXNDAGXWXB6XRA/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EDZOEKHLF52MG3JJ6HIVHLCGXQJRIM34/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org

[Python-Dev] Re: Please be more precise when commenting on PEP 611.

[Mark Shannon]

On 09/12/2019 10:29 pm, Gregory P. Smith wrote:
My overall problem with the PEP and a reason I'd reject it by default it 
that it is asking to pre-emptively impose limits on things, some of 
which we believe would cause problems to existing long running 
applications (limiting the total number of types for example), without 
having actually demonstrated practical benefits from such a change.  
Having an implementation that takes limits and runs with them to 
practical effect would provide motivation to consider adopting them.


Otherwise it doesn't feel like it solves a real problem, and could cause 
some existing users pain.  So what's our real motivation?


picking on some nits within the PEP 611 text:

 > Motivation: "poorly generated code could cause values to exceed 2^32"

Malicious or poorly generated code can always do anything so this isn't 
a strong motivation.  If you want a nice failure when we come near 
actual existing physical limits, that could be done today as a PR 
without a PEP.


If correctness is the motivation, we could be correct without changing 
the existing unchecked limits.


 > : "there are two ways to use a 32 bit refcount on a 64 bit machine. 
One is to limit each sub-interpreter to 32Gb of memory. The other is to 
use a saturating reference count, which would be a little bit slower, 
but allow unlimited memory allocation."


Please do not arbitrarily cap sub-interpreter memory usage to a small 
value.  32GiB is very small today.


I wasn't planning on doing so, merely pointing out that there is a very 
simple way to make a 32 bit reference count correct.
Limiting the number of objects or using a saturating count would be more 
sensible.




Also, at least one existing eternal refcount implementation I've had 
experience with demonstrated a notable hit to interpreter cpu 
performance as it required an additional test+branch within the 
extremely widely used Py_INCREF and Py_DECREF macro code.


Did they also reduce the size of object header, improving cache locality?
Do you have a reference?




-gps

On Mon, Dec 9, 2019 at 6:10 AM Mark Shannon > wrote:


Hi everyone,

Thanks again for all your comments on PEP 611.

I would like to ask a favour; please be more specific in your comments.

Ideally state which part of the PEP you are disagreeing with and why
you
disagree with the relevant part of the rationale/motivation.

Also, when asking for limits to be raised or removed entirely, could
you
state what you perceive to be the costs and benefits of larger limits.
What do you believe is an acceptable cost in memory or runtime for
larger limits?

For example, you might say that the limit of one million lines of code
per module is too small, and that it is worth a small, say 1%,
impact on
speed to allow a larger of limit of 100 million.

If you believe a limit would have no cost, then please give a
explanation of why that is so.

Merely saying that you would like a larger limit is pointless.
If there were no cost to arbitrarily large limits, then I wouldn't have
proposed the PEP in the first place.

Bear in mind that the costs of higher limits are paid by everyone, but
the benefits are gained by few.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org

To unsubscribe send an email to python-dev-le...@python.org

https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/ZXDAJKRVSF6UUE5UEPE5PMXYXOLJ5A4V/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3AUUCWOSLFXUR3DAU7LJ5RHODRIETVR7/
Code of Conduct: http://python.org/psf/codeofconduct/