Re: [Python-Dev] PEP 550 v4

[David Mertz]

This is now looking really good and I can understands it.

One question though. Sometimes creation of a context variable is done with
a name argument, other times not. E.g.

  var1 = new_context_var('var1')
  var = new_context_var()

The signature is given as:

  sys.new_context_var(name: str)

But it seems like it should be:

  sys.new_context_var(name: Optional[str]=None)

On Aug 25, 2017 3:35 PM, "Yury Selivanov"  wrote:

Hi,

This is the 4th iteration of the PEP that Elvis and I have
rewritten from scratch.

The specification section has been separated from the implementation
section, which makes them easier to follow.

During the rewrite, we realized that generators and coroutines should
work with the EC in exactly the same way (coroutines used to be
created with no LC in prior versions of the PEP).

We also renamed Context Keys to Context Variables which seems
to be a more appropriate name.

Hopefully this update will resolve the remaining questions
about the specification and the proposed implementation, and
will allow us to focus on refining the API.

Yury



PEP: 550
Title: Execution Context
Version: $Revision$
Last-Modified: $Date$
Author: Yury Selivanov ,
Elvis Pranskevichus 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 11-Aug-2017
Python-Version: 3.7
Post-History: 11-Aug-2017, 15-Aug-2017, 18-Aug-2017, 25-Aug-2017


Abstract


This PEP adds a new generic mechanism of ensuring consistent access
to non-local state in the context of out-of-order execution, such
as in Python generators and coroutines.

Thread-local storage, such as ``threading.local()``, is inadequate for
programs that execute concurrently in the same OS thread.  This PEP
proposes a solution to this problem.


Rationale
=

Prior to the advent of asynchronous programming in Python, programs
used OS threads to achieve concurrency.  The need for thread-specific
state was solved by ``threading.local()`` and its C-API equivalent,
``PyThreadState_GetDict()``.

A few examples of where Thread-local storage (TLS) is commonly
relied upon:

* Context managers like decimal contexts, ``numpy.errstate``,
  and ``warnings.catch_warnings``.

* Request-related data, such as security tokens and request
  data in web applications, language context for ``gettext`` etc.

* Profiling, tracing, and logging in large code bases.

Unfortunately, TLS does not work well for programs which execute
concurrently in a single thread.  A Python generator is the simplest
example of a concurrent program.  Consider the following::

def fractions(precision, x, y):
with decimal.localcontext() as ctx:
ctx.prec = precision
yield Decimal(x) / Decimal(y)
yield Decimal(x) / Decimal(y**2)

g1 = fractions(precision=2, x=1, y=3)
g2 = fractions(precision=6, x=2, y=3)

items = list(zip(g1, g2))

The expected value of ``items`` is::

[(Decimal('0.33'), Decimal('0.67')),
 (Decimal('0.11'), Decimal('0.22'))]

Rather surprisingly, the actual result is::

[(Decimal('0.33'), Decimal('0.67')),
 (Decimal('0.11'), Decimal('0.22'))]

This is because Decimal context is stored as a thread-local, so
concurrent iteration of the ``fractions()`` generator would corrupt
the state.  A similar problem exists with coroutines.

Applications also often need to associate certain data with a given
thread of execution.  For example, a web application server commonly
needs access to the current HTTP request object.

The inadequacy of TLS in asynchronous code has lead to the
proliferation of ad-hoc solutions, which are limited in scope and
do not support all required use cases.

The current status quo is that any library (including the standard
library), which relies on TLS, is likely to be broken when used in
asynchronous code or with generators (see [3]_ as an example issue.)

Some languages, that support coroutines or generators, recommend
passing the context manually as an argument to every function, see [1]_
for an example.  This approach, however, has limited use for Python,
where there is a large ecosystem that was built to work with a TLS-like
context.  Furthermore, libraries like ``decimal`` or ``numpy`` rely
on context implicitly in overloaded operator implementations.

The .NET runtime, which has support for async/await, has a generic
solution for this problem, called ``ExecutionContext`` (see [2]_).


Goals
=

The goal of this PEP is to provide a more reliable
``threading.local()`` alternative, which:

* provides the mechanism and the API to fix non-local state issues
  with coroutines and generators;

* has no or negligible performance impact on the existing code or
  the code that will be using the new mechanism, including
  libraries like ``decimal`` and ``numpy``.


High-Level Specification


The full specification of this PEP is broken down into three parts:

* 

Re: [Python-Dev] PEP 550 and other python implementations

[Jim Baker]

re other implementations: the model presented in
https://www.python.org/dev/peps/pep-0550/#implementation seems perfectly
compatible with Jython. It's just one more thing we would add to
PyThreadState (which is what it's also called in the Jython runtime).

On Fri, Aug 25, 2017 at 7:45 PM, Jim J. Jewett  wrote:

> Should PEP 550 discuss other implementations?  E.g., the object space used
> in pypy?
>
> -jJ
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> jbaker%40zyasoft.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 leak-in vs leak-out, why not just a ChainMap

[Jim J. Jewett]

On Aug 24, 2017 11:02 AM, "Yury Selivanov"  wrote:

On Thu, Aug 24, 2017 at 10:05 AM, Jim J. Jewett 
wrote:
> On Thu, Aug 24, 2017 at 1:12 AM, Yury Selivanov > On Thu, Aug 24, 2017
> at 12:32 AM, Jim J. Jewett  wrote:


If you look at this small example:

foo = new_context_key()

async def nested():
await asyncio.sleep(1)
print(foo.get())

async def outer():
foo.set(1)
await nested()
foo.set(1000)

l = asyncio.get_event_loop()
l.create_task(outer())
l.run_forever()

If will print "1", as "nested()" coroutine will see the "foo" key when
it's awaited.

Now let's say we want to refactor this snipped and run the "nested()"
coroutine with a timeout:

foo = new_context_key()

async def nested():
await asyncio.sleep(1)
print(foo.get())

async def outer():
foo.set(1)
await asyncio.wait_for(nested(), 10)  # !!!
foo.set(1000)

l = asyncio.get_event_loop()
l.create_task(outer())
l.run_forever()

So we wrap our `nested()` in a `wait_for()`, which creates a new
asynchronous tasks to run `nested()`.  That task will now execute on
its own, separately from the task that runs `outer()`.  So we need to
somehow capture the full EC at the moment `wait_for()` was called, and
use that EC to run `nested()` within it.  If we don't do this, the
refactored code would print "1000", instead of "1".


I would expect 1000 to be the right answer!  By the time it runs, 1000 (or
mask_errors=false, to use a less toy example) is what its own controlling
scope requested.

If you are sure that you want the value frozen earlier, please make this
desire very explicit  ... this example is the first I noticed it.  And
please explain what this means for things like signal or warning masking.


ContextKey is declared once for the code that uses it. Nobody else
will use that key. Keys have names only for introspection purposes,
the implementation doesn't use it, iow:

var = new_context_key('aa')
var.set(1)
# EC = [..., {var: 1}]

# Note the that EC has a "var" object itself as the key in the
mapping, not "a".


This I had also not realized.  So effectively, they keys are based on
object identity, with some safeguards to ensure that even starting with the
same (interned) name will *not* produce the same object unless you passed
it around explicitly, or are in the same same code unit (file, typically).

This strikes me as reasonable, but still surprising.  (I think of variables
as typically named, rather than identified by address.)  So please make
this more explicit as well.

-jJ
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v4

[Ethan Furman]

All in all, I like it.  Nice job.

On 08/25/2017 03:32 PM, Yury Selivanov wrote:


A *context variable* is an object representing a value in the
execution context.  A new context variable is created by calling
the ``new_context_var()`` function.  A context variable object has
two methods:

* ``lookup()``: returns the value of the variable in the current
   execution context;

* ``set()``: sets the value of the variable in the current
   execution context.


Why "lookup" and not "get" ?  Many APIs use "get" and it's functionality is 
well understood.



Conceptually, an *execution context* (EC) is a stack of logical
contexts.  There is one EC per Python thread.

A *logical context* (LC) is a mapping of context variables to their
values in that particular LC.


Still don't like the name of "logical context", but we can bike-shed on that 
later.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 550 and other python implementations

[Jim J. Jewett]

Should PEP 550 discuss other implementations?  E.g., the object space used
in pypy?

-jJ
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] bpo-30987 review

[Pier-Yves Lessard]

Hello there!
I've created a pull request 
almost a month ago that seems to have very little attention and I was
wondering what could be done (if any).

The suggested approach on the "Lifecycle of a pull request" page is said to
wait for a month then remind the nosy list. Unfortunately, I am the only
one on the nosy list, so I skipped to this email directly. I saw some pull
request more recent than mine getting a look at, so I thought maybe
something was wrong with mine.

In any case, let me know if I can do anything to help on that. The feature
I added will unlock a quite nice features for the automotive industry.

Thank you,
Keep up the awesome work!

-- 
*Pier-Yves Lessard*
*438-862-5914*
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 550 v4

[Yury Selivanov]

Hi,

This is the 4th iteration of the PEP that Elvis and I have
rewritten from scratch.

The specification section has been separated from the implementation
section, which makes them easier to follow.

During the rewrite, we realized that generators and coroutines should
work with the EC in exactly the same way (coroutines used to be
created with no LC in prior versions of the PEP).

We also renamed Context Keys to Context Variables which seems
to be a more appropriate name.

Hopefully this update will resolve the remaining questions
about the specification and the proposed implementation, and
will allow us to focus on refining the API.

Yury



PEP: 550
Title: Execution Context
Version: $Revision$
Last-Modified: $Date$
Author: Yury Selivanov ,
Elvis Pranskevichus 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 11-Aug-2017
Python-Version: 3.7
Post-History: 11-Aug-2017, 15-Aug-2017, 18-Aug-2017, 25-Aug-2017


Abstract


This PEP adds a new generic mechanism of ensuring consistent access
to non-local state in the context of out-of-order execution, such
as in Python generators and coroutines.

Thread-local storage, such as ``threading.local()``, is inadequate for
programs that execute concurrently in the same OS thread.  This PEP
proposes a solution to this problem.


Rationale
=

Prior to the advent of asynchronous programming in Python, programs
used OS threads to achieve concurrency.  The need for thread-specific
state was solved by ``threading.local()`` and its C-API equivalent,
``PyThreadState_GetDict()``.

A few examples of where Thread-local storage (TLS) is commonly
relied upon:

* Context managers like decimal contexts, ``numpy.errstate``,
  and ``warnings.catch_warnings``.

* Request-related data, such as security tokens and request
  data in web applications, language context for ``gettext`` etc.

* Profiling, tracing, and logging in large code bases.

Unfortunately, TLS does not work well for programs which execute
concurrently in a single thread.  A Python generator is the simplest
example of a concurrent program.  Consider the following::

def fractions(precision, x, y):
with decimal.localcontext() as ctx:
ctx.prec = precision
yield Decimal(x) / Decimal(y)
yield Decimal(x) / Decimal(y**2)

g1 = fractions(precision=2, x=1, y=3)
g2 = fractions(precision=6, x=2, y=3)

items = list(zip(g1, g2))

The expected value of ``items`` is::

[(Decimal('0.33'), Decimal('0.67')),
 (Decimal('0.11'), Decimal('0.22'))]

Rather surprisingly, the actual result is::

[(Decimal('0.33'), Decimal('0.67')),
 (Decimal('0.11'), Decimal('0.22'))]

This is because Decimal context is stored as a thread-local, so
concurrent iteration of the ``fractions()`` generator would corrupt
the state.  A similar problem exists with coroutines.

Applications also often need to associate certain data with a given
thread of execution.  For example, a web application server commonly
needs access to the current HTTP request object.

The inadequacy of TLS in asynchronous code has lead to the
proliferation of ad-hoc solutions, which are limited in scope and
do not support all required use cases.

The current status quo is that any library (including the standard
library), which relies on TLS, is likely to be broken when used in
asynchronous code or with generators (see [3]_ as an example issue.)

Some languages, that support coroutines or generators, recommend
passing the context manually as an argument to every function, see [1]_
for an example.  This approach, however, has limited use for Python,
where there is a large ecosystem that was built to work with a TLS-like
context.  Furthermore, libraries like ``decimal`` or ``numpy`` rely
on context implicitly in overloaded operator implementations.

The .NET runtime, which has support for async/await, has a generic
solution for this problem, called ``ExecutionContext`` (see [2]_).


Goals
=

The goal of this PEP is to provide a more reliable
``threading.local()`` alternative, which:

* provides the mechanism and the API to fix non-local state issues
  with coroutines and generators;

* has no or negligible performance impact on the existing code or
  the code that will be using the new mechanism, including
  libraries like ``decimal`` and ``numpy``.


High-Level Specification


The full specification of this PEP is broken down into three parts:

* High-Level Specification (this section): the description of the
  overall solution.  We show how it applies to generators and
  coroutines in user code, without delving into implementation details.

* Detailed Specification: the complete description of new concepts,
  APIs, and related changes to the standard library.

* Implementation Details: the description and analysis of data
  structures and algorithms used to implement this PEP, as well as the
  

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2017-08-18 - 2017-08-25)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open6149 (+16)
  closed 36880 (+28)
  total  43029 (+44)

Open issues with patches: 2342 


Issues opened (28)
==

#30871: Add test.pythoninfo
http://bugs.python.org/issue30871  reopened by haypo

#31233: socketserver.ThreadingMixIn leaks running threads after server
http://bugs.python.org/issue31233  opened by haypo

#31234: Make support.threading_cleanup() stricter
http://bugs.python.org/issue31234  opened by haypo

#31237: test_gdb disables 25% of tests in optimized builds
http://bugs.python.org/issue31237  opened by lukasz.langa

#31241: ast col_offset wrong for list comprehensions, generators and t
http://bugs.python.org/issue31241  opened by samuelcolvin

#31242: Add SSLContext.set_verify_callback()
http://bugs.python.org/issue31242  opened by rfinnie

#31243: checks whether PyArg_ParseTuple returned a negative int
http://bugs.python.org/issue31243  opened by Oren Milman

#31244: IDLE: work around shortcuts bug in Windows' IMEs and tk
http://bugs.python.org/issue31244  opened by Constantine Ketskalo

#31245: Asyncio UNIX socket and SOCK_DGRAM
http://bugs.python.org/issue31245  opened by qdawans

#31246: [2.7] test_signal.test_setitimer_tiny() fails randomly on x86-
http://bugs.python.org/issue31246  opened by haypo

#31248: method wrapper type has invalid __name__/__qualname__ 'method-
http://bugs.python.org/issue31248  opened by bup

#31249: test_concurrent_futures leaks dangling threads
http://bugs.python.org/issue31249  opened by haypo

#31250: test_asyncio leaks dangling threads
http://bugs.python.org/issue31250  opened by haypo

#31252: Operator.itemgetter documentation should include dictionary ke
http://bugs.python.org/issue31252  opened by ashishnitinpatil

#31254: WeakKeyDictionary/Mapping doesn't call __missing__
http://bugs.python.org/issue31254  opened by Antony.Lee

#31256: xml.etree.ElementTree: add support for doctype in tostring met
http://bugs.python.org/issue31256  opened by bastik

#31258: [2.7] Enhance support reap_children() and threading_cleanup()
http://bugs.python.org/issue31258  opened by haypo

#31259: [3.6] Enhance support reap_children() and threading_cleanup()
http://bugs.python.org/issue31259  opened by haypo

#31260: [2.7] Enhance PC/VS9.0/ project to produce python.bat, as PCbu
http://bugs.python.org/issue31260  opened by haypo

#31265: Remove doubly-linked list from C OrderedDict
http://bugs.python.org/issue31265  opened by inada.naoki

#31267: threading.Timer object is affected by changes to system time
http://bugs.python.org/issue31267  opened by winfreak

#31270: Simplify documentation of itertools.zip_longest
http://bugs.python.org/issue31270  opened by rami

#31271: an assertion failure in io.TextIOWrapper.write
http://bugs.python.org/issue31271  opened by Oren Milman

#31272: typing module conflicts with  __slots__-classes
http://bugs.python.org/issue31272  opened by grayfall

#31273: Unicode support in TestCase.skip
http://bugs.python.org/issue31273  opened by Nathan Buckner

#31274: Support building against homebrew on macOS
http://bugs.python.org/issue31274  opened by barry

#31275: Check fall-through in _codecs_iso2022.c
http://bugs.python.org/issue31275  opened by skrah

#31276: PyObject_CallFinalizerFromDealloc is undocumented
http://bugs.python.org/issue31276  opened by pv



Most recent 15 issues with no replies (15)
==

#31276: PyObject_CallFinalizerFromDealloc is undocumented
http://bugs.python.org/issue31276

#31273: Unicode support in TestCase.skip
http://bugs.python.org/issue31273

#31272: typing module conflicts with  __slots__-classes
http://bugs.python.org/issue31272

#31267: threading.Timer object is affected by changes to system time
http://bugs.python.org/issue31267

#31260: [2.7] Enhance PC/VS9.0/ project to produce python.bat, as PCbu
http://bugs.python.org/issue31260

#31256: xml.etree.ElementTree: add support for doctype in tostring met
http://bugs.python.org/issue31256

#31250: test_asyncio leaks dangling threads
http://bugs.python.org/issue31250

#31248: method wrapper type has invalid __name__/__qualname__ 'method-
http://bugs.python.org/issue31248

#31245: Asyncio UNIX socket and SOCK_DGRAM
http://bugs.python.org/issue31245

#31242: Add SSLContext.set_verify_callback()
http://bugs.python.org/issue31242

#31241: ast col_offset wrong for list comprehensions, generators and t
http://bugs.python.org/issue31241

#31224: Missing definition of frozen module
http://bugs.python.org/issue31224

#31207: IDLE, configdialog: Factor out ExtPage class from ConfigDialog
http://bugs.python.org/issue31207

#31202: Windows pathlib.Path.glob(pattern) fixed part of the pattern c
http://bugs.python.org/issue31202

#31196: Blank line inconsistency between InteractiveConsole and 

Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)

[Guido van Rossum]

I think the issue with sys.std* is a distraction for this discussion. The
issue also seems overstated, and I wouldn't want to change it. The ability
to set these is mostly used in small programs that are also
single-threaded. Libraries should never mess with them -- it's easy to
explicitly pass an output file around, and for errors you should use
logging or in a pinch you can write to sys.stderr, but you shouldn't set it.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)

[Eric Snow]

On Fri, Aug 25, 2017 at 9:10 AM, Barry Warsaw  wrote:
> It’s ideas like this that do make me think of scopes when talking about 
> global state and execution contexts.  I understand that the current PEP 550 
> invokes an explicit separate namespace,

Right.  The observation that PEP 550 proposes a separate stack of
scopes from the lexical scope is an important one.

> but thinking bigger, if the usual patterns of just writing to 
> sys.std{out,err}  still worked and in the presence of single “threaded” 
> execution it just did the normal thing, but in the presence of threads, 
> async, etc. it *also* did the right thing, then code wouldn’t need to change 
> just because you started to adopt async.  That implies some scoping rules to 
> make “sys.stdout” refer to the local execution’s sys.stdout if it were set, 
> and the global sys.stdout if it were not.

Yeah, at the bottom of the PEP 550 stack there'd need to be a proxy to
the relevant global state.  While working on the successor to PEP 406
(import state), I realized I'd need something like this.

>
> This would of course be a much deeper change to Python, with lots of tricky 
> semantics and corner cases to get right.  But it might be worth it to provide 
> an execution model and an  API that would be harder to get wrong because 
> Python just Does the Right Thing.  It’s difficult because you also have to be 
> able to reason about what’s going on, and it’ll be imperative to be able to 
> debug and examine the state of your execution when things go unexpected.  
> That’s always harder when mixing dynamic scope with lexical scope, which I 
> think is what PEP 550 is ultimately getting at.

+1  Thankfully, PEP 550 is targeted more at a subset of library
authors than at the general population.

-eric
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)

[Yury Selivanov]

On Fri, Aug 25, 2017 at 11:10 AM, Barry Warsaw  wrote:
[..]
> It’s ideas like this that do make me think of scopes when talking about 
> global state and execution contexts.  I understand that the current PEP 550 
> invokes an explicit separate namespace, but thinking bigger, if the usual 
> patterns of just writing to sys.std{out,err}  still worked and in the 
> presence of single “threaded” execution it just did the normal thing, but in 
> the presence of threads, async, etc. it *also* did the right thing, then code 
> wouldn’t need to change just because you started to adopt async.  That 
> implies some scoping rules to make “sys.stdout” refer to the local 
> execution’s sys.stdout if it were set, and the global sys.stdout if it were 
> not.

The problem here is exactly the "usual patterns of just writing to
sys.std{out,err}". The usual pattern assumes that it's a global
variable, and there are no ways of getting around this. None. There
are many applications out there that are already written with the
assumption that setting sys.stdout changes it for all threads, which
means that we cannot change this already established semantics.

The sys.std* API just needs to be slowly deprecated and replaced with
a new API that uses context managers and does things differently under
the hood, *if* and only if, we all agree that we even need to solve
this problem.

This is a completely separate problem from the one that PEP 550
solves, which is providing a better TLS that is aware of generators
and async code.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)

[Eric Snow]

On Fri, Aug 25, 2017 at 8:18 AM, Yury Selivanov  wrote:
> Another idea:
>
> 1. We alter PyModule to make it possible to add properties (descriptor
> protocol, or we implement custom __getattr__). I think we can make it
> so that only sys module would be able to actually use it, so it's not
> going to be a new feature -- just a hack for CPython.

FWIW, I've been toying with a similar problem and solution for a
while.  I'd like to clean up the sys module, including grouping some
of the attributes (e.g. the import state), turn the get/set pairs into
properties, and deprecate direct usage of some of the attributes.

Though supporting descriptors on module objects would work [1] and be
useful  (particularly deprecating module attrs), it's sufficiently
worthy of a PEP that I haven't taken the time.  Instead, the approach
I settled on was to rename sys to _sys and add a sys written in Python
that proxies _sys.  Here's a rough first pass:

https://github.com/ericsnowcurrently/cpython/tree/sys-module

It's effectively the same thing as ModuleType supporting descriptors,
but specific only to sys.  One problem with both approaches is that
we'd be changing the type of the sys module.  There's a relatively
common idiom in the stdlib (and elsewhere) of using "type(sys)" to get
ModuleType.  Changing the type of the sys module breaks that.

-eric


[1] This is doable with a custom __getattribute__ on ModuleType,
though it will impact attribute lookup on all modules.  I suppose
there could be a subclass that does the right thing...  Anyway, this
is more python-ideas territory.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)

[Barry Warsaw]

On Aug 25, 2017, at 10:18, Yury Selivanov  wrote:
> 
> I has a similar idea when I discovered that PEP 550 can't be used
> directly to fix sys.std* streams redirection.  Another idea:
> 
> 1. We alter PyModule to make it possible to add properties (descriptor
> protocol, or we implement custom __getattr__). I think we can make it
> so that only sys module would be able to actually use it, so it's not
> going to be a new feature -- just a hack for CPython.
> 
> 2. We make sys.std* attributes properties, with getters and setters.
> 
> 3. sys.std* setters will: issue a DeprecationWarning; set whatever the
> user wants to set in a global variable + set a flag (let's call it
> "sys.__stdout_global_modified") that sys.std* were modified.
> 
> 4. sys.std* getters will use PEP 550 to lookup when
> __stdout_global_modified is false.  If it's true -- we fallback to
> globals.
> 
> 5. We deprecate the current API and add new APIs for the redirection
> system that uses PEP 550 explicitly.
> 
> 6. In Python 4 we remove the old sys.std* API.
> 
> Thit is still *very* fragile: any code that writes to sys.stdout
> breaks all assumptions.  But it offers a way to raise a warning when
> old-API is being used - something that we'll probably need if we add
> new APIs to fix this problem.

It’s ideas like this that do make me think of scopes when talking about global 
state and execution contexts.  I understand that the current PEP 550 invokes an 
explicit separate namespace, but thinking bigger, if the usual patterns of just 
writing to sys.std{out,err}  still worked and in the presence of single 
“threaded” execution it just did the normal thing, but in the presence of 
threads, async, etc. it *also* did the right thing, then code wouldn’t need to 
change just because you started to adopt async.  That implies some scoping 
rules to make “sys.stdout” refer to the local execution’s sys.stdout if it were 
set, and the global sys.stdout if it were not.

This would of course be a much deeper change to Python, with lots of tricky 
semantics and corner cases to get right.  But it might be worth it to provide 
an execution model and an  API that would be harder to get wrong because Python 
just Does the Right Thing.  It’s difficult because you also have to be able to 
reason about what’s going on, and it’ll be imperative to be able to debug and 
examine the state of your execution when things go unexpected.  That’s always 
harder when mixing dynamic scope with lexical scope, which I think is what PEP 
550 is ultimately getting at.

Cheers,
-Barry


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)

[Yury Selivanov]

Right, Nick, I missed the part that you want to have a file-like
wrapper stored in sys.std* streams that would redirect lookups/calls
to the relevant real file-object in the current context (correct?)

I has a similar idea when I discovered that PEP 550 can't be used
directly to fix sys.std* streams redirection.  Another idea:

1. We alter PyModule to make it possible to add properties (descriptor
protocol, or we implement custom __getattr__). I think we can make it
so that only sys module would be able to actually use it, so it's not
going to be a new feature -- just a hack for CPython.

2. We make sys.std* attributes properties, with getters and setters.

3. sys.std* setters will: issue a DeprecationWarning; set whatever the
user wants to set in a global variable + set a flag (let's call it
"sys.__stdout_global_modified") that sys.std* were modified.

4. sys.std* getters will use PEP 550 to lookup when
__stdout_global_modified is false.  If it's true -- we fallback to
globals.

5. We deprecate the current API and add new APIs for the redirection
system that uses PEP 550 explicitly.

6. In Python 4 we remove the old sys.std* API.

Thit is still *very* fragile: any code that writes to sys.stdout
breaks all assumptions.  But it offers a way to raise a warning when
old-API is being used - something that we'll probably need if we add
new APIs to fix this problem.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)

[Nick Coghlan]

On 25 August 2017 at 23:36, Yury Selivanov  wrote:
> On Fri, Aug 25, 2017 at 9:23 AM, Nick Coghlan  wrote:
> [..]
>> And while PEP 550 doesn't handle the stream redirection case natively
>> (since it doesn't allow for suspend/resume callbacks the way PEP 525
>> does), it at least allows for the development of a context-aware
>> output stream wrapper API where:
>
> PEP 525 can't handle streams redirection -- it can do it only for
> single-threaded programs.

Good point, whereas the hypothetical context-aware wrapper I proposed
would be both thread-safe (since the process global would be changed
once before the program went multi-threaded and then left alone) *and*
potentially lower overhead (one context local lookup per stream
attribute access, rather than a global state change every time a frame
was suspended or resumed with a stream redirected)

> sys.stdout/stderr/stdin are global variables, that's how the API is
> specified. API users assume that the change is process-wide.

Yeah, I wasn't suggesting any implicit changes to the way those work.

However, it does occur to me that if we did add a new "contextlocals" API, then:

1. It could offer context-aware wrappers for stdin/stdout/stderr
2. If could offer context-aware alternatives to the stream redirection
context managers in contextlib

That approach would work even better than replacing sys.stdin/out/err
themselves with wrappers since the wrappers wouldn't be vulnerable to
being overwritten by other code that mutated the sys module.

Anyway, that's not a serious proposal right now, but I do think it's
decent validation of the power and flexibility of the proposed
implicit state management model.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)

[Yury Selivanov]

On Fri, Aug 25, 2017 at 9:23 AM, Nick Coghlan  wrote:
[..]
> And while PEP 550 doesn't handle the stream redirection case natively
> (since it doesn't allow for suspend/resume callbacks the way PEP 525
> does), it at least allows for the development of a context-aware
> output stream wrapper API where:

PEP 525 can't handle streams redirection -- it can do it only for
single-threaded programs.

sys.stdout/stderr/stdin are global variables, that's how the API is
specified. API users assume that the change is process-wide.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)

[Nick Coghlan]

On 25 August 2017 at 20:28, Antoine Pitrou  wrote:
> On Fri, 25 Aug 2017 15:36:55 +1000
> Nick Coghlan  wrote:
>> On 24 August 2017 at 23:52, Barry Warsaw  wrote:
>> > Guido van Rossum wrote:
>> >> On Tue, Aug 22, 2017 at 7:12 PM, Nathaniel Smith  wrote:
>> >>
>> >> I worry that that's going to lead more people astray thinking this has
>> >> something to do with contextlib, which it really doesn't (it's much more
>> >> closely related to threading.local()).
>> >
>> > This is my problem with using "Context" for this PEP.  Although I can't
>> > keep up with all names being thrown around, it seems to me that in
>> > Python we already have a well-established meaning for "contexts" --
>> > context managers, and the protocols they implement as they participate
>> > in `with` statements.  We have contextlib which reinforces this.  What's
>> > being proposed in PEP 550 is so far removed from this concept that I
>> > think it's just going to cause confusion (well, it does in me anyway!).
>>
>> While I understand the concern, I think context locals and contextlib
>> are more closely related than folks realise, as one of the main
>> problems that the PEP is aiming to solve is that with statements (and
>> hence context managers) *do not work as expected* when their body
>> includes "yield", "yield from" or "await" .
>
> If I write:
>
> def read_chunks(fn, chunk_size=8192):
> with open(fn, "rb") as f:
> while True:
> data = f.read(chunk_size)
> if not data:
>break
> yield data
>
> The "with" statement here works fine even though its body includes a
> "yield" (and if there had been an added "await" things would probably
> not be different).
>
> The class of context managers you're talking about is in my experience a
> small minority (I've hardly ever used them myself, and I don't think I
> have ever written one).

I actually agree with this, as the vast majority of context managers
are about managing the state of frame locals, rather than messing
about with implicitly shared state. It's similar to the way that
thread locals are vastly outnumbered by ordinary frame locals.

That's part of what motivated my suggested distinction between
explicit context (such as your example here) and implicit context (the
trickier cases that PEP 550 aims to help handle).

>  So I don't think the two concepts are as
> closely related as you seem to think.

Of the 12 examples in
https://www.python.org/dev/peps/pep-0343/#examples, two of them
related to manipulating the decimal thread local context, and a third
relates to manipulating a hidden "queue signals for later or process
them immediately?" flag, so the use cases that PEP 550 covers have
always been an intended subset of the use cases the PEP 343 covers.

It's just that the explicit use cases either already work sensibly in
the face of frame suspension (e.g. keeping a file open, since you're
still using it), or have other reasons why you wouldn't want to
suspend the frame after entering that particular context (e.g. if you
suspend a frame with a lock held, there's no real way for the
interpreter to guess whether that's intentional or not, so it has to
assume keeping it locked is intentional, and expect you to release it
explicitly if that's what you want)

And while PEP 550 doesn't handle the stream redirection case natively
(since it doesn't allow for suspend/resume callbacks the way PEP 525
does), it at least allows for the development of a context-aware
output stream wrapper API where:

* you replace the target stream globally with a context-aware wrapper
that delegates attribute access to a particular context local if
that's set and to the original stream otherwise
* you have a context manager that sets & reverts the context local
variable rather than manipulating the process global state directly

> That said, I also think "context" is the best term (barring
> "environment" perhaps) to describe what PEP 550 is talking about.
> Qualifying it ("logical", etc.) helps disambiguate

+1

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Scope, not context? (was Re: PEP 550 v3 naming)

[Antoine Pitrou]

On Fri, 25 Aug 2017 15:36:55 +1000
Nick Coghlan  wrote:
> On 24 August 2017 at 23:52, Barry Warsaw  wrote:
> > Guido van Rossum wrote:  
> >> On Tue, Aug 22, 2017 at 7:12 PM, Nathaniel Smith  wrote:
> >>
> >> I worry that that's going to lead more people astray thinking this has
> >> something to do with contextlib, which it really doesn't (it's much more
> >> closely related to threading.local()).  
> >
> > This is my problem with using "Context" for this PEP.  Although I can't
> > keep up with all names being thrown around, it seems to me that in
> > Python we already have a well-established meaning for "contexts" --
> > context managers, and the protocols they implement as they participate
> > in `with` statements.  We have contextlib which reinforces this.  What's
> > being proposed in PEP 550 is so far removed from this concept that I
> > think it's just going to cause confusion (well, it does in me anyway!).  
> 
> While I understand the concern, I think context locals and contextlib
> are more closely related than folks realise, as one of the main
> problems that the PEP is aiming to solve is that with statements (and
> hence context managers) *do not work as expected* when their body
> includes "yield", "yield from" or "await" .

If I write:

def read_chunks(fn, chunk_size=8192):
with open(fn, "rb") as f:
while True:
data = f.read(chunk_size)
if not data:
   break
yield data

The "with" statement here works fine even though its body includes a
"yield" (and if there had been an added "await" things would probably
not be different).

The class of context managers you're talking about is in my experience a
small minority (I've hardly ever used them myself, and I don't think I
have ever written one).  So I don't think the two concepts are as
closely related as you seem to think.

That said, I also think "context" is the best term (barring
"environment" perhaps) to describe what PEP 550 is talking about.
Qualifying it ("logical", etc.) helps disambiguate

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com