Re: [Python-Dev] PEP 562

[Serhiy Storchaka]

14.11.17 22:34, Ivan Levkivskyi пише:

This function will be called only if ``name`` is not found in the module
through the normal attribute lookup.


It is worth to mention that using name as a module global will bypass 
__getattr__. And this is intentional, otherwise calling __getattr__ for 
builtins will harm a performance.



Backwards compatibility and impact on performance
=


What is affect on pydoc, word completion, inspect, pkgutil, unittest?


   def keep_pickleable(func):
       func.__name__ = func.__name__.replace('_deprecated_', '')
       func.__qualname__ = func.__qualname__.replace('_deprecated_', '')
       return func

   @keep_pickleable
   def _deprecated_old_function(arg, other):
       ...


I would create more standard helpers (for deprecation, for lazy 
importing). This feature is helpful not by itself, but because it will 
be used for implementing new features. Using __getattr__ directly will 
need to write a boilerplate code. Maybe when implementing these helper 
you will discover that this PEP needs some additions.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 560

[Nick Coghlan]

On 15 November 2017 at 06:26, Ivan Levkivskyi  wrote:

> After some discussion on python-ideas, see https://mail.python.org/
> pipermail/python-ideas/2017-September/047220.html, this PEP received
> positive comments. The updated version that takes into account the comments
> that appeared in the discussion so far is available at
> https://www.python.org/dev/peps/pep-0560/
>

I don't have anything to add to the python-ideas comments you already
incorporated, so +1 for this version from me.

* ``importlib.reload(typing)`` is up to 7x faster
>

Nice! That's getting much closer to the "negligible" range, even for
command line apps.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Eric Snow]

On Nov 7, 2017 08:12, "INADA Naoki"  wrote:

Additionally, class namespace should keep insertion order.  It's language
spec from 3.6.  So we should have two mode for such optimization.
It makes dict more complicated.


FWIW, PEP 520 (Preserving Class Attribute Definition Order) originally
specified leaving the class namespace alone.  Instead, the default class
*definition* namespace was changed to OrderedDict, and the ordering from
that namespace was stored as a tuple of names in a new __definition_order__
attribute on classes.  That approach effectively decoupled the final class
namespace from the proposed feature.

If it's an issue now then we might consider reviving __definition_order__
(which, as a bonus, has other minor benefits).  However, I expect we will
make the current dict implementation's behavior official, which renders any
changes unnecessary.

-eric
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Armin Rigo]

Hi Antoine,

On 8 November 2017 at 10:28, Antoine Pitrou  wrote:
> Yet, PyPy has no reference counting, and it doesn't seem to be a cause
> of concern.  Broken code is fixed along the way, when people notice.

It is a major cause of concern.  This is the main blocker for
pure-Python code compatibility between CPython and all other
implementations of Python, but most of the Python community plays nice
and says since long ago "don't do that".   As a result,
nowadays, people generally know better than rely on deterministic
__del__, and the language evolution usually cares not to add new
dependencies on that.  The problem is mostly confined to some
pre-existing large code bases (like OpenStack), where no good solution
exists.

It's roughly OK to have one big blocker that people need to know
about.  I don't think it's anywhere close to OK to have a miriad of
small ones.  PyPy has worked very hard to get where it is now, and
continues to regularly "fix" very obscure compatibility issues where
CPython's behaviour differs from its own documentation---by copying
the CPython actual behaviour, of course.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 562

[Ivan Levkivskyi]

After discussion on python-ideas, it looks this PEP moves towards a
favorable decision. For a recent discussion see
https://mail.python.org/pipermail/python-ideas/2017-November/047806.html.
The PEP is available at https://www.python.org/dev/peps/pep-0562/
The most important recent change is the addition of __dir__, as proposed by
Guido.

Here is the full text:

+

PEP: 562
Title: Module __getattr__ and __dir__
Author: Ivan Levkivskyi 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 09-Sep-2017
Python-Version: 3.7
Post-History: 09-Sep-2017


Abstract


It is proposed to support a ``__getattr__`` and ``__dir__`` functions
defined
+on modules to provide basic customization of module attribute access.



Rationale
=

It is sometimes convenient to customize or otherwise have control over
access to module attributes. A typical example is managing deprecation
warnings. Typical workarounds are assigning ``__class__`` of a module object
to a custom subclass of ``types.ModuleType`` or replacing the
``sys.modules``
item with a custom wrapper instance. It would be convenient to simplify this
procedure by recognizing ``__getattr__`` defined directly in a module that
would act like a normal ``__getattr__`` method, except that it will be
defined
on module *instances*. For example::

  # lib.py

  from warnings import warn

  deprecated_names = ["old_function", ...]

  def _deprecated_old_function(arg, other):
  ...

  def __getattr__(name):
  if name in deprecated_names:
  warn(f"{name} is deprecated", DeprecationWarning)
  return globals()[f"_deprecated_{name}"]
  raise AttributeError(f"module {__name__} has no attribute {name}")

  # main.py

  from lib import old_function  # Works, but emits the warning

Another widespread use case for ``__getattr__`` would be lazy submodule
imports. Consider a simple example::

  # lib/__init__.py

  import importlib

  __all__ = ['submod', ...]

  def __getattr__(name):
  if name in __all__:
  return importlib.import_module("." + name, __name__)
  raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

  # lib/submod.py

  print("Submodule loaded")
  class HeavyClass:
  ...

  # main.py

  import lib
  lib.submodule.HeavyClass  # prints "Submodule loaded"

There is a related proposal PEP 549 that proposes to support instance
properties for a similar functionality. The difference is this PEP proposes
a faster and simpler mechanism, but provides more basic customization.
An additional motivation for this proposal is that PEP 484 already defines
the use of module ``__getattr__`` for this purpose in Python stub files,
see [1]_.

In addition, to allow modifying result of a ``dir()`` call on a module
to show deprecated and other dynamically generated attributes, it is
proposed to support module level ``__dir__`` function. For example::

  # lib.py

  deprecated_names = ["old_function", ...]
  __all__ = ["new_function_one", "new_function_two", ...]

  def new_function_one(arg, other):
 ...
  def new_function_two(arg, other):
  ...

  def __dir__():
  return sorted(__all__ + deprecated_names)

  # main.py

  import lib

  dir(lib)  # prints ["new_function_one", "new_function_two",
"old_function", ...]


Specification
=

The ``__getattr__`` function at the module level should accept one argument
which is the name of an attribute and return the computed value or raise
an ``AttributeError``::

  def __getattr__(name: str) -> Any: ...

This function will be called only if ``name`` is not found in the module
through the normal attribute lookup.

The ``__dir__`` function should accept no arguments, and return
a list of strings that represents the names accessible on module::

  def __dir__() -> List[str]: ...

If present, this function overrides the standard ``dir()`` search on
a module.

The reference implementation for this PEP can be found in [2]_.


Backwards compatibility and impact on performance
=

This PEP may break code that uses module level (global) names
``__getattr__``
and ``__dir__``. The performance implications of this PEP are minimal,
since ``__getattr__`` is called only for missing attributes.


Discussion
==

Note that the use of module ``__getattr__`` requires care to keep the
referred
objects pickleable. For example, the ``__name__`` attribute of a function
should correspond to the name with which it is accessible via
``__getattr__``::

  def keep_pickleable(func):
  func.__name__ = func.__name__.replace('_deprecated_', '')
  func.__qualname__ = func.__qualname__.replace('_deprecated_', '')
  return func

  @keep_pickleable
  def _deprecated_old_function(arg, other):
  ...

One should be also careful to avoid recursion as one would do with
a class level ``__getattr__``.


References
==

.. [1] PEP 484 section about ``__getattr__`` 

[Python-Dev] PEP 560

[Ivan Levkivskyi]

After some discussion on python-ideas, see
https://mail.python.org/pipermail/python-ideas/2017-September/047220.html,
this PEP received positive comments. The updated version that takes into
account the comments that appeared in the discussion so far is available at
https://www.python.org/dev/peps/pep-0560/

Here I post the full text for convenience:

++

PEP: 560
Title: Core support for typing module and generic types
Author: Ivan Levkivskyi 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 03-Sep-2017
Python-Version: 3.7
Post-History: 09-Sep-2017


Abstract


Initially PEP 484 was designed in such way that it would not introduce
*any* changes to the core CPython interpreter. Now type hints and
the ``typing`` module are extensively used by the community, e.g. PEP 526
and PEP 557 extend the usage of type hints, and the backport of ``typing``
on PyPI has 1M downloads/month. Therefore, this restriction can be removed.
It is proposed to add two special methods ``__class_getitem__`` and
``__mro_entries__`` to the core CPython for better support of
generic types.


Rationale
=

The restriction to not modify the core CPython interpreter led to some
design decisions that became questionable when the ``typing`` module started
to be widely used. There are three main points of concern:
performance of the ``typing`` module, metaclass conflicts, and the large
number of hacks currently used in ``typing``.


Performance
---

The ``typing`` module is one of the heaviest and slowest modules in
the standard library even with all the optimizations made. Mainly this is
because of subscripted generic types (see PEP 484 for definition of terms
used in this PEP) are class objects (see also [1]_). The three main ways how
the performance can be improved with the help of the proposed special
methods:

- Creation of generic classes is slow since the ``GenericMeta.__new__`` is
  very slow; we will not need it anymore.

- Very long MROs for generic classes will be twice shorter; they are present
  because we duplicate the ``collections.abc`` inheritance chain
  in ``typing``.

- Time of instantiation of generic classes will be improved
  (this is minor however).


Metaclass conflicts
---

All generic types are instances of ``GenericMeta``, so if a user uses
a custom metaclass, then it is hard to make a corresponding class generic.
This is particularly hard for library classes that a user doesn't control.
A workaround is to always mix-in ``GenericMeta``::

  class AdHocMeta(GenericMeta, LibraryMeta):
  pass

  class UserClass(LibraryBase, Generic[T], metaclass=AdHocMeta):
  ...

but this is not always practical or even possible. With the help of the
proposed special attributes the ``GenericMeta`` metaclass will not be
needed.


Hacks and bugs that will be removed by this proposal


- ``_generic_new`` hack that exists because ``__init__`` is not called on
  instances with a type differing form the type whose ``__new__`` was
called,
  ``C[int]().__class__ is C``.

- ``_next_in_mro`` speed hack will be not necessary since subscription will
  not create new classes.

- Ugly ``sys._getframe`` hack. This one is particularly nasty since it looks
  like we can't remove it without changes outside ``typing``.

- Currently generics do dangerous things with private ABC caches
  to fix large memory consumption that grows at least as O(N\ :sup:`2`),
  see [2]_. This point is also important because it was recently proposed to
  re-implement ``ABCMeta`` in C.

- Problems with sharing attributes between subscripted generics,
  see [3]_. The current solution already uses ``__getattr__`` and
``__setattr__``,
  but it is still incomplete, and solving this without the current proposal
  will be hard and will need ``__getattribute__``.

- ``_no_slots_copy`` hack, where we clean up the class dictionary on every
  subscription thus allowing generics with ``__slots__``.

- General complexity of the ``typing`` module. The new proposal will not
  only allow to remove the above mentioned hacks/bugs, but also simplify
  the implementation, so that it will be easier to maintain.


Specification
=

``__class_getitem__``
-

The idea of ``__class_getitem__`` is simple: it is an exact analog of
``__getitem__`` with an exception that it is called on a class that
defines it, not on its instances. This allows us to avoid
``GenericMeta.__getitem__`` for things like ``Iterable[int]``.
The ``__class_getitem__`` is automatically a class method and
does not require ``@classmethod`` decorator (similar to
``__init_subclass__``) and is inherited like normal attributes.
For example::

  class MyList:
  def __getitem__(self, index):
  return index + 1
  def __class_getitem__(cls, item):
  return f"{cls.__name__}[{item.__name__}]"

  class MyOtherList(MyList):
  

Re: [Python-Dev] PEP 565: Show DeprecationWarning in __main__

[brent bejot]

On Mon, Nov 13, 2017 at 11:33 AM, Victor Stinner 
wrote:

> If the Python REPL is included in the "run an application" use case,
> the frontier between user and developer becomes blurry :-) Is REPL
> designed for users or developers? Should Python guess the intent of
> the human connected to the keyboard? ...
>
> Victor
>

I don't think folks are counting the Python REPL as "run an application".
People who use the REPL are at least writing python and should definitely
be informed of deprecations.  +1 for deprecations in the REPL from me; I
think it's a great way to inform a good percentage of the python devs of
upcoming changes.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python possible vulnerabilities in concurrency

[Jan Claeys]

On Tue, 2017-11-14 at 13:15 +0100, Antoine Pitrou wrote:
> On Mon, 13 Nov 2017 15:55:03 -0500
> Stephen Michell  wrote:
> > I am looking for one or two experts to discuss with me how Python
> > concurrency features fit together, and possible vulnerabilities
> > associated with that.
> > 
> > TR 24772 lists 5 vulnerabilities associated with 
> 
> Can you explain what "TR 24772" is?  (and/or give a link to a
> publicly-available resource)

Sounds like https://www.iso.org/standard/71094.html
which is updating https://www.iso.org/standard/61457.html
(which you can download from there if you search a bit; clearly either
ISO doesn't have a UI/UX "standard" or they aren't following it...)


-- 
Jan Claeys

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python possible vulnerabilities in concurrency

[Antoine Pitrou]

Hi Stephen,

On Mon, 13 Nov 2017 15:55:03 -0500
Stephen Michell  wrote:
> I am looking for one or two experts to discuss with me how Python concurrency 
> features fit together, and possible vulnerabilities associated with that.
> 
> TR 24772 lists 5 vulnerabilities associated with 

Can you explain what "TR 24772" is?  (and/or give a link to a
publicly-available resource)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 561 rework

[Ethan Smith]

A note was added [1] about the solution for module only distributions and
is live on Python.org.

[1] https://github.com/python/peps/pull/468

Ethan Smith

On Tue, Nov 14, 2017 at 1:02 AM, Sebastian Rittau 
wrote:

> Am 14.11.2017 um 02:38 schrieb Guido van Rossum:
>
> On Mon, Nov 13, 2017 at 3:50 PM, Sebastian Rittau 
> wrote:
>
>>
>
>> I am really looking forward to the implementation of this PEP and I am
>> glad that it is close to acceptance. One thing that is not really clear to
>> me is how module-only packages are handled. Say I have a package "foo" that
>> installs the file "foo.py" to site-packages, where would I install
>> "foo.pyi" and py.typed to? Or is this case not supported and I have to
>> convert the foo module into a package containing just __init__.py?
>>
>
> Good call. I think that conversion to a package is indeed the best
> approach -- it doesn't seem worth it to add more special-casing for this
> scenario.
>
> Ethan, if you agree, you should just add a sentence about this to the PEP.
>
> This seems like the best solution, especially since setuptools does not
> really support installing package data for pure modules.
>
>  - Sebastian
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> ethan%40ethanhs.me
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 561 rework

[Sebastian Rittau]

Am 14.11.2017 um 02:38 schrieb Guido van Rossum:
On Mon, Nov 13, 2017 at 3:50 PM, Sebastian Rittau > wrote:


I am really looking forward to the implementation of this PEP and
I am glad that it is close to acceptance. One thing that is not
really clear to me is how module-only packages are handled. Say I
have a package "foo" that installs the file "foo.py" to
site-packages, where would I install "foo.pyi" and py.typed to? Or
is this case not supported and I have to convert the foo module
into a package containing just __init__.py?


Good call. I think that conversion to a package is indeed the best 
approach -- it doesn't seem worth it to add more special-casing for 
this scenario.


Ethan, if you agree, you should just add a sentence about this to the PEP.

This seems like the best solution, especially since setuptools does not 
really support installing package data for pure modules.


 - Sebastian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com