Re: [Python-Dev] PEP 526 ready for review: Syntax for Variable and Attribute Annotations

[Nick Coghlan]

On 2 September 2016 at 02:21, Steven D'Aprano  wrote:
> Unless I've missed something, there's no way to pre-declare an instance
> attribute without specifying a type. (Even if that type is Any.) So how
> about we allow None as a type-hint on its own:
>
> NAME: None

None already has a meaning as an annotation - it's a shorthand for "type(None)".

While for variables and parameters, that's usually only seen in
combination with Union, and even though Union[T, None] has a preferred
spelling as Optional[T], there's also the "-> None" case to specify
that a function doesn't return a value. Having "-> None" mean "no
return value" and "NAME: None" mean "infer type from later assignment"
would be quite confusing.

However, a standalone Ellipsis doesn't currently have a meaning as a
type annotation (it's only meaningful when subscripting Tuple and
Callable), so a spelling like this might work:

NAME: ...

That spelling could then also be used in function definitions to say
"infer the return type from the return statements rather than assuming
Any":

def inferred_return_type(): -> ...
return some_other_function()

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Steve Dower]

My original plan was to bypass the utf8 encoding step, but that was going to 
cause major issues with code that blindly assumes it can do things like 
sys.stdout.buffer.write(b"\n") (rather than b"\n\0" - and who'd imagine you 
needed to do that). I didn't want to set up secret handshakes either, at least 
until there's a proven performance issue.

I'd need to test to be sure, but writing an incomplete code point should just 
truncate to before that point. It may currently raise OSError if that truncated 
to zero length, as I believe that's not currently distinguished from an error. 
What behavior would you propose?

Reads of less than four bytes fail instantly, as in the worst case we need four 
bytes to represent one Unicode character. This is an unfortunate reality of 
trying to limit it to one system call - you'll never get a full buffer from a 
single read, as there is no simple mapping between length-as-utf8 and 
length-as-utf16 for an arbitrary string.

Top-posted from my Windows Phone

-Original Message-
From: "Random832" 
Sent: ‎9/‎1/‎2016 16:31
To: "python-dev@python.org" 
Subject: Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

On Thu, Sep 1, 2016, at 18:28, Steve Dower wrote:
> This is a raw (bytes) IO class that requires text to be passed encoded
> with utf-8, which will be decoded to utf-16-le and passed to the Windows APIs.
> Similarly, bytes read from the class will be provided by the operating 
> system as utf-16-le and converted into utf-8 when returned to Python.

What happens if a character is broken across a buffer boundary? e.g. if
someone tries to read or write one byte at a time (you can't do a
partial read of zero bytes, there's no way to distinguish that from an
EOF.)

Is there going to be a higher-level text I/O class that bypasses the
UTF-8 encoding step when the underlying bytes stream is a console? What
if we did that but left the encoding as mbcs? I.e. the console is text
stream that can magically handle characters that aren't representable in
its encoding. Note that if anything does os.read/write to the console's
file descriptors, they're gonna get MBCS and there's nothing we can do
about it.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Random832]

On Thu, Sep 1, 2016, at 18:28, Steve Dower wrote:
> This is a raw (bytes) IO class that requires text to be passed encoded
> with utf-8, which will be decoded to utf-16-le and passed to the Windows APIs.
> Similarly, bytes read from the class will be provided by the operating 
> system as utf-16-le and converted into utf-8 when returned to Python.

What happens if a character is broken across a buffer boundary? e.g. if
someone tries to read or write one byte at a time (you can't do a
partial read of zero bytes, there's no way to distinguish that from an
EOF.)

Is there going to be a higher-level text I/O class that bypasses the
UTF-8 encoding step when the underlying bytes stream is a console? What
if we did that but left the encoding as mbcs? I.e. the console is text
stream that can magically handle characters that aren't representable in
its encoding. Note that if anything does os.read/write to the console's
file descriptors, they're gonna get MBCS and there's nothing we can do
about it.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Victor Stinner]

2016-09-02 0:04 GMT+02:00 Ethan Furman :
> - `fromord` to replace the mistaken purpose of the default constructor

To replace a bogus bytes(obj)? If someone writes bytes(obj) but expect
to create a byte string from an integer, why not using bchr() to fix
the code?

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 525, third round, better finalization

[Yury Selivanov]

Hi,

I've spent quite a while thinking and experimenting with PEP 525 trying 
to figure out how to make asynchronous generators (AG) finalization 
reliable.  I've tried to replace the callback for GCed with a callback 
to intercept first iteration of AGs.  Turns out it's very hard to work 
with weak-refs and make asyncio event loop to reliably track and 
shutdown all open AGs.


My new approach is to replace the 
"sys.set_asyncgen_finalizer(finalizer)" function with 
"sys.set_asyncgen_hooks(firstiter=None, finalizer=None)".


This design allows us to:

1. intercept first iteration of an AG.  That makes it possible for event 
loops to keep a weak set of all "open" AGs, and to implement a 
"shutdown" method to close the loop and close all AGs *reliably*.


2. intercept AGs GC.  That makes it possible to call "aclose" on GCed 
AGs to guarantee that 'finally' and 'async with' statements are properly 
closed.


3. in later Python versions we can add more hooks, although I can't 
think of anything else we need to add right now.


I'm posting below the only updated PEP section. The latest PEP revision 
should also be available on python.org shortly.


All new proposed changes are available to play with in my fork of 
CPython here: https://github.com/1st1/cpython/tree/async_gen



Finalization



PEP 492 requires an event loop or a scheduler to run coroutines.
Because asynchronous generators are meant to be used from coroutines,
they also require an event loop to run and finalize them.

Asynchronous generators can have ``try..finally`` blocks, as well as
``async with``.  It is important to provide a guarantee that, even
when partially iterated, and then garbage collected, generators can
be safely finalized.  For example::

async def square_series(con, to):
async with con.transaction():
cursor = con.cursor(
'SELECT generate_series(0, $1) AS i', to)
async for row in cursor:
yield row['i'] ** 2

async for i in square_series(con, 1000):
if i == 100:
break

The above code defines an asynchronous generator that uses
``async with`` to iterate over a database cursor in a transaction.
The generator is then iterated over with ``async for``, which interrupts
the iteration at some point.

The ``square_series()`` generator will then be garbage collected,
and without a mechanism to asynchronously close the generator, Python
interpreter would not be able to do anything.

To solve this problem we propose to do the following:

1. Implement an ``aclose`` method on asynchronous generators
   returning a special *awaitable*.  When awaited it
   throws a ``GeneratorExit`` into the suspended generator and
   iterates over it until either a ``GeneratorExit`` or
   a ``StopAsyncIteration`` occur.

   This is very similar to what the ``close()`` method does to regular
   Python generators, except that an event loop is required to execute
   ``aclose()``.

2. Raise a ``RuntimeError``, when an asynchronous generator executes
   a ``yield`` expression in its ``finally`` block (using ``await``
   is fine, though)::

async def gen():
try:
yield
finally:
await asyncio.sleep(1)   # Can use 'await'.

yield# Cannot use 'yield',
 # this line will trigger a
 # RuntimeError.

3. Add two new methods to the ``sys`` module:
   ``set_asyncgen_hooks()`` and ``get_asyncgen_hooks()``.

The idea behind ``sys.set_asyncgen_hooks()`` is to allow event
loops to intercept asynchronous generators iteration and finalization,
so that the end user does not need to care about the finalization
problem, and everything just works.

``sys.set_asyncgen_hooks()`` accepts two arguments:

* ``firstiter``: a callable which will be called when an asynchronous
  generator is iterated for the first time.

* ``finalizer``: a callable which will be called when an asynchronous
  generator is about to be GCed.

When an asynchronous generator is iterated for the first time,
it stores a reference to the current finalizer.  If there is none,
a ``RuntimeError`` is raised.  This provides a strong guarantee that
every asynchronous generator object will always have a finalizer
installed by the correct event loop.

When an asynchronous generator is about to be garbage collected,
it calls its cached finalizer.  The assumption is that the finalizer
will schedule an ``aclose()`` call with the loop that was active
when the iteration started.

For instance, here is how asyncio is modified to allow safe
finalization of asynchronous generators::

   # asyncio/base_events.py

   class BaseEventLoop:

   def run_forever(self):
   ...
   old_hooks = sys.get_asyncgen_hooks()
sys.set_asyncgen_hooks(finalizer=self._finalize_asyncgen)
   try:
   ...
   finally:
   

[Python-Dev] PEP 529: Change Windows filesystem encoding to UTF-8

[Steve Dower]

I'm about to be offline for a few days, so I wanted to get my current 
draft PEPs out for people can read and review.


I don't believe there is a lot of change as a result of either PEP, but 
the impact of what change there is needs to be weighed against the benefits.


We've already had some thorough discussion on this one and failed to 
reach agreement on whether we can make this change in 3.6 or if it needs 
a deprecation cycle that is more visible than the one we started in 3.3. 
In the latter case, we need to determine how visible that should be 
(i.e. warnings visible by default, visible for non-Windows platforms, 
value-dependent warnings/errors, etc.). IMHO, the argument about having 
the change be on-by-default or off-by-default is irrelevant until we 
decide on the deprecation issue, at which point it is obvious what the 
default should be.


See https://bugs.python.org/issue27781 for the current proposed patch. I 
do need to update it in order to merge against default it seems (work 
for my upcoming flight).


Cheers,
Steve


---
https://github.com/python/peps/blob/master/pep-0529.txt
---

PEP: 529
Title: Change Windows filesystem encoding to UTF-8
Version: $Revision$
Last-Modified: $Date$
Author: Steve Dower 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Aug-2016
Post-History: 01-Sep-2016

Abstract


Historically, Python uses the ANSI APIs for interacting with the Windows
operating system, often via C Runtime functions. However, these have 
been long
discouraged in favor of the UTF-16 APIs. Within the operating system, 
all text
is represented as UTF-16, and the ANSI APIs perform encoding and 
decoding using

the active code page.

This PEP proposes changing the default filesystem encoding on Windows to 
utf-8,

and changing all filesystem functions to use the Unicode APIs for filesystem
paths. This will not affect code that uses strings to represent paths, 
however

those that use bytes for paths will now be able to correctly round-trip all
valid paths in Windows filesystems. Currently, the conversions between 
Unicode

(in the OS) and bytes (in Python) were lossy and would fail to round-trip
characters outside of the user's active code page.

Notably, this does not impact the encoding of the contents of files. 
These will

continue to default to locale.getpreferredencoding (for text files) or plain
bytes (for binary files). This only affects the encoding used when users 
pass a
bytes object to Python where it is then passed to the operating system 
as a path

name.

Background
==

File system paths are almost universally represented as text with an 
encoding
determined by the file system. In Python, we expose these paths via a 
number of
interfaces, such as the ``os`` and ``io`` modules. Paths may be passed 
either

direction across these interfaces, that is, from the filesystem to the
application (for example, ``os.listdir()``), or from the application to the
filesystem (for example, ``os.unlink()``).

When paths are passed between the filesystem and the application, they are
either passed through as a bytes blob or converted to/from str using
``os.fsencode()`` or ``sys.getfilesystemencoding()``. The result of 
encoding a

string with ``sys.getfilesystemencoding()`` is a blob of bytes in the native
format for the default file system.

On Windows, the native format for the filesystem is utf-16-le. The 
recommended
platform APIs for accessing the filesystem all accept and return text 
encoded in

this format. However, prior to Windows NT (and possibly further back), the
native format was a configurable machine option and a separate set of APIs
existed to accept this format. The option (the "active code page") and these
APIs (the "*A functions") still exist in recent versions of Windows for
backwards compatibility, though new functionality often only has a 
utf-16-le API

(the "*W functions").

In Python, str is recommended because it can correctly round-trip all 
characters
used in paths (on POSIX with surrogateescape handling; on Windows 
because str

maps to the native representation). On Windows bytes cannot round-trip all
characters used in paths, as Python internally uses the *A functions and 
hence
the encoding is "whatever the active code page is". Since the active 
code page
cannot represent all Unicode characters, the conversion of a path into 
bytes can

lose information without warning or any available indication.

As a demonstration of this::
>>> open('test\uAB00.txt', 'wb').close()
>>> import glob
>>> glob.glob('test*')
['test\uab00.txt']
>>> glob.glob(b'test*')
[b'test?.txt']

The Unicode character in the second call to glob has been replaced by a '?',
which means passing the path back into the filesystem will result in a
``FileNotFoundError``. The same results may be observed with 
``os.listdir()`` or

any function that matches the return type to the parameter type.

While one 

[Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Steve Dower]

I'm about to be offline for a few days, so I wanted to get my current 
draft PEPs out for people can read and review.


I don't believe there is a lot of change as a result of either PEP, but 
the impact of what change there is needs to be weighed against the benefits.


If anything, I'm likely to have underplayed the impact of this change 
(though I've had a *lot* of support for this one). Just stating my 
biases up-front - take it as you wish.


See https://bugs.python.org/issue1602 for the current proposed patch for 
this PEP. I will likely update it after my upcoming flights, but it's in 
pretty good shape right now.


Cheers,
Steve


---
https://github.com/python/peps/blob/master/pep-0528.txt
---

PEP: 528
Title: Change Windows console encoding to UTF-8
Version: $Revision$
Last-Modified: $Date$
Author: Steve Dower 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Aug-2016
Post-History: 01-Sep-2016

Abstract


Historically, Python uses the ANSI APIs for interacting with the Windows
operating system, often via C Runtime functions. However, these have 
been long
discouraged in favor of the UTF-16 APIs. Within the operating system, 
all text
is represented as UTF-16, and the ANSI APIs perform encoding and 
decoding using

the active code page.

This PEP proposes changing the default standard stream implementation on 
Windows
to use the Unicode APIs. This will allow users to print and input the 
full range

of Unicode characters at the default Windows console. This also requires a
subtle change to how the tokenizer parses text from readline hooks, that 
should

have no backwards compatibility issues.

Specific Changes


Add _io.WindowsConsoleIO


Currently an instance of ``_io.FileIO`` is used to wrap the file descriptors
representing standard input, output and error. We add a new class 
(implemented
in C) ``_io.WindowsConsoleIO`` that acts as a raw IO object using the 
Windows

console functions, specifically, ``ReadConsoleW`` and ``WriteConsoleW``.

This class will be used when the legacy-mode flag is not in effect, when 
opening
a standard stream by file descriptor and the stream is a console buffer 
rather
than a redirected file. Otherwise, ``_io.FileIO`` will be used as it is 
today.


This is a raw (bytes) IO class that requires text to be passed encoded with
utf-8, which will be decoded to utf-16-le and passed to the Windows APIs.
Similarly, bytes read from the class will be provided by the operating 
system as

utf-16-le and converted into utf-8 when returned to Python.

The use of an ASCII compatible encoding is required to maintain 
compatibility
with code that bypasses the ``TextIOWrapper`` and directly writes ASCII 
bytes to
the standard streams (for example, [process_stdinreader.py]_). Code that 
assumes

a particular encoding for the standard streams other than ASCII will likely
break.

Add _PyOS_WindowsConsoleReadline


To allow Unicode entry at the interactive prompt, a new readline hook is
required. The existing ``PyOS_StdioReadline`` function will delegate to 
the new
``_PyOS_WindowsConsoleReadline`` function when reading from a file 
descriptor
that is a console buffer and the legacy-mode flag is not in effect (the 
logic

should be identical to above).

Since the readline interface is required to return an 8-bit encoded 
string with
no embedded nulls, the ``_PyOS_WindowsConsoleReadline`` function 
transcodes from

utf-16-le as read from the operating system into utf-8.

The function ``PyRun_InteractiveOneObject`` which currently obtains the 
encoding
from ``sys.stdin`` will select utf-8 unless the legacy-mode flag is in 
effect.

This may require readline hooks to change their encodings to utf-8, or to
require legacy-mode for correct behaviour.

Add legacy mode
---

Launching Python with the environment variable 
``PYTHONLEGACYWINDOWSSTDIO`` set

will enable the legacy-mode flag, which completely restores the previous
behaviour.

Alternative Approaches
==

The ``win_unicode_console`` package [win_unicode_console]_ is a pure-Python
alternative to changing the default behaviour of the console.

Code that may break
===

The following code patterns may break or see different behaviour as a 
result of
this change. All of these code samples require explicitly choosing to 
use a raw
file object in place of a more convenient wrapper that would prevent any 
visible

change.

Assuming stdin/stdout encoding
--

Code that assumes that the encoding required by ``sys.stdin.buffer`` or
``sys.stdout.buffer`` is ``'mbcs'`` or a more specific encoding may 
currently be
working by chance, but could encounter issues under this change. For 
example::


sys.stdout.buffer.write(text.encode('mbcs'))
r = sys.stdin.buffer.read(16).decode('cp437')

To correct this code, the encoding specified on the 

Re: [Python-Dev] PEP 467: last round (?)

[Ethan Furman]

On 09/01/2016 02:06 PM, Victor Stinner wrote:

2016-09-01 21:36 GMT+02:00 Ethan Furman:



Abstract


This PEP proposes five small adjustments to the APIs of the ``bytes`` and
``bytearray`` types to make it easier to operate entirely in the binary
domain:


You should add bchr() in the Abstract.


Done.


* Deprecate passing single integer values to ``bytes`` and ``bytearray``
* Add ``bytes.fromsize`` and ``bytearray.fromsize`` alternative constructors


I understand that main reason for this change is to catch bugs when
bytes(obj) is used and obj is not supposed to be an integer.

So I expect that bytes(int) will be quickly deprecated, but the PEP
doesn't schedule a removal of the feature. So it looks more than only
adding an alias to bytes(int).

I would prefer to either schedule a removal of bytes(int), or remove
bytes.fromsize() from the PEP.


The PEP states that ``bytes(x)`` will not be removed while 2.7 is supported.  
Once 2.7 is no longer a concern we can visit the question of removing that 
behavior.


* Add ``bytes.fromord`` and ``bytearray.fromord`` alternative constructors


Hum, you already propose to add a builtin function. Why would we need
two ways to create a single byte?


- `bchr` to mirror `chr`
- `fromord` to replace the mistaken purpose of the default constructor


* Add ``bytes.getbyte`` and ``bytearray.getbyte`` byte retrieval methods
* Add ``bytes.iterbytes`` and ``bytearray.iterbytes`` alternative iterators


I like these ones :-)


Cool.


In particular, there's a reasonable case to be made
that ``bytes(x)`` (where ``x`` is an integer) should behave like the
``bytes.fromint(x)`` proposal in this PEP.


"fromint"? Is it bytes.fromord()/bchr()?


Oops, fixed.

--
~Ethan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Patch reviews

[Victor Stinner]

2016-08-31 22:31 GMT+02:00 Christian Heimes :
> https://bugs.python.org/issue27744
> Add AF_ALG (Linux Kernel crypto) to socket module

This patch adds a new socket.sendmsg_afalg() method on Linux.

"afalg" comes from AF_ALG which means "Address Family Algorithm". It's
documented as "af_alg: User-space algorithm interface" in
crypto/af_alg.c.

IHMO the method should be just "sendmsg_alg()", beacuse "afalg" is
redundant. The AF_ prefix is only used to workaround a C limitation:
there is no namespace in the language, all symbols are in one single
giant namespace.

I don't expect that a platform will add a new sendmsg_alg() C
function. If it's the case, we will see how to handle the name
conflict ;-)

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Victor Stinner]

2016-09-01 21:36 GMT+02:00 Ethan Furman :
> Abstract
> 
>
> This PEP proposes five small adjustments to the APIs of the ``bytes`` and
> ``bytearray`` types to make it easier to operate entirely in the binary
> domain:

You should add bchr() in the Abstract.


> * Deprecate passing single integer values to ``bytes`` and ``bytearray``
> * Add ``bytes.fromsize`` and ``bytearray.fromsize`` alternative constructors

I understand that main reason for this change is to catch bugs when
bytes(obj) is used and obj is not supposed to be an integer.

So I expect that bytes(int) will be quickly deprecated, but the PEP
doesn't schedule a removal of the feature. So it looks more than only
adding an alias to bytes(int).

I would prefer to either schedule a removal of bytes(int), or remove
bytes.fromsize() from the PEP.


> * Add ``bytes.fromord`` and ``bytearray.fromord`` alternative constructors

Hum, you already propose to add a builtin function. Why would we need
two ways to create a single byte?

I'm talking about bchr(int)==bytes.fromord(int). I'm not sure that
there is an use case for bytearray.fromord(int).


> * Add ``bytes.getbyte`` and ``bytearray.getbyte`` byte retrieval methods
> * Add ``bytes.iterbytes`` and ``bytearray.iterbytes`` alternative iterators

I like these ones :-)



> In particular, there's a reasonable case to be made
> that ``bytes(x)`` (where ``x`` is an integer) should behave like the
> ``bytes.fromint(x)`` proposal in this PEP.

"fromint"? Is it bytes.fromord()/bchr()?


> Open Questions
> ==
>
> Do we add ``iterbytes`` to ``memoryview``, or modify
> ``memoryview.cast()`` to accept ``'s'`` as a single-byte interpretation?  Or
> do we ignore memory for now and add it later?

It's nice to have bytes.iterbytes() to help porting Python 2 code, but
I'm not sure that this function would be super popular in new Python 3
code.

I don't think that a memoryview.iterbytes() (or cast("s")) would be useful.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 526 ready for review: Syntax for Variable and Attribute Annotations

[Guido van Rossum]

On Thu, Sep 1, 2016 at 9:30 AM, Ivan Levkivskyi  wrote:
> On 1 September 2016 at 18:21, Steven D'Aprano  wrote:
[...]
>> Unless I've missed something, there's no way to pre-declare an instance
>> attribute without specifying a type. (Even if that type is Any.) So how
>> about we allow None as a type-hint on its own:
>>
>> NAME: None
>>
>> as equivalent to a declaration *without* a hint. The reader, and the
>> type-checker, can see that there's an instance attribute called NAME,
>> but in the absense of an actual hint, the type will have to be inferred,
>> just as if it wasn't declared at all.

> There is a convention for function annotations in PEP 484 that a missing
> annotation is equivalent to Any, so that I like your first option more.

But Steven wasn't proposing it to mean Any, he was proposing it to
mean "type checker should infer". Where I presume the inference should
be done based on the assignment in __init__ only.

I'm not sure if this needs special syntax (a type checker might behave
the same way without this, so we could just use a comment) but even if
we did decide we wanted to support

  NAME: None

for this case, we don't have to change Python, since this already
conforms to the syntax in PEP 526 (the type is None). We'd still have
to update the PEP to tell the authors of type checkers about this
special feature, since otherwise it would mean "NAME has type
NoneType" (remember that PEP 484 defines None as a shortcut for
NoneType == type(None)). But that's not a very useful type for a
variable...

But I'm not in a hurry for that -- I'm only hoping to get the basic
syntax accepted by Python 3.6 beta 1 so that we can start using this
in 5 years from now rather than 7 years from now.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 526 ready for review: Syntax for Variable and Attribute Annotations

[Guido van Rossum]

On Thu, Sep 1, 2016 at 10:01 AM, Koos Zevenhoven  wrote:
> On Thu, Sep 1, 2016 at 5:46 PM, Guido van Rossum  wrote:
>> IOW, PEP 3157 is not dead yet. Indeed.
>>
>
> PEP 3157? Is that a typo or is there such a thing somewhere?

Sorry, 3107 (the original Function Annotations PEP).

> [...]
> I hope there will at least be a recommendation somewhere (PEP 8?) to not
> mix the two styles of attribute annotation (beginning of class / in
> method). The whole readability benefit turns against itself if there
> are some non-ClassVar variables annotated outside __init__ and then
> the rest somewhere in __init__ and in whatever initialization helper
> methods __init__ happens to call.

Yeah, but then again, in general I don't believe you can legislate the
writing of readable code using crude syntactic means. Not mixing the
two in the same class sounds like pretty good advice though, and a
linter should be able to catch that easily.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 526 ready for review: Syntax for Variable and Attribute Annotations

[Guido van Rossum]

On Thu, Sep 1, 2016 at 10:30 AM, Steven D'Aprano  wrote:
> On Tue, Aug 30, 2016 at 02:20:26PM -0700, Guido van Rossum wrote:
>
>> - Whether (given PEP 484's relative success) it's worth adding syntax
>> for variable/attribute annotations.
>
> The PEP makes a good case that it does.

Thanks, I agree. :-)

>> - Whether the keyword-free syntax idea proposed here is best:
>>   NAME: TYPE
>>   TARGET: TYPE = VALUE
>
> I think so.
>
> That looks like similar to the syntax used by TypeScript:
>
> http://www.typescriptlang.org/docs/handbook/type-inference.html
>
> let zoo: Animal[] = [new Rhino(), new Elephant(), new Snake()];

And Rust.

In the tracker issue we're still tweaking this, e.g. the latest idea
is that after all we'd like to simplify the syntax to

TARGET: TYPE [= VALUE]

Please read the end of the tracker discussion:
https://github.com/python/typing/issues/258#issuecomment-244188268

> Some additional thoughts:
>
> Is it okay to declare something as both an instance and class attribute?
>
> class X:
> spam: int
> spam: ClassVar[Str] = 'suprise!'
>
> def __init__(self):
> self.spam = 999
>
>
> I would expect it should be okay.

I think it would be confusing because the class var would be used as
the default if the instance var is not defined.

> It is more common in Python circles to talk about class and instance
> *attributes* than "variables". Class variable might be okay in a
> language like Java where classes themselves aren't first-class values,
> but in Python "class variable" always makes me think it is talking about
> a variable which is a class, just like a string variable or list
> variable. Can we have ClassAttr[] instead of ClassVar[]?

We went back and forth on this. I really don't like to use the word
attribute here, because a method is also an attribute. And instance
variable sounds more natural to me than instance attribute.

Also we now have global variables, class variables, instance
variables, and local variables, all of which can be annotated.

(The PEP's language is actually a bit inconsistent here.)

> Other than that, +1 on the PEP.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 467: last round (?)

[Ethan Furman]

One more iteration.  PEPs repo not updated yet.  Changes are renaming of 
methods to be ``fromsize()`` and ``fromord()``, and moving ``memoryview`` to an 
Open Questions section.


PEP: 467
Title: Minor API improvements for binary sequences
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan , Ethan Furman 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 2014-03-30
Python-Version: 3.6
Post-History: 2014-03-30 2014-08-15 2014-08-16 2016-06-07 2016-09-01


Abstract


During the initial development of the Python 3 language specification, the
core ``bytes`` type for arbitrary binary data started as the mutable type
that is now referred to as ``bytearray``. Other aspects of operating in
the binary domain in Python have also evolved over the course of the Python
3 series.

This PEP proposes five small adjustments to the APIs of the ``bytes`` and
``bytearray`` types to make it easier to operate entirely in the binary domain:

* Deprecate passing single integer values to ``bytes`` and ``bytearray``
* Add ``bytes.fromsize`` and ``bytearray.fromsize`` alternative constructors
* Add ``bytes.fromord`` and ``bytearray.fromord`` alternative constructors
* Add ``bytes.getbyte`` and ``bytearray.getbyte`` byte retrieval methods
* Add ``bytes.iterbytes`` and ``bytearray.iterbytes`` alternative iterators


Proposals
=

Deprecation of current "zero-initialised sequence" behaviour without removal


Currently, the ``bytes`` and ``bytearray`` constructors accept an integer
argument and interpret it as meaning to create a zero-initialised sequence
of the given size::

>>> bytes(3)
b'\x00\x00\x00'
>>> bytearray(3)
bytearray(b'\x00\x00\x00')

This PEP proposes to deprecate that behaviour in Python 3.6, but to leave
it in place for at least as long as Python 2.7 is supported, possibly
indefinitely.

No other changes are proposed to the existing constructors.


Addition of explicit "count and byte initialised sequence" constructors
---

To replace the deprecated behaviour, this PEP proposes the addition of an
explicit ``fromsize`` alternative constructor as a class method on both
``bytes`` and ``bytearray`` whose first argument is the count, and whose
second argument is the fill byte to use (defaults to ``\x00``)::

>>> bytes.fromsize(3)
b'\x00\x00\x00'
>>> bytearray.fromsize(3)
bytearray(b'\x00\x00\x00')
>>> bytes.fromsize(5, b'\x0a')
b'\x0a\x0a\x0a\x0a\x0a'
>>> bytearray.fromsize(5, b'\x0a')
bytearray(b'\x0a\x0a\x0a\x0a\x0a')

``fromsize`` will behave just as the current constructors behave when passed a 
single
integer, while allowing for non-zero fill values when needed.


Addition of "bchr" function and explicit "single byte" constructors
---

As binary counterparts to the text ``chr`` function, this PEP proposes
the addition of a ``bchr`` function and an explicit ``fromord`` alternative
constructor as a class method on both ``bytes`` and ``bytearray``::

>>> bchr(ord("A"))
b'A'
>>> bchr(ord(b"A"))
b'A'
>>> bytes.fromord(65)
b'A'
>>> bytearray.fromord(65)
bytearray(b'A')

These methods will only accept integers in the range 0 to 255 (inclusive)::

>>> bytes.fromord(512)
Traceback (most recent call last):
  File "", line 1, in 
ValueError: integer must be in range(0, 256)

>>> bytes.fromord(1.0)
Traceback (most recent call last):
  File "", line 1, in 
TypeError: 'float' object cannot be interpreted as an integer

While this does create some duplication, there are valid reasons for it::

* the ``bchr`` builtin is to recreate the ord/chr/unichr trio from Python
  2 under a different naming scheme
* the class method is mainly for the ``bytearray.fromord`` case, with
  ``bytes.fromord`` added for consistency

The documentation of the ``ord`` builtin will be updated to explicitly note
that ``bchr`` is the primary inverse operation for binary data, while ``chr``
is the inverse operation for text data, and that ``bytes.fromord`` and
``bytearray.fromord`` also exist.

Behaviourally, ``bytes.fromord(x)`` will be equivalent to the current
``bytes([x])`` (and similarly for ``bytearray``). The new spelling is
expected to be easier to discover and easier to read (especially when used
in conjunction with indexing operations on binary sequence types).

As a separate method, the new spelling will also work better with higher
order functions like ``map``.


Addition of "getbyte" method to retrieve a single byte
--

This PEP proposes that ``bytes`` and ``bytearray`` gain the method ``getbyte``
which will always return ``bytes``::

>>> b'abc'.getbyte(0)
b'a'

If an index is asked for 

Re: [Python-Dev] PEP 526 ready for review: Syntax for Variable and Attribute Annotations

[Steven D'Aprano]

On Tue, Aug 30, 2016 at 02:20:26PM -0700, Guido van Rossum wrote:

> - Whether (given PEP 484's relative success) it's worth adding syntax
> for variable/attribute annotations.

The PEP makes a good case that it does.


> - Whether the keyword-free syntax idea proposed here is best:
>   NAME: TYPE
>   TARGET: TYPE = VALUE

I think so.

That looks like similar to the syntax used by TypeScript:

http://www.typescriptlang.org/docs/handbook/type-inference.html

let zoo: Animal[] = [new Rhino(), new Elephant(), new Snake()];


Some additional thoughts:

Is it okay to declare something as both an instance and class attribute?

class X:
spam: int
spam: ClassVar[Str] = 'suprise!'

def __init__(self):
self.spam = 999


I would expect it should be okay.



It is more common in Python circles to talk about class and instance 
*attributes* than "variables". Class variable might be okay in a 
language like Java where classes themselves aren't first-class values, 
but in Python "class variable" always makes me think it is talking about 
a variable which is a class, just like a string variable or list 
variable. Can we have ClassAttr[] instead of ClassVar[]?


Other than that, +1 on the PEP.



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 526 ready for review: Syntax for Variable and Attribute Annotations

[Koos Zevenhoven]

On Thu, Sep 1, 2016 at 5:46 PM, Guido van Rossum  wrote:
> On Thu, Sep 1, 2016 at 6:11 AM, Koos Zevenhoven  wrote:
>> While a large amount of Python programmers may not be interested in
>> type hinting local variables inside functions, I can see other
>> potential benefits in this.
>
> IOW, PEP 3157 is not dead yet. Indeed.
>

PEP 3157? Is that a typo or is there such a thing somewhere?

[...]
>> Also, when reading code, it may be hard to tell which (instance)
>> attributes the class implements. To have these listed in the beginning
>> of the class could therefore improve the readability.
>
> Right. That has been my observation using PEP 484's type comments
> extensively for annotating instance variables at the class level. E.g.
> much of mypy's own code is written this way, and it really is a huge
> help. But
>
> foo = None  # type: List[int]
>
> while it gives me the info I'm looking for, is not great
> notation-wise, and that's why I started thinking about an alternative:
>
> foo: List[int]
>
> (in either case, the __init__ contains something like `self.foo = []`).
>
>> In this light, I'm not sure it's a good idea to allow attribute type
>> hints inside methods.
>
> Those are meant for the coding style where all attributes are
> initialized in the method and people just want to add annotations
> there. This is already in heavy use in some PEP-484-annotated code
> bases I know of, using # type comments, and I think it will be easier
> to get people to switch to syntactic annotations if they can
> mechanically translate those uses. (In fact we are planning an
> automatic translator.)

I suppose the translator would be somewhat more complicated if it were
to move the type hints to the beginning of the class suite. Anyway, I
hope there will at least be a recommendation somewhere (PEP 8?) to not
mix the two styles of attribute annotation (beginning of class / in
method). The whole readability benefit turns against itself if there
are some non-ClassVar variables annotated outside __init__ and then
the rest somewhere in __init__ and in whatever initialization helper
methods __init__ happens to call.

[...]
>> Note that we could then also have this:
>>
>> def NAME
>>
>> Which would, again for readability (see above), be a way to express
>> that "there is an instance variable called X, but no type hint for
>> now". I can't think of a *good* way to do this with the keyword-free
>> version for people that don't use type hints.
>>
>> And then there could also be a simple decorator like
>> @slotted_attributes that automatically generates "__slots__" from the
>> annotations.
>
> This I like, or something like it. It can be a follow-up design. (I.e.
> a separate PEP, once we have experiece with PEP 526.)

I think there should be a syntax for this that does not involve type
hints, but I can't seem to come up with anything that works with the
keyword-free version :(.

-- Koos


> --
> --Guido van Rossum (python.org/~guido)

-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 526 ready for review: Syntax for Variable and Attribute Annotations

[Ivan Levkivskyi]

On 1 September 2016 at 18:21, Steven D'Aprano  wrote:

> The simplest way would be to say "go on, one type hint won't hurt,
> there's no meaningful runtime cost, just do it".
>
> from typing import Any
>
> class X:
> NAME: Any
>
> Since I'm not running a type checker, it doesn't matter what hint I use,
> but Any is probably the least inaccurate.
>
> But I think there's a better way.
>
> Unless I've missed something, there's no way to pre-declare an instance
> attribute without specifying a type. (Even if that type is Any.) So how
> about we allow None as a type-hint on its own:
>
> NAME: None
>
> as equivalent to a declaration *without* a hint. The reader, and the
> type-checker, can see that there's an instance attribute called NAME,
> but in the absense of an actual hint, the type will have to be inferred,
> just as if it wasn't declared at all.
>

There is a convention for function annotations in PEP 484 that a missing
annotation
is equivalent to Any, so that I like your first option more.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Patch reviews

[Christian Heimes]

On 2016-08-31 22:31, Christian Heimes wrote:
> Hi,
> 
> I have 7 patches for 3.6 ready for merging. The new features were
> discussed on Security-SIG and reviewed by Victor or GPS. The patches
> just need one final review and an ACK. The first three patches should
> land in 2.7, 3.4 and 3.5, too.
> 
> http://bugs.python.org/issue26470
> Make OpenSSL module compatible with OpenSSL 1.1.0
> 
> https://bugs.python.org/issue27850
> Remove 3DES from cipher list (sweet32 CVE-2016-2183)
> Also adds ChaCha20 Poly1305
> 
> http://bugs.python.org/issue27691
> X509 cert with GEN_RID subject alt name causes SytemError
> 
> http://bugs.python.org/issue27866
> ssl: get list of enabled ciphers
> 
> https://bugs.python.org/issue27744
> Add AF_ALG (Linux Kernel crypto) to socket module
> 
> http://bugs.python.org/issue16113
> Add SHA-3 and SHAKE (Keccak) support
> 
> http://bugs.python.org/issue26798
> add BLAKE2 to hashlib

And another one:

http://bugs.python.org/issue27928
Add hashlib.scrypt

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 526 ready for review: Syntax for Variable and Attribute Annotations

[Steven D'Aprano]

On Thu, Sep 01, 2016 at 04:11:05PM +0300, Koos Zevenhoven wrote:

> Maybe it's just me, but I've always thought 'def' is Python's least
> logically used keyword. It seems to come from 'define', but what is it
> about 'define' that makes it relate to functions only.

Convention.

You can't use "def" to define both functions and classes:

def function(x): ...
def Class(x): ...

is ambiguous, which is the function and which is the class? So we cannot 
avoid at least one limitation: "def is for functions, or classes, but 
not both". Given that, it isn't that weird to make the rule "def is only 
for functions".



[...]
> Note that we could then also have this:
> 
> def NAME
> 
> Which would, again for readability (see above), be a way to express
> that "there is an instance variable called X, but no type hint for
> now". I can't think of a *good* way to do this with the keyword-free
> version for people that don't use type hints.

The simplest way would be to say "go on, one type hint won't hurt, 
there's no meaningful runtime cost, just do it".

from typing import Any

class X:
NAME: Any

Since I'm not running a type checker, it doesn't matter what hint I use, 
but Any is probably the least inaccurate.

But I think there's a better way.

Unless I've missed something, there's no way to pre-declare an instance 
attribute without specifying a type. (Even if that type is Any.) So how 
about we allow None as a type-hint on its own:

NAME: None

as equivalent to a declaration *without* a hint. The reader, and the 
type-checker, can see that there's an instance attribute called NAME, 
but in the absense of an actual hint, the type will have to be inferred, 
just as if it wasn't declared at all.

The risk is that somebody will "helpfully" correct the "obvious typo" 
and change it to NAME = None, but I think that will usually be harmless. 
I can invent examples where they will behave differently, but they feel 
contrived to me:

class X:
spam: None  # declaration only, without a hint

def method(self):
if not hasattr(self, "spam"):
raise XError("spam not set")
return self.spam

def setup(self, arg):
if hasattr(self, "spam"):
raise XError("spam already set")
self.spam = arg


Changing the declaration to an assignment does change the behaviour of 
the class, but I think that will be obvious when it happens.


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 526 ready for review: Syntax for Variable and Attribute Annotations

[Guido van Rossum]

On Thu, Sep 1, 2016 at 6:11 AM, Koos Zevenhoven  wrote:
> While a large amount of Python programmers may not be interested in
> type hinting local variables inside functions, I can see other
> potential benefits in this.

IOW, PEP 3157 is not dead yet. Indeed.

> When I start sketching a new class, I'm often tempted to write down
> the names of the attributes first, before starting to implement
> ``__init__``. Sometimes I even write temporary comments for this
> purpose. This syntax would naturally provide a way to sketch the list
> of attributes. Yes, there is already __slots__, but I'm not sure that
> is a good example of readability.

Agreed, it can't get much cleaner than NAME: TYPE.

> Also, when reading code, it may be hard to tell which (instance)
> attributes the class implements. To have these listed in the beginning
> of the class could therefore improve the readability.

Right. That has been my observation using PEP 484's type comments
extensively for annotating instance variables at the class level. E.g.
much of mypy's own code is written this way, and it really is a huge
help. But

foo = None  # type: List[int]

while it gives me the info I'm looking for, is not great
notation-wise, and that's why I started thinking about an alternative:

foo: List[int]

(in either case, the __init__ contains something like `self.foo = []`).

> In this light, I'm not sure it's a good idea to allow attribute type
> hints inside methods.

Those are meant for the coding style where all attributes are
initialized in the method and people just want to add annotations
there. This is already in heavy use in some PEP-484-annotated code
bases I know of, using # type comments, and I think it will be easier
to get people to switch to syntactic annotations if they can
mechanically translate those uses. (In fact we are planning an
automatic translator.)

>> - Whether the keyword-free syntax idea proposed here is best:
>>   NAME: TYPE
>>   TARGET: TYPE = VALUE
>
> I wonder if this would be better:
>
> def NAME: TYPE
> def NAME: TYPE = VALUE
>
> Maybe it's just me, but I've always thought 'def' is Python's least
> logically used keyword. It seems to come from 'define', but what is it
> about 'define' that makes it relate to functions only. Adding an
> optional 'def' for other variables might even be a tiny bit of added
> consistency.

Here I strongly disagree. Everyone will be confused.

> Note that we could then also have this:
>
> def NAME
>
> Which would, again for readability (see above), be a way to express
> that "there is an instance variable called X, but no type hint for
> now". I can't think of a *good* way to do this with the keyword-free
> version for people that don't use type hints.
>
> And then there could also be a simple decorator like
> @slotted_attributes that automatically generates "__slots__" from the
> annotations.

This I like, or something like it. It can be a follow-up design. (I.e.
a separate PEP, once we have experiece with PEP 526.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 526 ready for review: Syntax for Variable and Attribute Annotations

[Koos Zevenhoven]

On Wed, Aug 31, 2016 at 12:20 AM, Guido van Rossum  wrote:
> I'm happy to present PEP 526 for your collective review:
> https://www.python.org/dev/peps/pep-0526/ (HTML)
> https://github.com/python/peps/blob/master/pep-0526.txt (source)
>
> There's also an implementation ready:
> https://github.com/ilevkivskyi/cpython/tree/pep-526
>
> I don't want to post the full text here but I encourage feedback on
> the high-order ideas, including but not limited to
>
> - Whether (given PEP 484's relative success) it's worth adding syntax
> for variable/attribute annotations.

While a large amount of Python programmers may not be interested in
type hinting local variables inside functions, I can see other
potential benefits in this.

When I start sketching a new class, I'm often tempted to write down
the names of the attributes first, before starting to implement
``__init__``. Sometimes I even write temporary comments for this
purpose. This syntax would naturally provide a way to sketch the list
of attributes. Yes, there is already __slots__, but I'm not sure that
is a good example of readability.

Also, when reading code, it may be hard to tell which (instance)
attributes the class implements. To have these listed in the beginning
of the class could therefore improve the readability.

In this light, I'm not sure it's a good idea to allow attribute type
hints inside methods.

>
> - Whether the keyword-free syntax idea proposed here is best:
>   NAME: TYPE
>   TARGET: TYPE = VALUE
>

I wonder if this would be better:

def NAME: TYPE
def NAME: TYPE = VALUE

Maybe it's just me, but I've always thought 'def' is Python's least
logically used keyword. It seems to come from 'define', but what is it
about 'define' that makes it relate to functions only. Adding an
optional 'def' for other variables might even be a tiny bit of added
consistency.

Note that we could then also have this:

def NAME

Which would, again for readability (see above), be a way to express
that "there is an instance variable called X, but no type hint for
now". I can't think of a *good* way to do this with the keyword-free
version for people that don't use type hints.

And then there could also be a simple decorator like
@slotted_attributes that automatically generates "__slots__" from the
annotations.

-- Koos


> Note that there's an extensive list of rejected ideas in the PEP;
> please be so kind to read it before posting here:
> https://www.python.org/dev/peps/pep-0526/#rejected-proposals-and-things-left-out-for-now
>
>
> --
> --Guido van Rossum (python.org/~guido)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com